I have an express route that needs to retrieve a row from a db, check if a field has a value set and if not set it. The row data then gets sent back to the client.
I know that node runs on a single thread but I/O operations do actually run asynchronously so I think I may have a problem if whilst the first client is waiting to write to db, a second client comes and reads a null value and performs the write a second time.
I can't have this happen as the value written is a shared value that there can only be one of.
Am I correct that this could happen and if so what is a recommended way to handle this?
Thanks.
let express = require('express');
let router = express.Router();
router.post('/getRoomStateByRoomUrl', async (req, res, next) => {
const roomUrl = req.body.room_url;
try {
//READ FROM DB
const roomState = await RoomStateModel.getRoomStateByRoomUrl(roomUrl);
if(!roomState.tokboxSessionID) {
const newSessionID = await TokboxService.createSession();
//WRITE TO DB
await RoomStateModel.setTokboxSessionID(newSessionID);
roomState.tokboxSessionID = newSessionID;
}
res.status(200).json(roomState);
} catch (error) {
next(error);
}
});
Related
I'm working on making an API using express and MySQL. I'm having issues with my routes. I have it in a separate folder, requiring the different controller files and exporting the router at the end. Here's the issue. The last router.get function will not work. I can swap it and whatever is last will not work. I'll get back an empty array. I'm at a loss as to what can be wrong. Here's my code, the routes file:
const express = require('express');
const router = express.Router();
const getEmployeesController = require('../controllers/getEmployees');
const getEmployeesByIdController = require('../controllers/getEmployeesById');
const getEmployeesByFirstNameController = require('../controllers/getEmployeesByFirstName');
router.get('/', getEmployeesController.getEmployees);
router.get('/:id', getEmployeesByIdController.getEmployeesById);
router.get('/:first_name', getEmployeesByFirstNameController.getEmployeesByFirstName);
module.exports = router;
The 'first_name' router worked when it was second, after the '/', but now it won't. Same with the 'id', worked when its second, but not when it's third.
Here's the controller function, one as an example:
const mysql = require('mysql')
const pool = require('../mysql/connection')
const { handleSQLError } = require('../mysql/error')
const getEmployeesById = (req, res) => {
let sql = "SELECT ?? FROM ?? WHERE ?? = ?"
sql = mysql.format(sql, ['*', 'employees', 'emp_no', req.params.id])
pool.query(sql, (err, rows) => {
if (err) return handleSQLError(res, err)
return res.json(rows);
})
}
module.exports = { getEmployeesById };
/:first_name and /:id match the exact same URLs so only the first one you register is going to get all the matching URLs. They both match /anything.
You really can't define routes like that. There's no way Express knows which route handler you want to use with /anything is the requested URL.
Instead, you need to define a route structure where those two types of URLs are different and you can design a route handler that will uniquely catch each one. I personally don't ever use top level wildcard routes like this because they match every top level URL and they prohibit you using top level URLs for any other purpose in your site.
Instead, you might change your URL design to do this:
router.get('/id/:id', ...)
router.get('/firstname/:firstname', ...);
Then, it would be completely clear from the URL what type of resource was being requested and each route would match only the appropriate URLs.
I'm currently working on an HTTPS Cloud Function using Firebase, consisting in deleting the post my Android user requested.
General idea
The workflow is (the whole code is available at the end of this SO Question): 1) Firebase checks the user identity (admin.auth().verifyIdToken) ; 2) Firestore gets data from the post that must be deleted (deleteDbEntry.get().then()) ; 3) Cloud Storage prepares itself to delete the file found in the gotten data (.file(filePath).delete()) ; 4) Firestore prepares a batch to delete the post (batch.delete(deleteDbEntry);) and to update the likes/unlikes using the gotten data (batch.update(updateUserLikes,) ; 5) executes the promise of the deletion of the file and of the batch (return Promise.all([deleteFile, batch_commit])).
Expected behavior
I would want to check the user identity. If it's successful, to get the requested post to delete's data using Firebase. If it's successful, I would want to execute the Firestore batch plus the Cloud Storage file deletion in the same promise (that's why I use Promise.all([deleteFile, batch_commit]).then()). If the identity check fails, or if the data get fails, or if the batch fails, I would want to tell the Android app. If all successes, idem.
As all of these operations are in a Cloud HTTPS Function, I must return a promise. This promise, I think, would correspond to all that operations if they are successful, or to an error if at least one is not (?).
Actual behavior
For the moment, I just return the promise of the Firebase user identity check.
My problem & My question
I can't go from the actual behavior to the expected behavior because:
I think it's not very clear in my mind whether I should return the promise corresponding to "all these operations are successful, or at least one is not" in this Cloud HTTPS Function
As these operations are nested (except the Firestorage file deletion + Firestore post deletion which are present in a batch), I can't return something like Promise.all().
My question
Could you please tell me if I'm right (point 1.) and, if not: what should I do? If yes: how could I do it, because of point 2.?
Whole Firebase Cloud HTTPS Function code
Note: I've removed my input data controls to make my code more lisible.
exports.deletePost = functions.https.onCall((data, context) => {
return admin.auth().verifyIdToken(idToken)
.then(function(decodedToken) {
const uid = decodedToken.uid;
const type_of_post = data.type_of_post;
const the_post = data.the_post;
const deleteDbEntry = admin_firestore.collection('list_of_' + type_of_post).doc(the_post);
const promise = deleteDbEntry.get().then(function(doc) {
const filePath = type_of_post + '/' + uid + '/' + data.stored_image_name;
const deleteFile = storage.bucket('android-f.appspot.com').file(filePath).delete();
const batch = admin.firestore().batch();
batch.delete(deleteDbEntry);
if(doc.data().number_of_likes > 0) {
const updateUserLikes = admin_firestore.collection("users").doc(uid);
batch.update(updateUserLikes, "likes", FieldValue.increment(-doc.data().number_of_likes));
}
const batch_commit = batch.commit();
return Promise.all([deleteFile, batch_commit]).then(function() {
return 1;
}).catch(function(error) {
console.log(error);
throw new functions.https.HttpsError('unknown', 'Unable to delete the post. (2)');
});
}).catch(function(error) {
console.log(error);
throw new functions.https.HttpsError('unknown', 'Unable to delete the post. (1)');
});
return promise;
}).catch(function(error) {
console.log(error);
throw new functions.https.HttpsError('unknown', 'An error occurred while verifying the token.');
});
});
You should note that you are actually defining a Callable Cloud Function and not an HTTPS one, since you do:
exports.deletePost = functions.https.onCall((data, context) => {..});
One of the advantages of a Callable Cloud Function over an HTTPS one is that it "automatically deserializes the request body and validates auth tokens".
So you can simply get the user uid with context.auth.uid;.
Now, regarding the way of "orchestrating" the different calls, IMHO you should just chain the different Promises returned by the asynchronous Firebase methods (the ones of Firestore and the one of Cloud Storage), as follows:
exports.deletePost = functions.https.onCall((data, context) => {
//....
const uid = context.auth.uid;
let number_of_likes;
const type_of_post = data.type_of_post;
const the_post = data.the_post;
const deleteDbEntry = admin_firestore.collection('list_of_' + type_of_post).doc(the_post);
return deleteDbEntry.get()
.then(doc => {
number_of_likes = doc.data().number_of_likes;
const filePath = type_of_post + '/' + uid + '/' + data.stored_image_name;
return storage.bucket('android-f.appspot.com').file(filePath).delete();
})
.then(() => {
const batch = admin.firestore().batch();
batch.delete(deleteDbEntry);
if (number_of_likes > 0) {
const updateUserLikes = admin_firestore.collection("users").doc(uid);
batch.update(updateUserLikes, "likes", FieldValue.increment(-doc.data().number_of_likes));
}
return batch.commit();
}).catch(function (error) {
console.log(error);
throw new functions.https.HttpsError('....', '.....');
});
});
I don't think using Promise.all() will bring any interest, in your case, because, as explained here, "if any of the passed-in promises reject, Promise.all asynchronously rejects with the value of the promise that rejected, whether or not the other promises have resolved".
At the time of writing, there is no way to group all of these asynchronous calls to different Firebase services into one atomic operation.
Even if the batched write at the end is atomic, it could happen that the file in Cloud Storage is correctly deleted but that the batched write to Firestore is not executed, for example because there is a problem with the Firestore service.
Also, note that you only need one exception handler at the end of the Promise chain. If you want to differentiate the cause of the exception, in such a way that you send a different error message to the front-end you could use the approach presented in this article.
The article shows how to define different custom Error classes (derived from the standard built-in Error object) which are used to check the kind of error in the exception handler.
I am currently working on building a backend using Express and MongoDB Native Node. I have been researching attempting to find the best "best practice" for managing connections to a Mongo database and using that connection across the app.
My current solution is working and I get desired results via tests in Postman. I have come up with this after being unable to find concrete answers on handling the connections with MongoDB 3.x (without Mongoose) and still modular.
Would someone be willing to give some feedback on my current solution?
My main concern would be that this setup would not be performant. I suspect it may not be, potentially due to opening/closing connections frequently, but I am not sure if the way I am doing it is good or bad practice.
I created a db.js file to serve my connection:
const assert = require("assert");
const MongoClient = require("mongodb").MongoClient;
const base = process.env.PWD;
const config = require(base + "/config");
let db;
let client;
const connect = async () => {
const url = config.url
const dbName = config.dbName
client = await MongoClient.connect(url)
db = await client.db(dbName)
return db
}
const disconnect = () => {
client.close()
}
module.exports = {
connect: connect,
disconnect: disconnect
}
I then set the routes for my 'todos' in index.js inside of my todos folder. Following a best practice suggestion to have all component files in their own folders(open to feedback on folder structure):
const express = require('express'),
base = process.env.PWD,
router = express.Router(),
todos = require(base + '/todos/todosController')
/* GET All Todos */
router.get('/all', todos.getTodos)
/* GET One Todo */
router.get('/todo/:id', todos.getTodo)
/* POST One Todo */
router.post('/todo/:id', todos.addTodo)
/* DELETE One Todo */
router.delete('/todo/:id', todos.deleteTodo)
module.exports = router;
Lastly the actual todosController.js which requires db.js
This is where I suspect some improvement could happen but I am just not sure. I connect within the routes via async function and await the connection and assign it to db I do my CRUD queries (all currently working properly) and then disconnect at the end.
If this is considered performant and a good practice I am happy with that answer but if there is a way to do this better with current driver and syntax I would be happy for any feedback.
'use strict';
const base = process.env.PWD,
client = require(base + '/db.js'),
assert = require('assert')
let db
const getTodos = async (req, res) => {
db = await client.connect()
const collection = await db.collection('documents')
// Find all todos
collection.find({}).toArray((err, todos) => {
assert.equal(err, null)
res.status(200).json(todos)
})
client.disconnect()
}
It seems that this is a common misconception that opening and closing a connection on every request is more efficient. Opening connection is expensive and this is one of the reasons for the existence of the connection pools. MongoDB supports those and you should consider them.
Here is an article on the subject of Express/MongoDB connection handling which starts right away with:
A common mistake developers make when connecting to the database is to
call MongoClient.connect() in every route handler to get a database
connection.
I've been creating a small node js app that iterates through an array of names and queries an API for the names. The issue I have is that the array is very large (400,000+ words) and my application runs out of memory before the forEach is complete.
I've been able to diagnose the issue by researching about how JS works with the call stack, web api, and callback queue. What I believe the issue to be is that the forEach loop is blocking the call stack and so the http requests continue to clog up the callback queue without getting resolved.
If anyone can provide a solution for unblocking the forEach loop or an alternative way of coding this app I would be very greatful.
Node JS App
const mongoose = require("mongoose");
const fs = require("fs");
const ajax = require("./modules/ajax.js");
// Bring in Models
let Dictionary = require("./models/dictionary.js");
//=============================
// MongoDB connection
//=============================
// Opens connection to database "test"
mongoose.connect("mongodb://localhost/bookCompanion");
let db = mongoose.connection;
// If database test encounters an error, output error to console.
db.on("error", (err)=>{
console.error("Database connection failed.");
});
db.on("open", ()=>{
console.info("Connected to MongoDB database...");
}).then(()=>{
fs.readFile("./words-2.json", "utf8", (err, data)=>{
if(err){
console.log(err);
} else {
data = JSON.parse(data);
data.forEach((word)=>{
let search = ajax.get(`API url Here?=${word}`);
search.then((response)=>{
let newWord = new Dictionary ({
word: response.word,
phonetic: response.phonetic,
meaning: response.meaning
}).save();
console.log("word saved");
}).catch((err)=>{
console.log("Word not found");
});
});
};
});
});
Check
Check whether api accepts multiple query params.
Try to use async Promises.
Resolve the promises and try to perform the save operation on the Promises by Promise#all
I'm having a little problem with thread blocking algorithms.
I have a route that generates a zip with huge size files.
The flow works like this:
GetUrls > ObtainHugeSizeBuffer > GenerateZIP > UploadZIPToCloud
I cannot modify the timeout response default time of 2 minutes with the express-timeout module. I've also been trying to break the loopholes in the .nextTick() function.
I've even tried to look over queueing but I don't think that applies in this situation.
Do you guys have any idea how to expand the response time? - I strictly need to for one route only.
// start the server
const server = app.listen(8080);
// increase the timeout to 4 minutes
server.timeout = 240000;
This is the easiest way to extend server timeout, but it affects everything, not just one method.
In your case (you wan't it only for on specific route):
'use strict';
const ms = require('ms');
const express = require('express');
const router = express.Router();
router.route('/upload-files')
.post(
setConnectionTimeout('12h'),
require('./actions/upload-files').responseHandler
);
function setConnectionTimeout(time) {
var delay = typeof time === 'string'
? ms(time)
: Number(time || 5000);
return function (req, res, next) {
res.connection.setTimeout(delay);
next();
}
}
exports.router = router;
Not my code, found it in this thead: Node Express specific timeout value per route
You need to google better :)