I've just discovered that my API is doing weird things when 2 requests are triggered at almost the same time.
I figured out that the issue was me missing the "var" declaration before my "user" variable below, but I'm really curious about the root issue that caused the bug described below:
I have two API endpoints that call the same function as follow:
router.get('/refresh_session_token', function (req, res) {
let user_id = req.body.user_id // The value sent is 8
findUserWithId(user_id)
.then(user_data => {
user = user_data // I forgot 'var' here
})
.then(() => {
console.log(user) // This should always show user data from user_id = 8
})
}
router.get('/resend_invite', function (req, res) {
let user_id = req.body.user_id // The value sent is 18
findUserWithId(user_id)
.then(user_data => {
user = user_data // I forgot 'var' here
})
.then(() => {
console.log(user) // This should always show user data from user_id = 18
})
}
const findUserWithId = (id) => {
return knex.raw(`SELECT * FROM users WHERE id = ?`, [id]).then((data) => data.rows[0])
}
All this code is in the same file that I export through module.exports = router;
What I discovered is that if I trigger the endpoints /refresh_session_token and /resend_invite at almost the same time each with two different user_id, it happens that sometimes, my console.log returns the same result for both as if I was using the same user_id.
Adding var to user fixed the issue but I'm very surprised as to what is actually happening on the background.
Do you have any idea?
When you don't declare your variable and you aren't running your module in Javascript's strict mode, then the first assignment to that variable with:
user = user_data
creates an automatic global variable named user. This means that your two routes are then sharing that same variable.
And, since your two routes both have asynchronous operations in them, even with the single-threadedness of things, your two routes can still be in-flight at the same time and both trying to use the same global variable. One route will overwrite the value from the other. This is a disaster in server-based code because usually, the bug won't show until you get into production and it will be really, really hard to find a reproducible case.
The best answer here is to always run your code in strict mode and then the JS interpreter will make this an error and you will never be allowed to run your code this way in the first place. The error will be found very quickly and easily.
Then obviously, always declare variables with let or const. There are very, very few reasons to ever use var any more as let and const give you more control over the scope of your variable.
To run your module in strict mode, insert this:
'use strict';
before any other Javascript statements.
Or, use something like TypeScript that doesn't let you do sloppy things like not declare your variables.
Related
I tried finding the solution to my problem, but couldnt find it, and was looking for some "best practice examples". I have a nodejs express application and my functions are split in files. For example I have this controller (oktacontroller.js):
var okta_api_key = <here some await data, getting from db>;
const OKTA_ORG_URL = '<here too>';
exports.createUser = async (req, res) => {
console.log(okta_api_key);
}
exports.getGroups = async (req, res) => {
console.log(okta_api_key);
}
In both exported functions (which are express routes) I need the var okta_api_key. I know I can get them by querying them in both functions, because they are async functions and I can use await there, but it feels dumb to query this every time (because it wont ever change).
How can I manage this? I know I can do this:
var okta_api_key;
(async () => {
okta_api_key = await <getting key async>
})()
But this feels off as well..
Is there any way too make some sort of large function, which is async, and exports the two functions? In other words: How do I use await on the top level of a file (module). It doesnt need to be on top level (its impossible), but some sort of method to have my "top level" variables exposed to my exported functions.
EDIT: Some other usecase, because I got the suggestion of putting it in a config file. Yes, for this one it is possible, but for example: I have some other api key which gets his access token from the service itself, on every startup (because it expires). That token cannot be stored in the config file, so I need some async work to get the value. I know top-level-await is not working (or even not desirable), but I just want an example of how you guys would do this if it were your project :)
You are close:
var okta_api_key = (async () => {
return await <getting key async>
})();
Create a promise, then await that promise whenever you want to use it.
How do I use await on the top level of a file (module).
Top level await might look great: You just add one await and then you can access the variable synchronously. But that simplifies things too much: It will stop all modules depending on that module from executing. In most cases you don't want that¹. Instead create a promise of the async task, then await it when needed. That way you limit the asynchronous execution to the code pieces that actually need it.
¹ Those rare cases are:
1) Loading some global config, that you have to access everywhere in your code, so it makes no sense to start the service if the config isn't ready.
2) awaiting in the top level file of your service: As no module depends on it, this won't cause any problems.
Side note: Top level await is not yet specified, and the NodeJS support is also not there yet. To use it in production you have to wait a few months (?).
I have a lambda function that connects to a MongoDB instance, and am adding a connection pool reuse logic so that when a container is reused by lambda, the previous connection is reused instead of creating a new one.
In my database.ts file, previously there was a single export const db = mongo.createConnection(..) that was being used everywhere in the code, which created a new connection on every function invocation.
Now, as per online tutorials, I stored the actual connection variable in a global variable in the file and converted the export to a function which checks whether the above-mentioned connection object is not null, and if so returns it without creating a new connection. If it is indeed null, then it creates a connection and assigns the result of createConnection to the variable and also returns it.
Of course, I needed to convert all the usages of db to db(), since it was a function now. Logs added in the file indicated that the connection was indeed being reused for all subsequent calls to db after the first one.
Then, I converted db into an IIFE and back to a normal export. I imagined that instead of having to go and replace all usages to a function call, it'd be better to call it in the file instead and then export it.
However now, what's happening is strange. The first invocation shows that a new connection is being created. (=> Creating a new database connection) in the code. However on subsequent invocations, none of the log statements are printed. Ideally, shouldn't (=> Reusing an existing connection) be printed on the console? I suppose the connection is still being reused since the first log does not appear, but why does this phenomenon occur?
I know it has something to do with how modules are required or cached and how AWS Lambda handles JS modules. Any further insight will be helpful.
let connection: mongoose.Connection | null = null;
let isPreviousConnectionAvailable = false;
export const db = (() => {
if (!isPreviousConnectionAvailable) {
log.debug('=> Creating a new database connection.');
connection = mongoose.createConnection(dbConnectionURL, { useNewUrlParser: true });
isPreviousConnectionAvailable = true;
} else {
log.debug('=> Reusing an existing database connection');
}
return connection;
})();
The logs are not going to be printed. As it gets called the first time it gets imported and then subsequently you are accessing the same constant when the container is warm. It is the expected behavior.
I have quite a complex express route in my node.js server, that makes many modifications to the database via mongoose.
My goal is to implement a mechanism, that reverts all changes when any error occurs. My idea was implementing functions for undoing into the catch block.
But this is quite ugly, as I have to know what the previous values were and what if an error occurs in the catch block? It's especially difficult to revert those changes, when an error occurred during a Promise.all(array.map( /* ... */ ))
My route looks akin to this:
module.exports = (req, res) => {
var arr1, var2, var3
try {
const body = req.body
arr1 = await fetchVar1(body)
const _data = await Promise.all([
Promise.all(
arr1.map(async x => {
const y = await fetchSometing(x)
return doSometing(y)
})
),
doSomething3(arr1),
])
var2 = _data[1]
var3 = _data[2]
return res.json({ arr1, var2, var3 })
} catch (err) {
/**
* If an error occurs I have to undo
* everything that has been done
* in the try block
*/
}
}
Preferably I would like to implement something that "batches" all changes and "commits" the changes if no errors occurred.
What you are looking for is transactions: https://mongoosejs.com/docs/transactions.html
Manually undoing stuff after doing them won't protect you from every issue, so you should not rely on that. For example, exactly as you wrote: what happens if there is a crash after a partial write (some data is written, some is not), then another crash during your "rollback" code, which does not cleanup everything? If your code depends on your data being absolutely clean, then you have a problem. Your code should either be able to handle partial data correctly, or you must have some way to guarantee that your data is perfectly good at all times.
Transactions is the way to go, because it only commits everything at once if everything works.
What you’re looking for is called Transactions.
Transactions are new in MongoDB 4.0 and Mongoose 5.2.0. Transactions let you execute multiple operations in isolation and potentially undo all the operations if one of them fails. This guide will get you started using transactions with Mongoose.
For more information check the link below:
https://mongoosejs.com/docs/transactions.html
I'll try to make this concise...
With all the ES6 hype, I feel like using the var keyword is becoming more and more frowned upon by the JS community at large. Due to this (perceived) general consensus, I've been getting in the habit of avoiding using var in lieu of the newer const and let keywords for creating variables.
Below is an example of a block of code I wrote where I couldn't use const because I wanted to redefine a variable. I was also using async / await so all the code was in the same block-level scope, which apparently prevents you from redefining let variables either. So my only option was to use var (or create yet another let variable).
Here's the code. Saving a new user account to a MongoDB via Mongoose...
router.post('/signup', async (req, res, next) => {
const { email, password } = req.body;
const user = await userModel.findOne({ email });
if (user) { res.send('Email already in use.'); return; }
// here is where the issue occurs - use var instead?
let newUser = new userModel({ email, password });
let newUser = await newUser.save();
console.log(newUser); // just checking everything works
res.send({ message: 'New user account created!' });
});
MDN mentions this behavior... with just ONE SENTENCE! That's not helpful. :( And then they go on to discuss hoisting behavior, which isn't my issue, or at least I don't see how that's related. MDN Source Here.
So in conclusion...
Is this an example of an instance when var would be the most appropriate keyword to use?
If this isn't a time for using var, is there any instance when var is the most appropriate keyword to use, even in the age of ES6 (7, 8, etc)?
Disclaimer: I know I don't absolutely need to save those variables, but in my opinion, it's worth the trade off to write slightly more verbose code that's also more readable and understandable. I'm fine with using var, I just though this was an interesting case.
There is no reason to ever use var in ES6. const should be your default. The only reason to use let over const is when you want to reassign the variable later in the same block. In that case you don't declare it again, just assign the new value to it:
let newUser = new userModel({ email, password });
newUser = await newUser.save();
// allowed to reassign because `newUser` is not `const`
Depending on what save does, I suspect you could just do:
const newUser = new userModel({ email, password });
await newUser.save();
newUser.save(); should reject the Promise it returns if the save failed, which await will convert to a thrown error. So there should be no question about whether or not the save succeeded in the code after that await. If those lines are reached, the save succeeded.
This is not the matter of using let , var, or const.
Within the same scope a variable name cannot be declared more than one time, or you will get the following Error;
SyntaxError: Identifier 'newUser' has already been declared
You can declare once and assign value as many as you want.
let newUser = new userModel({ email, password });
newUser = await newUser.save();
var usage is often restricted in linter rules because it's prone to be misused in ES6. This isn't a valid case for var. As another answer mentions, the variable should be reassigned, not redeclared:
let newUser = new userModel({ email, password });
newUser = await newUser.save();
The only valid case for var in ES6 is to define a global in top-level scope:
<script>var FOO = 1;</script>
let and const will result in Identifier has already been declared error if a variable was defined multiple times for some reason:
<script>let FOO = 1;</script>
...
<script>let FOO = 1;</script>
And if (typeof FOO === 'undefined) safeguard isn't applicable because it creates block scope.
This is applicable to browser scripts only. Node.js scripts are evaluated in module scope.
Alternatives may involve referring this (in loose mode) or window properties.
I am using web3 version 1.0.0-beta.27 where all accesses to the blockchain will be asynchronous, clearly this opens up the possibility of race conditions, ie:
var Web3 = require("web3");
// connect to etherum blockchain
var ether_port = 'http://localhost:8545'
var web3 = new Web3(new Web3.providers.HttpProvider(ether_port));
// this is how we set the value, note there is the possiblity of race condidtions here
var accounts = []
web3.eth.getAccounts().then(function(accts){
console.log("printing account: ", accts)
accounts = accts
})
// observe race condition
console.log("assert race condition: ", accounts[0])
The last line above is contrived, it is there to demonstrate that I would like to use accounts after it has been evaluated. Ie, eventually I would like modify/read the blockchain from a front end express.js web app or even a mobile app, so in the interest of being rigorous, what are the common tools in node.js to ensure race conditions never occur? Do these tools exist? If not what are some common practices. I am new to node.js as well.
One idea is to not attempt to directly store the data because code trying to access the data has no idea when it's valid due to the uncertain nature of asynchronous results. So, instead you store the promise and any code that wants access to the data, just uses .then()/.catch() on the promise. This will always work, regardless of the async timing. If the data is already there, the .then() handler will be called quickly. If the data is not yet there, then the caller will be in line to be notified when the data arrives.
let accountDataPromise = web3.eth.getAccounts().then(function(accts){
console.log("printing account: ", accts)
return accts;
});
// then, elsewhere in the code
accountDataPromise.then(accts => {
// use accts here
}).catch(err => {
// error getting accts data
});
FYI, assigning data from a .then() handler to a higher scoped variable that you want to generally use in other code outside the promise chain is nearly always a sign of troublesome code - don't do it. This is because other code outside the promise chain has no idea when that data will or will not be valid.