We Keep Coding

Javascript / Nodejs use await on top level in nodejs module

I tried finding the solution to my problem, but couldnt find it, and was looking for some "best practice examples". I have a nodejs express application and my functions are split in files. For example I have this controller (oktacontroller.js):
var okta_api_key = <here some await data, getting from db>;
const OKTA_ORG_URL = '<here too>';
exports.createUser = async (req, res) => {
console.log(okta_api_key);
}
exports.getGroups = async (req, res) => {
console.log(okta_api_key);
}
In both exported functions (which are express routes) I need the var okta_api_key. I know I can get them by querying them in both functions, because they are async functions and I can use await there, but it feels dumb to query this every time (because it wont ever change).
How can I manage this? I know I can do this:
var okta_api_key;
(async () => {
okta_api_key = await <getting key async>
})()
But this feels off as well..
Is there any way too make some sort of large function, which is async, and exports the two functions? In other words: How do I use await on the top level of a file (module). It doesnt need to be on top level (its impossible), but some sort of method to have my "top level" variables exposed to my exported functions.
EDIT: Some other usecase, because I got the suggestion of putting it in a config file. Yes, for this one it is possible, but for example: I have some other api key which gets his access token from the service itself, on every startup (because it expires). That token cannot be stored in the config file, so I need some async work to get the value. I know top-level-await is not working (or even not desirable), but I just want an example of how you guys would do this if it were your project :)

You are close:
var okta_api_key = (async () => {
return await <getting key async>
})();
Create a promise, then await that promise whenever you want to use it.
How do I use await on the top level of a file (module).
Top level await might look great: You just add one await and then you can access the variable synchronously. But that simplifies things too much: It will stop all modules depending on that module from executing. In most cases you don't want that¹. Instead create a promise of the async task, then await it when needed. That way you limit the asynchronous execution to the code pieces that actually need it.
¹ Those rare cases are:
1) Loading some global config, that you have to access everywhere in your code, so it makes no sense to start the service if the config isn't ready.
2) awaiting in the top level file of your service: As no module depends on it, this won't cause any problems.
Side note: Top level await is not yet specified, and the NodeJS support is also not there yet. To use it in production you have to wait a few months (?).

Is it safe to rely on Node.js require behavior to implement Singletons?

Suppose I have a module implemented like this:
const dbLib = require('db-lib');
const dbConfig = require('db-config');
const dbInstance = dbLib.createInstance({
host: dbConfig.host,
database: dbConfig.database,
user: dbConfig.user,
password: dbConfig.password,
});
module.exports = dbInstance;
Here an instance of database connection pool is created and exported. Then suppose db-instance.js is required several times throughout the app. Node.js should execute its code only once and always pass the same one instance of database pool. Is it safe to rely on this behavior of Node.js require command? I want to use it so that I don't need to implement dependency injection.

Every single file that you require in Node.js is Singleton.
Also - require is synchronous operation and it also works deterministically. This means it is predictable and will always work the same.
This behaviour is not for require only - Node.js has only one thread in Event-Loop and it works like this (little simplified):
Look if there is any task that it can do
Take the task
Run the task synchronously from the beginning to the end
If there are any asynchronous calls, it just push them to "do later", but it never starts them before synchronous part of the task is done (unless there is worker spawned, but you dont need to know details about this)
Repeat the whole process
For example imagine this code is file infinite.js:
setTimeout(() => {
while(true){
console.log('only this');
}
}, 1000)
setTimeout(() => {
while(true){
console.log('never this');
}
}, 2000)
while(someConditionThatTakes5000Miliseconds){
console.log('requiring');
}
When you require this file, it first register to "doLater" the first setTimeout to "after 1000ms be resolved", the second for "after 2000ms be resolved" (note that it is not "run after 1000ms").
Then it run the while cycle for 5000ms (if there is condition like that) and nothing else happens in your code.
After 5000ms the require is completed, the synchronous part is finished and Event Loop looks for new task to do. And the first one to see is the setTimeout with 1000ms delay (once again - it took 1000ms to just mark as "can be taken by Event-Loop", but you dont know when it will be run).
There is neverending while cycle, so you will see in console "only this". The second setTimeout will never be taken from Event-Loop as it is marked after 2000ms to "can be taken", but Event Loop is stuck in never-ending while loop already.
With this knowledge, you can use require (and other Node.js aspects) very confidently.
Conclusion - the require is synchronous, deterministic. Once it finishes with requiring file (the output of it is a object with methods and properties you export, or empty object if you dont export anything) the reference to this object is saved to Node.js core memory. When you require file from somewhere else, it firsts look into the core memory and if it finds the require there, it just use the reference to the object and therefore never execute it twice.
POC:
Create file infinite.js
const time = Date.now();
setTimeout(() => {
let i=0;
console.log('Now I get stuck');
while(true){
i++;
if (i % 100000000 === 0) {
console.log(i);
}
}
console.log('Never reach this');
}, 1000)
setTimeout(() => {
while(true){
console.log('never this');
}
}, 2000)
console.log('Prepare for 5sec wait')
while(new Date() < new Date(time + 5*1000)){
// blocked
}
console.log('Done, lets allow other')
Then create server.js in same folder with
console.log('start program');
require('./infinite');
console.log('done with requiring');
Run it with node server
This will be the output(with numbers neverending):
start program
Prepare for 5sec wait
Done, lets allow other
done with requiring
Now I get stuck
100000000
200000000
300000000
400000000
500000000
600000000
700000000
800000000
900000000

The documentation of Node.js about modules explains:
Modules are cached after the first time they are loaded. This means (among other things) that every call to require('foo') will get exactly the same object returned, if it would resolve to the same file.
Multiple calls to require('foo') may not cause the module code to be executed multiple times.
It is also worth mentioning the situations when require produce Singletons and when this goal could not reached (and why):
Modules are cached based on their resolved filename. Since modules may resolve to a different filename based on the location of the calling module (loading from node_modules folders), it is not a guarantee that require('foo') will always return the exact same object, if it would resolve to different files.
Additionally, on case-insensitive file systems or operating systems, different resolved filenames can point to the same file, but the cache will still treat them as different modules and will reload the file multiple times. For example, require('./foo') and require('./FOO') return two different objects, irrespective of whether or not ./foo and ./FOO are the same file.
To summarize, if your module name is unique inside the project then you'll always get Singletons. Otherwise, when there are two modules having the same name, require-ing that name in different places may produce different objects. To ensure they produce the same object (the desired Singleton) you have to refer the module in a manner that is resolved to the same file in both places.
You can use require.resolve() to find out the exact file that is resolved by a require statement.

mdg:validated-method _execute asynchronous issues

I'm running into problems with the validated method package in my app tests. I'm calling my methods through the _execute function in order to be able to pass a userId to simulate a logged-in user while testing. The problem is that my asserts right underneath that _execute are called before the method has a chance of completing. I know my test works though because it only happens sometimes, mostly because mongo isn't always returning results quite as fast.
I looked around and found a todos app that uses the _execute function in its tests. I can't get those tests to fail no matter how many times I rerun them, though.
This is an example of my test code.
describe('clients.add', function() {
it('should add an empty (draft) client', function() {
const res = clients_add._execute({ userId: 'CURRENTUSERID' }, { company_id: c1._id });
assert.instanceOf(res, Mongo.ObjectID, 'method returns the newly created clients ID');
const db_client = Clients.findOne(res);
assert.isTrue(db_client.draft, 'client is drafted');
assert.isDefined(db_client.created, 'there\'s a created date');
});
});
clients_add does quite a few permission checks and can therefor take a little while before completing. Rerunning this test 20 times will fail about 5 times and pass the other 15.
Shouldn't the _execute function be synchronous? How do I make it? What am I missing?

In server code, if you provide a callback to database modification functions like insert, it returns the created ID instantaneously, and runs the callback only once the database has acknowledged the write. If you don't provide a callback, the insert call is synchronous and throws an error if the operation fails. See more about this in Meteor docs.
It seems that you have provided an error-handling callback to the insert-function in your method code. This causes the inconsistent behavior, since the database might not actually have had time to do the write before you call findOne in your test. Also, this is redundant since if an error occurs in the insert, the method has already returned and the error is never shown to the user. It's better to simply omit the error-handling callback altogether:
return Clients.insert(new_client);

Module returning asynchronously initialited object

I'm having this "design" problem that's driving me crazy.
My goal is having a decoupled RabbitMQ client. It has to be able to init it's connection and "return" a created channel so my publishing module can use it.
Code looks like this (i know that is not the better code but i expect it serves for this explanation).
var createConnection = (function() {
var channel;
var connect = function(){
// amqp connect
// error handling
createChannel();
});
}
var createChannel = function(){
//amqpConn.createConfirmChannel...
}
//pseudo
return{
getChannel : function(){
if(!initiated)
connect();
return channel;
}
}
})();
module.exports = createConnection;
Now, important things:
1- I know this ain't gonna work and i know why, its a simplification.
2- I'm aware that i can accomplish my goals by using async or promises.
3- Maybe has no sense decoupling a rabbit client but is for understanding purposes
That said, my questions:
1- Is there any way i can accomplish this without using other modules?
2- if so, can be accomplished in a fancy and stylish way?
3- Is there any fancy solution that allows 3rd party code executing a simple "publish(exchange, channel, msg)" been sure that connection has been established?
I feel able to work with JS but sometimes you just need do things one way only to know that you can but this is giving me some headache.
Truly thanks and i hope the question was understood :)

One way I've found to handle this is to wrap your asynchronous object in an object that is aware of the asynchronous state of your object and presents the same API regardless of whether or not the asynchronous object has finished initializing.
For example, you could wrap your channel object in another object that presents the same methods but internally checks if the actual channel object is initialized. If it is, use it as normal. If it isn't, wait for it to be initialized and use it as normal. The user of the wrapper object wouldn't need to know if the channel is actually initialized. The main drawback of this is every wrapper method that needs to access channel must be asynchronous even if the method it's accessing on channel is synchronous.
Example:
function initializeChannel() {
return new Promise((resolve, reject) => {
// create, initialize, and resolve channel
});
}
module.exports = { // wrapper
channelPromise: initializeChannel(),
foo(a) {
return this.channelPromise.then((channel) => channel.foo(a));
}
};

When working with NodeJS FS mkdir, what is the importance of including callbacks?

I'm playing with the NodeJS REPL console and following this tutorial.
http://www.tutorialspoint.com/nodejs/nodejs_file_system.htm
I'm focusing on the File System(FS) module. Let's look at the mkdir function used for creating directories.
According to TutorialsPoint, this is how you create a directory with FS
var fs = require("fs");
console.log("Going to create directory /tmp/test");
fs.mkdir('/tmp/test',function(err){
if (err) {
return console.error(err);
}
console.log("Directory created successfully!");
});
They specifically say you need this syntax
fs.mkdir(path[, mode], callback)
Well I just tried using less code without the callback and it worked.
var fs = require('fs');
fs.mkdir('new-directory');
And the directory was created. The syntax should just be
fs.mkdir(path);
I have to ask, what is the purpose of the callback and do you really need it? For removing a directory I could understand why you would need it, in case the directory didn't exist. But I can't see what could possibly go wrong with the mkdir command. Seems like a lot of unnecessary code.

As of node v10.0, the callback to fs.mkdir() is required. You must pass it, even if you just pass a dummy function that does nothing.
The point of the callback is to let you know if and when the call succeeded and if it didn't succeed, what the specific error was.
Remember, this type of function is asynchronous. It completes some unknown time in the future so the only way to know when it is done or if it completed successfully is by passing a callback function and when the callback is called, you can check the error and see that it has completed.
As it turns out, there are certainly things that can go wrong with mkdir() such as a bad path, a permissions error, etc... so errors can certainly happen. And, if you want to immediately use that new directory, you have to wait until the callback is called before using it.
In response to one of your other comments, the fs.mkdir() function is always asynchronous whether you pass the callback or not.
Here's an example:
var path = '/tmp/test';
fs.mkdir(path, function (err) {
if (err) {
console.log('failed to create directory', err);
} else {
fs.writeFile(path + "/mytemp", myData, function(err) {
if (err) {
console.log('error writing file', err);
} else {
console.log('writing file succeeded');
}
});
}
});
Note: Modern versions of nodejs, include fs.promises.mkdir() which returns a promise that resolves/rejects instead of using plain callbacks. This allows you to use await with try/catch or .then() and .catch() instead of the plain callback to know when it's done and promises make it typically easier to sequence in with other asynchronous operations and to centralize error handling.

Because mkdir is async.
Example:
If you do:
fs.mkdir('test');
fs.statSync('test').isDirectory();//might return false cause it might not be created yet
But if you do:
fs.mkdir('test', function() {
fs.statSync('test').isDirectory();//will be created at this point
});
You can still use mkdirSync if you need a sync version.

Many things could go wrong by using mkdir, and you should probably handle exceptions and errors and return them back to the user, when possible.
e.g. mkdir /foo/bar could go wrong, as you might need root (sudo) permissions in order to create a top-level folder.
However, the general idea behind callbacks is that the method you're using is asynchronous, and given the way Javascript works you might want to be notified and continue your program execution once the directory has been created.
Update: bare in mind that if you need — let's say — to save a file in the directory, you'll need to use that callback:
fs.mkdir('/tmp/test', function (err) {
if (err) {
return console.log('failed to write directory', err);
}
// now, write a file in the directory
});
// at this point, the directory has not been created yet
I also recommend you having a look at promises, which are now being used more often than callbacks.

Because it's an async call, it may be that further execution of the program depends on the outcome of the operation (dir created sucessfully). When the callback executes is the first point in time when this can be checked.
However, this operation is really fast, it may seem as it's happening instantly, but really (because it's async), the following line after fs.mkdir(path); will be executed without waiting for any feedback from the fs.mkdir(path); thus w/o any guarantee that the directory creation finished already, or if it failed.