We Keep Coding

Javascript / Nodejs use await on top level in nodejs module

I tried finding the solution to my problem, but couldnt find it, and was looking for some "best practice examples". I have a nodejs express application and my functions are split in files. For example I have this controller (oktacontroller.js):
var okta_api_key = <here some await data, getting from db>;
const OKTA_ORG_URL = '<here too>';
exports.createUser = async (req, res) => {
console.log(okta_api_key);
}
exports.getGroups = async (req, res) => {
console.log(okta_api_key);
}
In both exported functions (which are express routes) I need the var okta_api_key. I know I can get them by querying them in both functions, because they are async functions and I can use await there, but it feels dumb to query this every time (because it wont ever change).
How can I manage this? I know I can do this:
var okta_api_key;
(async () => {
okta_api_key = await <getting key async>
})()
But this feels off as well..
Is there any way too make some sort of large function, which is async, and exports the two functions? In other words: How do I use await on the top level of a file (module). It doesnt need to be on top level (its impossible), but some sort of method to have my "top level" variables exposed to my exported functions.
EDIT: Some other usecase, because I got the suggestion of putting it in a config file. Yes, for this one it is possible, but for example: I have some other api key which gets his access token from the service itself, on every startup (because it expires). That token cannot be stored in the config file, so I need some async work to get the value. I know top-level-await is not working (or even not desirable), but I just want an example of how you guys would do this if it were your project :)

You are close:
var okta_api_key = (async () => {
return await <getting key async>
})();
Create a promise, then await that promise whenever you want to use it.
How do I use await on the top level of a file (module).
Top level await might look great: You just add one await and then you can access the variable synchronously. But that simplifies things too much: It will stop all modules depending on that module from executing. In most cases you don't want that¹. Instead create a promise of the async task, then await it when needed. That way you limit the asynchronous execution to the code pieces that actually need it.
¹ Those rare cases are:
1) Loading some global config, that you have to access everywhere in your code, so it makes no sense to start the service if the config isn't ready.
2) awaiting in the top level file of your service: As no module depends on it, this won't cause any problems.
Side note: Top level await is not yet specified, and the NodeJS support is also not there yet. To use it in production you have to wait a few months (?).

Node Express writing API functions styles/design patterns best practice

I use the MEAN stack and I am writing some REST API functions to manipulate data with routing functions...
I just have the problem that I am quite new to REST Api and the MEAN stack and also Javascript...
so I was wondering, that there are many different styles of writing those javascript functions, e.g. to get some data (and even how to do data manipulation, e.g by mongoose /mongoDB driver api function like aggregate / find, or doing the selection or filtering of data manually with javascript).
Just to show some examples:
exports.getTestbyIDRange = function(req, res) {
Test.aggregate([{"$match": {"$and": [{ "_id": {"$gte": +req.params.start, "$lte": +req.params.end}}]}}])
//Test.find({"_id": {"$gte": Number(req.params.start), "$lte": Number(req.params.end)}})
.exec(function(err, demo) {
console.log("Anzahl gefundener Dokumente: ", demo.length)
res.json(demo)
for (let elem of demo) {
console.log("ID: ", elem["_id"])
} } ) }
here, after the query I am using .exec(function..., but I also have seen ..,function(err, Test) {...} or a then-clause:
}).then( doc => {
// do something with matched document
}).catch(e => { console.err(e); res.send(e); })
So I am quite confused, is there a "best practice" style approach / design pattern for writing those functions? Or are all those styles equivalent? Which is best in terms of error handling? Or in case of errors / error handling, which is best to continue your app in case of some minor errors? I don't want my backend to get stuck or hang up.... Is there maybe some good literature to find online about styles?
Second question: When writing queries or data aggregation operations, should I do it with mongodb operations like find and aggregate, or can I also do it with plain javascript code ? (I find that sometimes easier because I am quite new to MongoDB)
The thing here is that my backend functions are all running on the same server like where my MongoDB is sitting, so its basically localhost, that means it shouldnt be too bad performance when retrieving complete documents from the server and then doing javascript code on it, otherwise if it would be by network, then it's understandable that I should do database operations with db drivers first...

Some tips from my side
1.Try to break big codes into smaller functions like some function to get data from db and another to process data
Make a common function for querying the database
There should be only one res.send or res.json etc in function and no code should be below that as the code below that will also exexute
Use promises for avoiding callback hell plus it will also help you to write a good maintainable code and will also help with better error handling
For generating a structure you can use swagger create project it will provide you a nice project structure
Use one middle ware with first param as err to cath unhadeled errors while executing api for example
app.use((err, req, res, next) => {
console.log('-----Something broke!---', err)
res.status(500).send('Something broke!')
})
Use process.on in index file to handle errors
For example
process.on('unhandledRejection', (reason, p) =>{console.log('Unhandled Rejection at:', p, 'reason:', reason)})
process.on('uncaughtException', err => {console.log('-----uncaughtException----- ', err)})

Functional Programming and async/promises

I'm refactoring some old node modules into a more functional style. I'm like a second year freshman when it comes to FP :) Where I keep getting hung up is handling large async flows. Here is an example where I'm making a request to a db and then caching the response:
// Some external xhr/promise lib
const fetchFromDb = make => {
return new Promise(resolve => {
console.log('Simulate async db request...'); // just simulating a async request/response here.
setTimeout(() => {
console.log('Simulate db response...');
resolve({ make: 'toyota', data: 'stuff' });
}, 100);
});
};
// memoized fn
// this caches the response to getCarData(x) so that whenever it is invoked with 'x' again, the same response gets returned.
const getCarData = R.memoizeWith(R.identity, (carMake, response) => response.data);
// Is this function pure? Or is it setting something outside the scope (i.e., getCarData)?
const getCarDataFromDb = (carMake) => {
return fetchFromDb(carMake).then(getCarData.bind(null, carMake));
// Note: This return statement is essentially the same as:
// return fetchFromDb(carMake).then(result => getCarData(carMake, result));
};
// Initialize the request for 'toyota' data
const toyota = getCarDataFromDb('toyota'); // must be called no matter what
// Approach #1 - Just rely on thenable
console.log(`Value of toyota is: ${toyota.toString()}`);
toyota.then(d => console.log(`Value in thenable: ${d}`)); // -> Value in thenable: stuff
// Approach #2 - Just make sure you do not call this fn before db response.
setTimeout(() => {
const car = getCarData('toyota'); // so nice!
console.log(`later, car is: ${car}`); // -> 'later, car is: stuff'
}, 200);
<script src="https://cdnjs.cloudflare.com/ajax/libs/ramda/0.25.0/ramda.min.js"></script>
I really like memoization for caching large JSON objects and other computed properties. But with a lot of asynchronous requests whose responses are dependent on each other for doing work, I'm having trouble keeping track of what information I have and when. I want to get away from using promises so heavily to manage flow. It's a node app, so making things synchronous to ensure availability was blocking the event loop and really affecting performance.
I prefer approach #2, where I can get the car data simply with getCarData('toyota'). But the downside is that I have to be sure that the response has already been returned. With approach #1 I'll always have to use a thenable which alleviates the issue with approach #2 but introduces its own problems.
Questions:
Is getCarFromDb a pure function as it is written above? If not, how is that not a side-effect?
Is using memoization in this way an FP anti-pattern? That is, calling it from a thenable with the response so that future invocations of that same method return the cached value?

Question 1
It's almost a philosophical question here as to whether there are side-effects here. Calling it does update the memoization cache. But that itself has no observable side-effects. So I would say that this is effectively pure.
Update: a comment pointed out that as this calls IO, it can never be pure. That is correct. But that's the essence of this behavior. It's not meaningful as a pure function. My answer above is only about side-effects, and not about purity.
Question 2
I can't speak for the whole FP community, but I can tell you that the Ramda team (disclaimer: I'm a Ramda author) prefers to avoid Promises, preferring more lawful types such Futures or Tasks. But the same questions you have here would be in play with those types substituted for Promises. (More on these issues below.)
In General
There is a central point here: if you're doing asynchronous programming, it will spread to every bit of the application that touches it. There is nothing you will do that changes this basic fact. Using Promises/Tasks/Futures helps avoid some of the boilerplate of callback-based code, but it requires you to put the post response/rejection code inside a then/map function. Using async/await helps you avoid some of the boilerplate of Promise-based code, but it requires you to put the post reponse/rejection code inside async functions. And if one day we layer something else on top of async/await, it will likely have the same characteristics.
(While I would suggest that you look at Futures or Tasks instead of Promises, below I will only discuss Promises. The same ideas should apply regardless.)
My suggestion
If you're going to memoize anything, memoize the resulting Promises.
However you deal with your asynchrony, you will have to put the code that depends on the result of an asynchronous call into a function. I assume that the setTimeout of your second approach was just for demonstration purposes: using timeout to wait for a DB result over the network is extremely error-prone. But even with setTimeout, the rest of your code is running from within the setTimeout callback.
So rather than trying to separate the cases for when your data has already been cached and when it hasn't, simply use the same technique everywhere: myPromise.then(... my code ... ). That could look something like this:
// getCarData :: String -> Promise AutoInfo
const getCarData = R.memoizeWith(R.identity, make => new Promise(resolve => {
console.log('Simulate async db request...')
setTimeout(() => {
console.log('Simulate db response...')
resolve({ make: 'toyota', data: 'stuff' });
}, 100)
})
)
getCarData('toyota').then(carData => {
console.log('now we can go', carData)
// any code which depends on carData
})
// later
getCarData('toyota').then(carData => {
console.log('now it is cached', carData)
})
<script src="//cdnjs.cloudflare.com/ajax/libs/ramda/0.25.0/ramda.min.js"></script>
In this approach, whenever you need car data, you call getCarData(make). Only the first time will it actually call the server. After that, the Promise is served out of the cache. But you use the same structures everywhere to deal with it.
I only see one reasonable alternative. I couldn't tell if your discussion about having to have to wait for the data before making remaining calls means that you would be able to pre-fetch your data. If that's the case, then there is one additional possibility, one which would allow you to skip the memoization as well:
// getCarData :: String -> Promise AutoInfo
const getCarData = make => new Promise(resolve => {
console.log('Simulate async db request...')
setTimeout(() => {
console.log('Simulate db response...')
resolve({ make: 'toyota', data: 'stuff' });
}, 100)
})
const makes = ['toyota', 'ford', 'audi']
Promise.all(makes.map(getCarData)).then(allAutoInfo => {
const autos = R.zipObj(makes, allAutoInfo)
console.log('cooking with gas', autos)
// remainder of app that depends on auto data here
})
<script src="//cdnjs.cloudflare.com/ajax/libs/ramda/0.25.0/ramda.min.js"></script>
But this one means that nothing will be available until all your data has been fetched. That may or may not be all right with you, depending on all sorts of factors. And for many situations, it's not even remotely possible or desirable. But it is possible that yours is one where it is helpful.
One technical point about your code:
const getCarDataFromDb = (carMake) => {
return fetchFromDb(carMake).then(getCarData.bind(null, carMake));
};
Is there any reason to use getCarData.bind(null, carMake) instead of () => getCarData(carMake)? This seems much more readable.

Is getCarFromDb a pure function as it is written above?
No. Pretty much anything that uses I/O is impure. The data in the DB could change, the request could fail, so it doesn't give any reliable guarantee that it will return consistent values.
Is using memoization in this way an FP anti-pattern? That is, calling it from a thenable with the response so that future invocations of that same method return the cached value?
It's definitely an asynchrony antipattern. In your approach #2 you are creating a race condition where the operation will succeed if the DB query completes in less than 200 ms, and fail if it takes longer than that. You've labeled a line in your code "so nice!" because you're able to retrieve data synchronously. That suggests to me that you're looking for a way to skirt the issue of asynchrony rather than facing it head-on.
The way you're using bind and "tricking" memoizeWith into storing the value you're passing into it after the fact also looks very awkward and unnatural.
It is possible to take advantage of caching and still use asynchrony in a more reliable way.
For example:
// Some external xhr/promise lib
const fetchFromDb = make => {
return new Promise(resolve => {
console.log('Simulate async db request...')
setTimeout(() => {
console.log('Simulate db response...')
resolve({ make: 'toyota', data: 'stuff' });
}, 2000);
});
};
const getCarDataFromDb = R.memoizeWith(R.identity, fetchFromDb);
// Initialize the request for 'toyota' data
const toyota = getCarDataFromDb('toyota'); // must be called no matter what
// Finishes after two seconds
toyota.then(d => console.log(`Value in thenable: ${d.data}`));
// Wait for 5 seconds before getting Toyota data again.
// This time, there is no 2-second wait before the data comes back.
setTimeout(() => {
console.log('About to get Toyota data again');
getCarDataFromDb('toyota').then(d => console.log(`Value in thenable: ${d.data}`));
}, 5000);
<script src="https://cdnjs.cloudflare.com/ajax/libs/ramda/0.25.0/ramda.min.js"></script>
The one potential pitfall here is that if a request should fail, you'll be stuck with a rejected promise in your cache. I'm not sure what would be the best way to address that, but you'd surely need some way of invalidating that part of the cache or implementing some sort of retry logic somewhere.

Mocking Redis Constructor with Sinon

I'm trying to figure out a way to mock redis in this module:
const Redis = require('ioredis');
const myFunction = {
exists: (thingToCheck) {
let redis_client = new Redis(
6379,
process.env.REDIS_URL,
{
connectTimeout: 75,
dropBufferSupport: true,
retryStrategy: functionHere
});
redis_client.exists(thingToCheck, function (err, resp) {
// handlings in here
});
}
};
Using this test-code:
const LambdaTester = require('lambda-tester');
const chai = require('chai');
const expect = chai.expect;
const sinon = require('sinon');
const mockRedis = sinon.mock(require('ioredis'));
describe( 'A Redis Connection error', function() {
before(() => {
mockRedis.expects('constructor').returns({
exists: (sha, callback) => {
callback('error!', null);
}
});
});
it( 'It returns a database error', function() {
return LambdaTester(lambdaToTest)
.event(thingToCheck)
.expectError((err) => {
expect(err.message).to.equal('Database error');
});
});
});
I also tried a few variations, but I'm kind of stuck as I basically need to mock the constructor and I'm not sure Sinon supports this?
mockRedis.expects('exists').returns(
(thing, callback) => {
callback('error!', null);
}
);
sinon.stub(mockRedis, 'constructor').callsFake(() => console.log('test!'));
sinon.stub(mockRedis, 'exists').callsFake(() => console.log('test!'));
Not sure what else to try here, I also tried using rewire as suggested here, but using mockRedis.__set__("exists", myMock); never set that private variable.
I want to fake my error paths ultimately.
I would love to hear what others are doing to test redis in node js 😄.

Your problem is not whether Sinon supports this or that, but rather a missing understanding of how "classes" work in Ecmascript, as shown by the attempt at stubbing constructor property in the test code. That will never work, as that property has nothing to do with how any resulting objects turn out. It is simply a reference to the function that was used to create the object. I have covered a very similar topic on the Sinon tracker that you might have interest in reading to gain some core JS foo :-) Basically, it is not possible to stub a constructor, but you can probably coerce your code to use another constructor function in its place through either DI or link seams.
As a matter of fact, a few answers down in the same thread, you will see me covering an example of how I myself designed a Redis using class to be easily testable by supporting dependency injection. You might want to check it out, as it is more or less directly applicable to your example module above.
Another technique, which you already has tried getting to work, is using link seams (using rewire). The Sinon homepage has a nice article on doing this. Both rewire and proxyquire will do the job just fine here: I think you have just complicated the matter a bit by wrapping the require statement with a mock.
Even though I am on the Sinon maintainer team, I never use the mock functionality, so I cannot tell you how to use that, as I think it obscures the testing, but to get the basic link seams working using rewire I would basically remove all the Sinon code first and get the basic case going (removing redis for a stubbed module you have created).
Only then, add Sinon code as required.

node jasmine - how to unit test on function that involve redis call?

I just started playing around Jasmine and I'm still struggling on the spyon/mocking things, e.g., I have a function
module.exports = (() => {
....
function getUserInfo(id) {
return new Promise((resolve, reject) => {
redis.getAsync(id).then(result => {
resolve(result)
})
})
}
return { getUserInfo: getUserInfo }
})()
Then I start writing the Jasmine spec
describe('Test user helper', () => {
let userInfo
beforeEach(done => {
userHelper.getUserInfo('userid123')
.then(info => {
userInfo = info
done()
})
})
it('return user info if user is found', () => {
expect(userInfo).toEqual('info of userid 123')
})
})
It runs well, but my question is how can I mock the redis.getAsync call, so it can become a real isolated unit test?
Thanks.

Good question. You can mock out the redis dependency but only if you rewrite you code, slightly, to be more testable.
Here, that means making redis a parameter to the factory that returns the object containing getUserInfo.
Of course, this changes the API, callers now need to call the export to get the object. To fix this, we can create a wrapper module that calls the function with the standard redis object, and returns the result. Then we move the actual factory into an inner module, which still allows it to be tested.
Here is what that might well look like
user-helper/factory.js
module.exports = redis => {
....
function getUserInfo(id) {
return redis.getAsync(id); // note simplified as new Promise was not needed
}
return {getUserInfo};
};
user-helper/index.js
// this is the wrapper that preserves existing API
module.exports = require('./factory')(redis);
And now for the test
const userHelperFactory = require('./user-helper/factory');
function createMockRedis() {
const users = [
{userId: 'userid123'},
// etc.
];
return {
getAsync: function (id) {
// Note: I do not know off hand what redis returns, or if it throws,
// if there is no matching record - adjust this to match.
return Promise.resolve(users.find(user => user.userId === id));
}
};
}
describe('Test user helper', () => {
const mockRedis = createMockRedis();
const userHelper = userHelperFactory(mockRedis);
let userInfo;
beforeEach(async () => {
userInfo = await userHelper.getUserInfo('userid123');
});
it('must return user info when a matching user exists', () => {
expect(userInfo).toEqual('info of userid 123');
});
});
NOTE: As discussed in comments, this was just my incidental approach to the situation at hand. There are plenty of other setups and conventions you can use but the primary idea was just based on the existing export of the result of an IIFE, which is a solid pattern, and I leveraged the NodeJS /index convention to preserve the existing API. You could also use one file and export via both module.exports = factory(redis) and module.exports.factory = factory, but that would, I believe, be less idiomatic in NodeJS. The broader point was that being able to mock for tests, and testability in general is just about parameterization.
Parameterization is wonderfully powerful, and its simplicity is why developers working in functional languages sometimes laugh at OOP programmers, such as yours truly, and our clandestine incantations like "Oh glorious Dependency Injection Container, bequeath unto me an instanceof X" :)
It is not that OOP or DI get it wrong it is that testability, DI, IOC, etc. are just about parameterization.
Interestingly, if we were loading redis as a module, and if we were using a configurable module loader, such as SystemJS, we could do this by simply using loader configuration at the test level. Even Webpack lets you do this to some extent, but for NodeJS you would need to monkey patch the Require Function, or create a bunch of fake packages, which are not good options.
To the OP's specific response
Thanks! That's a good idea, but practically, it seems it's quite strange when I have tons of file to test in which I will need to create a factory and index.js for each of them.
You would need to restructure your API surface and simply export factories that consuming code must call, rather than the result of applying those factories, to reduce the burden, but there are tradeoffs and default instances are helpful to consumers.