We Keep Coding

Sequential async operations for each in Node JS

I am working on a database migration. This requires querying one DB, getting an array of records and perform a set of async operations to insert the data in the new DB. In order to keep data consistency, I want to insert the records one at the time so I want each operation to run sequentially. The only way I have found to do these is using recursion.
Is there a cleaner way of doing this same thing? I know there is a library called async https://caolan.github.io/async/v3/ which I never tried before.
The recursive method I have written looks like this:
const insertItem = async (data) => {
let item = data[0];
if (!item) {
//I am done return
return;
}
try {
//Do multiple await calls to insert record into new database
} catch (e) {
//Recover from error (DB rollbacks, etc)
} finally {
//Remove inserted or failed item from collection.
data.shift();
await insertItem(data);
}
};
//Query original database
getInfo().then((data) => insertItem(data));

You can use sync loop for...of it will wait for HTTP response.
const dataArr = ['data1', 'data2', 'data3'];
async function processItems(arr){
for(const el of arr) {
const response = await insertData(el);
// add some code here to process the response.
}
};
processItems(dataArr);

The posted code achieves iterating through the data collection (an array) retrieved from the first data base by using data.shift to mutate the argument array before calling the single function handling everything recursively.
To clean this up, remove the shift and recursive calls by splitting the data processing function into two:
one function to step through data retrieved from the first data base,
a second function to insert records in the second data base as required.
This removes the need for the .finally clause and leaves the structure of the code looking more like
async function insertData(data) {
for( let index = 0 ; index < data.length; ++index) {
await insertItem( data[index]);
}
}
async function insertItem( data) {
try {
//Do multiple await calls to insert record into new database
} catch (e) {
//Recover from error (DB rollbacks, etc)
// throwing an error here aborts the caller, insertData()
}
}
getInfo().then( insertData).catch( /*... handle fatal error ...*/);
Depending on preferred style, insertItem could be declared as nested function within insertData to keep it looking neat, and insertData could be written as an anonymous function argument of the then call after getInfo().
It is possible, of course, to perform asynchronous operations sequentially be other means, but using await inside an async function is perhaps the simplest coding method.

Firebase cloud functions promise.all

After going through simple examples on cloud functions that worked, I want to test something more complex. I have created a table (BTDevices) table in the Firebase realtime database, populated with few data:
Now I want my cloud function to use the MAC addresses for retrieving the "userId" fields. This is a snippet of my JavaScript cloud function:
var getUserIdPromisesArray = [];
for (var i = 0; i < nearbyMACsArray.length; i++) {
var tmpNearbyMAC = nearbyMACsArray[i];
var promise = admin.database().ref(`${DB_BT_DEVICES}/${tmpNearbyMAC}/userId`).once('value');
getUserIdPromisesArray.push(promise);
}
return Promise.all(getUserIdPromisesArray).then(results => {
results.forEach(result => {
var userId = result.val();
console.log('result.val() = ', result.val());
});
});
From other posts like this, this, and this, I know that I need to first create all my promises, and when they are done, then process the fetched data. So the for-loop iterates the nearbyMACsArray that contains the MAC addresses, creates a promise, and pushes it to the getUserIdPromisesArray.
Then I wait for all promises to finish and iterate the "results" array to print the read value, but the cloud function log shows a null value in one of the outputs:
Any ideas on why this is happening? Am I missing something with the promises?
Thanks.

According to the documentation, val() returns null when there is no data at the requested location. This is most likely what's going on. It's not likely a problem with your promises.

Get Value out of Callback

I have a callback, where I need to get the array out of my callback.
I am trying to return the awaited array into the predefined one.
let documents = [];
socket.on('echo', async function(documents, data){
documents = await data;
console.log(documents); // shows me, what I want to see
});
console.log(documents); // empty Array
I need the result in my predefined Array documents
I have read several Tuts, but I dont get it. I know on stackoverflow it is sked several times. But all threads seem to be more complex then my situation. So I hope to get it cleared out with an more incomplex one.

You need to understand something first. When this runs what is inside the callback doesn't run unit the server will emit that event, in your case 'echo'.
What I think you want to do is use documents outside the callback. You can create a function and call it when the event is emitted.
Something like this:
const manageDocuments = (documents) => {
// do what you want with documents array
console.log(documents);
}
socket.on('echo', async function(documents, data){
documents = await data;
manageDocuments(documents);
});
Of course you can also get rid of the async/await

let documents = await socket.on('echo', async function(documents, data){
console.log(documents);
return data;
});
console.log(documents);

The problem is that the code outside the socket function executes with the empty array because is only executed once at runtime.
If you want to have access to the documents inside the socket function you have to make then persist, or use the socket.on inside of another loop.

Cannot get data needed due to Mongoose Queries/Promises

I am trying to create a Jeopardy like game. I have the topics for the game in a database. When the page is opened I want it to get 6 topics randomly from the database. I am putting them in an array which I will then pass to my ejs file. The problem is that when I go to pass the array to the ejs file, it is always empty. I understand that it is because of promises(actually queries) from Mongoose. My problem is that I cannot figure out how to handle this. I have read the Mongoose docs and searched everywhere but I can't figure it out.
I have tried using callbacks, but they usually just make the program hang and do nothing. I have tried using .then, but I must be using it wrong because it doesn't do what I want.
app.get("/", function(request, response){
//My array to put the topics into
var questionArr = [];
//I need to know the number of items in db for random
var getQuestions = Questions.countDocuments({}, function(err, count){
for(var i = 0; i < 6; i++){
!function(i){
var rand = math.randomInt(count);
//Here I get a random topic and add to array
//Seems to work and actually get it
Questions.findOne().skip(rand).exec(function(err, result){
questionArr.push(result);
});
}(i);
}
});
//I thought this would make it wait for the function to finish so
//that I could have my full array, but apparently not
getQuestions.then(() => {
//this runs before the other functions and give me a length of 0
console.log(questionArr.length);
response.render("jeopardy.ejs", {questions: questionArr});
});
});
I simply need to have the render run after it gets the information from the database. However, it still just runs with an empty array. Thanks for any help, I'm pretty new to async.

I see few issues with your code:
1) You're mixing promises and callbacks which makes things more complicated. The code doesn't work mainly because you're not awaiting for Questions.findOne() results.
2) There is no Math.randomInt
In order to make it work it has to be similar to below:
Questions.countDocuments()
.then((count) => {
return Promise.all([...new Array(6)].map(() => {
const rand = Math.floor(Math.random() * Math.floor(count));
return Questions.findOne().skip(rand).exec();
}))
})
.then((questionArr) => {
response.render("jeopardy.ejs", {questions: questionArr});
});
Best is to use async/await which will make it even more readable
app.get("/", async function(request, response){
const count = await Questions.countDocuments();
const questionArr = await Promise.all([...new Array(6)].map(() => {
const rand = Math.floor(Math.random() * Math.floor(count));
return Questions.findOne().skip(rand).exec();
}));
response.render("jeopardy.ejs", {questions: questionArr});
});
Also keep in mind that you better do proper error handling, but that's a subject of a separate post :)

Sequelize unable to assign data after query with .then()

I have this function but I am unable to put it inside my item variable. How do I put it outside the item variable?
function getItems(id) {
let item;
sequelize.query("SELECT * FROM table WHERE id = " + id, {type: Sequelize.QueryTypes.SELECT})
.then(myTableRows => {
// item = JSON.stringify(myTableRows);
});
return item;
}

Your sequelize query returns a promise, and you cannot set the value of the item inside the promise and expect an immediate response. If you want it to work this way, you need to handle the promise correctly, or use an async-await. You can reconfigure your function to look like this, which will handle the issue:
async function getItems(id) {
let item;
item = await sequelize.query("SELECT * FROM table WHERE id = " + id, {type: Sequelize.QueryTypes.SELECT});
return item;
}
===========================================================================
Update:
I am not sure your experience level with node, but many aspects of node leverage promises. At a high level, what that means is perform some action, then do something once its done. With sequelize, you are performing a SQL query, and once you get a response, you can now do something with this data.
Async-Await is a way to make promises a bit more straightforward in understanding, but act the same. There are plenty of articles online to help you better understand the concepts.
Now the function that is calling getItems needs to also be an async-await, or call it as a promise in order to actually get the values. For example:
async function test(id) {
let items = await getItems(id);
console.log(items)
}
Hopefully that clears up some confusion, though you many need to learn a bit more about promises if you are unclear about how they work. Hopefully this helps.