I am working on a database migration. This requires querying one DB, getting an array of records and perform a set of async operations to insert the data in the new DB. In order to keep data consistency, I want to insert the records one at the time so I want each operation to run sequentially. The only way I have found to do these is using recursion.
Is there a cleaner way of doing this same thing? I know there is a library called async https://caolan.github.io/async/v3/ which I never tried before.
The recursive method I have written looks like this:
const insertItem = async (data) => {
let item = data[0];
if (!item) {
//I am done return
return;
}
try {
//Do multiple await calls to insert record into new database
} catch (e) {
//Recover from error (DB rollbacks, etc)
} finally {
//Remove inserted or failed item from collection.
data.shift();
await insertItem(data);
}
};
//Query original database
getInfo().then((data) => insertItem(data));
You can use sync loop for...of it will wait for HTTP response.
const dataArr = ['data1', 'data2', 'data3'];
async function processItems(arr){
for(const el of arr) {
const response = await insertData(el);
// add some code here to process the response.
}
};
processItems(dataArr);
The posted code achieves iterating through the data collection (an array) retrieved from the first data base by using data.shift to mutate the argument array before calling the single function handling everything recursively.
To clean this up, remove the shift and recursive calls by splitting the data processing function into two:
one function to step through data retrieved from the first data base,
a second function to insert records in the second data base as required.
This removes the need for the .finally clause and leaves the structure of the code looking more like
async function insertData(data) {
for( let index = 0 ; index < data.length; ++index) {
await insertItem( data[index]);
}
}
async function insertItem( data) {
try {
//Do multiple await calls to insert record into new database
} catch (e) {
//Recover from error (DB rollbacks, etc)
// throwing an error here aborts the caller, insertData()
}
}
getInfo().then( insertData).catch( /*... handle fatal error ...*/);
Depending on preferred style, insertItem could be declared as nested function within insertData to keep it looking neat, and insertData could be written as an anonymous function argument of the then call after getInfo().
It is possible, of course, to perform asynchronous operations sequentially be other means, but using await inside an async function is perhaps the simplest coding method.
Related
I am using NodeJs, Express and async libarary. I am trying to fetch data from elasticsearch and return that information to the user.
Here is my code:
1. // function that generates the ES query I want
const generateEsQuery = (source,target)=> (callback)=>{
let query = {} // some query that I generated
callback(null,callback)
}
2. // functions that I want to call after the fetching the query
const readElasticData =(data)=> async(callback) =>{
const trafficData =await elasticClient.search({"ignoreUnavailable":true,index:data.indexes,body:{"query":data.query}},{ignore: [404]});
callback(null ,trafficData)
}
async function readEsData (data,callback){
const trafficData =await elasticClient.search({"ignoreUnavailable":true,index:data.indexes,body:{"query":data.query}},{ignore: [404]});
callback(null ,trafficData)
}
3. // Calling my funtions
function myFunction(information){
// async.waterfall([generateEsQuery(information[0],information[1]),readElasticData],// The second function doesnt even run
async.waterfall([generateEsQuery(information[0],information[1]),readEsData] // the second functions runs but doesn't return the results to the callback
function(err, results) {
console.log("All Results",results);
return results;
});
}
I have two functions one to generate a ElasticQuery (1) and another to execute the query (2), I am trying to use the waterfall function of async library to execute them once after another , but I am unable to fetch the results at the final callback.
Out of the two functions that I use to read data , the second one "readEsData" atleast runs after the "generateEsQuery" function. I am new to Node js and I am not sure what I am doing wrong.I also do not want to combine the two functions as I will be reusing the first one in other places.
I am following this approach : https://medium.com/velotio-perspectives/understanding-node-js-async-flows-parallel-serial-waterfall-and-queues-6f9c4badbc17
Question : How do I send the result of the second function through the callback.
I have to compare few rows derived from potgres with data derived from the dynamo DB. problem is in both cases i have to iterate over data & unless i can assign the value to a variable i will have to query the same data again from either one of two databases.
function postgressdata(sql){
return new Promise(function(resolve, reject) {
client_postgres.query(sql, (err, res) => {
resolve(res);
});
});
}
let get_creatives = postgressdata(sql);
get_creatives.then(function(result) {
console.log(result); // prints the result
let new_result = result;
}
console.log(new_result) // does not print result
How Can I assign the data to a variable here & use it anywhere?
I am a newbie to NODEJS so pardon me if I am asking silly question.
Nodejs is non-blocking.
so when this block of code runs
let get_creatives = postgressdata(sql);
get_creatives.then(function(result) {
console.log(result); // prints the result
let new_result = result;
}
console.log(new_result) // does not print result
what will happen is you will get nothing printed first because .then() will be called when the promise is resolved so this line
console.log(new_result) // does not print result
will be executed first which doesn't have the value yet obviously.
after that when the promise is resolved .then will be called and your value will be set.
now how to solve this?
I am assuming you're sending the value back to the callee so you can either use the async/await structure or you can use a callback, depending on how you need the value.
You can just keep a reference to the promise as your cache. A promise in JavaScript represents the result of an async operation, not the operation itself.
If you have something like:
async function runQuery() {
const result = await db.query(sql`SELECT * FROM creatives;`);
console.log('ran query');
return result;
}
const creativesPromise = runQuery();
async function useResult() {
console.log(await creativesPromise);
}
useResult();
useResult();
useResult();
even though we use the resulting list of creatives three times, "ran query" will only get logged once. Every use of the query results will need to be async, because you never know whether the query has finished running when you request the data, but the query will only be run once.
I have a callback, where I need to get the array out of my callback.
I am trying to return the awaited array into the predefined one.
let documents = [];
socket.on('echo', async function(documents, data){
documents = await data;
console.log(documents); // shows me, what I want to see
});
console.log(documents); // empty Array
I need the result in my predefined Array documents
I have read several Tuts, but I dont get it. I know on stackoverflow it is sked several times. But all threads seem to be more complex then my situation. So I hope to get it cleared out with an more incomplex one.
You need to understand something first. When this runs what is inside the callback doesn't run unit the server will emit that event, in your case 'echo'.
What I think you want to do is use documents outside the callback. You can create a function and call it when the event is emitted.
Something like this:
const manageDocuments = (documents) => {
// do what you want with documents array
console.log(documents);
}
socket.on('echo', async function(documents, data){
documents = await data;
manageDocuments(documents);
});
Of course you can also get rid of the async/await
let documents = await socket.on('echo', async function(documents, data){
console.log(documents);
return data;
});
console.log(documents);
The problem is that the code outside the socket function executes with the empty array because is only executed once at runtime.
If you want to have access to the documents inside the socket function you have to make then persist, or use the socket.on inside of another loop.
I am trying to iterate through an array which pushes a new Thing to a list, inside the Thing it does some async calls of its own. How would I iterate through an array in a synchronous way, as the callback requires the the data from the list to work. Since my for loop is synchronous and does some asynchronous calls, the callback is called before the the list if finished being made.
I do not know how I would iterate through the array and do all the work before doing the callback
load(file, callback) {
fs.readFile(file, (err, fd) => {
var data = JSON.parse(fd);
for(var i of data.array){
this.list.push(new Thing(i.id)); // new Thing is Asynchronous
}
callback(); // Needs a finished list
return;
});
}
Solved it:
By converting my Thing class to synchronous, by removing the asynchronous calls to a function inside the class, and first instantiating all the Things in the loop then calling Promise.all calling the function I solved the issue:
load(file, callback) {
fs.readFile(file, (err, fd) => {
var data = JSON.parse(fd);
for(var i of data.array){
this.list.push(new Thing(i.id));
}
Promise.all(this.list.map(i => i.load()).then(callback);
});
}
You'd have to have some state inside of Thing to track it's doneness for example you could have an instance variable that's a promise. So given this hacked together example of Thing
class Thing {
constructor(id) {
this.id = id;
this.done = new Promise((resolve, reject) => {
asyncThingWithCallback((err, data) {
if (err) {
this.asyncVal = null;
reject(err);
} else {
this.asyncVal = data;
resolve()
}
})
});
}
}
You can use the done property inside of your callback like this:
load(file, callback) {
fs.readFile(file, (err, fd) => {
var data = JSON.parse(fd);
for(var i of data.array){
this.list.push(new Thing(i.id)); // new Thing is Asynchronous
}
Promise.all(this.list.map((thing) => thing.done))
.then(callback)
});
}
First off, it's generally not advisable to have a constructor that needs some asynchronous operation to finish creating a valid object. That just doesn't lead to easily writable or maintainable code because the constructor has to return the object reference and because the operation is asynchronous, it has to return that object reference before you're done creating a valid object. That just leads to messy, partially created objects. You can make it work by requiring a completion callback be passed to the constructor and making sure the calling code does not attempt to use the object until after the completion callback has been called, but this is just not a clean way to do things. It also makes it impossible to have your async operation return a promise (which is the future of async design) because the constructor has to return the object reference so it can't return a promise.
You could embed the promise in the object, but that's messy too because the promise is only really useful during the initial async operation.
What is often done instead is to make the constructor be only synchronous and then have a .init() method that does the async parts. That makes for cleaner code and is compatible with implementations using promises.
Or, you can create a factory function that returns a promise that resolves to the object reference.
Second off, as you already seem to know, your for loop runs synchronously. It doesn't "wait" for any async operations inside it to complete before going onto the next part of the loop. As long as each invocation of the loop is separate and doesn't depend upon the prior iteration, that's all fine. All you need to know is when all the async operations in the loop are done and making your async operations return promises and using Promise.all() is generally the best tool for that.
So, let's supposed you use the .init() method scheme where .init() does the async part of the initialization and the constructor is synchronous and .init() returns a promise. Then, you could do this:
// create all the things objects
let things = data.array.map(i => new Thing(i.id));
// initialize them all asynchronously
Promise.all(things.map(item => {
return item.init();
})).then(function() {
// all things are asynchronously initialized here
});
Or, using the concept of a factory function that returns a promise that resolves to the object:
function newThing(i) {
let o = new Thing(i.id);
return o.init().then(function() {
// resolve to the object itself
return o;
});
}
Promise.all(data.array.map(i => newThing(i))).then(things => {
// all things in the array ready to be used here
});
If you need to sequence your array iteration so the 2nd iteration did not start until the async part of the first iteration was done and 3rd waited until the 2nd iteration was done and so on, then you can't use a for loop because it simply doesn't work that way. There are several different ways to do such a serialized async iteration. You can see several different schemes in these other posts:
How to synchronize a sequence of promises?
JavaScript: Perform a chain of promises synchronously
ES6 Promises - something like async.each?
How can I execute shell commands in sequence?
You can use primise.all to run all the promises after the for loop .Then you can resolve the promise.all .
load(file) {
fs.readFile(file).Then(function (fd){
var data = JSON.parse(fd);
var EachPromise = [ ];
for(var i of data.array){
EachPromise.push(new Thing(i.id)); // new Thing is Asynchronous
}
Promise.all(EachPromise) .then(function (result){
console.log('this is result',result);
}).Catch(function (error){
console.log('this is error', error);
});
}
I am using Node.js to send some data to a SQL Server table. The code has a for loop, which calls a function, which runs an insert statement (will eventually be a stored procedure). The foor loop runs through a series of objects, in order by date. Most of the time, the insert statements finish in the correct order but some of the entries are in the wrong order.
Is there some way to insure that the order of dates in the database is the same order as I call them (chronologically). I know about SORT in T-SQL, but I think the program should do it in the correct order. I have always heard to avoid thread sleeping and timers whenever possible, so I don't like the idea of setTimeout. Is there another way, perhaps with event handling?
Example code below. An element of the jsonArray is a single object that has a date key as well as several other numbers. The SQL Server table has an autoincrementing primary key integer, and I want the dates to be entered chronologically (like I call them), so that the primary key as well as the dates are in order.
I am also using the MSSQL import for the SQL manipulation.
// Main class
for (let j = 0; j < jsonArray.length; j++) {
let myObj = jsonArray[j]
WriteToDB.myWriteFunction(myObj)
}
// Different class
const sql = require('mssql')
let request = new sql.Request()
let query = (...)
function myWriteFunction () {
request.query(query) // Just the actual insert statement, no issue here
request.on('error', function (err) {
console.log('REQ ERROR')
console.dir(err)
})
// These dates are out of order, due to when the done event is triggered
request.on('done', function () {
console.log('QUERY RAN FOR ' + e1)
})
}
I am assuming its some sort of multithreading that is causing this
those database operations are done asynchronously and they all happen on the same thread. If the D1 (first database operation) takes longer than D2 (second database operation) and D1 is inserted after the D2 (resulting you problem), it's not a problem that involves multi-threading, maybe D2 has more data, etc. ..
I would totally recommend you NOT to use the bellow code sample unless you really believe the order in the sql in indeed important. req2 is done only after req1 is finish and so on .. => wasted time
let jsonArray = [1,2,3,4];
if (checkArray(jsonArray)) {
createReq(jsonArray[0])
}
function createReq(req) {
new Promise((resolve, reject) => {
setTimeout(() => {
resolve();
jsonArray.shift();
console.log(req);
if (checkArray(jsonArray)) {
createReq(jsonArray[0])
}
}, 100)
})
}
function checkArray(arr) {
return (jsonArray && jsonArray[0])
}