I am using NodeJs, Express and async libarary. I am trying to fetch data from elasticsearch and return that information to the user.
Here is my code:
1. // function that generates the ES query I want
const generateEsQuery = (source,target)=> (callback)=>{
let query = {} // some query that I generated
callback(null,callback)
}
2. // functions that I want to call after the fetching the query
const readElasticData =(data)=> async(callback) =>{
const trafficData =await elasticClient.search({"ignoreUnavailable":true,index:data.indexes,body:{"query":data.query}},{ignore: [404]});
callback(null ,trafficData)
}
async function readEsData (data,callback){
const trafficData =await elasticClient.search({"ignoreUnavailable":true,index:data.indexes,body:{"query":data.query}},{ignore: [404]});
callback(null ,trafficData)
}
3. // Calling my funtions
function myFunction(information){
// async.waterfall([generateEsQuery(information[0],information[1]),readElasticData],// The second function doesnt even run
async.waterfall([generateEsQuery(information[0],information[1]),readEsData] // the second functions runs but doesn't return the results to the callback
function(err, results) {
console.log("All Results",results);
return results;
});
}
I have two functions one to generate a ElasticQuery (1) and another to execute the query (2), I am trying to use the waterfall function of async library to execute them once after another , but I am unable to fetch the results at the final callback.
Out of the two functions that I use to read data , the second one "readEsData" atleast runs after the "generateEsQuery" function. I am new to Node js and I am not sure what I am doing wrong.I also do not want to combine the two functions as I will be reusing the first one in other places.
I am following this approach : https://medium.com/velotio-perspectives/understanding-node-js-async-flows-parallel-serial-waterfall-and-queues-6f9c4badbc17
Question : How do I send the result of the second function through the callback.
Related
I have an async function that makes an API call as so:
const getModel = async () => {
const model = await fetch model...
return model;
};
I also have another function that uses the tfjs model to analyze the sentiment of text:
export const analyzeSentiment = async (text) => {
const model = await getModel();
const score = await predict(text, model);
return score;
};
My problem is I'm mapping multiple sets of text to the analyzeSentiment function, which requires getModel to make a fetch call for every set of text. Is there any way to set the return of getModel()
to a variable that will persist allowing it to be only called once and used for every time analyzeSentiment is called? I have also tried declaring model outside of analyzeSentiment and awaiting it inside the function, however that did not work either. If it matters, I'm using React/Node.
You can create a local cache variable that remembers the value from the first time it was requested and then uses that value from then on:
let cachedModel;
export const analyzeSentiment = async (text) => {
if (!cachedModel) {
cachedModel = getModel();
}
return predict(text, await cachedModel);
};
In this implementation cachedModel will contain a promise from the first time that analyzeSentiment() was called. If analyzeSentiment() is called multiple times in sequence, the code will still only call getModel() once - all the successive calls will just await the same promise. This all works whether the promise is pending or already fulfilled.
Or, if you just want to use the same model for a series of analyzeSentiment() calls, but only for that specific series of calls, then get the model once in the caller and pass the same one into a saquence of analyzeSentiment() calls as a function argument.
I am working on a database migration. This requires querying one DB, getting an array of records and perform a set of async operations to insert the data in the new DB. In order to keep data consistency, I want to insert the records one at the time so I want each operation to run sequentially. The only way I have found to do these is using recursion.
Is there a cleaner way of doing this same thing? I know there is a library called async https://caolan.github.io/async/v3/ which I never tried before.
The recursive method I have written looks like this:
const insertItem = async (data) => {
let item = data[0];
if (!item) {
//I am done return
return;
}
try {
//Do multiple await calls to insert record into new database
} catch (e) {
//Recover from error (DB rollbacks, etc)
} finally {
//Remove inserted or failed item from collection.
data.shift();
await insertItem(data);
}
};
//Query original database
getInfo().then((data) => insertItem(data));
You can use sync loop for...of it will wait for HTTP response.
const dataArr = ['data1', 'data2', 'data3'];
async function processItems(arr){
for(const el of arr) {
const response = await insertData(el);
// add some code here to process the response.
}
};
processItems(dataArr);
The posted code achieves iterating through the data collection (an array) retrieved from the first data base by using data.shift to mutate the argument array before calling the single function handling everything recursively.
To clean this up, remove the shift and recursive calls by splitting the data processing function into two:
one function to step through data retrieved from the first data base,
a second function to insert records in the second data base as required.
This removes the need for the .finally clause and leaves the structure of the code looking more like
async function insertData(data) {
for( let index = 0 ; index < data.length; ++index) {
await insertItem( data[index]);
}
}
async function insertItem( data) {
try {
//Do multiple await calls to insert record into new database
} catch (e) {
//Recover from error (DB rollbacks, etc)
// throwing an error here aborts the caller, insertData()
}
}
getInfo().then( insertData).catch( /*... handle fatal error ...*/);
Depending on preferred style, insertItem could be declared as nested function within insertData to keep it looking neat, and insertData could be written as an anonymous function argument of the then call after getInfo().
It is possible, of course, to perform asynchronous operations sequentially be other means, but using await inside an async function is perhaps the simplest coding method.
I'm not sure if this is possible and can't wrap my head around it. I have a larger project where I want to combine a couple of callbacks in different files into one and simply get the data from the different functions. My issue is that I can't seem to fetch the last part which I'll demonstrate in the code.
File one, importing file three and calling the sendUpdate function with 'note' as param.
const three = require("./three")
const note = "test"
three.sendUpdate(note);
File two,
async function getUser(user) {
return await otherfileFarAway.otherFunctionFarAway(user);
}
module.exports = {
getUser
}
File three, where I want to bundle these two up. I'm importing file two
const two = require("two");
async function sendUpdate(note) {
const total = await two.getUser(user);
console.log(note); // Works
console.log(total); // Undefined, duuh
try {
const url
.concat(total)
.concat("/Notes");
const result = await axios.post(
url,
{
Comment: note
},
);
return result.data;
} catch (error) {
logger.axiosError(error);
return null;
}
}
module.exports = {
sendUpdate
}
How would I actually just call getUser in file two from file three and get the value it is getting in file two? Is that possible? If I call it without parameters I get nothing, if I send it with something I get undefined. If I define it beforehand with for example "let user;" I get nothing.
What am I doing wrong or is it just simply not possible to get the returned value?
I suppose you are using Node? If that is that case you need to use exports to make them available via require. Here's some doc
Modify file two to the following to add some debug messages, so you'll see what's wrong:
async function getUser(user) {
console.log(`====== executing getUser() in filetwo with param: `, user);
const result = await otherfileFarAway.otherFunctionFarAway(user);
console.log(`====== returning `, result);
return result;
}
module.exports = {
getUser
}
Your questions seems to be How would I actually just call getUser in file two from file three and get the value it is getting in file two?.
The answer is that you're already calling that filetwo.getUser(...) properly, because otherwise you'd get a syntax error. It's probably that you're not sending the right parameter to it, so otherFunctionFarAway(...) returns nothing/undefined/null.
Edit
After your comment regarding overriding the user var in filetwo, a solution would be, something along the lines:
create and export another function in filetwo which will export the correct user variable
in sendUpdate(...), before calling the present getUser(note) function, make another call to the above function
call getUser() with the result from point 2.
I have a callback, where I need to get the array out of my callback.
I am trying to return the awaited array into the predefined one.
let documents = [];
socket.on('echo', async function(documents, data){
documents = await data;
console.log(documents); // shows me, what I want to see
});
console.log(documents); // empty Array
I need the result in my predefined Array documents
I have read several Tuts, but I dont get it. I know on stackoverflow it is sked several times. But all threads seem to be more complex then my situation. So I hope to get it cleared out with an more incomplex one.
You need to understand something first. When this runs what is inside the callback doesn't run unit the server will emit that event, in your case 'echo'.
What I think you want to do is use documents outside the callback. You can create a function and call it when the event is emitted.
Something like this:
const manageDocuments = (documents) => {
// do what you want with documents array
console.log(documents);
}
socket.on('echo', async function(documents, data){
documents = await data;
manageDocuments(documents);
});
Of course you can also get rid of the async/await
let documents = await socket.on('echo', async function(documents, data){
console.log(documents);
return data;
});
console.log(documents);
The problem is that the code outside the socket function executes with the empty array because is only executed once at runtime.
If you want to have access to the documents inside the socket function you have to make then persist, or use the socket.on inside of another loop.
I'd like to call multiple times the same API but with different keys to get results faster.
The thing is I need to not wait to receive the result from the first call to start the second call, etc...
The steps are :
1) I have an array with all the different keys.
2) This gets data from the API ("APIKeys" is the array that contains all the keys) :
_.map(APIKeys,function(value, index){
var newCount = count+(25*index);
parseResult(Meteor.http.get("http://my.api.com/content/search/scidir?query=a&count=25&start="+newCount+"&apiKey="+value+""));
});
3) I call a function (named "parseResult") that will format and filter the result I get from the API and save it into the database.
I want to call the function (step 3) without having to wait that I get the data from the API and continue with the other keys while the request is being made.
Do you know how I could do that with meteor ?
Thanks
Do something like this to use HTTP.get() in an async manner:
HTTP.get("http://my.api.com/content/search/scidir?query=a&count=25&start="+newCount+"&apiKey="+value+"", function (error, result) {
// parse the result here
});
And see the docs here:
http://docs.meteor.com/#/full/http_get