Calculating sum of multiple async callbacks - javascript

I am still learning Javascript properly and read a few threads on here about asynchronity (1 2).
Unfortunately, I have still trouble to find a solution to my problem. I am calling an API in async fashion and would like to create a sum out of the response that I am receiving of multiple async calls.
Let's say I want my code to perform an action if my "sum" has a value of 40. Async call "A" returns 6, async call "B" returns 8, and so on until we finally reach the value of 40.
How can I create such a sum, as my calls are all async?
Would it require to write each async result to a database and pull the value up in the next async call?
Is there a better solution for this?
Thank you for your help.
EDIT: To make things easier to understand I will add some source code:
Webhook.js
router.post('/', (req, res, next) => {
if (middleware.active)
middleware.handle(req.body) // <--- this gives me one result
res.sendStatus(200)
});
Basically, I will receive multiple webhook calls. "middleware.handle" will send an API call to third party application. I want to take the result from that API call and add it to the result of another API call from another webhook request.
As you can see, I don't know when my webhook will be triggered, neither how many times it will be triggered before reaching my desired total of 40.

You can use Promise.all([apiCallOne, apiCallTwo]).then(values => sum(values))
sum being a function that sums an array of numbers

You can await for both in a do/while loop then do the rest of the logic
async function myFunc() {
let valA = 0;
let valB = 0;
do {
valA += await fetch('https://api.url.com/a');
valB += await fetch('https://api.url.com/b');
} while((valA + valB) < 40 );
You can also do each one in its do/while loop if you want to request value from one only at a time then check condition.

Related

How to improve performance of my app ? Heavy calculations in Nodejs

I am using a trading bot I created using Nodejs (Express), Postgres and React. React is only for the UI and all the work is done with Node. Postgres is used to store in DB trades information.
The application only runs on my localhost.
The workflow is like this :
Fetch the list of every crypto there is on the trading platform (1 external API request)
Loop on each crypto fetched and perform some "heavy" calculations on them (+150 external API requests and call lot of async helper functions)
If one crypto meets required conditions, buy it (2 external API requests per bought crypto) then insert the trade data in DB (1 internal API request)
When all cryptos have been looped through, repeat
I tried to simplify it the most I could because it is a little more complex than that but this is the basic workflow.
This works fine but is really slow. The loop on each crypto (2) takes about a minute to complete (around 2.5 seconds per crypto).
Here is some code to give you an idea of the structure (again, simplified to the max because it uses lot of other helpers functions and is a few thousands lines of code total) :
// Controller :
exports.strategyBacktesting = async (req, res, next) => {
//... Some code ...
const markets = await getMarkets();
const calculationsResults = await calculations(markets);
//... More code ...
res.status(200).json({
// Return calculationResults and other stuff for the UI
)};
}
//----- Files in utils folders
// Called from controller
exports.getMarkets = async () => {
// Get all the listed cryptos via an external API request
// Loop through all the fetched data to filter the cryptos I want to work with
// Return a Promise (array containing all the cryptos I will do the calculations on)
}
// Called from controller
exports.calculations = async (markets) => {
// Loop through all the results from getMarkets() (markets param) (+150 results) = all the cryptos to work with
for (let cryptoPair in markets) ...
// In the loop:
// Call the getPairCandles() function (below) to get all the data needed per crypto :
// Note that the cryptoPair param is the object from the loop
const { /* data needed for the calculation */ } = await getPairCandles(cryptoPair)
// Perform all the needed calculations (one iteration = one crypto) => This is where it takes a long time (around 2.5 seconds per crypto)
// Calls lots of async helper functions and do the calculations
// If some required conditions are met, do :
//- 1 external async API call to buy the crypto
//- 1 external async API call to set a sell limit
//- One internal async API call to save the trade info in the DB
// When the loop is done on the +150 cryptos, return a Promise for the controller to return to UI
}
// Called from calculations() function above
const getPairCandles = async (cryptoPair) => {
// Make an external API request to get specific real-time data about one crypto (the cryptoPair param of the function)
// Loop through some data in the object received to filter the wanted properties
// Return a Promise containing 5 arrays (the data I need to calculate on this crypto)
}
I read about workers and such in Nodejs that could help with heavy calculations. That being said, how could I implement it ? Is it even the good solution in my case ? The thing is, the loop that performs the calculations is the same that contains lots of calls to async functions returning Promises and async API calls. This is the loop that takes around a minute to complete.
So I think I can greatly improve the performance, but don't know how. What can I do about it ? Could someone please guide me in the good direction ?
Thank you very much

Firestore not writing all documents

I'm using forEach to write over 300 documents with data from an object literal.
It works 80% of the time -- all documents get written, the other times it only writes half or so, before the response gets sent and the function ends. Is there a way to make it pause and always work correctly?
Object.entries(qtable).forEach(([key, value]) => {
db.collection("qtable").doc(key).set({
s: value.s,
a: value.a
}).then(function(docRef) {
console.log("Document written with ID: ", docRef.id);
res.status(200).send(qtable);
return null;
})
Would it be bad pratice to just put a 2 second delay?
You are sending the response inside your loop, before the loop is complete. If you are using Cloud Functions (you didn't say), sending the response will terminate the function an clean up any extra work that hasn't completed.
You will need to make sure that you only send the response after all the async work is complete. This means you will have to pay attention to the promises returned by set() and use them to determine when to finally send the response. Leaning how promises work in JavaScript is crucial to writing functions that work properly.
You need to wait for the set() calls to conclude. They return a promise that you should deal with.
For instance, you can do this by pushing the return of set() to a promise array and awaiting for them outside the loop (with Promise.all()).
Or you can await each individual call, but in this case you need to change the forEach() to a normal loop, otherwise the await will not work inside a forEach() arrow function.
Also, you should probably set the response status just once, and outside the loop.

losing internet connection while sending multiple API calls one by one

I want to make multiple API calls one by one and the second one will be dependent on the result of the first one. What if I lost my internet connection in between each call?
Example:
const test1 = async () => {
const var1 = await function1()
// loss of internet connection or browser gets shut down
const var2 = await function2(var1)
}
Will var2 get executed or will it have expected result returned to me? If not, what are the workarounds to make sure two calls get executed?
If you try to call api without connection, you get an error. In your case error will be thrown when you call await someApiCall(). Workaround is to put some kind of retry logic.

How to handle async code within a loop in JavaScript?

I am reaching out to a SongKick's REST API in my ReactJS Application, using Axios.
I do not know how many requests I am going to make, because in each request there is a limit of data that I can get, so in each request I need to ask for a different page.
In order to do so, I am using a for loop, like this:
while (toContinue)
{
axios.get(baseUrl.replace('{page}',page))
.then(result => {
...
if(result.data.resultsPage.results.artist === undefined)
toContinue = false;
});
page++;
}
Now, the problem that I'm facing is that I need to know when all my requests are finished. I know about Promise.All, but in order to use it, I need to know the exact URLs I'm going to get the data from, and in my case I do not know it, until I get empty results from the request, meaning that the last request is the last one.
Of course that axios.get call is asynchronous, so I am aware that my code is not optimal and should not be written like this, but I searched for this kind of problem and could not find a proper solution.
Is there any way to know when all the async code inside a function is finished?
Is there another way to get all the pages from a REST API without looping, because using my code, it will cause extra requests (because it is async) being called although I have reached empty results?
Actually your code will crash the engine as the while loop will run forever as it never stops, the asynchronous requests are started but they will never finish as the thread is blocked by the loop.
To resolve that use async / await to wait for the request before continuing the loop, so actually the loop is not infinite:
async function getResults() {
while(true) {
const result = await axios.get(baseUrl.replace('{page}', page));
// ...
if(!result.data.resultsPage.results.artist)
break;
}
}

Simultaneous API calls with independent callbacks

I'd like to call multiple times the same API but with different keys to get results faster.
The thing is I need to not wait to receive the result from the first call to start the second call, etc...
The steps are :
1) I have an array with all the different keys.
2) This gets data from the API ("APIKeys" is the array that contains all the keys) :
_.map(APIKeys,function(value, index){
var newCount = count+(25*index);
parseResult(Meteor.http.get("http://my.api.com/content/search/scidir?query=a&count=25&start="+newCount+"&apiKey="+value+""));
});
3) I call a function (named "parseResult") that will format and filter the result I get from the API and save it into the database.
I want to call the function (step 3) without having to wait that I get the data from the API and continue with the other keys while the request is being made.
Do you know how I could do that with meteor ?
Thanks
Do something like this to use HTTP.get() in an async manner:
HTTP.get("http://my.api.com/content/search/scidir?query=a&count=25&start="+newCount+"&apiKey="+value+"", function (error, result) {
// parse the result here
});
And see the docs here:
http://docs.meteor.com/#/full/http_get

Categories