We Keep Coding

Using RxJS for unknown number of consequtive HTTP Requests

I need to fetch a large number of data points from our API.
These can't however all be fetched at once, as the response time would be too long, so I want to break it into multiple requests. The response looks something like this:
{
href: www.website.com/data?skip=0&limit=150,
nextHref: www.website.com/data?skip=150&limit=150,
maxCount: 704,
skip: 0,
count: 150,
limit:150,
results: [...]
}
So, ultimately I need to continually call the nextHref until we actually reach the last one.
After each request, I want to take the results and concatenate them into a list of data, which will be updated on the UI.
I am relatively new to the world of Obervables but would like to create a solution with RxJS. Does anyone have an idea of how to implement this?
The part that gets me the most is that I don't know how many requests I will have to do in advance. It just needs to keep looping until it's done.

It looks like you can determine the number of calls to make after the first response is received. So, we can make the first call, and build an observable that returns the results of the "first call" along with the results of all subsequent calls.
We can use scan to accumulate the results into a single array.
const results$ = makeApiCall(0, 150).pipe(
switchMap(firstResponse => {
const pageCount = Math.ceil(firstResponse.maxCount / firstResponse.limit);
const pageOffsets = Array(pageCount - 1).fill(0).map((_, i) => (i + 1) * firstResponse.limit);
return concat(
of(firstResponse),
from(pageOffsets).pipe(
mergeMap(offset => makeApiCall(offset, firstResponse.limit), MAX_CONCURRENT_CALLS)
)
);
}),
scan((acc, cur) => acc.concat(cur.results), [])
);
Here's a breakdown of what this does:
we first call makeApiCall() so we can determine how many other calls need made
from creates an observable that emits our array of offsets one at a time
mergeMap will execute our subsequent calls to makeApiCall() with the passed in offsets and emit the results. Notice you can provide a "concurrency" limit, to control how many calls are made at a time.
concat is used to return an observable that emits the first response, followed by the results of the subsequent calls
switchMap subscribes to this inner observable and emits the results
scan is used to accumulate the results into a single array
Here's a working StackBlitz demo.

Unable to send data through callbacks in Javascript

I am using NodeJs, Express and async libarary. I am trying to fetch data from elasticsearch and return that information to the user.
Here is my code:
1. // function that generates the ES query I want
const generateEsQuery = (source,target)=> (callback)=>{
let query = {} // some query that I generated
callback(null,callback)
}
2. // functions that I want to call after the fetching the query
const readElasticData =(data)=> async(callback) =>{
const trafficData =await elasticClient.search({"ignoreUnavailable":true,index:data.indexes,body:{"query":data.query}},{ignore: [404]});
callback(null ,trafficData)
}
async function readEsData (data,callback){
const trafficData =await elasticClient.search({"ignoreUnavailable":true,index:data.indexes,body:{"query":data.query}},{ignore: [404]});
callback(null ,trafficData)
}
3. // Calling my funtions
function myFunction(information){
// async.waterfall([generateEsQuery(information[0],information[1]),readElasticData],// The second function doesnt even run
async.waterfall([generateEsQuery(information[0],information[1]),readEsData] // the second functions runs but doesn't return the results to the callback
function(err, results) {
console.log("All Results",results);
return results;
});
}
I have two functions one to generate a ElasticQuery (1) and another to execute the query (2), I am trying to use the waterfall function of async library to execute them once after another , but I am unable to fetch the results at the final callback.
Out of the two functions that I use to read data , the second one "readEsData" atleast runs after the "generateEsQuery" function. I am new to Node js and I am not sure what I am doing wrong.I also do not want to combine the two functions as I will be reusing the first one in other places.
I am following this approach : https://medium.com/velotio-perspectives/understanding-node-js-async-flows-parallel-serial-waterfall-and-queues-6f9c4badbc17
Question : How do I send the result of the second function through the callback.

Calculating sum of multiple async callbacks

I am still learning Javascript properly and read a few threads on here about asynchronity (1 2).
Unfortunately, I have still trouble to find a solution to my problem. I am calling an API in async fashion and would like to create a sum out of the response that I am receiving of multiple async calls.
Let's say I want my code to perform an action if my "sum" has a value of 40. Async call "A" returns 6, async call "B" returns 8, and so on until we finally reach the value of 40.
How can I create such a sum, as my calls are all async?
Would it require to write each async result to a database and pull the value up in the next async call?
Is there a better solution for this?
Thank you for your help.
EDIT: To make things easier to understand I will add some source code:
Webhook.js
router.post('/', (req, res, next) => {
if (middleware.active)
middleware.handle(req.body) // <--- this gives me one result
res.sendStatus(200)
});
Basically, I will receive multiple webhook calls. "middleware.handle" will send an API call to third party application. I want to take the result from that API call and add it to the result of another API call from another webhook request.
As you can see, I don't know when my webhook will be triggered, neither how many times it will be triggered before reaching my desired total of 40.

You can use Promise.all([apiCallOne, apiCallTwo]).then(values => sum(values))
sum being a function that sums an array of numbers

You can await for both in a do/while loop then do the rest of the logic
async function myFunc() {
let valA = 0;
let valB = 0;
do {
valA += await fetch('https://api.url.com/a');
valB += await fetch('https://api.url.com/b');
} while((valA + valB) < 40 );
You can also do each one in its do/while loop if you want to request value from one only at a time then check condition.

API using Hapi & Async - How to empty array / execute something right after "reply" or at each new "get"?

I'm building a small API that fetches data and perform tasks on it (using async), stores some of this data in an array using push, and then shows it to a client with Hapi's reply().
I'm looking to empty my array (for example using arrayname.length = 0) right after the server has sent an answer to the client.
Current code follows this logic:
let data = []
server.route({
method: 'GET',
path: '/api/{data}',
handler: function (request, reply) {
async.parallel([
function(callback) {
// fetch some data, work with it and then add it to our array using push
data.push ({ // example data
bla:'blabla',
number:'10'
});
callback();
},
function(callback) { // another time (...)
data.push ({
bla:'blabla',
number:'2'
});
callback();
}
],
function(err) { // This is the final callback, sort data and send it using reply
data.sort(function (a, b) {
return a.number - b.number
})
reply(data)
console.log(request.params.data)
})
}
});
server.start((err) => {
if (err) {
throw err;
}
console.log(`Server running at: ${server.info.uri}`);
});
With the current code if I refresh my page, data gets added to the already existing data array.
I tried for several hours and find no way to empty this array right after reply(data) has been called.
Is there any way to do that, or would I have to nest my parallel async functions with an async serie and perform the emptying that way? (Haven's been able to succeed either + sounds overkill).
Is there a way to simply execute something once "reply" has been called or when there is a new "get" so that each client can have the only the data generated by it's request and not also the data that was in the array before?
Thanks!

I'm in agreement with all of the answers suggesting to put the array within the handler. With the array situated out side of the request handler, all request coming through will be writing to the same array. Even if you clear this array at the end of your handler logic, it is not guaranteed to be empty for the next request context since you can have many request context in play.

Following #piotrbienias's advice I created the array inside handler rather than at the beginning of my file and it's cleared each time I do a new request on my API, so I don't need to empty it once the reply is sent.
Thanks, #piotrbienias!

AngularJs: $http.get request inside another $http.get request

I'm really new to AngularJS, and actually relatively new to programming altogether. Basically, I want to request the JSON api from Jenkins, for the list of running jobs in 2 different folders. Inside that data, there is a url to each individual job, which I want to get the data for, as well. So, I need to do another $http.get request for each job, passing the url (which is a value inside the data of the first request) as the parameters.
Initially, I had one request inside another, inside a couple loops to iterate between
The folders, and in each folder
The jobs.
After doing some research, I realized that due to $http requests being async, and for loops being sync, that method was not going to work. So I have a service, which, using $q, collects the promise of the first request, but I don't know how to use the data from the first request as a parameter for the second request. Can someone help, please?

So assume you have two calls, a and b. They both return a list of Jenkins jobs. You can group these using an array of promises and then using $q.all(promises) to group these responses:
var jenkinsPromises = [];
// You could loop over this part if you have an array of calls for example.
jenkinsPromises.push($http.get{ // call folder one });
jenkinsPromises.push($http.get{ // call folder two });
// Now wait for all calls to finish and iterate over their responses.
$q.all(jenkinsPromises).then(function (jenkinsResponses) {
// jenkinsResponses is an array with objects
// each object being a response from Jenkins
}
In the above example, jenkinsResponses, you will find a combined result of the, lets say 'first layer' calls to Jenkins. This array contains response-objects which, as you say, contain the urls you need to make calls to.
Using the same practise as above, you can then group the calls in a promise array. Then we use $q.all() again to group their responses.
$q.all(jenkinsPromises).then(function (jenkinsResponses) {
var i, j, current, urlPromises = [];
for (i = 0, j = jenkinsResponses.length; i < j; i++) {
current = jenkinsResponses[i];
// Push the call to the urlPromises array.
// Note that this is a dummy call.
urlPromises.push($http.get{ current.url] };
}
$q.all(urlPromises).then(function (urlResponses) {
// urlResponses contains a result of all calls to the urls.
}
}

You should use async waterfall https://github.com/caolan/async for async request.
Example:
async.waterfall([
function(callback) {
callback(null, 'one', 'two');
},
function(arg1, arg2, callback) {
// arg1 now equals 'one' and arg2 now equals 'two'
callback(null, 'three');
},
function(arg1, callback) {
// arg1 now equals 'three'
callback(null, 'done');
}
], function (err, result) {
// result now equals 'done'
});

If I'm understanding right, this is the same thing I had to learn in the last few months. Basically, this:
requestFunction(...)
.then(function(response)
{
nextRequest(...).then(...);
}
);
The catch and finally methods are often implemented for this sort of control flow, which is called promises. It's worth looking into; I personally can't stand AngularJS, but understanding it is important to my day job, and this is pretty fundamental to it.

If I unterstand you need to loop twice. So, in your first loop you call your first request. And in the callback of this request, you can put the code for your second loop.
e.g:
for(var i=0; i<cpt1; cpt1++){
request1.then(function(response1){
for(var j=0; j<cpt2; cpt2++){
request2.then(function(response2){
});
}
});
}