I'm guessing this should be somewhat easy to achieve but I've having trouble (conceptually, I guess) figuring out how to tackle it.
What I have is an API that returns an array of JSON objects. I need to step through these objects, and, for each object, make another AJAX call. The issue is the system that handles each AJAX call can only handle two active calls at a time (as it's quite a CPU-intensive task that hooks out into a desktop application).
I was wondering how I could achieve this using RxJS (either using version 5 or 4)?
EDIT: In addition, is it possible to have a chain of steps running concurrently. i.e.
Downloading File: 1
Processing File: 1
Converting File: 1
Uploading File: 1
Downloading File: 2
Processing File: 2
Converting File: 2
Uploading File: 2
Downloading File: 3
Processing File: 3
Converting File: 3
Uploading File: 3
I've tried doing something like:
Rx.Observable.fromPromise(start())
.concatMap(arr => Rx.Observable.from(arr))
.concatMap(x => downloadFile(x))
.concatMap((entry) => processFile(entry))
.concatMap((entry) => convertFile(entry))
.concatMap((entry) => UploadFile(entry))
.subscribe(
data => console.log('data', new Date().getTime(), data),
error => logger.warn('err', error),
complete => logger.info('complete')
);
However that doesn't seem to work. The downloadFile, for example doesn't wait for processFile, convertFile and uploadFile to all complete, rather, the next one will run again as soon as the previous one completes.
Here are 2 approaches, if you want the sequence of requests exactly like this
Downloading File: 1
Processing File: 1
Converting File: 1
Uploading File: 1
Downloading File: 2
Processing File: 2
...
You need to resolve all promises inside single concatMap method, like this
Rx.Observable.fromPromise(getJSONOfAjaxRequests())
.flatMap(function(x) { return x;})
.concatMap(function(item) {
return downloadFile(item)
.then(processFile)
.then(convertFile);
})
.subscribe(function(data) {
console.log(data);
});
see the working plunkr here: https://plnkr.co/edit/iugdlC2PpW3NeNF2yLzS?p=preview
This way, the new ajax call will be sent only when the previous is finished.
Another approach is that allow the files to send requests in parallel but the operations 'downloading,processing,converting,uploading' will be in sequence. For this you can get it working by
Rx.Observable.fromPromise(getJSONOfAjaxRequests())
.flatMap(function(x) { return x;})
.merge(2) // in case maximum concurrency required is 2
.concatMap(function(item) {
return downloadFile(item);
})
.concatMap(function(item) {
return processFile(item);
})
.concatMap(function(item) {
return convertFile(item)
})
.subscribe(function(data) {
//console.log(data);
});
see plunkr here: https://plnkr.co/edit/mkDj6Q7lt72jZKQk8r0p?p=preview
You could use merge operator with the maxConcurrency overload (Rxjs v4), so something like :
Rx.Observable.fromArray(aJSONs)
.map (function (JSONObject) {
return ajaxRequest(JSONObject) // that would be an observable (or a promise)
})
.merge(2)
You can have a look to see other examples of use at :
Limit number of requests at a time with RxJS,
or How to limit the concurrency of flatMap?
Official documentation :
merge(maxConcurrency)
How about something like this? You could use from to break the array into bite sized chunks and process them one by one using concatMap.
function getArr() {
return Rx.Observable.of([1, 2, 3, 4, 5, 6, 7, 8]);
}
function processElement(element) {
return Rx.Observable.of(element)
.delay(500);
}
getArr()
.concatMap(arr => {
return Rx.Observable.from(arr);
})
.concatMap(element => {
return processElement(element);
})
.subscribe(res => {
console.log(res);
});
Old post but I believe this could work, for console log we could use tap. Note editor would through intellisense error since from expects an array, but the code should work.
from(start()).pipe(
switchMap(files => from(files).pipe(
switchMap(file => from(downloadFile(file)).pipe(
map(_ => ({file: file, downloaded: true}))
)),
switchMap(attr => from(processFile(attr.file)).pipe(
map(_ => ({...attr, downloaded: true}))
)),
switchMap(attr => from(convertFile(attr.file)).pipe(
map(_ => ({...attr, converted: true}))
)),
switchMap(attr => from(uploadFile(attr.file)).pipe(
map(_ => ({...attr, uploaded: true}))
))
))
).subscribe(_ => {})
Related
I'm looking to simplify my project's testing API where I am aiming for something like this:
testThing((t) => {
t(33);
t(44);
t(42);
})
Now I don't know how to get Jest to show the correct code frames for failed expect's. This is my current stab at an implementation:
const testThing = (callback: any) => {
callback((n: any) => {
test(n.toString(), () => {
expect(n).toBe(42);
});
});
};
Which results in the testThing definition to be shown for every failed test case. Here's a replit if you want to see it in action: https://replit.com/#grgr/jest-frame-issue#thing.test.js
I've finished writing my first Cypress test. Everything is good except I'm struggling to post the result data to a website. Because I want to send the result data and also if any errors occurs the result screenshot to our coworker telegram group.
For the last two days I've tried everything and couldn't find any solution.
I've tried those in my test script (cypress/integration/test.js);
Cypress.on('test:after:run', (test, runnable) => {
console.log('test,runnable', test, runnable)
const details = {
projectKey: Cypress.env('zephyr-project-key'),
testName: test.invocationDetails.relativeFile,
status: test.status,
error: runnable.err.message,
retries: runnable.retries.length,
duration: test.wallClockDuration,
startTime: test.wallClockStartedAt
}
cy.request('POST', 'http://mywebsite.com/notify.php', { body: details })
fetch('http://mywebsite.com/notify.php')
})
Also this didn't work (cypress/plugins/index.js);
module.exports = (on, config) => {
// `on` is used to hook into various events Cypress emits
// `config` is the resolved Cypress config
on('after:run', (results) => {
if (results) {
// results will be undefined in interactive mode
console.log(results.totalPassed, 'out of', results.totalTests, 'passed')
fetch('http://mywebsite.com/notify.php');
}
})
}
Edit: This is day 3 and I still couldn't solve this. What I've seen from Cypress help page is that cy.task() calls do not fire in 'test:after:run' event block;
https://github.com/cypress-io/cypress/issues/4823
I've seen some telegram groups who can do what I'm trying to do. All I need is to be able to get the results and post it to my website.
The third parameter to cy.request() is body, you don't have to wrap it.
Cypress.on('test:after:run', (test, runnable) => {
const details = {
projectKey: Cypress.env('zephyr-project-key'),
testName: test.invocationDetails.relativeFile,
status: test.status,
error: runnable.err?.message, // need err?.message if there is no error
retries: runnable.retries.length,
duration: test.wallClockDuration,
startTime: test.wallClockStartedAt
}
cy.request('POST', 'http://mywebsite.com/notify.php', details) // don't wrap details
.then(res => expect(res.status).to.eq(201)) // confirm result
})
I'm working with a Gulp file. In this file, I have two functions that perform two separate tasks. These functions are called using the following:
gulp.task('build', gulp.parallel(buildJs, buildCss));
When the build task is executed, I receive the following error:
The following tasks did not complete: default, build, buildCss
Did you forget to signal async completion?
My buildCss function is defined like this:
const files= [
{ input:'core.scss', output:'core.css' },
{ input:'theme.scss', output:'theme.css' },
{ input:'controls.scss', output:'controls.css'}
];
function buildCss() {
files.forEach(function(f) {
return gulp.src(`./scss/${f.input}`)
.pipe(sass())
.pipe(concat(f.output))
.pipe(gulp.dest('./'))
;
});
}
I suspect that each iteration of the loop is spinning up it's own thread. So, Gulp never knows when buildCss is finished. Which means, somehow, I need to know when all of the .css files are generated and call something. I'm unsure how to do the last piece if my understanding is correct though.
How do I address async completion of items in a loop in Gulp?
Thank you!
You can use .on('end') and create a Promise for each task, and then check that all promises went ok, sth like this should work:
function buildCss() {
return Promise.all(files.map(task => {
return new Promise((resolve,reject) => {
return gulp.src(`./scss/${task.input}`)
.pipe(sass())
.pipe(concat(task.output))
.pipe(gulp.dest('./'))
.on('end', () => resolve())
;
});
}));
}
Hi i am working on my Angular 7 project. I am getting one response from a api and i want to integrate that response to other api which result success in all other apis.
Here is my code :
ngOnInit() {
this.second()
}
first() {
this.service.getId(id).resp.subscribe((res => {
console.log(res);
this.firstresp = res;
});
}
second() {
this.service.getId(this.firstresp).resp.subscribe((res => {
console.log(res)
});
}
Here the problem is first function executed properly and second function i am getting response only after refreshing the page. Any solution? TIA.
This is actually an RXJS question, not an angular one. You want to use switchMap:
this.service.getId(id).resp.pipe(
switchMap((res) => {
return this.service.getId(res).resp;
})
).subscribe((rep) => {
....
});
switchMap above pipes the result of the first call into the second one then emits that result to the subscribe. No need for the firstresp etc.
I need to call an API recursively using request promise after getting result from API need to write in an excel file , API sample response given below
{
"totalRecords": 9524,
"size": 20,
"currentPage": 1,
"totalPages": 477,
"result": [{
"name": "john doe",
"dob": "1999-11-11"
},
{
"name": "john1 doe1",
"dob": "1989-12-12"
}
]
}
Now I want to call this API n times, here n is equal to totalPages, after calling each API I want to write response result to the excel files.
First write page 1 response result to excel then append page 2 response result to excel file and so on..
I have written some sample code given below
function callAPI(pageNo) {
var options = {
url: "http://example.com/getData?pageNo="+pageNo,
method: 'GET',
headers: {
'Content-Type': 'application/json'
},
json: true
}
return request(options)
}
callAPI(1).then(function (res) {
// Write res.result to excel file
}).catch(function (err) {
// Handle error here
})
But facing problem calling recursively API and maintaining sequentially like write page 1 result first to excel file then page 2 result append to excel and so on..
Any code sample how to achieve in nodejs
You want to do something like this:
function getAllPages() {
function getNextPage(pageNo) {
return callAPI(pageNo).then(response => {
let needNextPage = true;
if (pageNo === 1) {
// write to file
} else {
// append to file
}
if (needNextPage) {
return getNextPage(pageNo+1);
} else {
return undefined;
}
});
}
return getNextPage(1);
}
Obviously change that 'needNextPage' to false to stop the recursion when you're done
So you want to do 477 requests in sequence? How long do you wanna wait for this to finish? Even in paralell, this would be still too long for me.
Best: write an API that can return you a batch of pages at once. Reducing the number of requests to the backend. Maybe something like http://example.com/getData?pages=1-100 and let it return an Array; maybe like
[
{
"totalRecords": 9524,
"currentPage": 1,
"totalPages": 477,
"result": [...]
},
{
"totalRecords": 9524,
"currentPage": 2,
"totalPages": 477,
"result": [...]
},
...
]
or more compact
{
"totalRecords": 9524,
"totalPages": 477,
"pages": [
{
"currentPage": 1,
"result": [...]
},
{
"currentPage": 2,
"result": [...]
},
...
]
}
Sidenote: writing the size of the results array into the json is unnecessary. This value can easily be determined from data.result.length
But back to your question
Imo. all you want to run in sequence is adding the pages to the sheet. The requests can be done in paralell. That already saves you a lot of overall runtime for the whole task.
callApi(1).then(firstPage => {
let {currentPage, totalPages} = firstPage;
//`previous` ensures that the Promises resolve in sequence,
//even if some later request finish sooner that earlier ones.
let previous = Promise.resolve(firstPage).then(writePageToExcel);
while(++currentPage <= totalPages){
//make the next request in paralell
let p = callApi(currentPage);
//execute `writePageToExcel` in sequence
//as soon as all previous ones have finished
previous = previous.then(() => p.then(writePageToExcel));
}
return previous;
})
.then(() => console.log("work done"));
or you wait for all pages to be loaded, before you write them to excel
callApi(1).then(firstPage => {
let {currentPage, totalPages} = firstPage;
let promises = [firstPage];
while(++currentPage < totalPages)
promises.push(callApi(currentPage));
//wait for all requests to finish
return Promise.all(promises);
})
//write all pages to excel
.then(writePagesToExcel)
.then(() => console.log("work done"));
or you could batch the requests
callApi(1).then(firstPage => {
const batchSize = 16;
let {currentPage, totalPages} = firstPage;
return Promise.resolve([ firstPage ])
.then(writePagesToExcel)
.then(function nextBatch(){
if(currentPage > totalPages) return;
//load a batch of pages in paralell
let batch = [];
for(let i=0; i<batchSize && ++currentPage <= totalPages; ++i){
batch[i] = callApi(currentPage);
}
//when the batch is done ...
return Promise.all(batch)
//... write it to the excel sheet ...
.then(writePagesToExcel)
//... and process the next batch
.then(nextBatch);
});
})
.then(() => console.log("work done"));
But don't forget to add the error handling. Since I'm not sure how you'd want to handle errors with the approaches I've posted, I didn't include the error-handling here.
Edit:
can u pls modify batch requests, getting some error, where you are assigning toalPages it's not right why the totalPages should equal to firstPage
let {currentPage, totalPages} = firstPage;
//is just a shorthand for
let currentPage = firstPage.currentPage, totalPages = firstPage.totalPages;
//what JS version are you targeting?
This first request, callApi(1).then(firstPage => ...) is primarily to determine currentIndex and totalLength, as you provide these properties in the returned JSON. Now that I know these two, I can initiate as many requests in paralell, as I'd want to. And I don't have to wait for any one of them to finish to determine at what index I am, and wether there are more pages to load.
and why you are writing return Promise.resolve([ firstPage ])
To save me some trouble and checking, as I don't know anything about how you'd implement writePagesToExcel.
I return Promise.resolve(...) so I can do .then(writePagesToExcel). This solves me two problems:
I don't have to care wether writePagesToExcel returns sync or a promise and I can always follow up with another .then(...)
I don't need to care wether writePagesToExcel may throw. In case of any Error, it all ends up in the Promise chain, and can be taken care of there.
So ultimately I safe myself a few checks, by simply wrapping firstPage back up in a Promise and continue with .then(...). Considering the amounts of data you're processing here, imo. this ain't too much of an overhead to get rid of some potential pitfalls.
why you are passing array like in resolve
To stay consistent in each example. In this example, I named the function that processes the data writePagesToExcel (plural) wich should indicate that it deals with multiple pages (an array of them); I thought that this would be clear in that context.
Since I still need this seperate call at the beginning to get firstPage, and I didn't want to complicate the logic in nextBatch just to concat this first page with the first batch, I treat [firstPage] as a seperate "batch", write it to excel and continue with nextBatch
function callAPI(pageNo) {
var options = {
url: "http://example.com/getData?pageNo="+pageNo,
method: 'GET',
headers: {
'Content-Type': 'application/json'
},
json: true
}
return request(options)
}
function writeToExcel(res){console.log(res)} //returns promise.
callAPI(1).then(function (res) {
if(res){
writeToExcel(res).then(() => {
var emptyPromise = new Promise(res => setTimeout(res, 0));
while(res && res.currentPage < res.totalPages){
emptyPromise = emptyPromise.then(() => {
return callAPI(res.currentPage).then(function (res){
if(res){
writeToExcel(res)
}
});
}
}
return emptyPromise;
});
}
}).catch(function (err) {
// Handle error here
})