I am streaming in a large response dataset from an http request. I am taking the response and parsing out url's that need to be fetched for subsequent data. When I do this without any control flow it just ends up crashing. How would I use a function like async's foreachlimit but with an iterator being a stream rather than an actual array? Am I thinking about this all wrong?
Found out the concept of "pausing" a stream and this served me well.
stream.pause()
stream.resume()
Related
I have code similar to following in my app. I pipe a readable stream into multiple writable streams, that are server response objects:
readableStream.pipe(fs.createWriteStream('/dev/null')); //Adding this at the beginning so readableStream keeps sending the data
//res objects are added later
readableStream.pipe(res1);
readableStream.pipe(res2);
readableStream.pipe(res3);
.
.
.
A readable stream in JS waits for its piped writable streams to be ready for receiving data, then it sends.
There's some questions.
Does a readable stream waits until all its piped writable streams to be ready, or it sends the data even if one of them is ready? For instance, what happens if one of res objects can't get the data?
In what condition a res object can't (not ready to) get the data?
What happens if one of piped writable streams can't get sent chunks?
I don't want any data to be piled up in memory, to prevent memory increasing. Many thanks.
I want to get a response from node js API to client-side on every step the API completes till API completes its full step.
Suppose I call localhost/api/doStepCompleate on my backend(Node js) API will complete multiple steps but want to send every step response to the client-side till the API completes its full task.
Thank you
If this is an http request/response, then you can use res.write() on the server to send partial responses and finally call res.end() when you're done.
A challenge here will be that many clients are not set up to process partial responses. Libraries such as fetch() will wait for the entire response and not notify their caller of partial responses unless you configure the response as a stream and you have client-side code to parse arbitrary chunks of data as it arrives. See here for some idea how that can work.
I like to save the following parameters in a CSV file in single line for HTTP REST call in Jmeter.
Endpoint URL, request body, request header, response code, response body . could someone help with JS code or groovy code. Thanks in advance!
I don't think you will be able to save request and response "in single line" because:
for sure it will have line breaks
most probably it will have delimiters (i.e. CSV stands for "comma-separated values" and if your response will contain a comma - it will create another "column")
If you're going to use JSR223 Test Elements for this it can be done, but the correct work is guaranteed only for a one thread (virtual user), if you have more than 1 user concurrently writing the data into the same file you'll face the race condition resulting in data corruption or loss
So I would recommend considering using Flexible File Writer which provides full freedom to choose what to store, where to store and it "flushes" the metrics periodically in a thread-safe manner.
I have an array in memory (nodejs server side) that I am updating every 10s and a client that do a request every 10s also. The request parses the array to get it in a specific string format. Also, the update process is in a setInterval function.
I want to run a stress test to that endpoint. I thought that if I move the process of parsing the array to string to the place where I am updating the array, then service only will return a string (like a cache) and stress test will not be a problem to pass.
Here, my problem is: if the time required to update my array and parsing is so long until reach the assignation of a new value for the string cached, then client will receive a non correct value from the service because it continues updating. So my question is how can I be sure client will receive the correct value always. That is, how can I avoid race condition in this context.
The good news is; unless you have spawned a worker or another process Node is single threaded. So it is quite impossible for Node (under normal circumstances) to encounter race conditions.
However, from your description it sounds like your are concerned about the asynchronous nature of your http requests.
Client makes a request to the server
Server begins work
10 seconds pass (server is still working)
Client makes another request to the server using outdated information since server isn't done working.
Server returns old data but, at this point it is too late.
Fortunately, there is more good news. Javascript has a TON of built in support for asynchronous programming. Usually you would wrap your requests in a promise to avoid such pitfalls. Resulting in a process that looks like this:
Client makes a request to the server
Server begins work
Client waits until server comes back before continuing
Server finishes work and returns data to client
Client send another request to the server (ad-infinitum)
You can also make your Promises look like synchronous programming that you're used to via the new(-ish) async functions. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
I figured out I can get the timings of the any XHR request from chrome using the performance API like this below which gives an array as an result (shows ho many requests of this resource has been made):
performance.getEntriesByType('resource').filter(item => item.name.includes("https://myurl"))
If I make 20 XHR requests parallely/concurrently, how do I differentiate which request performance is which object. Is there a way to get specific reference? I am using plain XHR request and am planning to use a RXJS stream if I am able to get a reference. Any help is welcome, I am new to this API. I have attached the request and the result screenshot below:
maybe you can add a query param in the url you are getting with a random string and so you can use the query param to match the request with the performance, even this can create server side cache issue.