I was trying to get reader of ReadableStream using getReader() and response.body of fetch.
Even when I am not consuming data by calling read method, the buffer of stream is getting filled and it is even not applying any kind of backpressure and this is leading to huge memory usage. Is there any way to set WaterMark size on the ReadableStream returned by fetch. Is there any other way to control the buffer size of stream.
Related
I have code similar to following in my app. I pipe a readable stream into multiple writable streams, that are server response objects:
readableStream.pipe(fs.createWriteStream('/dev/null')); //Adding this at the beginning so readableStream keeps sending the data
//res objects are added later
readableStream.pipe(res1);
readableStream.pipe(res2);
readableStream.pipe(res3);
.
.
.
A readable stream in JS waits for its piped writable streams to be ready for receiving data, then it sends.
There's some questions.
Does a readable stream waits until all its piped writable streams to be ready, or it sends the data even if one of them is ready? For instance, what happens if one of res objects can't get the data?
In what condition a res object can't (not ready to) get the data?
What happens if one of piped writable streams can't get sent chunks?
I don't want any data to be piled up in memory, to prevent memory increasing. Many thanks.
I have an AudioBuffer client-side that I'd like to AJAX to a express server.
This link shows how a XMLHttpRequest can send/receive binary data - ArrayBuffer.
An ArrayBuffer is different to an AudioBuffer (or so I believe) as I decoded an ArrayBuffer to make the AudioBuffer in the first place. This was done using decodeAudioData() as part of the Web Audio API.
So my question is, can I convert an AudioBuffer back to an ArrayBuffer?
If this is possible, I'd like to then send the ArrayBuffer to my express server and then forward it on to a S3 bucket as shown in this example.
In that example, the code PUTs a Buffer in a S3 bucket created via a fs.readFileSync(). I assume the difference between a Buffer and ArrayBuffer wont be an issue with S3 :S
Use the copyFromChannel method, then convert the Float32Array to an ArrayBuffer.
var myArrayBuffer = audioCtx.createBuffer(2, frameCount, audioCtx.sampleRate);
var anotherArray = new Float32Array();
myArrayBuffer.copyFromChannel(anotherArray,1,0);
I am streaming in a large response dataset from an http request. I am taking the response and parsing out url's that need to be fetched for subsequent data. When I do this without any control flow it just ends up crashing. How would I use a function like async's foreachlimit but with an iterator being a stream rather than an actual array? Am I thinking about this all wrong?
Found out the concept of "pausing" a stream and this served me well.
stream.pause()
stream.resume()
Is there any way I can query the image from SalesForce server as a blob object ?
We already have forcetk client queries which is retrieving all the data, but the image alone is returned as a link (salesforce link).
Can we retrieve image as blob in the same REST query call ?
The methods I saw required to make an extra call to fetch images, but here I have images in each row of the result, it would have been better if images come as a part of result object itself.
If you're using the REST API, then the blob data can't be returned with the rest of the object, it has to be a separate request. If you use SOAP then it can inline the blob data with the rest of the fields for the object, but it will be limited to returning one row at a time.
I am currently researching the possibility of reading partial XHR responses with binary data. Our current approach is based on the 'responseText' property and base64 encoding. Clearly, this is far from optimal.
How could we read partial Blob/ArrayBuffer responses using XHR? When I try in Chrome, the entire ArrayBuffer/Blob is made available when readyState = 4, but not before that.
To summarize, it seems to me that:
Reading XHR's responseText property: Responses can be read before readyState = 4, and we can stream base64 encoded binary data back to the client
Reading XHR's response property with responseType = 'arraybuffer': No partial response reading, but the entire buffer is made available when readyState = 4
Am I missing something here? What approach could we take to read partial binary responses?
Keep your eyes on the fetch API, currently supported by Firefox and Chrome.
There is a way, though it's not standard yet. Firefox allows you to set responseType on XHR to "moz-blob", "moz-chunked-text" or "moz-chunked-arraybuffer", depending on which one works for you. Then, when you listen for the progress event, you will be able to access partial data as it comes in. MDN has more info on that here and here.
Chrome will support the Streams API, but it's not ready yet. Firefox may also eventually support it. I read somewhere that IE does already, though I can't seem to find any official documentation to confirm that.
The best API to use as an XHR replacement is fetch with readableStream.
This is explained here:
https://developers.google.com/web/fundamentals/primers/async-functions#example_streaming_a_response
Chrome already supports it. Firefox implements it but it must be activated manually for the moment (it will be activated by default in a future version).
While waiting for this activation, Firefox implements the XHR with the non-standard response type moz-chunked-arraybuffer
The library https://www.npmjs.com/package/fetch-readablestream proposes an API that implements these two methods. It uses https://www.npmjs.com/package/web-streams-polyfill.