What are the limits to pipe a stream to server response objects? - javascript

I have code similar to following in my app. I pipe a readable stream into multiple writable streams, that are server response objects:
readableStream.pipe(fs.createWriteStream('/dev/null')); //Adding this at the beginning so readableStream keeps sending the data
//res objects are added later
readableStream.pipe(res1);
readableStream.pipe(res2);
readableStream.pipe(res3);
.
.
.
A readable stream in JS waits for its piped writable streams to be ready for receiving data, then it sends.
There's some questions.
Does a readable stream waits until all its piped writable streams to be ready, or it sends the data even if one of them is ready? For instance, what happens if one of res objects can't get the data?
In what condition a res object can't (not ready to) get the data?
What happens if one of piped writable streams can't get sent chunks?
I don't want any data to be piled up in memory, to prevent memory increasing. Many thanks.

Related

How to upload and download any file in the browser?

This is part of an experiment I am working on.
Let's say I upload a file eg: .psd (photoshop file) or .sketch (sketch) through the input type file tag, it displays the name of the file and can be downloaded as a .psd / .sketch on click of a button (without data corruption)
How would this be achieved?
Edit 1:
I'm going to add a bit more info as the above was not completely clear.
This is the flow:
User uploads any file
File gets encrypted in the client before sending to a sockets.io server
User on the other end receives this file and is able to decrypt and download.
Note: There is not database connected with the sockets.io. It just listens and responds to whoever connected to the server.
I got the enc/dec part covered. Only thing is uploading and store as ? in a variable so it can be encrypted and doing the opposite on the recepient end (dec and downlodable)
Thanks again in advance :)
I think these are your questions:
How to read a file that was opened/dropped into a <file> element
How to send a file to a server
How to receive a file from a server
When a user opens a file on your file element, you'll be able to use its files property:
for (const file of fileInputEl.files) {
// Do something with file here...
}
Each file implements the Blob interface, which means you can call await file.arrayBuffer() to get an ArrayBuffer, which you can likely use directly in your other library. At a minimum, you can create your byte array from it.
Now, to send data, I strongly recommend that you use HTTP rather than Socket.IO. If you're only sending data one way, there is no need for a Web Socket connection or Socket.IO. If you make a normal HTTP request, you offload all the handling of it to the browser. On the upload end, it can be as simple as:
fetch('https://files.example.com/some-id-here', {
method: 'PUT'
body: file
});
On the receive end, you can simply open a link <a href="https://files.example.com/some-id-here">.
Now, the server part... You say that you want to just pass this file through. You didn't specify at all what you're doing on the server. So, speaking abstractly, when you receive a request for a file, you can just wait and not reply with data until the sending end connects and start uploading. When the sending end sends data, send that data immediately to the receiving end. In fact, you can pipe the request from the sending end to the response on the receiving end.
You'll probably have some initial signalling to choose an ID, so that both ends know where to send/receive from. You can handle this via your usual methods in your chat protocol.
Some other things to consider... WebRTC. There are several off-the-shelf tools for doing this already, where the data can be sent peer-to-peer, saving you some bandwidth. There are some complexities with this, but it might be useful to you.

MarkLogic 8 - Stream large result set to a file - JavaScript - Node.js Client API

Let's say I have a query that is going to return a very large response. Possibly thousands of records and possibly gigabytes of data.
Normally in the UI, we just show a single page of this data. Now I need an option to take the entire result set and stream it out to a file. Then the user can go download this at their leisure.
So how do I select all results from a query using query builder and then stream it out to a file in chunks without running out of memory?
If you want the document descriptors, you can open an object stream as in the following example:
https://github.com/marklogic/node-client-api/blob/develop/examples/query-builder.js#L38
If you only want the content of the documents, you can use a chunked stream as shown in the following example (the same approach can be used for a query):
https://github.com/marklogic/node-client-api/blob/develop/examples/read-stream.js#L27
The general approach would be as follows:
open the destination file as a write stream
https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options
query for the first page of documents, piping the read stream for the documents to the write stream for the file, taking care to set the end option to false:
https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options
loop on reading documents, incrementing the start page by the page length until finished reading
call end() on the write stream to close the file
https://nodejs.org/api/stream.html#stream_writable_end_chunk_encoding_callback
Hoping that helps

NodeJS async foreachlimit on stream

I am streaming in a large response dataset from an http request. I am taking the response and parsing out url's that need to be fetched for subsequent data. When I do this without any control flow it just ends up crashing. How would I use a function like async's foreachlimit but with an iterator being a stream rather than an actual array? Am I thinking about this all wrong?
Found out the concept of "pausing" a stream and this served me well.
stream.pause()
stream.resume()

Sending a file chunk WITH meta data over a WebRTC Data Channel

I have a file chunking operation that splits a file via File Reader into slices which are read via readAsArrayBuffer. I would like to send those chunks one at a time over my data channel WITH meta information attached (a chunk id, for instance). Like:
// Build chunk wrapper
var block = {
chunkId: id,
data: buffer
};
// Send the chunk to peer
channel.send(JSON.stringify(block));
Now when I send that data as is demonstrated above the data in the ArrayBuffer buffer is lost. I would like to emphasize I'm not having any trouble sending data over my data channel.
I would like to know how I can send that data with its associated meta information so that the file chunks can be reassembled in the correct order on the other side?
Do I need to do something like make an ArrayBuffer with two sub arrays, one with the meta information, and the other with the actual data or is there a simpler way?
There are many ways you can solve this, but basically you'll need to serialize, encode and deserialize, decode your data.
If you want to send your metadata with the data, you'll need to either serialize both to a uint8array or to a string, and do the inverted operation on the receiver side.
For example Sharefest utilizes a TLV protocol: https://github.com/Peer5/ShareFest/blob/master/core/protocol/BinaryProtocol.js

how to parse both data & binary websocket messages?

I've found this great page that shows how to stream video.
His code assumes that the onmessage data is jpg like so
ws.onmessage = function (e) {
$("#image").attr('src', 'data:image/jpg;base64,'+e.data);
}
but I'd like to have audio and other page data as well.
How can WebSockets be made to parse message types and binary types?
The data in Websocket messages can be either string (text messages) or binary data. In your case it's string data, where the string content is the base64 encoding of a binary image.
In a better optimized program the image could be also transferred binary.
Depending on the message type, e.data will be of another type.
If the message is a text message then e.data will be of type string.
If the message is a binary message then e.data will contain either an ArrayBuffer or a Blob object. You can by yourself decide which representation of binary data you want to receive by setting the WebSocket.binaryType property (e.g. to "arraybuffer").
You can use instanceof to check if you received a binary or text message.
If you want to transfer different types of binary data you can use a header at the start of the message to tell the client what the message contains. E.g. declare the first byte of the binary data as header. If the first byte contains 1 then the remaining data is a picture, if the first byte contains 2 then the remaining data is a short audio sample, etc.
However it's a little more difficult on both sides since you need to split or combine the binary data array.
Alternatively you could use multiple parallel websocket connections between client and server. On for video data, one for audio data, one for general messages, etc.

Categories