I have a file chunking operation that splits a file via File Reader into slices which are read via readAsArrayBuffer. I would like to send those chunks one at a time over my data channel WITH meta information attached (a chunk id, for instance). Like:
// Build chunk wrapper
var block = {
chunkId: id,
data: buffer
};
// Send the chunk to peer
channel.send(JSON.stringify(block));
Now when I send that data as is demonstrated above the data in the ArrayBuffer buffer is lost. I would like to emphasize I'm not having any trouble sending data over my data channel.
I would like to know how I can send that data with its associated meta information so that the file chunks can be reassembled in the correct order on the other side?
Do I need to do something like make an ArrayBuffer with two sub arrays, one with the meta information, and the other with the actual data or is there a simpler way?
There are many ways you can solve this, but basically you'll need to serialize, encode and deserialize, decode your data.
If you want to send your metadata with the data, you'll need to either serialize both to a uint8array or to a string, and do the inverted operation on the receiver side.
For example Sharefest utilizes a TLV protocol: https://github.com/Peer5/ShareFest/blob/master/core/protocol/BinaryProtocol.js
Related
I know that blob is a data type for binary data as integer is a datatype for int. As they say, It's used to store files directly in database (we move our audio file into blob, and save that blob in database).
Question 1) why to store blob for audio if I can just put the audio in storage for example path /var/www/audio.mp3 and in database I store path_name /var/www/audio.mp3?
Question 2) which is better ? how netflix stores movies? just blobs or what?
Question 3) Curious if there're any cons or prons if you could just give me ideas so that I know when to use them .
Putting the blob in the database, rather than a file, allows you to grow to multiple servers with load balancing. If you put the data in files, you would have to replicate the files between the server. Most databases have built-in replication features, this isn't as easy for regular files.
Better to use external storage/cdn for serving such kind of large content.
How Netflix and our works? They upload content on external bucket i. e. S3 and write file name in db for identification. According to user file access frequency that file cache on CDN/edge location. User will get awesome experience while content server from their nearest edge location
With blob you can store all kinds of stuff.
Do you communicate with an API via SOAP or JSON and want to store it in the database? Use a blob. Want to log what a user filled into a form when it threw an exception? Store the entire post as a blob. You can save everything as is. It's handy for logging if you have different data formats. I know an API which expects some data via SOAP and some as JSON. To log the communication I use blob because the response may be in XML, JSON, a number (http code 203 for empty but accepted) or an exception as array.
I have a GraphQL server, hosted on express. I want to return images to the client by sending back nodejs buffer objects. How can i config graphql server, to return bytes, instead of json? I don't wish to do this through base64, as the image are large in size.
You have to return JSON, but there's still a way. We're using GraphQL to return images stored in Blob fields in a legacy sql database. We're using sequelize, graphql-sequelize, and graphql-js.
We've defined the Blob fields to be String types in our graphql schema and so they come through in the json reply just fine.
Then we convert to a buffer before delivering, like
const imgBuffer = new Buffer.from(imgObj.data, 'ascii');
The only problem is we're now having trouble saving image data back to the database via the graphql interface. Our mutation function gives us a syntax error when it finds certain bad unicode characters in the strings, like \U0000 and whatnot (so I found your question looking for a solution to that).
There's a way, but it's complicated, and very manual, and I'm only going to give you an overview of what I've done in ApolloServer, but I think it should be enough.
First, you need to use the "Accept" header in your request to send a binary mime type, and send a matching "Content-Type" in your response. This is nessisary to be efficient, but not nessisary to work, as you'll see (with EJSON).
To serialize and deserialize respecting the headers you may need to write an express middleware, and you'll need to handle base64 encoding with a {$data: "..."} encapsulating object (as EJSON does) or just (strangely) returning null, if someone makes a request for binary data using "application/json" for their "accept" header. You'll also want to choose what binary formats that you'll support. I only use 1: "application/x-msgpack", but I hear that "application/cbor" is becoming more popular. You can use a library for EJSON, MessagePack, and CBOR to do your serialization, so this isn't as hard as it sounds.
I would then strongly recommend using the #defer on any images. See this post for more information on #defer: https://www.apollographql.com/blog/introducing-defer-in-apollo-server-f6797c4e9d6e/
I've done it. It wasn't easy, and it would be better if ApolloServer worked this way "out of the box".
It's better to send a hashed & temporary link to download it
The URL can be hashed to be non-accessible by other users.
Backend can expire the link or remove the binary file on the static server.
There might be an answer to your question by using the node module found here.
Let's say I have a query that is going to return a very large response. Possibly thousands of records and possibly gigabytes of data.
Normally in the UI, we just show a single page of this data. Now I need an option to take the entire result set and stream it out to a file. Then the user can go download this at their leisure.
So how do I select all results from a query using query builder and then stream it out to a file in chunks without running out of memory?
If you want the document descriptors, you can open an object stream as in the following example:
https://github.com/marklogic/node-client-api/blob/develop/examples/query-builder.js#L38
If you only want the content of the documents, you can use a chunked stream as shown in the following example (the same approach can be used for a query):
https://github.com/marklogic/node-client-api/blob/develop/examples/read-stream.js#L27
The general approach would be as follows:
open the destination file as a write stream
https://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options
query for the first page of documents, piping the read stream for the documents to the write stream for the file, taking care to set the end option to false:
https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options
loop on reading documents, incrementing the start page by the page length until finished reading
call end() on the write stream to close the file
https://nodejs.org/api/stream.html#stream_writable_end_chunk_encoding_callback
Hoping that helps
I've found this great page that shows how to stream video.
His code assumes that the onmessage data is jpg like so
ws.onmessage = function (e) {
$("#image").attr('src', 'data:image/jpg;base64,'+e.data);
}
but I'd like to have audio and other page data as well.
How can WebSockets be made to parse message types and binary types?
The data in Websocket messages can be either string (text messages) or binary data. In your case it's string data, where the string content is the base64 encoding of a binary image.
In a better optimized program the image could be also transferred binary.
Depending on the message type, e.data will be of another type.
If the message is a text message then e.data will be of type string.
If the message is a binary message then e.data will contain either an ArrayBuffer or a Blob object. You can by yourself decide which representation of binary data you want to receive by setting the WebSocket.binaryType property (e.g. to "arraybuffer").
You can use instanceof to check if you received a binary or text message.
If you want to transfer different types of binary data you can use a header at the start of the message to tell the client what the message contains. E.g. declare the first byte of the binary data as header. If the first byte contains 1 then the remaining data is a picture, if the first byte contains 2 then the remaining data is a short audio sample, etc.
However it's a little more difficult on both sides since you need to split or combine the binary data array.
Alternatively you could use multiple parallel websocket connections between client and server. On for video data, one for audio data, one for general messages, etc.
Is there any way I can query the image from SalesForce server as a blob object ?
We already have forcetk client queries which is retrieving all the data, but the image alone is returned as a link (salesforce link).
Can we retrieve image as blob in the same REST query call ?
The methods I saw required to make an extra call to fetch images, but here I have images in each row of the result, it would have been better if images come as a part of result object itself.
If you're using the REST API, then the blob data can't be returned with the rest of the object, it has to be a separate request. If you use SOAP then it can inline the blob data with the rest of the fields for the object, but it will be limited to returning one row at a time.