I'm writing WebSocket based clients servers.
I want to send raw binary data that is then decoded by the client.
My problem is two-fold:
while doing manipulations on binary data in c++ (the language I'm writing the servers in) is straight forward, it seems hard in Javascript.
I have found that you can use ArrayBuffers and UInt8Arrays to do most of the manipulations, and that works fine for sending messages.
My problem is that when I try receiving the messages if I declare the message as binary on the server side, it shows up as a Blob on the client. I have tried converting the Blob to an ArrayBuffer like so:
ws.onmessage = function(evt) {
var data = null;
data = await new Response(evt.data).arrayBuffer();
}
But this gives me the error:
SyntaxError: await is only valid in async functions and async generators
It seems that this method is asynchronous, and while I'm sure I could do it this way, it doesn't really seem that good in the first place.
I have realized that sending the data as the text makes evt.data appear as a string, which makes me believe that I can use a JSON format for the packets instead of a binary one.
I really don't want to use JSON though, as some of the packets will be full of a lot of information and I'm scared it will add unnecessary bloat.
I think what I really want is just to be able to read the evt.data as an ArrayBuffer, because it seems like that would be the most performant.
Use the then method of the promise
new Response(evt.data).arrayBuffer().then(buffer=> {
//here you have the buffer
})
then
arrayBuffer
Note: The await can be used inside functions declared with async keyword.
Related
I understood why we are using JSON to exchange data between browser and server in place of XML but I could not understand why we are using only string type of JSON even we have six different value datatype, I mean why we can't use integer or Boolean or any other value datatype.
Hope you guys understand what I'm trying to say, Thanks in advance.
If I understand correctly, the limitation is because of the way data needs to be encoded to be sent over HTTP and ultimately over the wire. You json object (or xml,etc) is ultimately just a payload for HTTP (which is just a payload for tcp in turn and so on).
HTTP inherently does not and should not identify data types in payload, it is just an array for HTTP. You can select how to represent this array i.e. how to encode it; It can be string (ascii, utf-8, etc) or binary but it has to be uniform for the whole payload.
HTTP does offer different encoding methods of payload which can be interpreted by the receiver by looking at the content-type header and accordingly decode the data.
Hope this helps.
why we are using only string type of JSON
Uhm, we're not. I believe you're misunderstanding something here. HTTP responses can really contain anything; every time you download a PDF or an image from a web server, the web server is sending a binary payload, which can literally be anything. So it's not even true that all HTTP bodies must be text.
To exchange data between systems, you send bytes. For these bytes to mean anything, you need an encoding scheme. Image formats have a particular way in which bytes need to be arranged, and when properly doing so, you can send pictures with them. Same for PDFs, video, audio, and anything else (including text).
If you want to send structured data, you need to express that structure somehow. How do you send a, for example, PHP array over HTTP…? (Substitute your equivalent list data structure in your language of choice.) You can't. A PHP array is a specific data structure in memory of a PHP runtime, sending that as is over HTTP has no meaning (because it deals with internal pointers and such). This array needs to be serialised first. There are many possible serialisation methods, some of them using binary data, and some using formats which are human readable to varying degrees. You could simply join all array elements with commas and .split(',') them again on the other end, but that's rather simplistic and misses many more complex cases and edge cases.
JSON and XML (and YAML and whatnot) are human readable formats which can serialise data structures like arrays (and dictionaries and numbers and booleans etc), and which happen to be text-based (purposely, to make them developer-friendly). You can use any of those data types JSON allows. Nothing prevents you from doing so, and not using them is insane. JSON and XML also happen to be two formats easily parsed with tools built into every browser. You could use any other binary format too, but then you'd have to manually parse it in Javascript.
Communication between browser and server can be done in many ways. It's just that JSON comes out of the box. You can use protobuf, xml and plethora of other data serialization techniques to communicate with the server as long as both sides understand what's the communication medium. On the browser side, you have to probably implement protobuf, xml etc serialization/deserialization on your own in javascript.
Any valid JSON is permitted for data exchange. The keys are string quoted but the values can be strings, numbers, booleans, array or other objects itself. Though before transmission, everything is converted into a string and the receiving side parses it into the correct format.
I have a large JSON file that I would like to store it in Redis. The problem is when I parse it I run out of memory in Node.Js
I extended the heap memory from 1.39GB from 4GB it's still happening and I believe I am not doing it properly.
With a lot of search, I found out that streaming is my best bet. The thing is I am not really fluent with Streaming and I am not sure even this would resolve my problem
I read a lot and there is a lot of scattered information. I was wondering to ask if you think if this is even approachable or if this is correct?
Would I be able to stream a JSON object into Redis?
Do I have to Staingifiying it or it will be automatically?
Should I stringify my json chunk by chunk?
or streaming into redis will end up being a string?
I am using ioRedis client to interact with Redis.
I appreciate your help in advance.
If you can guarantee that only one processor will be updating that key, you could possibly use SETRANGE. As you parse the file, you can keep a reference to the next offset:
(psuedo-code)
offset = 0
offset = redis.set_range(key, offset, "string")
Then you can load pieces of the file up to Redis without having to load everything into memory at once.
SETRANGE returns the length of the string after it was modified.
This also assumes that you can load pieces of the file contents without having to parse everything as JSON then convert it back to a string. Also assumes that only one process is updating that key -- if multiple processes try to update it, the JSON value can get corrupted.
I'm writing a browser game and am looking for a way to send raw array buffers to and from the node.js server.
I don't like the idea of sending JSON strings over WebSockets because:
I have to specify keys when the receiver already knows the format
There is no validation or checking if you send malformed structure (not required though)
Parsing JSON strings in to objects is inherently slower than reading binary
Wasted bandwidth from the entire payload being a string (instead of packed ints, for example)
Ideally, I would be able to have a schema for every message type, construct that message, and send its raw array buffer to the server. Something like this:
schema PlayerAttack {
required int32 damage;
required int16[] coords;
required string name;
}
var message = new PlayerAttack(5, [24, 32], "John");
websockets.send(message.arrayBuffer());
And it would then arrive at the Node.js server as a Buffer with the option to decode in to an object.
Google's Protocol Buffers almost fits this use-case, but are too slow and have too much overhead (7x slower than JSON.parse from my benchmark and includes features like tagging which I have no use for).
I have a class Message which can be serialized when the data goes through the network, I currently use JSON, mostly because I use JSON for everything. (webservice, sockets).
I want to improve the serialization to make it as good as possible, I believe improvments are possible here.
The aim is to make the transport string lighter, especially when used by sockets (video game), because it will be used for everything, every response client/server or server/client and even inside the server or client methods, it's the usual way to provide data.
The Message is a complex object that can also contain other object instances, like a MessageLang, which will be responsable to translate a sentence on the client based on a code.
So far it works fine, here are the results:
Socket server emit with simple string:
verbose: websocket writing 5:::{"name":"user.newAuthenticated","args":["Respond to emitter"]}
Socket server emit with simple message instance:
verbose: websocket writing 5:::{"name":"user.newAuthenticated","args":["{\"m\":\"Respond to all clients\",\"d\":{},\"s\":1,\"t\":\"m\"}"]}
Socket server emit with complex message instance:
verbose: websocket writing 5:::{"name":"user.newAuthenticated","args":["{\"m\":{\"m\":\"__12\",\"a\":{\"field\":\"name\",\"min\":3,\"max\":20}},\"d\":{\"key\":\"fakeKey\"},\"s\":1,\"t\":\"m\"}"]}
The complexe message would render the following sentence:
The min length of name is 3. Max length is 20. and would contain the key: "fakeKey" in data. Just to explain how it works.
As you see, the message get bigger and bigger and it is normal, but I would like to know what I can do to make a better serialization here:
Delete the message itself when there aren't (empty)
Delete the data when it's empty as well
Delete the status when it's false (because it's the default value)
I see a lot of \ in the socket log because it is JSON, I believe that's a problem, because each time I'll add something I'll get extra characters that I do not want. Maybe the JSON isn't a good choice and I should serialize differently, first in JSON like the examples at the top, but then in something else, maybe kind of binary, if it takes less space.
What do you think?
And if it would be a good idea to encrypt somehow the message in another format, would the cost of the encryption be worth it? Because encrypt it would take a bit of time as well, so I'm just wondering if it wouldn't just move the issue, like it would take less time to send the message through socket because it would be lighter, but we would use more time to encrypt it. Just wondering.
My guess is that your message object has two fields (name and args).
The first stop to reduce the length of the message is to get rid of the (pretty useless) outer object and replace it with an array. So an empty message
{"name":"empty","args":[]}
would become
["empty",[]]
or even
["empty"]
The next thing is that you have a bug in the serialization of the arguments. Instead of sending JSON, you wrap the JSON data in a string. Example: In the authenticated case, you send
{"name":"user.newAuthenticated","args":["{\"m\":\"Respond to all clients\",\"d\":{},\"s\":1,\"t\":\"m\"}"]}
but you should send
{"name":"user.newAuthenticated","args":[{"m":"Respond to all clients","d":{},"s":1,"t":"m"}]}
instead. Now the question is whether args is a list of a single object. If it's always a single object, then you could get rid of the [] as well. With my suggested change from above, that would give you:
["user.newAuthenticated",{"m":"Respond to all clients","d":{},"s":1,"t":"m"}]
which is pretty good IMO. If you can make the (de-)serializer handle default values properly, you can reduce this to:
["user.newAuthenticated",{"m":"Respond to all clients","s":1,"t":"m"}]
(i.e. we can omit the empty d property).
For a MMO, I think a minimum of data must be sent to the client. If a socket is called 2xx/3xx by sec, you must reduce the size of the data sent through the socket as most as possible.
On another hand, it also consummes resource to encrypt the object on the server side to send a minified version of the object... Wouldn't it be better not to reduce it and to send an object not reduced so we don't spent resource to encrypt it?
I am receiving protobuf data from a remote host over a websocket. I know how to decode the protobuf in JS using Protobuf.js but I cannot determine which part of the result coming through the websocket is in fact the protobuf data I need to decode. Attempting to decode the entire result gives me the JS error:
Cannot wrap null or non-object
If I log to console the object returned from the websocket, it looks like this:
I've tried decoding various things like e.data (where e is the total result returned) but not sure if I'm going about this the right way.
What part of this holds the actual protobuf content?
Yes, the data is in .data, and appears to be represented as a Blob. Working with blobs in javascript is not trivial. To get an object from that, you would need a javascript protobuf implementation that accepts a Blob and returns the deserialized object.