I am receiving protobuf data from a remote host over a websocket. I know how to decode the protobuf in JS using Protobuf.js but I cannot determine which part of the result coming through the websocket is in fact the protobuf data I need to decode. Attempting to decode the entire result gives me the JS error:
Cannot wrap null or non-object
If I log to console the object returned from the websocket, it looks like this:
I've tried decoding various things like e.data (where e is the total result returned) but not sure if I'm going about this the right way.
What part of this holds the actual protobuf content?
Yes, the data is in .data, and appears to be represented as a Blob. Working with blobs in javascript is not trivial. To get an object from that, you would need a javascript protobuf implementation that accepts a Blob and returns the deserialized object.
Related
I'm pretty new to the Pickle Library and JS. I am writing code in JS to interact with a Python server; when I make a POST to the Python server, I am returned a pickled float. I want to convert the float into a JS-readable object.
The server returns something like pickle.dumps(3.14159,0). When I print this value in Python, I get b'F3.14159\n.' However, the library I'm working with, JPickle, cannot interpret the 'b' in the return (or at least, that's what I'm guessing...when I try to unpickle my response using console.log(jpickle.loads(response)), I get a "unhandled opcode" error). Separate from the POST request, jpickle.loads('F3.14159\n.') gives me 3.14159, but jpickle.loads(b'F3.14159\n.') fails to compile. I'm not sure how to change the response I get from the POST because it is unreadable in JS, so I'm unsure how to go about solving this problem.
I'm open to any solutions that don't involve changing the server-side code (I don't have control over it), including using another library. I've tried JsonPickle with no luck.
Use just JSON, no pickle library required; Python comes with the json module and browser JS can, out of the box, parse json into values.
Serialize your data via something like json.dump and use JSON.parse in the browser to deserialize it.
A little late, but the answers in attached might help:: the code allows Javascript on a browser to traverse and decode compressed pickled objects coming from Python via Ajax.
Is there an already published Javascript solution to traverse Python pickled objects without using Node.js
Make sure that the mime type is allowing you to pass binary numbers.
I understood why we are using JSON to exchange data between browser and server in place of XML but I could not understand why we are using only string type of JSON even we have six different value datatype, I mean why we can't use integer or Boolean or any other value datatype.
Hope you guys understand what I'm trying to say, Thanks in advance.
If I understand correctly, the limitation is because of the way data needs to be encoded to be sent over HTTP and ultimately over the wire. You json object (or xml,etc) is ultimately just a payload for HTTP (which is just a payload for tcp in turn and so on).
HTTP inherently does not and should not identify data types in payload, it is just an array for HTTP. You can select how to represent this array i.e. how to encode it; It can be string (ascii, utf-8, etc) or binary but it has to be uniform for the whole payload.
HTTP does offer different encoding methods of payload which can be interpreted by the receiver by looking at the content-type header and accordingly decode the data.
Hope this helps.
why we are using only string type of JSON
Uhm, we're not. I believe you're misunderstanding something here. HTTP responses can really contain anything; every time you download a PDF or an image from a web server, the web server is sending a binary payload, which can literally be anything. So it's not even true that all HTTP bodies must be text.
To exchange data between systems, you send bytes. For these bytes to mean anything, you need an encoding scheme. Image formats have a particular way in which bytes need to be arranged, and when properly doing so, you can send pictures with them. Same for PDFs, video, audio, and anything else (including text).
If you want to send structured data, you need to express that structure somehow. How do you send a, for example, PHP array over HTTP…? (Substitute your equivalent list data structure in your language of choice.) You can't. A PHP array is a specific data structure in memory of a PHP runtime, sending that as is over HTTP has no meaning (because it deals with internal pointers and such). This array needs to be serialised first. There are many possible serialisation methods, some of them using binary data, and some using formats which are human readable to varying degrees. You could simply join all array elements with commas and .split(',') them again on the other end, but that's rather simplistic and misses many more complex cases and edge cases.
JSON and XML (and YAML and whatnot) are human readable formats which can serialise data structures like arrays (and dictionaries and numbers and booleans etc), and which happen to be text-based (purposely, to make them developer-friendly). You can use any of those data types JSON allows. Nothing prevents you from doing so, and not using them is insane. JSON and XML also happen to be two formats easily parsed with tools built into every browser. You could use any other binary format too, but then you'd have to manually parse it in Javascript.
Communication between browser and server can be done in many ways. It's just that JSON comes out of the box. You can use protobuf, xml and plethora of other data serialization techniques to communicate with the server as long as both sides understand what's the communication medium. On the browser side, you have to probably implement protobuf, xml etc serialization/deserialization on your own in javascript.
Any valid JSON is permitted for data exchange. The keys are string quoted but the values can be strings, numbers, booleans, array or other objects itself. Though before transmission, everything is converted into a string and the receiving side parses it into the correct format.
I'm writing WebSocket based clients servers.
I want to send raw binary data that is then decoded by the client.
My problem is two-fold:
while doing manipulations on binary data in c++ (the language I'm writing the servers in) is straight forward, it seems hard in Javascript.
I have found that you can use ArrayBuffers and UInt8Arrays to do most of the manipulations, and that works fine for sending messages.
My problem is that when I try receiving the messages if I declare the message as binary on the server side, it shows up as a Blob on the client. I have tried converting the Blob to an ArrayBuffer like so:
ws.onmessage = function(evt) {
var data = null;
data = await new Response(evt.data).arrayBuffer();
}
But this gives me the error:
SyntaxError: await is only valid in async functions and async generators
It seems that this method is asynchronous, and while I'm sure I could do it this way, it doesn't really seem that good in the first place.
I have realized that sending the data as the text makes evt.data appear as a string, which makes me believe that I can use a JSON format for the packets instead of a binary one.
I really don't want to use JSON though, as some of the packets will be full of a lot of information and I'm scared it will add unnecessary bloat.
I think what I really want is just to be able to read the evt.data as an ArrayBuffer, because it seems like that would be the most performant.
Use the then method of the promise
new Response(evt.data).arrayBuffer().then(buffer=> {
//here you have the buffer
})
then
arrayBuffer
Note: The await can be used inside functions declared with async keyword.
I used protostuff to transform to byte array a json input i have. The code in java is:
LinkedBuffer buffer = LinkedBuffer.allocate(1024);
Schema<String> orderSchema = RuntimeSchema.getSchema(String.class);
int i = 1 ;
for(String p:poligonsStr) {
buffer.clear();
byteslist.add(ProtostuffIOUtil.toByteArray(p, orderSchema, buffer));
}
The problem is I don't know the algorithm that is used and how I can decode with the JavaScript client (Node.js). Also I saw there is a very good algorithm called Smile implemented for protostuff in project com.dyuproject.protostuff but I would like to know how to get schema with that library- I didn't manage that yet.
I would like to know what's the best to use: ProtostuffIOUtil or SmileIOUtil?
And how to use? And how to decode with JavaScript?
protostuff binary encoding is different from protobuf, and as far as I know there is no JavaScript library to decode protostuff-encoded data at the moment.
smile is not supported by web browsers out of the box, but there are libraries that can decode it.
As for me, there are two optiomal ways how you can encode data on server using Protostuff library, and decode it using JavaScript on client side:
Use protobuf encoding, it is good if size of encoded data is important. On server side, you should use ProtobufIOUtil to serialize your data to protobuf binary format. On client side, you can use https://github.com/dcodeIO/ProtoBuf.js/ to decode binary data from server.
Use JSON encoding, it is native format for JavaScript and usually it will be parsed faster than binary protobuf-encoded data. On server side, you should use JsonIOUtil (from protostuff-json module) to serialize your data to JSON text format. On client side, it is supported out of the box.
Here is an example how to serialize your POJO into protobuf binary using Protostuff: HelloService.java
I have a very long list of points (lat/long) in my C# program and I need to pass it to my webgl code. I have done it before using json but I thought I could reduce the bandwidth if I sent the data in binary format. Based on my research, on the client side I should use XMLHttpRequest with arraybuffer as response type. I just do not know what to do on the server side. I mean how to prepare the binary data in C# that could be interpreted as arraybuffer in javascript.
I am new to web programming, so if I am not clear on any part, please let me know.
I am not an expert, but I found this solution to work:
On the server side, convert the array of numbers to a byte array and then send the byte array to javascript as a binary file (Set the Mime type to "application/octet-stream")