Is there any way to send text and binary in one request via websocket? For example: file name (text) and file content (binary)
I can send them as string like:
JSON.stringify({filename: "test.dat", filecontent: data});
But it is a lot slower than sending only file content as binary (arraybuffer).
Remember that binary is just encoded data. This is less a JavaScript question and more an encoding question. Here's how I would do it.
Set aside 32 bits (representing one integer) at the beginning of your request to specify the bit length of test.dat. Then combine this with your two data sources. Your payload will look like this:
TEXT_LENGTH + TEST.DAT AS BINARY + FILECONTENT AS BINARY
Then get back the data as an array buffer. Use
textLengthBits = parseInt(arrBuffer.slice(0,32), 2);
To get the length of the text. Then slice again,
textBits = arrBuffer.slice(32, 32 + textLengthBits)
To get the text. The remaining bits are your file.
fileBits = arrBuffer.slice(32 + textLengthBits);
Related
Overview:
I'm building a Javascript tool inside a web page. Except for loading that page, the tool will run without server communication. A user will select a local file containing multiple binary records, each with a x'F0 start byte and x'F0 end byte. The data in between is constrained to x'00 - x'7F and consists of:
bit maps
1-byte numbers
2-byte numbers, low order byte first
a smattering of ASCII characters
The records vary in lengths and use different formats.
[It's a set of MIDI Sysex messages, probably not relevant].
The local file is read via reader.readAsArrayBuffer and then processed thus:
var contents = event.target.result;
var bytes = new Uint8Array(contents);
var rawAccum = '';
for (x = 0; x < bytes.length; x++) {
rawAccum += bytes[x];
}
var records = rawAccum.split(/\xF0/g);
I expect this to split the string into an array of its constituent records, deleting the x'F0 start byte in the process.
It actually does very little. records.length is 1 and records[0] contains the entire input stream.
[The actual split code is: var records = rawAccum.split(/\xF0\x00\x00\x26\x02/g); which should remove several identical bytes from the start of each record. When this failed I tried the abbreviated version above, with identical (non)results.]
I've looked at the doc on split( and at several explanations of \xXX among regex references. Clearly something does not work as I have deduced. My experience with JavaScript is minimal and sporadic.
How can I split a string of binary data at the occurrence of a specific binary byte?
The splitting appears to work correctly:
var rawAccum = "\xf0a\xf0b\xf0c\xf0"
console.log( rawAccum.length); // 7
var records = rawAccum.split(/\xF0/g);
console.log(records); // "", "a", "b", "c", ""
but the conversion of the array buffer to a string looks suspicious. Try converting the unsigned byte value to a string before appending it to rawAccum:
for (x = 0; x < bytes.length; x++) {
rawAccum += String.fromCharCode( bytes[x]);
}
Data conversions (update after comment)
The filereader reads the file into an array buffer in memory, but JavaScript does not provide access to array buffers directly. You can either create and initialize a typed array from the buffer (e.g. using the Uint8Array constructor as in the post), or access bytes in the buffer using a DataView object. Methods of DataView objects can convert sequences of bytes at specified positions to integers of varying types, such as the 16 bit integers in the Midi sysex records.
JavaScript strings use sequences of 16 bit values to hold characters, where each character uses one or two 16 bit values encoded using UTF-16 character encoding. 8 bit characters use only the lower 8 bits of a single 16 bit value to store their Unicode code point.
It is possible to convert an array buffer of octet values into a "binary string", by storing each byte value from the buffer in the low order bits of a 16 bit character and appending it to an existing string. This is what the post attempts to do. But in JavaScript strings (and individual characters which have a string length of 1) are not a subset of integer numbers and have their own data type, "string".
So to convert an unsigned 8 bit number to a JavaScript 16 bit character of type "string", use the fromCharCode static method of the global String object, as in
rawAccum += String.fromCharCode( bytes[x]);
Calling String.fromCharCode is also how to convert an ASCII character code located within MIDI data to a character in JavaScript.
To convert a binary string character derived from an 8 bit value back into a number, use the String instance method charCodeAt on a string value and provide the character position:
var byteValue = "\xf0".charCodeAt(0);
returns the number 0xf0 or 250 decimal.
If you append a number to a string, as in the question, the number is implicitly converted to a decimal string representation of its value first:
"" + 0xf0 + 66 // becomes the string "24066"
Note that an array buffer can be inspected using a Uint8Array created from it, sliced into pieces using the buffer's slice method and have integers of various types extracted from the buffer using data views. Please review if creating a binary string remains the best way to extract and interpret Midi record contents.
I want to send the data captured from my microphone (converted to unsigned 16bit integers) in the browser to my server as a binary string, but I'm having a hard time doing that.
I tried using String.fromCodePoint on my array, but it seems that the resulting String is not a valid one. I also tried using DataView, but not sure how to get a binary String out of that either.
Does anyone know how that can be achieved?
EDIT: I am referring to binary data, as is "binary file", and not to "binary representation of an integer".
This did not answer the question.
If only in the browser you can use the btoa function to encode to a base64 string.
Edit:
I am assuming you have an array of integers.
Final Edit:
Maybe try something like:
arr.map(function(item) {
var value = item.toString(2);
var padLength = 16 - value.length;
var binaryVal = "0".repeat(padLength) + value;
return binaryVal;
}).join("");
Context
I know that in python you can create an a string representing the packed bytes of an array doing something like such:
import numpy as np
np.array([1, 2], dtype=np.int32).tobytes()
# returns '\x01\x00\x00\x00\x02\x00\x00\x00'
np.array([1, 2], dtype=np.float32).tobytes()
# returns '\x00\x00\x80?\x00\x00\x00#'
And they can be decoded using np.fromstring
Question
Currently, my Javascript is receiving a string of packed bytes that encodes an array of floats (i.e. '\x00\x00\x80?\x00\x00\x00#') and I need to decode the array -- what is the best way to do this?
(if it were an array of ints I imagine I could use text-encoding to pull the bytes and then just multiply and add appropriately...)
Thanks,
First, you have to convert a string into a buffer, and then create a Float32Array on that buffer. Optionally, spread it to create a normal Array:
str = '\x00\x00\x80?\x00\x00\x00#'
bytes = Uint8Array.from(str, c => c.charCodeAt(0))
floats = new Float32Array(bytes.buffer)
console.log(floats)
console.log([...floats]);
Floating-point representation can vary from platform to platform. It isn't a good idea to use a binary serialized form of a floating-point number to convey a number from one machine to another.
That said, the answers to this question might help you, provided the encoding used by the JavaScript matches that of the source of the numbers:
Read/Write bytes of float in JS
In my JS-application I'm uploading documents to a server.
The documents are stored in Uint8Array's. That means documents are represented as Arrays consisting of Integers.
Before uploading the documents I JSON.stringify the documents.
Here you have an example:
var document = [0,5,5,7,2,234,1,4,2,4]
JSON.stringify(document)
=> "[0,5,5,7,2,234,1,4,2,4]"
Then I send the JSON-representation to the Server.
My problem is that the JSON-representation has a much bigger file-size than the original Integer-Array. I guess that's because the Array is transformed to a JSON-String. I send much more data to the server then needed. How can I store the data more compressed in JSON?
I thought, the JSON-representation is maybe smaller, if I convert the Array first to Base64 and then to JSON.
What are your ideas? Thanks
The integer array JSON representation results in an average 3.57 ratio between input and output:
10 values are represented by 2 bytes (one digit, one comma)
90 values by 3 bytes (2 digits, one comma)
156 values by 4 bytes
On the other hand, base64 will result in an average 1.333... ratio (3 bytes are encoded as 4).
If you have mostly ASCII-like characters in your array (i.e. in the range 32-126), you would probably be better off just sending them as strings (with a few characters escaped), but not if you have random 8-bit data.
You could use some kind of base94 representation to get a better ratio over base64, but is it really worth the cost?
Also note that if you may also consider compression of the data.
I got a strange experience. When I send data of this arraybuffer setting:
var f32s = Float32Array(2048);
for (var i = 0; i < f32s.length; i++) {
f32s[i] = buffer[i]; // fill the array
ws.send(f32s[i]);
}
The buffer size I got at the other end is 8192 bytes.
But when I send chunck of buffer in JSON format like bellow:
var obj = {
buffer_id: 4,
data: f32s[i]
};
var json = JSON.stringify({ type:'buffer', data: obj });
ws.send(json);
The buffer size I got at the other end bloat to 55,xxx bytes with data filled in and 17,xxx bytes with no data filled.
Why this happen and how do I keep the buffer size low?
I want to do this because the stream is choppy when I render it at the other end.
Thank you.
I would expect this is happening because a float 32 array requires exactly 32 bits per number within the data structure, however json being an ascii format represents each number with a serious of 8bit characters, and then another 8bits for the comma and maybe again for the decimal and again for the delimiting whitespace.
Therefore the data [0.1234545, 111.3242, 523.12341] for instance requires 3 * 32 => 96 bits to represent within a float32array but as a json string requires 8bits for each of the 32 characters in this example which comes to 256 bits.