This is doing my head in.
I have a jquery ajax post returning output from a mysql database. One field is binary. I wish to proccess that binary field using Javascript. I have been around and around in circles trying different approaches non of which seem satisfactory and most hit a road block.
I can pass the data up base64 encoded and then decode it in Javascript.
The data is a packet so I wish to decode it using a DataView as it has all the Get... calls for treating the data as a packet ( like a C structure ).
I have tried Blobs and Typed Arrays but everwhere I go it seems to just not be the right thing.
What I am after is something like this:
var ab = new ArrayBuffer ( atob( my_base64_encoded_packet_data )
var dv = new DataView ( ab );
Simple! No. The only way I have managed to get close is to using fileReader, like this ( roughly ):
fileReader.onload = function ( ) {
var newBuf = this.result;
callback( newBuf );
return;
};
fileReader.readAsArrayBuffer( buf );
Where buf is a blob and newBuf is the ArrayBuffer.
Here is another approach which does what I want, but seems labour intensive.
function str2ab(str) {
var buf = new ArrayBuffer(str.length);
var bufView = new Uint8Array(buf);
for (var i=0, var strLen=str.length; i<strLen; i++) {
bufView[i] = str.charCodeAt(i);
}
return buf;
}
Help please. Thanks
The problem is that you are using jQuery's ajax. jQuery can't handle binary since it wants everything to be strings. Browser tries to be smart with it and tries to transform the response if you want to receive it as text
What you want is the raw binary so a responseType that is either arraybuffer or a blob is what you want.
The solution is either stick to doing xhr manually or use the new fetch method
fetch(url, opts).then(res => res.arrayBuffer()).then(buffer => {
// code here
})
Sending and Receiving Binary Data - MDN
Related
I have an ArrayBuffer object that I need to be able to convert to String to JSON, but I can't get the value of the [Int8Array] out of the object, even though it is clearly there.
I've tried all variations on this, but they all return undefined
console.log(result);//Returns the array buffer
//Following methods all return undefined?
console.log(result["[[Int8Array]]"]);
console.log(result[[[Int8Array]]]);
console.log(result[[["Int8Array"]]]);
console.log(result[Int8Array]);
console.log(result["Int8Array"]);
How can I get all the Int8Array or UInt8Array values that are clearly available in the object?
You can use the textDecoder that accepts ArrayBuffer (as well as uint8array) without having to deal with Uint8array's:
var str = new TextDecoder().decode(arrayBuffer)
var json = JSON.parse(str)
if you want to get straight to json
var json = await new Response(arrayBuffer).json()
You need to intiantiate a new Uint8Array to get their values, you can't access them directly using your ArrayBuffer instance.
var buf = new ArrayBuffer(8);
var int8view = new Uint8Array(buf);
console.log(int8view)
JSFiddle : https://jsfiddle.net/v8m7pjqb/
I have a Uint8Array with an offset that I receive from another function. This contains the data I need, but there is a bit of other stuff at the start of the buffer backing this typed array.
The actual data are 32bit integers, and I'd like to have that data in an Int32Array. But converting this doesn't seem to be straightforward, I'm currently doing it manually the following way:
var outputBuffer = new ArrayBuffer(data.length);
var inputByteArray = new Uint8Array(outputBuffer);
for (var i=0; i < data.length; ++i) {
inputByteArray[i] = data[i]
}
var outputInt32Array= new Int32Array(outputBuffer);
The straightforward way of just creating a new Int32Array and passing the source Uint8Array doesn't work:
var outputInt32Array = new Int32Array(data) // data is the Uint8Array with offset
This results in a typed array that still behaves like a Uint8Array and hands out individual bytes, not 32bit integers.
Trying it by passing in the offset also doesn't work, I get the error "RangeError: start offset of Int32Array should be a multiple of 4":
var outputInt32Array = new Int32Array(data.buffer, data.byteOffset, length)
Is manually copying each byte the only way to get an Int32Array out of an Int8Array with an offset?
No, you don't need to manually copy the bytes from one to the other array. Using new Int32Array(data.buffer, …) is the best approach, but if you have a weird offset you will need to use a second buffer that is properly aligned. Still, you don't have to copy it manually, you can just use the slice method:
var outputInt32Array = new Int32Array(data.buffer.slice(data.byteOffset), 0, length);
If you need to access Int32s on the same buffer as data, you can also use a DataView.
How do I convert an array of bytes into ArrayBuffer in Nashorn? I am trying to insert binary data into a pure JavaScript environment (i.e., it doesn't have access to Java.from or Java.to) and so would like to create an instance out an array of bytes.
Looks like I was going about this the wrong way. It made more sense to convert it into Uint8Array since what I'm sending in is an array of bytes.
I created the following function:
function byteToUint8Array(byteArray) {
var uint8Array = new Uint8Array(byteArray.length);
for(var i = 0; i < uint8Array.length; i++) {
uint8Array[i] = byteArray[i];
}
return uint8Array;
}
This will convert an array of bytes (so byteArray is actually of type byte[]) into a Uint8Array.
I think you're right about using a Uint8Array, but this code might be preferable:
function byteToUint8Array(byteArray) {
var uint8Array = new Uint8Array(byteArray.length);
uint8Array.set(Java.from(byteArray));
return uint8Array;
}
Also, if you really need an ArrayBuffer you can use uint8Array.buffer.
How to save (with the lowest file size) pixels data in javascript?
For now I'm using JSON and it is looking like this:
{"SIZE":[120,120],
"FRAME":["SHINE",""],
"LEDS":[
{"X":0,"Y":0,"R":16,"G":30,"B":58},
{"X":1,"Y":0,"R":13,"G":23,"B":43},
{"X":2,"Y":0,"R":14,"G":26,"B":50},
etc...
So for little pictures like 16x16, 90x32 the file size is ok but now I need do pictures 120x120 and file size is around 500kb. Is there any better solution for that?
You have few options to send your pixels data in minimal size. The easiest way is to pack all this pixel information in one binary data.
MIME type
You can send image data with http request using i.e. image/bmp MIME type. You can view https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Sending_and_Receiving_Binary_Data to check how to read that kind of data
var oReq = new XMLHttpRequest();
oReq.open("GET", "/myfile.png", true);
oReq.responseType = "arraybuffer";
oReq.onload = function (oEvent) {
var arrayBuffer = oReq.response; // Note: not oReq.responseText
if (arrayBuffer) {
var byteArray = new Uint8Array(arrayBuffer);
for (var i = 0; i < byteArray.byteLength; i++) {
// do something with each byte in the array
}
}
};
oReq.send(null);
Convert to binary and send it as BASE64
You can convert your pixels data to binary format (similar to pure BMP) and then use BASE64 algorithm to represent it with letters. Then you can send it with regular JSON
This is very good explanation about packing complex structures to binary: http://danthedev.com/2015/07/25/binary-in-javascript/
What is the preferable way of appending/combining ArrayBuffers?
I'm receiving and parsing network packets with a variety of data structures. Incoming messages are read into ArrayBuffers. If a partial packet arrives I need to store it and wait for the next message before re-attempting to parse it.
Currently I'm doing something like this:
function appendBuffer( buffer1, buffer2 ) {
var tmp = new Uint8Array( buffer1.byteLength + buffer2.byteLength );
tmp.set( new Uint8Array( buffer1 ), 0 );
tmp.set( new Uint8Array( buffer2 ), buffer1.byteLength );
return tmp.buffer;
}
Obviously you can't get around having to create a new buffer as ArrayBuffers are of a fixed length, but is it necessary to initialize typed arrays? Upon arrival I just want is to be able to treat the buffers as buffers; types and structures are of no concern.
Why not using a Blob ? (I realize it might not have been available at that time).
Just create a Blob with your data, like var blob = new Blob([array1,array2,string,...]) and turn it back into an ArrayBuffer (if needed) using a FileReader (see this).
Check this : What's the difference between BlobBuilder and the new Blob constructor?
And this : MDN Blob API
EDIT :
I wanted to compare the efficiency of these two methods (Blobs, and the method used in the question) and created a JSPerf : http://jsperf.com/appending-arraybuffers
Seems like using Blobs is slower (In fact, I guess it's the use of Filereader to read the Blob that takes the most time). So now you know ;)
Maybe it would me more efficient when there are more than 2 ArrayBuffer (like reconstructing a file from its chunks).
function concat (views: ArrayBufferView[]) {
let length = 0
for (const v of views)
length += v.byteLength
let buf = new Uint8Array(length)
let offset = 0
for (const v of views) {
const uint8view = new Uint8Array(v.buffer, v.byteOffset, v.byteLength)
buf.set(uint8view, offset)
offset += uint8view.byteLength
}
return buf
}
It seems you've already concluded that there is no way around creating a new array buffer. However, for performance sake, it could be beneficial to append the contents of the buffer to a standard array object, then create a new array buffer or typed array from that.
var data = [];
function receive_buffer(buffer) {
var i, len = data.length;
for(i = 0; i < buffer.length; i++)
data[len + i] = buffer[i];
if( buffer_stream_done())
callback( new Uint8Array(data));
}
Most javascript engines will already have some space set aside for dynamically allocated memory. This method will utilize that space instead of creating numerous new memory allocations, which can be a performance killer inside the operating system kernel. On top of that you'll also shave off a few function calls.
A second, more involved option would be to allocate the memory beforehand.
If you know the maximum size of any data stream then you could create an array buffer of that size, fill it up (partially if necessary) then empty it when done.
Finally, if performance is your primary goal, and you know the maximum packet size (instead of the entire stream) then start out with a handful of array buffers of that size. As you fill up your pre-allocated memory, create new buffers between network calls -- asynchronously if possible.
You could always use DataView (http://www.khronos.org/registry/typedarray/specs/latest/#8) rather than a specific typed array, but as has been mentioned in the comments to your question, you can't actually do much with ArrayBuffer on its own.