Appending ArrayBuffers - javascript

What is the preferable way of appending/combining ArrayBuffers?
I'm receiving and parsing network packets with a variety of data structures. Incoming messages are read into ArrayBuffers. If a partial packet arrives I need to store it and wait for the next message before re-attempting to parse it.
Currently I'm doing something like this:
function appendBuffer( buffer1, buffer2 ) {
var tmp = new Uint8Array( buffer1.byteLength + buffer2.byteLength );
tmp.set( new Uint8Array( buffer1 ), 0 );
tmp.set( new Uint8Array( buffer2 ), buffer1.byteLength );
return tmp.buffer;
}
Obviously you can't get around having to create a new buffer as ArrayBuffers are of a fixed length, but is it necessary to initialize typed arrays? Upon arrival I just want is to be able to treat the buffers as buffers; types and structures are of no concern.


Why not using a Blob ? (I realize it might not have been available at that time).
Just create a Blob with your data, like var blob = new Blob([array1,array2,string,...]) and turn it back into an ArrayBuffer (if needed) using a FileReader (see this).
Check this : What's the difference between BlobBuilder and the new Blob constructor?
And this : MDN Blob API
EDIT :
I wanted to compare the efficiency of these two methods (Blobs, and the method used in the question) and created a JSPerf : http://jsperf.com/appending-arraybuffers
Seems like using Blobs is slower (In fact, I guess it's the use of Filereader to read the Blob that takes the most time). So now you know ;)
Maybe it would me more efficient when there are more than 2 ArrayBuffer (like reconstructing a file from its chunks).


function concat (views: ArrayBufferView[]) {
let length = 0
for (const v of views)
length += v.byteLength
let buf = new Uint8Array(length)
let offset = 0
for (const v of views) {
const uint8view = new Uint8Array(v.buffer, v.byteOffset, v.byteLength)
buf.set(uint8view, offset)
offset += uint8view.byteLength
}
return buf
}


It seems you've already concluded that there is no way around creating a new array buffer. However, for performance sake, it could be beneficial to append the contents of the buffer to a standard array object, then create a new array buffer or typed array from that.
var data = [];
function receive_buffer(buffer) {
var i, len = data.length;
for(i = 0; i < buffer.length; i++)
data[len + i] = buffer[i];
if( buffer_stream_done())
callback( new Uint8Array(data));
}
Most javascript engines will already have some space set aside for dynamically allocated memory. This method will utilize that space instead of creating numerous new memory allocations, which can be a performance killer inside the operating system kernel. On top of that you'll also shave off a few function calls.
A second, more involved option would be to allocate the memory beforehand.
If you know the maximum size of any data stream then you could create an array buffer of that size, fill it up (partially if necessary) then empty it when done.
Finally, if performance is your primary goal, and you know the maximum packet size (instead of the entire stream) then start out with a handful of array buffers of that size. As you fill up your pre-allocated memory, create new buffers between network calls -- asynchronously if possible.


You could always use DataView (http://www.khronos.org/registry/typedarray/specs/latest/#8) rather than a specific typed array, but as has been mentioned in the comments to your question, you can't actually do much with ArrayBuffer on its own.

Related

Javascript Webworker how to put json information into array buffer

I have thousands of small strings that I have to pass from a webworker back to the main page, each one is something like this:
"this string needs to be sent"
How would I be able to include it into an array buffer in order to increase the transfer speed? I understand how to use numbers with array buffers, but how do you use strings? I am looking for something like this:
var strings = ["str1","str2","str3",...]
for (var i = 0; i < strings.length; i++) {
arraybuffer[i] = //Whatever operation works to add strings[i]
}

It's worth measuring and comparing performance of various techniques. The worker could use SharedArrayBuffer if supported in your target browsers (not exemplified below), otherwise Transferrable objects can be used with postMessage(). TextEncoder creates ArrayBuffers from strings.
Individual strings can be transferred as they are encoded:
const encoder = new TextEncoder()
strings.forEach(s => {
const encoded = encoder.encode(s)
postMessage(encoded, [encoded.buffer])
})
An array of strings could be transferred in batches:
const encoded = strings.map(s => encoder.encode(s))
postMessage(encoded, encoded.map(bytes => bytes.buffer))

ArrayBuffer madness - how to get binary into an arraybuffer

This is doing my head in.
I have a jquery ajax post returning output from a mysql database. One field is binary. I wish to proccess that binary field using Javascript. I have been around and around in circles trying different approaches non of which seem satisfactory and most hit a road block.
I can pass the data up base64 encoded and then decode it in Javascript.
The data is a packet so I wish to decode it using a DataView as it has all the Get... calls for treating the data as a packet ( like a C structure ).
I have tried Blobs and Typed Arrays but everwhere I go it seems to just not be the right thing.
What I am after is something like this:
var ab = new ArrayBuffer ( atob( my_base64_encoded_packet_data )
var dv = new DataView ( ab );
Simple! No. The only way I have managed to get close is to using fileReader, like this ( roughly ):
fileReader.onload = function ( ) {
var newBuf = this.result;
callback( newBuf );
return;
};
fileReader.readAsArrayBuffer( buf );
Where buf is a blob and newBuf is the ArrayBuffer.
Here is another approach which does what I want, but seems labour intensive.
function str2ab(str) {
var buf = new ArrayBuffer(str.length);
var bufView = new Uint8Array(buf);
for (var i=0, var strLen=str.length; i<strLen; i++) {
bufView[i] = str.charCodeAt(i);
}
return buf;
}
Help please. Thanks

The problem is that you are using jQuery's ajax. jQuery can't handle binary since it wants everything to be strings. Browser tries to be smart with it and tries to transform the response if you want to receive it as text
What you want is the raw binary so a responseType that is either arraybuffer or a blob is what you want.
The solution is either stick to doing xhr manually or use the new fetch method
fetch(url, opts).then(res => res.arrayBuffer()).then(buffer => {
// code here
})
Sending and Receiving Binary Data - MDN

How can I convert a Uint8Array with offset into an Int32Array?

I have a Uint8Array with an offset that I receive from another function. This contains the data I need, but there is a bit of other stuff at the start of the buffer backing this typed array.
The actual data are 32bit integers, and I'd like to have that data in an Int32Array. But converting this doesn't seem to be straightforward, I'm currently doing it manually the following way:
var outputBuffer = new ArrayBuffer(data.length);
var inputByteArray = new Uint8Array(outputBuffer);
for (var i=0; i < data.length; ++i) {
inputByteArray[i] = data[i]
}
var outputInt32Array= new Int32Array(outputBuffer);
The straightforward way of just creating a new Int32Array and passing the source Uint8Array doesn't work:
var outputInt32Array = new Int32Array(data) // data is the Uint8Array with offset
This results in a typed array that still behaves like a Uint8Array and hands out individual bytes, not 32bit integers.
Trying it by passing in the offset also doesn't work, I get the error "RangeError: start offset of Int32Array should be a multiple of 4":
var outputInt32Array = new Int32Array(data.buffer, data.byteOffset, length)
Is manually copying each byte the only way to get an Int32Array out of an Int8Array with an offset?

No, you don't need to manually copy the bytes from one to the other array. Using new Int32Array(data.buffer, …) is the best approach, but if you have a weird offset you will need to use a second buffer that is properly aligned. Still, you don't have to copy it manually, you can just use the slice method:
var outputInt32Array = new Int32Array(data.buffer.slice(data.byteOffset), 0, length);
If you need to access Int32s on the same buffer as data, you can also use a DataView.

Why is Javascript TypeDArray not working?

I have 4 bytes ArrayBuffer and I assigned a number at the index 0 using dataview. When I try to get the value using dataview that gives result correctly but it does not give correct result when I try to get value using typed array. Can anyone help on this? Here are the code:
var buffer = new ArrayBuffer(4);
var dataview = new DataView(buffer);
dataview.setUint32(0,5000);
var unint32 = new Uint32Array(buffer);
console.log(unint32[0]); //2282946560 instead of 5000
console.log(dataview.getUint32(0)); //shows correctly 5000

That's because you're using the wrong endian-type when using setUint32. Specifically, you need it to store a little-endian representation, because your hardware behaves like that.
You can see this more easily using hexadecimal values. Now, 5000 === 0x1388, while 2282946560 = 0x88130000. Can you see the pattern here?
Try this instead:
dataview.setUint32(0, 5000, true);
var unint32 = new Uint32Array(buffer);
console.log(unint32[0]); // 5000, yay!
As apsillers pointed out, if you're going to use dataview to retrieve the value too, you'll also have to use dataview.getUint32(0, true).
As a last word, if you just need to work with Uint32 numbers, forget about the DataView and use the typed array right away:
var buffer = new ArrayBuffer(4);
var unint32 = new Uint32Array(buffer);
unint32[0] = 5000;
You'll never get it wrong with this.
On the other hand, if you need to fill an ArrayBuffer with raw binary data, you'd like to check then endiannes of your hardware with this little trick.

Is there a way to read binary data in JavaScript?

I would like to inject binary data into an object in JavaScript. Is there a way to do this?
i.e.
var binObj = new BinaryObject('101010100101011');
Something to that effect. Any help would be great.

You can use parseInt:
var bin = parseInt('10101010', 2);
The second argument (the radix) is the base of the input.

There's this binary ajax library that is explained here and there's also another binary parser library that can handle more data types.
You could also look at Google Gears which has a binary Blob object or take a look at making a javascript wrapper for Flash which provides a native ByteArray implementation.
Or... you can sit and wait and hope that all these things become standard :)

On all recent browsers you can do:
xhr.overrideMimeType('text/plain; charset=x-user-defined');
And retrieve a string. To get the binary result you will have to do
data.charCodeAt(pos) & 0xff;
On the nightly builds of Firefox and Chrome you can retrieve the value as an ArrayBuffer
xhr.responseType = "arraybuffer";
The result is then accessible there
xhr.mozResponseArrayBuffer // Firefox
xhr.response // Chrome
Then you can apply a TypedArray (eg: Int32Array) or a DataView on the buffer to read the result.
In order to make this process easier, I made a jQuery Patch to support the binary type and a DataView Wrapper that uses the latest available reading feature of the browser.

JavaScript has very little support for raw binary data. In general, it's best to live within this restriction. However, there's a trick I'm considering trying for a project of mine that involves manipulating huge bitmaps to do set operations in an OLAP database. This won't work in IE.
The basic idea is this: coerce the binary data into a PNG to send it to JavaScript, For example, a bitmap might be a black and white PNG, with black being 100% transparent. Then use Canvas operations to do bitwise data manipulation.
The HTML5 Canvas includes a pixel array type, which allows access to bytes in an image. Canvas also supports compositing operations, such as XOR. Lighten and darken should be able to do AND and OR. These operations are likely to be well optimized in any browser that supports them-- probably using the GPU.
If anyone tries this, please let me know how well it works.

That would be the other way around... pow and squareroot might be calculated by the Math-Class... I don't know if it is the fastest way, but it's as fast as the Windows Calculator in the "Programmer View".
AlertFormatedBin();
function AlertFormatedBin()
{
var vals = decToBinArr(31,8);
var i;
var s = "";
var mod = vals.length % 4;
for(i= 0; i <mod;i++)
{
s+=vals[i];
}
if(i>0)
s+=" ";
var j = i;
for(i;i<vals.length;i++)
{
s+=vals[i];
if(i-j != 0 && (i+1-j)%4 == 0)
{
s+=" ";
}
}
alert(s);
}
function decToBinArr(dec, minSize)
{
var mod = dec%2;
var r = new Array();
if(dec > 1)
{
dec-=mod;
var bd = squareRootRoundedDown(dec);
if(minSize && minSize-1 > bd)
bd = minSize-1;
else
var i;
for(i = bd; i>0;i--)
{
var nxt = pow(2,i);
if(dec >= nxt)
{
r[i] = 1;
dec-=nxt;
}
else
{
r[i] = 0;
}
}
}
r[0]= mod;
r.reverse();
return r;
}
function squareRootRoundedDown(dec)
{
if(dec<2)
return 0;
var x = 2;
var i;
for(i= 1;true;i++)
{
if(x>=dec)
{
i = x == dec ? i : i-1;
break;
}
x= x*2;
}
return i;
}
function pow(b,exp)
{
if(exp == 0)
return 0;
var i = 1;
var r= b;
for(i = 1; i < exp;i++)
r=r*b;
return r;
}

In the near future you will be able to use ArrayBuffers and File API Blobs.

As #Zippy pointed out in a comment, the more recent (late 2016) solutions include:
DataView (Standard)
jDataView (polyfill/extension of DataView)
jBinary (built on jDataView)

Javascript doesn't provide a mechanism to load an object in any form other than simple strings.
Closest you can do is serializing the object to a string, optionally encrypting/compressing it, sending it to the browser, and decrypting/decompressing if necessary, checking for sanity, eval() and pray().
Instead of using eval (which is not quite safe), you can use your own format (alternatively, xml or json for which there are plenty of libs) and parse it yourself.
As a side note, if you want this for obfuscation after the browser gets the usable data (after decrypting/decompressing), it is too easy to circumvent.

Percent encoding can unescape strings into a direct 1<->1 representaion of any binary blob and is also portable across browsers;
unescape("%uFFFF%uFFFF%uFFFF");
Most browser exploit's use this technique for embedding shellcode into HTML pages, it works very well for creating arbitrary binary streams.

jBinary "makes it easy to create, load, parse, modify and save complex binary files and data structures in both browser and Node.js."
I haven't used it, but it's what I found when asking the same question asked here...

Welcome to everyone who found this older post on Google. I figured out a solution that works in Chrome as of 2019, so hopefully this is just some added feature or something most people missed.
You can use a 0b prefix to your number. It won't quite get the binary representation, but you can easily convert it to an integer for storage. For example, you could store the binary number 1010 as such:
var binNum = 0b1010 //Stores as an integer, which would be 10
If you're curious, it also works for hexadecimal with the 0x prefix:
var binNum = 0x1010 //Stores as an integer, which would be 4112

Categories