I read a file using FileReader.readAsArrayBuffer and then do something like this:
var compressedData = pako.gzip(new Uint8Array(this.result));
var blob1 = new Blob([compressedData]); // size = 1455338 bytes
var blob2 = new Blob(compressedData); // size = 3761329 bytes
As an example: if result has 4194304 bytes, after compression it will be size 1455338 bytes. But for some reason the Uint8Array needs to be wrapped in an Array. Why is this?
Cf. documentation for BLOB constructor:
https://developer.mozilla.org/en-US/docs/Web/API/Blob/Blob
[the first argument] is an Array of ArrayBuffer, ArrayBufferView, Blob, DOMString objects, or a mix of any of such objects, that will be put inside the Blob. DOMStrings are encoded as UTF-8.
I'm not sure how it works under the hood, but basically the constructor expects an array of things it will pack into the BLOB. So, in the first case, you're constructing a BLOB of a single part (i.e. your ArrayBuffer), whereas in the second you're constructing it from 1455338 parts (i.e. each byte separately).
Since the documentation says the BLOB parts can only be arrays or strings, it probably ends up converting each of the byte values inside your ArrayBuffer into UTF-8 strings, which means instead of using 1 byte per number, it uses 1 byte per decimal digit (the ratio of the two result sizes seems to support this, since single byte values are 1-3 digits long, and the larger BLOB is about 2.5 times the size of the smaller). Not only is that wasteful, I'm pretty sure it also renders your ZIP unusable.
So, bottom line is, the first version is the correct way to go.
Unfortunately, MDN article is almost wrong here, and at best misleading.
From the specs:
The Blob() constructor can be invoked with the parameters below:
A blobParts sequence
which takes any number of the following types of elements, and in any order:
BufferSource elements.
Blob elements.
USVString elements.
... [BlobPropertyBag, none of our business here]
So a sequence here can be a lot of things, from an Array to a Set going through an multi-dimensional Array.
Then the algorithm is to traverse this sequence until it finds one of the three types of elements above.
So what happens in your case is that a TypedArray can be converted to a sequence. This means that when you pass it as the direct parameter, it will not be able to see its ArrayBuffer and the algorithm will traverse its content and pick up the values (here 8 bit numbers converted to Strings), which is probably not what you expected.
In the other hand, when you wrap your Uint8Array through an Array, the algorithm is able to find the BufferSource your Uint8Array points to. So it will use it instead (binary data, and probably what you want).
var arr = new Uint8Array(25);
arr.fill(255);
var nowrap = new Blob(arr);
var wrapped = new Blob([arr]);
test(nowrap, 'no wrap');
test(wrapped, 'wrapped');
function test(blob, msg) {
var reader = new FileReader();
reader.onload = e => console.log(msg, reader.result);
reader.readAsText(blob);
}
Related
Take the following snippet:
const arr = [1.1, 2.2, 3.3]
const arrBuffer = (Float32Array.from(arr)).buffer
How would one cast this ArrayBuffer to a SharedArrayBuffer?
const sharedArrBuffer = ...?
Note that both ArrayBuffer and SharedArrayBuffer are backing data pointers that you only interact with through a typed array (like Float32Array, in your example). Array Buffers represent memory allocations and can't be "cast" (only represented with a typed array).
If you have one typed array already, and need to copy it into a new SharedArrayBuffer, you can do that with set:
// Create a shared float array big enough for 256 floats
let sharedFloats = new Float32Array(new SharedArrayBuffer(1024));
// Copy floats from existing array into this buffer
// existingArray can be a typed array or plain javascript array
sharedFloats.set(existingArray, 0);
(In general, you can have a single array buffer and interact with it through multiple "typed lenses" - so, basically, casting an array buffer into different types, like Float32 and Uint8. But you can't cast an ArrayBuffer to a SharedArrayBuffer, you'll need to copy its contents.)
I have a c# application that converts a double array to a byte array of data to a node.js server which is converted to a Buffer (as convention seems to recommend). I want to convert this buffer into an array of the numbers originally stored in the double array, I've had a look at other questions but they either aren't applicable or just don't work ([...buf], Array.prototype.slice.call(buf, 0) etc.).
Essentially I have a var buf which contains the data, I want this to be an array of integers, is there any way I can do this?
Thank you.
First, you need to know WHAT numbers are in the array. I'll assume they are 32bit integers. So first, create encapsulating Typed Array around the buffer:
// #type {ArrayBuffer}
var myBuffer = // get the bufffer from C#
// Interprets byte array as 32 bit int array
var myTypedArray = new Int32Array(myBuffer);
// And if you really want standard JS array:
var normalArray = [];
// Push all numbers from buffer to Array
normalArray.push.apply(normalArray, myTypedArray);
Note that stuff might get more complicated if the C#'s array is in Big Endian, but I assume it's not. According to this answer, you should be fine.
I managed to do this with a DataView and used that to iterate over the buffer, something I'd tried before but for some reason didn't work but does now.
I am using jsmodbus npm library to retrieve 16 bit registers using readHoldingRegisters() function.
// array of values returned by jsmodbus readHoldingRegisters for two 16 bit registers
data = [ 17008, 28672 ]
I am using an ArrayBuffer and DataView to set and get the data in the required format:
const buffer = new ArrayBuffer(4)
const view = new DataView(buffer)
I understand that data returned from registers is always 16-bit integers split over two 8-bit byte values, so should I set the two bytes to the view as two consecutive ints even though it may be later retrieved from the view as an Int16 or Float32. Is this correct?
Secondly, if I expect to retrieve the data as signed Int16 or Float32, is it necessary to set the high byte as signed and the low byte as unsigned, like so:
view.setInt16(0, data[0])
view.setUint16(0, data[1])
And third: notwithstanding the need to set proper endianness, does it even matter which method you use when setting the data against the view, as the order of bytes and bits in the view isn't affected by which method you set against, only when you retrieve that data back out?
Certainly, it doesn't appear to:
view.setInt16(0, data[0])
view.setInt16(1, data[1]) // notice: setInt16
val = view.getFloat32(0)
// val = 122.880859375
view.setInt16(0, data[0])
view.setUint16(1, data[1]) // notice: set*U*int16
val = view.getFloat32(0)
// val = 122.880859375
Sanity check appreciated!
For some Int32Array a, I have tried this:
var d = zlib.deflateSync(new Buffer(a));
But when I do
var b = new Int32Array(zlib.inflateSync(d));
I get b which is different than a.
I am aware of idea of turning the array into a string before deflating. This is just not memory efficient.
Is there a way to do this efficiently?
The main problem here is not in deflating and inflating, as those two actions just transfer data from one Buffer element to another Buffer element, and vice versa, as expected. Problem here is how to transfer typed array (e.g. Int32Array) to Buffer, and from Buffer back to typed array. To do that we have to deal with Buffer, ArrayBuffer, and typed arrays here.
Let's have Int32Array a to start with:
var a = new Int32Array([3,-2,258]);
If you create a Buffer from this typed array directly with var b = new Buffer(b);, you will get a buffer which contain 3 bytes instead of 12 (each element in Int32Array has 4 bytes), which means you will lose some information. The right way to do this is to create Buffer from ArrayBuffer. You can do it like so:
var b = new Buffer(a.buffer);
That way you get a Buffer which contains 12 bytes. Now it is easy to deflate and inflate this buffer b:
var c = zlib.deflateSync(b);
var d = zlib.inflateSync(c);
Now buffers d and b contain the same elements. Nothing special about this.
Now if you create Int32Array directly from Buffer d, you will get 4 times more elements that you should because each byte in the Buffer d will be mapped to one element in Int32Array. To overcome this problem, we will use Uint8Array:
var e = new Uint8Array(d);
We used Uint8Array just to map each byte of the buffer d to one byte of an typed array of which we now have access to its ArrayBuffer. The last step is to create the final Int32Array from ArrayBuffer of typed array e:
var f = new Int32Array(e.buffer);
Finally we have f that looks the same as a.
Note that some steps above depend on Endianness. If you do all this on the same machine you should be fine with the steps above.
I'd like to know if there's a way of merging multiple typed arrays of different types into one arraybuffer.
At the moment I'm sending binary data to a server via websockets. But now I'm just using Uint8 Arrays. So I assemble a new Uint8Array([with, the, data, I, need]) and send the buffer of this array (Unint8Array.buffer) to the server.
But now it could happen, that I need some Uint16 or Float32 Data within my request, mixed inside my Uint8 Data. Is there any way to achieve this and how.
EDIT: The length of the request data is unknown as I have to assemble the requests dynamic.
Thanks for any reply :)
Yes, you can do that, that's why types arrays have the separation of the array and the buffer it stores things in. Here's an example of two views of a buffer, one with bytes and one with words: Live Copy
// Create the buffer
var buf = new ArrayBuffer(10);
// Create a byte view of it
var a8 = new Int8Array(buf);
// Create a word view of it
var a16 = new Int16Array(buf, 2, 2);
// Set a couple of words
a16[0] = 0x1221;
a16[1] = 0x2442;
// Show the contents of the byte array
var i;
for (i = 0; i < a8.length; ++i) {
console.log("0x" + a8[i].toString(16));
}
Output:
0x0
0x0
0x21
0x12
0x42
0x24
0x0
0x0
0x0
0x0
There I've created the ArrayBuffer explicitly for clarity, but you can also do it implicitly: Live Copy (same output)
// Create a byte view and underlying buffer
var a8 = new Int8Array(10);
// Create a word view of the byte view's buffer
var a16 = new Int16Array(a8.buffer, 2, 2);
// Set a couple of words
a16[0] = 0x1221;
a16[1] = 0x2442;
// Show the contents of the byte array
var i;
for (i = 0; i < a8.length; ++i) {
console.log("0x" + a8[i].toString(16));
}
EDIT: The length of the request data is unknown as I have to assemble the requests dynamic.
If you have allocated a large enough ArrayBuffer, that's not necessarily a problem, you just need to create the view at the relevant starting point for (say) the rest of the length of the buffer. But you'll have to keep track of how big what you've written is at least after you've written it, so you know where to continue with other data.