We Keep Coding

Hash large files with crypto.subtle.digest("SHA-256", buffer)

i have developed a web application where a user can select multiple files via a input field. Then the sha-256 checksums are calculated by the following code. The code (taken from developer.mozilla.org) only works for small files. What do I have to change to handle large files (e.g. 1GB+) too?
function sha256(buffer){
return crypto.subtle.digest("SHA-256", buffer).then(function (hash) {
return hex(hash);
});
}
function hex(buffer) {
var hexCodes = [];
var view = new DataView(buffer);
for (var i = 0; i < view.byteLength; i += 4) {
// Using getUint32 reduces the number of iterations needed (we process 4 bytes each time)
var value = view.getUint32(i)
// toString(16) will give the hex representation of the number without padding
var stringValue = value.toString(16)
// We use concatenation and slice for padding
var padding = '00000000'
var paddedValue = (padding + stringValue).slice(-padding.length)
hexCodes.push(paddedValue);
}
// Join all the hex strings into one
return hexCodes.join("");
}

Javascript/node.js Object that can store arbitrary binary data and do appending very fast

I need to process lots of binary data in a Node.js app. As data arrive in many small chunks (in the format of Buffer object) that comes to my part of code via a callback, I have to do MANY operations per second like appending, slicing etc.
I was tempted to store the binary data in Javascript string, which support appending, slicing etc. But unfortunately I can't really convert the (arbitrary) binary data to string, which has to have an valid encoding, like UTF8.
To use Buffer object, the appending operation become very expensive. For example, the following code snippet took 1.5 seconds on my P7 processor.
var a = new Buffer([1])
var b = new Buffer([2])
var start = new Date()
for (i=0; i<100000; i++) {
a = Buffer.concat([a, b], a.length + 1)
}
console.log(new Date() - start)
If I were doing simple string appending a += b assuming a and b are strings, it will take only 0.01 second.
I wonder if there is an object in Javascript that can store arbitrary binary data and support appending very efficiently.
Thanks in advance
Update1
Tried TypeArray, the speed is a little better, but it's still far slower than string appending.
var a = new Uint8Array(),
b = new Uint8Array(1);
var c
b[0] = 11
var start = new Date()
for (i=0; i<100000; i++) {
c = new Uint8Array (a.length + b.length)
c.set(a,0)
c.set(b, a.length)
a = c
}
console.log(new Date() - start)
console.log(a.length)

I think smart-buffer might be what you're after? It allows you to write other buffers into it and will dynamically resize as needed.
Testing script:
const SmartBuffer = require('smart-buffer').SmartBuffer;
// set up buffers
var a = new Buffer([1])
var smart_a = new SmartBuffer();
smart_a.writeInt8(1);
var b = new Buffer([2])
// time buffer concatenation method
console.time("Buffer concatenation");
for (let i = 0; i < 100000; i++) {
a = Buffer.concat([a, b], a.length + 1)
}
console.timeEnd("Buffer concatenation");
// time smart buffer writeBuffer method
console.time("Smart Buffer writing");
for (let i = 0; i < 100000; i++) {
smart_a.writeBuffer(b);
}
let final_smart_a = smart_a.toBuffer();
console.timeEnd("Smart Buffer writing");
// check that resulting buffers match
for (let i = 0; i < 100000; i++) {
console.assert(a[i] == final_smart_a[i]);
}
Results (1 trial):
Buffer concatenation: 2110.282ms
Smart Buffer writing: 14.971ms

Convert a byteArray into an IntegerArray using javascript

I receive a bytearray and I want to convert it into a intarray.
Is this possible in NodeJS?
Reason to do that:
A proxy receives values from a serialconnection which is a bytearray and I try to decode that and put it into a more specific JSON object.
var uint8Message = new Uint8Array(8),
output = [];
uint8Message[0] = 40;
uint8Message[1] = 40;
uint8Message[2] = 40;
uint8Message[3] = 40;
uint8Message[4] = 40;
uint8Message[5] = 40;
uint8Message[6] = 40;
uint8Message[7] = 40;
var counter = 0;
var intermediate = [];
for (var i = 0; i< uint8Message.byteLength; i++) {
if (counter < 4) {
intermediate.push(uint8Message[i]);
}
counter++;
if (counter === 3 ){
output.push(new Uint16Array(intermediate));
counter = 0;
intermediate = [];
}
}
console.log(output);
I am sending a intArray which is converted to byteArray from arduino to a NodeJS serialport handler. I want to get 8 integer values in array:
Status and value for four engines.
So in the end I want to have this:
[1,89,2,49,1,28,3,89]
Which is not complete correct with the example. But the example above is for testing.

Still not sure I understand your question correctly but if you want to convert Uint8Array to say Uint32Array you could do
const uint8Array = new Uint8Array(8).fill(40)
console.log(new Uint32Array(uint8Array.buffer))
If you need plain old js array you could do
const uint8Array = new Uint8Array(8).fill(40)
const intArray = Array.from(new Uint32Array(uint8Array.buffer))
console.log(Array.isArray(intArray))
Also you might want to take a look at what is called DataView that allows low level access to buffers' contents.

Convert large array of integers to unicode string and then back to array of integers in node.js

I have some data which is represented as an array of integers and can be up to 200 000 elements. The integer value can vary from 0 to 200 000.
To emulate this data (for debugging purposes) I can do the following:
let data = [];
let len = 200000
for (let i = 0; i < len; i++) {
data[i] = i;
}
To convert this array of integers as an unicode string I perform this:
let dataAsText = data.map((e) => {
return String.fromCodePoint(e);
}).join('');
When I want to convert back to an array of integers the array appears to be longer:
let dataBack = dataAsText.split('').map((e) => {
return e.codePointAt(e);
});
console.log(dataBack.length);
How does it come ? What is wrong ?
Extra information:
I use codePointAt/fromCodePoint because it can deal with all unicode values (up to 21 bits) while charCodeAt/fromCharCode fails.
Using, for example, .join('123') and .split('123') will make that the variable dataBack is the same length as data. But this isn't an elegant solution because the size of the string dataAsText will unnecessarily be very large.
If let len is equal or less to 65536 (which is 2^16 or 16 bits max value) then everything works fine. Which is strange ?
EDIT:
I use codePoint because I need to convert the data as unicode text so that the data is short.
More about codePoint vs charCode with an example:
If we convert 150000 to a character then back to an integer with codePoint:
console.log(String.fromCodePoint("150000").codePointAt(0));
this gives us 150000 which is correct. Doing the same with charCode fails and prints 18928 (and not 150000):
console.log(String.fromCharCode("150000").charCodeAt(0));

That's because higher code point values will yield 2 words, as can be seen in this snippet:
var s = String.fromCodePoint(0x2F804)
console.log(s); // Shows one character
console.log('length = ', s.length); // 2, because encoding is \uD87E\uDC04
var i = s.codePointAt(0);
console.log('CodePoint value at 0: ', i); // correct
var i = s.codePointAt(1); // Should not do this, it starts in the middle of a sequence!
console.log('CodePoint value at 1: ', i); // misleading
In your code things go wrong when you do split, as there the words making up the string are all split, discarding the fact that some pairs are intended to combine into a single character.
You can use the ES6 solution to this, where the spread syntax takes this into account:
let dataBack = [...dataAsText].map((e, i) => {
// etc.
Now your counts will be the same.
Example:
// (Only 20 instead of 200000)
let data = [];
for (let i = 199980; i < 200000; i++) {
data.push(i);
}
let dataAsText = data.map(e => String.fromCodePoint(e)).join("");
console.log("String length: " + dataAsText.length);
let dataBack = [...dataAsText].map(e => e.codePointAt(0));
console.log(dataBack);
Surrogates
Be aware that in the range 0 ... 65535 there are ranges reserved for so-called surrogates, which only represent a character when combined with another value. You should not iterate over those expecting that these values represent a character on their own. So in your original code, this will be another source for error.
To fix this, you should really skip over those values:
for (let i = 0; i < len; i++) {
if (i < 0xd800 || i > 0xdfff) data.push(i);
}
In fact, there are many other code points that do not represent a character.

I have a feeling split doesn't work with unicode values, a quick test above 65536 shows that they become double the length after splitting
Perhaps look at this post and answers, as they ask a similar question

I don't think you want charPointAt (or charCodeAt) at all. To convert a number to a string, just use String; to have a single delimited string with all the values, use a delimiter (like ,); to convert it back to a number, use the appropriate one of Number, the unary +, parseInt, or parseFloat (in your case, Number or + probably):
// Only 20 instead of 200000
let data = [];
for (let i = 199980; i < 200000; i++) {
data.push(i);
}
let dataAsText = data.join(",");
console.log(dataAsText);
let dataBack = dataAsText.split(",").map(Number);
console.log(dataBack);
If your goal with codePointAt is to keep the dataAsText string short, then you can do that, but you can't use split to recreate the array because JavaScript strings are UTF-16 (effectively) and split("") will split at each 16-bit code unit rather than keeping code points together.
A delimiter would help there too:
// Again, only 20 instead of 200000
let data = [];
for (let i = 199980; i < 200000; i++) {
data.push(i);
}
let dataAsText = data.map(e => String.fromCodePoint(e)).join(",");
console.log("String length: " + dataAsText.length);
let dataBack = dataAsText.split(",").map(e => e.codePointAt(0));
console.log(dataBack);

If you're looking for a way to encode a list of integers so that you can safely transmit it over a network, node Buffers with base64 encoding might be a better option:
let data = [];
for (let i = 0; i < 200000; i++) {
data.push(i);
}
// encoding
var ta = new Int32Array(data);
var buf = Buffer.from(ta.buffer);
var encoded = buf.toString('base64');
// decoding
var buf = Buffer.from(encoded, 'base64');
var ta = new Uint32Array(buf.buffer, buf.byteOffset, buf.byteLength >> 2);
var decoded = Array.from(ta);
// same?
console.log(decoded.join() == data.join())
Your original approach won't work because not every integer has a corresponding code point in unicode.
UPD: if you don't need the data to be binary-safe, no need for base64, just store the buffer as is:
// saving
var ta = new Int32Array(data);
fs.writeFileSync('whatever', Buffer.from(ta.buffer));
// loading
var buf = fs.readFileSync('whatever');
var loadedData = Array.from(new Uint32Array(buf.buffer, buf.byteOffset, buf.byteLength >> 2));
// same?
console.log(loadedData.join() == data.join())

Address to WCHAR_T to pass to ReadProcessMemory

I'm having trouble passing a WCHAR_T to ReadProcessMemory
This is how to succesfully pass a pointers address to ReadProcessMemory, I can do it with structures:
remote_tbb = ralloc_alloc(struct_TBButton.size);
var rez = SendMessage(hToolbar, TB_GETBUTTON, i, ctypes.voidptr_t(remote_tbb));
if (!rez) { throw new Error('Failed on SendMessage of TB_GETBUTTON') }
var local_tbb = new struct_TBButton();
var retRead = ralloc_read(remote_tbb, local_tbb.address());
var freed = ralloc_free(remote_tbb);
But now I need to do with WCHAR_T, so this is what I have:
var chars = SendMessage(hToolbar, TB_GETBUTTONTEXTW, local_tbb.idCommand, ctypes.voidptr_t(0));
console.log('chars=', chars, chars.toString(), uneval(chars));
if (chars && parseInt(chars.toString()) > 0) {
var remote_buf = ralloc_alloc(parseInt(chars.toString()));
var charsRe = SendMessage(hToolbar, TB_GETBUTTONTEXTW, local_tbb.idCommand, ctypes.voidptr_t(remote_buf));
console.log('charsRe=', charsRe);
var local_buf = ctypes.jschar; //WCHAR_T
var retRead = ralloc_read(remote_buf, local_buf.address()); ///PROBLEM LINE
console.log('retRead=', retRead);
var freed = ralloc_free(remote_buf);
console.log('freed=', freed);
console.log('Button Text = ', local_buf, local_buf.toString());
} else {
console.log('Button Text = NONE');
}
So my problem is on line:
var retRead = ralloc_read(remote_buf, local_buf.address());`
and it is specifically on the local_buf.address()
Errors in my experimenting that get thrown are:
expected type pointer, got ctypes.jschar
local_buf.address is not a function
So how to pass WCHAR_T as reference?
Edit:
Here is my ralloc_read implemetnation:
function ralloc_read(remote_address, local_buffer) {
var found_addr;
for (var i = 0; i < buffers.length; i++) {
if (buffers[i][0] == remote_address) {
found_addr = buffers[i]
break;
}
}
if (!found_addr) {
return null;
}
/*using the found remote address(found_addr[0]),
*i read size bytes (found_addr[1]) into my local_buffer*/
//console.info('found_addr[0]', found_addr[0].toString());
var rez = ReadProcessMemory(proc, found_addr[0], local_buffer, found_addr[1], 0);
return rez;
}

If ralloc_read calls ReadProcessMemory, then you'll need to allocate a jschar array that will receive the result.
var local_buf = ctypes.jschar.array()(chars);
ralloc_read(remote_buf, local_buf.address());
var str = local_buf.readString();
Edit However, the allocation call is wrong:
ralloc_alloc(parseInt(chars.toString()));
This will allocate chars bytes, e.g. chars = 11, 11 bytes.
A wchar_t/jschar however is not 1 byte but 2 bytes.
ctypes.jschar.size
// 2
So you'll actually need to allocate a remote memory buffer that is larger:
ralloc_alloc(parseInt(chars.toString()) * ctypes.jschar.size);
// That would be ralloc_alloc(count * sizeof(wchar_t*)) in C/C++
The local_buf stuff is correct, though as js-ctypes arrays will automatically calculate the required storage if it knows the size of the array element type, so a ctypes.jschar.array()(11) buffer will actually have 11 elements of size 2 bytes, i.e. 11 items * 2 bytes/item == 22 bytes.