I'm getting a binary file from my computer in javascript. And this binary file contains some long numbers in the following:
20
56599565 6559565
65656589 6595956
26989565 5656595
...
My question is that how can I get these numbers?
I have a binary file that contains numbers, its created with serialization on c++. And the first 40 bytes of this binary file is empty. After these 40 bytes have passed, my numbers are starting to come.
My solution:
After reading the file successfully I have an ArrayBuffer right? And I know that my numbers start from the 40. bytes. So I sliced it.
// rawData ArrayBuffer(197283) {}
var frameCount = rawData.slice(40,48);
var frameCountView = new Float64Array(frameCount);
And then what should I do?
I think we can get the Int8Array from Arraybuffer;
// rawData ArrayBuffer(197283) {}
var fileContent = rawData.slice(40);
var frameCountView = new Int64Array(fileContent);
var byteArray = new BigUint64Array(fileContent);
Related
Say I have an array containing a million random numbers:
[ 0.17309080497872764, 0.7861753816498267, ...]
I need to save them to disk, to be read back later. I could store them in a text format like JSON or csv, but that will waste space. I'd prefer a binary format where each number takes up only 8 bytes on disk.
How can I do this using node?
UPDATE
I did not find an answer to this specific question, with a full example, in the supposedly duplicate question. I was able to solve it myself, but in a verbose way that could surely be improved:
// const a = map(Math.random, Array(10));
const a = [
0.9651891365487693,
0.7385397746441058,
0.5330173086062189,
0.08100066198727673,
0.11758119861500771,
0.26647845473863674,
0.0637438360410223,
0.7070151519015955,
0.8671093412761386,
0.20282735866103718
];
// write the array to file as raw bytes (80B total)
var wstream = fs.createWriteStream('test.txt');
a.forEach(num => {
const b = new Buffer(8);
b.writeDoubleLE(num);
wstream.write(b);
})
wstream.end(() => {
// read it back
const buff = fs.readFileSync('test.txt');
const aa = a.map((_, i) => buff.readDoubleLE(8*i));
console.log(aa);
});
I think this was answered in Read/Write bytes of float in JS
The ArrayBuffer solution is probably what you are looking for.
I have a BMP image and I need to send it to a device via TCP/IP. We already have the C library in our company, which can handle this, but I need to do it in JavaScript. Unfortunately, I don't have access to the source code of the library, nor to the device system.
This is a Base64 string of a sample image (a black&white checkmark):
Qk2+AAAAAAAAAD4AAAAoAAAAIAAAACAAAAABAAEAAAAAAIAAAADEDgAAxA4AAAAAAAAAAAAAAAAAAP///wD//////+H////A////gH///wB///4AP//8AD//+AAf//AAD//gAA//yAAH/5wAB/+8EAP/fjAB/394Af+++AD/vPwAf8P+AH///gA///9AH///v5///7/P///fx///7+P//+/z///38f//+/n///38///+/f///v3///97////hw==
I use window.atob and I am encoding it into HEX. I use this JS function:
function toHex(str) {
var result = '';
for (var i = 0; i < str.length; i++) {
result += str.charCodeAt(i).toString(16);
}
return result;
}
var str = window.atob(base64img);
var result = toHex(str);
It gives me this result, which is almost expected result:
424dbe00000003e0002800020000200001010000080000c4e00c4e00000000000000ffffff0ffffffffffe1ffffffc0ffffff807fffff07ffffe03ffffc03ffff801ffff00fffe00fffc807ff9c07ffbc103ff7e301ff7f781ffbef80ffbcfc07fc3fe07ffffe03fffff401fffffbf9fffffbfcfffffdfc7ffffefe3ffffeff3fffff7f1fffffbf9fffffdfcfffffefdfffffefdffffff7bffffff87
The library sends the exact same image correctly (the device accepts the message). This is how it looks like (copied from log):
be00424dbe000000000000003e000000280000002000000020000000010001000000000080000000c40e0000c40e0000000000000000000000000000ffffff00ffffffffffe1ffffffc0ffffff807fffff007ffffe003ffffc003ffff8001ffff0000fffe0000fffc80007ff9c0007ffbc10101003ff7e3001ff7f7801ffbef800ffbcfc007fc3fe007ffffe003fffff401fffffbf9fffffbfcfffffdfc7ffffefe3ffffeff3fffff7f1fffffbf9fffffdfcfffffefdfffffefdffffff7bffffff87
So this is what I need to get from Base64 in my JavaScript. Is it even possible? Or am I missing something?
Documentation for the library says the image has to be 2B binary data (Little Endian). I don't understand it. Should I encode the image any other way?
One option would be encode each byte separately to ensure correct endianness.
img = "Qk2+AAAAAAAAAD4AAAAoAAAAIAAAACAAAAABAAEAAAAAAIAAAADEDgAAxA4AAAAAAAAAAAAAAAAAAP///wD//////+H////A////gH///wB///4AP//8AD//+AAf//AAD//gAA//yAAH/5wAB/+8EAP/fjAB/394Af+++AD/vPwAf8P+AH///gA///9AH///v5///7/P///fx///7+P//+/z///38f//+/n///38///+/f///v3///97////hw=="
str = atob(img)
buf = []
function hex(str, pos) {
return ('000' + (str.charCodeAt(pos) || 0).toString(16)).substr(-2);
}
for (var i = 0; i < str.length; i+= 4) {
buf.push(hex(str, i+2));
buf.push(hex(str, i+3));
buf.push(hex(str, i+0));
buf.push(hex(str, i+1));
}
console.log(buf.join(''))
This doesn't match your desired output exactly, are you sure it corresponds to the given base64 string?
On the other side, your initial output looks better, 424d are correct bytes to start a BMP file (BM signature), while be00 are not.
This is a Base64 string of a sample image (a black&white checkmark)...
It gives me this result, which is almost expected result:
424dbe00000003e0002800020000200001010000080000c4e00c4e00000000000000ffffff0ffffffffffe1ffffffc0ffffff807fffff07ffffe03ffffc03ffff801ffff00fffe00fffc807ff9c07ffbc103ff7e301ff7f781ffbef80ffbcfc07fc3fe07ffffe03fffff401fffffbf9fffffbfcfffffdfc7ffffefe3ffffeff3fffff7f1fffffbf9fffffdfcfffffefdfffffefdffffff7bffffff87
What do you mean by "almost expected result"?, those bytes make the image below (save as .bmp).
Your expected "a black&white checkmark" image :
PS:
The correct .bmp file has 190 bytes. Your library version (copied from logs) gives result of 194 bytes.
The library's version of bytes has a beginning 2-byte short for file size. That is BE 00 (= 190 when read as endian of 00 BE) followed by rest of Bitmap file bytes (190 total bytes). This makes total of 190 bytes + the 2 bytes of short. Then it adds these two mysterious 10 and 10 bytes starting at position 114. Grand total of 194 bytes.
To me... the library is corrupting the already fine image bytes, but you say the device accepts it?
Does it accept the "almost expected result" hex bytes too?
I'm creating a Buffer from a long string of hex but getting some size mismatching (when I try to decrypt it). Sure enough when I run the following code:
var hexData = fs.readFileSync(fileName).toString().trim();
var bufferData = new Buffer(hexData, 'hex');
console.log(Buffer.byteLength(hexData, 'hex'));
console.log(bufferData.length);
my output is:
232548
30
Why is the whole string not being loaded into the Buffer?
EDIT: I noticed I was being silly and should be doing
var bufferData = fs.readFileSync(fileName, 'hex');
But the length of that is 930194!
Assuming you meant hexData and not data when you build bufferData, Buffer.byteLength seems to happily accept malformed hex, whereas the Buffer constructor will strip it out of the buffer. Consider:
> Buffer.byteLength('ff00junk', 'hex')
4
> var b = new Buffer('ff00junk', 'hex')
> b.length
2
> b
<Buffer ff 00>
Perhaps your file contains invalid hex?
I want to generate a file containing strings alongside Float32Arrays in my web app. I can write this data into a blob and then a file like this:
var myFirstFloat32Array = new Float32Array([42, 1, 2, 3]);
var myString = 'This is my string. It is a beautiful string.';
var blob = new Blob([
new Uint8Array([myFloat32Array.length]), // Store the size.
myFloat32Array,
new Uint8Array([myString.length]), // Store the size.
myString);
var downloadLink = document.createElement('a');
downloadLink.href = window.URL.createObjectURL(blob);
downloadLink.download = 'generatedFile';
downloadLink.innerHTML = 'Click to download generated file.';
So ... that successfully writes the file, but how do I read the string following the float 32 array?
I can read the first float32 array easily by reading the number from the first 4 bytes (which gives me the size of the array) and then constructing a Float32Array of that length.
However, how do I read the string? The string could contain characters outside of the ASCII range, so I can't just btoa the string, convert to a Uint8Array, and then write it.
Maybe I could convert the string to a blob first, get its size from the blob, and then somehow decode it later while reading (but how do I know the character encoding used later)?
What's the standard way to store string data alongside byte data in javascript for later decoding?
As a personal project, I'm writing a 6502 emulator in JavaScript (HTML5 based). I'm porting some bits of it from a predecessor I created in C. To load in the files (ROMs in my case), I could use this C code:
unsigned char* buffer = calloc(1, 4096);
FILE* file = fopen("xyz", "rb");
fread(buffer, 1, 4096, file);
fclose(file);
and access it like this:
char firstChar = buffer[0];
short nextShort = (buffer[2] << 8) | buffer[1];
free(buffer);
Now, I need to be able to port this to JavaScript. For input, I can do something like this using a file input:
var file = document.getElementById("picker").files[0];
var reader = new FileReader();
reader.readAsDataURL(file);
From here, I have reader.result as a giant, base64 encoding of the file. I need a way to access the binary file as I did in the beginning C example, where I can simply get the values (or use simple bitwise operations)
I'm guessing the most feasible solution would be an array of values 0-255, but I just need access, regardless of how.
#bfavaretto pointed me in the correct direction
var file = document.getElementById("picker").files[0];
var reader = new FileReader();
reader.readAsArrayBuffer(file);
var buffer = new Uint8Array(reader.result);
Which can then be accessed as buffer[0] for the first byte.
I'm the author of simplebuf library. It allows quick binary parsing/serialization and is purely written in javascript.
Example:
var layout = [
sb.field("len", sb.type.uint(32)),
sb.field("padding", sb.type.uint(32)),
sb.field("id", sb.type.string_dynamic("len"))
];
var original = {len: 4, "padding": 999, "id": "1234"};
sb.write(buffer, 0, original, layout);
https://github.com/conceptacid/simplebuf.js