I am facing issue in getting data from a file as a buffer and then converting it into a long sample code is
var fs = require('fs');
fs.open('24.mkt', 'r', function(status, fd) {
if (status) {
console.log(status.message);
return;
}
var buffer = new Buffer(4);
fs.read(fd, buffer, 0, 4, 0, function(err, num) {
console.log(buffer.values());
});
});
file link - > https://archive.org/download/kamo_24/24.mkt
first 4 bytes contains Timestamp in Long in 4 Bytes
You can use node.js's Buffer.readInt32BE function. It reads 4 bytes in the given order (Big, or Little endian) to a variable, starting at the offset parameter:
// Unix timestamp now: 1492079016
var buffer = Buffer.from([0x58, 0xEF, 0x51, 0xA8]);
var timestamp = buffer.readInt32BE(0);
process.stdout.write(timestamp.toString());
You probably want to use readUInt32BE and/or readUInt32LE (of buffer) to convert buffer values to number.
You may also try to converting a values in a buffer to numeric value using node-bigint or node-bignum (it's probably overkill for a 4 bytes case but if you need to deal with a bigger numbers it may suit the need), both allow creation from buffer in a similar form (just be aware of options differences):
bignum.fromBuffer(buf, opts)
// or
bigint.fromBuffer(buf, opts)
Related
I'm sending messages to SQS using the AWS-SDK for JavaScript. Each message needs to be 256kb in size tops.
Each message is a JSON object that gets decoded on another service.
Option 1: JSON Object as string: Count length and make sure it's less than 262144?
function* getStuff(rows, someConfig) {
let totesPayload = 0
let payload = []
for (const row of rows) {
const singleItemInPayload = rowToPayload(row, someConfig)
if (singleItemInPayload.length + totesPayload < 262144 - (enclosingObjectSize())) {
payload.push(singleItemInPayload)
totesPayload += singleItemInPayload.length
} else {
yield({ payload })
payload = []
}
}
Option 2: Buffer.from(JSON Object as string): Count length of JSON Object and make sure it's less than 262144?
Most of the data is text, so I'm not sure I'm going to get any good help from putting it in a byte array.
Is option 2 necessary?
SQS uses UTF-8 for strings, so if any part of your message can contain non-ASCII characters, then you will need to measure the size by converting to bytes because UTF-8 is a variable width encoding which uses 1 to 4 bytes for a single character.
FireFox 46.0.1: I am using 3rd-party (easyrtc) software to send 15KB chunks of Float32Arrays between peers. Easyrtc insists that the data be JSON-able. Unfortunately, JSON.stringify yields a string more than twice as long as the original data: 16384 bytes of data becomes a string of length 35755. Below is my test code followed by the console output. What if anything can I do to reduce the stringify'd size? Is there a way to send the values only (no keys)? Can I use the 'replacer' argument to send only the values, and if so, don't I need to use a replacer on the corresponding JSON.parse on the receiving end?
var g_testBufferNBytes = 4096 * 4;
var g_testBuffer = new ArrayBuffer(g_testBufferNBytes);
var g_testBufferView = new Float32Array(g_testBuffer);
console.log("array byte length " + g_testBuffer.byteLength);
console.log("view byte length " + g_testBufferView.byteLength);
var j = JSON.stringify(g_testBufferView);
console.log("j length " + j.length);
var newBuf = JSON.parse(j);
console.log("newBuf length " + Object.keys(newBuf).length);
CONSOLE:
array byte length 16384
view byte length 16384
j length 35755
newBuf length 4096
Yes
ES6: Assume that your data are in let f32 = g_testBufferView (array Float32Array) ) - whe can save it as JSON array in at leas 4 ways:
// code
let f32json = JSON.stringify(f32);
let f32jsonArr = JSON.stringify(Array.from(f32));
let f32base64 = btoa(String.fromCharCode(...(new Uint8Array(f32.buffer))));
let f32base128 = ... // not trivial, look below
// decode
let df32json = new Float32Array(Object.values(JSON.parse(f32json)));
let df32jsonArr = new Float32Array(JSON.parse(f32jsonArr));
let df32base64 = new Float32Array(new Uint8Array([...atob(f32base64)].map(c => c.charCodeAt(0))).buffer);
let df32base128 = ... // not trivial, look below
Note that Object.values return values sorted by numeric keys (look here).
Here is working example. You can also use base128 do decode but I not use in this example (to not complicate it) - more details here.
If your Float32Array- f32 has 4096 elements equals to 0.3 then:
f32 has 16384 bytes,
f32json (j from your question) has 109483 bytes (which is >6x bigger than f32)
f32jsonArr has 81921 bytes (which is >5x bigger than f32)
f32base64 has 21848 bytes(which is ~1.3x bigger than f32)
f32base128 has 18725 bytes (whis is <1.15x bigger than f32) but chrome will send ~2x bigger request (depends on input data)
If your Float32Array- f32 has 4096 elements equals integer from 1 to 9 then:
f32 has 16384 bytes - CONST,
f32json (j from your question) has 35755 bytes (which is >2x bigger than f32)
f32jsonArr has 8193 bytes (which is 2x SMALLER (sic!) than f32)
f32base64 has 21848 bytes - CONST (which is ~1.3x bigger than f32)
f32base128 has 18725 bytes - CONST (whis is <1.15x bigger than f32) but chrome will send ~2x bigger request (depends on input data)
Conclusion
The smallest result which not depends of array values (result size is constant) we get for f32base64 ~33% bigger than input array size. For f32base128 - it contains valid JSON (string) which is something about <15% bigger than input, but chrome during sending increase this size (look here - on 'update' section). So use f32base64 - this is probably the smallest JSON that you can get without more sophisticated methods.
I want convert an integer (signed) to 32 bit (big endian) into a octet stream and give the octet stream as a array value to the constructor of a
Buffer Object.
I can create it in the console for example for the value -2000:
<code>
buf = Buffer(4)
buf.writeInt32BE(-2000)
buf // is <Buffer ff ff f8 30>
buf1 = new Buffer([0xff, 0xff, 0xf8, 0x30])
</code>
The value -3000 is for example -3000 : 0xff ,0xff, 0xf4, 0x48
But the framework i use accepts not the writeInt32BE function and throws exception.
How can i convert a 32 bit integer value signed to a octet Array stream without the writeInt32BE ?
A function that takes a value and returns an array of octet stream.
Using a 4 byte array buffer, converted to a data view and calling setInt32 on the view seems to work. This approach supports specification of both little endian and big endian (the default) formats independent of machine architecture.
function bigEnd32( value) {
var buf = new ArrayBuffer(4);
var view = new DataView(buf);
view.setInt32( 0, value);
return view;
}
// quick test (in a browser)
var n = prompt("Signed 32: ");
var view = bigEnd32( +n);
for(var i = 0 ; i < 4; ++i)
console.log(view.getUint8( i));
Documentation was located searching for "MDN ArrayBuffer" "MDN Dataview" etc. Check out DataView in detail for properties that access the underlying array buffer - you may be able to tweak the code to suite your application.
I want to convert a float value like 26,4 into a byte array. Something like: [51,51,-45,65]. Another example: 32,2 > [-51, -52, 0, 66].
I can do it in Arduino C++ and Python, but I don't see how to do it in JavaScript.
I have an input field in my HTML page where I want to write my float value and decode it before sending it to a Python script.
Is it possible?
Typed Arrays
First create a Float32Array which is a view on top of an actual byte-buffer (ArrayBuffer). We can allocate the buffer and view in a single operation:
var farr = new Float32Array(2); // two indexes each 4 bytes
Store the floating point numbers. These will be stored in IEEE-754 format:
farr[0] = 26.4;
farr[1] = 32.2;
Now we can add another view for the underlying buffer, this time signed 8-bit view:
var barr = new Int8Array(farr.buffer); // use the buffer of Float32Array view
console.log(barr);
// -> 51,51,-45,65,-51,-52,0,66
Live demo below
var farr = new Float32Array(2); // two indexes each 4 bytes
farr[0] = 26.4;
farr[1] = 32.2;
var barr = new Int8Array(farr.buffer); // use the buffer of Float32Array view
console.log(barr);
document.querySelector("div").innerHTML = barr[0]+","+barr[1]+","+barr[2]+","+barr[3]+","+barr[4]+","+barr[5]+","+barr[6]+","+barr[7];
<div></div>
I have to read a binary file which is said to be encoded 4 byte single format and never having to work with binary data, I don't know what this means.
I can do this reading a file with binary data in JavaScript:
d = new FileReader();
d.onload = function (e) {
var i, len;
// grab a "chunk"
response_buffer = e.target.result.slice(0, 1024);
view = new DataView(response_buffer);
for (i = 0, len = response_buffer.byteLength; i < len; i += 1) {
// hmhm
console.log(view.getUint8(i));
}
}
d.readAsArrayBuffer(some_file);
Which runs a loop from 0 to 1023 and I am getting numbers on the console, but I don't know if this is my decoded data :-)
Question:
What is 4 byte single format and how do I access the data correctly? What is the difference between say getUint8() and getint8() or getInt32() in "human understandable language"?
Thanks!
4 byte single format is not a commonly understood term in computer science.
If you could expect your file to be a series of single precision floating point numbers, then I might guess that "4 byte single format" means single precision floating point because each of those is four bytes long.
You will want to use getFloat32() to parse single precision floating point numbers from the binary stream.
If you want 1024 numbers parsed with getFloat32(), then you need 1024*4 bytes and you need to advance your for loop by four bytes each time since getFloat32() processes four bytes at a time:
d = new FileReader();
d.onload = function (e) {
var i, len;
// grab a "chunk"
response_buffer = e.target.result.slice(0, 1024 * 4);
view = new DataView(response_buffer);
for (i = 0, len = response_buffer.byteLength; i < len; i += 4) {
// hmhm
console.log(view.getFloat32(i));
}
}
d.readAsArrayBuffer(some_file);
Also, please note that IE10 and IOS 5 do not have the .slice() method for an ArrayBuffer if you're planning on using this in a general web page.