Javascript Convert int value to octet stream Array - javascript

I want convert an integer (signed) to 32 bit (big endian) into a octet stream and give the octet stream as a array value to the constructor of a
Buffer Object.
I can create it in the console for example for the value -2000:
<code>
buf = Buffer(4)
buf.writeInt32BE(-2000)
buf // is <Buffer ff ff f8 30>
buf1 = new Buffer([0xff, 0xff, 0xf8, 0x30])
</code>
The value -3000 is for example -3000 : 0xff ,0xff, 0xf4, 0x48
But the framework i use accepts not the writeInt32BE function and throws exception.
How can i convert a 32 bit integer value signed to a octet Array stream without the writeInt32BE ?
A function that takes a value and returns an array of octet stream.

Using a 4 byte array buffer, converted to a data view and calling setInt32 on the view seems to work. This approach supports specification of both little endian and big endian (the default) formats independent of machine architecture.
function bigEnd32( value) {
var buf = new ArrayBuffer(4);
var view = new DataView(buf);
view.setInt32( 0, value);
return view;
}
// quick test (in a browser)
var n = prompt("Signed 32: ");
var view = bigEnd32( +n);
for(var i = 0 ; i < 4; ++i)
console.log(view.getUint8( i));
Documentation was located searching for "MDN ArrayBuffer" "MDN Dataview" etc. Check out DataView in detail for properties that access the underlying array buffer - you may be able to tweak the code to suite your application.

Related

How to use javascript (in Angular) to get bytes encoded by java.util.Base64? [duplicate]

I need to convert a base64 encode string into an ArrayBuffer.
The base64 strings are user input, they will be copy and pasted from an email, so they're not there when the page is loaded.
I would like to do this in javascript without making an ajax call to the server if possible.
I found those links interesting, but they didt'n help me:
ArrayBuffer to base64 encoded string
this is about the opposite conversion, from ArrayBuffer to base64, not the other way round
http://jsperf.com/json-vs-base64/2
this looks good but i can't figure out how to use the code.
Is there an easy (maybe native) way to do the conversion? thanks
Try this:
function _base64ToArrayBuffer(base64) {
var binary_string = window.atob(base64);
var len = binary_string.length;
var bytes = new Uint8Array(len);
for (var i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
return bytes.buffer;
}
Using TypedArray.from:
Uint8Array.from(atob(base64_string), c => c.charCodeAt(0))
Performance to be compared with the for loop version of Goran.it answer.
For Node.js users:
const myBuffer = Buffer.from(someBase64String, 'base64');
myBuffer will be of type Buffer which is a subclass of Uint8Array. Unfortunately, Uint8Array is NOT an ArrayBuffer as the OP was asking for. But when manipulating an ArrayBuffer I almost always wrap it with Uint8Array or something similar, so it should be close to what's being asked for.
Goran.it's answer does not work because of unicode problem in javascript - https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding.
I ended up using the function given on Daniel Guerrero's blog: http://blog.danguer.com/2011/10/24/base64-binary-decoding-in-javascript/
Function is listed on github link: https://github.com/danguer/blog-examples/blob/master/js/base64-binary.js
Use these lines
var uintArray = Base64Binary.decode(base64_string);
var byteArray = Base64Binary.decodeArrayBuffer(base64_string);
Async solution, it's better when the data is big:
// base64 to buffer
function base64ToBufferAsync(base64) {
var dataUrl = "data:application/octet-binary;base64," + base64;
fetch(dataUrl)
.then(res => res.arrayBuffer())
.then(buffer => {
console.log("base64 to buffer: " + new Uint8Array(buffer));
})
}
// buffer to base64
function bufferToBase64Async( buffer ) {
var blob = new Blob([buffer], {type:'application/octet-binary'});
console.log("buffer to blob:" + blob)
var fileReader = new FileReader();
fileReader.onload = function() {
var dataUrl = fileReader.result;
console.log("blob to dataUrl: " + dataUrl);
var base64 = dataUrl.substr(dataUrl.indexOf(',')+1)
console.log("dataUrl to base64: " + base64);
};
fileReader.readAsDataURL(blob);
}
Javascript is a fine development environment so it seems odd than it doesn't provide a solution to this small problem. The solutions offered elsewhere on this page are potentially slow. Here is my solution. It employs the inbuilt functionality that decodes base64 image and sound data urls.
var req = new XMLHttpRequest;
req.open('GET', "data:application/octet;base64," + base64Data);
req.responseType = 'arraybuffer';
req.onload = function fileLoaded(e)
{
var byteArray = new Uint8Array(e.target.response);
// var shortArray = new Int16Array(e.target.response);
// var unsignedShortArray = new Int16Array(e.target.response);
// etc.
}
req.send();
The send request fails if the base 64 string is badly formed.
The mime type (application/octet) is probably unnecessary.
Tested in chrome. Should work in other browsers.
Pure JS - no string middlestep (no atob)
I write following function which convert base64 in direct way (without conversion to string at the middlestep). IDEA
get 4 base64 characters chunk
find index of each character in base64 alphabet
convert index to 6-bit number (binary string)
join four 6 bit numbers which gives 24-bit numer (stored as binary string)
split 24-bit string to three 8-bit and covert each to number and store them in output array
corner case: if input base64 string ends with one/two = char, remove one/two numbers from output array
Below solution allows to process large input base64 strings. Similar function for convert bytes to base64 without btoa is HERE
function base64ToBytesArr(str) {
const abc = [..."ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"]; // base64 alphabet
let result = [];
for(let i=0; i<str.length/4; i++) {
let chunk = [...str.slice(4*i,4*i+4)]
let bin = chunk.map(x=> abc.indexOf(x).toString(2).padStart(6,0)).join('');
let bytes = bin.match(/.{1,8}/g).map(x=> +('0b'+x));
result.push(...bytes.slice(0,3 - (str[4*i+2]=="=") - (str[4*i+3]=="=")));
}
return result;
}
// --------
// TEST
// --------
let test = "Alice's Adventure in Wonderland.";
console.log('test string:', test.length, test);
let b64_btoa = btoa(test);
console.log('encoded string:', b64_btoa);
let decodedBytes = base64ToBytesArr(b64_btoa); // decode base64 to array of bytes
console.log('decoded bytes:', JSON.stringify(decodedBytes));
let decodedTest = decodedBytes.map(b => String.fromCharCode(b) ).join``;
console.log('Uint8Array', JSON.stringify(new Uint8Array(decodedBytes)));
console.log('decoded string:', decodedTest.length, decodedTest);
Caution!
If you want to decode base64 to STRING (not bytes array) and you know that result contains utf8 characters then atob will fail in general e.g. for character 💩 the atob("8J+SqQ==") will give wrong result . In this case you can use above solution and convert result bytes array to string in proper way e.g. :
function base64ToBytesArr(str) {
const abc = [..."ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"]; // base64 alphabet
let result = [];
for(let i=0; i<str.length/4; i++) {
let chunk = [...str.slice(4*i,4*i+4)]
let bin = chunk.map(x=> abc.indexOf(x).toString(2).padStart(6,0)).join('');
let bytes = bin.match(/.{1,8}/g).map(x=> +('0b'+x));
result.push(...bytes.slice(0,3 - (str[4*i+2]=="=") - (str[4*i+3]=="=")));
}
return result;
}
// --------
// TEST
// --------
let testB64 = "8J+SqQ=="; // for string: "💩";
console.log('input base64 :', testB64);
let decodedBytes = base64ToBytesArr(testB64); // decode base64 to array of bytes
console.log('decoded bytes :', JSON.stringify(decodedBytes));
let result = new TextDecoder("utf-8").decode(new Uint8Array(decodedBytes));
console.log('properly decoded string :', result);
let result_atob = atob(testB64);
console.log('decoded by atob :', result_atob);
Snippets tested 2022-08-04 on: chrome 103.0.5060.134 (arm64), safari 15.2, firefox 103.0.1 (64 bit), edge 103.0.1264.77 (arm64), and node-js v12.16.1
I would strongly suggest using an npm package implementing correctly the base64 specification.
The best one I know is rfc4648
The problem is that btoa and atob use binary strings instead of Uint8Array and trying to convert to and from it is cumbersome. Also there is a lot of bad packages in npm for that. I lose a lot of time before finding that one.
The creators of that specific package did a simple thing: they took the specification of Base64 (which is here by the way) and implemented it correctly from the beginning to the end. (Including other formats in the specification that are also useful like Base64-url, Base32, etc ...) That doesn't seem a lot but apparently that was too much to ask to the bunch of other libraries.
So yeah, I know I'm doing a bit of proselytism but if you want to avoid losing your time too just use rfc4648.
I used the accepted answer to this question to create base64Url string <-> arrayBuffer conversions in the realm of base64Url data transmitted via ASCII-cookie [atob, btoa are base64[with +/]<->js binary string], so I decided to post the code.
Many of us may want both conversions and client-server communication may use the base64Url version (though a cookie may contain +/ as well as -_ characters if I understand well, only ",;\ characters and some wicked characters from the 128 ASCII are disallowed). But a url cannot contain / character, hence the wider use of b64 url version which of course not what atob-btoa supports...
Seeing other comments, I would like to stress that my use case here is base64Url data transmission via url/cookie and trying to use this crypto data with the js crypto api (2017) hence the need for ArrayBuffer representation and b64u <-> arrBuff conversions... if array buffers represent other than base64 (part of ascii) this conversion wont work since atob, btoa is limited to ascii(128). Check out an appropriate converter like below:
The buff -> b64u version is from a tweet from Mathias Bynens, thanks for that one (too)! He also wrote a base64 encoder/decoder:
https://github.com/mathiasbynens/base64
Coming from java, it may help when trying to understand the code that java byte[] is practically js Int8Array (signed int) but we use here the unsigned version Uint8Array since js conversions work with them. They are both 256bit, so we call it byte[] in js now...
The code is from a module class, that is why static.
//utility
/**
* Array buffer to base64Url string
* - arrBuff->byte[]->biStr->b64->b64u
* #param arrayBuffer
* #returns {string}
* #private
*/
static _arrayBufferToBase64Url(arrayBuffer) {
console.log('base64Url from array buffer:', arrayBuffer);
let base64Url = window.btoa(String.fromCodePoint(...new Uint8Array(arrayBuffer)));
base64Url = base64Url.replaceAll('+', '-');
base64Url = base64Url.replaceAll('/', '_');
console.log('base64Url:', base64Url);
return base64Url;
}
/**
* Base64Url string to array buffer
* - b64u->b64->biStr->byte[]->arrBuff
* #param base64Url
* #returns {ArrayBufferLike}
* #private
*/
static _base64UrlToArrayBuffer(base64Url) {
console.log('array buffer from base64Url:', base64Url);
let base64 = base64Url.replaceAll('-', '+');
base64 = base64.replaceAll('_', '/');
const binaryString = window.atob(base64);
const length = binaryString.length;
const bytes = new Uint8Array(length);
for (let i = 0; i < length; i++) {
bytes[i] = binaryString.charCodeAt(i);
}
console.log('array buffer:', bytes.buffer);
return bytes.buffer;
}
made a ArrayBuffer from a base64:
function base64ToArrayBuffer(base64) {
var binary_string = window.atob(base64);
var len = binary_string.length;
var bytes = new Uint8Array(len);
for (var i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
return bytes.buffer;
}
I was trying to use above code and It's working fine.
The result of atob is a string that is separated with some comma
,
A simpler way is to convert this string to a json array string and after that parse it to a byteArray
below code can simply be used to convert base64 to an array of number
let byteArray = JSON.parse('['+atob(base64)+']');
let buffer = new Uint8Array(byteArray);
Solution without atob
I've seen many people complaining about using atob and btoa in the replies. There are some issues to take into account when using them.
There's a solution without using them in the MDN page about Base64. Below you can find the code to convert a base64 string into a Uint8Array copied from the docs.
Note that the function below returns a Uint8Array. To get the ArrayBuffer version you just need to do uintArray.buffer.
function b64ToUint6(nChr) {
return nChr > 64 && nChr < 91
? nChr - 65
: nChr > 96 && nChr < 123
? nChr - 71
: nChr > 47 && nChr < 58
? nChr + 4
: nChr === 43
? 62
: nChr === 47
? 63
: 0;
}
function base64DecToArr(sBase64, nBlocksSize) {
const sB64Enc = sBase64.replace(/[^A-Za-z0-9+/]/g, "");
const nInLen = sB64Enc.length;
const nOutLen = nBlocksSize
? Math.ceil(((nInLen * 3 + 1) >> 2) / nBlocksSize) * nBlocksSize
: (nInLen * 3 + 1) >> 2;
const taBytes = new Uint8Array(nOutLen);
let nMod3;
let nMod4;
let nUint24 = 0;
let nOutIdx = 0;
for (let nInIdx = 0; nInIdx < nInLen; nInIdx++) {
nMod4 = nInIdx & 3;
nUint24 |= b64ToUint6(sB64Enc.charCodeAt(nInIdx)) << (6 * (3 - nMod4));
if (nMod4 === 3 || nInLen - nInIdx === 1) {
nMod3 = 0;
while (nMod3 < 3 && nOutIdx < nOutLen) {
taBytes[nOutIdx] = (nUint24 >>> ((16 >>> nMod3) & 24)) & 255;
nMod3++;
nOutIdx++;
}
nUint24 = 0;
}
}
return taBytes;
}
If you're interested in the reverse operation, ArrayBuffer to base64, you can find how to do it in the same link.

javascript: smallest JSON.stringify for Float32Array?

FireFox 46.0.1: I am using 3rd-party (easyrtc) software to send 15KB chunks of Float32Arrays between peers. Easyrtc insists that the data be JSON-able. Unfortunately, JSON.stringify yields a string more than twice as long as the original data: 16384 bytes of data becomes a string of length 35755. Below is my test code followed by the console output. What if anything can I do to reduce the stringify'd size? Is there a way to send the values only (no keys)? Can I use the 'replacer' argument to send only the values, and if so, don't I need to use a replacer on the corresponding JSON.parse on the receiving end?
var g_testBufferNBytes = 4096 * 4;
var g_testBuffer = new ArrayBuffer(g_testBufferNBytes);
var g_testBufferView = new Float32Array(g_testBuffer);
console.log("array byte length " + g_testBuffer.byteLength);
console.log("view byte length " + g_testBufferView.byteLength);
var j = JSON.stringify(g_testBufferView);
console.log("j length " + j.length);
var newBuf = JSON.parse(j);
console.log("newBuf length " + Object.keys(newBuf).length);
CONSOLE:
array byte length 16384
view byte length 16384
j length 35755
newBuf length 4096
Yes
ES6: Assume that your data are in let f32 = g_testBufferView (array Float32Array) ) - whe can save it as JSON array in at leas 4 ways:
// code
let f32json = JSON.stringify(f32);
let f32jsonArr = JSON.stringify(Array.from(f32));
let f32base64 = btoa(String.fromCharCode(...(new Uint8Array(f32.buffer))));
let f32base128 = ... // not trivial, look below
// decode
let df32json = new Float32Array(Object.values(JSON.parse(f32json)));
let df32jsonArr = new Float32Array(JSON.parse(f32jsonArr));
let df32base64 = new Float32Array(new Uint8Array([...atob(f32base64)].map(c => c.charCodeAt(0))).buffer);
let df32base128 = ... // not trivial, look below
Note that Object.values return values sorted by numeric keys (look here).
Here is working example. You can also use base128 do decode but I not use in this example (to not complicate it) - more details here.
If your Float32Array- f32 has 4096 elements equals to 0.3 then:
f32 has 16384 bytes,
f32json (j from your question) has 109483 bytes (which is >6x bigger than f32)
f32jsonArr has 81921 bytes (which is >5x bigger than f32)
f32base64 has 21848 bytes(which is ~1.3x bigger than f32)
f32base128 has 18725 bytes (whis is <1.15x bigger than f32) but chrome will send ~2x bigger request (depends on input data)
If your Float32Array- f32 has 4096 elements equals integer from 1 to 9 then:
f32 has 16384 bytes - CONST,
f32json (j from your question) has 35755 bytes (which is >2x bigger than f32)
f32jsonArr has 8193 bytes (which is 2x SMALLER (sic!) than f32)
f32base64 has 21848 bytes - CONST (which is ~1.3x bigger than f32)
f32base128 has 18725 bytes - CONST (whis is <1.15x bigger than f32) but chrome will send ~2x bigger request (depends on input data)
Conclusion
The smallest result which not depends of array values (result size is constant) we get for f32base64 ~33% bigger than input array size. For f32base128 - it contains valid JSON (string) which is something about <15% bigger than input, but chrome during sending increase this size (look here - on 'update' section). So use f32base64 - this is probably the smallest JSON that you can get without more sophisticated methods.

JavaScript: reading 3 bytes Buffer as an integer

Let's say I have a hex data stream, which I want to divide into 3-bytes blocks which I need to read as an integer.
For example: given a hex string 01be638119704d4b9a I need to read the first three bytes 01be63 and read it as integer 114275. This is what I got:
var sample = '01be638119704d4b9a';
var buffer = new Buffer(sample, 'hex');
var bufferChunk = buffer.slice(0, 3);
var decimal = bufferChunk.readUInt32BE(0);
The readUInt32BE works perfectly for 4-bytes data, but here I obviously get:
RangeError: index out of range
at checkOffset (buffer.js:494:11)
at Buffer.readUInt32BE (buffer.js:568:5)
How do I read 3-bytes as integer correctly?
If you are using node.js v0.12+ or io.js, there is buffer.readUIntBE() which allows a variable number of bytes:
var decimal = buffer.readUIntBE(0, 3);
(Note that it's readUIntBE for Big Endian and readUIntLE for Little Endian).
Otherwise if you're on an older version of node, you will have to do it manually (check bounds first of course):
var decimal = (buffer[0] << 16) + (buffer[1] << 8) + buffer[2];
I'm using this, if someone knows something wrong with it, please advise;
const integer = parseInt(buffer.toString("hex"), 16)
you should convert three byte to four byte.
function three(var sample){
var buffer = new Buffer(sample, 'hex');
var buf = new Buffer(1);
buf[0] = 0x0;
return Buffer.concat([buf, buffer.slice(0, 3)]).readUInt32BE();
}
You can try this function.

How to convert float into byte array?

I want to convert a float value like 26,4 into a byte array. Something like: [51,51,-45,65]. Another example: 32,2 > [-51, -52, 0, 66].
I can do it in Arduino C++ and Python, but I don't see how to do it in JavaScript.
I have an input field in my HTML page where I want to write my float value and decode it before sending it to a Python script.
Is it possible?
Typed Arrays
First create a Float32Array which is a view on top of an actual byte-buffer (ArrayBuffer). We can allocate the buffer and view in a single operation:
var farr = new Float32Array(2); // two indexes each 4 bytes
Store the floating point numbers. These will be stored in IEEE-754 format:
farr[0] = 26.4;
farr[1] = 32.2;
Now we can add another view for the underlying buffer, this time signed 8-bit view:
var barr = new Int8Array(farr.buffer); // use the buffer of Float32Array view
console.log(barr);
// -> 51,51,-45,65,-51,-52,0,66
Live demo below
var farr = new Float32Array(2); // two indexes each 4 bytes
farr[0] = 26.4;
farr[1] = 32.2;
var barr = new Int8Array(farr.buffer); // use the buffer of Float32Array view
console.log(barr);
document.querySelector("div").innerHTML = barr[0]+","+barr[1]+","+barr[2]+","+barr[3]+","+barr[4]+","+barr[5]+","+barr[6]+","+barr[7];
<div></div>

What does it mean if binary data is "4 byte single format" and how do I read it in JavaScript?

I have to read a binary file which is said to be encoded 4 byte single format and never having to work with binary data, I don't know what this means.
I can do this reading a file with binary data in JavaScript:
d = new FileReader();
d.onload = function (e) {
var i, len;
// grab a "chunk"
response_buffer = e.target.result.slice(0, 1024);
view = new DataView(response_buffer);
for (i = 0, len = response_buffer.byteLength; i < len; i += 1) {
// hmhm
console.log(view.getUint8(i));
}
}
d.readAsArrayBuffer(some_file);
Which runs a loop from 0 to 1023 and I am getting numbers on the console, but I don't know if this is my decoded data :-)
Question:
What is 4 byte single format and how do I access the data correctly? What is the difference between say getUint8() and getint8() or getInt32() in "human understandable language"?
Thanks!
4 byte single format is not a commonly understood term in computer science.
If you could expect your file to be a series of single precision floating point numbers, then I might guess that "4 byte single format" means single precision floating point because each of those is four bytes long.
You will want to use getFloat32() to parse single precision floating point numbers from the binary stream.
If you want 1024 numbers parsed with getFloat32(), then you need 1024*4 bytes and you need to advance your for loop by four bytes each time since getFloat32() processes four bytes at a time:
d = new FileReader();
d.onload = function (e) {
var i, len;
// grab a "chunk"
response_buffer = e.target.result.slice(0, 1024 * 4);
view = new DataView(response_buffer);
for (i = 0, len = response_buffer.byteLength; i < len; i += 4) {
// hmhm
console.log(view.getFloat32(i));
}
}
d.readAsArrayBuffer(some_file);
Also, please note that IE10 and IOS 5 do not have the .slice() method for an ArrayBuffer if you're planning on using this in a general web page.

Categories