Hex String to INT32 - Little Endian (DCBA Format) Javascript - javascript

Implementing something based on a pathetic documentation without no info nothing.
The example is just this
(7F02AAF7)H => (F7AA027F)H = -139853185
Let's say even if I convert 7F02AAF7 to F7AA027F, then still the output via
'parseInt('F7AA027F', 16)' is different from what I am expecting.
I did some google search and found this website http://www.scadacore.com/field-tools/programming-calculators/online-hex-converter/
Here when you input 7F02AAF7 then it is processed to wanted number under INT32 - Little Endian (DCBA) system. I tried this search term but no luck.
Can you please tell me what exactly am I supposed to do here and is there any node.js library which can help me with this.

You could adapt the excellent answer of T.J. Crowder and use DataView#setUint8 for the given bytes with DataView#getInt32 and an indicator for littleEndian.
var data = '7F02AAF7'.match(/../g);
// Create a buffer
var buf = new ArrayBuffer(4);
// Create a data view of it
var view = new DataView(buf);
// set bytes
data.forEach(function (b, i) {
view.setUint8(i, parseInt(b, 16));
});
// get an int32 with little endian
var num = view.getInt32(0, 1);
console.log(num);

Node can do that pretty easily using buffers:
Buffer.from('7F02AAF7', 'hex').readInt32LE()

JavaScript integers are usually stored as a Number value:
4.3.19
primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value
So the result of parseInt is a float where the value losslessly fits into the fraction part of the float (52 bits capacity). parseInt also doesn't parse it as two-complement notation.
If you want to force anything that you read into 32 bit, then the easiest would be two force it to be automatically converted to 32 bit by applying a binary operation. I would suggest:
parseInt("F7AA027F", 16) | 0
The binary OR (|) with 0 is essentially a no-op, but it converts any integer to 32 bit. This trick is often used in order to convert numbers to 32 bit in order to make calculations on it faster.
Also, this is portable.

In my case, I am trying to send accelerometer data from Arduino to Pi.
The raw data I am reading is that [0x0, 0x0, 0x10, 0xBA].
If you lack knowledge about the topic as me, use the scadacore.com website to find out find your data should correspond to. In my case it is Float - Little Endian (DCBA) which outputs: -0.03126526. Now we know what kind of a conversion that we need.
Then, check available functions based on the language. In my case, Node.js buffer library offers buf.readFloatLE([offset]) function which is the one I need.

Related

How to calculate logarithm with base N for big integer?

I'm using this BigInteger.js for some calculations:
let myBigInt = bigInt(20).pow(200) // gets 160693804425899027554196209234116260252220299378279283530137600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
I'd like to apply the logarithm to the big integer but in the docs I could not find any matching function. How can I implement a log(baseN, valueX) function for the BigInteger.js library?
Note: let myLogarithm = myBigInt.log(baseN) is not a valid implementation.
Note: After a lot of try&error I did found a working solution my own and I will post it here because I'm pretty sure there are a few more people then me that also gots faced with the same issue right there. So I hope, I could help :)
Have a look at wikipedia, as I did because theres a very nice article about baseConversion.
Below you can find a function for Math.log(base, value) that is able to calculate the log(base) from a value.
Math.log = (function() {
var log = Math.log;
return function(base, n) {
return log(n)/(base ? log(base) : 1);
};
})();
To calculate the logarithmToBaseN for bigInt-values just use this line of code:
let logarithmToBaseN = (myBigInt.toString().length * Math.log(baseN, 10) + Math.log(baseN, parseFloat("0." + myBigInt))) - 1);
Edit: This soltuion is a tiny workaround bacause parseFloat("0." + myBigInt) converts a big value like 100000 to a really small one like 0.100000,... what causes that it will be in integer precision.
According to #Jonas W's comment: The solution is very accurate for lower bases like (5, 10, ...) combined with low values like 10, 1000, 100000 - but for really big values like bigInt(20).pow(200) is it not.
Note: Using parseFloat (IEEE 754 double precision floating-point) means, you have a maximum of 52 bits of precision, which is a bit more than 15 decimal places. After that - the accuracy will be killed.
Note: For really big values bigInt(20).pow(200) combined with really big Bases like 100*(and more) it seems to be pretty accurate again.
Greetings, jonas.

Portable hashCode implementation for binary data

I am looking for a portable algorithm for creating a hashCode for binary data. None of the binary data is very long -- I am Avro-encoding keys for use in kafka.KeyedMessages -- we're probably talking anywhere from 2 to 100 bytes in length, but most of the keys are in the 4 to 8 byte range.
So far, my best solution is to convert the data to a hex string, and then do a hashCode of that. I'm able to make that work in both Scala and JavaScript. Assuming I have defined b: Array[Byte], the Scala looks like this:
b.map("%02X" format _).mkString.hashCode
It's a little more elaborate in JavaScript -- luckily someone already ported the basic hashCode algorithm to JavaScript -- but the point is being able to create a Hex string to represent the binary data, I can ensure the hashing algorithm works off the same inputs.
On the other hand, I have to create an object twice the size of the original just to create the hashCode. Luckily most of my data is tiny, but still -- there has to be a better way to do this.
Instead of padding the data as its hex value, I presume you could just coerce the binary data into a String so the String has the same number of bytes as the binary data. It would be all garbled, more control characters than printable characters, but it would be a string nonetheless. Do you run into portability issues though? Endian-ness, Unicode, etc.
Incidentally, if you got this far reading and don't already know this -- you can't just do:
val b: Array[Byte] = ...
b.hashCode
Luckily I already knew that before I started, because I ran into that one early on.
Update
Based on the first answer given, it appears at first blush that java.util.Arrays.hashCode(Array[Byte]) would do the trick. However, if you follow the javadoc trail, you'll see that this is the algorithm behind it, which is as based on the algorithm for List and the algorithm for byte combined.
int hashCode = 1;
for (byte e : list) hashCode = 31*hashCode + (e==null ? 0 : e.intValue());
As you can see, all it's doing is creating a Long representing the value. At a certain point, the number gets too big and it wraps around. This is not very portable. I can get it to work for JavaScript, but you have to import the npm module long. If you do, it looks like this:
function bufferHashCode(buffer) {
const Long = require('long');
var hashCode = new Long(1);
for (var value of buff.values()) { hashCode = hashCode.multiply(31).add(value) }
return hashCode
}
bufferHashCode(new Buffer([1,2,3]));
// hashCode = Long { low: 30817, high: 0, unsigned: false }
And you do get the same results when the data wraps around, sort of, though I'm not sure why. In Scala:
java.util.Arrays.hashCode(Array[Byte](1,2,3,4,5,6,7,8,9,10))
// res30: Int = -975991962
Note that the result is an Int. In JavaScript:
bufferHashCode(new Buffer([1,2,3,4,5,6,7,8,9,10]);
// hashCode = Long { low: -975991962, high: 197407, unsigned: false }
So I have to take the low bytes and ignore the high, but otherwise I get the same results.
This functionality is already available in Java standard library, look at the Arrays.hashCode() method.
Because your binary data are Array[Byte], here is how you can verify it works:
println(java.util.Arrays.hashCode(Array[Byte](1,2,3))) // prints 30817
println(java.util.Arrays.hashCode(Array[Byte](1,2,3))) // prints 30817
println(java.util.Arrays.hashCode(Array[Byte](2,2,3))) // prints 31778
Update: It is not true that the Java implementation boxes the bytes. Of course, there is conversion to int, but there's no way around that. This is the Java implementation:
public static int hashCode(byte a[]) {
if (a == null) return 0;
int result = 1;
for (byte element : a) result = 31 * result + element;
return result;
}
Update 2
If what you need is a JavaScript implementation that gives the same results as a Scala/Java implementation, than you can extend the algorithm by, e.g., taking only the rightmost 31 bits:
def hashCode(a: Array[Byte]): Int = {
if (a == null) {
0
} else {
var hash = 1
var i: Int = 0
while (i < a.length) {
hash = 31 * hash + a(i)
hash = hash & Int.MaxValue // taking only the rightmost 31 bits
i += 1
}
hash
}
}
and JavaScript:
var hashCode = function(arr) {
if (arr == null) return 0;
var hash = 1;
for (var i = 0; i < arr.length; i++) {
hash = hash * 31 + arr[i]
hash = hash % 0x80000000 // taking only the rightmost 31 bits in integer representation
}
return hash;
}
Why do the two implementations produce the same results? In Java, integer overflow behaves as if the addition was performed without loss of precision and then bits higher than 32 got thrown away and & Int.MaxValue throws away the 32nd bit. In JavaScript, there is no loss of precision for integers up to 253 which is a limit the expression 31 * hash + a(i) never exceeds. % 0x80000000 then behaves as taking the rightmost 31 bits. The case without overflows is obvious.
This is the meat of algorithm used in the Java library:
int result 1;
for (byte element : a) result = 31 * result + element;
You comment:
this algorithm isn't very portable
Incorrect. If we are talking about Java, then provided that we all agree on the type of the result, then the algorithm is 100% portable.
Yes the computation overflows, but it overflows exactly the same way on all valid implementations of the Java language. A Java int is specified to be 32 bits signed two's complement, and the behavior of the operators when overflow occurs is well-defined ... and the same for all implementations. (The same goes for long ... though the size is different, obviously.)
I'm not an expert, but my understanding is that Scala's numeric types have the same properties as Java. Javascript is different, being based on IEE 754 double precision floating point. However, with case you should be able to code the Java algorithm portably in Javascript. (I think #Mifeet's version is wrong ...)

About the binary format of JavaScript TypedArray float32

I need to write a piece of hardware emulator in JavaScript. It has its own floating point format, so I do lots of conversion between JS numerics and that format, which is slow. I have the idea to use JavaScript TypedArray float32 since I have direct access of bytes forming the float32 floating point value which is not so far from the desired format, so the conversion would be much faster this way (only some shifts, etc, using Uint8 view of the Float32).
However, I am not sure how portable solution it would be. Various documents about TypedArray topics states that float32 is like "the native C format on that hardware" or such. But can I expect that the exact binary format of float32 is the same on all platforms running some browser/JS? I can guess the endiannes can be a problem, but I can deal with that if at least there are no other differences. As far as I can tell, the format used seems to be IEE754, but there can be others used to implement float32 in JS on some (well, at least not so exotic ...) platforms?
I could test at least x86 and Apple A7 CPUs, and it seems they are the very same, which is good, but also odd, as I thought the byte order of these CPUs are different (maybe not the floating format at least?). However it's far from being a global truth just checking two platforms/OSes/browsers/whatever ...
The Float32Array will internally represent the values bases on the endianess of the host system, typically little-endian.
And yes, the format is IEEE 754 (this has been around since before FPUs came along, and the variations of it deals with more the width, ie. 64-bit, 80-bit and so on). All numbers (Number) in JavaScript is internally represented as 64-bit IEEE 754. For typed arrays both 32-bit and 64-bit IEEE 754 is of course available.
PowerPC and 68k CPUs uses big-endian (the way it ought to be! :) ). The so-called network order is also big-endian, and many platform-independent file formats are stored in big-endian byte order (in particular with audio and graphics). Most mainstream computers uses little-endian CPUs such as the x86. So in cases with a combination of these you very likely have to deal with different byte orders.
To deal with endianess you can instead of using a Float32Array use a DataView.
For example:
var buffer = new ArrayBuffer(10240); // some raw byte buffer
var view = new DataView(buffer); // flexible view supporting endianness
Now you can now read and write to any position in the buffer with endianess in mind (DataView also allow reading/writing from/to non-aligned positions, ie. you can write a Float32 value to position 3 if you need to. You cannot do this with Float32/Uint32/Int16 etc.).
The browser will internally convert to the correct order - you just provide the value as-is:
view.setFloat32(pos, 0.5); // big-endian
view.setFloat32(pos, 0.5, false) // big-endian
view.setFloat32(pos, 0.5, true); // little-endian
And likewise when reading:
var n = view.getFloat32(pos); // big-endian
var n = view.getFloat32(pos, false) // big-endian
var n = view.getFloat32(pos, true); // little-endian
Tip: You can use a native Float32Array internally and later read/write to it using endianess. This tend to be speedier but it requires a conversion using DataView at the end if the resulting buffer's endianess is different from the host system:
var f32 = new Float32Array(buffer); // or use a size
f32[0] = 0.5;
Then to make sure you have big-endian representation:
var view = new DataView(f32.buffer);
var msbVal = view.getFloat32(0): // returns 32-bit repres. in big-endian
Hope that gave some inputs! Just throw me questions about it if you want me to elaborate on some part.

Interpreting Web Audio API FFT results

The Web Audio API has an analyser node which allows you to get FFT data on the audio you're working with and has byte and float ways of getting the data. The byte version makes a bit of sense, returning what looks like a normalized (depending on min and max decibel values) intensity spectrum with 0 being no component of the audio at a specific frequency bin and 255 being the max.
But I'd like a bit more detail than 8 bit, using the float version however, gives weird results.
freqData = new Float32Array(analyser.frequencyBinCount);
analyser.getFloatFrequencyData(freqData);
This gives me values between -891.048828125 and 0. -891 shows up corresponding to silence, so it's somehow the minimum value while I'm guessing 0 is equivalent to the max value.
What's going on? Why is -891.048828125 significant at all? Why a large negative being silence and zero being maximum? Am I using the wrong FloatArray or is there misconfiguration? Float64 gives 0 values.
Since there seems to be zero documentation on what the data actually represents, I looked into the relevant source code of webkit: RealtimeAnalyser.cpp
Short answer: subtract analyser.minDecibels from every value of the Float32Array to get positive numbers and multiply with (analyser.maxDecibels - analyser.minDecibels) to get a similar representation as with getByteFrequencyData, just with more resolution.
Long answer:
Both getByteFrequencyData and getFloatFrequencyData give you the magnitude in decibels. It's just scaled differently and for getByteFrequencyData a minDecibels constant is subtracted:
Relevant code in webkit for getByteFrequencyData:
const double rangeScaleFactor = m_maxDecibels == m_minDecibels ? 1 : 1 / (m_maxDecibels - m_minDecibels);
float linearValue = source[i];
double dbMag = !linearValue ? minDecibels : AudioUtilities::linearToDecibels(linearValue);
// The range m_minDecibels to m_maxDecibels will be scaled to byte values from 0 to UCHAR_MAX.
double scaledValue = UCHAR_MAX * (dbMag - minDecibels) * rangeScaleFactor;
Relevant code in webkit for getFloatFrequencyData:
float linearValue = source[i];
double dbMag = !linearValue ? minDecibels : AudioUtilities::linearToDecibels(linearValue);
destination[i] = float(dbMag);
So, to get positive values, you can simply subtract minDecibels yourself, which is exposed in the analyzer node:
//The minimum power value in the scaling range for the FFT analysis data for conversion to unsigned byte values.
attribute double minDecibels;
Another detail is that by default, the analyser node does time smoothing, which can be disabled by setting smoothingTimeConstant to zero.
The default values in webkit are:
const double RealtimeAnalyser::DefaultSmoothingTimeConstant = 0.8;
const double RealtimeAnalyser::DefaultMinDecibels = -100;
const double RealtimeAnalyser::DefaultMaxDecibels = -30;
Sadly, even though the analyser node computes a complex fft, it doesn't give access to the complex representations, just the magnitudes of it.
Correct on both points in the previous answer and comments - the numbers are in decibels, so 0 is max and -infinity is min (absolute silence). -891.0... is, I believe, just a floating point conversion oddity.
You are correct in using a Float32Array. I found an interesting tutorial on using the Audio Data API, which while it is different than the Web Audio API, gave me some useful insight to me about what you are trying to do here. I had a quick peek to see about why the numbers are negative, and didn't notice anything obvious, but I wondered if these numbers might be in decibels, dB, which commonly is given in negative numbers, and zero is the peak. The only problem with that theory is that -891 seems to be a really small number for dB.

Can you get powers of 10 faster than O(log n)?

I know that exponentiation is O(log n) or worse for most cases, but I'm getting lost trying to understand of how numbers are represented themselves. Take JavaScript, for example, because it has several native number formats:
100000 === 1E5 && 100000 === 0303240
>>> true
Internally, don't they all end up being stored and manipulated as binary values stored in memory? If so, is the machine able to store the decimal and scientific-notation representations as fast as it does the octal?
And thus, would you expect +("1E" + n) to be faster than Math.pow(10, n)?
Mostly this question is about how 1E(n) works, but in trying to think about the answer myself I became more curious about how the number is parsed and stored in the first place. I would appreciate any explanation you can offer.
I don't think string manipulation could be faster, because at least concatenation creates a new object (memory allocation, more job for GC), Math.pow usually comes to single machine instruction.
Moreover, some modern JS VMs do hotspot optimisation, producing machine code from javascript. There is chance of it for Math.pow, but nearly impossible for the string magic.
If you 100% sure that Math.pow works slow in your application (I just cannot believe in it), you could use array lookup, it should work fastest possible: [1,10,100,1000,10000,...][n]. Array will be relatively small and complexity is O(1).
but I'm getting lost trying to understand of how numbers are represented themselves. Take JavaScript, for example, because it has several native number formats:
Internally, don't they all end up being stored and manipulated as binary values stored in memory?
Yep In javascript, there is only one number type a 64bit float type therefore
1 === 1.0
http://www.hunlock.com/blogs/The_Complete_Javascript_Number_Reference
If so, is the machine able to store the decimal and scientific-notation representations as fast as it does the octal?
Yes again because there is only one type. (Maybe there is a minute difference but it should be negligible)
However, for this specific case there is a limit on the numbers that can be represented ~ 1e300, therefore the runtime is O(~300) = O(1) all other numbers are represented as +/- Infinity.
And thus, would you expect +("1E" + n) to be faster than Math.pow(10, n)?
Not quite! 1E100 is faster than Math.pow(10,n)
However +("1E"+n) is slower than Math.pow(10,n);
Not because of string and memory allocation, but because the JS interpreter has to parse the string and convert it into a number, and that is slower than the native Math.pow(num,num) operation.
jsperf test
I ran a jsperf on the options.
var sum = 0;
for (var i = 1; i < 20; ++i){
sum += +("1E" + i);
}
is slow because of string concatenation.
var sum = 0;
for (var i = 0; i < 20; ++i){
Math.pow(10, i);
}
is therefore faster, since it operates on numbers only.
var sum = 0;
sum += 1e0;
sum += 1e1;
...
sum += 1e19;
is fastest, but only likely since 1ex for a constant are precomputed values.
To get the best peformance, you might want to precompute the answers for yourself.
Math.pow doesn't distinguish between numbers so it is just as slow for every number, provided that the interpreter doesn't optimize for integers. It is likely to allocate just a few floats to run. I am ignoring parsing time.
"1E"+n will allocate 2~3 string objects which might have quite a substantial memory overhead, destroy intermediates, and reparse it as a number. Unlikely to be faster than pow. I am again ignoring the parse time.

Categories