We Keep Coding

What does OR 0x80 do?

In Bitwise operation, what does | 0x80 do? I know (& 0xFF) is convert value to 8 bit integer but how about (| 0x80) ?
I have the following code:
const buf = createHash('sha256').update('test').digest()
for (let i = 0; i < n; i++) {
const ubyte = buf.readUInt8(i)
const shifted = (ubyte >> 1) | mask
destBuf.writeUInt8(shifted, i)
mask = (ubyte & 1) * 0x80 // mask is 0 or 128
}
Can anyone explain that for me?

0x... means that what comes next is an hexadecimal number.
0x80 is the hexadecimal representation of the number 128. In binary, this equals 10000000.
The | character is the bitwise or operator. Suppose you have a 8-bit number:
a = xxxxxxxx
with x being either a 0 or a 1. Now, masking this number with 0x80 means:
xxxxxxxx | 10000000 = 1xxxxxxx
So it basically means you will have a 1 for your leftmost significant bit, while keeping all the other bits the same.
Now, in your code you use this mask in the line:
const shifted = (ubyte >> 1) | mask
What this does is takes the number ubyte:
ubyte = xxxxxxxy // x and y can be either 1 or 0
It shifts it right by onw digit:
ubyte >> 1 = zxxxxxxx // y gets lost, and z is a 0 if ubyte was unsigned.
Now it masks this number with your mask. When the mask is 128, the result is:
(ubyte >> 1) | 10000000 = 1xxxxxxx
So you will have a 1 as your most significant bit, and the other bits are unchanged.

It combines the bits of both participating numbers with the logical "or":
const b= 0x7;
function tst(a,b){
console.log(a.toString(2).padStart(8)+" first number: "+a)
console.log(b.toString(2).padStart(8)+" second number: "+b)
console.log((a | b).toString(2).padStart(8)+" bitwise overlay: "+(a|b))
console.log("----")
}
[0x80,0x6A,0x70,0x8f].forEach(a=>tst(a,b))

How to add bits in JavaScript

Say you have two integers 10 and 20. That is 00001010 and 00010100. I would then like to just basically concat these as strings, but have the result be a new integer.
00001010 + 00010100 == 0000101000010100
That final number is 2580.
However, I am looking for a way to do this without actually converting them to string. Looking for something more efficient that just does some bit twiddling on the integers themselves. I'm not too familiar with that, but I imagine it would be along the lines of:
var a = 00001010 // == 10
var b = 00010100 // == 20
var c = a << b // == 2580
Note, I would like for this to work with any sequences of bits. So even:
var a = 010101
var b = 01110
var c = a + b == 01010101110

You basic equation is:
c = b + (a << 8).
The trick here is that you need to always shift by 8. But since a and b do not always use all 8 bits in the byte, JavaScript will automatically omit any leading zeros. We need to recover the number of leading zeros (of b), or trailing zeros of a, and prepend them back before adding. This way, all the bits stay in their proper position. This requires an equation like this:
c = b + (a << s + r)
Where s is the highest set bit (going from right to left) in b, and r is the remaining number of bits such that s + r = 8.
Essentially, all you are doing is shifting the first operand a over by 8 bits, to effectively add trailing zeros to a or equally speaking, padding leading zeros to the second operand b. Then you add normally. This can be accomplishing using logarithms, and shifting, and bitwise OR operation to provide an O(1) solution for some arbitrary positive integers a and b where the number of bits in a and b do not exceed some positive integer n. In the case of a byte, n = 8.
// Bitwise log base 2 in O(1) time
function log2(n) {
// Check if n > 0
let bits = 0;
if (n > 0xffff) {
n >>= 16;
bits = 0x10;
}
if (n > 0xff) {
n >>= 8;
bits |= 0x8;
}
if (n > 0xf) {
n >>= 4;
bits |= 0x4;
}
if (n > 0x3) {
n >>= 2;
bits |= 0x2;
}
if (n > 0x1) {
bits |= 0x1;
}
return bits;
}
// Computes the max set bit
// counting from the right to left starting
// at 0. For 20 (10100) we get bit # 4.
function msb(n) {
n |= n >> 1;
n |= n >> 2;
n |= n >> 4;
n |= n >> 8;
n |= n >> 16;
n = n + 1;
// We take the log here because
// n would otherwise be the largest
// magnitude of base 2. So, for 20,
// n+1 would be 16. Which, to
// find the number of bits to shift, we must
// take the log base 2
return log2(n >> 1);
}
// Operands
let a = 0b00001010 // 10
let b = 0b00010100 // 20
// Max number of bits in
// in binary number
let n = 8
// Max set bit is the 16 bit, which is in position
// 4. We will need to pad 4 more zeros
let s = msb(b)
// How many zeros to pad on the left
// 8 - 4 = 4
let r = Math.abs(n - s)
// Shift a over by the computed
// number of bits including padded zeros
let c = b + (a << s + r)
console.log(c)
Output:
2580
Notes:
This is NOT commutative.
Add error checking to log2() for negative numbers, and other edge cases.
References:
https://www.geeksforgeeks.org/find-significant-set-bit-number/
https://github.com/N02870941/java_data_structures/blob/master/src/main/java/util/misc/Mathematics.java

so the problem:
a is 10 (in binary 0000 1010)
b is 20 (in binary 0100 0100)
you want to get 2580 using bit shift somehow.
if you right shift a by 8 using a<<=8 (this is the same as multiplying a by 2^8) you get 1010 0000 0000 which is the same as 10*2^8 = 2560. since the lower bits of a are all 0's (when you use << it fills the new bits with 0) you can just add b on top of it 1010 0000 0000 + 0100 0100 gives you 1010 0001 0100.
so in 1 line of code, it's var result = a<<8 + b. Remember in programming languages, most of them have no explicit built-in types for "binary". But everything is binary in its nature. so int is a "binary", an object is "binary" ....etc. When you want to do some binary operations on some data you can just use the datatype you have as operands for binary operations.
this is a more general version of how to concatenate two numbers' binary representations using no string operations and data
/*
This function concate b to the end of a and put 0's in between them.
b will be treated starting with it's first 1 as its most significant bit
b needs to be bigger than 0, otherwise, Math.log2 will give -Infinity for 0 and NaN for negative b
padding is the number of 0's to add at the end of a
*/
function concate_bits(a, b, padding) {
//add the padding 0's to a
a <<= padding;
//this gets the largest power of 2
var power_of_2 = Math.floor(Math.log2(b));
var power_of_2_value;
while (power_of_2 >= 0) {
power_of_2_value = 2 ** power_of_2;
a <<= 1;
if (b >= power_of_2_value) {
a += 1;
b -= power_of_2_value;
}
power_of_2--;
}
return a;
}
//this will print 2580 as the result
let result = concate_bits(10, 20, 3);
console.log(result);

Note, I would like for this to work with any sequences of bits. So even:
var a = 010101
var b = 01110
var c = a + b == 01010101110
This isn't going to be possible unless you convert to a string or otherwise store the number of bits in each number. 10101 010101 0010101 etc are all the same number (21), and once this is converted to a number, there is no way to tell how many leading zeroes the number originally had.

Understanding converting big endian to/from string

Looking at these implementations, I am wondering if one could explain the reason behind the specific operations. Not coming from computer science, I am not sure why these decisions were made.
function binb2rstr(input) {
var str = []
for (var i = 0, n = input.length * 32; i < n; i += 8) {
var code = (input[i >> 5] >>> (24 - i % 32)) & 0xFF
var val = String.fromCharCode(code)
str.push(val)
}
return str.join('')
}
function rstr2binb(input) {
var output = Array(input.length >> 2)
for (var i = 0, n = output.length; i < n; i++) {
output[i] = 0
}
for (var i = 0, n = input.length * 8; i < n; i += 8) {
output[i >> 5] |= (input.charCodeAt(i / 8) & 0xFF) << (24 - i % 32)
}
return output
}
What I understand so far are:
i += 8 is for iterating through bytes.
0xFF is 255, which is 2^8 - 1, so 1 byte.
32 which is the size of a word, or 4 bytes
| is bitwise OR, <<, >>>, and & are likewise bit operators.
The % modulus keeps the value within that max value of x = x % max.
What I don't understand is:
i >> 5, how that was picked.
& 0xFF, how that was picked.
24 - i % 32, where the 24 came from.
var code = (input[i >> 5] >>> (24 - i % 32)) & 0xFF, how the character code is computed from that.
input.length >> 2
Wondering if this is just a standard computer science function because it's hard to tell where these variables come from and how this was learned. It seems like these values must be a standard algorithm based on byte length but I can't tell how to get there with these open questions. Thank you for your help.

This code consists of some pretty clever bit-fiddling based on 32-bit values.
But let's work on your points:
i >> 5, how that was picked.
This divides i by 32 --- corresponding to the n = input.length * 32 overall length. Considering the whole algorithm this means that one value is processed four times (0,8,16,24) before selecting the next input value.
& 0xFF, how that was picked.
This simply selects the lowest 8-bit of a n-bit value.
24 - i % 32, where the 24 came from.
This relates to i += 8. The i % 32 indicates four different iterations (32/8=4) which are temp= (0, 8, 16, 24). So 24-temp results in (24,16,8,0).
var code = (input[i >> 5] >>> (24 - i % 32)) & 0xFF, how the character code is computed from that.
1. 1st iteration: i=0 ;24-0=24; input[0] >>> 24 & 0xFF = highest byte of input[0] shifted to lowest
2. 2nd iteration: i=8 ;24-8=16; input[0] >>> 16 & 0xFF = 2nd highest byte of input[0] shifted to 2nd lowest
3. 3rd iteration: i=16;24-16=8; input[0] >>> 8 & 0xFF = 2nd lowest byte of input[0] shifted to 2nd highest
4. 4th iteration: i=8 ;24-24=0; input[0] >>> 0 & 0xFF = lowest byte of input[0] shifted to highest
This was the Big-Endian-Conversion.
The next iteration has i=32 and starts the next iteration input[32/32]=input[1].
Overall this algorithm shifts the 32-bit code to the right and masks the lowest 8-bit to be used as a CharCode by String.fromCharCode(code).
The last one is from a different algorithm and so input.length >> 2 simply does a division by 2 discarding the possible rest of 1.
Concerning your last question:
It seems like these values must be a standard algorithm based on byte length but I can't tell how to get there with these open questions.
This is far from a standard algorithm. It is just a clever bit-manipulation based on bytes.
In assembler this code would be even easier to understand.
There is even one instruction called BSWAP to swap between 32-bit Big-Endian and Little-Endian values in a register.

Decreasing byte size of integer block in socket communication

I am developing a multiplayer game server.
On my case, every single byte that really matter for gaming experience and saving bandwith.
Client and server will send some integer values each other.
Integers mostly will have values lower than 100.
In some cases, that integers could have values between 0 and 100000.
All that integers will be send in same sequence. (Imagine that they are integer array)
Using 8 bit integer array or 16 bit integer array is not an option to me because of possible values greater than 65535.
And, I do not want to use 32 bit integer array just for the values what be in action rarely.
So, I developed an algorithm for that (here is the javascript port):
function write(buffer, number){
while(number > 0x7f){
buffer.push(0x80 | (number & 0x7f));
number >>= 7;
}
buffer.push(number);
}
function read(buffer){
var cur, result = 0, shift = 0x8DC54E1C0; // ((((((28 << 6) | 21) << 6) | 14) << 6) | 7) << 6;
while((cur = buffer.shift()) > 0x7f)
{
result |= (cur & 0x7f) << shift;
shift >>= 6;
}
return result | (cur << shift);
}
var d = [];
var number = 127;
write(d, number);
alert("value bytes: " + d);
var newResult = read(d);
alert("equals : " + (number === newResult));
My question is: Is there a better way to solve that problem ?
Thanks in advance

Bitwise operations but what is the meaning (Espruino)

What is the meaning of this line of code
n = (n<<1) | ((d>=0.0004)?1:0);
Trying to understand code from here in function sigOff()
http://www.espruino.com/Remote+Control+Sockets

This snippet seems to use the bitwise OR (|) and left shift (<<) operators:
Bitwise OR: a | b;
Returns a one in each bit position for which the corresponding bits of either or both operands are ones.
Left shift: a << b;
Shifts a in binary representation b (< 32) bits to the left, shifting in zeros from the right.
The left shift by 1 (<< 1) basically doubles the value of n.
Then, the or (|) basically "adds" 1 to the result to make it uneven, if d >= 0.0004.
If d < 0.0004, the result from the left shift isn't changed.
So, for n == 3 and d == 0.0004, this happens:
n << 1 // 6
(d>=0.0004)?1:0 // 1
6 | 1 // 7
For n == 5 and d == 0.0002, this happens:
n << 1 // 10
(d>=0.0004)?1:0 // 0
10 | 0 // 10