We Keep Coding

Manually "compressing" a very large number of boolean values in JSON

We have a data model where each entity has 600 boolean values. All of this data needs to travel over the wire from a node.js backend to an Angular frontend, via JSON.
I was thinking about various ways to optimize it (this is an internal API and is not public, so adherence to best practices is less important than performance and saving bandwidth).
I am not a native Javascript speaker, so was hoping to get some feedback on some of the options I was considering, which are:
Turning it into a bitfield and using a huge (600-bit) BigInt.
Is this a feasible approach? I can imagine it would probably be pretty horrific in terms of performance
Splitting the 600 bits into 10 integers (since JS integers are 64 bit), and putting those into an array in the JSON
Base64 encoding a binary blob (will be decoded to a UInt8Array I'm assuming?)
Using something like Protobuf? It might be overkill because I don't want more than 1-2 hours spent on this optimization; definitely don't want to make major changes to the architecture either
Side note: We don't have compression on the server end due to infrastructure reasons, which makes this more complicated and is the reason for us implementing this on the data level.
Thanks!

as Evan points out, transforming your boolean for example into a single character for true="t" and false="f", the 600 boolean will become a joined string of 600 chars which can very well be split into 3 strings of 200 of the sizes, then once received on the front just concatenate the transit and if you want to recover your Bollean values ​​from the string, with a simple reg it becomes possible.
I don't know how the data is set and then obtained, just changing this parameter to which I think needs to be automated.
Once the final string is obtained on the front here is an example of reg ex which can convert your string to an array with your 600 boolean. It is also possible to define indexes by defining an object instead of the array.
function convert_myBool(str)
{
/*var reg = new RegExp('.{1}', 'g');
var tmpTab = str.replace(reg, function(matched){
return matched == "t"?true:false;
});*/
//map is best
tmpTab = str.split('').map((value) =>{
return value == "t"?true:false;
});
return tmpTab;
};
I wrote this dynamically so of course it can be pondered, improved replaced etc. Hoping to have helped :)

Can it be sorted in any way? If there are boolean values that always occur in conjunction with a related value you may be able to group them and simplify.
Depending on what your use for that data is, you may be able to cache some of the it or memoize based on usage frequency. There would be a space tradeoff with caching, however.

How to parse phone number as JSON

The main stuff
I receive an MQTT packet (which appears as an ASCII array to the buffer in format [after being printed using stringify]):
packet = {
"cmd": "publish",
"retain": true,
"qos": 1,
"dup": false,
"length": 36,
"topic": "topic/subtopic",
"payload": {
"type": "Buffer",
"data": [123,34,50,57,34,58,43,52,52,55,56,53,49,54,56,53,50,51,52,125]
},
"messageId": 3
}
packet.payload converts to {"29":+447...} which is the beginning of a mobile number. When passed into JSON.parse(packet.payload), it throws SyntaxError: Unexpected token + in JSON at position 6.
My question is how can I extract the value (and less importantly, the key)?
Extra (probably unnecessary) context you can skip over
The error makes sense since + isn't a valid number type and it's not encased as a string.
I attempted convert the value to a string using a reviver function and a filter, with no success, nor was I able to get the whole thing treated as a string with JSON.stringify(packet.payload), packet.payload.toString() or String(packet.payload). I can do a manual conversion from ASCII chars to string between {, : and } but I'm looking for a more robust, generalised solution that doesn't manipulate the ASCII.
Parsing works as expected on a different packet {"6000":100} to give an expected key value pair return.
Using .entries() returned some weird behaviour - (on the other packet) it returned 6000,100 which was of array type, but unusable as an array or number (as far as I could figure out). This might hint as to what's going on?
I was also wondering whether it was possible to extend the behaviour of .parse() to accept a number in the form +44... or to take that value and return it as "+44...".
Edit for extra clarity:
The first thing I wanted to do was limit the output datatypes from the upstream device (a PLC) to primitives (like int) only, however this isn't possible or even enforceable, so my broker needs to be able to handle whatever is thrown at it even if it's invalid type.
With regards to why the packet comes in a JSON format? I honestly don't know - but I'd think it's either how MQTT natively works, or the Aedes broker service I'm running on Node.
The broker has an event publish which will emit the client and the packet whenever a publisher publishes a packet. This is how I am receiving the packet on the server end of things.

The following code will take the array and convert it back to a string via a buffer.
var buffer = Buffer.from(packet.payload.array)
var string = buffer.toString('utf8')
In that case that packet.payload is already a buffer then the following should work:
var buffer = Bufffer.from(packet.payload)
EDIT:
var regex = /{ \"\d+\": (\+\d+) }/
var matches = packet.payload.toString('utf8').match(regex)
var phoneNumber = matches[1]

How do I pass parameters to the ldapjs exop() function?

I'm trying to use the "Change User's Password" extended operation, as defined in this RFC which states that it takes a sequence of three optional parameters. However, it seems that ldapjs's client.exop() function only allows me to provide it with a string or buffer.
This is my attempt:
const dn = `uid=${username},ou=People,dc=${orginization},dc=com`
client.exop('1.3.6.1.4.1.4203.1.11.1', [dn, null, newPassword], (err, value, res) => {
// ...
})
And this is the resulting error:
TypeError: options.requestValue must be a buffer or a string
How am I supposed to encode those values into a string? The ldapjs documentation gives very little information about passing parameters to an extended operation.

TL:DR; The extended operation parameters need to be ASN.1 values encoded with the BER standard. This isn't a trivial task, so you might want an additional npm library, such as asn1 to help with this process.
After combing through ldapjs's code, reading up a bunch about ASN.1 and how LDAP uses the ASN.1 standard, and some trial and error, I was finally able to resolve this issue. Because of the distinct lack of documentation for this, I thought I would share what I learned on stackoverflow so others don't need to go through as much trouble as I did.
A working example
This uses the asn1 npm library to encode the data being sent.
const { Ber } = require('asn1')
// ...
const CTX_SPECIFIC_CLASS = 0b10 << 6
const writer = new Ber.Writer()
writer.startSequence()
writer.writeString(dn, CTX_SPECIFIC_CLASS | 0) // sequence item number 0
// I'm choosing to omit the optional sequence item number 1
writer.writeString(newPassword, CTX_SPECIFIC_CLASS | 2) // sequence item number 2
writer.endSequence()
client.exop('1.3.6.1.4.1.4203.1.11.1', writer.buffer, (err, value, res) => {
// ...
})
What is ASN.1?
ASN.1 is a language that's used to describe an interface for an object. These interfaces are special in that they are language agnostic - i.e. javascript can create an object that conforms to one of these interfaces, encode it, and send it to a python server which then decodes and validates the object against the same interface. Much of ASN.1 is not relevant to what we're trying to accomplish, but it's important to note that what we're trying to do is make an object that conforms to one of these ASN.1 interfaces (LDAP is built all around them).
What is BER?
BER describes a standard way to represent an object that conforms to an ASN.1 interface. Using the BER standard, we can encode javascript data into a buffer that can be understood by an LDAP server.
BER basics
BER is designed to be a very compact encoding standard. I'll go over the basics here, but I highly recommend this article if you want to get into more details about the binary representation of BER (it's tailored to LDAP users). A Layman's Guide to a Subset of ASN.1, BER, and DER is another great resource.
ASN.1 describes a number of basic object types, such as strings and numbers, and It describes structured object types such as a sequence or set. It also provides the power for users to use their own custom types.
In BER, each piece of data is prefixed by two bytes (usually): An identifier byte and a length-of-data byte. The identifier byte tags the data with information about the kind of data it contains (A string? sequence? custom type?). There are four "classes" of tags: universal (such as a string), application (LDAP defined some application tags which you might encounter), context-specific (See the "BER Sequences" section below), and private (not likely applicable here). A string tag's bit sequence will always be interpreted as a string tag, but the meaning for the bit sequence of a custom tag may vary on the environment, or even within a request.
In the asn1 npm library, you can write out a string element as follows:
writer.writeString('text')
To find all of the available functions, the author of this library asks that you peek into the source code.
BER Sequences
A sequence is used to describe an object (a set of key-value pairs) with a particular shape. Some elements may be optional while others are required. The RFC I was following gave the following description for its parameters. We need to conform to this sequence's interface in order to send our password reset parameters to LDAP.
PasswdModifyRequestValue ::= SEQUENCE {
userIdentity [0] OCTET STRING OPTIONAL
oldPasswd [1] OCTET STRING OPTIONAL
newPasswd [2] OCTET STRING OPTIONAL }
The [0], [1], and [2] all refer to context-specific tag numbers. A value tagged with the context-specific tag of 1 will be interpreted as the value for the oldPasswd argument. We don't need to use the global string tag to indicate that our value is of type string - LDAP can already infer that information using the interface we're conforming to. This means when writing a string in this sequence, instead of doing writer.writeString('text') as done before (which automatically used the global string tag), a tag number must be provided as follows:
const CTX_SPECIFIC_CLASS = 0b10 << 6
writer.writeString(newPassword, CTX_SPECIFIC_CLASS | 2) // The second optional parameter allows you to set a custom tag on the data being set (instead of the default string tag).
The first two bits of the tag byte are reserved for specifying the tag class (in this case, it's the context-specific class, or bits "10"). So, CTX_SPECIFIC_CLASS | 2 refers to the newPasswd sequence item described by the RFC. Note that if I want to omit an optional sequence entry, I just don't write out a value tagged with that sequence id.
Concluding Remarks
Hopefully this should give readers enough information to be able to format and send BER-encoded parameters for an extended LDAP operation. I do want to note that I am no ASN.1/BER expert - all of this information above is just how I understood these concepts from my own research over the past couple of days. So, there are likely a few things mis-explained in this post. Feel free to edit it if you happen to be more knowledgeable than me about this topic.

Convert UUID to/from binary in Node

The project I work on switched to MySQL. The keys we use are UUID strings (like 43d597d7-2323-325a-90fc-21fa5947b9f3), but the database field, rather than be a string, is defined as binary(16) - 16-byte unsigned binary.
I understand that a UUID is basically a 16-byte binary, but I have no idea how to convert from/to a binary number.
I'm using node-mysql to access the database, and I tried using node-uuid to parse the UUID, but that yields an array of integers. I also tried using Node's Buffer, but that just yields a buffer object.
How do I convert a UUID string to fit into that field? And how do I turn a value I read from that field into a UUID?

Due to lack of time, I'll paste the comment that provided valid result(s) and modify the answer later so it's clearer.
Right, if you have a UUID 43d597d7-2323-325a-90fc-21fa5947b9f3 in that string format already in your JS app, you'd send the following query to MySQL:
SELECT col FROM table WHERE uuid_col = UNHEX(REPLACE('43d597d7-2323-325a-90fc-21fa5947b9f3', '-', ''));
If you want to pull data out and have UUID in readable format, you have to convert it to hexadecimal notation.
SELECT HEX(uuid_col) FROM table;
That one will give you the UUID without dashes. It appears that the node-uuid.parse method works if you give it hex string without dashes.

While N.B.'s answer works I stumbled upon another solution.
UUID v1 starts with character segments that are time based; however, the smallest units come first making distribution rather scattered in an index.
If you aren't stuck on the precise UUID v1 format than there is a NodeJS module that can generate unique IDs based on UUID v1 that also monotonically increase and scale about as well as auto incremented IDs. It also works with node-mysql.
Checkout: monotonic-id
An example with node-mysql:
var MID = require('monotonic-id');
var mid = new MID();
client.query('INSERT INTO `...` SET `mid`=?', mid.toBuffer(), function(err, res) {
...
})

Extract values from a 'binary format' file and create text file with fixed field sizes

I've used JavaScript to get the data through the internet (I'm interfacing with a brokerage firm's API functions), but unlike most of the rest of their API's, this one returns the data in a 'binary' like format. Here is the layout of the file I get back:
Field -------- Type ------------ Length(8 bit bytes) --------------- Description
Symbol Count----Integer------------- 4--------------------------- Number of symbols for which data is being returned. The subsequent sections are repeated this many times
REPEATING SYMBOL DATA
Symbol Length Short 2 Length of the Symbol field
Symbol String Variable The symbol for which the historical data is returned
Error Code Byte 1 0=OK, 1=ERROR
Error Length Short 2 Only returned if Error Code=1. Length of the Error string
Error Text String Variable Only returned if Error Code=1. The string describing the error
Bar Count Integer 4 # of chart bars; only if error code=0
REPEATING PRICE DATA
close Float 4
high Float 4
Low Float 4
open Float 4
volume Float 4 in 100's
timestamp Long 8 time in milliseconds from 00:00:00 UTC on January 1, 1970
END OF REPEATING PRICE DATA
Terminator Bytes 2 0xFF, 0XFF
END OF REPEATING SYMBOL DATA
As you can see, this file is a mixture of different types of fields. My requirement is to convert this file from the way it is into a fixed field text file (or CSV file). I'm not very good at JavaScript, but I know enough to get by. My main language is MAPPER from Unisys (it is actually called "Business Information Server"). Currently I get all HTTP responses as text files, but this one is a 'binary' file, and MAPPER can not process it because it is a text-based language (a 4GL). I've spent days trying to find a snippet of JavaScript code that I could use, but to no avail. I think this is really simple stuff for a guy that knows JavaScript.

I'm a fellow UNISYS programmer. 25 years of FORTRAN 77 on a 2200 mainframe. Happily, I rarely had anything to do with MAPPER.
I'd like to help, but you're not providing enough information.
Where is this JavaScript code running? In a browser, or is it an extension to whatever you're using to access MAPPER?
Are you using some kind of terminal emulator? AttachMate?
Is your data really arriving in a file, or is it in memory? How are you receiving it, how are you passing on the contents?
Is it vital that your processing happen in JavaScript? There are dozens of languages that would make very short work of the task if the data were lying around as a file and the output should be a file too.
One problem I see is that, AFAIK, JavaScript doesn't know about file IO. That's why I'm asking where it's running.
EDIT:
OK, somehow you have a browser-like environment and JavaScript running in it.
First, the problem of getting binary data out of your response. Here's a bit of help:
https://developer.mozilla.org/en/using_xmlhttprequest
This is Mozilla documentation, under "Receiving binary data," but I'm hoping there will be enough overlap for it to be useful:
function load_binary_resource(url) {
var req = new XMLHttpRequest();
req.open('GET', url, false);
//XHR binary charset opt by Marcus Granado 2006 [http://mgran.blogspot.com]
req.overrideMimeType('text/plain; charset=x-user-defined');
req.send(null);
if (req.status != 200) return '';
return req.responseText;
}
The above lets you fiddle with the connection a bit to hopefully obtain binary data.
That function is called like so:
var filestream = load_binary_resource(url);
var abyte = filestream.charCodeAt(x) & 0xff;
...and if I understand this correctly, your responseText is a JavaScript string (as usual) but thanks to the fiddling and the binary content, it's not containing printable text but binary data. Heh, as long as you don't try to interpret it, it's just a series of bytes just like any old text.
The second line lets you extract a single byte from any position in the string. That byte will be a value between 0 and 255; or if you're unlucky, between -128 and 127. Not sure how JavaScript deals with signed bytes.
This may look like it's doing you a lot of no good. Let's see how you could get to your data:
Your data starts off with a short called symbolLength. I'm guessing a short is 2 bytes, and I'm guessing that offsets for charCodeAt() begin at 0. So you'll be wanting the first two bytes, or bytes 0 and 1. I'm not sure if your data will be coming in high-endian or low-endian, but you should be able to reconstruct that short from either
var symbolLength = fileStream.charCodeAt(0) + 256 * fileStream.charCodeAt(1);
or
var symbolLength = 256 * fileStream.charCodeAt(0) + fileStream.charCodeAt(1);
In other words, using multiplication to re-assemble bytes into integers.
Integers are presumably 4 bytes, so you'll be multiplying by 4 powers of 256: 16777216, 65536, 256 and 1 - again, either in that order or reversed.
The String data is, of course, just that, and once you've taken into account the number of bytes taken up by the preceding fields, you should be able to dig it out of your response string simply using substring operators.
Now for the yucky part - conversion of floats. The internal structure of those numbers is defined by IEEE 754. float probably corresponds to binary32 and double (if you have any) to binary64. The links from the Wikipedia article I linked explain these formats well enough that you could write your own conversion routine if you were desparate, but in your shoes I'd go looking for ready-built coding for this. Surely you're not the first person faced with converting a handful of bytes into a floating point number. Maybe you can find some C or Java code you could hand-convert, or you could even find a routine already written in JavaScript.
Finally, once you have in hand methods to convert all the data types you mentioned, all you need to do is to format that data in whatever format you want to see downstream in MAPPER. Loop through the structures, incrementing the pointers for the offsets... probably nothing new for you.
Admittedly, I've done a lot of guessing and handwaving here. This could be the beginning of a solution but you'll probably want to do a bit of experimenting and hit SO with some detail questions. Don't mention UNISYS, phrase your question as if you wanted to do this in IE :)
As a first step, I'd try dumping out your incoming binary string, byte-wise, preferrably in hex, to some medium where you can read it and compare the bytes you see with the bytes you're expecting from the input data.