Firestore can store 64 bit signed integers. However Javascript's number type has a 53 bit mantissa. How to read/write big numbers with Firestore without losing precision?
I tried:
Using BigInt: The Javascript SDK doesn't support it.
Using strings: This could do but I need sort operations for numbers.
For some reason the admin SDK has BigInt support, but the client library not.
Convert BigInt to a string and write it as a string to firestore. This modifies all clients you use to convert from string to BigInt and vice versa but will allow you to store large numbers as needed in the browser at least (due to JavaScript being 53 bit precision-ish). Not ideal but it works. If you need to query on that field in Firestore, I would then use a Cloud function to store the number in another field for querying. I realize this is not the ideal solution, but may help solve your issue in a pinch.
Finally, I would recommend doing is logging a feature request to the FireStore SDK for web to support BigInt.
Related
I have some concerns about the autogenerated datastore keys.
The documentation claims that the result is a 64-bit integer. And for the REST API Google uses a string type to send the key.
From the other hand, Chris Ramsdale, Product Manager says that
These IDs are large, well-distributed integers, but are guaranteed to be small enough to be completely represented as 64-bit floats so they can be stored as Javascript numbers or JSON.
But Chris made this post in 2013.
So I am confused by this discrepancy. Has anyone had any issues with representing autogenerated datastore IDs with JavaScript numbers?
Because otherwise, it is going to be pretty difficult to do the conversion in all HTTP handlers, both for parsing requests and returning responses.
Thank you!
It is possible to manually assign the key ids, so even if GAE never generates ones javascript can't represent, it might be possible for it to be created another way. Best be safe and use strings.
I have numbers generated from Javascript code and I want to store them in PostgreSQL table. I have legacy table where the whole JSON object is stored as JSONB type and in the new table I'd like to flatten the JSON to separate columns.
Ideally I want to avoid loss of precision as much as possible. Especially I'd like to avoid turning JS integer numbers into float numbers and vice versa. In other words inserting integer and getting back float is something I'd like to mitigate (if possible).
So far I've experimented with DOUBLE PRECISION and NUMERIC types. I think NUMERIC is better fit because documentation states that within the implementation limits there is no loss in precision. On the other hand DOUBLE PRECISION will be probably faster for numeric operations. I plan to do a lot of statistical operations.
I am not sure which one to choose. What is the optimal or recommended PostgreSQL data type with regards to maximum compatibility JavaScript Number type?
I am not JavaScript expert, but what I found on net, then JavaScript uses 64bit floats. It is same like DOUBLE PRECISION type - 8bytes like 8bytes.
this might seem a very naive question but I am having a hard time to figure this out. I have a float value 37.50378 in my PostgreSQL database. When I am trying to fetch this value in my Nodejs application it gives me 37.5038. I want to fetch the exact number without rounding off the decimal digits. How do I do that?
The data type of the column in Postgres is Real.
EDIT
I am using Knex schema builder and using float(column, precision, scale) to create a column(to store above-said value). I have tried different numbers for precision and scale just in case that's causing the above-said behavior. But every time I tried to fetch the value 37.50378, all I get back is 37.5038.
Thanks.
You may want to use double(column) in knex, which is translated to double precision in postgres.
This is because of the real 4-byte precision. See PostgreSQL Numeric Types.
It's got nothing to do with Node.js or its PostgreSQL driver.
The number is larger than 9223372036854775807 - too big for NumberLong, which is mongo's native 64-bit long type. What's the best way to do this/the best field type?
Is it possible to preserve any of the querying functionality of a smaller integer (such as {$lt})?
The big numbers are being generated by bignumber.js, and I'm using mongoose to interact with mongoDb.
I'm afraid that the only viable/safe option would be to store such big numbers as a string and serialize it back and forth between the application and MongoDB when read/written. However, you will loose the ability to use MongoDB built-in functions that work with numbers (you can still cast the values to numbers, but it won't be safe anymore).
I am currently working on a project that will involve credit card swipes for admissions based on database rows. Like a will call system, the SHA-256 hash of the CC number must match the hash in the DB row in order to be considered the "proper pickup".
However, because the box office system is based in the browser, the CC number on pickup must be hashed client-side, using Javascript, and then compared to the previously downloaded will call data.
However when trying to hash the numbers, the hash always ends up different than what was hashed when the DB row was created (using VB.NET and SQL Server 2008 R2). For example, if a CC number in the database happened to be 4444333322221111, then the resulting hash from .NET would become xU6sVelMEme0N8aEcCKlNl5cG25kl8Mo5pzTowExenM=.
However, when using any SHA-256 hash library for Javascript I could find, the resulting hash would always be NbjuSagE7lHVQzKSZG096bHtQoMLscYAXyuCXX0Wtw0=.
I'm assuming this is some kind of Unicode/UTF-8 issue, but no matter what I try I cannot get the hashes to come out the same and it's starting to drive me crazy. Can anyone offer any advice?
Here's something that may provide some insight. Please go to http://www.insidepro.com/hashes.php?lang=eng and insert "4444333322221111" without quotes into the Password box. Afterwards, scroll down to the SHA-256 section.
You can see that there are four results, two of them are the hash codes I posted (the second from the top being the Javascript hash and the bottom one being the SQL hash). According to that page, the bottom hash result is generated using a base 64 string, as well as making the password into unicode format.
I've investigated this and tried many different functions to encode the password into unicode format, but no matter what little tweaks I try or other functions I make, I could never get it to match the hash code I need.
I am currently investigating the parameters used when calling the SHA-256 function on the server side.
UPDATE:
So just to make sure I wasn't crazy, I ran the Hash method I'm using for the CC numbers in the immediate window while debugging. Again, the result remains the same as before. You can see a screenshot here: http://i.imgur.com/raEyX.png
According to online SHA-256 hash calculator and a base-64 to hex decoder, it is the .NET implementation that has not calculated the hash correctly. You may want to double check the parameters you pass to the hashing functions.
When you are dealing with two untrusted implementations, it is always a good idea to find another independent implementation, and choose the one that matches the third one as correct. Either that, or find some test vectors, and validate the implementations individually.
EDIT:
A quick experiment shows that the SHA-256 hash you get from .NET matches the hext string 3400340034003400330033003300330032003200320032003100310031003100 - little endian 16-bit characters. Make sure you pass in ASCII.
Adam Liss had it right when he mentioned the byte arrays between strings in .NET/SQL Server are different than strings in Javascript. The array in .NET for the string 4444333322221111 would look like [52 0 52 0 52 0 52 0 51 0 51 0... etc.] and the same thing in Javascript would just look like [52 52 52 52 51 51 51 51...]. Thus, with different byte arrays, different hashes were generated.
I was able to remedy this for my application by modifying the base 64 SHA-256 hashing algorithm from here, where each character is pulled from the string one at a time in order to generate the hash.
Rather than having it do it this way, I first converted the string into a unicode-like byte array (like the .NET example above, 52 0 52 0 etc), fed that array to the hashing algorithm instead of the string, and did some very minor tweaks in order for it to grab each array member to generate the hash. Low and behold, it worked and now I have a very convenient method of hashing CC numbers in the same fashion as the .NET framework for quick and easy order lookup.
Are you sure about your JavaScript SHA256 function ?
And your firstly generated hash ?
SHA-256("4444333322221111"); // 35b8ee49a804ee51d5433292646d3de9b1ed42830bb1c6005f2b825d7d16b70d
hex: 35b8ee49a804ee51d5433292646d3de9b1ed42830bb1c6005f2b825d7d16b70d
HEX: 35B8EE49A804EE51D5433292646D3DE9B1ED42830BB1C6005F2B825D7D16B70D
h:e:x: 35:b8:ee:49:a8:04:ee:51:d5:43:32:92:64:6d:3d:e9:b1:ed:42:83:0b:b1:c6:00:5f:2b:82:5d:7d:16:b7:0d
base64: NbjuSagE7lHVQzKSZG096bHtQoMLscYAXyuCXX0Wtw0=