I am creating an app where I sometimes need to allow user to generate some random strings. I would like to force that to be generated in the following format:
xxxx-xxxx-xxxx
Where "x" is some number [0-9] or character [A-Z]. What would the most efficient way to do this? When generated, I would also need to check does it already exist in database so I am a little bit worried about the time which it would take.
We can make it so simple like
require("crypto").randomBytes(64).toString('hex')
You can use crypto library.
var crypto = require('crypto');
//function code taken from http://blog.tompawlak.org/how-to-generate-random-values-nodejs-javascript
function randomValueHex (len) {
return crypto.randomBytes(Math.ceil(len/2))
.toString('hex') // convert to hexadecimal format
.slice(0,len).toUpperCase(); // return required number of characters
}
var string = randomValueHex(4)+"-"+randomValueHex(4)+"-"+randomValueHex(4);
console.log(string);
Check these threads: Generate random string/characters in JavaScript
You can check if the field exists in the database. If it does, just generate a new token. Then check again. The probability of it existing is really low if you don't have large user base. Hence, probability of long loop of checks is low as well.
Related
We have a data model where each entity has 600 boolean values. All of this data needs to travel over the wire from a node.js backend to an Angular frontend, via JSON.
I was thinking about various ways to optimize it (this is an internal API and is not public, so adherence to best practices is less important than performance and saving bandwidth).
I am not a native Javascript speaker, so was hoping to get some feedback on some of the options I was considering, which are:
Turning it into a bitfield and using a huge (600-bit) BigInt.
Is this a feasible approach? I can imagine it would probably be pretty horrific in terms of performance
Splitting the 600 bits into 10 integers (since JS integers are 64 bit), and putting those into an array in the JSON
Base64 encoding a binary blob (will be decoded to a UInt8Array I'm assuming?)
Using something like Protobuf? It might be overkill because I don't want more than 1-2 hours spent on this optimization; definitely don't want to make major changes to the architecture either
Side note: We don't have compression on the server end due to infrastructure reasons, which makes this more complicated and is the reason for us implementing this on the data level.
Thanks!
as Evan points out, transforming your boolean for example into a single character for true="t" and false="f", the 600 boolean will become a joined string of 600 chars which can very well be split into 3 strings of 200 of the sizes, then once received on the front just concatenate the transit and if you want to recover your Bollean values from the string, with a simple reg it becomes possible.
I don't know how the data is set and then obtained, just changing this parameter to which I think needs to be automated.
Once the final string is obtained on the front here is an example of reg ex which can convert your string to an array with your 600 boolean. It is also possible to define indexes by defining an object instead of the array.
function convert_myBool(str)
{
/*var reg = new RegExp('.{1}', 'g');
var tmpTab = str.replace(reg, function(matched){
return matched == "t"?true:false;
});*/
//map is best
tmpTab = str.split('').map((value) =>{
return value == "t"?true:false;
});
return tmpTab;
};
I wrote this dynamically so of course it can be pondered, improved replaced etc. Hoping to have helped :)
Can it be sorted in any way? If there are boolean values that always occur in conjunction with a related value you may be able to group them and simplify.
Depending on what your use for that data is, you may be able to cache some of the it or memoize based on usage frequency. There would be a space tradeoff with caching, however.
I have this issue with Google's Firestore and Google's Realtime DB ids/duplicates but I think it is more general problem and it may have multiple solutions without even considering Firebase.
Right now, I create IDs from my JSON object like this:
// Preparing data
let data = {
workouts: [{...}],
movements: [{...}]
}
// Creating ID from data
const id = btoa(JSON.stringify(data))
// Deleting workouts and movements
delete data.workouts
delete data.movements
// Adding multiple other properties to data objects for example:
data.date = new Date()
... and much more
// Creating new document in DB here or
alerting user it already exists if data.id already exists
When I load the data object from Firestore, I decode it like this:
const data = JSON.parse(atob(this.workout.id))
My goal is to have only unique workouts + movements combinations in my database and generating id based on data from workouts + movements solves it.
The issue is that Realtime DB has limit of 750 Bytes (750 UTF-8 chars per id) and Firestore has limit of 1500 Bytes per id. I have just discovered that by having id that has ~1000 chars. And I believe I would be able to hit even the 1500 chars limit with data from users.
My ideas:
1) Use some different encoding (supporting UTF-8) that will make even long (1000 chars) string to something like 100 chars max. It will still need to be decodable. Is it even possible or Base64 is the shortest it could be?
2) Use autogenerated IDs + save encoded string as data.id parameter to db and when creating new workout always compare this data.id to table of already created workout data.id(s).
Is it possible to solve without looping through all existing workouts?
3) Any other idea? I am still in the realm of decoding/encoding but I believe it must has a different more simple solution.
Do not btoa
First off, Base64 string is probably gonna be longer than the stringified JSON, so if you're struggling with character limit and you can use entire UTF-8, do not btoa anything.
IDs
You're looking for a hash. You could (not recommended) try to roll your own by writing hashing functions for JSON primitives, each must return number:
{ ... } object shall have it's properties sorted by name then hashed
string string shall construct the it's hash from individual characters (.charCodeAt())
number probably can just be kept as-is
[ ... ] Not really sure what would I do with arrays, probably assume different order is different hash and hash them as is
Then you'd deal with the json recursively, constructing the value as:
let hash = 0;
hash += hashStringValue("ddd");
hash *= 31;
hash += hashObjectValue({one:1, test:"text"});
return hash
The multiplication by a prime before addition is a cheap trick, but this only works fir limited depth of the object.
Use library for hash
I googled javascript hash json and found this: https://www.npmjs.com/package/json-hash which looks like what you want:
// If you're not on babel use:
// require('babel/polyfill')
npm install json-hash
var assert = require('assert')
var hash = require('json-hash')
// hash.digest(any, options?)
assert.equal(hash.digest({foo:1,bar:1}), hash.digest({bar:1,foo:1}))
Storage
For the storage of JSON data, if you really need it, use compression algorithm such as LZString. You could also filter the JSON and only keep the values you really need.
On sql server : Out put : 0x5C8C8AAFE7AE37EA4EBDF8BFA01F82B8
SELECT HASHBYTES('MD5', convert(varchar,getdate(),112)+'mytest#+')
On JavaScript : Out put : 5c8c8aafe7ae37ea4ebdf8bfa01f82b8
//to get Md5 Hash bytes
vm.getMd5Hashbytes = function () {
var currentDate = moment().format('YYYYMMDD');
var md5Hash = md5.createHash(currentDate + 'mytest#+');
return md5Hash;
}
angular-md5 module
Q : Can you tell me why this difference ? SQL server shows 0x as prefix.Why ?
This is purely a formatting issue. Both versions are producing an identical sequence of bytes. SQL Server and node just have different conventions when it comes to presenting these bytes in a human readable format.
You can get similar formatting by specifically telling SQL Server how to format your binary data
declare #hashAsBinary varbinary(max)
declare #hashAsText char(32)
set #hashAsBinary = HASHBYTES('MD5', '20160818mytest#+')
set #hashAsText = LOWER(CONVERT(varchar(max), #hashAsBinary, 2))
select #hashAsText
Which outputs:
5c8c8aafe7ae37ea4ebdf8bfa01f82b8
See SQL Server converting varbinary to string
I am not sure how else to explain it but it will take more space than a comment allows for so I will post it as an answer.
Look at the source code that you are referencing. At the end (lines 210 and 212) you will see it converts the binary value to a hex string (and then to lower case which does not matter unless you opt for a string comparison at the end). End result = your JavaScript library returns a representation using the type string formatted as hex.
Your Sql function HASHBYTES on the other hand produces a varbinary typed result (which is a different type than string (varchar)).
So you have 2 different data types (each living on their own space as you have not pulled one to the other). You never mention where you are doing the comparison, ie: on the database or are you pulling from the database to script. Either way to do a comparison you need to convert one type so you are either comparing 2 strings types OR comparing two binary types. If you do not compare similar types you will get unexpected results or run time exceptions.
If you are comparing using strings AND in JavaScript then look at your library that you are referencing, it already has a call named wordToHex, copy and paste it and reuse it to convert your Sql result to a string and then do a string comparison (do not forget to compare case insensitive or also make it lower case).
Edit
WebApi is black box for me.It is a 3rd party service.I just need to send the security token as mentioned above.
Assuming that the type accepted by that web api is byt[] appending 0x to your string in javascript and then sending it to the web api should work as in the web api will then translate the incoming parameter as a byte array and execute the comparison using the correct types. As this is a black box there is no way to know for certain unless you either ask them if the accepted type is indeed a byte array or to test it.
I'm writing a Node API and got a model for which I gotta generate a random number of 15 digits. This must be unique and should not look trivial (I can't get an autoincrement).
I really don't want to generate a number and query against Mongo database for existance checking. I would have to generate some kind of while loop based on promises that way.
I thought about simply using new Date().epoch but, is this going to be unique? could I ever get a duplicate?
Then I also thought on appending something like:
function privateKey (howMany, chars) {
chars = chars
|| "0123456789";
var rnd = crypto.randomBytes(howMany)
, value = new Array(howMany)
, len = chars.length;
for (var i = 0; i < howMany; i++) {
value[i] = chars[rnd[i] % len]
};
return parseInt(value.join(''));
}
To include a duplicity avoiding. How should I implement this?
Edit, this should be a number.
I know there's uuid and Mongo ObjectId but they're not only numbers.
I don't think it's a good idea. One of the reasons is system time skew. Upon synchronizing time with some benchmark server you would get duplicates. In fact this can happen on the runtime every couple of hours. Some servers have serious time drift and they sync time every one in a while. Any time this happens you can get duplicates.
Why not use ObjectId?
According to the MongoDB documentation, ObjectId is a 12-byte BSON type, constructed using:
a 4-byte value representing the seconds since the Unix epoch,
a 3-byte machine identifier,
a 2-byte process id, and
a 3-byte counter, starting with a random value.
ObjectIds are small and fast to generate. MongoDB clients should add an _id field with a unique ObjectId. However, if a client does not add an _id field, mongod will add an _id field that holds an ObjectId.
Edit :
You can convert ObjectId to a number of length of your choice using the below code.
var idNum = parseInt(ObjectId.valueOf(), 15);
Using time for generating unique IDs is not safe. As ak. suggested it you could get duplicates due to bad synchro.
If not having a number is not critical, you should use node-uuid which is based on RFC4122 for unique ID generation.
The project I work on switched to MySQL. The keys we use are UUID strings (like 43d597d7-2323-325a-90fc-21fa5947b9f3), but the database field, rather than be a string, is defined as binary(16) - 16-byte unsigned binary.
I understand that a UUID is basically a 16-byte binary, but I have no idea how to convert from/to a binary number.
I'm using node-mysql to access the database, and I tried using node-uuid to parse the UUID, but that yields an array of integers. I also tried using Node's Buffer, but that just yields a buffer object.
How do I convert a UUID string to fit into that field? And how do I turn a value I read from that field into a UUID?
Due to lack of time, I'll paste the comment that provided valid result(s) and modify the answer later so it's clearer.
Right, if you have a UUID 43d597d7-2323-325a-90fc-21fa5947b9f3 in that string format already in your JS app, you'd send the following query to MySQL:
SELECT col FROM table WHERE uuid_col = UNHEX(REPLACE('43d597d7-2323-325a-90fc-21fa5947b9f3', '-', ''));
If you want to pull data out and have UUID in readable format, you have to convert it to hexadecimal notation.
SELECT HEX(uuid_col) FROM table;
That one will give you the UUID without dashes. It appears that the node-uuid.parse method works if you give it hex string without dashes.
While N.B.'s answer works I stumbled upon another solution.
UUID v1 starts with character segments that are time based; however, the smallest units come first making distribution rather scattered in an index.
If you aren't stuck on the precise UUID v1 format than there is a NodeJS module that can generate unique IDs based on UUID v1 that also monotonically increase and scale about as well as auto incremented IDs. It also works with node-mysql.
Checkout: monotonic-id
An example with node-mysql:
var MID = require('monotonic-id');
var mid = new MID();
client.query('INSERT INTO `...` SET `mid`=?', mid.toBuffer(), function(err, res) {
...
})