We Keep Coding

ES2015:multikeys Map : where is the bottleneck?

I have an algorithm coding issue in ES2017 with an async generator. I have to extract some data from a list of randomly-distributed maps.
My initial goal is to code a multi-keys, multiple values (a project I called mukemuva) data structure, with the ability of compacting the overlapping information.
(this is a long post, with some code reviews needed to answers it, so please take a while to answer...)
The problem I attempt to solve is the following:
in a regular ES2015 Map, keys should be scalar values (0, 1, 2, 3, 'avc', true, ...) but not litteral property bags objects (such as 3D points { x:1, y:2, z: 3 } of functions; because JS Maps use some internal hash of the JS engine, not the shape of the property bag itself.
For example, in the node REPL type
m = new Map()
m.set({ x: 1, y: 2, z: 3 } , 123)
m.set({ x: 1, y: 2, z: 3 } , 456)
m.set({ x: 1, y: 2, z: 3 } , 789)
... and you'll get a 3 pairs (key/value) resulting Map.
In a SQL RDBMS, doing the same will probably result in a 1 row table with just the ending value (789) and it is this behaviour that I expect.
NAIVE IMPLEMENTATION
a fast, but memoty conuming implemetnation is the maintain an internal list of pairs:
const pairs = [
{ bag: { x: 0, y: 0, z: 0 }, with_data: { r: 'ba', g: 'da', b: '55' } },
{ bag: { x: 1, y: 2, z: 3 }, with_data: 123 }
]
and loop the whole list to determine how to fill the array:
if a bag is present inside the list, replace its "with_data" value (UPDATE)
if not, create a new entriy in the pairs list, with appropriate "bag" and "with_data" fileds (INSERT/CREATE)
This is super fast but can be troubled or messed up while the size of the entry list increases, especially for high numbers of entries (typically 500K to 1000K)
ADVANCED IMPLEMENTATION
First I have conceptualized and named things :
m.set({ x: 1 y: 2, z: 3 }, 'c0ffee' )
'x', ''y and z: 'z' are so called "roles"
1, 2, and 3 are called "role partitions" or "parts"
the 'coffee' string is the stored "item"
duck blocks
For the storage I have developped a blocks memoty pool allocator that stores data under pseudo random keys generated with a MWC (Multiply-With-Carry) genertor, such as :
pool.set(data) --> uid, a key 5 digits of 37 values each (0-9, _ and A-Z)
pool.get(uid) --> arbitrary data, brought back to user
Using the pool, I can store duck blocks with this shape:
type DuckBlock = [
uid: KEY37(5),
ducktype: ROLE | PART | ITEM,
counter: 0, # refs counter
related: array of KEY37(5), # related block UIDs
fellow: KEY37(5) # "parent" block UID, for PART roles
data: <any> # user arbitrary data object
}
Using lookup Maps to store:
roles: <role_name> => UID,
parts: <part_value> => [UID] (lsit of parts)
items
querying ststem:
const generator = await multimap.select({
x: 0, // single value search
y: '*' // wildcard search,
z: (val) => -5 < val && val < 5 // filter function search
})
Brief Algorithm:
for each role, find the correponding partitions that martches filters
for each part duck block, get the items UIDs
carteian product parts_for_role_X * parts_for_role_Y * parts_for_role_Z
get the item_uids list for each entry
intersect the list s of items to get selecteed items
grab the items by their UID
insert / set up values
For the setting up of the value :
multimap.set({ x: 0, y:0, z: 0}, { rgb: 'bada55'})
... I compute the selection generator, with the values of the key as filters of ther select method, then :
if empty: set anew value
if not empty, overwrite the existing value
THE QUESTION
the naive code is surprisingly fast (3s for 1M pointsrecorded) but I suspect dlete oprations to be costful (imagine an array.splice of a such array)
The advanced version could make a save of up to 25% space (tested) but it is sadly slow (almost 30 min for 1 M records)
I suspect the mechansim of collecting uids to have a too big algorithmic complexity...
But where is really the bootleneck and how could I proceed to fix it ?
(will have its own repo sooner...)
https://github.com/hefeust/data-manipulation-software/mukemuva
Thanks for replies.

Javascript - Adding new label and data to existing array

As I know, it is possible to push more data into an array. Fe, I have an array:
G = [12, 34, 5].
Right now, I can access the nth element like this:
G[n]
I'd now like to push new data in it with a label, so I want the array to look like
G = [12, 34, 5, label:567856, other: Infinity]
where I can get 567856 by calling
G["label"] //(or Infinity by calling G["other"]). How can I achieve this?
I've found
G[i].push({
label:567856,
other: Infinity
})
but this way it adds it as a whole new element, and I'm only able to call G[4]["other"], instead of G["other"]. How can I add the element as I've described?
Thank you!

To add onto Andriy's answer, you need to use Javascript Objects rather than arrays. An object can have indices with custom names. For example, given
var newObj = {"hello": "world", "value":1, "inf": Infinity}
you can do
newObj['hello'] // "world"
newObj['value'] // 1
The problem with
G[i].push({
label:567856,
other: Infinity
})
is that you are pushing an object with 2 attributes, not pushing 2 objects, that's why you need to use G[4]["other"]
See running JSFiddle example.

G["other"] = "something";
With this you will keep the original array, and now have the attribute other, but it is not in [12, 34, 5]
Whit this one you can add an object to the array:
G.push({other: 123})
console.log(G);//[12, 34, 5, object]
console.log(G[3].other);//123
The problem with
G[i].push({
label:567856,
other: Infinity
})
is that you are pushing an object with 2 attributes, not pushing 2 objects, that's why you need to use G[4]["other"]

Arrays in JavaScript are a type of object. As such, they can contain properties:
G.label = 567856;
G.other = Infinity;
The advantage of arrays over other objects is that their indexed elements are ordered.
If you'd like the fourth and fifth elements in the array to be 567856 and Infinity and you want to be able to refer to those values with G.label and G.other, you can do so as follows:
var G = [12, 34, 5];
G.push(G.label = 567856); //same as G.label = 567856; G.push(G.label);
G.push(G.other = Infinity);
You can still iterate through the array using a loop:
var G = [12, 34, 5];
G.push(G.label = 567856);
G.push(G.other = Infinity);
G.forEach(function(val) {
console.log(val); // 12 ... 34 ... 5 ... 567856 ... Infinity
});
console.log(G.label); //567856
console.log(G.other); //Infinity
Note that this does create duplicates. If you change G.label or G.other afterwards, those changes will not be reflected in the fourth and fifth elements of the array.
However, you can overcome that by creating setters on G.label and G.other using Object.defineProperty():
var G = [12, 34, 5];
G.push(G.label = 567856);
G.push(G.other = Infinity);
G.forEach(function(val) {
console.log(val); // 12 ... 34 ... 5 ... 567856 ... Infinity
});
console.log(G.label); //567856
console.log(G.other); //Infinity
Object.defineProperty(G, 'label', {
set: function(x) {
this[3] = x;
}
});
Object.defineProperty(G, 'other', {
set: function(x) {
this[4] = x;
}
})
G.label = 99999;
G.other = 11111;
G.forEach(function(val) {
console.log(val); // 12 ... 34 ... 5 ... 99999 ... 11111
});

Arrays isn't designed to suit your case.
See Array element accessing flow from ECMAScript 262, 5.1 15.4
Array objects give special treatment to a certain class of property
names. A property name P (in the form of a String value) is an array
index if and only if ToString(ToUint32(P)) is equal to P and
ToUint32(P) is not equal to 2^32−1.
So you simply cannot access Array element by alphabetical name because that key won't be parsed to integer by ToUint32.
You can add object to array and store it's index after pushing into array ( Array.prototype.push would return you size of your array):
var G = [1,3,4];
var labelIndex = G.push({'label': 123}) - 1;
console.log(G[labelIndex]["label"]);
Actually that's solution would suite case when you have two or more objects inside your array with same property.
Suggestion below not recommended!
However, you can use code below to define your G Array properties, but it's not value of property of item from your array, it's array property:
G.other = Infinity;
G.label = 567856;
// Access newly created properties
console.log(G["other"]);
console.log(G["label"]);
Good Luck !

Time-series data in JSON

I need to model 1,000,000+ data points in JSON. I am thinking of two ways of doing this:
a) Array of objects:
[{time:123456789,value:1432423},{time:123456790,value:1432424},....]
or
b) Nested arrays
[[123456789,1432423],[123456790,1432424],....]
Naively comparing these two approaches, it feels like the latter is faster because it uses less characters but less descriptive. Is b really faster than a ? Which one would you choose and why ?
Is there a 3rd approach ?

{time:[123456789,123456790,...], value:[1432423,1432424,...]}
why?
iterating over a primitive array is faster.
comparable to "JSON size" with b) but you will not lose the "column" information
this npm could be of interest: https://github.com/michaelwittig/fliptable

If your time series data models some continuous function, especially over regular time intervals, there could be much more efficient representation with delta compression, even if you are still using JSON:
[
{time:10001,value:12345},
{time:10002,value:12354},
{time:10003,value:12354},
{time:10010,value:12352}
]
Can be represented as:
[[10001,1,1,7],[12345,9,,-2]]
Which is a 4 times shorter representation.
The original could be reconstructed with:
[{time:a[0][0],value:a[1][0]},{time:a[0][0] + a[0][1]||1, value: a[1][0] + a[1][1]||0 ...

To add another example (idea: 'time is a key'):
ts1 = {123456789: 1432423, 123456790: 1432424}
One could imagine even:
ts2 = {"2017-01-01": {x: 2, y: 3}, "2017-02-01": {x: 1, y: 5}}
Quite compact in notation.
When you want to get the keys, use Object.keys:
Object.keys(ts2) // ["2017-01-01", "2017-02-01"]
You can then either get the values by iterating using these keys or use the more experimental Object.values:
Object.values(ts2) // [{x: 2, y: 3}, {x: 1, y: 5}
In terms of speed: A quick test with 10.000.000 items in an array worked here:
obj3 = {};
for(var i=0; i < 10000000; i++) {obj3[i] = Math.random()};
console.time("values() test");
Object.values(obj3);
console.timeEnd("values() test");
console.time("keys() test");
Object.keys(obj3);
console.timeEnd("keys() test");
Results at my machine (Chrome, 3.2Ghz Xeon):
values() test: 181.77978515625ms
keys() test: 1230.604736328125ms

underscore/lodash unique by multiple properties

I have an array of objects with duplicates and I'm trying to get a unique listing, where uniqueness is defined by a subset of the properties of the object. For example,
{a:"1",b:"1",c:"2"}
And I want to ignore c in the uniqueness comparison.
I can do something like
_.uniq(myArray,function(element) { return element.a + "_" + element+b});
I was hoping I could do
_.uniq(myArray,function(element) { return {a:element.a, b:element.b} });
But that doesn't work. Is there something like that I can do, or do I need to create a comparable representation of the object if I'm comparing multiple properties?

Use Lodash's uniqWith method:
_.uniqWith(array, [comparator])
This method is like _.uniq except that it accepts comparator which is invoked to compare elements of array. The order of result values is determined by the order they occur in the array. The comparator is invoked with two arguments: (arrVal, othVal).
When the comparator returns true, the items are considered duplicates and only the first occurrence will be included in the new array.
Example:
I have a list of locations with latitude and longitude coordinates -- some of which are identical -- and I want to see the list of locations with unique coordinates:
const locations = [
{
name: "Office 1",
latitude: -30,
longitude: -30
},
{
name: "Office 2",
latitude: -30,
longitude: 10
},
{
name: "Office 3",
latitude: -30,
longitude: 10
}
];
const uniqueLocations = _.uniqWith(
locations,
(locationA, locationB) =>
locationA.latitude === locationB.latitude &&
locationA.longitude === locationB.longitude
);
// Result has Office 1 and Office 2

There doesn't seem to be a straightforward way to do this, unfortunately. Short of writing your own function for this, you'll need to return something that can be directly compared for equality (as in your first example).
One method would be to just .join() the properties you need:
_.uniqBy(myArray, function(elem) { return [elem.a, elem.b].join(); });
Alternatively, you can use _.pick or _.omit to remove whatever you don't need. From there, you could use _.values with a .join(), or even just JSON.stringify:
_.uniqBy(myArray, function(elem) {
return JSON.stringify(_.pick(elem, ['a', 'b']));
});
Keep in mind that objects are not deterministic as far as property order goes, so you may want to just stick to the explicit array approach.
P.S. Replace uniqBy with uniq for Lodash < 4

Here there's the correct answer
javascript - lodash - create a unique list based on multiple attributes.
FYI var result = _.uniqBy(list, v => [v.id, v.sequence].join());

I do think that the join() approach is still the simplest. Despite concerns raised in the previous solution, I think choosing the right separator is the key to avoiding the identified pitfalls (with different value sets returning the same joined value). Keep in mind, the separator need not be a single character, it can be any string that you are confident will not occur naturally in the data itself. I do this all the time and am fond of using '~!$~' as my separator. It can also include special characters like \t\r\n etc.
If the data contained is truly that unpredictable, perhaps the max length is known and you could simply pad each element to its max length before joining.

There is a hint in #voithos and #Danail combined answer. How I solved this was to add a unique key on the objects in my array.
Starting Sample Data
const animalArray = [
{ a: 4, b: 'cat', d: 'generic' },
{ a: 5, b: 'cat', d: 'generic' },
{ a: 4, b: 'dog', d: 'generic' },
{ a: 4, b: 'cat', d: 'generic' },
];
In the example above, I want the array to be unique by a and b but right now I have two objects that have a: 4 and b: 'cat'. By combining a + b into a string I can get a unique key to check by.
{ a: 4, b: 'cat', d: 'generic', id: `${a}-${b}` }. // id is now '4-cat'
Note: You obviously need to map over the data or do this during creation of the object as you cannot reference properties of an object within the same object.
Now the comparison is simple...
_.uniqBy(animalArray, 'id');
The resulting array will be length of 3 it will have removed the last duplicate.

late to the party but I found this in lodash docs.
var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }];
_.uniqWith(objects, _.isEqual);
// => [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]

index of array that is passed

How do I retrieve the index of the array that is passed? The solution I currently use is sending the index too, but that doesn't feel right.
jsFiddle
var obj = {arr: [{x: 1, y: 2},{x: 3, y: 4},{x: 5, y: 6}]};
function myFunction(myObj)
{
alert(myObj); // 5
// alert(the index of the array that is passed); // 2
}
myFunction(obj.arr[2].x);

There's no real way to do what you're asking. JavaScript is purely call-by-value, which means that before a function call is made the arguments are completely evaluated. All that's left after the evaluation is the final value, and a copy of that is passed to the function.
You can of course write code that searches for a value in some relatively-global array, but that would be a waste of CPU cycles if you can instead simply pass the array index to the function.

You can use Array.indexOf jsfiddle
var obj = {arr: [{x: 1, y: 2},{x: 3, y: 4},{x: 5, y: 6}]};
function myFunction(myObj)
{
alert(obj.arr.indexOf(myObj));
}
myFunction(obj.arr[2]);

You can't do that, because the array or the index is not passed to the function, or even the object.
The array is read from the arr property of the object, then an object is read from the array, then the value is read from the x property of that object, and the function is called with that value.
Once inside the function, you can't tell that the value came from a property of an object, or that the object was stored in an array, or that the array was in turn a property in an object.