I need to model 1,000,000+ data points in JSON. I am thinking of two ways of doing this:
a) Array of objects:
[{time:123456789,value:1432423},{time:123456790,value:1432424},....]
or
b) Nested arrays
[[123456789,1432423],[123456790,1432424],....]
Naively comparing these two approaches, it feels like the latter is faster because it uses less characters but less descriptive. Is b really faster than a ? Which one would you choose and why ?
Is there a 3rd approach ?
{time:[123456789,123456790,...], value:[1432423,1432424,...]}
why?
iterating over a primitive array is faster.
comparable to "JSON size" with b) but you will not lose the "column" information
this npm could be of interest: https://github.com/michaelwittig/fliptable
If your time series data models some continuous function, especially over regular time intervals, there could be much more efficient representation with delta compression, even if you are still using JSON:
[
{time:10001,value:12345},
{time:10002,value:12354},
{time:10003,value:12354},
{time:10010,value:12352}
]
Can be represented as:
[[10001,1,1,7],[12345,9,,-2]]
Which is a 4 times shorter representation.
The original could be reconstructed with:
[{time:a[0][0],value:a[1][0]},{time:a[0][0] + a[0][1]||1, value: a[1][0] + a[1][1]||0 ...
To add another example (idea: 'time is a key'):
ts1 = {123456789: 1432423, 123456790: 1432424}
One could imagine even:
ts2 = {"2017-01-01": {x: 2, y: 3}, "2017-02-01": {x: 1, y: 5}}
Quite compact in notation.
When you want to get the keys, use Object.keys:
Object.keys(ts2) // ["2017-01-01", "2017-02-01"]
You can then either get the values by iterating using these keys or use the more experimental Object.values:
Object.values(ts2) // [{x: 2, y: 3}, {x: 1, y: 5}
In terms of speed: A quick test with 10.000.000 items in an array worked here:
obj3 = {};
for(var i=0; i < 10000000; i++) {obj3[i] = Math.random()};
console.time("values() test");
Object.values(obj3);
console.timeEnd("values() test");
console.time("keys() test");
Object.keys(obj3);
console.timeEnd("keys() test");
Results at my machine (Chrome, 3.2Ghz Xeon):
values() test: 181.77978515625ms
keys() test: 1230.604736328125ms
Related
By default the indexing of every JavaScript array starts from 0. I want to create an array whose indexing starts from 1 instead.
I know, must be very trivial... Thanks for your help.
It isn't trivial. It's impossible. The best you could do is create an object using numeric properties starting at 1 but that's not the same thing.
Why exactly do you want it to start at 1? Either:
Start at 0 and adjust your indices as necessary; or
Start at 0 and just ignore index 0 (ie only use indices 1 and up).
A simple solution is to fill the zeroth item:
var map = [null, 'January', 'February', 'March'];
'First month : ' + map[1];
Semantically it would be better to use an object:
var map = {1:'January', 2:'February', 3:'March'};
'First month : ' + map[1];
Note these keys are not ints actually, object keys are always strings.
Also, we can't use dot notation for accessing. (MDN - Property Accessors)
I'd choose the first solution, which I think is less confusing.
Since this question also pops up for a Google search like "javascript start array at 1" I will give a different answer:
Arrays can be sliced. So you can get a sliced version of the Array like this:
var someArray = [0, 1, 2, 3];
someArray.slice(1);
[1, 2, 3]
someArray.slice(2, 4);
[2, 3]
Source: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/slice
You could use delete to remove the first element like so:
let arr = ['a','b','c'];
delete arr[0];
console.log(arr[0]);
console.log(arr[1]);
Or just not define it at all:
let arr = [,'b','c'];
console.log(arr[0]);
console.log(arr[1]);
If you want to make sure that you always get the first truthy element regardless of the index and have access to ES6 you can use:
arr.find(Boolean)
The question asks "How to create an array in JavaScript whose indexing starts at 1". The accepted answer states "It isn't trivial. It's impossible."
This is true, and should be understood for good reason. However, you can create an array and omit setting the first element, in which case it will still exist (hence the accepted answer being correct) but it'll be marked as empty for you.
let usernames = ['bob', 'sally', 'frank']
let myArray = [];
let arrayIndex = 1;
usernames.map(username => {
myArray[arrayIndex] = username;
arrayIndex++;
})
console.log(myArray);
Array(4) [ <1 empty slot>, "bob", "sally", "frank" ]
1: "bob"
2: "sally"
3: "frank"
length: 4
Notice that the length is "4".
console.log(myArray[0]);
undefined
Using this, there's a quirk in our favour whereby using Object.keys() on an array doesn't return empty (undefined) elements. So with the above array:
console.log(Object.keys(myArray).length);
3
Note: This is arguably a little hacky so use it with caution.
As zero of something rarely exists in our world, doing this might be useful where you are only going to access pre-defined indexes. An example would be if you have pages of a book. There isn't a page 0 as that makes no sense. And if you are always access a value directly, e.g.
const currentPage = pages[1];
Then this is fine in my opinion, as long as the code shows intent. Some will argue that ignoring a valid array index is futile, and I don't fully disagree. However, it's also futile and very annoying when I want to get page 35 of a book and the array index is 34. Meh!
When you loop your (dodgy) array with map it ignores the 0 index you didn't want:
myArray.map((value, index) => {
console.log(index);
console.log(value);
})
1
bob
2
sally
3
frank
For general use however, you should use index 0, so when you loop some data and spit things out you're not going to get caught out by the first one being empty.
Okay, according to #cletus you couldn't do that because it's a built-in javascript feature but you could go slightly different way if you still want that. You could write your own index-dependent functions of Array (like reduce, map, forEach) to start with 1. It's not a difficult task but still ask yourself: why do I need that?
Array.prototype.mapWithIndexOne = function(func) {
const initial = []
for (let i = 1; i < this.length + 1; i++) {
initial.push(func(this[i - 1], i))
}
return initial
}
const array = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
console.log(array.mapWithIndexOne((element, index) => `${element}-${index}`))
// => ["First-1", "Second-2", "Third-3", "Fourth-4", "Fifth-5"]
Codepen: https://codepen.io/anon/pen/rvbNZR?editors=0012
Using Array.map
[,1,2,3].map((v, i) => ++i)
Just wanted to point out that an index in c ish languages is also the offset from the first element. This allows all sorts of offset math where you don't have to subtract 1 before doing the math, only to add the 1 back later.
if you want a "1" array because the indexes are mapped to other values, that's the case for an enumeration or a hash.
First add this function to your javascript codes:
var oneArray = function(theArray)
{
theArray.splice(0,0,null);
return theArray
}
Now use it like this:
var myArray= oneArray(['My', 'name', 'is', 'Ram']);
alert(myArray[1]); << this line show you: My
See live demo
Just prepend a null:
a = [1, 2, 3, 4]
a.unshift(null)
a[3] // 3
Simple, just make two changes to the classic Javascript for loop.
var Array = ['a', 'b', 'c'];
for (var i = 1; i <= Array.length; i++) {
//"i" starts at 1 and ends
//after it equals "length"
console.log(i);
}
There are multiple questions here in SO that are similar to my question, but none really answers it completely.
Basically, I want to have objects as keys in JavaScript. I know it is possible with Map, however, it doesn't fully apply to my use-case. Here is the reason:
Let's say I have two objects, var d1 = {a: 1, b: 2} and var d2 = {a: 1, b: 2}, and a Map var m = new Map(). When adding d1 to m, when I call m.get(d2) I will get undefined. I assume it is because Map works in a reference-like manner.
However, I want the following functionality:
m.set(d1, 'hit');
m.get(d2) // should return 'hit' because properties and values of d1 = d2
One approach I thought of was not to use Map, but a simple JS object. Keys would be JSON.stringify(obj) and when I want to get a value from the object, I use JSON.parse(key) and then perform object equality check (Object.getOwnPropertyNames and then checking one-by-one).
However, I feel that this approach is way more complex and time-consuming than it should be. What I am looking for is probably a hash function of some sort that would efficiently map a object with keys of type String to values of type Number (integer in my case). Also, the ordering can be different (d1 = {a: 1, b: 2} should equal d2 = {b: 2, a: 1}).
How to design an efficient hash function to work with either JS Objects/Maps to perform the above-mentioned operations?
Write a function that turns the object into a string with the keys in a consistent order.
function objToKey(obj) {
Object.keys(obj).sort().map(k => `${k}:${obj[k]}`).join(',');
}
var d1 = {a: 1, b: 2},
d2 = {b: 2, a: 1},
m = new Map();
m.set(objToKey(d1), "foo");
console.log(m.get(objToKey(d2)));
This is a MWE based on some templates going from v3 to v4 of the amazing d3.js.
The data is in csv file, both examples load the same file (its clean):
day,movie1,movie2,movie3,movie4,movie5,movie6
1,20,8,3,0,0,0
2,18,5,1,13,0,0
3,14,3,1,10,0,0
4,7,3,0,5,27,15
5,4,3,0,2,20,14
6,3,1,0,0,10,13
7,2,0,0,0,8,12
8,0,0,0,0,6,11
9,0,0,0,0,3,9
10,0,0,0,0,1,8
here is MWE in question:
d3.csv("../data/source/movies.csv", function (error, data) {
dataViz(data)});
function dataViz(incData) {
expData = incData;
stackData =[];
for (x in incData[0]) {
if (x != "day") {
var newMovieObject = {
name: x, values:[]
};
for (y in incData) {
newMovieObject
.values
.push({
x: parseInt(incData[y][ "day"]),
y: parseInt(incData[y][x])
})
}
stackData
.push(newMovieObject);
}}}
Now in v3 the stackData array has 6 objects with 10 values each e.g.:
{name: "movie1" values:[
{x: 1, y:20} //0
...
{x:10, y:0} //9
]
…
}
In v4 for however I get an array with 6 objects with 11 values each, the last one annoyingly being:
{name: "movie1" values:[
{x: 1, y:20} //0
...
{x:10, y:0} //9
{x: NaN, y: NaN} //10 *ouch*
]
…
}
As a js noob, I don't understand why this vanilla JS function returns different results, and what to do about it? Any help would be greatly appreciated.
The reason for this difference is that D3 v4.x creates an additional property named columns to the data array when it parses the CSV (look at the documentation).
So, for instance, given your data:
day,movie1,movie2,movie3,movie4,movie5,movie6
1,20,8,3,0,0,0
2,18,5,1,13,0,0
...
D3 creates, after the "normal" objects, this additional object (technically speaking, an additional property to the array):
columns: ["day", "movie", "movie2", "movie3", "movie4", "movie5", "movie6"]
Which you can call using data.columns.
The problem you're facing right now is that when you use a for...in loop you end up iterating this property as well, getting a lot of NaN.
Solution: you can simply avoid iterating over columns or, if you don't need it, you can remove it from your data. There are several ways for removing an property from an array in JavaScript, the simpler way being this:
delete incData.columns;
To check this columns property, simply console.log(data) using D3 v3 and v4, comparing the results.
I'm trying to construct an array in JavaScript, but I'm not sure of the correct way to do it, or if this type of array is even possible...
Lets say I have a key for each item in the array starting with 'a' and ending with 'z'. For each item, the array key will correspond with another multidimensional array. These new multidimensional arrays are a series of coordinates (x and y). Each item in the original array can have many sets of coordinates. For example:
How can I construct such an array with Javascript? What is the proper syntax?
Just to add another possible option to your list, on the same lines as #SMcCrohan's answer, mixing objects and arrays.
var coords = {
a: [{
x: 20,
y: 15
},
{
x: 25,
y: 17
}],
b: [{
x: 10,
y: 30
}],
....
};
This assumes you will always use coordinates x and y, It means you can access the values like so:
var value1 = coords.a[1].x; // 25
var value2 = coords.b[0].y; // 30
For the data you've provided:
var arr = {
a: [[20,15],[25,17],[10,45]],
b: [[10,33],[12,2],[14,9],[72,103],[88,12]],
c: [[2,2],[41,21]],
d: [[0,0],[21,2],[44,44],[19,99],[1,1],[100,100]],
e: [[1,1],
f: [[3,40],[41,86]]
}
The first structure you want, a keyed array, isn't an array in JavaScript - it's an object. Objects contain key-value pairs. In this case, the values are arrays, and the objects in those arrays are themselves arrays.
An important thing to note here if you're coming from another language that defines 'regular' multi-dimensional arrays is that there is no expectation or guarantee that the 'rows' of this structure are all the same length.
I have an array of objects with duplicates and I'm trying to get a unique listing, where uniqueness is defined by a subset of the properties of the object. For example,
{a:"1",b:"1",c:"2"}
And I want to ignore c in the uniqueness comparison.
I can do something like
_.uniq(myArray,function(element) { return element.a + "_" + element+b});
I was hoping I could do
_.uniq(myArray,function(element) { return {a:element.a, b:element.b} });
But that doesn't work. Is there something like that I can do, or do I need to create a comparable representation of the object if I'm comparing multiple properties?
Use Lodash's uniqWith method:
_.uniqWith(array, [comparator])
This method is like _.uniq except that it accepts comparator which is invoked to compare elements of array. The order of result values is determined by the order they occur in the array. The comparator is invoked with two arguments: (arrVal, othVal).
When the comparator returns true, the items are considered duplicates and only the first occurrence will be included in the new array.
Example:
I have a list of locations with latitude and longitude coordinates -- some of which are identical -- and I want to see the list of locations with unique coordinates:
const locations = [
{
name: "Office 1",
latitude: -30,
longitude: -30
},
{
name: "Office 2",
latitude: -30,
longitude: 10
},
{
name: "Office 3",
latitude: -30,
longitude: 10
}
];
const uniqueLocations = _.uniqWith(
locations,
(locationA, locationB) =>
locationA.latitude === locationB.latitude &&
locationA.longitude === locationB.longitude
);
// Result has Office 1 and Office 2
There doesn't seem to be a straightforward way to do this, unfortunately. Short of writing your own function for this, you'll need to return something that can be directly compared for equality (as in your first example).
One method would be to just .join() the properties you need:
_.uniqBy(myArray, function(elem) { return [elem.a, elem.b].join(); });
Alternatively, you can use _.pick or _.omit to remove whatever you don't need. From there, you could use _.values with a .join(), or even just JSON.stringify:
_.uniqBy(myArray, function(elem) {
return JSON.stringify(_.pick(elem, ['a', 'b']));
});
Keep in mind that objects are not deterministic as far as property order goes, so you may want to just stick to the explicit array approach.
P.S. Replace uniqBy with uniq for Lodash < 4
Here there's the correct answer
javascript - lodash - create a unique list based on multiple attributes.
FYI var result = _.uniqBy(list, v => [v.id, v.sequence].join());
I do think that the join() approach is still the simplest. Despite concerns raised in the previous solution, I think choosing the right separator is the key to avoiding the identified pitfalls (with different value sets returning the same joined value). Keep in mind, the separator need not be a single character, it can be any string that you are confident will not occur naturally in the data itself. I do this all the time and am fond of using '~!$~' as my separator. It can also include special characters like \t\r\n etc.
If the data contained is truly that unpredictable, perhaps the max length is known and you could simply pad each element to its max length before joining.
There is a hint in #voithos and #Danail combined answer. How I solved this was to add a unique key on the objects in my array.
Starting Sample Data
const animalArray = [
{ a: 4, b: 'cat', d: 'generic' },
{ a: 5, b: 'cat', d: 'generic' },
{ a: 4, b: 'dog', d: 'generic' },
{ a: 4, b: 'cat', d: 'generic' },
];
In the example above, I want the array to be unique by a and b but right now I have two objects that have a: 4 and b: 'cat'. By combining a + b into a string I can get a unique key to check by.
{ a: 4, b: 'cat', d: 'generic', id: `${a}-${b}` }. // id is now '4-cat'
Note: You obviously need to map over the data or do this during creation of the object as you cannot reference properties of an object within the same object.
Now the comparison is simple...
_.uniqBy(animalArray, 'id');
The resulting array will be length of 3 it will have removed the last duplicate.
late to the party but I found this in lodash docs.
var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }];
_.uniqWith(objects, _.isEqual);
// => [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]