I am working on something where I take data from 2 different APIs that I have no control of and I want to combine the results in the most efficient way.
One of the arrays hold some assets, lets say books, the other one holds a transaction for the said book. Here is an example:
{
author: {name: 'J.K. Rowling', },
assetName: 'Book1'
}]
const array2 = [
{from: 'John',
to: 'Sarah,
price: 10,
timeStamp: 123,
assetName: 'Book1',
authorName: 'J.K. Rowling'
}]
Note that to find the corresponding transaction for a given book, you need both assetName and authorName to match - you can own more than one book of the same author and you can own two books with the same name but a different author but an author has only one book with a given name, thus finding the corresponding transaction to an asset requires both fields to match and there are no other unique identifiers.
The naive approach is to iterate over one of the arrays and for each entry to check in the second array to find the transaction but that looks like it will take too long to execute if the arrays are of substantial size.
I was wondering what better solutions can you think of for merging two objects with different structure that is efficient?
Well, if author.name + assetName form an id, you could iterate over array1 once & create a Map with keys being author.name + assetName & vales being original objects.
Then you could iterate over array2 once as well & enrich it whatever way you want. All lookups in the second iteration will be fast since you will access the Map instead of searching in array.
const indexedArray1 = new Map();
array1.forEach(data => indexedArray1.set(data.author.name + data.assetName, data);
const enrichedArray2 = array2.map(transaction => {
const relatedBook = indexedArray1.get(transaction.authorName + transaction.assetName);
// Merge relatedBook & transaction the way you want here
});
I often do the following when merging arrays
The time complexity is O(n)
const array1 = [{
author: {name: 'J.K. Rowling' },
assetName: 'Book1'
}]
const array2 = [{
from: 'John',
to: 'Sarah',
price: 10,
timeStamp: 123,
assetName: 'Book1',
authorName: 'J.K. Rowling'
}]
const array2_map = {}
array2.forEach(e => {
const key = `${e.assetName}:${e.authorName}`
if (!array2_map[key]) array2_map[key] = []
const { from, to, price, timeStamp } = e
array2_map[key].push({
from,
to,
price,
timeStamp
})
})
const merged_array = array1.map(e => ({
...e,
transaction: array2_map[`${e.assetName}:${e.authorName}`] || []
}))
Related
I am looking for the way to remove duplicates. I found a common way is to create a Set and then spread into a new Array.
How could I Set to acomplish this purpose? For instance, I have the following code:
const tmp1=[];
const tmp2=[{
guid:"e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
}];
const tmp = [...new Set([...tmp1],[...tmp2])]; //This should remove duplicates, but gets empty array
const x = [...tmp1, ...tmp2]; // This would keep duplicates
The issue is that because tmp1 is an empty array, then I am getting empty result. However, if I do the following, then getting correct result:
const tmp = [...new Set(...tmp1,[...tmp2])];
I think something is missing in here.
This is an example of duplicated entries where Set is working like a charm just keeping one record:
const tmp1=[{
guid:"e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
}];
const tmp2=[{
guid:"e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
}];
const tmp = [...new Set([...tmp1],[...tmp2])];
This was the original idea, but how about if one of the lists is empty. Then, I am getting an empty array if this occurs.
Thank you
Set should take 1 argument but it's taking 2, merge them into one:
const tmp = [...new Set([...tmp1, ...tmp2])];
Note: that this method won't remove duplicates because you are passing an object to the set and not a reference for it, in this case, you can do this instead:
const tmp1 = [];
const tmp2 = [{
guid: "e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
},
{
guid: "e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
}
];
// pass a special unique key that differs object from each other to item, in this case i passed guid
const tmp = [...new Map([...tmp1, ...tmp2].map(item => [item['guid'], item])).values()]
Just to explain why this example in the question seemingly work:
const tmp1=[{
guid:"e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
}];
const tmp2=[{
guid:"e695d848-7188-4741-9c95-44bec634940f",
name: "Spreading.pdf",
code: "G1"
}];
const tmp = [...new Set([...tmp1],[...tmp2])];
It results in one object because Set() only take one iterable object, so it takes the first one.
If const tmp2=[1, 2, 3] in the above example, tmp will still result in the one object in tmp1.
More about Set()
const users = [
{id:1, email:"abc#email.com"},
{id:2, email:"xyz#email.com"},
{....}(~70,000 objects)
]
function a(){
const id = 545
users.filter((value)=>{
if(value.id === id)
return true
})
}
We have 70,000 users' objects. we need to filter the email based on the id.
users= [{id: '1001', email: "abc#gmail.com"}, {{id: '1002', email: "spc#gmail.com"} , ..];
Using array and array.filter() ends up in the error.
Error
what's the best way of approach for this?
It might be best to convert your array into a Map so that lookups can be made without scanning the entire array. As such:
const lookupMap = new Map(users.map((u) => [u.id, u]));
so now you can
const user = lookupMap.get(userId)
without having to scan all 70000 user objects.
Let's say I have the following two arrays:
let arr1 = [ { id: "1234567890", name: "Someone", other: "unneeded", props: 123 } ... ];
let arr2 = [ { id: "1234567890", points: 100, other: "unneeded", props: 456 } ... ];
I need to combine these based on the name and points by id which looks like:
[ { id: "1234567890", name: "Someone", points: 100 } ... ]
One option would be to map them like so:
let final = arr1.map(u => ({
id: u.id,
name: u.name,
points: arr2.find(uu => uu.id === u.id)
}));
However, this is inefficient for larger arrays (thousands of entries), since find() iterates through the array each time. I'm trying to make this more efficient. I read over Array.reduce(), but that doesn't seem like the answer (unless I'm wrong). How can I do this?
You can create a Map from the objects of the second array so that you can access the corresponding object directly by id in constant time:
let arr1 = [
{id: 1234567890, name: "Someone", other: "unneeded", props: 123},
{id: 1234567891, name: "Someone1", other: "unneeded1", props: 124},
{id: 1234567892, name: "Someone2", other: "unneeded2", props: 125}
];
let arr2 = [
{id: 1234567890, points: 100, other: "unneeded", props: 456},
{id: 1234567891, points: 101, other: "unneeded", props: 457},
{id: 1234567892, points: 102, other: "unneeded", props: 458}
];
let arr2Map = arr2.reduce((a, c) => {
a.set(c.id, c);
return a;
}, new Map());
let final = arr1.map(({id, name, points}) =>
({id, name, points: arr2Map.get(id).points || points}));
console.log(final);
One solution you could try is Webworkers. If you haven't used them before, they can run script in a separate thread, essentially allowing you to use multiple cores in order to process something. They are great for parallelizable tasks, meaning tasks that can broken up without any disruption. They would fit your use case pretty well since you are just doing a big map on your data.
Before going down this path though you should be forewarned that there is some overhead with webworkers. In order to get data into a webworker you have to serialize it into the thread and then deserialize it when it returns. However, if you split up your tasks into separate works and have them operating in parallel you should be able to mitigate some of that.
// app.js
let largeArray = []; // contains the large amount of arrays to map
let outputArray = [];
while(largeArray.length){
let worker = new Worker('/path/to/worker/script.js');
worker.postMessage({
chunk: largeArray.splice(0,1000)
});
worker.onmessage = evt => {
let data = evt.data;
outputArray = outputArray.concat(data);
worker.terminate();
}
}
Separately have a worker script, maybe similar to this one available to be referenced.
// workerScript.js
self.onmessage = ({data: {chunk}}) => {
let final = chunk.map(u => ({
id: u.id,
name: u.name,
points: arr2.find(uu => uu.id === u.id)
}));
self.postMessage(final);
}
You might ask about the serializing, and it's something that happens automatically. Webworkers have their own global scope, and as a result of serializing you can send Objects, Arrays, and primitives. All of this can be serialized. Objects with custom properties and classes will throw errors though. Behind the scenes it's basically taking your data and doing JSON.stringify({{your data}}) and then within the webworker data becomes the result of JSON.parse({{serialized data}}).
Because this work is occurring in separate threads you won't see any blocking on the main thread where your app is running. If you are trying to process too much at once though, the serializing will be noticeable as it's blocking until completion.
If I have a list of objects, eg:
const poeple = [
{name: "Johhny", born:1980, passed:2080},
{name: "Jenny", born:1990, passed:2093},
...{name: "Jan", born:1999, passed:2096}
];
I found out how to make a new list, with the ages like this:
const ages = people.map(person => person.passed - person.born);
And I can add the property 'age' to the original people, like this:
people.map(person => person["age"] = person.passed - person.born);
But, is there a (quick/efficient/eloquent) way to just make a new array, with each object having the additional property, as I don't want to change the original array?
I tried to use the array push method, but that just adds objects one at a time, with a loop, is there any other way to do this with less code?
You can use map method with spread syntax in object.
const people = [{name: "Johhny", born:1980, passed:2080},{name: "Jenny", born:1990, passed:2093},{name: "Jan", born:1999, passed:2096}];
const result = people.map(person => ({
...person,
age: person.passed - person.born
}))
console.log(result)
You can use Array.map() and create the new object by appending the age to the existing object using Object.assign():
const people = [{name: "Johhny", born:1980, passed:2080},{name: "Jenny", born:1990, passed:2093},{name: "Jan", born:1999, passed:2096}];
const result = people.map(o => Object.assign({
age: o.passed - o.born
}, o));
console.log(result);
I am setting up my tests for the results to a REST endpoint that returns me an array of Mongo database objects.
[{_id: 5, title: 'Blah', owner: 'Ted', description: 'something'...},
{_id: 70, title: 'GGG', owner: 'Ted', description: 'something'...}...]
What I want my tests to verify is that in the return array it conatins the specific titles that should return. Nothing I do using Chai/Chai-Things seems to work. Things like res.body.savedResults.should.include.something.that.equals({title: 'Blah'}) error out I'm assuming since the record object contains other keys and values besides just title.
Is there a way to make it do what I want? I just need to verify that the titles are in the array and don't care what the other data might be (IE _id).
Thanks
This is what I usually do within the test:
var result = query_result;
var members = [];
result.forEach(function(e){
members.push(e.title);
});
expect(members).to.have.members(['expected_title_1','expected_title_2']);
If you know the order of the return array you could also do this:
expect(result).to.have.deep.property('[0].title', 'expected_title_1');
expect(result).to.have.deep.property('[1].title', 'expected_title_2');
As stated here following code works now with chai-like#0.2.14 and chai-things. I just love the natural readability of this approach.
var chai = require('chai'),
expect = chai.expect;
chai.use(require('chai-like'));
chai.use(require('chai-things')); // Don't swap these two
expect(data).to.be.an('array').that.contains.something.like({title: 'Blah'});
Probably the best way now a days would be to use deep.members property
This checks for unordered complete equality. (for incomplete equality change members for includes)
i.e.
expect([ {a:1} ]).to.have.deep.members([ {a:1} ]); // passes
expect([ {a:1} ]).to.have.members([ {a:1} ]); // fails
Here is a great article on testing arrays and objects
https://medium.com/building-ibotta/testing-arrays-and-objects-with-chai-js-4b372310fe6d
DISCLAIMER: this is to not only test the title property, but rather a whole array of objects
ES6+
Clean, functional and without dependencies, simply use a map to filter the key you want to check
something like:
const data = [{_id: 5, title: 'Blah', owner: 'Ted', description: 'something'},{_id: 70, title: 'GGG', owner: 'Ted', description: 'something'}];
expect(data.map(e=>({title:e.title}))).to.include({title:"Blah"});
or even shorter if you only check one key:
expect(data.map(e=>(e.title))).to.include("Blah");
https://www.chaijs.com/api/bdd/
Here is another approach that I found to be more helpful. Basically, use string interpolation and map your array of objects to an array of string literals. Then you can write expectations against the array of strings.
const locations: GeoPoint[] = [
{
latitude: 10,
longitude: 10
},
{
latitude: 9,
longitude: 9
},
{
latitude: -10,
longitude: -10
},
{
latitude: -9,
longitude: -9
}
];
const stringLocations: string[] = locations.map((val: GeoPoint) =>
`${val.latitude},${val.longitude}`
);
expect(stringLocations).to.contain('-9.5,-9.5');
expect(stringLocations).to.contain('9.5,9.5');
I loved the suggestion from #sebastien-horin
But another way with Should syntax (for the specific property):
const data = [
{ _id: 5, title: 'Blah', owner: 'Ted', description: 'something' },
{ _id: 7, title: 'Test', owner: 'Ted', description: 'something' },
];
data.map((e) => e.title).every((title) => title.should.equal('Blah'));
An alternative solution could be extending the array object with a function to test if an object exists inside the array with the desired property matching the expected value, like this
/**
* #return {boolean}
*/
Array.prototype.HasObjectWithPropertyValue = function (key, value) {
for (var i = 0; i < this.length; i++) {
if (this[i][key] === value) return true;
}
return false;
};
(i put this in my main test.js file, so that all other nested tests can use the function)
Then you can use it in your tests like this
var result = query_result;
// in my case (using superagent request) here goes
// var result = res.body;
result.HasObjectWithPropertyValue('property', someValue).should.equal(true);