Let's say I have the following two arrays:
let arr1 = [ { id: "1234567890", name: "Someone", other: "unneeded", props: 123 } ... ];
let arr2 = [ { id: "1234567890", points: 100, other: "unneeded", props: 456 } ... ];
I need to combine these based on the name and points by id which looks like:
[ { id: "1234567890", name: "Someone", points: 100 } ... ]
One option would be to map them like so:
let final = arr1.map(u => ({
id: u.id,
name: u.name,
points: arr2.find(uu => uu.id === u.id)
}));
However, this is inefficient for larger arrays (thousands of entries), since find() iterates through the array each time. I'm trying to make this more efficient. I read over Array.reduce(), but that doesn't seem like the answer (unless I'm wrong). How can I do this?
You can create a Map from the objects of the second array so that you can access the corresponding object directly by id in constant time:
let arr1 = [
{id: 1234567890, name: "Someone", other: "unneeded", props: 123},
{id: 1234567891, name: "Someone1", other: "unneeded1", props: 124},
{id: 1234567892, name: "Someone2", other: "unneeded2", props: 125}
];
let arr2 = [
{id: 1234567890, points: 100, other: "unneeded", props: 456},
{id: 1234567891, points: 101, other: "unneeded", props: 457},
{id: 1234567892, points: 102, other: "unneeded", props: 458}
];
let arr2Map = arr2.reduce((a, c) => {
a.set(c.id, c);
return a;
}, new Map());
let final = arr1.map(({id, name, points}) =>
({id, name, points: arr2Map.get(id).points || points}));
console.log(final);
One solution you could try is Webworkers. If you haven't used them before, they can run script in a separate thread, essentially allowing you to use multiple cores in order to process something. They are great for parallelizable tasks, meaning tasks that can broken up without any disruption. They would fit your use case pretty well since you are just doing a big map on your data.
Before going down this path though you should be forewarned that there is some overhead with webworkers. In order to get data into a webworker you have to serialize it into the thread and then deserialize it when it returns. However, if you split up your tasks into separate works and have them operating in parallel you should be able to mitigate some of that.
// app.js
let largeArray = []; // contains the large amount of arrays to map
let outputArray = [];
while(largeArray.length){
let worker = new Worker('/path/to/worker/script.js');
worker.postMessage({
chunk: largeArray.splice(0,1000)
});
worker.onmessage = evt => {
let data = evt.data;
outputArray = outputArray.concat(data);
worker.terminate();
}
}
Separately have a worker script, maybe similar to this one available to be referenced.
// workerScript.js
self.onmessage = ({data: {chunk}}) => {
let final = chunk.map(u => ({
id: u.id,
name: u.name,
points: arr2.find(uu => uu.id === u.id)
}));
self.postMessage(final);
}
You might ask about the serializing, and it's something that happens automatically. Webworkers have their own global scope, and as a result of serializing you can send Objects, Arrays, and primitives. All of this can be serialized. Objects with custom properties and classes will throw errors though. Behind the scenes it's basically taking your data and doing JSON.stringify({{your data}}) and then within the webworker data becomes the result of JSON.parse({{serialized data}}).
Because this work is occurring in separate threads you won't see any blocking on the main thread where your app is running. If you are trying to process too much at once though, the serializing will be noticeable as it's blocking until completion.
Related
I am working on something where I take data from 2 different APIs that I have no control of and I want to combine the results in the most efficient way.
One of the arrays hold some assets, lets say books, the other one holds a transaction for the said book. Here is an example:
{
author: {name: 'J.K. Rowling', },
assetName: 'Book1'
}]
const array2 = [
{from: 'John',
to: 'Sarah,
price: 10,
timeStamp: 123,
assetName: 'Book1',
authorName: 'J.K. Rowling'
}]
Note that to find the corresponding transaction for a given book, you need both assetName and authorName to match - you can own more than one book of the same author and you can own two books with the same name but a different author but an author has only one book with a given name, thus finding the corresponding transaction to an asset requires both fields to match and there are no other unique identifiers.
The naive approach is to iterate over one of the arrays and for each entry to check in the second array to find the transaction but that looks like it will take too long to execute if the arrays are of substantial size.
I was wondering what better solutions can you think of for merging two objects with different structure that is efficient?
Well, if author.name + assetName form an id, you could iterate over array1 once & create a Map with keys being author.name + assetName & vales being original objects.
Then you could iterate over array2 once as well & enrich it whatever way you want. All lookups in the second iteration will be fast since you will access the Map instead of searching in array.
const indexedArray1 = new Map();
array1.forEach(data => indexedArray1.set(data.author.name + data.assetName, data);
const enrichedArray2 = array2.map(transaction => {
const relatedBook = indexedArray1.get(transaction.authorName + transaction.assetName);
// Merge relatedBook & transaction the way you want here
});
I often do the following when merging arrays
The time complexity is O(n)
const array1 = [{
author: {name: 'J.K. Rowling' },
assetName: 'Book1'
}]
const array2 = [{
from: 'John',
to: 'Sarah',
price: 10,
timeStamp: 123,
assetName: 'Book1',
authorName: 'J.K. Rowling'
}]
const array2_map = {}
array2.forEach(e => {
const key = `${e.assetName}:${e.authorName}`
if (!array2_map[key]) array2_map[key] = []
const { from, to, price, timeStamp } = e
array2_map[key].push({
from,
to,
price,
timeStamp
})
})
const merged_array = array1.map(e => ({
...e,
transaction: array2_map[`${e.assetName}:${e.authorName}`] || []
}))
I'm not sure what to name this, but basically I'm new to React and Redux and looking for a more correct/cleaner way to do this or just how to do this with my current set up.
I have a state that looks like this
--Character
---id
---name
---race
----id
----raceName
----traits
-----trait
------id
------name
------description
-----trait
------id
------name
------description
---classes
----class
-----id
-----className
-----classLevel
-----traits
------trait
-------id
-------name
-------description
------trait
-------id
-------name
-------description
----class
-----id
-----className
-----classLevel
-----traits
------trait
-------id
-------name
-------description
------trait
-------id
-------name
-------description
---traits
----trait
-----id
-----name
-----description
----trait
-----id
-----name
-----description
As you can see(hopefully) traits is an array of object TRAIT and classes is an array of object CLASS, in the end the whole state is quite a messy deal. I've read that I can somehow reference them by ID's but I'm not sure how if IDs are autogenerated.
So I kind of have two questions:
How do I simplify/flatten this structure if it even could be done?
If I can't simplify this structure is there anyway I can find a specific Trait with a specific ID without looping through all the objects that have property traits?
Yes. You can find Trait with a specific ID easily. Let know if this is what you are asking.
// Search in traits directly under Character.
const traitForId = this.state.Character.traits.find((trait) => {
return trait.id = "<SPECIFIC_ID>"
})
// Search in the list of traits under each Class.
const classTraits = this.state.Character.classes.map((class) => class.traits).flat();
const classTraitsForId = classTraits.find((trait) => {
return trait.id = "<SPECIFIC_ID>"
})
Find below recursive way to find a Trait irrespective of where it's present in the state.
function findTraitForId(state, specific_id){
if(state.traits){
const traitForId = state.traits.find((trait) => {
return trait.id == specific_id
});
if(traitForId)
return traitForId;
}
return Object.keys(state).filter((key) => key != 'traits').map((stateItem) => {
return findTraitForId(state[stateItem], specific_id);
}).flat();
}
Tried above function for the input
findTraitForId({'classes':[{traits: [{id: 1, name: "A"}, {id: 2, name: "AB"}]}, {traits: [{id: 3, name: "ABC"}, {id: 4, name: "ABCD"}]}], traits: [{id: 5, name: "ABCDE"}, {id: 6, name: "ABCDEF"}]}, 3)
which return
[{id: 3, name: "ABC"}]
I'm struggeling to understand the different rxjs operators. For example when i've got a collection of objects and i want to add additional data from an api to each object.
e.g.
people = [{id: 1, name: null}, {id:2, name: null}]
from(people).pipe(
map(person => {
return api.getName(person.id).pipe(
map(name => person.name = name)
)
})
).subscribe(people =>
console.log(people) // should be [{id: 1, name: bob}, {id:2, name: alice}]
)
I tried using mergeMap, map, switchMap in different variations but i never figured out how to get map the additional data into the array.
Your issue is likely when you are using map/switch map you are returning the result of the assignment person.name = name;
When you use an arrow function without curly braces like this
() => something;
It is actually shorthand for
() => {
return something
}
Here is a working example, with some sloppy class creation to make it work...
https://stackblitz.com/edit/angular-ivy-xfunnc?file=src%2Fapp%2Fapp.component.ts
people = [{id: 1, name: null}, {id:2, name: null}];
peopleWithNames = from(this.people).pipe(
switchMap(person => {
return this.api.getName(person.id).pipe(
map(name => {
// set the person's name property
person.name = name;
// but return the whole person object, not just the retrun from the assignment
return person;
})
)
})
).subscribe(p => {
console.log(p)
})
If you want a list, I think do you need use forkJoin. You map each element of people to a call
people = [{id: 1, name: null}, {id:2, name: null}];
forkJoin(people.map(p=>this.api.getName(person.id))
.subscribe((res:any[])=>{
res.forEach((p:any,index)=>{
people[index]={...people,..p}
})
})
Well you can use map too to get the response with the full elements
people = [{id: 1, name: null}, {id:2, name: null}];
forkJoin(this.people.map(p=>this.api.getName(p.id))).pipe(
map((res:any[])=>{
const result=this.people.map((p,index)=>
//if the response is an object map to
//({...p,...res[index]})
//if the response is a string,
({...p,name:res[index]})
)
return result
})
).subscribe(res=>{console.log(res)})
this is bit more theoretical question. I originally intented to call this question Is it possible to iterate over map twice, but just from the sound of it, it sounds like an anti-pattern. So I assume I'm just approaching this wrong.
Also note: Data here servers as an abstraction. I know that what I'm doing here with data provided here is unnecessary, but please don't get fixated too much on data-structure and what-not. It does not represent the real (more complex, which furthermore is provided by client and I can't alter) data I'm working with. Instead approach the problem as how to return structured async calls for each array item please! :-)
My problem boils down to this:
I have array of ids on which I need to execture two separate asynchronous calls
Both of these callls need to pass (and in all id instances)
So as an example, imagine I have these two data-sets:
const dataMessages = [
{ id: "a", value: "hello" },
{ id: "b", value: "world" }
];
const dataNames = [
{ id: "a", name: "Samuel" },
{ id: "b", name: "John" },
{ id: "c", name: "Gustav"},
];
And an API-call mock-up:
const fetchFor = async (collection: Object[], id: string): Promise<Object> => {
const user = collection.find(user => user.id === id);
if (!user) {
throw Error(`User ${id} not found`);
}
return user;
};
Now I need to call the fetchFor() function for both the data-sets, presumably inside the inside the Promise.all, given forEach is not asynchronous from a predetermined array of ids.
I was thinking something akin to maping a list of Promises for the Promise.all to execute. This works fine, when you only need to map a single api-call:
const run = async () => {
const result = await Promise.all(
['a', 'b'].map(id => fetchFor(dataMessages, id)
)
console.log(result) // [{id: 'a', value: 'hello'}, {id: 'b', value: 'world}]
}
However I somehow need to return both promises for the
fetchFor(dataMessages, id)
fetchFor(dataNames, id)
inside the Promise.all array of Promises.
I guess I could always simply do a flatMap of two maps for both instances of API calls, but that sounds kinda dumb, given
I'd be doing array.map on same array twice
My data structure would not be logically connected (two separate array items for the same user, which would not even by followed by eachother)
So ideally I'd like to return dat in form of
const result = await Promise.all([...])
console.log(result)
/* [
* {id: 'a', message: 'hello', name: 'Samuel'},
* {id: 'b', message: 'world', name: 'John'},
* ]
Or do I simply have to do flatmap of promises and then do data-merging to objects based on id identifier inside a separate handler on the resolved Promise.all?
I've provided a working example of the single-api-call mockup here, so you don't have to copy-paste.
What would be the correct / common way of approaching such an issue?
You could nest Promise.all calls:
const [messages, names] = await Promise.all([
Promise.all(
['a', 'b'].map(id => fetchFor(dataMessages, id)
),
Promise.all(
['a', 'b', 'c'].map(id => fetchFor(dataNames, id)
)
]);
If you're wanting to then merge the results after retrieved, it's just a matter of standard data manipulation.
I am setting up my tests for the results to a REST endpoint that returns me an array of Mongo database objects.
[{_id: 5, title: 'Blah', owner: 'Ted', description: 'something'...},
{_id: 70, title: 'GGG', owner: 'Ted', description: 'something'...}...]
What I want my tests to verify is that in the return array it conatins the specific titles that should return. Nothing I do using Chai/Chai-Things seems to work. Things like res.body.savedResults.should.include.something.that.equals({title: 'Blah'}) error out I'm assuming since the record object contains other keys and values besides just title.
Is there a way to make it do what I want? I just need to verify that the titles are in the array and don't care what the other data might be (IE _id).
Thanks
This is what I usually do within the test:
var result = query_result;
var members = [];
result.forEach(function(e){
members.push(e.title);
});
expect(members).to.have.members(['expected_title_1','expected_title_2']);
If you know the order of the return array you could also do this:
expect(result).to.have.deep.property('[0].title', 'expected_title_1');
expect(result).to.have.deep.property('[1].title', 'expected_title_2');
As stated here following code works now with chai-like#0.2.14 and chai-things. I just love the natural readability of this approach.
var chai = require('chai'),
expect = chai.expect;
chai.use(require('chai-like'));
chai.use(require('chai-things')); // Don't swap these two
expect(data).to.be.an('array').that.contains.something.like({title: 'Blah'});
Probably the best way now a days would be to use deep.members property
This checks for unordered complete equality. (for incomplete equality change members for includes)
i.e.
expect([ {a:1} ]).to.have.deep.members([ {a:1} ]); // passes
expect([ {a:1} ]).to.have.members([ {a:1} ]); // fails
Here is a great article on testing arrays and objects
https://medium.com/building-ibotta/testing-arrays-and-objects-with-chai-js-4b372310fe6d
DISCLAIMER: this is to not only test the title property, but rather a whole array of objects
ES6+
Clean, functional and without dependencies, simply use a map to filter the key you want to check
something like:
const data = [{_id: 5, title: 'Blah', owner: 'Ted', description: 'something'},{_id: 70, title: 'GGG', owner: 'Ted', description: 'something'}];
expect(data.map(e=>({title:e.title}))).to.include({title:"Blah"});
or even shorter if you only check one key:
expect(data.map(e=>(e.title))).to.include("Blah");
https://www.chaijs.com/api/bdd/
Here is another approach that I found to be more helpful. Basically, use string interpolation and map your array of objects to an array of string literals. Then you can write expectations against the array of strings.
const locations: GeoPoint[] = [
{
latitude: 10,
longitude: 10
},
{
latitude: 9,
longitude: 9
},
{
latitude: -10,
longitude: -10
},
{
latitude: -9,
longitude: -9
}
];
const stringLocations: string[] = locations.map((val: GeoPoint) =>
`${val.latitude},${val.longitude}`
);
expect(stringLocations).to.contain('-9.5,-9.5');
expect(stringLocations).to.contain('9.5,9.5');
I loved the suggestion from #sebastien-horin
But another way with Should syntax (for the specific property):
const data = [
{ _id: 5, title: 'Blah', owner: 'Ted', description: 'something' },
{ _id: 7, title: 'Test', owner: 'Ted', description: 'something' },
];
data.map((e) => e.title).every((title) => title.should.equal('Blah'));
An alternative solution could be extending the array object with a function to test if an object exists inside the array with the desired property matching the expected value, like this
/**
* #return {boolean}
*/
Array.prototype.HasObjectWithPropertyValue = function (key, value) {
for (var i = 0; i < this.length; i++) {
if (this[i][key] === value) return true;
}
return false;
};
(i put this in my main test.js file, so that all other nested tests can use the function)
Then you can use it in your tests like this
var result = query_result;
// in my case (using superagent request) here goes
// var result = res.body;
result.HasObjectWithPropertyValue('property', someValue).should.equal(true);