We Keep Coding

NodeJS calculating difference between timestamps based on specific conditions

Have an interesting problem.
I have a SQL database, 50000+ entries containing data that looks as follows when queried with KnexJS:
[{
id: 65532,
status: 0.00,
timestamp: 1656124205000,
},
{
id: 65533,
status: 49.81,
timestamp: 1656124503000,
},
{
id: 65534,
status: 0.00,
timestamp: 1656124909000,
}
.... 60000+ more entries
]
I'm using Knex as the query builder.
I want be able to find every object where status is 0 and then get the following object where status is not 0 and then calculate the time difference for each occurrence where that happened.
Usually 0.00 won't be consecutive, but there's no guarantee for that, else I would've just grabbed the id/index where status is 0.00 and + 1.
Any suggestions where to get started with this kinda logic?

How do I do a MongoDB left join, keeping documents with no match on the right?

I'm a little new to this, so forgive me if this is actually trivial. I'm trying to do a left outer join in MongoDB/NodeJS. I would like all matching documents from the left table, even if they have no match on the right. I have a collection of questions:
{_id: 'question1', text: 'This is a question.'}
{_id: 'question2', text: 'This is another question.'}
And a collection of responses, each tied to a user and a question:
{_id: 'response1', QID: 'question1', UID: 'player1', response: 'This is my answer.'}
Now, I'd like to get a list of questions and a user's response to each one, including ones where there is no recorded response, so what I'd want from the documents above might be...
{_id: 'question1', text: 'This is a question.', response: {_id: 'response1', QID: 'question1', UID: 'player1', response: 'This is my answer'}}
{_id: 'question2', text: 'This is another question.', response: []}
Is there a way to do this in the aggregation pipeline? When I use lookup to join responses to questions, and then match the UID, the second question disappears because there's no response tied to it (and thus, fails to satisfy the UID match).
Edit: another thing I tried was to use a let/pipeline in the lookup stage:
const month = 11;
const year = 2020;
const UID = 'aaaaaaa';
//Only get responses by UID 'aaaaaaa'
const myResponses = await Question.aggregate([
{ $match: { year, month } },
{
$lookup: {
from: Response.collection.name,
let: { questionID: '$_id' },
pipeline: [
{
$match: {
$expr: {
$and: [{ $eq: ['$$questionID', '$QID'] }, { $eq: ['$UID', UID] }],
},
},
},
],
as: 'Response',
},
},
This was close, but for some reason, matching the UIDs doesn't seem to work, as it returns an empty array for the Response on every clue. Taking out the last $eq condition and just matching $$questionID and $QID gets every response to every question, as I would've expected, but trying to check for equality to a constant isn't working.
Thanks in advance!

https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/ per the docs
Performs a left outer join

Split object into multiple ordered arrays based on identity of 1 spesific value

Not quite sure if that title was the best I could do.
I'm a pretty new to js and keep running into problems ... I hope some of you have the time to give me a pointer or two on this scenario.
I have several objects that looks pretty much like this - except from the fact that there are 28 instances of every "room" type. I need to split this object into multiple objects - one for each "room" type. In some of my objects there are only one room type - whilst in others there are 3 or 4.
[ { id: 1
created: 2018-12-29T13:18:05.788Z,
room: 'Double Room'
type: 'Standard'
price: 500
},
{ id: 29
created: 2018-12-29T13:18:05.788Z,
room: 'Twin Room'
type: 'Standard'
price: 500
},
{ id: 58
created: 2018-12-29T13:18:05.788Z,
room: 'Family Room'
type: 'Standard'
price: 900
},
]
Oh, and it's important that the instances don't "loose" their order in the array - since it's date related and need to be presentet in an ascending order. And vanilla js only.
Is array.map() the function I'm looking for to solve this problem? Is it posible to do this without iteration?
My final goal is to create some kind of generic function that can sort this out for all my objects.
And guys: happy hollidays!

You could take an object as hash table for the wanted groups. Then iterate the objects and assign the object to the group. If the group does not exist, create a new group with an array.
function groupBy(array, key) {
var groups = Object.create(null);
array.forEach(o => (groups[o[key]] = groups[o[key]] || []).push(o));
return groups;
}
var data = [{ id: 1, created: '2018-12-29T13:18:05.788Z', room: 'Double Room', type: 'Standard', price: 500 }, { id: 29, created: '2018-12-29T13:18:05.788Z', room: 'Twin Room', type: 'Standard', price: 500 }, { id: 58, created: '2018-12-29T13:18:05.788Z', room: 'Family Room', type: 'Standard', price: 900 }],
groupedByRoom = groupBy(data, 'room');
console.log(groupedByRoom);
.as-console-wrapper { max-height: 100% !important; top: 0; }

How to get certain number in this JavaScript array? [duplicate]

This question already has answers here:
How can I access and process nested objects, arrays, or JSON?
(31 answers)
Closed 5 years ago.
I'm new to JavaScript so my array mapping skills are bad, how would I find the assetid which is 47243781293 in this array? Thank you.
EconItem {
appid: 440,
contextid: '2',
assetid: '4723781293',
classid: '2674',
instanceid: '11040547',
amount: 1,
missing: false,
currency: false,
background_color: '3C352E',
icon_url: '...',
icon_url_large: '...',
tradable: false,
actions:
[ { link: 'http://wiki.teamfortress.com/scripts/itemredirect.php?id=5002&lang=en_US',
name: 'Item Wiki Page...' } ],
name: 'Refined Metal',
name_color: '7D6D00',
type: 'Level 3 Craft Item',
market_name: 'Refined Metal',
market_hash_name: 'Refined Metal',
commodity: false,
market_tradable_restriction: 7,
market_marketable_restriction: 0,
id: '4723781293',
fraudwarnings: [],
descriptions: [],
owner_descriptions: [],
owner_actions: [],
tags: [],
marketable: false
}

I think you mean to ask "..which is 47243781293 in this JSON" and not "...which is 47243781293 in this array". The object which you pasted above is a JSON representation. If that's what you are meaning to ask, please read below -
Given that the key id will always be present and the object value pasted above is assigned to variable EconItem, I would try something like this
If( EconItem['id'] === '47243781293' )
{
console.log('Asset id: 47243781293 present in the JSON object');
}
If you are not sure that key id will always be present in the object, I would first check for the key to be present using Object.keys(). Details can be found here - https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/keys
Hope that helps!

To get a value from the object, just refer to it's key: EconItem.assetid
For objects with multiple results, You can iterate on it, and output a particular value from your Object like so:
for (var i in EconItem)
{
console.log(EconItem[i].assetid);
//do more here
}

to find this in an array, you use filter which is a function that ships with every javascript array, using your object above for each EconItem
let array = [EconItem, EconItem]
search = array.filter(eachItem=>eachItem.assetId==='47243781293');
--> returns an array of items with assetId as 47243781293, now the first item should be your EconItem, i.e search[0];
Check example snippet
var items = [{
appid: 440,
contextid: '2',
assetid: '4723781293'
}, {
appid: 441,
contextid: '2',
assetid: '4723781292'
}];
// now we search with this 4723781293
var search = items.filter(function(item){
return item.assetid === '4723781293';
});
//show our search result
alert ("item appid is:"+search[0].appid+", context:"+search[0].contextid+", assetid:"+search[0].assetid);

Prevent Javascript function running out of memory because too many objects

I'm building a web scraper in nodeJS that uses request and cheerio to parse the DOM. While I am using node, I believe this is more of a general javascript question.
tl;dr - creating ~60,000 - 100,000 objects, uses up all my computer's RAM, get an out of memory error in node.
Here's how the scraper works. It's loops within loops, I've never designed anything this complex before so there might be way better ways to do this.
Loop 1: Creates 10 objects in array called 'sitesArr'. Each object represents one website to scrape.
var sitesArr = [
{
name: 'store name',
baseURL: 'www.basedomain.com',
categoryFunct: '(function(){ // do stuff })();',
gender: 'mens',
currency: 'USD',
title_selector: 'h1',
description_selector: 'p.description'
},
// ... x10
]
Loop 2: Loops through 'sitesArr'. For each site it goes to the homepage via 'request' and gets a list of category links, usually 30-70 URLs. Appends these URLs to the current 'sitesArr' object to which they belong, in an array property whose name is 'categories'.
var sitesArr = [
{
name: 'store name',
baseURL: 'www.basedomain.com',
categoryFunct: '(function(){ // do stuff })();',
gender: 'mens',
currency: 'USD',
title_selector: 'h1',
description_selector: 'p.description',
categories: [
{
name: 'shoes',
url: 'www.basedomain.com/shoes'
},{
name: 'socks',
url: 'www.basedomain.com/socks'
} // x 50
]
},
// ... x10
]
Loop 3: Loops through each 'category'. For each URL it gets a list of products links and puts them in an array. Usually ~300-1000 products per category
var sitesArr = [
{
name: 'store name',
baseURL: 'www.basedomain.com',
categoryFunct: '(function(){ // do stuff })();',
gender: 'mens',
currency: 'USD',
title_selector: 'h1',
description_selector: 'p.description',
categories: [
{
name: 'shoes',
url: 'www.basedomain.com/shoes',
products: [
'www.basedomain.com/shoes/product1.html',
'www.basedomain.com/shoes/product2.html',
'www.basedomain.com/shoes/product3.html',
// x 300
]
},// x 50
]
},
// ... x10
]
Loop 4: Loops through each of the 'products' array, goes to each URL and creates an object for each.
var product = {
infoLink: "www.basedomain.com/shoes/product1.html",
description: "This is a description for the object",
title: "Product 1",
Category: "Shoes",
imgs: ['http://foo.com/img.jpg','http://foo.com/img2.jpg','http://foo.com/img3.jpg'],
price: 60,
currency: 'USD'
}
Then, for each product object I'm shipping them off to a MongoDB function which does an upsert into my database
THE ISSUE
This all worked just fine, until the process got large. I'm creating about 60,000 product objects every time this script runs, and after a little while all of my computer's RAM is being used up. What's more, after getting about halfway through my process I get the following error in Node:
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
I'm very much of the mind that this is a code design issue. Should I be "deleting" the objects once I'm done with them? What's the best way to tackle this?