We Keep Coding

How to process an array of objects containing sensor readings and timestamps and validate how many times and how long the sensor had a certain value?

I am processing some sensor data and my input looks something like this (the values are always 1 or 0, but the length of the values & timestamps arrays is a lot longer, as they contain the readings of a sensor for 24 hours & the data ingestion happens every second):
const input= {
values: [ 0, 1, 0, 0, 1 ],
timestamps: [
'2022-08-19T08:01:21.000Z',
'2022-08-19T08:01:22.000Z',
'2022-08-19T08:01:23.000Z',
'2022-08-19T08:01:24.000Z',
'2022-08-19T08:01:25.000Z'
]
}
or I could easily convert the input to the following format (I couldn't decide, which one would be more suitable):
const input= [{value: 0, timestamp: '2022-08-19T08:01:21.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:22.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:23.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:24.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:25.000Z'}]
My goal is to identify all the periods when the sensor reading was 0 and also validate if these periods were shorter or longer than 1 minute, i.e., for the case above, I'd like to get the following result:
result = [
{startDate: '2022-08-19T08:01:21.000Z', endDate= '2022-08-19T08:01:22.000Z', isShorterThanOneMin: true},
{startDate: '2022-08-19T08:01:23.000Z', endDate= '2022-08-19T08:01:25.000Z', isShorterThanOneMin: true}]
Could you advise a time-efficient way to solve this?

Simple JavaScript version
class Processor {
constructor(inputs) {
this.inputs = inputs;
this.outputs = [];
this.process();
}
process() {
while (this.inputs.length > 0) {
const input = this.inputs.shift();
if (this.previousInput === undefined) {
this.previousInput = input.value === 0 ? input : undefined;
continue;
}
if (this.previousInput.value === 0) {
if (input.value === 0)
continue;
this.outputs.push({
startDate: this.previousInput.date,
endDate: input.date,
isShorterThanOneMinute: (input.date.getTime() - this.previousInput.date.getTime()) < 60000
});
this.previousInput = undefined;
}
}
}
}
const inputs = [
{ value: 0, date: new Date("2022-08-19T08:01:21.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:22.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:23.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:24.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:25.000Z") }
];
const processor = new Processor(inputs);
console.log(processor.outputs);
Fancier, longer TypeScript version
interface Input {
value: number;
date: Date;
}
namespace Input {
export type List = Input[];
export const clone = (input: Input): Input => {
return {
value: input.value,
date: new Date(input.date.getTime())
}
}
}
interface Output {
startDate: Date;
endDate: Date;
isShorterThanOneMinute: boolean;
}
namespace Output {
export type List = Output[];
export const clone = (output: Output): Output => {
return {
startDate: new Date(output.startDate.getTime()),
endDate: new Date(output.endDate.getTime()),
isShorterThanOneMinute: output.isShorterThanOneMinute
}
}
}
class Processor {
private previousInput?: Input;
private outputs: Output.List = [];
private inputs: Input.List;
constructor(inputs: Input.List) {
this.inputs = inputs.map(Input.clone);
this.process();
}
private process() {
while (this.inputs.length > 0) {
const input = this.inputs.shift()!;
if (this.previousInput === undefined) {
this.previousInput = input.value === 0 ? input : undefined;
continue;
}
if (this.previousInput.value === 1) {
throw new Error(`This is not possible, because we never store an input with value = 1.`);
}
if (input.value === 0) continue;
this.outputs.push({
startDate: this.previousInput.date,
endDate: input.date,
isShorterThanOneMinute: (input.date.getTime() - this.previousInput.date.getTime()) < 60000
});
this.previousInput = undefined;
}
}
getOutputs(): Output.List {
return this.outputs.map(Output.clone);
}
append(input: Input): this {
this.inputs.push(Input.clone(input));
this.process();
return this;
}
}
const inputs: Input.List = [
{ value: 0, date: new Date("2022-08-19T08:01:21.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:22.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:23.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:24.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:25.000Z") }
];
const processor = new Processor(inputs);
console.log(processor.getOutputs());
// Continue using the instance as more entries because available...
processor.append({ value: 1, date: new Date("2022-08-19T08:02:25.000Z") });
processor.append({ value: 1, date: new Date("2022-08-19T08:03:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:04:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:05:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:06:25.000Z") });
processor.append({ value: 1, date: new Date("2022-08-19T08:07:25.000Z") });
console.log(processor.getOutputs());

EDIT - #focorner has the fastest code for it methodsName other.
Source: https://sourceb.in/dXDPkOliQr [Made few changes.]
#focorner's answer should be accepted answer as the difference is too high. Idk If I am doing something wrong, but the difference in unbelievable
Ignore -
The current fastest way would be the one I tested called methodTwo. I tested answers from here, and the answer from #focorner doesn't work, not sure why, Maybe due to timestamps.
Of the two methods that I wrote, first, one called Process is actually O(n) time complexity and it's slower than the second version of its. The method processTwo is the fastest method. Note that, these tests were performed on intel i9 with 128GB Ram with an input array length of 360_000.
Having the timestamps as number is the fastest way to do this as new Date().getTime() or Date.now() are going to make it slower.
Date.now() is faster than Date().getTime()
Also destructuring the objects like { value, timestamp } = input is going to increase the time. Destructured assignments are less computationally efficient than traditional.
I used a package called benchmark to test the methods.
processTwo works on very simple logic, that is it checks two elements in a single loop ie element on index i and i + 1.
Json Source: https://rentry.co/x9vtb [360000 is too huge for internet, so smaller bersion]
Source: https://sourceb.in/c9garaf4Ld
The seconds test took only the test data provided by the author and here processTwo is the fastest. I might be wrongly testing but I tried and found out processTwo faster.
I also saw that the output of the Author's approach is just { start: Date, end: Date } and doesn't actually calculate isShorterThanOneMin so I added it to make it even.
You should start storing the number timestamps in an array instead of ISO date or strings.
Source: https://sourceb.in/vqTq2dBMmT

I figured it out, not sure if it's the fastest way, but seems to work:
const input= [{value: 1, timestamp: '2022-08-19T08:01:21.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:22.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:23.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:24.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:25.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:29.000Z'}]
let result = input.reduce((finalArray, currentValue,index, arr) => {
if (currentValue.value === 0) {
let resultElement = {};
resultElement.start = currentValue.timestamp;
//find end Date
let nextToCheck = index;
let endFound = false;
do {
nextToCheck = nextToCheck + 1;
if (nextToCheck === arr.length) {
break;
}
if ((arr[nextToCheck].value) === 1) {
resultElement.end = arr[nextToCheck].timestamp;
endFound = true;
}
} while (!endFound)
// find out if previous one was 1 --> if it was 0, it should not be pushed to avoid having sub-arrays
if (index !== 0 && arr[index - 1 ].value !== 0) {
finalArray.push(resultElement)
}
if (index === 0) {
finalArray.push(resultElement)
}
}
return finalArray;
}, [])
console.log(result)

Comparing two arrays and fill in missing values

I want to compare the dates from 2 arrays and push the name from the matching date inside a new array. And then push '0' for missing dates.
This is what I've tried
var data = [{
name: 'Amy',
date: '2020-01-01'
}, {
name: 'John',
date: '2020-01-02'
}, {
name: 'Sara',
date: '2020-01-04'
}];
var fulldate = ['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-04', '2020-01-05'];
var newData = [];
var len = data.length;
for (var i = 0; i < len; i++) {
if (data[i].date == fulldate[i]) {
newData.push(data[i].name);
} else if (data[i].date != fulldate[i]) {
newData.push("0")
}
}
console.log(newData);
The problem is that it stops after encountering the unmatched date:
Amy,John,0
This is what I need
Amy, John, 0, Sara, 0

The best approach is always to use map and filter. Use map function on the fulldate The obj filters out the object (if present) with same date, as of current el value. I have used ternary operator in the return statement which returns name if object is present, and 0 otherwise.
var data = [{
name: 'Amy',
date: '2020-01-01'
}, {
name: 'John',
date: '2020-01-02'
}, {
name: 'Sara',
date: '2020-01-04'
}];
var fulldate = ['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-04', '2020-01-05'];
var result = fulldate.map((el) => {
let obj = data.filter(item => (item.date == el))
return (obj[0]) ? obj[0].name : 0;
})
console.log(result);

On your code, you have used for-loop based on data variable length but the result has the same length with fulldate so it will be looped based on fulldate variable length.
Inside loop, you have compared fulldate[i] == data[i].date so compared only same index. Inside the loop, you need to use another loop to find the index of the matched date.
Instead of using for loop, you can simply do it using Array.prototype.map function. (Inside map function, using Array.prototype.findIndex, you can find the matched date index.)
var data = [{
name: 'Amy',
date: '2020-01-01'
}, {
name: 'John',
date: '2020-01-02'
}, {
name: 'Sara',
date: '2020-01-04'
}];
var fulldate = ['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-04', '2020-01-05'];
const result = fulldate.map((date) => {
const existed = data.findIndex(item => item.date === date);
return existed >= 0 ? data[existed].name : 0
});
console.log(result);

var data = [{
name: 'Amy',
date: '2020-01-01'
}, {
name: 'John',
date: '2020-01-02'
}, {
name: 'Sara',
date: '2020-01-04'
}];
var fulldate = [
'2020-01-01',
'2020-01-02',
'2020-01-03',
'2020-01-04',
'2020-01-05'
];
var newData = [];
for(var i = 0; i < fulldate.length; i++) {
var found = false;
for(var k in data) {
if(data[k].date == fulldate[i]) {
newData.push(data[k].name);
found = true;
break;
}
}
if(!found)
newData.push("0");
}
console.log(newData);

Suppose the arrays are ordered by date already, use a variable dataIdx to iterate data and Array.map() to simplify the for loop.
var data = [{name:'Amy', date:'2020-01-01'}, {name:'John', date:'2020-01-02'}, {name:'Sara', date:'2020-01-04'}];
var fulldate = ['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-04', '2020-01-05'];
var dataIdx = 0;
var newData = fulldate.map(date => data[dataIdx] && data[dataIdx].date == date ? data[dataIdx++].name : '0');
console.log(newData);

for variable declaration, use 'let' or 'const' instead of var. reason: var vs let vs const
And, the result you are looking for can be done using map & find js func.
const fulldate = ["2020-01-01", "2020-01-02", "2020-01-03", "2020-01-04", "2020-01-05"];
const data = [
{name: "Amy", date: "2020-01-01"},
{name: "John", date: "2020-01-02"},
{name: "Sara", date: "2020-01-04"}
];
const result = fulldate.map(date => {
const matchingDate = data.find(nameDateObj => nameDateObj['date'] === date);
return matchingDate ? matchingDate['name'] : 0;
});
console.log(result)
fyi: this can also be done using findIndex instead of find too.
fulldate.map(date => {
const matchingDateIndex = data.findIndex(nameDateObj => nameDateObj['date'] === date);
return matchingDateIndex > -1 ? data[matchingDateIndex]['name'] : 0;
});

For shortest solution you should combine map and find. Here is a oneliner:
const data = [{
name: 'Amy',
date: '2020-01-01'
}, {
name: 'John',
date: '2020-01-02'
}, {
name: 'Sara',
date: '2020-01-04'
}];
const fulldate = ['2020-01-01', '2020-01-02', '2020-01-03', '2020-01-04', '2020-01-05'];
const result = fulldate.map(x => (data.find(y => y.date === x) || {name: 0}).name)
console.log(result)

NodeJS: Get Count By Month-Year

I have some transactional data which looks as below:
[{UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z", …},
{UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z", …},
{UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z", …},
{UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z", …}]
My requirement it to get count by month-year, so that the output looks like below:
[{period: '2014-03', count:2}
{period: '2014-04', count:1},
{period: '2014-10', count:1}]
I'm doing this in Nodejs, and am just not able to work with the date to make this happen.
Can you please help?

You can use the code given below to group based on period year and month.
let array = [{ UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z" },
{ UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z" },
{ UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z" },
{ UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z" }]
function count(array) {
return array.reduce((total, elem) => {
let temp = elem.createdAt.split("-")
let groupKey = temp[0] + "-" + temp[1];
total[groupKey] ? total[groupKey] +=1: total[groupKey] = 1;
// total[groupKey] += 1;
return total
}, {})
}
console.log(count(array))
The output of code above will be
{ '2014-03': 2, '2014-04': 1, '2014-10': 1 }
Of course you can easily convert from JSON format to array format using code given below
function convertToArray(json_data) {
let result = [];
for (var i in json_data)
result.push({ period: i, count: json_data[i] });
return result;
}
The output will be
[ { period: '2014-03', count: 2 },
{ period: '2014-04', count: 1 },
{ period: '2014-10', count: 1 } ]

You can take the substring of date , take the unique and count the frequency of each date
const arr = [{UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z",},
{UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z"},
{UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z"},
{UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z"}]
//take substring and just grab unique date
let distict_dates = [...new Set(arr.map(a => a.createdAt.substring(0, 7)))];
//count each date frequency
let reduced = distict_dates.map(a => {
return {
userCount: arr.filter(a1 => a1.createdAt.startsWith(a)).length,
createdAt: a
}
}
)
console.log(reduced);

You need this method basically:
.reduce((acc = {}, i) => {
let period = i.createdAt.slice(0,7);
acc[period] = {period, count: acc[period] ? acc[period].count+1: 1}
return acc;
}, {})
let t = [{UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z"},
{UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z"},
{UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z"},
{UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z"}].reduce((acc = {}, i) => {
let period = i.createdAt.slice(0,7);
acc[period] = {period, count: acc[period] ? acc[period].count+1: 1}
return acc;
}, {})
console.log(t);
// if you need exact same result then do it like this
console.log(Object.values(t));

using aggregation:
db.collectionName.aggregate([
{
$project:{
period:{$dateToString:{format:"%Y-%m",date:"$createdAt"}}
}
},{
$group:{ _id : {period : "$period"},count:{$sum:1}}
},
{ $sort : { _id : 1 } }
]);

You can try this:
db.doc.aggregate([{ $group: {
_id:null,
period: {
$dateToString: { format: "%Y-%m", date: "$createdAt" } },
count: { $sum: 1 } },
]
).then()

If the createdAt is a string, convert string to date as follows
var t = Date.parse("2015-04-01T18:30:00.000Z");
var date = new Date(d)
Now get the year and month as follows
date.getMonth() // zero indexed
date.getFullYear()
Append them and form your required string format for createdAt

Eliminate array entries based on date

I have an array of data similar to this:
var items = [
{ id: 84, "completedDate":"2019-01-26T17:45:07.895Z" },
{ id: 92, "completedDate":"2019-02-26T17:45:07.895Z" },
{ id: 123, "completedDate":"2019-03-26T17:45:07.895Z" },
{ id: 2353, "completedDate":"2019-04-26T17:45:07.895Z" }
];
I would like to return an array with only objects less than 30 days old.
I have tried to filter
var filtered = items.filter(function(item) {
return moment(item.completedDate) > moment.subtract(30, 'days');
});
Is this what I need to do, or is there a better way to do this?

You don't need moment to compare dates:
const compareDate = new Date();
compareDate.setDate(compareDate.getDate() - 30);
const filtered = items.filter(item => new Date(item.completedDate) > compareDate);

Here's a similar way to do this without moment. here we just get the current day, reset the time back to the start of the day (you may or may not need this for your use case) and then we just use plain JS date objects to compare
var items = [
{ id: 84, "completedDate":"2019-01-26T17:45:07.895Z" },
{ id: 92, "completedDate":"2019-02-26T17:45:07.895Z" },
{ id: 123, "completedDate":"2019-03-26T17:45:07.895Z" },
{ id: 2353, "completedDate":"2019-04-26T17:45:07.895Z" }
];
var thirtyDaysAgo = new Date();
thirtyDaysAgo.setHours(0, 0, 0, 0);
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
var filtered = items.filter(function(item) {
var d = new Date(item.completedDate).getTime();
return d > thirtyDaysAgo;
});
console.log(filtered);
Or, an even smaller filter function (if you don't need IE 11 support) would be:
var filtered = items.filter((item) => new Date(item.completedDate).getTime() > thirtyDaysAgo);

try
items.filter( x=> x.completedDate > today.toISOString() );
var items = [
{ id: 84, "completedDate":"2019-01-26T17:45:07.895Z" },
{ id: 92, "completedDate":"2019-02-26T17:45:07.895Z" },
{ id: 123, "completedDate":"2019-03-26T17:45:07.895Z" },
{ id: 2353, "completedDate":"2019-04-26T17:45:07.895Z" }
];
var today = new Date("2019-04-20T17:45:07.895Z") // or: new Date()
today = new Date(+today - 30 *86400000)
let r= items.filter( x=> x.completedDate > today.toISOString() );
console.log(r);

JavaScript: Compare dates in an array and sum the "price" for each month/year

I have a json file with multiple transactions with a date and a price attribute. Now I want to compare the dates and if they are in the same month and year I want to sum up the prices.
JSON:
transactions: [
{
date: "2017-11-17",
price: "28",
},
{
...
}
JavaScript:
request.onload = function() {
for(const transaction of request.response.transactions) {
let year = new Date(transaction.date).getFullYear();
let month = new Date(transaction.date).getMonth();
console.log(year + ' ' + month); // output: 2017-11 ...
}
};
I tried to loop over the json object but I struggle to find a solution to compare the dates.

Edit: Edited example with Object.assign instead of Object spread.
You'll need to use reduce to sum the prices. See comments for details.
const transactions = [{
date: "2017-11-17",
price: "28",
},
{
date: "2017-12-17",
price: "23",
},
{
date: "2017-11-17",
price: "12",
},
{
date: "2017-10-17",
price: "55",
},
{
date: "2017-11-17",
price: "09",
},
];
const sumTransactions = (transactions) => {
const summed = transactions.reduce((acc, current) => {
// Get the current date object
const date = new Date(current.date);
// Create your key/identifier
const key = `${date.getFullYear()}-${date.getMonth() + 1}`;
// Retreive the previous price from the accumulator
const previousPrice = acc[key]; // Might also return undefined
// Create your temp current price value, and be sure to deal with numbers.
let currentPrice = Number(current.price);
// If you had a previous value (and not undefined)
if (previousPrice) {
// Add it to our value
currentPrice += Number(previousPrice);
}
// Return the future accumulator value
return Object.assign(acc, {
[key]: currentPrice, // new values will overwrite same old values
})
}, {})
// Once we have all values, get the dates, and sort them (default: earlier first)
// Return an array of each value from the summed object to our sortedArray
const sortedArray = Object.keys(summed).sort().map((val) => {
return summed[val];
});
console.log("sortedArray", sortedArray);
};
sumTransactions(transactions);

I experimented a bit and came up with this solution:
var transactions = [
{
date: "2017-11-17",
price: "28",
},
{
date: "2017-12-17",
price: "22",
},
{
date: "2017-12-17",
price: "20",
}
]
var sumedUpDates = [];
var prices = [];
function isDateSumedUp(date) {
return sumedUpDates.indexOf(date.substring(0, 7)) !== -1;
}
function sumUpDate(date) {
var sum = 0;
transactions.forEach(t => {
if(t.date.substring(0, 7) === date.substring(0, 7)) {
sum += parseInt(t.price);
}
});
sumedUpDates.push(date.substring(0, 7));
prices.push(sum);
}
transactions.forEach(t => {
if(!isDateSumedUp(t.date)) {
sumUpDate(t.date);
}
});
var obj = {};
sumedUpDates.forEach((d, i) => obj[d] = prices[i]);
console.log(obj);

This solutions uses map to format your dates into year/month format for each object entry and then reduce to sum them by those separated dates.
const transactions = [
{date:"2017-11-17", price: "28",},
{date:"2017-12-17", price: "28",},
{date:"2017-11-17", price: "20",},
{date:"2017-12-17", price: "2",},
{date:"2017-11-17", price: "58",},
{date:"2017-11-17", price: "8",},
{date:"2017-10-17", price: "30",},
{date:"2018-11-17", price: "1",},
];
const mapper = single => {
let d = single.date.split('-');
let p = Number(single.price);
return { year: d[0], month: d[1], price: p };
}
const reducer = (group, current) => {
let i = group.findIndex(single => (single.year == current.year && single.month == current.month));
if (i == -1) {
return [ ...group, current ];
}
group[i].price += current.price;
return group;
};
const sumPrices = transactions.map(mapper).reduce(reducer, []);
console.log(sumPrices);

var array = [];
for (var i = 0; i < transactions.length; i++) {
var date = new Date(transactions[i].date);
var ym = date.getFullYear() + "-" + date.getMonth();
if (array[ym] == null) {
array[ym] = 0;
}
array[ym] += parseInt(transactions[i].price);
}
With this data
var transactions = [{
date: "2017-11-17",
price: "28",
},
{
date: "2017-12-17",
price: "5",
},
{
date: "2016-02-17",
price: "28",
},
{
date: "2015-11-17",
price: "25",
},
{
date: "2016-02-17",
price: "12",
},
{
date: "2017-11-17",
price: "50",
}
];
This will give you the sum of all of the year-months duplicates like this :
[
2017-10: 78,
2017-11: 5,
2016-1: 40,
2015-10: 25
]

Another solution is reduce:
var transactions = [
{date: "2017-11-17",price: "28"},
{date: "2017-12-17",price: "22"},
{date: "2017-12-17",price: "20"}
];
var result = transactions.reduce(function(acc, obj) {
var key = obj.date.substr(0,7);
acc[key] = (acc[key] || 0) + +obj.price;
return acc;
}, Object.create(null));
console.log(result);