We Keep Coding

How to process an array of objects containing sensor readings and timestamps and validate how many times and how long the sensor had a certain value?

I am processing some sensor data and my input looks something like this (the values are always 1 or 0, but the length of the values & timestamps arrays is a lot longer, as they contain the readings of a sensor for 24 hours & the data ingestion happens every second):
const input= {
values: [ 0, 1, 0, 0, 1 ],
timestamps: [
'2022-08-19T08:01:21.000Z',
'2022-08-19T08:01:22.000Z',
'2022-08-19T08:01:23.000Z',
'2022-08-19T08:01:24.000Z',
'2022-08-19T08:01:25.000Z'
]
}
or I could easily convert the input to the following format (I couldn't decide, which one would be more suitable):
const input= [{value: 0, timestamp: '2022-08-19T08:01:21.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:22.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:23.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:24.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:25.000Z'}]
My goal is to identify all the periods when the sensor reading was 0 and also validate if these periods were shorter or longer than 1 minute, i.e., for the case above, I'd like to get the following result:
result = [
{startDate: '2022-08-19T08:01:21.000Z', endDate= '2022-08-19T08:01:22.000Z', isShorterThanOneMin: true},
{startDate: '2022-08-19T08:01:23.000Z', endDate= '2022-08-19T08:01:25.000Z', isShorterThanOneMin: true}]
Could you advise a time-efficient way to solve this?

Simple JavaScript version
class Processor {
constructor(inputs) {
this.inputs = inputs;
this.outputs = [];
this.process();
}
process() {
while (this.inputs.length > 0) {
const input = this.inputs.shift();
if (this.previousInput === undefined) {
this.previousInput = input.value === 0 ? input : undefined;
continue;
}
if (this.previousInput.value === 0) {
if (input.value === 0)
continue;
this.outputs.push({
startDate: this.previousInput.date,
endDate: input.date,
isShorterThanOneMinute: (input.date.getTime() - this.previousInput.date.getTime()) < 60000
});
this.previousInput = undefined;
}
}
}
}
const inputs = [
{ value: 0, date: new Date("2022-08-19T08:01:21.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:22.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:23.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:24.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:25.000Z") }
];
const processor = new Processor(inputs);
console.log(processor.outputs);
Fancier, longer TypeScript version
interface Input {
value: number;
date: Date;
}
namespace Input {
export type List = Input[];
export const clone = (input: Input): Input => {
return {
value: input.value,
date: new Date(input.date.getTime())
}
}
}
interface Output {
startDate: Date;
endDate: Date;
isShorterThanOneMinute: boolean;
}
namespace Output {
export type List = Output[];
export const clone = (output: Output): Output => {
return {
startDate: new Date(output.startDate.getTime()),
endDate: new Date(output.endDate.getTime()),
isShorterThanOneMinute: output.isShorterThanOneMinute
}
}
}
class Processor {
private previousInput?: Input;
private outputs: Output.List = [];
private inputs: Input.List;
constructor(inputs: Input.List) {
this.inputs = inputs.map(Input.clone);
this.process();
}
private process() {
while (this.inputs.length > 0) {
const input = this.inputs.shift()!;
if (this.previousInput === undefined) {
this.previousInput = input.value === 0 ? input : undefined;
continue;
}
if (this.previousInput.value === 1) {
throw new Error(`This is not possible, because we never store an input with value = 1.`);
}
if (input.value === 0) continue;
this.outputs.push({
startDate: this.previousInput.date,
endDate: input.date,
isShorterThanOneMinute: (input.date.getTime() - this.previousInput.date.getTime()) < 60000
});
this.previousInput = undefined;
}
}
getOutputs(): Output.List {
return this.outputs.map(Output.clone);
}
append(input: Input): this {
this.inputs.push(Input.clone(input));
this.process();
return this;
}
}
const inputs: Input.List = [
{ value: 0, date: new Date("2022-08-19T08:01:21.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:22.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:23.000Z") },
{ value: 0, date: new Date("2022-08-19T08:01:24.000Z") },
{ value: 1, date: new Date("2022-08-19T08:01:25.000Z") }
];
const processor = new Processor(inputs);
console.log(processor.getOutputs());
// Continue using the instance as more entries because available...
processor.append({ value: 1, date: new Date("2022-08-19T08:02:25.000Z") });
processor.append({ value: 1, date: new Date("2022-08-19T08:03:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:04:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:05:25.000Z") });
processor.append({ value: 0, date: new Date("2022-08-19T08:06:25.000Z") });
processor.append({ value: 1, date: new Date("2022-08-19T08:07:25.000Z") });
console.log(processor.getOutputs());

EDIT - #focorner has the fastest code for it methodsName other.
Source: https://sourceb.in/dXDPkOliQr [Made few changes.]
#focorner's answer should be accepted answer as the difference is too high. Idk If I am doing something wrong, but the difference in unbelievable
Ignore -
The current fastest way would be the one I tested called methodTwo. I tested answers from here, and the answer from #focorner doesn't work, not sure why, Maybe due to timestamps.
Of the two methods that I wrote, first, one called Process is actually O(n) time complexity and it's slower than the second version of its. The method processTwo is the fastest method. Note that, these tests were performed on intel i9 with 128GB Ram with an input array length of 360_000.
Having the timestamps as number is the fastest way to do this as new Date().getTime() or Date.now() are going to make it slower.
Date.now() is faster than Date().getTime()
Also destructuring the objects like { value, timestamp } = input is going to increase the time. Destructured assignments are less computationally efficient than traditional.
I used a package called benchmark to test the methods.
processTwo works on very simple logic, that is it checks two elements in a single loop ie element on index i and i + 1.
Json Source: https://rentry.co/x9vtb [360000 is too huge for internet, so smaller bersion]
Source: https://sourceb.in/c9garaf4Ld
The seconds test took only the test data provided by the author and here processTwo is the fastest. I might be wrongly testing but I tried and found out processTwo faster.
I also saw that the output of the Author's approach is just { start: Date, end: Date } and doesn't actually calculate isShorterThanOneMin so I added it to make it even.
You should start storing the number timestamps in an array instead of ISO date or strings.
Source: https://sourceb.in/vqTq2dBMmT

I figured it out, not sure if it's the fastest way, but seems to work:
const input= [{value: 1, timestamp: '2022-08-19T08:01:21.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:22.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:23.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:24.000Z'},
{value: 0, timestamp: '2022-08-19T08:01:25.000Z'},
{value: 1, timestamp: '2022-08-19T08:01:29.000Z'}]
let result = input.reduce((finalArray, currentValue,index, arr) => {
if (currentValue.value === 0) {
let resultElement = {};
resultElement.start = currentValue.timestamp;
//find end Date
let nextToCheck = index;
let endFound = false;
do {
nextToCheck = nextToCheck + 1;
if (nextToCheck === arr.length) {
break;
}
if ((arr[nextToCheck].value) === 1) {
resultElement.end = arr[nextToCheck].timestamp;
endFound = true;
}
} while (!endFound)
// find out if previous one was 1 --> if it was 0, it should not be pushed to avoid having sub-arrays
if (index !== 0 && arr[index - 1 ].value !== 0) {
finalArray.push(resultElement)
}
if (index === 0) {
finalArray.push(resultElement)
}
}
return finalArray;
}, [])
console.log(result)

NodeJS: Get Count By Month-Year

I have some transactional data which looks as below:
[{UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z", …},
{UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z", …},
{UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z", …},
{UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z", …}]
My requirement it to get count by month-year, so that the output looks like below:
[{period: '2014-03', count:2}
{period: '2014-04', count:1},
{period: '2014-10', count:1}]
I'm doing this in Nodejs, and am just not able to work with the date to make this happen.
Can you please help?

You can use the code given below to group based on period year and month.
let array = [{ UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z" },
{ UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z" },
{ UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z" },
{ UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z" }]
function count(array) {
return array.reduce((total, elem) => {
let temp = elem.createdAt.split("-")
let groupKey = temp[0] + "-" + temp[1];
total[groupKey] ? total[groupKey] +=1: total[groupKey] = 1;
// total[groupKey] += 1;
return total
}, {})
}
console.log(count(array))
The output of code above will be
{ '2014-03': 2, '2014-04': 1, '2014-10': 1 }
Of course you can easily convert from JSON format to array format using code given below
function convertToArray(json_data) {
let result = [];
for (var i in json_data)
result.push({ period: i, count: json_data[i] });
return result;
}
The output will be
[ { period: '2014-03', count: 2 },
{ period: '2014-04', count: 1 },
{ period: '2014-10', count: 1 } ]

You can take the substring of date , take the unique and count the frequency of each date
const arr = [{UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z",},
{UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z"},
{UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z"},
{UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z"}]
//take substring and just grab unique date
let distict_dates = [...new Set(arr.map(a => a.createdAt.substring(0, 7)))];
//count each date frequency
let reduced = distict_dates.map(a => {
return {
userCount: arr.filter(a1 => a1.createdAt.startsWith(a)).length,
createdAt: a
}
}
)
console.log(reduced);

You need this method basically:
.reduce((acc = {}, i) => {
let period = i.createdAt.slice(0,7);
acc[period] = {period, count: acc[period] ? acc[period].count+1: 1}
return acc;
}, {})
let t = [{UserId: 19156, createdAt: "2014-03-01T18:30:00.000Z"},
{UserId: 19150, createdAt: "2014-03-09T18:30:00.000Z"},
{UserId: 18459, createdAt: "2014-04-09T18:30:00.000Z"},
{UserId: 19666, createdAt: "2014-10-24T07:12:05.000Z"}].reduce((acc = {}, i) => {
let period = i.createdAt.slice(0,7);
acc[period] = {period, count: acc[period] ? acc[period].count+1: 1}
return acc;
}, {})
console.log(t);
// if you need exact same result then do it like this
console.log(Object.values(t));

using aggregation:
db.collectionName.aggregate([
{
$project:{
period:{$dateToString:{format:"%Y-%m",date:"$createdAt"}}
}
},{
$group:{ _id : {period : "$period"},count:{$sum:1}}
},
{ $sort : { _id : 1 } }
]);

You can try this:
db.doc.aggregate([{ $group: {
_id:null,
period: {
$dateToString: { format: "%Y-%m", date: "$createdAt" } },
count: { $sum: 1 } },
]
).then()

If the createdAt is a string, convert string to date as follows
var t = Date.parse("2015-04-01T18:30:00.000Z");
var date = new Date(d)
Now get the year and month as follows
date.getMonth() // zero indexed
date.getFullYear()
Append them and form your required string format for createdAt

convert string to date in mongodb?

I am using find to get the result then I'm applying Array.prototype.filter in javascript to filter the dates in between 2 given date strings in format dd-mm-yyyy.
let query = {};
req.query.query && (query.name = new RegExp(req.query.query, 'i'));
req.query.gender && (query.gender = new RegExp(`^${req.query.gender}$`, 'i'));
Student.find(query, { roll: 1, name: 1, _id: 0, email: 1, DOB: 1, gender: 1 }, (err, students) => {
if (err) {
next(err);
}
else {
if (req.query.DOB_from && req.query.DOB_to) {
let from = toDate(req.query.DOB_from);
let to = toDate(req.query.DOB_to)
res.send(filterBetweenDates(students, from, to));
}
else {
res.send(students)
}
}
});
/**
* Returns time val in mil secs from a date string in Indian format i.e dd-mm-yyyy
* #param {string} dateStr
*/
function toDate(dateStr) {
let a = dateStr.split('-').map(Number);
let d = new Date(a[2], a[1] - 1, a[0]);
return d.getTime();
}
/**
* Filters the result which matches the date
* #param {Student1[]} students resultant students array
* #param {Number} from time val in millisecs(from)
* #param {Number} to time val in millisecs(from)
*/
function filterBetweenDates(students, from, to) {
return students.filter(({ DOB }) => {
return (from <= toDate(DOB)) && (to >= toDate(DOB));
});
}
The issue I'm facing is that the dates are in dd-mm-yyyy format saved as a string And also the input is in the same format as string. So, I'm first applying other queries and then filtering according to the dates. Is there a way I can do this in the query itself?
Edit:
Sample Collection
[
{
"name": "vib",
"roll": 413,
"email": "abc#example.com",
"DOB": "25-07-1997",
"gender": "Male"
}
{
"name":"abc",
"roll":123,
"email": "abc#xyz.com",
"DOB": "07-11-2000",
"gender": "Female"
}
]

You can use $dateFromString aggregation which change your string DOB to date and then you can $match the date and for changing user date format you can use moment libaray
db.collection.aggregate([
{
"$addFields": {
"date": {
"$dateFromString": {
"dateString": "$DOB"
}
}
}
},
{ "$match": { "date": { "$lte": date, "$gte": date }}}
])

How do I change date formatting in Javascript to enable sorting?

I am writing my dates of birth in the following manner:
4.02.1976 14:37
1.7.1990 11:35
10.10.1910 18:00
I wanted to sort it using something similar to this code (enabling sorting by birth):
var obj = [{"id":1,"dateOfBirth":"1.7.1990 11:35"},
{"id":4,"dateOfBirth":"4.02.1976 14:37"},{"id":2,"dateOfBirth":"28.10.1950
2:15"},{"id":3,"dateOfBirth":"03.01.1963 23:10"}]
obj.sort(function(a,b) { return new Date(a.dateOfBirth).getTime() - new
Date(b.dateOfBirth).getTime() } );
I am unsure if I need to reformat the dates of birth to achieve this.

Since you just have the year/month/day, it's pretty trivial to split up the dateOfBirth string, convert to a single number, and sort by that number, without any need to mess with Dates:
var obj = [{
"id": 1,
"dateOfBirth": "1.7.1990"
}, {
id: 2,
dateOfBirth: "28.10.1950"
}, {
"id": 4,
"dateOfBirth": "4.02.1976"
}];
function valueFromDOBString(str) {
const values = str.split('.').map(Number);
return values[0] + values[1] * 100 + values[2] * 10000;
}
const sortedObj = obj.sort((a, b) => {
return valueFromDOBString(b.dateOfBirth) - valueFromDOBString(a.dateOfBirth);
});
console.log(sortedObj);

A working version with optional time values.
var obj = [{
"id": 1,
"dateOfBirth": "1.7.1990"
},
{
"id": 4,
"dateOfBirth": "4.02.1976 14:37"
}, {
"id": 2,
"dateOfBirth": "28.10.1950 2:15"
}
];
console.log(
obj.sort(function(a, b) {
return parseDate(a.dateOfBirth) -
parseDate(b.dateOfBirth);
})
);
function parseDate(str) {
var tokens = str.split(/\D/);
while (tokens.length < 5)
tokens.push(0);
return new Date(tokens[2], tokens[1]-1, tokens[0], tokens[3]||0, tokens[4]||0);
}

Format Json, remove duplicates and count elements in javascript

I'm a little lost, I have a dataset in json format with timestamps and ids like this:
[{
"date":"2016-11-18 19:20:42","id_pa":"7"
},{
"date":"2016-11-18 19:04:55","id_pa":"5"
},{
"date":"2016-11-19 20:53:42","id_pa":"7"
},{
"date":"2016-11-19 20:53:43","id_pa":"7"
},{
"date":"2016-11-19 20:53:43","id_pa":"7"
},{
"date":"2016-11-20 20:49:42","id_pa":"7"
},{
"date":"2016-11-20 20:50:45","id_pa":"7"
},{
"date":"2016-11-20 20:50:46","id_pa":"7"
}]
And I want to build a json that displays the date and the number of IDs each day. The new Json would be like this:
[{
"date":"18-11-2016","num_pa":"2"
},{
"date":"19-11-2016","num_pa":"1"
},{
"date":"20-11-2016","num_pa":"1"
}]
I figured I had to do a .map to format the date so it shows dd-mm-yyyy, then a .filter to remove duplicates and finally a .reduce to count the diferent ids for every date. So far I've done only the .map procedure but I'm not sure how to do the next steps and either my solution is the best solution or not.
This is a piece of my code:
SwapSvc
.getUsage (vm.id_fi)
.then((data)=>{
//console.log(`lreceived data: `+ JSON.stringify(data) );
vm.fdata = data.map((elem) => {
//console.log(`date: ${elem.date}`);
//console.log(`id_pa: ${elem.id_pa}`);
var d = new Date (elem.date);
return{
date:d.getDate()+'-'+d.getMonth()+'-'+d.getFullYear()/*elem.date*/,
id_pa:elem.id_pa
}})
var temp = [];
vm.filteredData = vm.fdata.filter((elem, index) => {
if(temp.indexOf(elem.date)<0){
temp.push(elem);
return true;
}
else return false;
});
console.log(`data after parsing and ordering: `+ JSON.stringify(vm.filteredData) );
return data;
})
.catch((err)=>{
//error
console.log(`error, no response`);
throw err;
});
PS: I'm using angular 1.6 with ES6.
Thanks in advance
BRJ

You could use a hash table for the date and collect all id_pa for later count.
var data = [{ date: "2016-11-18 19:20:42", id_pa: "7" }, { date: "2016-11-18 19:04:55", id_pa: "5" }, { date: "2016-11-19 20:53:42", id_pa: "7" }, { date: "2016-11-19 20:53:43", id_pa: "7" }, { date: "2016-11-19 20:53:43", id_pa: "7" }, { date: "2016-11-20 20:49:42", id_pa: "7" }, { date: "2016-11-20 20:50:45", id_pa: "7" }, { date: "2016-11-20 20:50:46", id_pa: "7" }],
hash = Object.create(null),
result;
data.forEach(function (a) {
var date = a.date.slice(0, 10);
hash[date] = hash[date] || Object.create(null);
hash[date][a.id_pa] = true;
});
result = Object.keys(hash).map(date => ({ date, num_pa: Object.keys(hash[date]).length }));
console.log(result);
.as-console-wrapper { max-height: 100% !important; top: 0; }

You can just chain reduce onto your map call and use the ES6 findIndex method to see if the object currently exists within the array being created by the reduce function.
SwapSvc
.getUsage (vm.id_fi)
.then((data)=>{
//console.log(`lreceived data: `+ JSON.stringify(data) );
vm.fdata = data.map((elem) => {
//console.log(`date: ${elem.date}`);
//console.log(`id_pa: ${elem.id_pa}`);
var d = new Date (elem.date);
return{
date:d.getDate()+'-'+d.getMonth()+'-'+d.getFullYear()/*elem.date*/,
id_pa:elem.id_pa
}}).reduce((p, c, i) => {
var index = p.findIndex(x => x.date === c.date);
if (index !== -1) p[index].num_pa++;
else p.push({"date": c.date, "num_pa": 1})
return p;
}, [])
console.log(`data after parsing and ordering: `+ JSON.stringify(vm.fData) );
return data;
})