We Keep Coding

InfluxDB 2.0: Group Periods and Sum Up Duration

I have an InfluxDB (version 2), where an entry is written every second into my bucket, together with an identifier (uuid of the source) and a side value (some domain-specific measurement from 1 to 6). Instead of having a long list of such by-second logs:
2020-05-18T15:57:18 side=1
2020-05-18T15:57:19 side=1
2020-05-18T15:57:20 side=3
2020-05-18T15:57:21 side=3
2020-05-18T15:57:22 side=3
2020-05-18T15:57:23 side=2
2020-05-18T15:57:24 side=2
I'd like to condense those entries, so that I can calculate the duration, for how long side=x held true:
from 2020-05-18T15:57:18 to 2020-05-18T15:57:19 side=1 duration=2s
from 2020-05-18T15:57:20 to 2020-05-18T15:57:22 side=3 duration=3s
from 2020-05-18T15:57:23 to 2020-05-18T15:57:24 side=2 duration=2s
I also restrict the time period by a from/to range. This is my current query:
from(bucket: "mybucket")
|>range(start:2020-01-01T00:00:00.000Z, stop:2020-12-31T00:00:00.000Z
|>filter(fn:(r) => r.identifier == "3c583530-0152-4ed1-b15f-5bb0747e771e")
My approach is to read the raw data, and then iterate over it, detect changes of the side (something like current.side != last.side), and then report that as a new entry to my logbook. This approach is very inefficient, of course (JavaScript):
const data = fetchFromInfluxDB(from, to):
const terminator = { 'time': new Date(), 'identifier': '', 'side': -1 };
data.push(terminator); // make sure last item is also reported
const logbook = new Array();
let lastSide = -1;
let from = undefined;
for (let e of data) {
if (e.side != lastSide) {
if (from != undefined) {
let to = e.time;
let diff = Math.floor((new Date(to) - new Date(from)) / 1000);
if (diff > 0) {
logbook.push({'from': from, 'to': to, 'side': lastSide, 'seconds': diff});
}
}
lastSide = e.side;
from = e.time;
}
}
Is there a way to group and sum up that data using the InfluxDB query language?

Combine array of dates sorted in ascending order into array of date ranges

I have one array of dates, I want to create object containing start and end by checking continue dates.
EX.
dateArray = [
"2020-01-22T00:00:00.000Z",
"2020-01-23T00:00:00.000Z",
"2020-01-28T00:00:00.000Z",
"2020-01-29T00:00:00.000Z",
"2020-01-30T00:00:00.000Z",
"2020-01-31T00:00:00.000Z",
"2020-02-01T00:00:00.000Z",
"2020-02-02T00:00:00.000Z",
"2020-02-03T00:00:00.000Z",
"2020-02-04T00:00:00.000Z",
"2020-02-05T00:00:00.000Z",
"2020-02-06T00:00:00.000Z",
"2020-02-07T00:00:00.000Z",
"2020-02-16T00:00:00.000Z",
"2020-02-17T00:00:00.000Z",
"2020-02-18T00:00:00.000Z",
"2020-02-19T00:00:00.000Z",
"2020-02-20T00:00:00.000Z"
]
myRequirement = [{
start: "2020-01-22T00:00:00.000Z",
end: "2020-01-23T00:00:00.000Z"
},
{
start: "2020-01-28T00:00:00.000Z",
end: "2020-02-07T00:00:00.000Z"
},
{
start: "2020-02-16T00:00:00.000Z",
end: "2020-02-20T00:00:00.000Z"
}
]
I want to do this using in node.js.
I tried this using some nested for loops.
First i am running loop on main dateArray, Then checking is it first date or not, If it is first date then storing it as first objects start date, Then in next date case checking is it next most date of previous date or not.
let gapArray = [];
let startEndObj = {};
let tempStartDate;
let tempEndDate;
let tempNextDate;
await asyncForEach(finalAvailablityDatesArrayOFi.availeblityDatesArray, async (availeblityDatesArrayOFi) => {
console.log("availeblityDatesArrayOFi", availeblityDatesArrayOFi);
if (!tempStartDate) {
console.log("In if");
startEndObj.startDate = availeblityDatesArrayOFi;
tempStartDate = availeblityDatesArrayOFi;
let oneDatePlus = new Date(availeblityDatesArrayOFi).setDate(new Date(availeblityDatesArrayOFi).getDate() + 1);
tempNextDate = new Date(oneDatePlus);
console.log("startEndObj", startEndObj);
}
else if (tempStartDate) {
console.log("in else");
if (new Date(availeblityDatesArrayOFi).getTime() == new Date(tempNextDate).getTime()) {
console.log("Do nothing!");
tempStartDate = availeblityDatesArrayOFi;
tempEndDate = availeblityDatesArrayOFi;
let oneDatePlus = new Date(availeblityDatesArrayOFi).setDate(new Date(availeblityDatesArrayOFi).getDate() + 1);
tempNextDate = new Date(oneDatePlus);
}
else {
startEndObj.endDate = new Date(tempEndDate);
gapArray.push(startEndObj);
tempStartDate = '';
tempEndDate = '';
startEndObj = {};
}
}
});
Thank you!

Looks like a job for Array.prototype.reduce().
Note: hereafter assumption is made that few prerequisites are met:
source array items are valid ISO-formatted date strings or others, parseable by new Date() constructor, otherwise should be brought to one of supported format
source array items are sorted in ascending order, otherwise Array.prototype.sort() method must be applied in advance
array items do not include time of the day part (or this part is exactly the same for all items), otherwise consecutive date records may happen to have difference greater than 864e5 milliseconds (1 day) and more complex comparison is required
You may walk through your array and compare current items with previous/following, once you have a gap greater than 1 day you push new range into resulting array or modify end date for the last one:
const src = ["2020-01-22T00:00:00.000Z","2020-01-23T00:00:00.000Z","2020-01-28T00:00:00.000Z","2020-01-29T00:00:00.000Z","2020-01-30T00:00:00.000Z","2020-01-31T00:00:00.000Z","2020-02-01T00:00:00.000Z","2020-02-02T00:00:00.000Z","2020-02-03T00:00:00.000Z","2020-02-04T00:00:00.000Z","2020-02-05T00:00:00.000Z","2020-02-06T00:00:00.000Z","2020-02-07T00:00:00.000Z","2020-02-16T00:00:00.000Z","2020-02-17T00:00:00.000Z","2020-02-18T00:00:00.000Z","2020-02-19T00:00:00.000Z","2020-02-20T00:00:00.000Z"],
ranges = src.reduce((res,date,idx,self) => {
const rangeStart = !idx || new Date(date) - new Date(self[idx-1]) > 864e5,
rangeEnd = idx == self.length-1 || new Date(self[idx+1]) - new Date(date) > 864e5
if(rangeStart) res.push({startdate:date,enddate:date})
else if(rangeEnd) res[res.length-1]['enddate'] = date
return res
}, [])
console.log(ranges)
.as-console-wrapper {min-height:100%}

You need to be careful with this type of processing to determine all the business rules exactly. If the time component is not to be considered, then it should be removed, otherwise when comparing say 2020-01-01T00:00:00 to 2020-01-02T012:00:00 you will get a difference greater than 1 day but might not want it to be treated as the start of a new range.
For that reason, the "days difference" logic should be in a separate function, which also makes it easier to change date libraries if you're using one. The days difference is also signed, so make sure they are passed in the right order.
Otherwise, the following is pretty much the same as Yevgen's answer but a little more efficient I think as it only creates two Dates on each iteration instead of four.
let dateArray = [
"2020-01-22T00:00:00.000Z",
"2020-01-23T00:00:00.000Z",
"2020-01-28T00:00:00.000Z",
"2020-01-29T00:00:00.000Z",
"2020-01-30T00:00:00.000Z",
"2020-01-31T00:00:00.000Z",
"2020-02-01T00:00:00.000Z",
"2020-02-02T00:00:00.000Z",
"2020-02-03T00:00:00.000Z",
"2020-02-04T00:00:00.000Z",
"2020-02-05T00:00:00.000Z",
"2020-02-06T00:00:00.000Z",
"2020-02-07T00:00:00.000Z",
"2020-02-16T00:00:00.000Z",
"2020-02-17T00:00:00.000Z",
"2020-02-18T00:00:00.000Z",
"2020-02-19T00:00:00.000Z",
"2020-02-20T00:00:00.000Z"
];
// Simple difference in days function
function daysDiff(d0, d1) {
return Math.round((d1 - d0) / 8.64e7);
}
let ranges = dateArray.reduce((acc, curr, i, arr) => {
// If first date, initialise first object
if (!acc.length) {
acc.push({start: curr, end: curr});
} else {
let d0 = new Date(curr);
let d1 = new Date(arr[i-1]);
// If difference greater than 1 day, end previous range
// and start a new range
if (daysDiff(d1, d0) > 1) {
acc[acc.length - 1].end = arr[i-1];
acc.push({start: curr, end: curr});
}
}
return acc;
}, []);
console.log(ranges);

How do I loop through objects and categorize by timestamps in Javascript?

I have an array of objects that have a keys called timestamp and motion. motion contains a value and timestamp contains a unix timestamp. I want to iterate over a number of the objects and find what "time of day" period they correspond to, I then want to total up the motion values for that given time of day and save the entire thing in an array of arrays. I want the duration to be changeable.
Let's say these are my objects;
{
timestamp: 1397160634,
motion: 2,
id: '534771d8c311731e21c75c9f'
},
{
timestamp: 1397160634,
motion: 3,
id: '534771d8c311731e21c75c9f'
}
Now I create my results array
var sampleDuration = 60; // Min
var minutesInDay = 1440;
var samplesPerDay = minutesInDay/sampleDuration;
var finalResultItem = []
for (var i = 0; i < samplesPerDay; i++) {
var IndividualresultArray = []
IndividualresultArray.push(60*i);
IndividualresultArray.push(0);
finalResultItem.push(IndividualresultArray);
}
I now have an array of arrays with each subarray's first item being a number (corresponding to a minute stamp) and the second value being zero.
I would now like to loop through all my objects and increment the second value (motion) based on the time of day range that is in the timestamp
_forEach(objects, function (object) {
{
// grab the timestamp
// figure out which minute range it coresponds to
// increment the array value that corresponds to the minute stamp
// rinse and repeat
}
this is where I go blank, I need the end result to look something like this
[[30, 5],[60, 20],[90, 5],[120, 0] .........]
or it could even look like this
[[000002400, 5],[000003000, 20],[000003600, 5],[000004200, 0] .........]
where the first value is a timestamp that ignores the year, month, and day, and only considers the time of day.
I have considered using moment.js in some capacity but I'm not sure how. Any help with this problem would be great.

I created a jsFiddle for you. The motion increment logic should look like (I'm using jQuery here but you get the point)
// Loop through and increment motion
$.each(objs, function (idx, obj) {
var date = new Date(obj.timestamp * 1000); // Convert to milliseconds
var minutesInDay = date.getUTCHours() * 60 + date.getUTCMinutes(); // Remove UTC for local time!
var minuteRange = Math.floor(minutesInDay / sampleDuration);
finalResultItem[minuteRange][1] += obj.motion;
});
EDIT: Removed some discussion after your edit. I also used more generic logic based on sampleDuration.

This should do it:
_forEach(objects, function (object) {
var date = new Date(objec.timestamp*1000);
var minuteOfDay = date.getUTCHours()*60+date.getUTCMinutes();
finalResultItem[minuteOfDay][1] += object.motion;
})
For a variable sample rate, employ a secondOfDay and divide that by your sampleDuration, then floor it to get your array index.

Find last document of the day for the last 7 days

I am adding entries to a schema every hour in order to track growth over the course of days while maintaining a current score for the current day. Now I would like to be able to pull the most recent record for each day for the past week. The results would be 6 records at or around midnight for 6 days previous and the 7th being the latest for the current day.
Here is my schema:
var schema = new Schema({
aid: { type: Number }
, name: { type: String }
, score: { type: Number }
, createdAt: { type: Date, default: Date.now() }
})
Edit
I've tried using this static, but it pulls the exact same record 7 times
schema.statics.getLastWeek = function(name, fn) {
var oneday = 60 * 60 * 24
, now = Date.now()
, docs = []
for (var i = 1; i <= 7; i++) {
this.where('name', new RegExp(name, 'i'))
.where('createdAt')
.gte(now - (i * oneday))
.desc('createdAt')
.findOne(function(err,doc){
docs.push(doc)
})
}
}
If I were using SQL I would do a subquery selecting MAXDATE and join it to my main query in order to retrieve the results I want. Anyway to do this here?

Kristina Chodorow gives a detailed recipe for this exact task in her book MongoDB: The Definitive Guide:
Suppose we have a site that keeps track of stock prices. Every few
minutes from 10 a.m. to 4 p.m., it gets the latest price for a stock,
which it stores in MongoDB. Now, as part of a reporting application,
we want to find the closing price for the past 30 days. This can be
easily accomplished using group.
I'm not familiar with Mongoose, however I've tried to adapt her example to your case below. Note I changed the createdAt default property from a value to a function and added an extra field datestamp to your schema:
var oneday = 24 * 60 * 60;
var schema = new Schema({
aid: { type: Number }
, name: { type: String }
, score: { type: Number }
// default: is a function and called every time; not a one-time value!
, createdAt: { type: Date, default: Date.now }
// For grouping by day; documents created on same day should have same value
, datestamp: { type: Number
, default: function () { return Math.floor(Date.now() / oneday); }
}
});
schema.statics.getLastWeek = function(name, fn) {
var oneweekago = Date.now() - (7 * oneday);
ret = this.collection.group({
// Group by this key. One document per unique datestamp is returned.
key: "datestamp"
// Seed document for each group in result array.
, initial: { "createdAt": 0 }
// Update seed document if more recent document found.
, reduce: function(doc, prev) {
if (doc.createdAt > prev.createdAt) {
prev.createdAt = doc.createdAt;
prev.score = doc.score;
// Add other fields, if desired:
prev.name = doc.name;
}
// Process only documents created within past seven days
, condition: { "createdAt" : {"$gt": oneweekago} }
}});
return ret.retval;
// Note ret, the result of group() has other useful fields like:
// total "count" of documents,
// number of unique "keys",
// and "ok" is false if a problem occurred during group()
);

A solution is to use group() to groups records by day. It's fancy, slow and can be blocking (meaning nothing else can run at the same time), but if your record set isn't too huge it's pretty powerful.
Group: http://www.mongodb.org/display/DOCS/Aggregation#Aggregation-Group
As for mongoose, I'm not sure if it supports group() directly, but you can use the node-mongodb-native implementation, by doing something like this (pseudo-code, mostly):
schema.statics.getLastWeek = function(name, cb) {
var keys = {} // can't remember what this is for
var condition = {} // maybe restrict to last 7 days
var initial = {day1:[],day2:[],day3:[],day4:[],day5:[],day6:[],day7:[]}
var reduce = function(obj, prev) {
// prev is basically the same as initial (except with whatever is added)
var day = obj.date.slice(0,-10) // figure out day, however it works
prev["day" + day].push(obj) // create grouped arrays
// you could also do something here to sort by _id
// which is probably also going to get you the latest for that day
// and use it to replace the last item in the prev["day" + 1] array if
// it's > that the previous _id, which could simplify things later
}
this.collection.group(keys, condition, initial, reduce, function(err, results) {
// console.log(results)
var days = results // it may be a property, can't remember
var lastDays = {}
days.forEach(function(day) {
// sort each day array and grab last element
lastDays[day] = days[day].sort(function(a, b) {
return a.date - b.date // check sort syntax, you may need a diff sort function if it's a string
}).slice(-1) // i think that will give you the last one
})
cb(lastDays) // your stuff
})
}
Some more comparisons between groups and map reduce from my blog:
http://j-query.blogspot.com/2011/06/mongodb-performance-group-vs-find-vs.html
There are no docs about the group command in the native driver, so you'll have to peer through the source code here:
https://github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/collection.js
Also for sort, check check https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Array/sort for exact syntax
Edit: Better Idea!!!
Just have a special collection called "lastRequestOfDay" and make the _id the day.
Overwrite the value with each new request. It will be super easy to query and fast to write and will always have the last value written each day!

Add another property to the schema named dateAdded or something.
schema.statics.getLastWeek = function(name, fn) {
var oneday = 60 * 60 * 24
, now = Date.now()
, docs = []
for (var i = 0; i < 7; i++) {
this.where('name', new RegExp(name, 'i'))
.where('createdAt')
.lt(now - (i * oneday))
.gte(now - ((i + 1) * oneday))
.desc('createdAt')
.findOne(function(err,doc){
// might not always find one
docs.push(doc)
})
}
return fn(null, docs)
}

Try something like this:
schema.statics.getLastWeek = function(name, fn) {
var oneday = 60 * 60 * 24
, now = Date.now()
, docs = []
, that = this
function getDay(day){
that.where('name', new RegExp(name, 'i'))
.where('createdAt')
.gte(now - (day * oneday))
.desc('createdAt')
.findOne(function(err,doc){
docs.push(doc)
})
}
for (var i = 1; i <= 7; i++) {
getDay(i);
}
}

Nobody seems to be trying for "close to midnight". :) The issue I saw with the original code was that it checked for a time greater than or equal to x days ago... which will always return the most recent time. I'm confused as to why DeaDEnD's solution returns the same record 7 times, though. Also, you never called fn, but that's not really the biggest of your concerns, is it?
Try adding on .lt(now - (now % oneday) - (i - 1) * oneday) (assuming 0-indexed; if it's 1-indexed, change that to say i - 2)

Best way to generate unique ids client-side (with Javascript)

I need to generate unique ids in the browser. Currently, I'm using this:
Math.floor(Math.random() * 10000000000000001)
I'd like to use the current UNIX time ((new Date).getTime()), but I'm worried that if two clients generate ids at the exact same time, they wouldn't be unique.
Can I use the current UNIX time (I'd like to because that way ids would store more information)? If not, what's the best way to do this (maybe UNIX time + 2 random digits?)

you can create a GUID using the following links:
http://softwareas.com/guid0-a-javascript-guid-generator
Create GUID / UUID in JavaScript?
This will maximise your chance of "uniqueness."
Alternatively, if it is a secure page, you can concatenate the date/time with the username to prevent multiple simultaneous generated values.

https://github.com/uuidjs/uuid provides RFC compliant UUIDs based on either timestamp or random #'s. Single-file with no dependencies, supports timestamp or random #-based UUIDs, uses native APIs for crypto-quality random numbers if available, plus other goodies.

In modern browser you can use crypto:
var array = new Uint32Array(1);
window.crypto.getRandomValues(array);
console.log(array);

var c = 1;
function cuniq() {
var d = new Date(),
m = d.getMilliseconds() + "",
u = ++d + m + (++c === 10000 ? (c = 1) : c);
return u;
}

Here is my javascript code to generate guid. It does quick hex mapping and very efficient:
AuthenticationContext.prototype._guid = function () {
// RFC4122: The version 4 UUID is meant for generating UUIDs from truly-random or
// pseudo-random numbers.
// The algorithm is as follows:
// Set the two most significant bits (bits 6 and 7) of the
// clock_seq_hi_and_reserved to zero and one, respectively.
// Set the four most significant bits (bits 12 through 15) of the
// time_hi_and_version field to the 4-bit version number from
// Section 4.1.3. Version4
// Set all the other bits to randomly (or pseudo-randomly) chosen
// values.
// UUID = time-low "-" time-mid "-"time-high-and-version "-"clock-seq-reserved and low(2hexOctet)"-" node
// time-low = 4hexOctet
// time-mid = 2hexOctet
// time-high-and-version = 2hexOctet
// clock-seq-and-reserved = hexOctet:
// clock-seq-low = hexOctet
// node = 6hexOctet
// Format: xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx
// y could be 1000, 1001, 1010, 1011 since most significant two bits needs to be 10
// y values are 8, 9, A, B
var guidHolder = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx';
var hex = '0123456789abcdef';
var r = 0;
var guidResponse = "";
for (var i = 0; i < 36; i++) {
if (guidHolder[i] !== '-' && guidHolder[i] !== '4') {
// each x and y needs to be random
r = Math.random() * 16 | 0;
}
if (guidHolder[i] === 'x') {
guidResponse += hex[r];
} else if (guidHolder[i] === 'y') {
// clock-seq-and-reserved first hex is filtered and remaining hex values are random
r &= 0x3; // bit and with 0011 to set pos 2 to zero ?0??
r |= 0x8; // set pos 3 to 1 as 1???
guidResponse += hex[r];
} else {
guidResponse += guidHolder[i];
}
}
return guidResponse;
};

You can always run a test against existing IDs in the set to accept or reject the generated random number recursively.
for example:
const randomID = function(){
let id = Math.floor(Math.random() * 10000000000000001) + new Date();
if (idObjectArray.contains(id)) {
randomID;
} else {
idObjectArray.push(id);
}
};
This example assumes you would just be pushing the id into a 1D array, but you get the idea. There shouldn't be many collisions given the uniqueness of the random number with the date, so it should be efficient.

There are two ways to achieve this
js const id = Date.now().toString()
While this does not guarantee uniqueness (When you are creating multiple objects within 1ms), this will work on a practical level, since it is usually not long before the objects on the client are sent to a real server.
If you wanted to create multiple records withing 1ms, I suggest using the code below
const { randomBytes } = require("crypto");
// 32 Characters
const id = randomBytes(16).toString("hex");
It works similar to a uuid4 without needing to add an external library (Assuming you have access to NodeJs at some point)