We Keep Coding

InfluxDB 2.0: Group Periods and Sum Up Duration

I have an InfluxDB (version 2), where an entry is written every second into my bucket, together with an identifier (uuid of the source) and a side value (some domain-specific measurement from 1 to 6). Instead of having a long list of such by-second logs:
2020-05-18T15:57:18 side=1
2020-05-18T15:57:19 side=1
2020-05-18T15:57:20 side=3
2020-05-18T15:57:21 side=3
2020-05-18T15:57:22 side=3
2020-05-18T15:57:23 side=2
2020-05-18T15:57:24 side=2
I'd like to condense those entries, so that I can calculate the duration, for how long side=x held true:
from 2020-05-18T15:57:18 to 2020-05-18T15:57:19 side=1 duration=2s
from 2020-05-18T15:57:20 to 2020-05-18T15:57:22 side=3 duration=3s
from 2020-05-18T15:57:23 to 2020-05-18T15:57:24 side=2 duration=2s
I also restrict the time period by a from/to range. This is my current query:
from(bucket: "mybucket")
|>range(start:2020-01-01T00:00:00.000Z, stop:2020-12-31T00:00:00.000Z
|>filter(fn:(r) => r.identifier == "3c583530-0152-4ed1-b15f-5bb0747e771e")
My approach is to read the raw data, and then iterate over it, detect changes of the side (something like current.side != last.side), and then report that as a new entry to my logbook. This approach is very inefficient, of course (JavaScript):
const data = fetchFromInfluxDB(from, to):
const terminator = { 'time': new Date(), 'identifier': '', 'side': -1 };
data.push(terminator); // make sure last item is also reported
const logbook = new Array();
let lastSide = -1;
let from = undefined;
for (let e of data) {
if (e.side != lastSide) {
if (from != undefined) {
let to = e.time;
let diff = Math.floor((new Date(to) - new Date(from)) / 1000);
if (diff > 0) {
logbook.push({'from': from, 'to': to, 'side': lastSide, 'seconds': diff});
}
}
lastSide = e.side;
from = e.time;
}
}
Is there a way to group and sum up that data using the InfluxDB query language?

Firebase dont write

I have a function that deletes matches from users if the match is not played after 15 minutes. if the user that create the match add coins to the match, the function retrieves that coins to the user profile.
here is the code
function deleteMatch(key,user,bet){
MatchesRef.child(key).remove();
UsersRef.child(user).once('value').then(userSnapshot => {
const credits = userSnapshot.val().credits || 0;
return UsersRef.child(user).child('credits').set(credits + parseInt(bet, 10));
})
}
The problem is if the function recieve in the parameters a user that create 2 matchs in the same minute, only add 1 coin.
User1 have 9 coins and user2 have 8 coins.
Example parameters: match1(key1, user1, 1), match2(key2,user2,1)
with this parameters the code work, and now user1 have 10 coins and user2 have 9.
But with differente paramenters
user1 have 9 coins, user2 have 8 coins.
Example parameters with problems:
match1(key1,user1,1), match2(key2,user2,1) match3(key3,user1,1).
With this the code read all the matches, but after the code the user1 have 10 coins, and user 2 have 9 coins.
The code is not giving 2 coins, 1 for every match that user1 create.
I think this is because firebase are trying to write to the user node 2 times at once. But no idea how to solve
The code that starts the action
module.exports = () =>
MatchesRef.once('value').then(snap => {
const now = new Date().getTime();
const records = [];
snap.forEach(row => {
const { date, hour, adversary1, bet } = row.val();
console.log("el adversario es "+ adversary1);
if (!date || !hour) return;
const [day, month] = date.split('/').map(p => parseInt(p, 10));
const [h, min] = hour.split(':').map(p => parseInt(p, 10));
const scheduledTime = (new Date((new Date()).getFullYear(), month - 1, day, h, min, 0, 0))
.getTime() + expireMinutes * 60 * 1000;
if (scheduledTime <= now) {
records.push({ key: row.key, user: adversary1, bet });
deleteMatch(row.key,adversary1,bet);
//records.map(({key,user,bet}) => deleteMatch({key,user, bet}))
}
});
return Promise.all(
// records.map(({key, user, bet}) => expireMatch(key, user, bet))
);
});

The problem was firebase try to write 2 times at the same location and to avoid this problem I implement transactions, who guarantee that firebase wait until the previous write was finished.
UsersRef.child(user).child('credits').transaction(function(currentCredits){
console.log(currentCredits);
return currentCredits + parseInt(bet, 10);
})

How to ask an API for many values (of the past 14 days) via iterative timestamp manipulation?

I want to calculate the RSI - (Relative Strength Index)(Last 14 Days) for the Bitcoin-Price. The necessary data for the past 14 days comes from an API. The result should be stored in a variable for further processing. I tried different methods, but the result wasn´t working. I think i don´t found the right way to solve this. I was not able to create a complete and working result, so I am here to ask my first question on Stack Overflow.
How can I ask the API for the course of the past 14 days via
iterative timestamp manipulation?
Timestamp now -1day/-2days.... (&ts=xxxxxxxxxxxx) - example: (ts=1452680400)
https://min-api.cryptocompare.com/data/pricehistorical?fsym=BTC&tsyms=USD&ts=1452680400
How can i put the values from the API in an array?:
var closePrices= {
var : 'text',
array: [BTCDay1, BTCDay2, BTCDay3, BTCDay4, BTCDay5, BTCDay6, BTCDay7, BTCDay8, BTCDay9, BTCDay10, BTCDay11, BTCDay12, BTCDay13, BTCDay14]
};
Then I want to put the array in this calculation formular:
public static double CalculateRsi(IEnumerable<double> closePrices)
{
var prices = closePrices as double[] ?? closePrices.ToArray();
double sumGain = 0;
double sumLoss = 0;
for (int i = 1; i < prices.Length; i++)
{
var difference = prices[i] - prices[i - 1];
if (difference >= 0)
{
sumGain += difference;
}
else
{
sumLoss -= difference;
}
}
if (sumGain == 0) return 0;
if (Math.Abs(sumLoss) < Tolerance) return 100;
var relativeStrength = sumGain / sumLoss;
return 100.0 - (100.0 / (1 + relativeStrength));
}

Several quite broad questions (it usually works better if you post some of the code you already tried to write yourself)... But fun ones to solve.
Assuming we can use es6 syntax, Promises and fetch (if you can't, look up how to polyfill & transpile).
Creating an array of timestamps
To get a timestamp for the current date, you write Date.now(). To change this time stamp to a day n days ago, we decrease it by the number of milliseconds in a day:
const timeStampForDaysAgo =
nrOfDays => Date.now() - 1000 * 60 * 60 * 24 * nrOfDays;
// e.g.: yesterday
const yesterday = timeStampForDaysAgo(1);
Now, if we fill an array of the integers 0...14, we can use map to create an array of timestamps!
const lastTwoWeeks = Array.from(Array(14), (_, i) => timeStampForDaysAgo(i))
Doing the requests
In modern browsers, you can use fetch to do requests. We'll need a list of URLs though, not just timestamps. To make the URLs, we again use map:
const urls = lastTwoWeeks.map(ts => `https://your-url.com?ts=${ts}`);
Now that we have URLs, we can create our requests (again, using map):
const btcRequests = urls.map(url => fetch(url).then(r => r.json()));
Calculating the result
We can't start calculating the result until all requests have finished. That's where Promise.all comes in:
Promise.all(btcRequests).then(calcRSI);
This makes sure we only call calcRSI until all requests have finished.
Because the API returns objects of { BTC: { USD: Number } }, we'll have to extract the numbers before we can do our math.
Promise
.all(btcRequests)
.then(responses => responses.map(obj => obj.BTC.USD))
Now that we have an array of numbers, we can finally call the calculate function you gave in C# code. Of course, you'll have to translate it to javascript first.
const calcRSI = (arrayOfValues) => /* do math, return value */
Promise
.all(btcRequests)
.then(responses => responses.map(obj => obj.BTC.USD))
.then(calcRSI)
.then(rsi => console.log("The RSI for the last 14 days, is:", rsi);
See the code in action in this fiddle (make sure to open the console)

Find last document of the day for the last 7 days

I am adding entries to a schema every hour in order to track growth over the course of days while maintaining a current score for the current day. Now I would like to be able to pull the most recent record for each day for the past week. The results would be 6 records at or around midnight for 6 days previous and the 7th being the latest for the current day.
Here is my schema:
var schema = new Schema({
aid: { type: Number }
, name: { type: String }
, score: { type: Number }
, createdAt: { type: Date, default: Date.now() }
})
Edit
I've tried using this static, but it pulls the exact same record 7 times
schema.statics.getLastWeek = function(name, fn) {
var oneday = 60 * 60 * 24
, now = Date.now()
, docs = []
for (var i = 1; i <= 7; i++) {
this.where('name', new RegExp(name, 'i'))
.where('createdAt')
.gte(now - (i * oneday))
.desc('createdAt')
.findOne(function(err,doc){
docs.push(doc)
})
}
}
If I were using SQL I would do a subquery selecting MAXDATE and join it to my main query in order to retrieve the results I want. Anyway to do this here?

Kristina Chodorow gives a detailed recipe for this exact task in her book MongoDB: The Definitive Guide:
Suppose we have a site that keeps track of stock prices. Every few
minutes from 10 a.m. to 4 p.m., it gets the latest price for a stock,
which it stores in MongoDB. Now, as part of a reporting application,
we want to find the closing price for the past 30 days. This can be
easily accomplished using group.
I'm not familiar with Mongoose, however I've tried to adapt her example to your case below. Note I changed the createdAt default property from a value to a function and added an extra field datestamp to your schema:
var oneday = 24 * 60 * 60;
var schema = new Schema({
aid: { type: Number }
, name: { type: String }
, score: { type: Number }
// default: is a function and called every time; not a one-time value!
, createdAt: { type: Date, default: Date.now }
// For grouping by day; documents created on same day should have same value
, datestamp: { type: Number
, default: function () { return Math.floor(Date.now() / oneday); }
}
});
schema.statics.getLastWeek = function(name, fn) {
var oneweekago = Date.now() - (7 * oneday);
ret = this.collection.group({
// Group by this key. One document per unique datestamp is returned.
key: "datestamp"
// Seed document for each group in result array.
, initial: { "createdAt": 0 }
// Update seed document if more recent document found.
, reduce: function(doc, prev) {
if (doc.createdAt > prev.createdAt) {
prev.createdAt = doc.createdAt;
prev.score = doc.score;
// Add other fields, if desired:
prev.name = doc.name;
}
// Process only documents created within past seven days
, condition: { "createdAt" : {"$gt": oneweekago} }
}});
return ret.retval;
// Note ret, the result of group() has other useful fields like:
// total "count" of documents,
// number of unique "keys",
// and "ok" is false if a problem occurred during group()
);

A solution is to use group() to groups records by day. It's fancy, slow and can be blocking (meaning nothing else can run at the same time), but if your record set isn't too huge it's pretty powerful.
Group: http://www.mongodb.org/display/DOCS/Aggregation#Aggregation-Group
As for mongoose, I'm not sure if it supports group() directly, but you can use the node-mongodb-native implementation, by doing something like this (pseudo-code, mostly):
schema.statics.getLastWeek = function(name, cb) {
var keys = {} // can't remember what this is for
var condition = {} // maybe restrict to last 7 days
var initial = {day1:[],day2:[],day3:[],day4:[],day5:[],day6:[],day7:[]}
var reduce = function(obj, prev) {
// prev is basically the same as initial (except with whatever is added)
var day = obj.date.slice(0,-10) // figure out day, however it works
prev["day" + day].push(obj) // create grouped arrays
// you could also do something here to sort by _id
// which is probably also going to get you the latest for that day
// and use it to replace the last item in the prev["day" + 1] array if
// it's > that the previous _id, which could simplify things later
}
this.collection.group(keys, condition, initial, reduce, function(err, results) {
// console.log(results)
var days = results // it may be a property, can't remember
var lastDays = {}
days.forEach(function(day) {
// sort each day array and grab last element
lastDays[day] = days[day].sort(function(a, b) {
return a.date - b.date // check sort syntax, you may need a diff sort function if it's a string
}).slice(-1) // i think that will give you the last one
})
cb(lastDays) // your stuff
})
}
Some more comparisons between groups and map reduce from my blog:
http://j-query.blogspot.com/2011/06/mongodb-performance-group-vs-find-vs.html
There are no docs about the group command in the native driver, so you'll have to peer through the source code here:
https://github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/collection.js
Also for sort, check check https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Array/sort for exact syntax
Edit: Better Idea!!!
Just have a special collection called "lastRequestOfDay" and make the _id the day.
Overwrite the value with each new request. It will be super easy to query and fast to write and will always have the last value written each day!

Add another property to the schema named dateAdded or something.
schema.statics.getLastWeek = function(name, fn) {
var oneday = 60 * 60 * 24
, now = Date.now()
, docs = []
for (var i = 0; i < 7; i++) {
this.where('name', new RegExp(name, 'i'))
.where('createdAt')
.lt(now - (i * oneday))
.gte(now - ((i + 1) * oneday))
.desc('createdAt')
.findOne(function(err,doc){
// might not always find one
docs.push(doc)
})
}
return fn(null, docs)
}

Try something like this:
schema.statics.getLastWeek = function(name, fn) {
var oneday = 60 * 60 * 24
, now = Date.now()
, docs = []
, that = this
function getDay(day){
that.where('name', new RegExp(name, 'i'))
.where('createdAt')
.gte(now - (day * oneday))
.desc('createdAt')
.findOne(function(err,doc){
docs.push(doc)
})
}
for (var i = 1; i <= 7; i++) {
getDay(i);
}
}

Nobody seems to be trying for "close to midnight". :) The issue I saw with the original code was that it checked for a time greater than or equal to x days ago... which will always return the most recent time. I'm confused as to why DeaDEnD's solution returns the same record 7 times, though. Also, you never called fn, but that's not really the biggest of your concerns, is it?
Try adding on .lt(now - (now % oneday) - (i - 1) * oneday) (assuming 0-indexed; if it's 1-indexed, change that to say i - 2)

Best way to generate unique ids client-side (with Javascript)

I need to generate unique ids in the browser. Currently, I'm using this:
Math.floor(Math.random() * 10000000000000001)
I'd like to use the current UNIX time ((new Date).getTime()), but I'm worried that if two clients generate ids at the exact same time, they wouldn't be unique.
Can I use the current UNIX time (I'd like to because that way ids would store more information)? If not, what's the best way to do this (maybe UNIX time + 2 random digits?)

you can create a GUID using the following links:
http://softwareas.com/guid0-a-javascript-guid-generator
Create GUID / UUID in JavaScript?
This will maximise your chance of "uniqueness."
Alternatively, if it is a secure page, you can concatenate the date/time with the username to prevent multiple simultaneous generated values.

https://github.com/uuidjs/uuid provides RFC compliant UUIDs based on either timestamp or random #'s. Single-file with no dependencies, supports timestamp or random #-based UUIDs, uses native APIs for crypto-quality random numbers if available, plus other goodies.

In modern browser you can use crypto:
var array = new Uint32Array(1);
window.crypto.getRandomValues(array);
console.log(array);

var c = 1;
function cuniq() {
var d = new Date(),
m = d.getMilliseconds() + "",
u = ++d + m + (++c === 10000 ? (c = 1) : c);
return u;
}

Here is my javascript code to generate guid. It does quick hex mapping and very efficient:
AuthenticationContext.prototype._guid = function () {
// RFC4122: The version 4 UUID is meant for generating UUIDs from truly-random or
// pseudo-random numbers.
// The algorithm is as follows:
// Set the two most significant bits (bits 6 and 7) of the
// clock_seq_hi_and_reserved to zero and one, respectively.
// Set the four most significant bits (bits 12 through 15) of the
// time_hi_and_version field to the 4-bit version number from
// Section 4.1.3. Version4
// Set all the other bits to randomly (or pseudo-randomly) chosen
// values.
// UUID = time-low "-" time-mid "-"time-high-and-version "-"clock-seq-reserved and low(2hexOctet)"-" node
// time-low = 4hexOctet
// time-mid = 2hexOctet
// time-high-and-version = 2hexOctet
// clock-seq-and-reserved = hexOctet:
// clock-seq-low = hexOctet
// node = 6hexOctet
// Format: xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx
// y could be 1000, 1001, 1010, 1011 since most significant two bits needs to be 10
// y values are 8, 9, A, B
var guidHolder = 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx';
var hex = '0123456789abcdef';
var r = 0;
var guidResponse = "";
for (var i = 0; i < 36; i++) {
if (guidHolder[i] !== '-' && guidHolder[i] !== '4') {
// each x and y needs to be random
r = Math.random() * 16 | 0;
}
if (guidHolder[i] === 'x') {
guidResponse += hex[r];
} else if (guidHolder[i] === 'y') {
// clock-seq-and-reserved first hex is filtered and remaining hex values are random
r &= 0x3; // bit and with 0011 to set pos 2 to zero ?0??
r |= 0x8; // set pos 3 to 1 as 1???
guidResponse += hex[r];
} else {
guidResponse += guidHolder[i];
}
}
return guidResponse;
};

You can always run a test against existing IDs in the set to accept or reject the generated random number recursively.
for example:
const randomID = function(){
let id = Math.floor(Math.random() * 10000000000000001) + new Date();
if (idObjectArray.contains(id)) {
randomID;
} else {
idObjectArray.push(id);
}
};
This example assumes you would just be pushing the id into a 1D array, but you get the idea. There shouldn't be many collisions given the uniqueness of the random number with the date, so it should be efficient.

There are two ways to achieve this
js const id = Date.now().toString()
While this does not guarantee uniqueness (When you are creating multiple objects within 1ms), this will work on a practical level, since it is usually not long before the objects on the client are sent to a real server.
If you wanted to create multiple records withing 1ms, I suggest using the code below
const { randomBytes } = require("crypto");
// 32 Characters
const id = randomBytes(16).toString("hex");
It works similar to a uuid4 without needing to add an external library (Assuming you have access to NodeJs at some point)