I'm trying to gather data from a MongoDB with Node JS to draw a graph later.
My goal is to collect all entries by hour of the day. In my collection there is a "created_at" field which stores a Date Object.
I'm trying to store the data in an array with 24 slots like this:
// padding the array for each hour of the day
var hours = new Array(23);
// defaulting the value for each hour to 0
for(var i=0; i<hours.length; i++){
hours[i] = 0;
}
db.collection.find({}, {"created_at": 1}, function(err, entry){
if (err){ doSomething(); }
else {
entry.forEach(function(item, index){
// get hour of the day from the Date object
h = item["created_at"].getHours();
h = parseInt(h);
// store hour of the day and count it up
hours[h]++;
console.log("#%s: %s", index, h);
});
}
});
console.log(hours);
Now when I log hours I get the array with the default values. ie [0, 0, 0, 0 ... 0]
I'm certain that the database has correct values as the console.log in the inner function gets correct data.
I suspect the problem is one of concurrency: the collection's .find method is asynchronous. So your console.log after the call to .find gets executed before the entry.forEach ever even executes. Try moving the console.log into the body of the else clause, after the forEach (which is synchronous), and you should see the result you're looking for.
Going forward, you'll have to employ promises or something to get the results you want.
Related
I have a data from an api request and tried to store it in a temporary array using foreach and filter the temporary array after the foreach is completed.... in my case, I added a counter to compare if the counter is equal the length of api data. However, I notice that it only worked when the api data is small, say, data.length = 15 but when it reaches 20+, somehow it ignores the counter.
here's my code:
var flag = 0;
dato.forEach((el) => {
temp.push({
id: el.id,
sex: el.sex,
name: el.name,
[self.momentFormat(el.date, 'YYYY-MM-DD').toString()]: ({
activity: el.activity,
status : el.status,
record: el.record
})
});
flag++;
});
setTimeout(() => {
//This is what I get when I pass a parameter with date from Apr 1 to Apr 15,
//But when I pass Apr 1 to Apr 20... None of the logs shows up even though the api data is not empty
console.log('dato length=', dato.length); //I get 8
console.log('temp length=', temp.length); //I get 13
console.log('flag length=', flag.length); //I get 8
},5000)
if(flag == dato.length) {
var result = temp.filter(function(v) {
return this[v.id]?
!Object.assign(this[v.id], v) : (this[v.id] = v)
}, {});
self.overtime_dtr_records = result;
}
I've tried setting a timeout but I dont think its the most optimal way. Aside from defeating the purpose of fast api request, I also noticed that some of the temporary array data are not included in the fitlering process.
UPDATE: I think comparing a counter to data length is correct.The problem is that when I log the length of api data, I get 45, but the loop only iterates 4 times
I want to make a cron job that deletes deeply nested objects in my realtime database that are older than 24 hours.
I have looped through and reached the deeply nested object, but I can't grab/target the value of "addedTime" in the object. How do I grab that value so I can run .remove on the parent? So far, it comes back as undefined or it throws an error.
.schedule("every 1 hours")
.onRun(context => {
const rootDatabaseRef = admin.database().ref("ghostData/");
return rootDatabaseRef.ref.once("value").then(function(snapshot) {
console.log("snap", snapshot.val());
snapshot.forEach(function(userSnapshot) {
let buckets = userSnapshot.val().buckets;
console.log("buckets", buckets);
buckets.forEach(function(bucket) {
let currentTimeYesterday = new Date(
new Date().getTime() - 24 * 60 * 60 * 1000
).getTime();
let addedTime = bucket.val().addedTime;
console.log("curr time", currentTimeYesterday);
console.log("addedTime", addedTime);
});
});
Here is the data in my realtime database as well as the logs from the serverless cloud functions:
I think you're having problems with looping, because when you do this "buckets.forEach(function(bucket)" --> bucket in your case is the first element of the list ,
and every element has a nested dictionary , so first you have to iterate the dictionary and for each key in the dictionary , you'll get another dictionary , and you've to grab
only the added-time value.
I know it's difficult to understand but I think it's happening because you're not looping correctly.
Try the code below or something similar.
buckets.forEach(function(bucket){
let currentTimeYesterday = new ......
bucket.forEach(function(dict){
Object.keys(dict).forEach(k => {
console.log(k, ':', dict[k].addedTime);
let addedTime = dict[k].addedTime;
});
....
}
....
}
I'd like all my Firebase content to load randomly every time you refresh, but I can't seem to get all my Firebase data into a dictionary where I can randomize them.
I have a global array and I'm trying to push my files in there and then iterate through them. But Javascript thinks the array is empty because the timing is off.
var randomStore = new Array;
function homeSetup() {
if(ref.toString() == featuredRef.toString()){
addFeaturedImages();
}
console.log('randomStore length is ' + randomStore.length);
}
function addFeaturedImages(){
featuredRef.on("child_added", function(snapshot) {
var doc = {
// 'name': snapshot.key, //name is the id
'artist': snapshot.val().artist,
'author': snapshot.val().author,
'projectTitle': snapshot.val().projectTitle,
'text': snapshot.val().text
};
randomStore.push(doc);
console.log('randomStore length HERE is ' + randomStore.length);
});
}
Considering how the code is typed, I would assume that the 'randomStore length HERE is' log would be typed first, but instead I get this:
randomStore length is 0
randomStore length HERE is 1
randomStore length HERE is 2
randomStore length HERE is 3
If I got my data into a different array, then I could manipulate it to sort randomly and stuff like that, but I can't seem to get it in there properly.
You mentioned the timing is off? What did you mean by that ?
Have you heard of Javascript Promise's ?
http://www.html5rocks.com/en/tutorials/es6/promises/
This post explains promises very good. Read this very carefully, because if you are working with firebase, you will be using promises on daily!
Instead of using featuredRef.on("child_added") use featuredRef.once('value'). This should get you the whole array at once. Attach a then listener where you continue with the work of homeSetup.
function homeSetup() {
var cb = function(randomStore) {
console.log('randomStore length is ' + randomStore.length);
//....
};
if(ref.toString() == featuredRef.toString()){
addFeaturedImages(cb);
} /* else if(...) {
addFooImages(cb)
}*/
}
function addFeaturedImages(cb){
featuredRef.once("value", function(snapshot) {
//TODO: transform elements of the array?
cb(snapshot.val());
});
}
Code untested, but I should get you started.
I have an array of data points, [date, value], like so...
data_points = [[1310279340, 1], [1310279340, 1]]
I need to create an array based on "milliseconds ago", with ten values for each second, or a value for every 100ms going backwards.
The new array's values will look like, [ms ago, value], like so...
ms_ago_points = [[0,3],[100,6],[200,7]]
So the last value in that array represents [200ms ago, value of 7].
I'm having a hard time figuring out how to get "value from Xms ago" based on date values in the first array. If needed I might be able to get the initial date/values in a format other then an array if it would be easier to poll for data.
Thanks for any guidance!!
edit: I'm looking to repeat the same value between dates for every 100ms tick that lands between them.
var now = Date.now(),
l = data_points.length,
ms_ago_points = [];
for ( var c = 0; c < l; c++ ) {
ms_ago_points.push([now - data_points[c][0],data_points[c][1]);
}
Maybe you need to sort the ms_ago_points afterwards, but this should give you your desired array.
However: I'd seriously suggest you not to work with multi-dimesional arrays but objects instead:
ms_ago_points= [{ms: 0, value: 3},{ms:100, value: 6}...]
*edit: This of course assumes that your data_points are already in 100ms steps, otherwise you'd have to implement an interpolation :)
I believe this meets your requirements:
function msFromDataPoints(nowTime, dataPoints) {
var d = dataPoints.length,
i = 0,
anchor = Math.floor(nowTime.getTime() / 100) * 100,
ms = anchor,
msArr = [];
while (d--) {
while (ms >= data_points[d][0] * 1000) {
msArr.push([anchor - ms, dataPoints[d][1]];
ms -= 100;
}
}
return msArr;
}
See it in a Js Fiddle.
Note: I find the data structures to be a little strange. An object for the input seems best:
[{date: 1310279340, value: 1}, {date: 1310279340, value: 1}]
And your output array doesn't need the milliseconds at all, as simply [1, 1, 5, 5, 3, 2 4] (or whatever) would be understood to mean millisecond indexes starting at 0 and increasing by 100. If absolutely required, you could use a sparse array:
result = [];
result[0] = 1;
result[100] = 1;
result[200] = 5; // and so on
I am trying to visualize a flickr dataset using protovis. I do understand the visualization part, but i have a question about accessing the data however. I was provided an example visualization and it accesses the data as folllowing:
var data = pv.range(250).map(function(row) {
return {
views: parseInt(Data.data(row, 2)), //refers to the 4 row and 2nd collumn in CSV
users: Data.data(row, 6),
date: Data.data(row, 8))), //more collumns excist but for now we only use these
};
});
As i understand a part of the data set is now stored in the variable data, namely views, users and date. Is this variable able to beaccessed like a dictionary?
What i am trying to do is checking whether there are date on which one user occurs more than 2 times. I thought of looping through the var data as follows:
dateUserDict {};
for (d=0; d < data.date.length; d++ ){
for (i=0; i < data.users.length; i++ ){
for (j=0; j < data.users.length; j++){
if (data.users[i] == data.users[j]){
userCounter++ //this should count the number of occurences of a specific user on a specific date
dateUserDict[data.date] = [data.user][userCounter]}
}
}
}
This does not seem to work. I am trying to store the events (the number of times a user occurs on a specific date) in a dictionary. If i get the dictionary as described i can easily visualise the whole thing. But it is this conversion from the first dict (data) to the second (dateUserDict) which bugs me!
Any help or a push is highly appreciated!
Thanks
jorrit
The function you provided will product a Javascript array of objects.
var data = pv.range(250).map(function(row) {
return {
views: parseInt(Data.data(row, 2)), //refers to the 4 row and 2nd collumn in CSV
users: Data.data(row, 6),
date: Data.data(row, 8))), //more collumns excist but for now we only use these
};
});
The result will look something like this:
var data = [ {views:10, users: 9, date: '09/13/1975'}, ... ]
So instead of using data.users.length, use data.length, and instead of data.users[i], you should be using data[i].users, etc.