I have a huge list of items about almost all the crops and these data is to be plotted using maps and charts. I would like to count the number of each crop, say how many times was cabbage planted. I use Firebase database to store the data and I retrieve it using this function below:
database = firebase.database()
var ref = database.ref('Planting-Calendar-Entries');
ref.on('value', gotData, errData);
function gotData(data){
console.log(data.val())
var veggie = data.val();
var keys = Object.keys(veggie);
console.log(keys);
let counter = 0
for (var i = 0; i < keys.length; i++){
var k = keys[i];
var Veg_planted = veggie[k].Veg_planted;
var coordinates = veggie[k].coordinates;
if (Veg_planted == 'Cabbage'){
counter++;
}
// vegAll = Veg_planted.count()
console.log(Veg_planted, coordinates)
}
console.log(counter)
}
function errData(err){
console.log('Error!');
console.log(err)
}
This data I retrieve it from the database where it gets updated whenever someone submits their planting information. The code I used above will only apply if my list is small, but I have a list of about 170 items and it would be hard to write code to count each crop individually using something like let counter = 0, counter++. Is there a way I could navigate around this?
I'm assuming data.val() returns an array, not an object, and you're misusing Object.keys() on an array instead of just looping over the array itself. If that's true, then it sounds like you want to group by the Veg_planted key and count the groupings:
const counts = Object.values(veggie).reduce((counts, { Veg_planted }) => ({
...counts,
[Veg_planted]: (counts[Veg_planted] || 0) + 1
}), {});
Usage:
const veggie = [{ Veg_planted: 'Cabbage' }, { Veg_planted: 'Cabbage' }, { Veg_planted: 'Corn' }];
// result of counts:
// {Cabbage: 2, Corn: 1}
Actually: the code to count the items is probably going to be the same, no matter how many items there are. The thing that is going to be a problem as you scale though is the amount of data that you have to retrieve that you're not displaying to the user.
Firebase does not support aggregation queries, and your approach only works for short lists of items. For a more scalable solution, you should store the actual count itself in the database too.
So:
Have a blaCount property for each bla that exists.
Increment/decrement the counter each time your write/remove a bla to/from the database.
Now you can read only the counters, instead of having to read the individual items.
Firestore would be better option. You can query based on the field value.
var plantingRef = db.collection("PlantingCalendarEntries");
var query = plantingRef.where("Veg_planted", "==", "Cabbage");
if you still want to stuck with realtime database.
Save Counters to database.
Or use cloud dunctions to count.
Related
First, I'm a newbie. No doubt, I've made some simple errors.
Using Node.js with MySQL Database, I'm building a basic web app that allows users to login. Once they've logged in they will be brought to their profile page and are displayed results of a quiz they've done in the form of a bar chart.
I want covert a row of mysql data into an array.
const mysql = require('mysql');
const dbconfig = require('/config/database');
const connection = mysql.createConnection(dbconfig.connection);
connection.query('USE ' + dbconfig.database);
// Create an array of scores for each category depedning on the user who's
// loggedin.
var category1scoreQuery =
"SELECT c1q1, c1q2, c1q3, c1q4, c1q5, c1q6, c1q7, c1q8
FROM nodejs_login.assessment_score
AS a JOIN users as u ON a.respondent_id = u.user_respondent_id
WHERE a.respondent_id = user.user_respondent_id;";
connection.connect(function(err){
if (err) throw err;
connection.query(category1scoreQuery, function(err, result, fields) {
if (err) throw err;
Object.keys(result).forEach(function(key){
var cat1Array = result[key];
// want to return array e.g. ["45/60", "60/60", "40/40","30/40","15/20",
// "30/40", "30/60", "20/40"];
console.log(cat1Array);
})
})
});
// I want to convert it to an array to parse the array of strings into
// totalUserScore over maxCategoryScore
var i;
var userCategoryScore1 = 0;
var maxCategoryScore = 0;
for(i=0; i < cat1Array.length;i++){
var splitScore = cat1Array[i].split("/");
console.log(splitScore);
myQuestionScore = parseInt(splitScore[0], 10);
userCategoryScore1 += myQuestionScore;
console.log(userCategoryScore);
maxQuestionScore = parseInt(splitScore[1]);
maxCategoryScore = maxCategoryScore + maxQuestionScore;
console.log(maxCategoryScore);
}
This is what I am actually getting which doesn't allow me to loop through.
RowDataPacket {
c1q1: '15/60',
c1q2: '15/60',
c1q3: '10/40',
c1q4: '10/40',
c1q5: '5/20',
c1q6: '10/40',
c1q7: '15/60',
c1q8: '10/40' }
This should work for you:
const RowDataPacket= {
c1q1: '15/60',
c1q2: '15/60',
c1q3: '10/40',
c1q4: '10/40',
c1q5: '5/20',
c1q6: '10/40',
c1q7: '15/60',
c1q8: '10/40' }
const values=Object.values(RowDataPacket);
console.log(values)
Reference[1st part] : https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Object/values
Description:
The Object.values() method returns an array of a given object's own enumerable property values, in the same order as that provided by a for...in loop (the difference being that a for-in loop enumerates properties in the prototype chain as well).
For the second part, to calculate the total scores:
//using the values array from first part
const scores=values.reduce((accum,value)=>{
const splitValues=value.split('/')
return {
score:accum.score + parseInt(splitValues[0]),
maxScore:accum.maxScore + parseInt(splitValues[1]),
}
},{score:0,maxScore:0})
console.log(scores)
Reference[2nd part]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce
Description:
The reduce() method executes a reducer function (that you provide) on each element of the array, resulting in a single output value.
This is fairly common problem when interacting with a database through javascript. To get what you want you can try using JSON library like this:
usersRows = JSON.parse(JSON.stringify(result));
Also this isn't exactly related to your question but it's something that made my life a lot easier when I was doing this: consider using the node Promisify module to transform your queries into promises (from here: https://www.npmjs.com/package/util-promisify). That way instead of having to use callbacks like you are doing your code would look something like this:
var results = await connection.query(category1scoreQuery);
//Process results here
Again this is only a suggestion but something that I found was very useful.
Cheers!
strucrure
enter image description here
I need to get data from the "Income" field, which is in each document in collection "values", and write them to array to calculate the sum of the elements of this array.
Do as follows if you want to sum up the values. You don't need an array.
var totalIncome = 0;
db.collection("values").get().then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
totalIncome += doc.data().Income;
});
console.log(totalIncome);
});
Be aware however that it will cost a document read for each document of the collection. If your values collection contains a lot of documents, you may use another strategy like updating the totalIncome on documents creation/deletion.
If you really need to populate an array, do as follows:
var totalIncomeArray = [];
db.collection("values").get().then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
totalIncomeArray.push(doc.data().Income);
});
//Do whatever you want with the array: it contains all the Income values
});
I have this database, which looks like this
so the first keys are user uid taken from auth, and then the username he/she provided and what did they score for each match are taken also..
I just wanted to get each user total points - for example Ray total points is 45 and Wood total points is 44 but after looking through for the docs all I was able to do was just for one user, I have to write each user name and the specific match for each line to get the value.. now think of how it will be if they are dozens of users? hmm a lot of lines..
here is the JSON
the javascript code
var query = firebase.database().ref();
query.once("value")
.then(function(snapshot) {
snapshot.forEach(function(childSnapshot) {
var key = childSnapshot.key;
var Data1 = childSnapshot.child("Ray/Match1/Points").val();
var Data2 = childSnapshot.child("Ray/Match2/Points").val();
console.log(Data1 + Data2);
});
})
which will let me display, Ray total points, but not for Wood obviously I have to repeat it and write it..
So how do i solve this?
I took a look at your problem and I think I have your solution, or at the very least a PATHWAY to your solution. Ok, first I'll explain the basic issue, then I'll attempt to provide you with some generic-ish code (I'll attempt to use some of the variables you used). And away we go!
Basically what I see is 2 steps...
STEP 1 - You need to use a "constructor function" that will create new user objects with their own name (and/or user ID) and their own set of properties.
With that line of thinking, you can have the constructor function include properties such as "user name", "match points 1", "match points 2" and then a function that console logs the summary of each name and their total points from match points 1 and 2.
STEP 2 - You need to put the constructor function inside of a loop that will go through the database looking for the specific properties you need to fill in the properties needed by the constructor function to spit out the info you're looking for.
So... and let's take a deep breath because that was a lot of words... let's try to code that. I'll use generic properties in a way that I think will make it easy for you to insert your own property/variable names.
var user = function(name, match1, match2){
this.name = name;
this.match1 = match1;
this.match2 = match2;
this.pointTotal = function(match1, match2) {
console.log(match1 + match2);};
this.summary = function(){
console.log(name + " has a total of " + pointTotal + "
points.");};
}
the "This" part of the code allows ANY user name to be used and not just specific ones.
Ok, so the code above takes care of the constructor function part of the issue. Now it doesn't matter how many users you need to create with unique names.
The next step is to create some kind of loop function that will go through the database and fill in the properties needed to create each user so that you can get the total points from EVERY user and not just one.
Again, I will use generic-ish property/variable names...
var key = childSnapshot.key;
while(i = 0; i < key.length + 1; i++) {
var user = function(name, match1, match2){
this.name = name;
this.match1 = match1;
this.match2 = match2;
this.pointTotal = function(match1, match2) {
console.log(match1 + match2);};
this.summary = function(){
console.log(name + " has a total of " + pointTotal + " points.");};
}
}
That is a whole lot of words and the code is a hybrid of generic property names/variables and of property names/variables used by you, but I'm certain that I am on the correct pathway.
I have a lot of confidence that if you used the code and EXPLANATION that I provided, that if you plug in your own variables you will get the solution that you need.
In closing I just want to say that I REALLY hope that helps and if it doesn't I'd like to help solve the problem one way or another because I need the practice. I work a job with weird hours and so if I don't answer right away I am likely at my job :(
Good luck and I hope I helped!
simply add total node to your db
|_Id
|_ $userId:
| |_ Ray
| | |_ Match1:24
| | |_ Match2:21
| |_ total:45
and then get user`s total
var query = firebase.database().ref();
query.once("value")
.then(function(snapshot) {
snapshot.forEach(function(childSnapshot) {
var total = childSnapshot.child("total").val();
console.log(total);
});
})
you can add the total node using cloud functions
Check out this implementation. No need for cloud function.
firebase().database().ref().on('value', function(snapshot) {
snapshot.forEach((user)=>{
user.forEach((matches)=> {
var total = 0;
matches.forEach((match)=> {
total += match.val().Points;
});
console.log(total);
});
});
})
If the key is the user's Id, why add yet another nested object with the user's name? Do you expect one user to have multiple usernames? That sounds weird and adds on complexity, as you have probably noticed. If you need to keep the user name somewhere in Firebase, it is recommended that you dedicate a user details section somewhere directly under the user Id key. Here is a JavaScript representation of the Firebase object structure:
{
a1230scfkls1240: {
userinfo: {
username: 'Joe'
},
matches: {
asflk12405: {
points: 123
},
isdf534853: {
points: 345
}
}
}
}
Now, getting to the total points seems a bit more straightforward, does it not? 😎
To help you without modifying your current database structure, all you need is to loop through all the userId+username+matches permutation in your database. Here is an example code to achieve just that, you do not need any special Firebase feature, just good old JavaScript for-of loop:
const query = firebase.database().ref();
query.once('value')
.then(snapshot => {
const points = {}
const users = snapshot.val()
for (const userId of Object.keys(users)) {
const userprofile = users[userId]
for (const username of Object.keys(userprofile)) {
const user = userprofile[username]
for (const matchId of Object.keys(user)) {
const match = user[matchId]
// Store the points per user, per profile, or per both, depending on your needs
points[username] = points[username] === undefined
? points[username] = match.points
: points[username] += match.points
}
}
}
})
I would like to store product information in a key, value array, with the key being the unique product url. Then I would also like to store the visit frequency of each of these products. I will store these objects as window.localStorage items, but that's not very important.
The thing I had in mind was two key value arrays:
//product information
prods["url"] = ["name:product_x,type:category_x,price:50"]
//product visits frequency
freq["url"] = [6]
Then I would like to sort these prods based on the frequency.
Is that possible?
Hope you guys can help! Thanks a lot
Well you seem to have made several strange choices for your data format/structure. But assuming the format of the "prod" is beyond your control but you can choose your data structure, here's one way to do it.
Rather than two objects both using url as a key and having one value field each I've made a single object still keyed on url but with the product and frequency information from each in a field.
Objects don't have any inherent order so rather than sorting the table object I sort the keys, your "url"s ordered by ascending frequency.
To show that it's sorted that way I print it out (not in the same format).
For descending frequency, change data[a].freq - data[b].freq to data[b].freq - data[a].freq
var data = {
"url": {
prod: "name:product_x,type:category_x,price:50",
freq: 6
},
"url2": {
prod: "name:product_y,type:category_y,price:25",
freq: 3
}
};
var sorted = Object.keys(data).sort((a, b) => data[a].freq - data[b].freq);
console.log(sorted.map(k => [data[k].freq, k, data[k].prod]));
There's more than one way to format the data, which would change the shape of the code here.
maybe something like this:
var prods = [
{url:1, val:[{name:'a',type:'x',price:60}]},
{url:2, val:[{name:'b',type:'x',price:30}]},
{url:3, val:[{name:'c',type:'x',price:50}]},
{url:4, val:[{name:'c',type:'x',price:20}]},
{url:5, val:[{name:'c',type:'x',price:10}]},
{url:6, val:[{name:'c',type:'x',price:40}]}
];
var freq = [
{url:1, freq:6},
{url:2, freq:3},
{url:3, freq:5},
{url:4, freq:2},
{url:5, freq:1},
{url:6, freq:4}
];
prods.sort(function (a, b) {
var aU = freq.filter(function(obj) {
return obj.url === a.url;
});
var bU = freq.filter(function(obj) {
return obj.url === b.url;
});
if (aU[0].freq > bU[0].freq) {
return 1;
}
if (aU[0].freq < bU[0].freq) {
return -1;
}
return 0;
});
I'm working with a large dataset that needs to be efficient with its Mongo queries. The application uses the Ford-Fulkerson algorithm to calculate recommendations and runs in polynomial time, so efficiency is extremely important. The syntax is ES6, but everything is basically the same.
This is an approximation of the data I'm working with. An array of items and one item being matched up against the other items:
let items = ["pen", "marker", "crayon", "pencil"];
let match = "sharpie";
Eventually, we will iterate over match and increase the weight of the pairing by 1. So, after going through the function, my ideal data looks like this:
{
sharpie: {
pen: 1,
marker: 1,
crayon: 1,
pencil: 1
}
}
To further elaborate, the value next to each key is the weight of that relationship, which is to say, the number of times those items have been paired together. What I would like to have happen is something like this:
// For each in the items array, check to see if the pairing already
// exists. If it does, increment. If it does not, create it.
_.each(items, function(item, i) {
Database.upsert({ match: { $exist: true }}, { match: { $inc: { item: 1 } } });
})
The problem, of course, is that Mongo does not allow bracket notation, nor does it allow for variable names as keys (match). The other problem, as I've learned, is that Mongo also has problems with deeply nested $inc operators ('The dollar ($) prefixed field \'$inc\' in \'3LhmpJMe9Es6r5HLs.$inc\' is not valid for storage.' }).
Is there anything I can do to make this in as few queries as possible? I'm open to suggestions.
EDIT
I attempted to create objects to pass into the Mongo query:
_.each(items, function(item, i) {
let selector = {};
selector[match] = {};
selector[match][item] = {};
let modifier = {};
modifier[match] = {};
modifier[match]["$inc"] = {};
modifier[match]["$inc"][item] = 1
Database.upsert(selector, modifier);
Unfortunately, it still doesn't work. The $inc breaks the query and it won't let me go more than 1 level deep to change anything.
Solution
This is the function I ended up implementing. It works like a charm! Thanks Matt.
_.each(items, function(item, i) {
let incMod = {$inc:{}};
let matchMod = {$inc:{}};
matchMod.$inc[match] = 1;
incMod.$inc[item] = 1;
Database.upsert({node: item}, matchMod);
Database.upsert({node: match}, incMod);
});
I think the trouble comes from your ER model. a sharpie isn't a standalone entity, a sharpie is an item. The relationship between 1 item and other items is such that 1 item has many items (1:M recursive) and each item-pairing has a weight.
Fully normalized, you'd have an items table & a weights table. The items table would have the items. The weights table would have something like item1, item2, weight (in doing so, you can have asymmetrical weighting, e.g. sharpie:pencil = 1, pencil:sharpie = .5, which is useful when calculating pushback in the FFA, but I don't think that applies in your case.
Great, now let's mongotize it.
When we say 1 item has many items, that "many" is probably not going to exceed a few thousand (think 16MB document cap). That means it's actually 1-to-few, which means we can nest the data, either using subdocs or fields.
So, let's check out that schema!
doc =
{
_id: "sharpie",
crayon: 1,
pencil: 1
}
What do we see? sharpie isn't a key, it's a value. This makes everything easy. We leave the items as fields. The reason we don't use an array of objects is because this is faster & cleaner (no need to iterate over the array to find the matching _id).
var match = "sharpie";
var items = ["pen", "marker", "crayon", "pencil"];
var incMod = {$inc:{}};
var matchMod = {$inc:{}};
matchMod.$inc[match] = 1;
for (var i = 0; i < items.length; i++) {
Collection.upsert({_id: items[i]}, matchMod);
incMod.$inc[items[i]] = 1;
}
Collection.upsert({_id: match}, incMod);
That's the easy part. The hard part is figuring out why you want to use an FFA for a suggestion engine :-P.