I'm working with EmberJS and SailsJS. Now I've been asked to make a statistics page and handle the filtering process in the SailsJS.
I have a model for departments and another model for requests the relationship between these models is (request belongsTo department). For some reason my manager prevented me to make a (hasMany) relationship.
Now what I want to do is to loop through all the departments and store them in new Object, inside that loop I want to loop through all the requests by Using Request.count({where : { department : department.id }}) and get the number of requests for each department in the departments Array Of Object.
I tried to do it as I explained but the problem is when I log the department inside the request loop it gives me the result as I imagined but when I add (.id) it shows me 'undefined'.
Here is my code:
Department.find().then((departments) => {
report.departments = departments;
Request.count({ department : departments.id}).exec(function countMe(err, count) {
console.log(count);
})
})
PS: if there's any other approach for this task please tell me, I'm kind of a beginner.
Your departments object in the .then callback is an array of department objects. To get an id you'd need to do something like departments[0].id.
I might not recommend using .count because that means a separate trip to the database for each department. Sails waterline may have some way for you to count up Requests grouped by department id, but just to get a steamroller working example, I would first just get everything and do some processing in your code:
Department.find().then((departments) => {
Request.find().then((requests) => {
var requestCounts = {}; // we'll store the counts in this object
for (var idx = 0; idx < requests.length; idx++) {
if (!requestCounts[requests[idx].department]) {
requestCounts[requests[idx].department] = 0;
}
requestCounts[requests[idx].department]++;
}
// use requestCounts...
});
});
Creating a separate object like that may not be what you want to do, but something like this should serve whatever purpose you have. Notice, the code I wrote did not require finding all the department objects, but whatever you are doing likely will...
If you're interested in efficiency, then once you get this working, you can see if there is some way you can query the database to directly get the counts you want instead of this in-code processing. But I would start with something simple like this.
EDIT:
It sounds like you may be able to use something like:
Request.find().groupBy('department').sum('count').exec(function (err, results){
console.log(results);
});
But I'm finding conflicting reports on whether this works with sails-mongo, so take this as a "maybe this will work" recommendation.
I assume Request has a property department which is an ObjectId related to departments.
The quick one is to correct the following:
the 2nd line does not make sense as reports is not used
do
let departmentIds = _.map(departments, 'id');
and use departmentIds in the count query object
However, you are anyway finding all departments in the initial query, I would assume you count all the Requests anyway. Finding the departments first only makes sense in case you have query object there limiting the number of returned departments.
In addition, if you know the departmentIds in question you might not need to query them. Otherwise you might use a projection to just return the departmentIds instead of all the properties of all the departments
Related
The structure I have for my firebase database is like this:
fruits:
apple,5
banana,6
I want to put apple and banana in an array so that when i give a command to Google Assistant, it would give me apple, 5 and banana, 6. The code I have is like the one below:
function handleCommand(agent) {
return admin.database().ref('Fruits').child().once("value").then((snapshot) =>{
var i;
var fruitlist=[];
//puts each snapshot child of 'Fruit' in an array
snapshot.forEach(function(item) {
var itemVal = item.val();
fruitlist.push(itemVal);
});
//outputs command in google assistant
for (i=0; i < fruitlist.length; i++) {
agent.add(fruitlist[i]);
}
})
The default response is "not available".
I get the following in the execution logs:
Firebase.child failed. Was called 0 aruguments. expects at least 1.
I do not know which argument to put inside the Firebase.child. if i want all fruits to be "spoken" by Google Assistant. Below is a picture of my firebase structure.
The error looks like the one below:
What I am currently doing now to just output the fruits are manually entering each child in the code like this and removed the ".child" in the return statement:
Which gives me the output below which is also what I want to see but using arrays as the solution I am using now is very much hardcoded:
As the error message suggests, and as you surmise, the child() call expects a parameter - in particular, the name of the child node you want to get information from. However, since you want all the children of the "Fruits" node - you don't need to specify it at all. The child() call just navigates down through the hierarchy, but you don't need to navigate at all if you don't want to.
The snapshot you get back will have a value of the entire object. In some cases, this can be pretty large, so it isn't a good idea to get it all at once. In your case, it is fairly small, so not as big a deal.
On the JavaScript side, you can now handle that value as an object with attributes and values. Your original code didn't quite do what you said you want it to, however - you're getting the value, but ignoring the name (which is the attribute name or key). You can iterate over the attributes of an object in a number of ways, but I like getting the keys of the object, looping over this, getting the value associated with the key, and then "doing something" with it.
While I haven't tested the code, it might look something like this:
function handleCommand(agent) {
return admin.database().ref('Fruits').once("value").then((snapshot) =>{
// Get an object with all the fruits and values
var fruits = snapshot.val();
// Get the keys for the attributes of this object as an array
var keys = Object.keys( fruits );
// Iterate over the keys, get the associated value, and do something with it
for( var i=0; i<keys.length; i++ ){
var key = keys[i];
var val = fruits[key];
agent.add( `The number of ${key} you have are: ${val}` );
}
})
While this is (or should be) working Firebase and JavaScript, there are a couple of problems with this on the Actions on Google side.
First, the message returned might have some grammar problems, so using your example, you may see a message such as "The number of Apple you have are: 1". There are ways to resolve this, but keep in mind my sample code is just a starter sample.
More significantly, however, the call to agent.add() with a string creates a "SimpleResponse". You're only allowed two simple responses per reply in an Action. So while this will work for your example, it will have problems if you have more fruit. You can solve this by concatenating the strings together so you're only calling agent.add() once.
Finally, you may wish to actually look at some of the other response options for different surfaces. So while you might read out this list on a speaker, you may read a shorter list on a device with a screen and show a table with the information. Details about these might be better addressed as a new StackOverflow question, however.
Dataloader is able to batch and cache requests, but it can only be used by either calling load(key) or loadMany(keys).
The problem I am having is that sometimes I do not know they keys of the items I want to load in advance.
I am using an sql database and this works fine when the current object has a foreign key from a belongsTo relation with another model.
For example a user that belongs to a group and so has a groupId. To resolve the group you would just call groupLoader.load(groupId).
On the other hand, if I wanted to resolve the users within a group, of which there could be many I would want a query such as
SELECT * from users where user.groupId = theParticularGroupId
but a query such as this doesn't use the keys of the users and so I am not sure how make use of dataloader.
I could do another request to get the keys like
SELECT id from users where user.groupId = theParticularGroupId
and then call loadMany with those keys... But I could have just requested the data directly instead.
I noticed that dataloader has a prime(key, value) function which can be used to prime the cache, however that can only be done once the data is already fetched. At which point many queries would already have been sent, and duplicate data could have been fetched.
Another example would be the following query
query {
groups(limit: 10) {
id
...
users {
id
name
...
}
}
}
I cannot know the keys if I am searching for say the first or last 10 groups. Then once I have these 10 groups. I cannot know the keys of their users, and if each resolver would resolve the users using a query such as
SELECT * from users where user.groupId = theParticularGroupId
that query will be executed 10 times. Once the data is loaded I could now prime the cache, but the 10 requests have already been made.
Is there any way around this issue? Perhaps a different pattern or database structure or maybe dataloader isn't even the right solution.
You'll want a dataloader instance for the lookup you can do, in this case you have a group ID and you want the users:
import DataLoader from 'dataloader';
const userIdsForGroupLoader = new DataLoader(groupIds => batchGetUsersIdsForGroups(groupIds));
Now your batchGetUsersForGroups function is essentially has to convert an array of group IDs to an array of arrays of users (one array of user IDs for each group).
You'd start off with an IN query:
SELECT id from users where user.groupId in (...groupIds)
This will give you a single result set of users, which you'll have to manipulate, by grouping them by their groupId, the array should be ordered according to the original array of groupIds. Make sure you return an empty array for groupIds that don't have any users.
Note that in this we're only returning the user ids, but you can batch fetch the users in one go once you have them. You could tweak it slightly to return the users themselves, you'll have to decide for yourself if that's the right approach.
Everything I mention in this article can be achieved using clever use of Dataloader. But the key takeaway is that the values you pass to the load/loadMany functions don't have to correspond to the IDs of the objects you're trying to return.
my receipe data in firebase
![my receipe data in firebase][1]
My receipe data looks like the following
receipe/1/12: "{"itemno":"1","receipeno":"12","receipedescript..."
How can I filter all receipes where receipeno = 12 or starting with receipeno = 12, irrespective of itemno?
I have tried the following, but got no results or errors?
Also tried startAt
this.query = this.db.database.ref("receipe").orderByChild("receipeno").equalTo("12").limitToFirst(100);
BTW: this.query = this.db.database.ref("receipe") this returns all data
and this.db.database.ref("receipe/1") returns all receipe for itemno == 1.
I have updated the data to not use opaque strings.
[Updated db so as not to use opaque strings][2]
and have tried the following.
this.query = this.db.database.ref("receipe").orderByChild("itemno").equalTo("1").limitToFirst(100);
And
this.query = this.db.database.ref("receipe").orderByChild("receipeno").equalTo("r1").limitToFirst(100);
Firebase is not going to help you with deep queries. It filters and sorts at one level. Therefore, there is no way to say, "find me all the recipes with an id (or recipeno) of 12, under any node under recipes.
If you had a single level, then you would do orderByKey().equalTo("12"). Of course this will limit you to one recipe per ID.
If you are going to have multiple recipes with the number 12 and want to do this kind of querying, you need to change your database structure--essentially getting rid of the intermediate level where you currently have keys of 1, 2 etc. Most likely in this case you would use automatically generated keys of the kind that result from push, such as:
+ recipes
+ autokey1: {itemno, recipeno, recipedescription}
+ autokey2: {itemno, recipeno, recipedescription}
Then you could say
orderByChild('recipeno').equalTo('12')
This assumes, of course, as pointed out in a comment, that you are saving the data as objects, not as stringified JSON, which Firebase will never be able to query against the insides of.
This is a good case study of the notion that you should design your Firebase database structure carefully in advance to allow you to do the kinds of querying you will need to do.
By the way, this question has nothing to do whatsoever with Angular.
Ok, so I've been reading and reading and searching and searching and strangely it doesn't seem like my scenario has been really covered anywhere.
I have an app that creates a list of products. I want a simple view that can sort the products and page through them.
Fore reference here is a simple representation of the data in Firebase.
app
stock
unique_id
name
url
imageUrl
price
When creating the list I have multiple threads using the push method on my firebase references:
new Firebase(firebaseUrl).child('stock').push({
name: "name",
price: 123
});
This gives me a lovely "hash" collection on the stock property of the app.
So what I'd now like to do is have a table to sort and page through the records that were placed in the stock hash.
I make a GET request to my server to a url like /stock?limit=10&skip=10&sort=name%20asc. This particular url would be the second page where the table contained 10 records per page and was sorted by the name property in ascending order.
Currently in my query handler I have this:
var firebaseRef = new Firebase(firebaseUrl).child('stock');
if (this.sortDesc) {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.endAt()
.limitToFirst(this.limitAmount);
} else {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.limitToFirst(this.limitAmount);
if (this.skipAmount > 0) {
firebaseRef = firebaseRef.startAt(this.skipAmount);
}
}
firebaseRef.once('value', function (snapshot) {
var results = [];
snapshot.forEach(function (childSnapshot) {
results.push(childSnapshot.val());
});
callback(null, results);
});
I'm running into a couple of problems. I'm going to split this into two cases, ascending and descending queries.
Ascending query
The orderByChild and limitToFirst seems to work correctly in the sorting ascending case. This means I can change which property has an ascending sort and how many results to return. What I am not able to get to work is skipping n records for paging to work. In the example query above I'm going to the second page. I do not get results 11-20, but I instead get the same 10 records as the first page.
Descending query
In this case I cannot begin to figure out how to tell Firebase to order by a property of the object identified by the unique key in a descending fashion. The closest I've read is to use endAt() and then limit. Docs say the limit is deprecated plus this still doesn't help me with any paging.
I tired to do doodles picturing how this would work. I came up with: order by the property, start at the 'end' of the collection, and then limit back to the page size. While this still wouldn't solve paging I would expect it to give me the last n records where n was the size of the page. I get no results.
I suppose I could say use firebaseRef = firebaseRef .orderByChild(this.sortProperty).limitToLast(this.limitAmount + this.skipAmount); and in the result callback use the forEach loop to take the first (or would it be the last; I'm not sure how that iteration would work) n records where n=this.limitAmount. This just seems inefficient. Wouldn't it be better to limit the query instead of using CPU cycles to limit data that had come over the wire or is this the relational DB query thought pattern overriding the correct thought process for NoSQL?
Further Confusion
After posting this I've still been working on a solution. I've had some things get close, but I'm also running into this filtering issue. How could I filter a set of items to one property by still sorting on another? Jeez! I want to have the ability for a user to get all the stock that isn't sold out and order it by price.
Finally
Why hasn't this basic example been fleshed out on any of the Firebase "Getting Started" pages? Being able to show tabular data, page through it, sort, and filter seem like something that EVERY web developer would come across. I'm using ng-table in an Angular app to drive the view, but it still seems that regardless of platform that the queries that I'm trying to generate would be practical on any platform that Firebase supports. Perhaps I'm missing something! Please educate me!
Firebase and NoSQL
I've come up with this simple scenario that I often run into with web applications. I want to show tabular data, filter, page, and sort it. Very simple. Very common. Writing a SQL statement for this would be dead easy. Why is the query so complicated for something like Firebase. Is this common with all NoSQL solutions? There is no relational data being stored thus the need for a relational database seems unnecessary. Yet, it seems like I could hack together a little flat file to do this storage since the ability to make Firebase do these simple tasks is not made clear in its API or Docs. FRUSTRATED!!!
I'm playing around with the idea of creating a global search that allows me to find any model in any of a number of collections by any of the model's attributes. For example:
I have the following collections:
Users
Applications
Roles
I don't know ahead of time what attributes each User, Applicaion and Role will have but for illustration purposes lets say I have:
User.name
User.last_name
User.email
Application.title
Application.description
Role.name
Role.description
Now, lets say I create a model called Site with a method called search. I want Site.search(term) to search through all the items in each collection where term matches any of the attributes. In essence, a global model search.
How would you suggest I approach this? I can brute-force it by iterating through all the collections' models and each model's attributes but that seems bloated and inefficient.
Any suggestions?
/// A few minutes later...
Here's a bit of code I tried just now:
find: function(query) {
var results = {}; // variable to hold the results
// iterate over the collections
_.each(["users", "applications", "roles"], _.bind(function(collection){
// I want the result to be grouped by type of model so I add arrays to the results object
if ( !_.isUndefined(results[collection]) || !_.isArray(results[collection]) ) {
results[collection] = [];
}
// iterate over the collection's models
_.each(this.get(collection).models, function(model){
// iterate over each model's attributes
_.each(model.attributes, function(value){
// for now I'm only considering string searches
if (_.isString(value)) {
// see if `query` is in the attribute's string/value
if (value.indexOf(query) > -1) {
// if so, push it into the result's collection arrray
results[collection].push(model);
}
};
});
});
// a little cleanup
results[collection] = _.compact(results[collection]);
// remove empty arrays
if (results[collection].length < 1) {
delete results[collection];
}
},this));
// return the results
return results;
}
This yields the expected result and I suppose it works fine but it bothers me that I'm iterating over three arrays. there may not be another solution but I have a feeling there is. If anyone can suggest one, thank you! Meanwhile I'll keep researching.
Thank you!
I would strongly discourage you from doing this, unless you have a very limited set of data and performance is not really a problem for you.
Iteration over everything is a no-no if you want to perform search. Search engines index data and make the process feasible. It is hard to build search, and there is no client-side library that does that effectively.
Which is why everybody is doing searching on the server. There exist easy (or sort of) to use search engines such as solr or the more recent and my personal preference elasticsearch. Presumably you already store your models/collections on the server, it should be trivial to also index them. Then searching becomes a question of making a REST call from your client.