I have created a small application for keeping track of how much time I spend in different courses (as a teacher) using Angular 5 and putting my data in Firestore via AngularFire2. Most things have worked out nicely, but for the final part of the application I am having serious problems. I am quite a newbie at Angular/Firebase and most importantly (perhaps) with reactive programming.
The data that is stored in Firestore is quite simple:
Data stored in Firebase It consists of six fields for information about the course, when the course was held and most importantly what I did (called Element) and for how long (called Duration).
Now, what I would like to do is to make a report that summaries all the durations for each thing I did (Element). In essence, what I would like to do is the equivalent of a GroupBy in SQL. As I understand it Firebase does not have an operation for grouping.
My, so far failed approach to this, has been to try to create an array into which I put the data and make calculations on that. It is here that I fail.
My first attempt is based on creating an observer as most examples for retrieving data in tutorials use. WorkItem is a simple class that has the same content as in Firebase.
this.lectureDoc = this.afs.collection(`users/${this.afAuth.auth.currentUser.uid}/regTime`,
ref => ref.where('course', '==', this.selectedCourse.courseName).orderBy('element') );
this.lectureItems = this.lectureDoc.snapshotChanges().pipe(
map (courses => courses.map(a => {
const data = a.payload.doc.data() as WorkItem;
const id = a.payload.doc.id;
return { id, ...data };
})
));
this.lectureItems.subscribe(item => {
item.forEach(i => {
this.allElements.push(i);
});
});
console.error(this.allElements);
console.error(this.allElements[0]);
The two last lines show the problem, namely that the first (of the two last lines) will return the complete array while the second will return 'undefined'.
I understand that reactive programming will make asynchronous calls and that I therefore cannot know for sure when the data is filled. However, I do not understand why I can see the content of the array in the first of the two last lines, but not in the second.
My second attempt is based on getting the data from the documents of the collection itself and also, to prevent empty arrays, using .then() but again the two last lines show exactly the same output as previously.
this.lectureDoc.ref.get().then(item => {
item.forEach(i => {
this.lectures.push(i.data() as WorkItem);
});
});
console.error(this.lectures);
console.error(this.lectures[0]);
The output can be seen in this image: Output from the running program
So, to recap, what I would like to do is to collect all the data in an array of WorkItem and then calculate how much time has been spent on different tasks and then display it in list on the web page. I have not come to the display part and I suspect that I will have trouble again with the array not being populated when bound to the list. I have a hard time to understand reactive programming...
Any help would be greatly appreciated!
//Tobias
This is a pretty common behaviour of the console as far as I'm aware, at least in Chrome. When you log data that's async but haven't been populated yet, the console will still show the array once the async operation is completed.
But when you log an object in the array, at the time that you log it that object isn't there yet, therefore the log will show as undefined.
Simply do the logging inside of the forEach and you will see the data being pushed properly.
An another case, example there: Console.log behavior
Related
I'm working with EmberJS and SailsJS. Now I've been asked to make a statistics page and handle the filtering process in the SailsJS.
I have a model for departments and another model for requests the relationship between these models is (request belongsTo department). For some reason my manager prevented me to make a (hasMany) relationship.
Now what I want to do is to loop through all the departments and store them in new Object, inside that loop I want to loop through all the requests by Using Request.count({where : { department : department.id }}) and get the number of requests for each department in the departments Array Of Object.
I tried to do it as I explained but the problem is when I log the department inside the request loop it gives me the result as I imagined but when I add (.id) it shows me 'undefined'.
Here is my code:
Department.find().then((departments) => {
report.departments = departments;
Request.count({ department : departments.id}).exec(function countMe(err, count) {
console.log(count);
})
})
PS: if there's any other approach for this task please tell me, I'm kind of a beginner.
Your departments object in the .then callback is an array of department objects. To get an id you'd need to do something like departments[0].id.
I might not recommend using .count because that means a separate trip to the database for each department. Sails waterline may have some way for you to count up Requests grouped by department id, but just to get a steamroller working example, I would first just get everything and do some processing in your code:
Department.find().then((departments) => {
Request.find().then((requests) => {
var requestCounts = {}; // we'll store the counts in this object
for (var idx = 0; idx < requests.length; idx++) {
if (!requestCounts[requests[idx].department]) {
requestCounts[requests[idx].department] = 0;
}
requestCounts[requests[idx].department]++;
}
// use requestCounts...
});
});
Creating a separate object like that may not be what you want to do, but something like this should serve whatever purpose you have. Notice, the code I wrote did not require finding all the department objects, but whatever you are doing likely will...
If you're interested in efficiency, then once you get this working, you can see if there is some way you can query the database to directly get the counts you want instead of this in-code processing. But I would start with something simple like this.
EDIT:
It sounds like you may be able to use something like:
Request.find().groupBy('department').sum('count').exec(function (err, results){
console.log(results);
});
But I'm finding conflicting reports on whether this works with sails-mongo, so take this as a "maybe this will work" recommendation.
I assume Request has a property department which is an ObjectId related to departments.
The quick one is to correct the following:
the 2nd line does not make sense as reports is not used
do
let departmentIds = _.map(departments, 'id');
and use departmentIds in the count query object
However, you are anyway finding all departments in the initial query, I would assume you count all the Requests anyway. Finding the departments first only makes sense in case you have query object there limiting the number of returned departments.
In addition, if you know the departmentIds in question you might not need to query them. Otherwise you might use a projection to just return the departmentIds instead of all the properties of all the departments
I have a simple chat application that I built and I want to be able to display user uploaded images (locally hosted) next to the user names on the channel's html page. Currently, I am using presence to track the users who are logged into the channel etc. I was able to override the fetch/2 function with the understanding that it would allow me to add a couple map fields to the :metas symbol with user model data.
From what I can tell based on extensive IO.inspecting of different parts of each of the functions; fetch/2, handle_info/2, and some console.logging on my JS layer, the fetch/2 function is not actually getting any data out of the database nor is it assigning it to the :metas map.
here is my current fetch/2 function:
def fetch(_topic, entries) do
query =
from u in User,
where: u.id in ^Map.keys(entries),
select: {u.id, u}
users = query |> Repo.all |> Enum.into(%{})
for {key, %{metas: metas}} <- entries, into: %{} do
{key, %{metas: metas, user: users[key]}}
end
It is basically ripped directly from the documentation. In Theory, the function above should query my User Model and grab all of the user data based on the User.id that is being passed to it through the entries map. Users[keys] comes back as empty despite users being a full map of my User model.
Also, according to the documentation, the query is only supposed to run on join so as not to overload the DB but it seems to run 4-5 times every time I refresh the page. Another thing to note, is that the user.id inside of entries seems to be a string type. Im not sure if this is important, I've tried passing a integer from the JS layer and also using Interger.parse from the actual fetch/2 function to change this to no avail.
When I inspect the users map I get this:
{"1" => %MyApp.User{__meta__: #Ecto.Schema.Metadata<:loaded, "users">,
email: "test#test.com", encrypt_pass: "$pbkdf2-
sha512$160000$ebfY956TgIXhEAF.mqLJAg$QWzBubfeiy4Xrf.EsFiU0jEZAuKvV4ZO5a
8QpeFr817C61DuaNfyo56WWzj6jak2homCFWAINbPrFtCSXUPWTw", gravatar: %
{file_name: "logo.png", updated_at: #Ecto.DateTime<2017-04-20 22:00:08>},
id: 1, inserted_at: ~N[2017-04-20 22:00:09.071000], password: nil,
updated_at: ~N[2017-04-20 22:00:09.090000]}}
My users[key] returns an empty map like this %{} and converting the input into an integer throws an error, (Poison.EncodeError) unable to encode value: {nil, "users"} if i convert it inside of the elixir code, and where: u.id in ^["undefined"], select: {u.id, u} (elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3 – if I convert it from the JS layer.
The original fetch/2 output is an array with online_at: 1492764577562 and phx_ref: "OAyzaGE82xc=" in the :metas map and my user id or email in the users var.
What is it that I am missing here? I know that the fetch/2 function only executes as a callback to the Presence.list/1 function which I am calling in my handle_info/2 channel function. I am also calling Presence.list in my JS layer and mapping it to my presences so that I can produce the list of usernames in the HTML. Am I just misunderstanding how this works or is there some other more simple way that I should be going about this? If you need to see more code I can supply more.
edit: I have a much better understanding of what is happening here. my entries map is actually this:
%{"1" => %{metas: [%{online_at: 1492798247818, phx_ref: "ELHwA+gWF+0="}]}}
So basically, the string for the user id, "1" is being mapped onto the metas map. When I try to just take that key out of the map with the Map.keys(entries) function it isn't able to pull anything out of the DB because it is a string, however, when I change it to an integer from the JavaScript side it throws an error because for whatever reason phoenix is expecting that key to be a string type. Strangely enough if I change the id from an id to an email and try to query the db with the email it doesn't work either. despite the email in the Database being string and the metas map expecting a string key for the entries map.
I am going to rebuild this channel part of the app from the ground up and see what is causing this problem. Then I will come back and see if I was able to fix the error.
You should validate the keys in the entries map first.
ids = Map.keys(entries)
true = Enum.all?(ids, &is_integer/1)
Ecto will convert strings to integers when interpolating into a query:
iex(40)> Ecto.Query.from(u in Users, where: u.id in ^[1, 2, "3", nil], select: u.id) |> Repo.all()
outputs the following debug log:
[debug] QUERY OK source="users" db=0.8ms
SELECT u0."id" FROM "users" AS r0 WHERE (r0."id" = ANY($1)) [[1, 2, 3, nil]]
Notice it coerced the string "3" to an integer and allowed the nil.
However a map will not be so kind:
iex(42)> users = %{1 => %{name: "joe"}, 2 => %{name: "jill"}}
%{1 => %{name: "joe"}, 2 => %{name: "jill"}}
iex(43)> users["1"]
nil
So in the code where you are using the keys from entries for database lookups and map lookups, it could be producing different results.
I've already figured out that the problem has very little to do with my fetch/2 function itself, rather, it had to do with my implementation of the presence module and channel in this case. Basically, the fetch/2 function was being called 4 times every time some one entered the chat room and two out of the four times it was being called with an empty list value [].
Obviously, you can't query a Ecto model with an empty list so it was throwing an error in that case as well. I tried putting guards on the fetch function to filter out the empty list calls but it would not show me the metas map data that I was looking for even when the query succeeded.
Also, the other main problem was my implementation or lack of implementation of a token. I wouldn't have to pass around the user model data through fetch function metas map if I was using a token for joining the chat room rather then just a user (aka the just a username). After making that realization, I was able to successfully connect the user model data with the channel and show it through the JS layer and ultimately, put it on the client.
Anyways guys, thanks for the suggestions. You may not have answered the question (It was my fault for asking the wrong question), but you certainly helped me get there. And also gave me the tools to form a much better understanding of the framework in general on the way.
If/When I have any more questions, I will make sure that I am asking the correct questions before posting them to stack overflow, that way I wont be wasting time.
Ok, so I've been reading and reading and searching and searching and strangely it doesn't seem like my scenario has been really covered anywhere.
I have an app that creates a list of products. I want a simple view that can sort the products and page through them.
Fore reference here is a simple representation of the data in Firebase.
app
stock
unique_id
name
url
imageUrl
price
When creating the list I have multiple threads using the push method on my firebase references:
new Firebase(firebaseUrl).child('stock').push({
name: "name",
price: 123
});
This gives me a lovely "hash" collection on the stock property of the app.
So what I'd now like to do is have a table to sort and page through the records that were placed in the stock hash.
I make a GET request to my server to a url like /stock?limit=10&skip=10&sort=name%20asc. This particular url would be the second page where the table contained 10 records per page and was sorted by the name property in ascending order.
Currently in my query handler I have this:
var firebaseRef = new Firebase(firebaseUrl).child('stock');
if (this.sortDesc) {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.endAt()
.limitToFirst(this.limitAmount);
} else {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.limitToFirst(this.limitAmount);
if (this.skipAmount > 0) {
firebaseRef = firebaseRef.startAt(this.skipAmount);
}
}
firebaseRef.once('value', function (snapshot) {
var results = [];
snapshot.forEach(function (childSnapshot) {
results.push(childSnapshot.val());
});
callback(null, results);
});
I'm running into a couple of problems. I'm going to split this into two cases, ascending and descending queries.
Ascending query
The orderByChild and limitToFirst seems to work correctly in the sorting ascending case. This means I can change which property has an ascending sort and how many results to return. What I am not able to get to work is skipping n records for paging to work. In the example query above I'm going to the second page. I do not get results 11-20, but I instead get the same 10 records as the first page.
Descending query
In this case I cannot begin to figure out how to tell Firebase to order by a property of the object identified by the unique key in a descending fashion. The closest I've read is to use endAt() and then limit. Docs say the limit is deprecated plus this still doesn't help me with any paging.
I tired to do doodles picturing how this would work. I came up with: order by the property, start at the 'end' of the collection, and then limit back to the page size. While this still wouldn't solve paging I would expect it to give me the last n records where n was the size of the page. I get no results.
I suppose I could say use firebaseRef = firebaseRef .orderByChild(this.sortProperty).limitToLast(this.limitAmount + this.skipAmount); and in the result callback use the forEach loop to take the first (or would it be the last; I'm not sure how that iteration would work) n records where n=this.limitAmount. This just seems inefficient. Wouldn't it be better to limit the query instead of using CPU cycles to limit data that had come over the wire or is this the relational DB query thought pattern overriding the correct thought process for NoSQL?
Further Confusion
After posting this I've still been working on a solution. I've had some things get close, but I'm also running into this filtering issue. How could I filter a set of items to one property by still sorting on another? Jeez! I want to have the ability for a user to get all the stock that isn't sold out and order it by price.
Finally
Why hasn't this basic example been fleshed out on any of the Firebase "Getting Started" pages? Being able to show tabular data, page through it, sort, and filter seem like something that EVERY web developer would come across. I'm using ng-table in an Angular app to drive the view, but it still seems that regardless of platform that the queries that I'm trying to generate would be practical on any platform that Firebase supports. Perhaps I'm missing something! Please educate me!
Firebase and NoSQL
I've come up with this simple scenario that I often run into with web applications. I want to show tabular data, filter, page, and sort it. Very simple. Very common. Writing a SQL statement for this would be dead easy. Why is the query so complicated for something like Firebase. Is this common with all NoSQL solutions? There is no relational data being stored thus the need for a relational database seems unnecessary. Yet, it seems like I could hack together a little flat file to do this storage since the ability to make Firebase do these simple tasks is not made clear in its API or Docs. FRUSTRATED!!!
I want to query object from Parse DB through javascript, that has only 1 of some specific relation object. How can this criteria be achieved?
So I tried something like this, the equalTo() acts as a "contains" and it's not what I'm looking for, my code so far, which doesn't work:
var query = new Parse.Query("Item");
query.equalTo("relatedItems", someItem);
query.lessThan("relatedItems", 2);
It seems Parse do not provide a easy way to do this.
Without any other fields, if you know all the items then you could do the following:
var innerQuery = new Parse.Query('Item');
innerQuery.containedIn('relatedItems', [all items except someItem]);
var query = new Parse.Query('Item');
query.equalTo('relatedItems', someItem);
query.doesNotMatchKeyInQuery('objectId', 'objectId', innerQuery);
...
Otherwise, you might need to get all records and do filtering.
Update
Because of the data type relation, there are no ways to include the relation content into the results, you need to do another query to get the relation content.
The workaround might add a itemCount column and keep it updated whenever the item relation is modified and do:
query.equalTo('relatedItems', someItem);
query.equalTo('itemCount', 1);
There are a couple of ways you could do this.
I'm working on a project now where I have cells composed of users.
I currently have an afterSave trigger that does this:
const count = await cell.relation("members").query().count();
cell.put("memberCount",count);
This works pretty well.
There are other ways that I've considered in theory, but I've not used
them yet.
The right way would be to hack the ability to use select with dot
notation to grab a virtual field called relatedItems.length in the
query, but that would probably only work for me because I use PostGres
... mongo seems to be extremely limited in its ability to do this sort
of thing, which is why I would never make a database out of blobs of
json in the first place.
You could do a similar thing with an afterFind trigger. I'm experimenting with that now. I'm not sure if it will confuse
parse to get an attribute back which does not exist in its schema, but
I'll find out, by the end of today. I have found that if I jam an artificial attribute into the objects in the trigger, they are returned
along with the other data. What I'm not sure about is whether Parse will decide that the object is dirty, or, worse, decide that I'm creating a new attribute and store it to the database ... which could be filtered out with a beforeSave trigger, but not until after the data had all been sent to the cloud.
There is also a place where i had to do several queries from several
tables, and would have ended up with a lot of redundant data. So I wrote a cloud function which did the queries, and then returned a couple of lists of objects, and a few lists of objectId strings which
served as indexes. This worked pretty well for me. And tracking the
last load time and sending it back when I needed up update my data allowed me to limit myself to objects which had changed since my last query.
I tried to find the answer a lot but no way. I have an Ext.data.ArrayStore store and want to get its data as string. I tried store.getRange(), store.getAt() but I couldn't figure out what these functions return. Is there any way to get ArrayStore data as string?
I am newbie to extjs, so if you have any example on this, I'd appreciate.
It really depends what you want to do with the data. For most UI widgets and that sort of thing, you'll want to just use the store directly. If you want to get a piece of data from the store for tweaking manually, that's a whole nother story.
store.getRange() will indeed return all of the records from the store, but they are returned as an Array of Record objects. Records contain an attribute called data which is an object containing any properties you defined in the record's config.
Example:
Ext.each(store.getRange(), function (item, idx, a) {
for (var i in item.data) {
console.log(item.data[i])
}
})
That should show you every item in every Record in store
EDIT: Changed my answer to not be totally wrong.