We Keep Coding

Building an Agenda / Calendar / Availability view with Firestore

I'm stuck. Stuck because Firestore is not (yet) capable to handle my (to me) relatively simple query. I don't want anything fancy. All I wish to get from the database are the meetings that have not ended yet. But, I would like the list to be limited to 5 items, and ordered by the starting date of the meeting, to display a small "Up next" style Agenda.
However, the following is not a valid query:
where("end", "<", now), orderBy("start", "asc"), limit(5));
So, how do I achieve this rather simple query for my Agenda?
And, while we're here, maybe we can dig into the other queries, too:
:: Display the last meeting (Already Over)
where("end", "<", now), orderBy("end", "desc"), limit(1))
:: Display the current meeting (Now) - Started, but not Ended.
.... ?????
:: Display the meetings which have not yet started (Next)
where("start", ">", now), orderBy("start", "asc"), limit(5))
The only thing that I can think of right now for the "current", is to grab the array of all the meetings that have ended. Grab the array of all the future meetings, and an array of all meetings. Substract the (previous) and (future) arrays from the (all) list, and I'll have the one meeting that hasn't ended, but has already started. Theres gotta be a more efficient way to do this. No?

The common approach for this is to define buckets, and then assign each meeting to the relevant buckets.
For example, say that you show the events that are going on today, and then style the events that have already started/ended differently.
In such a scenario, you could have an array of meetings_days that contains the days that the meeting is active: meeting_days: ['2022-12-01', '2022-12-02', '2022-12-03', '2022-12-04', '2022-12-05'].
Now you can use an array-contains filter to select the events for a given day, and then determine the styling in your client-side application code. If you don't show all events (e.g. not ones that have already finished), determining the right bucket size to limit overreads while keeping the data size reasonable.
An alternative data model could be a may of the meeting days: meeting_days: { '2022-12-01': true, '2022-12-02': true, '2022-12-03': true, '2022-12-04': true, '2022-12-05': true }. Now you can do an AND type query, like finding only events that run the first 2 days of December.
The correct data model here depends on the use-cases of your app, and will likely change/evolve as your app evolves.

react-native-firebase/firestore: query limit from result x to y [duplicate]

I would like to create two queries, with pagination option. On the first one I would like to get the first ten records and the second one I would like to get the other all records:
.startAt(0)
.limit(10)
.startAt(9)
.limit(null)
Can anyone confirm that above code is correct for both condition?

Firestore does not support index or offset based pagination. Your query will not work with these values.
Please read the documentation on pagination carefully. Pagination requires that you provide a document reference (or field values in that document) that defines the next page to query. This means that your pagination will typically start at the beginning of the query results, then progress through them using the last document you see in the prior page.

From CollectionReference:
offset(offset) → {Query}
Specifies the offset of the returned results.

As Doug mentioned, Firestore does not support Index/offset - BUT you can get similar effects using combinations of what it does support.
Firestore has it's own internal sort order (usually the document.id), but any query can be sorted .orderBy(), and the first document will be relative to that sorting - only an orderBy() query has a real concept of a "0" position.
Firestore also allows you to limit the number of documents returned .limit(n)
.endAt(), .endBefore(), .startAt(), .startBefore() all need either an object of the same fields as the orderBy, or a DocumentSnapshot - NOT an index
what I would do is create a Query:
const MyOrderedQuery = FirebaseInstance.collection().orderBy()
Then first execute
MyOrderedQuery.limit(n).get()
or
MyOrderedQuery.limit(n).get().onSnapshot()
which will return one way or the other a QuerySnapshot, which will contain an array of the DocumentSnapshots. Let's save that array
let ArrayOfDocumentSnapshots = QuerySnapshot.docs;
Warning Will Robinson! javascript settings is usually by reference,
and even with spread operator pretty shallow - make sure your code actually
copies the full deep structure or that the reference is kept around!
Then to get the "rest" of the documents as you ask above, I would do:
MyOrderedQuery.startAfter(ArrayOfDocumentSnapshots[n-1]).get()
or
MyOrderedQuery.startAfter(ArrayOfDocumentSnapshots[n-1]).onSnapshot()
which will start AFTER the last returned document snapshot of the FIRST query. Note the re-use of the MyOrderedQuery
You can get something like a "pagination" by saving the ordered Query as above, then repeatedly use the returned Snapshot and the original query
MyOrderedQuery.startAfter(ArrayOfDocumentSnapshots[n-1]).limit(n).get() // page forward
MyOrderedQuery.endBefore(ArrayOfDocumentSnapshots[0]).limit(n).get() // page back
This does make your state management more complex - you have to hold onto the ordered Query, and the last returned QuerySnapshot - but hey, now you're paginating.
BIG NOTE
This is not terribly efficient - setting up a listener is fairly "expensive" for Firestore, so you don't want to do it often. Depending on your document size(s), you may want to "listen" to larger sections of your collections, and handle more of the paging locally (Redux or whatever) - Firestore Documentation indicates you want your listeners around at least 30 seconds for efficiency. For some applications, even pages of 10 can be efficient; for others you may need 500 or more stored locally and paged in smaller chucks.

MongoDB: Can you include a collection as a field in a document of another collection?

I'm using Javascript, Node.js, Express and MongoDB to make a webapplication. In the application You can make an account with fields name and lastnames, but I also want a field that keeps track of completed steps (boolean: false if uncompleted, true if completed). It's not a possibility to hardcode the steps as separate boolean fields because I want to make the list of steps adjustable. So I was thinking of making another collection (=progress) with al the names of the steps and where you can add some and know I am searching for a way to include these boolean steps for each account I make.
sketch of how I think the MongoDB collection Schema's should look like:
account
- name (String)
- lastname (String)
- progress:
- stepName (String) , completed (boolean)
- stepName (String) , completed (boolean)
- stepName (String) , completed (boolean)
- ...
progress
- stepName (String)
- completed (boolean)
Many thanks in advance

You can -- since it looks like you are using Mongoose, you can see https://mongoosejs.com/docs/subdocs.html for some info on how to do something like you are outlining here.
However, based on my experience doing similar things (tracking progress through a flow), I would suggest setting things up a little differently. The biggest issue I see is that if you add a new step, you don't want to have to go through and update every account document to put the new step into its progress field. Or similarly, if you decide to remove a step, you don't want to have to go to each account and remove that step's entry from its progress field.
Instead of thinking of it as including the progress collection in each document of the account collection, I would think of it as having a step collection that contains any adjustable/configurable data about each step. There is no need to have a completed field there, because that's something specific to each account. Depending on your needs, I would suggest something like the following:
step
- id (String) -- something that will never change
- name (String) -- a user-facing name that you might want to change later
- position (number) -- used to control the order of steps presented to the user
- ... other things, depending on what you need/want
Now, for account, you will want a map (https://mongoosejs.com/docs/schematypes.html#maps) showing the progress (completion state of each step):
account
- name (String)
- lastname (String)
- progress: (Map of boolean)
- key: step id
- value: boolean showing whether it has been completed
- ...
This means any time you're checking an account's progress, you'll want to loop through the documents from the step collection (ordered based on the position field), and check that account's progress map against each step id to see if there are any that aren't completed. Although that sounds like extra work, there are a lot of advantages to this sort of approach. It means if you decide to add, remove, or reorder a step within the step collection, you don't need to go make the same change to every existing account document to match -- you'll just automatically pick up the change the next time you check the account's progress field.
One further note -- the way I'm suggesting you set up the schema, instead of having an explicit false value for each step that isn't completed, most of the time you just wouldn't have any value in the map for uncompleted steps. In your code, you can easily treat any missing step progress values the same as a false (if (!account.progress.firstStep) { works by default with javascript). This also gives you the option, if you want, of setting a step's progress to explicitly false only when a user has been presented with that step. Thus you have some easier visibility into what step they stopped on (even if you later change the order of the steps) and/or if they've skipped any steps (if some are considered optional), rather than only being able to see which steps have been completed.

What you describe is called an "Embedded Document" and is supported by mongodb.
Mongodb documentation includes an example of what you want:
{
...
name: { first: "Alan", last: "Turing" },
contact: { phone: { type: "cell", number: "111-222-3333" } },
...
}
And in mongodb it's accessed by:
"<embedded document>.<field>"
Source: https://docs.mongodb.com/manual/core/document/

Sorting and Paging A Collection With Unique IDs in Firebase

Ok, so I've been reading and reading and searching and searching and strangely it doesn't seem like my scenario has been really covered anywhere.
I have an app that creates a list of products. I want a simple view that can sort the products and page through them.
Fore reference here is a simple representation of the data in Firebase.
app
stock
unique_id
name
url
imageUrl
price
When creating the list I have multiple threads using the push method on my firebase references:
new Firebase(firebaseUrl).child('stock').push({
name: "name",
price: 123
});
This gives me a lovely "hash" collection on the stock property of the app.
So what I'd now like to do is have a table to sort and page through the records that were placed in the stock hash.
I make a GET request to my server to a url like /stock?limit=10&skip=10&sort=name%20asc. This particular url would be the second page where the table contained 10 records per page and was sorted by the name property in ascending order.
Currently in my query handler I have this:
var firebaseRef = new Firebase(firebaseUrl).child('stock');
if (this.sortDesc) {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.endAt()
.limitToFirst(this.limitAmount);
} else {
firebaseRef = firebaseRef
.orderByChild(this.sortProperty)
.limitToFirst(this.limitAmount);
if (this.skipAmount > 0) {
firebaseRef = firebaseRef.startAt(this.skipAmount);
}
}
firebaseRef.once('value', function (snapshot) {
var results = [];
snapshot.forEach(function (childSnapshot) {
results.push(childSnapshot.val());
});
callback(null, results);
});
I'm running into a couple of problems. I'm going to split this into two cases, ascending and descending queries.
Ascending query
The orderByChild and limitToFirst seems to work correctly in the sorting ascending case. This means I can change which property has an ascending sort and how many results to return. What I am not able to get to work is skipping n records for paging to work. In the example query above I'm going to the second page. I do not get results 11-20, but I instead get the same 10 records as the first page.
Descending query
In this case I cannot begin to figure out how to tell Firebase to order by a property of the object identified by the unique key in a descending fashion. The closest I've read is to use endAt() and then limit. Docs say the limit is deprecated plus this still doesn't help me with any paging.
I tired to do doodles picturing how this would work. I came up with: order by the property, start at the 'end' of the collection, and then limit back to the page size. While this still wouldn't solve paging I would expect it to give me the last n records where n was the size of the page. I get no results.
I suppose I could say use firebaseRef = firebaseRef .orderByChild(this.sortProperty).limitToLast(this.limitAmount + this.skipAmount); and in the result callback use the forEach loop to take the first (or would it be the last; I'm not sure how that iteration would work) n records where n=this.limitAmount. This just seems inefficient. Wouldn't it be better to limit the query instead of using CPU cycles to limit data that had come over the wire or is this the relational DB query thought pattern overriding the correct thought process for NoSQL?
Further Confusion
After posting this I've still been working on a solution. I've had some things get close, but I'm also running into this filtering issue. How could I filter a set of items to one property by still sorting on another? Jeez! I want to have the ability for a user to get all the stock that isn't sold out and order it by price.
Finally
Why hasn't this basic example been fleshed out on any of the Firebase "Getting Started" pages? Being able to show tabular data, page through it, sort, and filter seem like something that EVERY web developer would come across. I'm using ng-table in an Angular app to drive the view, but it still seems that regardless of platform that the queries that I'm trying to generate would be practical on any platform that Firebase supports. Perhaps I'm missing something! Please educate me!
Firebase and NoSQL
I've come up with this simple scenario that I often run into with web applications. I want to show tabular data, filter, page, and sort it. Very simple. Very common. Writing a SQL statement for this would be dead easy. Why is the query so complicated for something like Firebase. Is this common with all NoSQL solutions? There is no relational data being stored thus the need for a relational database seems unnecessary. Yet, it seems like I could hack together a little flat file to do this storage since the ability to make Firebase do these simple tasks is not made clear in its API or Docs. FRUSTRATED!!!

How can I know when a Firebase query has ended

I want to collect the most recent 10 items from my datastore. I believe I am supposed to do this using .child() and .limitToLast(), which emits an event every time a result has been added to the child.
I don't necessarily know if there are 10 items in total, so a simple counter won't work.
How do I know when Firebase is finished giving me results?
Example lookup code:
var plots = [];
firebaseDatastore.child("plots").orderByChild("unicode-timestamp").limitToLast(10).on("child_added", function(snapshot) {
// Add the new item to the list
plots.push(snapshot.val());
});
I need to know when the final plot has been added, regardless of whether it has hit the limit.

A Firebase query never really ends. It continues to monitor (in your case) the plots by unicode-timestamp and keep a "window" of the last 10 plots.
So:
child_added:timestamp1
child_added:timestamp2
...
child_added:timestamp9
child_added:timestamp10
And then when you add another plot:
child_removed:timestamp1
child_added:timestamp11
If you are not looking to use this behavior, you have two options:
use a value event and then snapshot.forEach over the child nodes
keep a counter and off your listener when you've reached the number of children you expect