We Keep Coding

Google Firestore - How to get several documents by multiple ids in one round-trip?

I am wondering if it's possible to get multiple documents by a list of ids in one round trip (network call) to the Firestore database.

if you're within Node:
https://github.com/googleapis/nodejs-firestore/blob/master/dev/src/index.ts#L978
/**
* Retrieves multiple documents from Firestore.
*
* #param {...DocumentReference} documents - The document references
* to receive.
* #returns {Promise<Array.<DocumentSnapshot>>} A Promise that
* contains an array with the resulting document snapshots.
*
* #example
* let documentRef1 = firestore.doc('col/doc1');
* let documentRef2 = firestore.doc('col/doc2');
*
* firestore.getAll(documentRef1, documentRef2).then(docs => {
* console.log(`First document: ${JSON.stringify(docs[0])}`);
* console.log(`Second document: ${JSON.stringify(docs[1])}`);
* });
*/
This is specifically for the server SDK
UPDATE: Cloud Firestore Now Supports IN Queries!
myCollection.where(firestore.FieldPath.documentId(), 'in', ["123","456","789"])

In practise you would use firestore.getAll like this
async getUsers({userIds}) {
const refs = userIds.map(id => this.firestore.doc(`users/${id}`))
const users = await this.firestore.getAll(...refs)
console.log(users.map(doc => doc.data()))
}
or with promise syntax
getUsers({userIds}) {
const refs = userIds.map(id => this.firestore.doc(`users/${id}`))
this.firestore.getAll(...refs).then(users => console.log(users.map(doc => doc.data())))
}

They have just announced this functionality, https://firebase.googleblog.com/2019/11/cloud-firestore-now-supports-in-queries.html .
Now you can use queries like, but mind that the input size can't be greater than 10.
userCollection.where('uid', 'in', ["1231","222","2131"])

With Firebase Version 9 (Dec, 2021 Update):
You can get multiple documents by multiple ids in one round-trip using "documentId()" and "in" with "where" clause:
import {
query,
collection,
where,
documentId,
getDocs
} from "firebase/firestore";
const q = query(
collection(db, "products"),
where(documentId(), "in",
[
"8AVJvG81kDtb9l6BwfCa",
"XOHS5e3KY9XOSV7YYMw2",
"Y2gkHe86tmR4nC5PTzAx"
]
),
);
const productsDocsSnap = await getDocs(q);
productsDocsSnap.forEach((doc) => {
console.log(doc.data()); // "doc1", "doc2" and "doc3"
});

You could use a function like this:
function getById (path, ids) {
return firestore.getAll(
[].concat(ids).map(id => firestore.doc(`${path}/${id}`))
)
}
It can be called with a single ID:
getById('collection', 'some_id')
or an array of IDs:
getById('collection', ['some_id', 'some_other_id'])

No, right now there is no way to batch multiple read requests using the Cloud Firestore SDK and therefore no way to guarantee that you can read all of the data at once.
However as Frank van Puffelen has said in the comments above this does not mean that fetching 3 documents will be 3x as slow as fetching one document. It is best to perform your own measurements before reaching a conclusion here.

If you are using flutter, you can do the following:
Firestore.instance.collection('your_collection_name')
.where(FieldPath.documentId, whereIn:["list", "of", "document", "ids"])
.getDocuments();
This will return a Future containing List<DocumentSnapshot> which you can iterate as you feel fit.

Surely the best way to do this is by implementing the actual query of Firestore in a Cloud Function? There would then only be a single round trip call from the client to Firebase, which seems to be what you're asking for.
You really want to be keeping all of your data access logic like this server side anyway.
Internally there will likely be the same number of calls to Firebase itself, but they would all be across Google's super-fast interconnects, rather than the external network, and combined with the pipelining which Frank van Puffelen has explained, you should get excellent performance from this approach.

You can perform an IN query with the document IDs (up to ten):
import {
query,
collection,
where,
getDocs,
documentId,
} from 'firebase/firestore';
export async function fetchAccounts(
ids: string[]
) {
// use lodash _.chunk, for example
const result = await Promise.all(
chunk(ids, 10).map(async (chunkIds) => {
const accounts = await getDocs(
query(
collection(firestore, 'accounts'),
where(documentId(), 'in', chunkIds)
));
return accounts.docs.filter(doc => doc.exists()).map(doc => doc.data());
})
);
return result.flat(1);
}

Here's how you would do something like this in Kotlin with the Android SDK.
May not necessarily be in one round trip, but it does effectively group the result and avoid many nested callbacks.
val userIds = listOf("123", "456")
val userTasks = userIds.map { firestore.document("users/${it!!}").get() }
Tasks.whenAllSuccess<DocumentSnapshot>(userTasks).addOnSuccessListener { documentList ->
//Do what you need to with the document list
}
Note that fetching specific documents is much better than fetching all documents and filtering the result. This is because Firestore charges you for the query result set.

For some who are stucked in same problem
here is a sample code:
List<String> documentsIds = {your document ids};
FirebaseFirestore.getInstance().collection("collection_name")
.whereIn(FieldPath.documentId(), documentsIds).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (DocumentSnapshot document : Objects.requireNonNull(task.getResult())) {
YourClass object = document.toObject(YourClass.class);
// add to your custom list
}
}
}
}).addOnFailureListener(new OnFailureListener() {
#Override
public void onFailure(#NonNull Exception e) {
e.printStackTrace();
}
});

For the ones who want to do it using Angular, here is an example:
First some library imports are needed: (must be preinstalled)
import * as firebase from 'firebase/app'
import { AngularFirestore, AngularFirestoreCollection } from '#angular/fire/firestore'
Some configuration for the collection:
yourCollection: AngularFirestoreCollection;
constructor(
private _db : AngularFirestore,
) {
// this is your firestore collection
this.yourCollection = this._db.collection('collectionName');
}
Here is the method to do the query: ('products_id' is an Array of ids)
getProducts(products_ids) {
var queryId = firebase.firestore.FieldPath.documentId();
this.yourCollection.ref.where(queryId, 'in', products_ids).get()
.then(({ docs }) => {
console.log(docs.map(doc => doc.data()))
})
}

I hope this helps you, it works for me.
getCartGoodsData(id) {
const goodsIDs: string[] = [];
return new Promise((resolve) => {
this.fs.firestore.collection(`users/${id}/cart`).get()
.then(querySnapshot => {
querySnapshot.forEach(doc => {
goodsIDs.push(doc.id);
});
const getDocs = goodsIDs.map((id: string) => {
return this.fs.firestore.collection('goods').doc(id).get()
.then((docData) => {
return docData.data();
});
});
Promise.all(getDocs).then((goods: Goods[]) => {
resolve(goods);
});
});
});
}

Yes, it is possible. Sample in .NET SDK for Firestore:
/*List of document references, for example:
FirestoreDb.Collection(ROOT_LEVEL_COLLECTION).Document(DOCUMENT_ID);*/
List<DocumentReference> docRefList = YOUR_DOCUMENT_REFERENCE_LIST;
// Required fields of documents, not necessary while fetching entire documents
FieldMask fieldMask = new FieldMask(FIELD-1, FIELD-2, ...);
// With field mask
List<DocumentSnapshot> documentSnapshotsMasked = await FirestoreDb.GetAllSnapshotsAsync(docRefList, fieldMask);
// Without field mask
List<DocumentSnapshot>documentSnapshots = await FirestoreDb.GetAllSnapshotsAsync(docRefList);
Documentation in .NET:
Get all snapshots
Field mask

This doesn't seem to be possible in Firestore at the moment. I don't understand why Alexander's answer is accepted, the solution he proposes just returns all the documents in the "users" collection.
Depending on what you need to do, you should look into duplicating the relevant data you need to display and only request a full document when needed.

if you are using the python firebase admin sdk this is how you query for multiple documents using their uids
from firebase_admin import firestore
import firebase_admin
from google.cloud.firestore_v1.field_path import FieldPath
app = firebase_admin.initialize_app(cred)
client = firestore.client(app)
collection_ref = client.collection('collection_name')
query = collection_ref.where(FieldPath.document_id(), 'in', listOfIds)
docs = query.get()
for doc in docs:
print(doc.id, doc.to_dict())
Instead of importing FieldPath you can also simply use the string __name__. Now your query will be collection_ref.where('__name__', 'in', listOfIds)

The best you can do is not use Promise.all as your client then must wait for .all the reads before proceeding.
Iterate the reads and let them resolve independently. On the client side, this probably boils down to the UI having several progress loader images resolve to values independently. However, this is better than freezing the whole client until .all the reads resolve.
Therefore, dump all the synchronous results to the view immediately, then let the asynchronous results come in as they resolve, individually. This may seem like petty distinction, but if your client has poor Internet connectivity (like I currently have at this coffee shop), freezing the whole client experience for several seconds will likely result in a 'this app sucks' experience.

Best method to string together variety of DB calls in Node js

I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}

Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.

EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!

I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.

Automatically populate() on Mongoose document fetch

Is it possible to automatically run populate() for referenced subdocuments for a particular model?
I have a City model that has referenced Region and Country documents that I would like to be automatically populated when a city/cities are fetched.

Well, there aren't docs for this in the Mongoose website; what I do is something like this:
schema.statics.createQuery = function( populates ) {
var query = this.find();
populates.forEach(function( p ) {
query.populate( p );
});
return query;
};
Of course, there is validation and some other stuff in this method, but essentially it's what I do with my models.
In your case, you could hard code the populates in such a method, if you strictly need them in every find call.

AFAIK there's no way to auto-populate all references to another model out of the box (there are plugins though). In a similar fashion to #gustavohenke's answer you can use a static, along with a small change to your find query.
Here's what I'd do:
citySchema.statics.fieldsToPopulate = function() {
return ['regionField', 'countryField'];
};
Where regionField and countryField are the fields which reference the models Region and Country respectively.
Then in your query you could populate accordingly:
var populate = city.fieldsToPopulate ? city.fieldsToPopulate() : [];
city.findById(id)
.populate(populate)
.exec(function(err, data) {
if (err) {
return next(err);
} else {
res.render('template', { city: data });
}
});

mongoose/mongodb query multiple sort

i'm quite new to mongodb. i manage to get a basic idea of a simple sort based only 1 parameter. what if there are more than 2 sort parameters. for instance, in a database made up of woodworking projects that have attributes totalCuttingTime and favorited.
Is the following a correct mongoose/mongodb function to find a list of projects that have the least totalCuttingTime and order in according to highest favoriteCounts to lowest.
var ProjectModel= mongoose.model('Project', schema);
exports.getMinCuttingTime = function(number, callback){
var leastCutTimeResult = ProjectModel.find().sort({totalCuttingTime: 1}).select({_id: 1}).limit(number).exec(
function(err, projects) {
callback(null, projects)
}
);
var result = leastCutTimeResult.find().sort({favoriteCount: -1}).select({_id: 1}).limit(number).exec(
function(err, projects) {
callback(null, projects)
}
);
return result;
}

You need to put both sort terms into one object:
exports.getMinCuttingTime = function(number, callback){
ProjectModel.find()
.sort({totalCuttingTime: 1, favoriteCount: -1})
.select({_id: 1})
.limit(number)
.exec(
function(err, projects) {
callback(null, projects)
}
);
};
It's worth noting that the ECMA-262 standard on which Node.js is based doesn't specify that an object's property order is maintained, and it's only a de facto standard to match insertion order. To eliminate any doubt, you can use an array instead:
.sort([['totalCuttingTime', 1], ['favoriteCount', -1]])

New to asynchronous programming in JavaScript. Any advice?

Asynchronous callbacks are great but when one callback depends on the result of another I have callbacks with api calls that have callbacks, and so on.
apiCall(function () { apiCall(function () { apiCall(function () ...
I can name the callback functions instead of including them inline. That looks prettier and has less nesting but I do not find it any easier to read.
Here is an example. I need to query the local sqlite database, use the result to query a server, then use the response to update the local database.
function sync() {
db.transaction(
function (transaction) {
execute(transaction, 'SELECT max(server_time) AS server_time FROM syncs;', [],
function (transaction, results) { // Query results callback
var t = results.rows.item(0).server_time;
$.post('sync.json', { last_sync_time: (t || '1980-01-01') },
function (data) { // Ajax callback
db.transaction(
function(transaction) {
$(data.thing).each(function () {
var thing = new Thing(this.thing);
thing.insert(transaction);
});
});
});
});
});
}
Is there a way to untagle this (other than naming the callbacks)?

I think you're too quick to discard un-nesting things by naming your functions rather than writing them inline. This is pretty much the only way to clean up that mess.
Instead of:
do_a(
function () {
// more nesting...
}
);
Use names to provide a bit of clarity and purpose to each function:
function on_a_complete() {
}
do_a(on_a_complete);

I think you answered your own question. Naming your callbacks is really the only way to clean this up anymore. Something like:
execute(transaction, 'SELECT max(server_time) AS server_time FROM syncs;', [],handleLocalResults, errorHandler);
handleLocalResults = function (transaction, results)...
handleServerResults = func...

Naming the callbacks is one thing you can do, but the other thing that you need to do is to have non-overlapping SQL transactions. (1)
Your first transaction should be like this:
// The whole thing starts here
db.transaction(selectTimeCB, null, ajaxPost);
The transaction starts with the callback to select the time, and when the transaction is complete, make a call to the ajaxPost operation.
// Initial transaction to get server_time
var selectTimeCB = function(t) {
var query = 'SELECT max(server_time) AS server_time FROM syncs';
t.executeSql(query, [], postLastSyncCB);
};
// This saves the results from the above select, and nothing else.
var server_time;
var postLastSyncCB = function(t, results) {
server_time = results.rows.item(0).server_time;
};
var ajaxPost = function() {
$.post('sync.json', { last_sync_time: (server_time || '1980-01-01') }, nextDbTransaction);
};
If you have overlapped SQL transactions, that can really kill the performance of the database. I recently did a test of 200 mixed transactions on a database with a bit over 500 rows, and found that keeping the transactions separate reduced a run time of over 90 seconds to 3-5 seconds.