We Keep Coding

Firestore not writing all documents

I'm using forEach to write over 300 documents with data from an object literal.
It works 80% of the time -- all documents get written, the other times it only writes half or so, before the response gets sent and the function ends. Is there a way to make it pause and always work correctly?
Object.entries(qtable).forEach(([key, value]) => {
db.collection("qtable").doc(key).set({
s: value.s,
a: value.a
}).then(function(docRef) {
console.log("Document written with ID: ", docRef.id);
res.status(200).send(qtable);
return null;
})
Would it be bad pratice to just put a 2 second delay?

You are sending the response inside your loop, before the loop is complete. If you are using Cloud Functions (you didn't say), sending the response will terminate the function an clean up any extra work that hasn't completed.
You will need to make sure that you only send the response after all the async work is complete. This means you will have to pay attention to the promises returned by set() and use them to determine when to finally send the response. Leaning how promises work in JavaScript is crucial to writing functions that work properly.

You need to wait for the set() calls to conclude. They return a promise that you should deal with.
For instance, you can do this by pushing the return of set() to a promise array and awaiting for them outside the loop (with Promise.all()).
Or you can await each individual call, but in this case you need to change the forEach() to a normal loop, otherwise the await will not work inside a forEach() arrow function.
Also, you should probably set the response status just once, and outside the loop.

How to combine arguments from separate Promises?

This problem should be clear: I want to write into a log file using a statement like the following. You can assume that this statement is inside a click handler for a button. There are two questions embedded in this pseudocode.
pLogInfo("local info").then(pauseUntilSettled).catch(err); // This is my goal
Here are my library functions (each returns a Promise):
// Get info asynchronously from a server (simple, should work)
function pGetServerInfo()
{
// then value: "server info"
} // pGetServerInfo
// Write into a file asynchronously (simple, should work)
function pWriteFile(path,string)
{
// no then value
} // pWriteFile
// Write formatted info into a log file asynchronously
function pLogInfo(localInfo)
{
pGetServerInfo().then(p2);
} // pLogInfo
function p2(serverInfo)
{
// Should write "local info, server info"
// But where is the local info? It got lost.
return pWriteFile('log',localInfo+','+serverInfo);
} // p2
Usage:
pLogInfo("local info").then(pauseUntilSettled).catch(err);
function pauseUntilSettled()
{
// How to wait before returning from
// the button click event handler?
} // pauseUntilSettled
ADDED 8/27/19:
Several possible solutions to this common problem occur to me:
Attach an object to one of the top functions in your chain (p.data={}). You can store any arguments or callbacks you wish in the object and reference them in any asynchronous 'then' code. This works because the function that is the parent of the object has global scope. It can fail if you fire another top-level Promise of the same thread while an existing Promise is not yet settled, because the new execution thread will share and overwrite the object. I have successfully used this approach, a variant of the above global formulation, but it is clearly unsafe.
Create a closure function to propagate the asynchronous thread. A closure function contains a snapshot of its arguments and referenced global variables. I have not yet gotten this idea to work, but it seems reasonable.
Create a new Promise, either as part of the thread of Promises or as a separate helper, that calls its resolve function with an object instead of a single value. Use that object to propagate more than one value to each "then" function. I have not gotten this idea to work either.
I hope these ideas inspires someone (including myself) to come up with a solution, because it is a very common problem that is not often discussed.

Satpal gave a great answer for promises. Another option is to use the RXJS library and utilize observables which are built on top of Promises. They have a next(), error() and complete() block where code will only execute once values are received in the stream. If you want to wait on multiple services to respond you can combine the streams together in multiple ways.
I know this is not the exact answer you are looking for but it is a very useful library. Here is the documentation. https://rxjs-dev.firebaseapp.com/api/index/class/Observable

Solution:
You can put intermediate values in scope in any later 'then' function explicitly, by using 'bind'. It is a nice solution that doesn't require changing how Promises work, and only requires a line or two of code to propagate the values just like errors are already propagated.
Here is a complete example:
// Get info asynchronously from a server
function pGetServerInfo()
{
// then value: "server info"
} // pGetServerInfo
// Write into a file asynchronously
function pWriteFile(path,string)
{
// no then value
} // pWriteFile
// The heart of the solution: Write formatted info into a log file asynchronously,
// using the pGetServerInfo and pWriteFile operations
function pLogInfo(localInfo)
{
var scope={localInfo:localInfo}; // Create an explicit scope object
var thenFunc=p2.bind(scope); // Create a temporary function with this scope
return (pGetServerInfo().then(thenFunc)); // Do the next 'then' in the chain
} // pLogInfo
// Scope of this 'then' function is {localInfo:localInfo}
function p2(serverInfo)
{
// Do the final 'then' in the chain: Writes "local info, server info"
return pWriteFile('log',this.localInfo+','+serverInfo);
} // p2
This solution can be invoked as follows:
pLogInfo("local info").then().catch(err);
(Note: a more complex and complete version of this solution has been tested, but not this example version, so it could have a bug as written.)

mdg:validated-method _execute asynchronous issues

I'm running into problems with the validated method package in my app tests. I'm calling my methods through the _execute function in order to be able to pass a userId to simulate a logged-in user while testing. The problem is that my asserts right underneath that _execute are called before the method has a chance of completing. I know my test works though because it only happens sometimes, mostly because mongo isn't always returning results quite as fast.
I looked around and found a todos app that uses the _execute function in its tests. I can't get those tests to fail no matter how many times I rerun them, though.
This is an example of my test code.
describe('clients.add', function() {
it('should add an empty (draft) client', function() {
const res = clients_add._execute({ userId: 'CURRENTUSERID' }, { company_id: c1._id });
assert.instanceOf(res, Mongo.ObjectID, 'method returns the newly created clients ID');
const db_client = Clients.findOne(res);
assert.isTrue(db_client.draft, 'client is drafted');
assert.isDefined(db_client.created, 'there\'s a created date');
});
});
clients_add does quite a few permission checks and can therefor take a little while before completing. Rerunning this test 20 times will fail about 5 times and pass the other 15.
Shouldn't the _execute function be synchronous? How do I make it? What am I missing?

In server code, if you provide a callback to database modification functions like insert, it returns the created ID instantaneously, and runs the callback only once the database has acknowledged the write. If you don't provide a callback, the insert call is synchronous and throws an error if the operation fails. See more about this in Meteor docs.
It seems that you have provided an error-handling callback to the insert-function in your method code. This causes the inconsistent behavior, since the database might not actually have had time to do the write before you call findOne in your test. Also, this is redundant since if an error occurs in the insert, the method has already returned and the error is never shown to the user. It's better to simply omit the error-handling callback altogether:
return Clients.insert(new_client);

Does .then(Promise.mapSeries(...)) behave differently than .then(function() { return Promise.mapSeries(...); })?

I initially assumed that passing a bare Promise.mapSeries(...) call as an argument to .then() would be the same as wrapping it in a function, like .then(function() { return Promise.mapSeries(...); }). Having written out this question, I'm no longer entirely sure why it works at all.
In the simplified code below, I open a couple of databases asynchronously (openDBAsync()) and then read a file containing a JS object. I use _.map() to iterate over all of the key/value pairs in the object and asynchronously update their values in a database, while keeping track of which ones meet certain criteria (whether the value is odd, in this toy example). Promise.all() waits for all of the async database calls to settle, and then Promise.mapSeries() is used to process each of the subset of keys, which makes another async database call for each one. Finally, I close all the databases.
function processData(path)
{
var oddKeys = [];
return openDBAsync()
.then(function() { return readFileAsync(path); })
.then(function(dataObject) {
return Promise.all(_.map(dataObject, function(value, key) {
if (value % 2) {
oddKeys.push(key);
}
return updateDBAsync(key, ++value);
}))
.then(Promise.mapSeries(
oddKeys,
function(key) {
return updateOddDBAsync(key);
}
))
})
.then(closeDBAsync);
}
The problem is that the database throws errors complaining that I'm trying to update the database after it's been closed. That means that some of the promises generated in the .mapSeries() call are being called after the final closeDBAsync(). I expected all of them to settle before the final .then() call.
If I wrap the call to Promise.mapSeries() in a function:
.then(function() {
return Promise.mapSeries(
oddKeys,
function(key) {
return updateOddDBAsync(key);
}
);
})
Then I don't get any errors. It also works if I put a .delay(2000) before the close database call, which indicates that Promise.mapSeries() isn't settling all of the promises before it finishes.
This seems like either a bug in Bluebird, or, much more likely, I'm not understanding something fundamental about how Promise.mapSeries() works. Any guidance would be much appreciated.

much more likely, I'm not understanding something fundamental about how Promise.mapSeries() works
Nope, this seems to be more a misunderstanding about how .then(…) works.
The then method of promises does always take a callback function (if you're passing anything else [but null], Bluebird should spit out a warning!). By calling .then(Promise.mapSeries(…)), you were passing a promise instead, which is simply ignored. Being ignored, it also is not awaited by anything, which leads to that error of the database being closed too early.
But the direct call to Promise.mapSeries() doesn't get applied to the array immediately. If it was, the array would be empty and this code wouldn't work at all.
Yes it does. Your array is filled from by the _.map callback, which is executed synchronously, before then and mapSeries are invoked.
So the solution is indeed to wrap the call in a function expression, which will only be executed when the Promise.all(…) fulfills, and whose result will then not be ignored but rather awaited. There might be more, different solutions, depending on what degree of parallel execution you want to allow.
Btw, given you are doing a database transaction, your current code is quite fragile. Look into the promise disposer pattern.

JavaScript Double Null Check and Locking

In a language with threads and locks it is easy to implement a lazy load by checking the value of a variable, if it's null then lock the next section of code, check the value again and then load the resource and assign. This prevents it from being loaded multiple times and causes threads after the first to wait for the first thread to complete the action that's needed.
Psuedo code:
if(myvar == null) {
lock(obj) {
if(myvar == null) {
myvar = getData();
}
}
}
return myvar;
JavaScript runs in a single thread, however, it still has this type of issue because of asynchronous execution while one call is waiting on a blocking resource. In this Node.js example:
var allRecords;
module.exports = getAllRecords(callback) {
if(allRecords) {
return callback(null,allRecords);
}
db.getRecords({}, function(err, records) {
if (err) {
return callback(err);
}
// Use existing object if it has been
// set by another async request to this
// function
allRecords = allRecords || partners;
return callback(null, allRecords);
});
}
I'm lazy loading all the records from a small DB table the first time this function is called and then returning the in-memory records on subsequent calls.
Problem: If multiple async requests are made to this function at the same time then the table is going to be loaded unnecessarily from the DB multiple times.
In order to solve this I could simulate a locking mechanism by creating a var lock; variable and setting it to true while the table is loading. I would then put the other async calls into a setTimeout() loop and check back on this variable every (say) 1 second until the data was available and then allow them to return.
The problems with that solution are:
It's fragile, what if the first async call throws and doesn't unset the lock.
How many times do we loop back into the timer before giving up?
How long should the timer be set for? In some environments 1 second might be way too long and inefficient.
Is there a best practise for solving this in JavaScript?

On the first call to the service, initialize an array. Start the fetch operation. Create a Promise, store it in the array.
On subsequent calls, if the data is there, return an already-fulfilled Promise. If not, add another Promise to the array and return that.
When the data arrives, resolve all the waiting Promise objects in the list. (You can throw away the list once the data's there.)

I really like the promise solution in the other answer -- very clever, very interesting. Promises aren't the dominent methodology, so you may need to educate the team. I'm going to go in another direction though.
What you're after is a memoize function -- an in-memory key/value cache of expensive results. JavaScript the Good Parts has a memoize sample towards the end. Lodash has a memoize function. These assume synchronous processing so don't account for your scenario -- which is to say they'd hit the database lots of times until one of the "threads" replied.
The async library also has a memoize function that does exactly what you want. In it's innards, it keeps a queue array of callbacks, and once it gets the answer, it both caches it and calls all the callbacks.
If you're into inventing, by all means, use promises. If you'd just like a plug-n-play answer, use async#memoize.