We Keep Coding

Firestore not writing all documents

I'm using forEach to write over 300 documents with data from an object literal.
It works 80% of the time -- all documents get written, the other times it only writes half or so, before the response gets sent and the function ends. Is there a way to make it pause and always work correctly?
Object.entries(qtable).forEach(([key, value]) => {
db.collection("qtable").doc(key).set({
s: value.s,
a: value.a
}).then(function(docRef) {
console.log("Document written with ID: ", docRef.id);
res.status(200).send(qtable);
return null;
})
Would it be bad pratice to just put a 2 second delay?

You are sending the response inside your loop, before the loop is complete. If you are using Cloud Functions (you didn't say), sending the response will terminate the function an clean up any extra work that hasn't completed.
You will need to make sure that you only send the response after all the async work is complete. This means you will have to pay attention to the promises returned by set() and use them to determine when to finally send the response. Leaning how promises work in JavaScript is crucial to writing functions that work properly.

You need to wait for the set() calls to conclude. They return a promise that you should deal with.
For instance, you can do this by pushing the return of set() to a promise array and awaiting for them outside the loop (with Promise.all()).
Or you can await each individual call, but in this case you need to change the forEach() to a normal loop, otherwise the await will not work inside a forEach() arrow function.
Also, you should probably set the response status just once, and outside the loop.

Will JS promisified function run alone after I return from the function in next line

My requirement is like this
I want to run an axios call.
I don't want to block the code until it finished.
Also I don't want to know it's 200 or 500
This is my experiment code.
function axios() {
console.log("axios calling...");
return new Promise((resolve, reject) => {
setTimeout(function() {
resolve(console.log("DONE!"));
}, 10000);
});
}
function run() {
axios();
console.log("Before");
return;
console.log("This should never log");
}
run();
According to this experiment I think even though I return from the function still that promisified function is run. Which means it guaranteed call the axios.
My concern is,
if axios take 10 mins established connection with the API (NOT SEND THE POST REQUEST) if I return from the next line will axios wait that 10 mins and send the request or break the connection establishing when I return?

A promise will wait for data regardless of weather you call .then() on it or not (note that there are some exceptions[1] ). And the page will continue processing and wait for events as long as it's opened.
So if the request takes 10 minutes it will continue waiting (barring timeouts on either the server, gateway, router or browser) until you close the page/tab.
[1] Some libraries only triggers promise creation when you call .then() for example knex will keep returning a query object instead of a promise until you call .then() on the query object)

How to handle async code within a loop in JavaScript?

I am reaching out to a SongKick's REST API in my ReactJS Application, using Axios.
I do not know how many requests I am going to make, because in each request there is a limit of data that I can get, so in each request I need to ask for a different page.
In order to do so, I am using a for loop, like this:
while (toContinue)
{
axios.get(baseUrl.replace('{page}',page))
.then(result => {
...
if(result.data.resultsPage.results.artist === undefined)
toContinue = false;
});
page++;
}
Now, the problem that I'm facing is that I need to know when all my requests are finished. I know about Promise.All, but in order to use it, I need to know the exact URLs I'm going to get the data from, and in my case I do not know it, until I get empty results from the request, meaning that the last request is the last one.
Of course that axios.get call is asynchronous, so I am aware that my code is not optimal and should not be written like this, but I searched for this kind of problem and could not find a proper solution.
Is there any way to know when all the async code inside a function is finished?
Is there another way to get all the pages from a REST API without looping, because using my code, it will cause extra requests (because it is async) being called although I have reached empty results?

Actually your code will crash the engine as the while loop will run forever as it never stops, the asynchronous requests are started but they will never finish as the thread is blocked by the loop.
To resolve that use async / await to wait for the request before continuing the loop, so actually the loop is not infinite:
async function getResults() {
while(true) {
const result = await axios.get(baseUrl.replace('{page}', page));
// ...
if(!result.data.resultsPage.results.artist)
break;
}
}

Run 1000 requests so that only 10 runs at a time

With node.js I want to http.get a number of remote urls in a way that only 10 (or n) runs at a time.
I also want to retry a request if an exception occures locally (m times), but when the status code returns an error (5XX, 4XX, etc) the request counts as valid.
This is really hard for me to wrap my head around.
Problems:
Cannot try-catch http.get as it is async.
Need a way to retry a request on failure.
I need some kind of semaphore that keeps track of the currently active request count.
When all requests finished I want to get the list of all request urls and response status codes in a list which I want to sort/group/manipulate, so I need to wait for all requests to finish.
Seems like for every async problem using promises are recommended, but I end up nesting too many promises and it quickly becomes uncypherable.

There are lots of ways to approach the 10 requests running at a time.
Async Library - Use the async library with the .parallelLimit() method where you can specify the number of requests you want running at one time.
Bluebird Promise Library - Use the Bluebird promise library and the request library to wrap your http.get() into something that can return a promise and then use Promise.map() with a concurrency option set to 10.
Manually coded - Code your requests manually to start up 10 and then each time one completes, start another one.
In all cases, you will have to manually write some retry code and as with all retry code, you will have to very carefully decide which types of errors you retry, how soon you retry them, how much you backoff between retry attempts and when you eventually give up (all things you have not specified).
Other related answers:
How to make millions of parallel http requests from nodejs app?
Million requests, 10 at a time - manually coded example
My preferred method is with Bluebird and promises. Including retry and result collection in order, that could look something like this:
const request = require('request');
const Promise = require('bluebird');
const get = Promise.promisify(request.get);
let remoteUrls = [...]; // large array of URLs
const maxRetryCnt = 3;
const retryDelay = 500;
Promise.map(remoteUrls, function(url) {
let retryCnt = 0;
function run() {
return get(url).then(function(result) {
// do whatever you want with the result here
return result;
}).catch(function(err) {
// decide what your retry strategy is here
// catch all errors here so other URLs continue to execute
if (err is of retry type && retryCnt < maxRetryCnt) {
++retryCnt;
// try again after a short delay
// chain onto previous promise so Promise.map() is still
// respecting our concurrency value
return Promise.delay(retryDelay).then(run);
}
// make value be null if no retries succeeded
return null;
});
}
return run();
}, {concurrency: 10}).then(function(allResults) {
// everything done here and allResults contains results with null for err URLs
});

The simple way is to use async library, it has a .parallelLimit method that does exactly what you need.

Structuring functions that encapsulate node.js request

I am making a call to an API within a function. At this point I get "undefined" as the returned value. I know the call to the API is successful since the URLs that I am trying to get in the call print to the term no problem. I am 99% sure that call to the encapsulating function gets triggered before the request function is done ("undefined" is returned before URLs are listed). Wanted to confirm this and ask if there is a tutorial or code snippet someone could point me to with a good description of a pattern I should follow in this case. <-- Obviously still struggling with the async nature of the beast :)
function GetPhotoURLs(url){
var photo_urls= new Array();
request({
url: url,
json: true
}, function (error, response, body) {
if (!error && response.statusCode === 200)
{
//console.log(body) // Print the json response
var inhale_jsonp=body;
var blog_stream= inhale_jsonp.substring(22,inhale_jsonp.length-2); //getting JSON out of the wrapper
blog_stream=JSON.parse(blog_stream);
for(var i=0;i<blog_stream.posts.length;i++)
{
photo_urls[i]=blog_stream['posts'][i]['photo-url-500'];
console.log(photo_urls[i]+"\n"); //checking that I am getting all the URLs
}
console.log("success!");
console.log(photo_urls[1]);
return photo_urls;
}
else
{
photo_urls[0]='none';
console.log("nope!");
console.log(photo_urls[0]);
return photo_urls;
}
});
}
output sequence -->
1. undefined
2. Listing of URLs
3. Success message + second element from URLs array

The request() function is asynchronous. As such, it finishes long after your original function has completed. Thus, you can't return the result from it (the result isn't even known yet when the function returns). Instead, you must process the result IN your callback or call a function from within that callback and pass the result to that function.
As for general design patterns for this type of work, you can either process the result in the callback as suggested above or switch to using promises to help you manage the asynchronous nature of the request. In all cases, you will be processing the result in some sort of callback.
You can read this answer for some more detail on handling async responses. That specific answer is written for client-side ajax calls, but the concepts for handling async responses are identical.
In your specific case, you may want to make getPhotoURLs() take a callback function which you can call with the result:
function GetPhotoURLs(url, callback){
.....
request(..., function(error, response, body) {
...
// when you have all the results
callback(photo_urls);
})
}
GetPhotoURLs(baseurl, function(urls) {
// process urls here
// other code goes here after processing the urls
});