I'm using forEach to write over 300 documents with data from an object literal.
It works 80% of the time -- all documents get written, the other times it only writes half or so, before the response gets sent and the function ends. Is there a way to make it pause and always work correctly?
Object.entries(qtable).forEach(([key, value]) => {
db.collection("qtable").doc(key).set({
s: value.s,
a: value.a
}).then(function(docRef) {
console.log("Document written with ID: ", docRef.id);
res.status(200).send(qtable);
return null;
})
Would it be bad pratice to just put a 2 second delay?
You are sending the response inside your loop, before the loop is complete. If you are using Cloud Functions (you didn't say), sending the response will terminate the function an clean up any extra work that hasn't completed.
You will need to make sure that you only send the response after all the async work is complete. This means you will have to pay attention to the promises returned by set() and use them to determine when to finally send the response. Leaning how promises work in JavaScript is crucial to writing functions that work properly.
You need to wait for the set() calls to conclude. They return a promise that you should deal with.
For instance, you can do this by pushing the return of set() to a promise array and awaiting for them outside the loop (with Promise.all()).
Or you can await each individual call, but in this case you need to change the forEach() to a normal loop, otherwise the await will not work inside a forEach() arrow function.
Also, you should probably set the response status just once, and outside the loop.
Related
This question concerns the Firestore database, but, more generally, it concerns making async requests in parallel.
Simply put, I wish to update multiple Firestore documents as quickly and efficiently as possible by mapping over an array of their document IDs.
the .set() method is async (returning a promise) and so I understand that I can wrap the multiple requests - i.e. the map() - in a Promise.all() function, which will return a single promise once all of the requests have resolved.
However, it is not at all clear to me whether I should await the set() within the map().
Whichever way I write the code (i.e. with or without await), it does not appear to make a difference to speed, but does it, or should it?
What is the correct way to achieve the greatest speed and efficiency in this instance?
const update_promises = array_of_ids.map(async id => {
const new_values = {
new_values_1: "val1",
last_update: new Date()
}
return await db.collection("my_collection").doc(id).set(new_values, { merge: true });
// OR SHOULD IT BE:
// return db.collection("my_collection").doc(id).set(new_values, { merge: true });
})
return await Promise.all(update_promises)
When get call set(), the SDK is going to immediately pipeline the write request over the a single managed connection. If you want to write a bunch of documents as fast as possible, you should kick them all off, then await the results at the end. You probably don't want to await each one individually, since you are causing the code to stop for a moment while it waits for the result before the next one can get sent. That said, the performance impact is probably negligible overall, unless you have a lot of work to be done in between each write.
My general rule is to only await an individual promise if its result is needed right away before moving on. Otherwise, collect all the promises together into an array for a single await with Promise.all().
I am running a post call inside a forEach loop. How do I see to that, it does not go to the next element in the list till the process for the first one is complete?
This is my array: arr = ['Hermione', 'Ron', 'Harry']
Here is my loop:
arr.forEach(d=>{
$('#the_table>tbody').append(`
<tr id='tr_${d}'>
<td'>${d}</td>
<td><i id='change' class='no'></i></td>
</tr>
`);
});
const myfunc = () => {
$.post(api, json_object,
function(data, status) {
if (data.value === 'started') {
setTimeout(myfunc, 5000);
} else {
$('#change').removeClass('no').addClass('yes');
}
}, 'json');
}
myfunc();
I want to start with the first element Hermione. Till the status changes from started to done, I don't want to start the process on the second element Ron and so on.
How do I do that?
You don't need a recursive function; you need to use the Promise api (async/await), with their buddies .then() .when() .next() etc. The basic problem you are facing is that you need results returned in a certain order, and ajax is asynchronous (things happen when they happen, not necessarily in order) - but you need them in a certain order.
In the beginning was the XMLHttpRequest (XHR), and (for the first time!) it allowed client-side javascript to communicate with a backend server, and receive a response with new data that could be parsed and added to the DOM without leaving or refreshing the page. And it was tricky and difficult, and few there be'd who used it - but it worked.
.then() along came John Resig and he invented jQuery and within it he included the $.ajax() construct, along with its shortcut cousins $.get() $.post() and $.load() - and XHR was significantly easier. And the world rejoiced! But because it was asynchronous, it was still kinda tricky to get the values out of the functions in the right order when you had several related ajax calls to make.
.then() along came the Promise api - because what could be easier than having your code make a Promise and wait for it to be fulfilled? And the world rejoiced once again, and there was no looting. You just set up a promise and .then() you can use .after() and .next() and .when() and, of course, .then() for truly readable code that was as easy to read as it was to write.
And nothing could possibly be easier! Until 2017, when async and await were added to ECMAScript. Now all you have to do is add the keyword async at the top of a function scope, and then within it you can use the await keyword to write those promises for you. Because async/await is really just a wrapper for the Promises api, you can still use .then() .when() .next() etc. - but your code is more readable and simpler to write!
And the world rejoiced, and still there was no looting.
WttW*: This (async/await and Promises api) is something you absolutely must get your head around - it will be more useful to you each day.
*Word to the Wise
References:
https://blog.bitsrc.io/understanding-javascript-async-and-await-with-examples-a010b03926ea
https://petetasker.com/using-async-await-jquerys-ajax/
I'm quite new to using Javascript and particularly JSON, and I've been struggling to do this:
There is a JSON file on my web server which I am trying to access and parse into a JavaScript object. What I am trying to do is parse that JSON into an array and then further manipulate that array based on other user variables.
This is a sample of what the JSON looks like:
{"log":
[{
"name":"Al",
"entries":[8,12,16,19]},
{"name":"Steve",
"entries":[11,17,22]}]}
What I need to do is retrieve the array for one of the entries and store it in an array as a JavaScript object. What I have tried to do is this:
var entriesLogged;
fetch ('url to the json file').then(function(response){
return response.json();
}).then(function(data){
entriesLogged = data.log[0].entries;
});
However, I can't seem to get this to work and to assign the value to the variable in a way that persists outside of this scope. I have been able to output the value of the array using console.log, but I have not been able to actually work with and manipulate that data like an object. I'd ideally like to parse the JSON file from the server onto a global array.
Most of the tutorials I've come across so far have used the JSON file to output console logs or change the contents of html elements, however I need to retrieve the values from the JSON into a global array first.
Am I missing something here? Does anyone have advice on how to do this?
Best wishes,
Dom
Are you trying to manipulate the entriesLogged variable after the fetch promise chain? The fetch is asynchronous so this means that any code after the fetch chain will run before the fetch chain finishes.
var entriesLogged;
fetch('url to the json file').then(function(response){
return response.json();
}).then(function(data){
entriesLogged = data.log[0].entries;
});
// Code down here will run before the fetch chain finishes
// So if you're using entriesLogged down here, it will be null
console.log(entriesLogged); // entriesLogged is null
You might want to do something like:
.then(function(data){
entriesLogged = data.log[0].entries;
myFunction(entriesLogged);
});
function myFunction(myData) {
console.log(myData); // can access entriesLogged here
}
What is happening is that the call is async, it means that when you make that call a new thread is created and javascript automatically passes to the next line.
var entriesLogged;
fecth(something).then(function(data){entriesLogged = data[0]}); // this goes to another thread an takes e.g. 5 miliseconds but javascript does not wait for this to complete, it goes to the next line
console.log(entriesLogged); // this is executed before the call is finished
There are TONS of "answers" for variations of this question that don't actually solve the problem, they just describe the issue. (tl;dr solution is last block of code)
The issue is that .fetch('URL.json') and .json() are "asynchronous" calls. So the script will start working on that call, then continue on to the rest of the commands in your script while the .fetch() and .json() calls are still being worked on. If you then hit a command BEFORE one of these asynchronous calls is done, the rest of your script will have no data to work with.
There are tons of places to look for fully understanding the ins and outs of how async calls work in JS, for example here:
How do I return the response from an asynchronous call?
If you're like me and you just want the code to function synchronously, the quick and dirty solution is to create an async main loop and then to make sure you use the await keyword when calling asynchronous functions that you need to WAIT for the data to be populated on.
Example main loop:
(async () => {
// put main loop code here
})();
You would then put your code inside that main loop BUT you have to make sure that you're adding the await keyword in front of every asynchronous call you use. in the original example provided, you'll have to declare your .then() functions that need to use await as async as well. This is confusing if you don't fully understand the asynchronous nature of javascript and I'll simplify it later, but I wanted to provide a working version of the original code provided so you can understand what changed:
(async () => {
var entriesLogged;
await fetch('url_to_the_file.json')
.then( async function(response){
return await response.json();
})
.then( function(data){
entriesLogged = data;
});
})();
The .then() call is typically used for post-processing so you can do work inside of the asynchronous call... but since in this example we're deliberately making it synchronous, we can avoid all of that and clean up the code by doing individual simple commands:
// Start main loop
(async () => {
let response = await fetch('url_to_the_file.json');
let entriesLogged = await response.json();
})();
That should get you what you're looking for and hopefully save anyone like us the heartache of spending hours of time trying track down something that there is a simple solution for.
Also want to give a huge call out to the place I got the solution for this:
https://www.sitepoint.com/delay-sleep-pause-wait/
I am reaching out to a SongKick's REST API in my ReactJS Application, using Axios.
I do not know how many requests I am going to make, because in each request there is a limit of data that I can get, so in each request I need to ask for a different page.
In order to do so, I am using a for loop, like this:
while (toContinue)
{
axios.get(baseUrl.replace('{page}',page))
.then(result => {
...
if(result.data.resultsPage.results.artist === undefined)
toContinue = false;
});
page++;
}
Now, the problem that I'm facing is that I need to know when all my requests are finished. I know about Promise.All, but in order to use it, I need to know the exact URLs I'm going to get the data from, and in my case I do not know it, until I get empty results from the request, meaning that the last request is the last one.
Of course that axios.get call is asynchronous, so I am aware that my code is not optimal and should not be written like this, but I searched for this kind of problem and could not find a proper solution.
Is there any way to know when all the async code inside a function is finished?
Is there another way to get all the pages from a REST API without looping, because using my code, it will cause extra requests (because it is async) being called although I have reached empty results?
Actually your code will crash the engine as the while loop will run forever as it never stops, the asynchronous requests are started but they will never finish as the thread is blocked by the loop.
To resolve that use async / await to wait for the request before continuing the loop, so actually the loop is not infinite:
async function getResults() {
while(true) {
const result = await axios.get(baseUrl.replace('{page}', page));
// ...
if(!result.data.resultsPage.results.artist)
break;
}
}
In a language with threads and locks it is easy to implement a lazy load by checking the value of a variable, if it's null then lock the next section of code, check the value again and then load the resource and assign. This prevents it from being loaded multiple times and causes threads after the first to wait for the first thread to complete the action that's needed.
Psuedo code:
if(myvar == null) {
lock(obj) {
if(myvar == null) {
myvar = getData();
}
}
}
return myvar;
JavaScript runs in a single thread, however, it still has this type of issue because of asynchronous execution while one call is waiting on a blocking resource. In this Node.js example:
var allRecords;
module.exports = getAllRecords(callback) {
if(allRecords) {
return callback(null,allRecords);
}
db.getRecords({}, function(err, records) {
if (err) {
return callback(err);
}
// Use existing object if it has been
// set by another async request to this
// function
allRecords = allRecords || partners;
return callback(null, allRecords);
});
}
I'm lazy loading all the records from a small DB table the first time this function is called and then returning the in-memory records on subsequent calls.
Problem: If multiple async requests are made to this function at the same time then the table is going to be loaded unnecessarily from the DB multiple times.
In order to solve this I could simulate a locking mechanism by creating a var lock; variable and setting it to true while the table is loading. I would then put the other async calls into a setTimeout() loop and check back on this variable every (say) 1 second until the data was available and then allow them to return.
The problems with that solution are:
It's fragile, what if the first async call throws and doesn't unset the lock.
How many times do we loop back into the timer before giving up?
How long should the timer be set for? In some environments 1 second might be way too long and inefficient.
Is there a best practise for solving this in JavaScript?
On the first call to the service, initialize an array. Start the fetch operation. Create a Promise, store it in the array.
On subsequent calls, if the data is there, return an already-fulfilled Promise. If not, add another Promise to the array and return that.
When the data arrives, resolve all the waiting Promise objects in the list. (You can throw away the list once the data's there.)
I really like the promise solution in the other answer -- very clever, very interesting. Promises aren't the dominent methodology, so you may need to educate the team. I'm going to go in another direction though.
What you're after is a memoize function -- an in-memory key/value cache of expensive results. JavaScript the Good Parts has a memoize sample towards the end. Lodash has a memoize function. These assume synchronous processing so don't account for your scenario -- which is to say they'd hit the database lots of times until one of the "threads" replied.
The async library also has a memoize function that does exactly what you want. In it's innards, it keeps a queue array of callbacks, and once it gets the answer, it both caches it and calls all the callbacks.
If you're into inventing, by all means, use promises. If you'd just like a plug-n-play answer, use async#memoize.