I have a function that receives an array of usernames. For each one I need to get the connections IDs and concatenate into another array. But I think the way I did have some problems of race condition that may colide and concatenate less results that it should...
const getConnections = async function (usernames) {
let connections = [];
await Promise.all(usernames.map(async (username) => {
try {
let connsFound = await dynamo.getConnectionsByUsername(username);
if (connsFound && connsFound.length > 0)
connections = connections.concat(connsFound);
} catch (error) {
console.error('ERROR GET CONNECTIONS', error)
}
}));
return connections;
};
the result of the connections its an array, so the result I would like to merge to the var connections...but .concat does not merge, it creates a new array, so I need to do a 'var = var.concat(newArray)'
I am affraid this is not safe and in some operation it will colide and overwrite some results...
Is there a better way to do it?
cheers
JavaScript is single threaded, so there is always at most one function that runs and accesses connections. You shouldn't have any problems.
However, there is no reason to do it that way. Since you are already using async/await, you can just create an array of arrays and flatten that:
const getConnections = async function (usernames) {
const connections = await Promise.all(usernames.map(async (username) => {
try {
return await dynamo.getConnectionsByUsername(username);
} catch (error) {
console.error('ERROR GET CONNECTIONS', error);
return [];
}
}));
return connections.flat();
};
.flat is relatively new, but it should be easy to write a custom helper function to achieve the same.
Related
I'm playing with the Rick and Morty API and I want to get all of the universe's characters
into an array so I don't have to make more API calls to work the rest of my code.
The endpoint https://rickandmortyapi.com/api/character/ returns the results in pages, so
I have to use recursion to get all the data in one API call.
I can get it to spit out results into HTML but I can't seem to get a complete array of JSON objects.
I'm using some ideas from
Axios recursion for paginating an api with a cursor
I translated the concept for my problem, and I have it posted on my Codepen
This is the code:
async function populatePeople(info, universePeople){ // Retrieve the data from the API
let allPeople = []
let check = ''
try {
return await axios.get(info)
.then((res)=>{
// here the current page results is in res.data.results
for (let i=0; i < res.data.results.length; i++){
item.textContent = JSON.stringify(res.data.results[i])
allPeople.push(res.data.results[i])
}
if (res.data.info.next){
check = res.data.info.next
return allPeople.push(populatePeople(res.data.info.next, allPeople))
}
})
} catch (error) {
console.log(`Error: ${error}`)
} finally {
return allPeople
}
}
populatePeople(allCharacters)
.then(data => console.log(`Final data length: ${data.length}`))
Some sharp eyes and brains would be helpful.
It's probably something really simple and I'm just missing it.
The following line has problems:
return allPeople.push(populatePeople(res.data.info.next, allPeople))
Here you push a promise object into allPeople, and as .push() returns a number, you are returning a number, not allPeople.
Using a for loop to push individual items from one array to another is really a verbose way of copying an array. The loop is only needed for the HTML part.
Also, you are mixing .then() with await, which is making things complex. Just use await only. When using await, there is no need for recursion any more. Just replace the if with a loop:
while (info) {
....
info = res.data.info.next;
}
You never assign anything to universePeople. You can drop this parameter.
Instead of the plain for loop, you can use the for...of syntax.
As from res you only use the data property, use a variable for that property only.
So taking all that together, you get this:
async function populatePeople(info) {
let allPeople = [];
try {
while (info) {
let {data} = await axios.get(info);
for (let content of data.results) {
const item = document.createElement('li');
item.textContent = JSON.stringify(content);
denizens.append(item);
}
allPeople.push(...data.results);
info = data.info.next;
}
} catch (error) {
console.log(`Error: ${error}`)
} finally {
section.append(denizens);
return allPeople;
}
}
Here is working example for recursive function
async function getAllCharectersRecursively(URL,results){
try{
const {data} = await axios.get(URL);
// concat current page results
results =results.concat(data.results)
if(data.info.next){
// if there is next page call recursively
return await getAllCharectersRecursively(data.info.next,results)
}
else{
// at last page there is no next page so return collected results
return results
}
}
catch(e){
console.log(e)
}
}
async function main(){
let results = await getAllCharectersRecursively("https://rickandmortyapi.com/api/character/",[])
console.log(results.length)
}
main()
I hesitate to offer another answer because Trincot's analysis and answer is spot-on.
But I think a recursive answer here can be quite elegant. And as the question was tagged with "recursion", it seems worth presenting.
const populatePeople = async (url) => {
const {info: {next}, results} = await axios .get (url)
return [...results, ...(next ? await populatePeople (next) : [])]
}
populatePeople ('https://rickandmortyapi.com/api/character/')
// or wrap in an `async` main, or wait for global async...
.then (people => console .log (people .map (p => p .name)))
.catch (console .warn)
.as-console-wrapper {max-height: 100% !important; top: 0}
<script>/* dummy */ const axios = {get: (url) => fetch (url) .then (r => r .json ())} </script>
This is only concerned with fetching the data. Adding it to your DOM should be a separate step, and it shouldn't be difficult.
Update: Explanation
A comment indicated that this is hard to parse. There are two things that I imagine might be tricky here:
First is the object destructuring in {info: {next}, results} = <...>. This is just a nice way to avoid using intermediate variables to calculate the ones we actually want to use.
The second is the spread syntax in return [...results, ...<more>]. This is a simpler way to build an array than using .concat or .push. (There's a similar feature for objects.)
Here's another version doing the same thing, but with some intermediate variables and an array concatenation instead. It does the same thing:
const populatePeople = async (url) => {
const response = await axios .get (url)
const next = response .info && response .info .next
const results = response .results || []
const subsequents = next ? await populatePeople (next) : []
return results .concat (subsequents)
}
I prefer the original version. But perhaps you would find this one more clear.
Problem
I'm using a for loop to create and save several documents and then push their ids as a reference to another document, which I later try to save.
However, when I execute the code, the final document save occurs before the for loop, which means the references never get saved.
How to get the for loop to finish before saving another document (in my case, the entry)? Or is there an entirely different, better way to do what I'm trying to accomplish?
What I've tried
I tried separating the code into two async-await functions, I also tried separating the code into a function with a callback that calls the other after its done. I'm currently trying to make sense of Promises more deeply. I'm also exploring whether it's possible to set up a flag or a small time delay.
Template.findById(templateId, function (err, template) {
if (err) { return console.log(err) }
let entry = new Entry();
entry.entryState = "Draft";
entry.linkedTemplateId = template._id;
for (var i = 0; i < template.components.length; i++) {
let component = template.components[i]
let entryComp = new entryComponent({
componentOrder: component.componentOrder,
componentType: component.componentType,
templateComponentId: component._id,
entryId: entry._id
})
entryComp.save(function(err){
if (err) { return console.log(err) }
entry.entryComponents.push(entryComp._id);
});
}
entry.save(function(err){
if (err) { return console.log(err) }
});
})
I like to use recursion for this purpose. Though I am very confident that someone else already thought of this before me, I did discover this solution on my own. Since I figured this out on my own, I am not sure if this is the best way to do things. But I have used it and it works well in my applications.
Template.findById(templateId, function (err, template) {
if (err) { return console.log(err) }
let entry = new Entry();
entry.entryState = "Draft";
entry.linkedTemplateId = template._id;
var fsm = function(i){
if (i<template.components.length)
{
let component = template.components[i]
let entryComp = new entryComponent({
componentOrder: component.componentOrder,
componentType: component.componentType,
templateComponentId: component._id,
entryId: entry._id
})
entryComp.save(function(err){
if (err) { return console.log(err) }
entry.entryComponents.push(entryComp._id);
//cause the fsm state change when done saving component
fsm(i+1)
});
}
else
{
//the last index i=length-1 has been exhausted, and now i= length, so
//so do not save a component, instead save the final entry
entry.save(function(err){
if (err) { return console.log(err) }
});
}
}
//start the fsm
fsm(0)
})
In short, I would use recursion to solve the problem. However, there may be a concern of exceeding the recursion stack limit if the array of components is very large. Consider limiting the number of items allowed in this application.
You can use the new "for of" loop in javascript along with async await to simplify your answer:-
Template.findById(templateId, async function (err, template) {
try {
if (err) { return console.log(err) }
let entry = new Entry();
entry.entryState = "Draft";
entry.linkedTemplateId = template._id;
for(const component of template.components) {
let entryComp = new entryComponent({
componentOrder: component.componentOrder,
componentType: component.componentType,
templateComponentId: component._id,
entryId: entry._id
})
await entryComp.save();
entry.entryComponents.push(entryComp._id);
}
await entry.save();
} catch (error) {
// handle error
}
})
Another way of doing it is using normal "for loop" with Promise.all and this one will be more fast then the other one as you won't be waiting for one entryComp to save before going further:-
Template.findById(templateId, async function (err, template) {
try {
if (err) { return console.log(err) }
let entry = new Entry();
entry.entryState = "Draft";
entry.linkedTemplateId = template._id;
const promises = [];
for (var i = 0; i < template.components.length; i++) {
let component = template.components[i]
let entryComp = new entryComponent({
componentOrder: component.componentOrder,
componentType: component.componentType,
templateComponentId: component._id,
entryId: entry._id
})
promises.push(entryComp.save());
entry.entryComponents.push(entryComp._id);
}
await Promise.all(promises);
await entry.save();
} catch (error) {
// handle error
}
})
Also, if you are performing multi document operations which are dependent on one another the correct way to perform these operations is through mongodb transactions which will provide atomicity. Take a look at them here https://docs.mongodb.com/master/core/transactions/
It seems like there is a cleaner and more optimised way to query a Firestore collection, call doc.data() on each doc, and then return an array as result. The order in which the docs are pushed into the result array feels haphazard.
There are many steps to this code:
A new result variable is created
A query is made to retrieve the 'stories' collection
For each doc, we call the doc.data()
We push each doc to the result array
Return the result array
function getStories() {
var result = [];
db.collection('stories').get().then(querySnapshot => {
querySnapshot.forEach(doc => result.push(doc.data()));
})
return result;
}
The code works fine but it seems like we can write this code in a cleaner way with less steps.
The code you shared actually doesn't work, as George comment. Since Firestore loads data asynchronously, you're always returns the array before the data has been loaded. So your array will be empty.
In code:
function getStories() {
var result = [];
db.collection('stories').get().then(querySnapshot => {
querySnapshot.forEach(doc => result.push(doc.data()));
})
return result;
}
var result = getStories();
console.log(result.length);
Will log:
0
To fix this, you want to return a promise, which resolves once the data has loaded. Something along these lines:
function getStories() {
var result = [];
return db.collection('stories').get().then(querySnapshot => {
querySnapshot.forEach(doc => result.push(doc.data()));
return result;
})
}
So this basically added two return statements, which makes your result bubble up and then be returned as the promise from getStories. To invoke this, you'd do:
getStories().then(result => {
console.log(result.length);
})
Which will then log the correct number of results.
First, define a map function for use with firebase (named mapSnapshot because it is meant specifically for use with firebase):
const mapSnapshot = f => snapshot => {
const r = [];
snapshot.forEach(x => { r.push(f(x)); });
return r;
}
Then, you can just work with Promise and mapSnapshot:
function getStories() {
return db.collection('stories').
get().
then(mapSnapshot(doc => doc.data()));
}
Usage example:
getStories().then(docs => ... do whatever with docs ...)
To be honest, it isn't less code if used as a one-time solution. But the neat thing here is, it allows to create reusable abstractions. So for example, you could use mapSnapshot to create a snapshotToArray function which can be reused whenever you need to convert the DataSnapshot from firebase to a normal Array:
const snapshotToArray = mapSnapshot(x => x);
Please note: This doesn't depend on any collection name whatsoever. You can use it with any DataSnapshot from any collection to convert it into an Array!
That's not all. You can just as easily create a function from the contents of a DataSnapshot into a regular Array with the contents in it:
const readDocsData = mapSnapshot(x => x.data());
Again, this doesn't look like a big deal – until you realize, you can also create a fromFirebase function, which allows to query various datasets:
const fromFirebase = (name, transform = x => x) => {
return db.collection(name).get().then(transform);
}
And which you can then use like this:
fromFirebase('stories', readDocsData).then(docs => {
// do what you want to do with docs
});
This way, the final result has less steps you as a programmer notice immediatly. But it produced (although reusable) several intermediate steps, each hiding a bit of abstraction.
I have the following code where the modifying the db item base on an inner promise is not working.
$('input[type=checkbox][name=checklist]:checked').each(function()
{
var collection=db.items.where("name").equals(checkbox.val());
var success;
collection.modify(function(item){
var invokePromise;
//invokePromise = a fucntion that returns a promise
//this invokepromise function needs the item from the db.
invokePromise.then(function(thirdPartyResponse){
item.date=new Date();
item.attempts= item.attempts+1; <-- this is not being updated.
}).catch(function(error){
delete this.value; <-- this is also not deleted
});
});
});
Considering #David Fahlander answer that Collection.modify() must synchronously update the item, you should collect the async responses first and after alter the database. You can use Promise.all() to asynchronously collect the responses from the invokePromise and modify the database in one go afterwards.
A callback given to Collection.modify() must synchronously update the item. You could also optimize the query using anyOf() instead of equasls. Here's an example that examplifies another strategy:
function yourAction () {
const checkedValues = $('input[type=checkbox][name=checklist]:checked')
.toArray() // Convert to a standard array of elements
.map(checkBox => checkBox.value); // Convert to an array of checked values
return invokePromise.then(thirdPartyResponse => {
return db.items.where("name").anyOf(checkedValues).modify(item => {
item.date = new Date();
++item.attempts;
}).catch(error => {
console.error("Failed to update indexedDB");
throw error;
});
}).catch(error => {
// Handle an errors from invokePromise
// If an error occurs, delete the values. Was this your intent?
console.error("Error occurred. Now deleting values instead", error);
return db.items.where("name").anyOf(checkedValues).delete();
});
}
You could retrieve all the entries, wait for the promises and then update them seperately:
(async function() {
const entries = await db.items
.where("name").equals(checkbox.val())
.toArray();
for(const entry of entries) {
//...
await invokePromise;
await db.items.put(entry);
}
})();
You might want to parallelize the whole thing with entries.map and Promise.all and Table.putAll.
Currently, I am trying to get the md5 of every value in array. Essentially, I loop over every value and then hash it, as such.
var crypto = require('crypto');
function userHash(userIDstring) {
return crypto.createHash('md5').update(userIDstring).digest('hex');
}
for (var userID in watching) {
refPromises.push(admin.database().ref('notifications/'+ userID).once('value', (snapshot) => {
if (snapshot.exists()) {
const userHashString = userHash(userID)
console.log(userHashString.toUpperCase() + "this is the hashed string")
if (userHashString.toUpperCase() === poster){
return console.log("this is the poster")
}
else {
..
}
}
else {
return null
}
})
)}
However, this leads to two problems. The first is that I am receiving the error warning "Don't make functions within a loop". The second problem is that the hashes are all returning the same. Even though every userID is unique, the userHashString is printing out the same value for every user in the console log, as if it is just using the first userID, getting the hash for it, and then printing it out every time.
Update LATEST :
exports.sendNotificationForPost = functions.firestore
.document('posts/{posts}').onCreate((snap, context) => {
const value = snap.data()
const watching = value.watchedBy
const poster = value.poster
const postContentNotification = value.post
const refPromises = []
var crypto = require('crypto');
function userHash(userIDstring) {
return crypto.createHash('md5').update(userIDstring).digest('hex');
}
for (let userID in watching) {
refPromises.push(admin.database().ref('notifications/'+ userID).once('value', (snapshot) => {
if (snapshot.exists()) {
const userHashString = userHash(userID)
if (userHashString.toUpperCase() === poster){
return null
}
else {
const payload = {
notification: {
title: "Someone posted something!",
body: postContentNotification,
sound: 'default'
}
};
return admin.messaging().sendToDevice(snapshot.val(), payload)
}
}
else {
return null
}
})
)}
return Promise.all(refPromises);
});
You have a couple issues going on here. First, you have a non-blocking asynchronous operation inside a loop. You need to fully understand what that means. Your loop runs to completion starting a bunch of non-blocking, asynchronous operations. Then, when the loop finished, one by one your asynchronous operations finish. That is why your loop variable userID is sitting on the wrong value. It's on the terminal value when all your async callbacks get called.
You can see a discussion of the loop variable issue here with several options for addressing that:
Asynchronous Process inside a javascript for loop
Second, you also need a way to know when all your asynchronous operations are done. It's kind of like you sent off 20 carrier pigeons with no idea when they will all bring you back some message (in any random order), so you need a way to know when all of them have come back.
To know when all your async operations are done, there are a bunch of different approaches. The "modern design" and the future of the Javascript language would be to use promises to represent your asynchronous operations and to use Promise.all() to track them, keep the results in order, notify you when they are all done and propagate any error that might occur.
Here's a cleaned-up version of your code:
const crypto = require('crypto');
exports.sendNotificationForPost = functions.firestore.document('posts/{posts}').onCreate((snap, context) => {
const value = snap.data();
const watching = value.watchedBy;
const poster = value.poster;
const postContentNotification = value.post;
function userHash(userIDstring) {
return crypto.createHash('md5').update(userIDstring).digest('hex');
}
return Promise.all(Object.keys(watching).map(userID => {
return admin.database().ref('notifications/' + userID).once('value').then(snapshot => {
if (snapshot.exists()) {
const userHashString = userHash(userID);
if (userHashString.toUpperCase() === poster) {
// user is same as poster, don't send to them
return {response: null, user: userID, poster: true};
} else {
const payload = {
notification: {
title: "Someone posted something!",
body: postContentNotification,
sound: 'default'
}
};
return admin.messaging().sendToDevice(snapshot.val(), payload).then(response => {
return {response, user: userID};
}).catch(err => {
console.log("err in sendToDevice", err);
// if you want further processing to stop if there's a sendToDevice error, then
// uncomment the throw err line and remove the lines after it.
// Otherwise, the error is logged and returned, but then ignored
// so other processing continues
// throw err
// when return value is an object with err property, caller can see
// that that particular sendToDevice failed, can see the userID and the error
return {err, user: userID};
});
}
} else {
return {response: null, user: userID};
}
});
}));
});
Changes:
Move require() out of the loop. No reason to call it multiple times.
Use .map() to collect the array of promises for Promise.all().
Use Object.keys() to get an array of userIDs from the object keys so we can then use .map() on it.
Use .then() with .once().
Log sendToDevice() error.
Use Promise.all() to track when all the promises are done
Make sure all promise return paths return an object with some common properties so the caller can get a full look at what happened for each user
These are not two problems: the warning you get is trying to help you solve the second problem you noticed.
And the problem is: in Javascript, only functions create separate scopes - every function you define inside a loop - uses the same scope. And that means they don't get their own copies of the relevant loop variables, they share a single reference (which, by the time the first promise is resolved, will be equal to the last element of the array).
Just replace for with .forEach.