This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 3 years ago.
I create an object and add properties and values to it. However, when I try to iterate through the keys it says I have a size of 0.
let hash = {};
this.props.user.user.following.forEach(async topicId => {
await API.graphql(graphqlOperation(queries.getTopic, {id: topicId})).then(data => {
if(data) {
const tweetId = data.data.getTopic.post.id;
if(!hash[tweetId]){
let post = data.data.getTopic.post;
post.topics = [{
id: data.data.getTopic.id,
name: data.data.getTopic.name
}]
hash[tweetId] = post;
} else {
console.log("Found tweet. Appending to topics array.");
let post = hash[tweetId];
let topicsArr = post.topics;
topicsArr.push({
id: data.data.getTopic.id,
name: data.data.getTopic.name
})
post.topics = topicsArr;
hash[tweetId] = post;
}
}})
});
console.log("Hash: ", hash);
console.log("Map size: ", Object.keys(hash).length);
let tweets = [];
for(var key in hash) {
tweets.push(hash[key]);
}
console.log("Tweets to go into state: ", tweets);
My output is as follows(the array on the last line is empty):
You're adding to the hash object with an async function.
So, each of runs through the forEach are done asynchronously, and the evaluation of the rest of the code continues on. The block where you use the data runs before your asynchronous functions have all completed.
You should have both the code where you fetch and the code where you process data from an API call in the same asynchronous block, since, when you think about it, a block of code can't depend on asynchronous code without itself being asynchronous.
You probably want to await all of the API calls you're doing in that loop. To do so, you can use Array.prototype.map() and Promise.all().
const tweets = async () => {
let hash = {};
await Promise.all(this.props.user.user.following.map(async topicID => {
const data = await API.graphql(graphqlOperation(queries.getTopic, { id: topicID }));
// No need to use .then(), since we're awaiting anyways
if (data) {
const tweedID = data.data.getTopic.post.id;
if (!hash[tweetID]) {
let post = data.data.getTopic.post;
post.topics = [{
id: data.data.getTopic.id,
name: data.data.getTopic.name
}]
hash[tweedID] = post;
} else {
console.log("Found tweet. Appending to topics array.");
let post = hash[tweedID];
let topicsArr = post.topics;
topicsArr.push({
id: data.data.getTopic.id,
name: data.data.getTopic.name
});
post.topics = topicsArr;
hash[tweedID] = post;
}
}
}));
console.log("Hash: ", hash);
console.log("Map size: ", Object.keys(hash).length);
let tweets = [];
for (const key in hash) {
tweets.push(hash[key]);
}
console.log("Tweets to go into state: ", tweets);
}
tweets();
// Or, put the whole thing in brackets and call it immediately:
(async () => {
// ...
})();
The reason that you see that the logged object has values is because of a feature of the dev-console that live-refreshes objects' properties, so they will no longer reflect the state when they've been logged, but instead display their current value.
Consider the following example:
const obj = {};
console.log(obj);
obj.foo = 'bar';
You'll see that the obj's foo property is 'bar', even though it was undefined when console.log was actually run. As #epascarello pointed out in the comments, you can tell when your browser has done this because of the little blue [i]. If you hover over it, you'll see that it says "This value was evaluated just now".
To avoid that behavior, JSON.stringify the object before logging:
const obj = {}
console.log(JSON.stringify(obj))
obj.foo = 'bar'
This will log {}, as you might expect, since it turns it into a string and spits that text out, instead of a reference to obj.
See more here.
Related
Context: I'm fetching 'car' data below (see the code that starts with "for") from a GET request and am pushing it to the 'array' array. And for each car.ID that I get, I need to run another GET request in sequence (the GET uses car.ID as a parameter and I have no problems in doing this).
Problem: after I fetch the results from the second GET, how to push the data to the same object of the array (i.e. I want to "complement" the object above that ended on car.BrandID by adding a few more key: value pairs to the same "line")?
THANK YOU IN ADVANCE.
for (let car of carsJustObtained) {
for (i=0; i<=2; i++){
array.push(
{
timestamp: epoch,
ID : car.ID,
BrandID : car.BrandID
})
//code continues but don't worry
FULL CODE BELOW:
function gotCars(carsJustObtained) {
for (let car of carsJustObtained) {
for (i=0; i<=2; i++){
array.push(
{
timestamp: epoch,
ID : car.ID,
BrandID : car.BrandID,
ModelID : car.ModelID,
}
);
//given car.ID the second GET will be triggered because the path depends on this variable!
let path_get_all_prices = `xxx=${car.ID}?securityToken=xxx&vehiclePriceTypeID=xxx`;
let get = https.get(
{
hostname: 'xxx.com',
path: path_get_all_prices
},
(getRes) => {
console.log(`getting prices for car ${car.ID}...`);
var reply = "";
getRes.on("data", (chunk) => (reply += chunk));
const obj = JSON.parse(reply);
gotPrices(obj.Response);
}
);
function gotPrices(pricesJustObtained) {
for (let price of pricesJustObtained){
array.push(
//how to add results to the same array of the 1st GET? There are three 'prices' for each car.ID
)};
};
};
You have to find the index of your object in your array, then you can add everything you can to this object :
array[index].name = 'Hello';
There are many ways to do this. I recommend you read about array.map()
This function lets you iterate your array and in each iteration perform the get request and extend the current element.
The key is to recognise that you are not pushing, the second time
What you are doing is reading each element of the array, and adding some information to that element.
for (let car of carsJustObtained) {
const newInfo=getFurtherInformationAboutCar(car) // this is your second getting
car.newInfo1 = newInfo.param1;
car.newInfo2 = newInfo.param2;
car.newInfo3 = newInfo.param3;
}
To answer your specific question about "merging" information
If you have one set of properties already defined for the car, and you want to merge in multiple new properties, a simple way to do it is as follows:
car = { ...car, ...objectContainingNewProperties};
If your original car was {a:2, b:3, c:4} and objectContainingNewProperties was {c: 10, d:20, e:30}, the result would be:
{ a:2,
b:3,
c:10,
d:20,
e:30 }
Any same-named properties in the second object will overwrite those in the original object.
Your second request is of course asynchronous, so by the time you get its response, you have already populated your array with all information from the first request.
I would suggest to use a promise-enabled alternative to http.get, as promises are a native feature in JavaScript that makes working with asynchronous events less messy. I will show here how it can work with node-fetch.
As fetch is natively supported in browser agents, you can run the snippet below to see the result. As a demo I have used https://jsonplaceholder.typicode.com/ as a server resource: it returns JSON for several sample datasets, including todos and users. A todo has some properties (like a title) and has a user id. A user has an email and a username. So we could make the todos-request the first request, and the users-request the second one (based on the user id received in the first). So the principle is the same as with your cars and prices.
This relies heavily on promises:
// For demo, we use these two URls:
// They both need a number following it
let url1 = "https://jsonplaceholder.typicode.com/todos/";
let url2 = "https://jsonplaceholder.typicode.com/users/";
let promises = [];
// Let's say we build an array with 5 objects:
for (let i = 1; i <= 5; i++) {
promises.push(
// Make the request
fetch(url1 + i*30)
// Parse the response as JSON
.then(resp => resp.json())
// Process this data
.then(data => {
// Create our own object from this data
let obj = {
user: data.userId,
todo: data.title
};
// Make second request, to get user's email, joining it with obj
return Promise.all([obj, fetch(url2 + obj.user)])
})
.then(([obj, resp2]) => Promise.all([obj, resp2.json()]))
// Merge the new data with the old
.then(([obj, data2]) => Object.assign(obj, {
email: data2.email,
logon: data2.username
}))
);
}
// Wait for all requests to finish...
Promise.all(promises).then(results => {
console.log(results); // The result!
});
With async/await
The above can be made even more readable, if you use the async/await syntax:
let url1 = "https://jsonplaceholder.typicode.com/todos/";
let url2 = "https://jsonplaceholder.typicode.com/users/";
async function getOne(i) {
let resp = await fetch(url1 + i*30);
let data = await resp.json();
// Create object from first request
let obj = {
user: data.userId,
todo: data.title
};
// Make second request, joining it with obj
let resp2 = await fetch(url2 + obj.user);
let data2 = await resp2.json();
return Object.assign(obj, {
email: data2.email,
logon: data2.username
});
}
let promises = [];
for (let i = 1; i <= 5; i++) {
promises.push(getOne(i));
}
Promise.all(promises).then(results => {
console.log(results);
});
This is my first question on stack overflow, so bear with me.
So I have a controller function that is rendering my google maps api, and I am trying to loop through the results from mongoDB and push them to an array so that I can pass it to a script tag on the ejs. I am able to console.log(myArr) within the for loop and get results, but not outside of it (just above the res.render). I am assuming my that my problem is my result within res.render is receiving an empty array.
Please help me I have been stuck on this problem for days now. Thank you, Andrew
function showMap(req, res) {
let myArr = [];
db.collection("poppies").find().toArray(function (err, result) {
for (let i = 0; i < result.length; i++) {
myArr.push(result[i].Address);
};
});
res.render('map', {
result: JSON.stringify(myArr)
})
};
Asynchronous Javascript allows you to execute operations without waiting for the processing thread to become free.
Imagine the data loads from your DB only after 3 seconds - you should "wait" to get the value from your DB before running the next line code.
In your case you use myArr "outside" without await -Or- promises -Or- callbacks) - So the value is an empty array.
IMPORTANT: The idea of Asynchronous Javascript is a topic for a course
(No way to cover this issue by StackOverflow Answer).
Not working
length is: undefined
function helloWorld() {
let items = db.collection("newspapers").find({}).toArray();
return (items);
};
const result = helloWorld();
console.log("length is: " + result.length); /* length is: undefined */
Working
await for the value from helloWorld() function.
function helloWorld() {
let items = db.collection("newspapers").find({}).toArray();
return (items);
};
const result = await helloWorld();
console.log("length is: " + result.length); /* length is: 337 */
In practice
Promise chaining for example:
For this database:
[
{
"Newspaper": "The New York Times"
},
{
"Newspaper": "Washington Post"
}
]
const dbName = 'my_database';
await client.connect();
const db = client.db(dbName);
myArr = [];
db.collection("newspapers").find({}).toArray()
.then(
res => {
for (let i = 0; i < res.length; i++) {
myArr.push(res[i].Newspaper);
};
console.log(`The length is ${res.length} documents`) /* The length is 2 documents */
console.log(myArr); /* [ 'The New York Times', 'Washington Post' ] */
},
err => console.error(`Something went wrong: ${err}`),
);
try/catch Async/await example (Readable code pattern):
const helloWorld = (async () => {
try {
let items = await db.collection("newspapers").find({}).limit(2).toArray();
return (items);
} catch (e) {
console.error(
`Unable to establish a collection: ${e}`,
)
}
})
const result = await helloWorld();
console.log("length is: " + result.length); /* length is: 2 */
More examples here:
https://docs.mongodb.com/drivers/node/fundamentals/promises
I'm working with mongodb stitch/realm and I'm trying to modify objects inside an array with a foreach and also pushing ids into a new array.
For each object that i'm modifying, I'm also doing a query first, after the document is found I start modifying the object and then pushing the id into another array so I can use both arrays later.
The code is something like this:
exports = function(orgLoc_id, data){
var HttpStatus = require('http-status-codes');
// Access DB
const db_name = context.values.get("database").name;
const db = context.services.get("mongodb-atlas").db(db_name);
const orgLocPickupPointCollection = db.collection("organizations.pickup_points");
const orgLocStreamsCollection = db.collection("organizations.streams");
const streamsCollection = db.collection("streams");
let stream_ids = [];
data.forEach(function(stream) {
return streamsCollection.findOne({_id: stream.stream_id}, {type: 1, sizes: 1}).then(res => { //if I comment this query it will push without any problem
if(res) {
let newId = new BSON.ObjectId();
stream._id = newId;
stream.location_id = orgLoc_id;
stream.stream_type = res.type;
stream.unit_price = res.sizes[0].unit_price_dropoff;
stream._created = new Date();
stream._modified = new Date();
stream._active = true;
stream_ids.push(newId);
}
})
})
console.log('stream ids: ' + stream_ids);
//TODO
};
But when I try to log 'stream_ids' it's empty and nothing is shown. Properties stream_type and unit_price are not assigned.
I've tried promises but I haven't had success
It's an asynchronous issue. You're populating the value of the array inside a callback. But because of the nature of the event loop, it's impossible that any of the callbacks will have been called by the time the console.log is executed.
You mentioned a solution involving promises, and that's probably the right tack. For example something like the following:
exports = function(orgLoc_id, data) {
// ...
let stream_ids = [];
const promises = data.map(function(stream) {
return streamsCollection.findOne({ _id: stream.stream_id }, { type: 1, sizes: 1 })
.then(res => { //if I comment this query it will push without any problem
if (res) {
let newId = new BSON.ObjectId();
// ...
stream_ids.push(newId);
}
})
})
Promise.all(promises).then(function() {
console.log('stream ids: ' + stream_ids);
//TODO
// any code that needs access to stream_ids should be in here...
});
};
Note the change of forEach to map...that way you're getting an array of all the Promises (I'm assuming your findOne is returning a promise because of the .then).
Then you use a Promise.all to wait for all the promises to resolve, and then you should have your array.
Side note: A more elegant solution would involve returning newId inside your .then. In that case Promise.all will actually resolve with an array of the results of all the promises, which would be the values of newId.
I've got two pages I'm working on, and both return an array of objects. When I use the following code, the new results work:
this.adminService.waiversGetAll()
.subscribe((data: Waiver[]) => {
this.waivers = data;
this.waivers.forEach((e) => {
if(e.has_signed === true) {
e.url = `View`
} else {
e.url = `${e.message}`;
}
return e;
});
console.log(this.waivers);
})
}
But when I try to do the same thing with a different array (where I need to update the values of an array nested inside) I don't get updated values:
this.adminService.GetUnsignedWaivers()
.subscribe((data: Player[]) => {
console.log("data",data);
data.forEach(e => {
let record: Object = {};
for(let i = 0; i < e.waivers.length; i++) {
console.log(e.waivers[i].has_signed);
if (e.waivers[i].has_signed === true) {
e.waivers[i].url = e.waivers[i].signatureUrl;
console.log(e.waivers[i].url);
e.waivers[i].message = "View Waiver";
} else {
e.waivers[i].url = e.waivers[i].url;
e.waivers[i].message = e.waivers[i].message;
}
console.log(e.waivers[i].message);
return;
};
return e;
});
this.size = this.players.length;
console.log(this.players);
})
}
When I look at the console.log of e.waivers[i].has_signed, the data is correct, but after that it's not right.
What do I have to do to make this work? I've tried using a for loop inside the foreach, and a bunch of other stuff.
The data supplied to the loop provides info like:
{
buyer: "email#someaddress.edu"
event: "COED A"
field: "Main"
net: null
player: {shirtSize: null, avp_id: 12345678, adult: true, …}
team: null
waivers: [{
email: "someemail#gmail.com",
has_signed: true,
message: "Liability Waiver",
signatureUrl: "https://somelink.pdf",
url: "https://somelink.com/somekeyidentifier"
}
IF the player has signed the waiver, there will be a signatureUrl field and the message should say "View Waiver" instead of the message telling me what type of waiver they will sign. I want the url to be set to signatureUrl if they signed, so I can use it in a table that doesn't like manipulation of data.
A visual of what is returned in my table:
All I get is 1600 records showing the url as though everyone hasn't signed, but when I console.log has_signed in the inner loop, it's showing TRUE for the ones that should show a signatureUrl instead.
Quickly looking at it, you have a return statement within your for loop, which would stop it from running after the first iteration.
First of all drop all the return statements in your code. Next, use map instead of forEach as the former returns you the new manipulated array and the latter is used just for iteration purpose.
Your code within subscribe then becomes:
data.waivers = data.waivers.map((waiver) => {
if (waiver.has_signed) {
// your logic goes here...
waiver.url = waiver.signatureUrl;
waivers.message = "View Waiver";
}
// No else is required as you are just reassigning with same values
});
this.playerDetails = data;
At last bind this modified data in your template.
I'm just starting to play around with Puppeteer (Headless Chrome) and Nodejs. I'm scraping some test sites, and things work great when all the values are present, but if the value is missing I get an error like:
Cannot read property 'src' of null (so in the code below, the first two passes might have all values, but the third pass, there is no picture, so it just errors out).
Before I was using if(!picture) continue; but I think it's not working now because of the for loop.
Any help would be greatly appreciated, thanks!
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
let title = document.querySelector('h1').innerText;
let article = document.querySelector('.c-entry-content').innerText;
let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
let source = "The Verge";
let categories = "Tech";
if (!picture)
continue; //throws error
return {
title,
article,
picture,
source,
categories
}
});
}
let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
If there is no picture, then document.querySelector() returns null, which does not have a src property. You need to check that your query found an element before trying to read the src property.
Moving the null-check to the top of the function has the added benefit of saving unnecessary calculations when you are just going to bail out anyway.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
const pictureElement = document.querySelector('.c-picture img');
if (!pictureElement) return null;
const picture = pictureElement.src;
const title = document.querySelector('h1').innerText;
const article = document.querySelector('.c-entry-content').innerText;
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
if (!result) continue;
// ... do stuff with result
}
Answering comment question: "Is there a way just to skip anything blank, and return the rest?"
Yes. You just need to check the existence of each element that could be missing before trying to read a property off of it. In this case we can omit the early return since you're always interested in all the results.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
const result = await page.evaluate(() => {
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1');
const content = document.querySelector('.c-entry-content');
const picture = img ? img.src : '';
const title = h1 ? h1.innerText : '';
const article = content ? content.innerText : '';
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
// ...
}
}
Further thoughts
Since I'm still on this question, let me take this one step further, and refactor it a bit with some higher level techniques you might be interested in. Not sure if this is exactly what you are after, but it should give you some ideas about writing more maintainable code.
// Generic reusable helper to return an object property
// if object exists and has property, else a default value
//
// This is a curried function accepting one argument at a
// time and capturing each parameter in a closure.
//
const maybeGetProp = default => key => object =>
(object && object.hasOwnProperty(key)) ? object.key : default
// Pass in empty string as the default value
//
const getPropOrEmptyString = maybeGetProp('')
// Apply the second parameter, the property name, making 2
// slightly different functions which have a default value
// and a property name pre-loaded. Both functions only need
// an object passed in to return either the property if it
// exists or an empty string.
//
const maybeText = getPropOrEmptyString('innerText')
const maybeSrc = getPropOrEmptyString('src')
async function scrape3() {
// ...
// The _ parameter name is acknowledging that we expect a
// an argument passed in but saying we plan to ignore it.
//
const evaluate = _ => page.evaluate(() => {
// Attempt to retrieve the desired elements
//
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1')
const content = document.querySelector('.c-entry-content')
// Return the results, with empty string in
// place of any missing properties.
//
return {
title: maybeText(h1),
article: maybeText(article),
picture: maybeSrc(img),
source: 'The Verge',
categories: 'Tech'
}
}))
// Start with an empty array of length 3
//
const evaluations = Array(3).fill()
// Then map over that array ignoring the undefined
// input and return a promise for a page evaluation
//
.map(evaluate)
// All 3 scrapes are occuring concurrently. We'll
// wait for all of them to finish.
//
const results = await Promise.all(evaluations)
// Now we have an array of results, so we can
// continue using array methods to iterate over them
// or otherwise manipulate or transform them
//
results
.filter(result => result.title && result.picture)
.forEach(result => {
//
// Do something with each result
//
})
}
Try-catch worked for me:
try {
if (await page.$eval('element')!==null) {
const name = await page.$eval('element')
}
}catch(error){
name = ''
}