Context: I'm fetching 'car' data below (see the code that starts with "for") from a GET request and am pushing it to the 'array' array. And for each car.ID that I get, I need to run another GET request in sequence (the GET uses car.ID as a parameter and I have no problems in doing this).
Problem: after I fetch the results from the second GET, how to push the data to the same object of the array (i.e. I want to "complement" the object above that ended on car.BrandID by adding a few more key: value pairs to the same "line")?
THANK YOU IN ADVANCE.
for (let car of carsJustObtained) {
for (i=0; i<=2; i++){
array.push(
{
timestamp: epoch,
ID : car.ID,
BrandID : car.BrandID
})
//code continues but don't worry
FULL CODE BELOW:
function gotCars(carsJustObtained) {
for (let car of carsJustObtained) {
for (i=0; i<=2; i++){
array.push(
{
timestamp: epoch,
ID : car.ID,
BrandID : car.BrandID,
ModelID : car.ModelID,
}
);
//given car.ID the second GET will be triggered because the path depends on this variable!
let path_get_all_prices = `xxx=${car.ID}?securityToken=xxx&vehiclePriceTypeID=xxx`;
let get = https.get(
{
hostname: 'xxx.com',
path: path_get_all_prices
},
(getRes) => {
console.log(`getting prices for car ${car.ID}...`);
var reply = "";
getRes.on("data", (chunk) => (reply += chunk));
const obj = JSON.parse(reply);
gotPrices(obj.Response);
}
);
function gotPrices(pricesJustObtained) {
for (let price of pricesJustObtained){
array.push(
//how to add results to the same array of the 1st GET? There are three 'prices' for each car.ID
)};
};
};
You have to find the index of your object in your array, then you can add everything you can to this object :
array[index].name = 'Hello';
There are many ways to do this. I recommend you read about array.map()
This function lets you iterate your array and in each iteration perform the get request and extend the current element.
The key is to recognise that you are not pushing, the second time
What you are doing is reading each element of the array, and adding some information to that element.
for (let car of carsJustObtained) {
const newInfo=getFurtherInformationAboutCar(car) // this is your second getting
car.newInfo1 = newInfo.param1;
car.newInfo2 = newInfo.param2;
car.newInfo3 = newInfo.param3;
}
To answer your specific question about "merging" information
If you have one set of properties already defined for the car, and you want to merge in multiple new properties, a simple way to do it is as follows:
car = { ...car, ...objectContainingNewProperties};
If your original car was {a:2, b:3, c:4} and objectContainingNewProperties was {c: 10, d:20, e:30}, the result would be:
{ a:2,
b:3,
c:10,
d:20,
e:30 }
Any same-named properties in the second object will overwrite those in the original object.
Your second request is of course asynchronous, so by the time you get its response, you have already populated your array with all information from the first request.
I would suggest to use a promise-enabled alternative to http.get, as promises are a native feature in JavaScript that makes working with asynchronous events less messy. I will show here how it can work with node-fetch.
As fetch is natively supported in browser agents, you can run the snippet below to see the result. As a demo I have used https://jsonplaceholder.typicode.com/ as a server resource: it returns JSON for several sample datasets, including todos and users. A todo has some properties (like a title) and has a user id. A user has an email and a username. So we could make the todos-request the first request, and the users-request the second one (based on the user id received in the first). So the principle is the same as with your cars and prices.
This relies heavily on promises:
// For demo, we use these two URls:
// They both need a number following it
let url1 = "https://jsonplaceholder.typicode.com/todos/";
let url2 = "https://jsonplaceholder.typicode.com/users/";
let promises = [];
// Let's say we build an array with 5 objects:
for (let i = 1; i <= 5; i++) {
promises.push(
// Make the request
fetch(url1 + i*30)
// Parse the response as JSON
.then(resp => resp.json())
// Process this data
.then(data => {
// Create our own object from this data
let obj = {
user: data.userId,
todo: data.title
};
// Make second request, to get user's email, joining it with obj
return Promise.all([obj, fetch(url2 + obj.user)])
})
.then(([obj, resp2]) => Promise.all([obj, resp2.json()]))
// Merge the new data with the old
.then(([obj, data2]) => Object.assign(obj, {
email: data2.email,
logon: data2.username
}))
);
}
// Wait for all requests to finish...
Promise.all(promises).then(results => {
console.log(results); // The result!
});
With async/await
The above can be made even more readable, if you use the async/await syntax:
let url1 = "https://jsonplaceholder.typicode.com/todos/";
let url2 = "https://jsonplaceholder.typicode.com/users/";
async function getOne(i) {
let resp = await fetch(url1 + i*30);
let data = await resp.json();
// Create object from first request
let obj = {
user: data.userId,
todo: data.title
};
// Make second request, joining it with obj
let resp2 = await fetch(url2 + obj.user);
let data2 = await resp2.json();
return Object.assign(obj, {
email: data2.email,
logon: data2.username
});
}
let promises = [];
for (let i = 1; i <= 5; i++) {
promises.push(getOne(i));
}
Promise.all(promises).then(results => {
console.log(results);
});
Related
Context
I'm retrieving data from the ESPN API to fetch weekly NFL matchup data. So, I'm making 18 api calls each time I need to fetch this data to account for all 18 weeks in the NFL season. I'm then creating an array with the data I need from the responses to those calls and writing out 18 files that align with each week in the NFL season (week1.json, week2.json, etc.).
Problem
The problem is that when I call my endpoint, I am seeing 2 things intermittently, and not necessarily at the same time:
(1) Some of the json files(week1.json, week2.json, etc.) include only a portion of the expected array. So, instead of 16 objects in the array, I may see only 4, or only 6, etc. Why would I only see a portion of the response data written to the array that's ultimately written to the .json files?
(2) Not all files are written to each time the endpoint is called. So, I may see that only week1-week5's .json files are written. Why aren't all of them updated?
Problem Code
// iterate 18 times
for (let i = 0; i < 18; i++) {
let weekNumber;
weekNumber = i + 1;
const week = fs.readFileSync(`./pickem/week${weekNumber}.json`, 'utf8');
const weekJson = JSON.parse(week);
// empty weekJson.games array
weekJson.games = []
// get all items
axios.get(`https://sports.core.api.espn.com/v2/sports/football/leagues/nfl/seasons/2022/types/2/weeks/${weekNumber}/events?lang=en®ion=us`)
.then(response => {
const schedule = [];
// get all items from response
const items = response.data.items
// console.log(response.data.items)
items.forEach(item => {
// make get call to $ref
axios.get(item.$ref)
.then(response => {
// get name
const name = response.data.name
// get date
const date = response.data.date
// get event id
const eventid = response.data.id
// get team ids
let team1 = response.data.competitions[0].competitors[0].id
let team2 = response.data.competitions[0].competitors[1].id
// create new object
const newObject = {
name: name,
date: date,
eventid: eventid,
team1: team1,
team2: team2
}
// add games for week
weekJson.games.push(newObject);
fs.writeFileSync(`./pickem/week${weekNumber}.json`, JSON.stringify(weekJson));
})
.catch(error => {
console.log(error)
})
})
}).catch(error => {
console.log(error)
})
}
Updated Code
router.get('/getschedules', (req, res) => {
async function writeGames() {
// iterate 18 times
for (let i = 0; i < 18; i++) {
let weekNumber;
weekNumber = i + 1;
const week = fs.readFileSync(`./pickem/week${weekNumber}.json`, 'utf8');
const weekJson = JSON.parse(week);
// empty weekJson.games array
weekJson.games = []
// get all items
// Add await keyword to wait for a week to be processed before going to the next one
await axios.get(`https://sports.core.api.espn.com/v2/sports/football/leagues/nfl/seasons/2022/types/2/weeks/${weekNumber}/events?lang=en®ion=us`)
.then(async (response) => { // add async to be able to use await
const schedule = [];
// get all items from response
const items = response.data.items
console.log(response.data.items)
// Use standard loop to be able to benefit from async/await
for (let item of items) {
// make get call to $ref
// wait for an item to be processed before going to the next one
await axios.get(item.$ref)
.then(response => {
// get name
const name = response.data.name
// get date
const date = response.data.date
// get event id
const eventid = response.data.id
// get team ids
let team1 = response.data.competitions[0].competitors[0].id
let team2 = response.data.competitions[0].competitors[1].id
// create new object
const newObject = {
name: name,
date: date,
eventid: eventid,
team1: team1,
team2: team2
}
// add games for week
weekJson.games.push(newObject);
})
.catch(error => {
console.log(error)
})
}
// moved out of the for loop since you only need to write this once
fs.writeFileSync(`./pickem/week${weekNumber}.json`, JSON.stringify(weekJson));
}).catch(error => {
console.log(error)
})
}
}
writeGames();
})
Your issue might come from the fact that you are looping over an array of item that triggers parallel asynchronous calls and write weekJson before you get the entire data. (But theoretically your code should work if writeSyncFile is really synchronous, maybe there are locks on the file system that prevents node to write properly?)
You could try to make everything sequential and only write weekJson once instead of everytime you go over an item:
EDIT
I updated my original code proposition by keeping parallel calls and it worked for me (it's similar to OP's code but I only write the json file once per week).
Then I tried to run OP's code and it was working fine as well. So this makes me think that the problem isn't from the code itself but rather how it's called. As a pure node script, there doesn't seem to be any issue. But I just noticed that OP is using it server side as the result of an API call.
Having an API write so many JSON concurrently is probably not the best idea (especially if the api is called multiple times almost simultaneously). You could either
just return the games in the response
or precompute the results
or fetch and write them only once then cache the result to be reused
Then I wonder if due to the server context, there is not some kind of timeout since OP said that with my initial solution, only the first week was created.
const axios = require("axios");
const fs = require("fs");
async function writeGames() {
const writeWeekGamesPromises = [];
// iterate 18 times
for (let weekNumber = 1; weekNumber < 19; weekNumber++) {
// give week a default value in case the json file doesn't exist (for repro purpose)
let week = "{}";
try {
week = fs.readFileSync(`./pickem/week${weekNumber}.json`, "utf8");
} catch (e) {
console.log(`error reading week ${weekNumber} json file:`, e);
// file doesn't exist yet
}
const weekJson = JSON.parse(week);
// empty weekJson.games array
const games = [];
weekJson.games = games;
// get all items
// Add await keyword to wait for a week to be processed before going to the next one
writeWeekGamesPromises.push(axios
.get(
`https://sports.core.api.espn.com/v2/sports/football/leagues/nfl/seasons/2022/types/2/weeks/${weekNumber}/events?lang=en®ion=us`
)
.then(async (eventListResponse) => {
// add async to be able to use await
const schedule = [];
console.log(JSON.stringify(eventListResponse.data),'\n');
// get all items from response
const items = eventListResponse.data.items;
// console.log(eventListResponse.data.items); // this seems to be useless since we log the whole data just above
// parallelize calls and wait for all games from a week to be fetched before writing the file
await Promise.all(
items.map((item) => {
// we return the promise so that Promise.all will wait for all games to be pushed before going on writing the file
return axios
.get(item.$ref)
.then((response) => {
// get name, date and eventid
const {name, date, id: eventid} = response.data;
// get team ids
let team1 = response.data.competitions[0].competitors[0].id;
let team2 = response.data.competitions[0].competitors[1].id;
games.push({ name, date, eventid, team1, team2 });
})
.catch((error) => {
console.log(error);
});
})
);
// Now that all game data is ready, write in the file
fs.writeFileSync(
`./pickem/week${weekNumber}.json`,
JSON.stringify(weekJson)
);
})
.catch((error) => {
console.log(error);
}));
}
// Waiting for all games from all weeks to be processed
await Promise.all(writeWeekGamesPromises);
}
async function runAndLogTime() {
const start = Date.now();
await writeGames();
console.log(`took ${(Date.now() - start) / 1000}s to write all json files`);
}
runAndLogTime();
I'm working with mongodb stitch/realm and I'm trying to modify objects inside an array with a foreach and also pushing ids into a new array.
For each object that i'm modifying, I'm also doing a query first, after the document is found I start modifying the object and then pushing the id into another array so I can use both arrays later.
The code is something like this:
exports = function(orgLoc_id, data){
var HttpStatus = require('http-status-codes');
// Access DB
const db_name = context.values.get("database").name;
const db = context.services.get("mongodb-atlas").db(db_name);
const orgLocPickupPointCollection = db.collection("organizations.pickup_points");
const orgLocStreamsCollection = db.collection("organizations.streams");
const streamsCollection = db.collection("streams");
let stream_ids = [];
data.forEach(function(stream) {
return streamsCollection.findOne({_id: stream.stream_id}, {type: 1, sizes: 1}).then(res => { //if I comment this query it will push without any problem
if(res) {
let newId = new BSON.ObjectId();
stream._id = newId;
stream.location_id = orgLoc_id;
stream.stream_type = res.type;
stream.unit_price = res.sizes[0].unit_price_dropoff;
stream._created = new Date();
stream._modified = new Date();
stream._active = true;
stream_ids.push(newId);
}
})
})
console.log('stream ids: ' + stream_ids);
//TODO
};
But when I try to log 'stream_ids' it's empty and nothing is shown. Properties stream_type and unit_price are not assigned.
I've tried promises but I haven't had success
It's an asynchronous issue. You're populating the value of the array inside a callback. But because of the nature of the event loop, it's impossible that any of the callbacks will have been called by the time the console.log is executed.
You mentioned a solution involving promises, and that's probably the right tack. For example something like the following:
exports = function(orgLoc_id, data) {
// ...
let stream_ids = [];
const promises = data.map(function(stream) {
return streamsCollection.findOne({ _id: stream.stream_id }, { type: 1, sizes: 1 })
.then(res => { //if I comment this query it will push without any problem
if (res) {
let newId = new BSON.ObjectId();
// ...
stream_ids.push(newId);
}
})
})
Promise.all(promises).then(function() {
console.log('stream ids: ' + stream_ids);
//TODO
// any code that needs access to stream_ids should be in here...
});
};
Note the change of forEach to map...that way you're getting an array of all the Promises (I'm assuming your findOne is returning a promise because of the .then).
Then you use a Promise.all to wait for all the promises to resolve, and then you should have your array.
Side note: A more elegant solution would involve returning newId inside your .then. In that case Promise.all will actually resolve with an array of the results of all the promises, which would be the values of newId.
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 3 years ago.
I create an object and add properties and values to it. However, when I try to iterate through the keys it says I have a size of 0.
let hash = {};
this.props.user.user.following.forEach(async topicId => {
await API.graphql(graphqlOperation(queries.getTopic, {id: topicId})).then(data => {
if(data) {
const tweetId = data.data.getTopic.post.id;
if(!hash[tweetId]){
let post = data.data.getTopic.post;
post.topics = [{
id: data.data.getTopic.id,
name: data.data.getTopic.name
}]
hash[tweetId] = post;
} else {
console.log("Found tweet. Appending to topics array.");
let post = hash[tweetId];
let topicsArr = post.topics;
topicsArr.push({
id: data.data.getTopic.id,
name: data.data.getTopic.name
})
post.topics = topicsArr;
hash[tweetId] = post;
}
}})
});
console.log("Hash: ", hash);
console.log("Map size: ", Object.keys(hash).length);
let tweets = [];
for(var key in hash) {
tweets.push(hash[key]);
}
console.log("Tweets to go into state: ", tweets);
My output is as follows(the array on the last line is empty):
You're adding to the hash object with an async function.
So, each of runs through the forEach are done asynchronously, and the evaluation of the rest of the code continues on. The block where you use the data runs before your asynchronous functions have all completed.
You should have both the code where you fetch and the code where you process data from an API call in the same asynchronous block, since, when you think about it, a block of code can't depend on asynchronous code without itself being asynchronous.
You probably want to await all of the API calls you're doing in that loop. To do so, you can use Array.prototype.map() and Promise.all().
const tweets = async () => {
let hash = {};
await Promise.all(this.props.user.user.following.map(async topicID => {
const data = await API.graphql(graphqlOperation(queries.getTopic, { id: topicID }));
// No need to use .then(), since we're awaiting anyways
if (data) {
const tweedID = data.data.getTopic.post.id;
if (!hash[tweetID]) {
let post = data.data.getTopic.post;
post.topics = [{
id: data.data.getTopic.id,
name: data.data.getTopic.name
}]
hash[tweedID] = post;
} else {
console.log("Found tweet. Appending to topics array.");
let post = hash[tweedID];
let topicsArr = post.topics;
topicsArr.push({
id: data.data.getTopic.id,
name: data.data.getTopic.name
});
post.topics = topicsArr;
hash[tweedID] = post;
}
}
}));
console.log("Hash: ", hash);
console.log("Map size: ", Object.keys(hash).length);
let tweets = [];
for (const key in hash) {
tweets.push(hash[key]);
}
console.log("Tweets to go into state: ", tweets);
}
tweets();
// Or, put the whole thing in brackets and call it immediately:
(async () => {
// ...
})();
The reason that you see that the logged object has values is because of a feature of the dev-console that live-refreshes objects' properties, so they will no longer reflect the state when they've been logged, but instead display their current value.
Consider the following example:
const obj = {};
console.log(obj);
obj.foo = 'bar';
You'll see that the obj's foo property is 'bar', even though it was undefined when console.log was actually run. As #epascarello pointed out in the comments, you can tell when your browser has done this because of the little blue [i]. If you hover over it, you'll see that it says "This value was evaluated just now".
To avoid that behavior, JSON.stringify the object before logging:
const obj = {}
console.log(JSON.stringify(obj))
obj.foo = 'bar'
This will log {}, as you might expect, since it turns it into a string and spits that text out, instead of a reference to obj.
See more here.
I'm trying to iterate and print out in order an array in Javascript that contains the title of 2 events that I obtained from doing web scraping to a website but it prints out in disorder. I know Javascript is asynchronous but I'm new in this world of asynchronism. How can I implement the loop for to print the array in order and give customized info?
agent.add('...') is like console.log('...'). I'm doing a chatbot with DialogFlow and NodeJs 8 but that's not important at this moment. I used console.log() in the return just for debug.
I tried the next:
async function printEvent(event){
agent.add(event)
}
async function runLoop(eventsTitles){
for (let i = 0; i<eventsTitles.length; i++){
aux = await printEvent(eventsTitles[i])
}
}
But i got this error error Unexpected await inside a loop no-await-in-loop
async function showEvents(agent) {
const cheerio = require('cheerio');
const rp = require('request-promise');
const options = {
uri: 'https://www.utb.edu.co/eventos',
transform: function (body) {
return cheerio.load(body);
}
}
return rp(options)
.then($ => {
//** HERE START THE PROBLEM**
var eventsTitles = [] // array of event's titles
agent.add(`This mont we have these events available: \n`)
$('.product-title').each(function (i, elem) {
var event = $(this).text()
eventsTitles.push(event)
})
agent.add(`${eventsTitles}`) // The array prints out in order but if i iterate it, it prints out in disorder.
// *** IMPLEMENT LOOP FOR ***
agent.add(`To obtain more info click on this link https://www.utb.edu.co/eventos`)
return console.log(`Show available events`);
}).catch(err => {
agent.add(`${err}`)
return console.log(err)
})
}
I would like to always print out Event's title #1 and after Event's title #2. Something like this:
events titles.forEach((index,event) => {
agent.add(`${index}. ${event}`) // remember this is like console.log(`${index}. ${event}`)
})
Thanks for any help and explanation!
There no async case here but if you still face difficultly than use this loop
for (let index = 0; index < eventsTitles.length; index++) {
const element = eventsTitles[index];
agent.add(${index}. ${element})
}
I need combine multple API calls on a final object, this because API have limits to be consumed, any have an idea how is possible combine multiple calls in same final object, next is an example of my code, I need all data in this.lista but is not working:
created(){
this.$http.get('/api/transactions?senderId=8642612272713533685S&limit=1&offset=000')
.then( function(res){
console.log(res.body.count);
let limit = Math.ceil(res.body.count/1000);
console.log(limit);
let m = {};
let off = 0;
for (var i = 0; i <= limit; i++) {
this.$http.get('/api/transactions?senderId=8642612272713533685S&limit=1000', {params:{offset: off}})
.then( function(data){
this.lista = { ...this.lista, ...data.body.transactions }
} )
off = off + 1000;
}
}
);
}
any help will be appreciated
Using Promise.all is most likely what you are looking for. I will write just enough code for you to understand where to go.
// Populate your array array with URLs you want to get
let urls = ["url1", "url2", "...and so on"];
// Make into web request promises
let httpReqPromises = urls.map( url => this.$http.get(url) )
// Wait for all of them to resolve
Promise.all(httpReqPromises).then(allResponses => {
// Put them all together
this.lista = allResponses.reduce((a, b) => ({...a, ...b}, {})
})
The only work I leave up to you is how you populate the url variable.