So I have a form, user searches a query.
Trying to do:
I make API calls consistently. The first query is entered by user, then the program makes new queries based on Entity Analysis, and makes API calls itself - over and over.
Issue: The results are the same, which means the code is making the
API calls based on the initial query over and over, instead of making
new queries from API results.
Result in Console:
queries length is zero...
Iterating... undefined
On Queue Searches #1: []
query length is zero
POST / 200 15.747 ms - -
etc. etc. etc.
code
Pushing the query entered by user into array, sending it to onQueueSearch to make API calls at a set interval.
// Global Queries
const queries = new Array();
let counter = 0; // Just to keep track
router.post('/', async (req, res) => {
// Assign first query
queries.push(req.body.searchTerm);
// Invoke the async iterator
const results = await queryIterator(queries);
res.send(results);
});
This function might be the problem?
For each query in the array, do pass it to sentimentAnalyze which calls API's, this function is suppose to make calls to API consistently at set interval. (Note: New queries are added to queries array in the API calls.)
async function onQueueSearch() {
if(queries.length === 0 ) {
console.log("queries length is zero...");
return; // return something if empty
}
// This will extract the next in queue and remove it from queries
return await sentimentAnalyze(queries.pop);
}
// Define functions for Async Iterator -----
async function* queryGenerator() {
while (true) {
yield await onQueueSearch(queries.pop());
}
}
async function queryIterator(queries) {
for await (const result of queryGenerator()) {
console.log('Iterating... ', result);
console.log(`On Queue Searches #${++counter}: `, queries);
if (!queries.length) {
console.log("query length is zero");
return result;
}
}
}
API calls function - Return a set of twitter results, then google api results - this async function try's to wait for returned promises/result
from twitter google api and returns
async function sentimentAnalyze(searchValue) {
try {
const tweetDocumentObject = await searchTwitter(searchValue);
const sentimentResults = await googleSentimentAnalyze(tweetDocumentObject);
const entitiesResults = await googleEntityAnalyze(tweetDocumentObject);
return {sentimentResults, entitiesResults};
} catch(err) {
console.log('sentimentAnalyze Error: ', err);
}
}
This function is where new queries are ADDED from entity analysis.
function googleEntityAnalyze(document) {
return new Promise((resolve, reject) => {
// Detects entities in the document
googleClient
.analyzeEntities({document: document})
.then(results => {
const entities = results[0].entities;
queries.unshift(req.body.searchTerm);
entities.forEach(entity => {
queries.unshift(entity.name);
// console.log("Entitiy name: " +entity.name);
// console.log(` - Type: ${entity.type} `);
resolve({ entity });
});
})
.catch(err => {
console.error('ENTITY ERROR:', err);
});
});
}
Ok, there's a couple of issues with your code, but easy to fix if you're carefull, I'll guide you through them.
First of:
You're pushing to queue always, this is the root cause of you getting the same data in queue:
queries.push(req.body.searchTerm); // searchTerm being '1'
... // Do more logic until googleEntityAnalyze
entities.forEach(entity => {
queries.push(entity.name);
resolve({ entity });
});
// Assume entities as [2, 3]
Now you have your queue as [1,2,3], being 1 as the first argument.
Then you redo the query logic from:
async function onQueueSearch() {
for(i = 0; i < queries.length; i++) {
return await sentimentAnalyze(queries[i]);
// queries[i].splice(i, 1);
}
if(queries.length === 0 ) {
console.log("queries length is zero...");
}
}
The return on the for loop will break the loop and return the first execution, which will be queries[0] === 1, your first argument.
So, to resolve this, keep the array in first-in-first-out, using:
queries.unshift(req.body.searchTerm);
entities.forEach(entity => {
queries.unshift(entity.name);
resolve({ entity });
});
This will keep your queries in order as they arrive [3,2,1], instead of [1,2,3], now you can extract your query using queries.pop() instead.
queries.unshift(4); // where queries was [3,2,1], will be [4,3,2,1]
queries.pop(); // 1
Change the sentiment analyze to:
async function onQueueSearch(){
if(queries.length === 0 ) {
console.log("queries length is zero...");
return; // Return something if empty!
}
return await sentimentAnalyze(queries.pop());
// This will extract the next in queue and remove it from queries
}
Secondly, your re-call logic
You're using an iterval to continuously recall the query, the problem with this:
setInterval(async function() {
results = await onQueueSearch();
}, 3000);
Is that you need to "estimate" the time the query will take to finish before re-executing. With the query unshift and pop above you'll make this work, however, you need a more elegant solution.
With NodeJS version < 10+:
recursion will be useful, here's a small sample of what you're doing in a simple manner:
const queries = [1];
let counter = 0; // Just to keep track
async function reRunQueries(){
counter++;
console.log(`queries #${counter}: `, queries);
results = await onQueueSearch();
console.log('Results: ', results);
if(!!queries.length){
return await reRunQueries();
}else{
return results;
}
}
async function onQueueSearch(){
return await longQuery(queries.pop());
}
async function longQuery(param){
if(param === 6){
// Use this as stop signal!
return param;
}else{
queries.unshift(++param);
return param;
}
}
const RES = reRunQueries();
RES.then(result => {
console.log('RES: ', result);
})
It's important to know your stop signal for recursion, otherwise it will never end.
With NodeJS > 10+
Use a Iterable Generator:
const queries = [];
let counter = 0; // Just to keep track
// EMULATE EXPRESS JS call ========================
const req = { body: { searchTerm: 1 } };
const res = {send: (val) => console.log('RECEIVED: ', val)};
const router = {
post: (route, callback) => callback(req, res)
}
router.post('/', async (req, res) => {
// Assign first query
queries.push(req.body.searchTerm);
// Invoke the async iterator
const results = await queryIterator(queries);
res.send(results);
});
// The Above is for nodeJS direct testing only,
// > you can plug it into your own router.post declaration "as is"
// -----------------------------------------
// Define functions for Async Iterator -----
async function* queryGenerator() {
while (true) {
yield await onQueueSearch(queries.pop());
}
}
async function queryIterator(queries) {
for await (const result of queryGenerator()) {
console.log('Iterating... ', result);
console.log(`On Queue Searches #${++counter}: `, queries);
if (!queries.length) {
return result;
}
}
}
// ------------------------------------------------------------
// Emulate your current async requests using queries array ----
async function onQueueSearch(param) {
return await longQuery(param);
}
async function longQuery(param) {
if (param === 6) {
// Use this as stop signal!
return Promise.resolve(param);
} else {
queries.unshift(++param);
return Promise.resolve(param);
}
}
Returns:
/*
Iterating... 2
On Queue Searches #1: [ 2 ]
Iterating... 3
On Queue Searches #2: [ 3 ]
Iterating... 4
On Queue Searches #3: [ 4 ]
Iterating... 5
On Queue Searches #4: [ 5 ]
Iterating... 6
On Queue Searches #5: [ 6 ]
Iterating... 6
On Queue Searches #6: []
RECEIVED: 6
*/
Related
I'm struggling a bit with JS promises.
I am using a library to pull data from Spotify that returns promises.
In my main function I can use an await to build an object from the response data and push it to an array (called nodes):
var nodes = [];
main();
async function main() {
var id = '0gusqTJKxtU1UTmNRMHZcv';
var artist = await getArtistFromSpotify(id).then(data => buildArtistObject(data));
nodes.push(artist);
When I debug here then all is good, nodes has my object.
However, when I introduce a 2nd await underneath to make another call:
nodes.forEach((node, i) => {
if (node.done == false) {
console.log(node.toString());
var related_artists = await getRelatedArtists(node.spotify_id);
I get the following error: SyntaxError: await is only valid in async function
I thought the first await statement would be resolved and the execution would continue until the next..?
Any help would be greatly appreciated.
EDIT
The other functions, if that helps, are just as follows:
function getArtistFromSpotify(id) {
let response = spotify
.request('https://api.spotify.com/v1/artists/' + id).then(function (data) {
return data;
})
.catch(function (err) {
console.error('Error occurred: ' + err);
return null;
});
return response;
}
function getRelatedArtists(id) {
let response = spotify
.request('https://api.spotify.com/v1/artists/' + id + '/related-artists').then(function (data) {
return data;
})
.catch(function (err) {
console.error('Error occurred: ' + err);
return null;
});
return response;
}
function buildArtistObject(data) {
var artist = {
node_id: nodes.length,
name: null,
genres: null,
popularity: null,
spotify_id: null,
done: false
}
artist.name = data.name;
artist.genres = data.genres;
artist.popularity = data.popularity > 0 ? data.popularity : 0;
artist.spotify_id = data.id;
return artist;
}
The code below has multiple problems.
var nodes = [];
main();
async function main() {
var id = '0gusqTJKxtU1UTmNRMHZcv';
var artist = await getArtistFromSpotify(id).then(data => buildArtistObject(data));
nodes.push(artist);
// ...
First of all, main mutates global scope nodes. Not only is this an antipattern even in synchronous code (functions should not rely on, or modify, global variable names; use parameters and return values instead), in asynchronous code, nodes will never be available for use anywhere but within main. See How do I return the response from an asynchronous call?.
Secondly, try to avoid combining then and await. It's confusing.
It's also a little odd that an array of nodes is used, yet only one artist is pushed onto it...
As for this code:
nodes.forEach((node, i) => {
if (node.done == false) {
console.log(node.toString());
var related_artists = await getRelatedArtists(node.spotify_id);
// ...
The error is self-explanatory. You must add async to the enclosing function if you want it to be asynchronous: nodes.forEach(async (node, i) => { // .... But that spawns a new promise chain per node, meaning future code that's dependent on the result won't be able to await all of the promises in the loop resolving. See Using async/await with a forEach loop. The likely solution is for..of or Promise.all.
While I'm not 100% sure what your final goal is, this is the general pattern I'd use:
async function main() {
const id = '0gusqTJKxtU1UTmNRMHZcv';
const data = await getArtistFromSpotify(id);
const artist = await buildArtistObject(data);
const nodes = [artist]; // odd but I assume you have more artists somewhere...
for (const node of nodes) {
if (!node.done) {
const relatedArtists = await getRelatedArtists(node.spotify_id);
}
}
/* or run all promises in parallel:
const allRelatedArtists = await Promise.all(
nodes.filter(e => !e.done).map(e => getRelatedArtists(e.spotify_id))
);
*/
// ...
}
main();
Since your code isn't runnable and some of the intent is unclear from the context, you'll likely need to adapt this a bit, so consider it pseudocode.
You have some misunderstandings of how to use promises -
let response = spotify
.request(url)
.then(function(data) { return data }) // this does nothing
.catch(function (err) { // don't swallow errors
console.error('Error occurred: ' + err);
return null;
})
return response
You'll be happy there's a more concise way to write your basic functions -
const getArtist = id =>
spotify
.request('https://api.spotify.com/v1/artists/' + id)
const getRelatedArtists = id =>
spotify
.request('https://api.spotify.com/v1/artists/' + id + '/related-artists')
Now in your main function, we can await as many things as needed. Let's first see how we would work with a single artist ID -
async function main(artistId) {
const artistData = await getArtist(artistId)
const relatedData = await getRelatedArtists(artistId)
return buildArtist(artistData, relatedData)
}
If you have many artist IDs -
async function main(artistIds) {
const result = []
for (const id of artistIds) {
const artistData = await getArtist(artistId)
const relatedData = await getRelatedArtists(artistId)
result.push(buildArtist(artistData, relatedData))
}
return result
}
Either way, the caller can handle errors as
main([693, 2525, 4598])
.then(console.log) // display result
.catch(console.error) // handle errors
Which is the same as -
main([693, 2525, 4598]).then(console.log, console.error)
The pattern above is typical but sub-optimal as the caller has to wait for all data to fetch before the complete result is returned. Perhaps you would like to display the information, one-by-one as they are fetched. This is possible with async generators -
async function* buildArtists(artistIds) {
for (const id of artistIds) {
const artistData = await getArtist(artistId)
const relatedData = await getRelatedArtists(artistId)
yield buildArtist(artistData, relatedData) // <- yield
}
}
async function main(artistIds) {
for await (const a of buildArtists(artistIds)) // <- for await
displayArtist(a)
}
main([693, 2525, 4598]).catch(console.error)
I am trying to run parallel requests in batches to an API using a bunch of keywords in an array. Article by Denis Fatkhudinov.
The problem I am having is that for each keyword, I need to run the request again with a different page argument for as many times as the number in the pages variable.
I keep getting Cannot read property 'then' of undefined for the return of the chainNext function.
The parallel request in batches on its own, without the for loop, works great, I am struggling to incorporate the for loop on the process.
// Parallel requests in batches
async function runBatches() {
// The keywords to request with
const keywords = ['many keyword strings here...'];
// Set max concurrent requests
const concurrent = 5;
// Clone keywords array
const keywordsClone = keywords.slice()
// Array for future resolved promises for each batch
const promises = new Array(concurrent).fill(Promise.resolve());
// Async for loop
const asyncForEach = async (pages, callback) => {
for (let page = 1; page <= pages; page++) {
await callback(page);
}
};
// Number of pages to loop for
const pages = 2;
// Recursively run batches
const chainNext = (pro) => {
// Runs itself as long as there are entries left on the array
if (keywordsClone.length) {
// Store the first entry and conviently also remove it from the array
const keyword = keywordsClone.shift();
// Run 'the promise to be' request
return pro.then(async () => {
// ---> Here was my problem, I am declaring the constant before running the for loop
const promiseOperation = await asyncForEach(pages, async (page) => {
await request(keyword, page)
});
// ---> The recursive invocation should also be inside the for loop
return chainNext(promiseOperation);
});
}
return pro;
}
return await Promise.all(promises.map(chainNext));
}
// HTTP request
async function request(keyword, page) {
try {
// request API
const res = await apiservice(keyword, page);
// Send data to an outer async function to process the data
await append(res.data);
} catch (error) {
throw new Error(error)
}
}
runBatches()
The problem is simply that pro is undefined, because you haven't initialized it.
You basically execute this code:
Promise.all(new Array(concurrent).fill(Promise.resolve().map(pro => {
// pro is undefined here because the Promise.resolve had no parameter
return pro.then(async () => {})
}));
I'm not completely sure about your idea behind that, but this is your problem in a more condensed version.
I got it working by moving actual request promiseOperation inside the for loop and returning the recursive function there too
// Recursively run batches
const chainNext = async (pro) => {
if (keywordsClone.length) {
const keyword = keywordsClone.shift()
return pro.then(async () => {
await asyncForEach(pages, (page) => {
const promiseOperation = request(keyword, page)
return chainNext(promiseOperation)
})
})
}
return pro
}
Credit for the parallel requests in batches goes to https://itnext.io/node-js-handling-asynchronous-operations-in-parallel-69679dfae3fc
I have an endpoint which loops through an array and updates the database as follows.
app.post('/create', Authenticate, async (req, res) => {
const {
products,
} = req.body;
const trxProvider = knex.transactionProvider();
const trx = await trxProvider();
try {
const formattedProduct = await Promise.all(products.map(async (product) => {
// Get Current value
const warehouseProducts = await trx('warehouse_products')
.select('available_qty as qty')
.where('product_code', product.product.code)
.first();
const finalQty = warehouseProducts.qty - product.orderQty;
// Update database
await trx('warehouse_products')
.update({ available_qty: finalQty })
.where('product_code', product.product.code);
}));
await trx('ordered_products')
.insert(formattedProduct);
trx.commit();
console.log('Transaction successful');
return send(res, 201, { success: true });
} catch (err) {
console.log(err);
trx.rollback();
const errors = {};
errors.message = 'something went wrong';
return send(res, 500, errors);
}
});
The issue arises when i try to update the same row of the warehouse_products table within the loop.
In the loop initially the qty value is taken from the warehouse_products table for a particular product then an arithmetic operation is done and the qty value is updated.
Ideally if both iterations access the same row, the second iteration's initial qty value should be what the first iteration updated. However the issue is that the second iteration too starts with the initial value of the first iteration. Its almost as if both iterations are happening parallel to each other instead of occurring sequentially.
Since you are using Promise.all it is supposed to happen in paralle. For sequential processing change this code
await Promise.all(products.map(async (product) => {
// logic here
});
to
for(const product of products) {
// logic here
}
Have a look at the definition for Promise.all()
It is typically used after having started multiple asynchronous tasks to run concurrently and having created promises for their results, so that one can wait for all the tasks being finished.
if you don't want to use an external library like Bluebird or Async
you can go with simple for a loop as following
let delay = (t) => new Promise((resolve) => {
setTimeout(() => {
return resolve(new Date().getMilliseconds())
}, t*1000);
});
let array = [1,1,1,1];
//using simple for loop
async function test() {
let out = [];
for (let i = 0; i < 4; i++) {
const res = await delay(array[i]);
out.push(res);
}
return out;
}
test().then(res => {
console.log(res)
})
I have the following code:
* Fetch stats from api
*/
fetchStats() {
this._isFetching = true;
// fetch stats after building url and replacing invalid characters
return new Promise(async (resolve, reject) => {
await API.fetchStats(this.rsn)
.then(jres => {
this.skills = jres.main.skills;
this._isFetching = false;
resolve('success');
})
.catch(err => {
console.log(err);
console.log('error retreiving stats');
this._isFetching = false;
reject('Failed to retreive stats');
})
.finally(() => {
this._isFetching = false;
});
});
}
I thought making it async with await would make it wait until it got the response before continuing. Returning the promise is something I added in testing to see if I could make it synchronous.
Then my code that consumes this method:
memberCollection.forEach(async el => {
await el.player.fetchStats()
.then(() => {
console.log(`Refreshed ${el.player.rsn}'s account`);
})
.catch(console.log(`Failed to refresh ${el.player.rsn}'s account`));
});
My thinking was that it would wait till it got a response then console.log either a successful refresh or a failed refresh. What I am instead seeing is a whole bunch of "success" messages followed by a string of failed messages indicating that it is running both the then and the catch message in the foreach. Does anyone know how I can make this work.
My issue is that Axios keeps timing out (my speculation is that it is due to the number of requests being sent off and the fact that there is a 5-10sec delay as it pulls from the db), if I navigate to the API URL manually it works as well as if I just do one member (as opposed to forEach) it works fine. So I'm trying to limit the number of requests fired off at once. I have tried setting my axios timeout to 10, 20, and 60 seconds, but it made no improvement.
Solution code:
const asyncForEach = async (arr, cb) => {
for(let i=0;i<arr.length;i++) {
let el = arr[i];
try {
let res = await cb(el);
} catch (err) { console.log(err) };
if(el.player && el.player.rsn) console.log(`Processed ${el.player.rsn}`);
}
console.log('done processing in asyncForEach');
}
not linked to axios but to async await.
consider
function slow(i){
return new Promise((ok,ko)=>{
return setTimeout(_=>ok(i), 1000)
})
}
async function asyncForEach(arr, cb){
for(var i = 0; i<arr.length; ++i){
let el = arr[i];
let res = await cb(el);
console.log('async', res, new Date)
}
}
/*
#foreach does not wait, but async and reduce are spaced by one second
foreach 4 2019-10-14T13:43:47.059Z
foreach 5 2019-10-14T13:43:47.071Z
foreach 6 2019-10-14T13:43:47.071Z
async 1 2019-10-14T13:43:47.071Z
async 2 2019-10-14T13:43:48.073Z
async 3 2019-10-14T13:43:49.074Z
reduce 7 2019-10-14T13:43:50.076Z
reduce 8 2019-10-14T13:43:51.078Z
reduce 9 2019-10-14T13:43:52.080Z
*/
async function main(){
await [4,5,6].forEach(async el=>{
let res = await slow(el);
console.log('foreach', res, new Date)
})
await asyncForEach([1,2,3], slow);
await [7,8,9].reduce((acc, el)=>acc.then(async _=>{
let res = await slow(el);
console.log('reduce', res, new Date);
return;
}), Promise.resolve())
}
main();
As you can see from timestamps, forEach does not wait for slow to finish
however, asyncForEach in its iteration does wait
What you may want to do is either
write a for loop as done with asyncForEach
use standard promises (stacking them):
[1,2,3].reduce((acc, el)=>acc.then(_=>{
return slow(el);
}), Promise.resolve())
I'm currently trying to fetch all messages in a guild, yet the .fetchMessages method only works on specific channels.
I've already tried using the .forEach function but have received multiple errors whilst doing so.
async function intervalFunc() {
var mainGuild = client.guilds.get("562324876330008576");
var messagesArray = [];
await mainGuild.channels.forEach(channel => {
if (channel.type == "text") {
channel.fetchMessages({ limit: 20 }).then(msg => {
messagesArray.push(msg);
});
}
});
console.log(messagesArray.length)
//....
}
The expected output is at least a few hundred but instead, it outputs 0.
You need to put await on the async operation:
async function intervalFunc() {
var mainGuild = client.guilds.get("562324876330008576");
var messagesArray = [];
for(channel in mainGuild.channels) {
if(channel.type == "text") {
const messages = await channel.fetchMessages({limit: 20});
messagesArray = [
...messagesArray,
...messages,
];
}
}
console.log(messagesArray.length);
}
So here it will wait for ferchMessages to return a value in an asynchronous manner and then proceed to next iteration.