We Keep Coding

How to inject a promise into a promise chain with an executor?

What I'm trying to accomplish
I am currently trying to create a wrapper for a db connection (to Neo4j) that works similar to the following:
Instantiate driver
Expose the executor for the driver so a session can be created
Pass my logic
Close the connection
Since there's no destructor in JavasScript, it's making this unnecessarily difficult with properly closing the session. The logic for creating and closing a connection is extremely repetitive and I'm trying to simplify repetitive scripts so that it's easier to call.
What I've tried.
Inject promise in chain
I thought something like the following could work, but I just cannot seem to create the logic properly. Passing session back to my inserted promise is challenging.
const connect = () => {
var driver;
var session;
return Promise.resolve(() => {
driver = my.driver(uri, creds);
}).then(() => {
// insert my promise here, exposing driver.session() function for executor
// if possible, bind it to the session var so we can properly close it after
// this would also determine the return value
}).catch((e) => console.error(e))
.finally(() => {
session.close();
driver.close();
})
});
Create class wrapper with procedural logic
I then also tried another approach, similar to:
var driver = my.driver(uri, creds);
var session;
function exitHandler(options) {
// ...
session.close();
driver.close();
}
// I have process.on for exit, SIGINT, SIGUSR1, SIGUSR2, and uncaughtException
process.on('exit', exitHandler.bind(null, options));
// ...
class Connection extends Promise<any> {
constructor(executor: Function) {
super((resolve, reject) => executor(resolve, reject));
executor(driver.session.bind(null, this));
}
}
export default Connection;
And calling it like
// ...
const handler = async () => await new Connection((session) => {
const s = session();
// do stuff here
});
The problem with this approach is that the driver is not instantiated before session is used (and so it's undefined). It also feels a little hacky with the process.on calls.
Question
Neither method works (or any of my other attempts). How can I properly wrap db connections to ensure they're consistent and deduplicate my existing code?
A sample of the Neo4j connection script can be found here. This is, essentially, what I'm trying to deduplicate across my scripts (pass everything from line 11 to 42 - inclusive) but have the init of driver, catch, finally, session.close(), driver.close() logic in my wrapper.
Ideally, I would like to expose the session function call so that I can pass parameters to it if needed: See the Session API for more info. If possible, I also want to bind the rxSession reactive session.

A sample of the Neo4j connection script can be found here. This is, essentially, what I'm trying to deduplicate across my scripts (pass everything from line 11 to 42 - inclusive) but have the init of driver, catch, finally, session.close(), driver.close() logic in my wrapper.
OK, the above part of what you are asking is what I was able to best parse and work with.
Taking the code you reference and factoring out lines 11 to 42 such that everything outside of those is shared and everything inside of those is customizable by the caller, this is what I get for the reusable part, designed to be in a module by itself:
// dbwrapper.js
const neo4j = require('neo4j-driver')
const uri = 'neo4j+s://<Bolt url for Neo4j Aura database>';
const user = '<Username for Neo4j Aura database>';
const password = '<Password for Neo4j Aura database>';
const driver = neo4j.driver(uri, neo4j.auth.basic(user, password));
let driverOpen = true;
async function runDBOperation(opCallback, sessOpts = {}) {
const session = driver.session(sessOpts);
try {
await opCallback(session);
} catch (e) {
console.log(e);
throw e;
} finally {
await session.close();
}
}
async function shutdownDb() {
if (driverOpen) {
driverOpen = false;
await driver.close();
}
}
process.on('exit', shutdownDb);
module.exports = { runDBOperation, shutdownDb };
Then, you could use this from some other module like this:
const { runDBOperation, shutdownDB } = require('./dbwrapper.js');
runDBOperation(async (session) => {
const person1Name = 'Alice'
const person2Name = 'David'
// To learn more about the Cypher syntax, see https://neo4j.com/docs/cypher-manual/current/
// The Reference Card is also a good resource for keywords https://neo4j.com/docs/cypher-refcard/current/
const writeQuery = `MERGE (p1:Person { name: $person1Name })
MERGE (p2:Person { name: $person2Name })
MERGE (p1)-[:KNOWS]->(p2)
RETURN p1, p2`
// Write transactions allow the driver to handle retries and transient errors
const writeResult = await session.writeTransaction(tx =>
tx.run(writeQuery, { person1Name, person2Name })
)
writeResult.records.forEach(record => {
const person1Node = record.get('p1')
const person2Node = record.get('p2')
console.log(
`Created friendship between: ${person1Node.properties.name}, ${person2Node.properties.name}`
)
})
const readQuery = `MATCH (p:Person)
WHERE p.name = $personName
RETURN p.name AS name`
const readResult = await session.readTransaction(tx =>
tx.run(readQuery, { personName: person1Name })
)
readResult.records.forEach(record => {
console.log(`Found person: ${record.get('name')}`)
})
}).then(result => {
console.log("all done");
}).catch(err => {
console.log(err);
});
This can be made more flexible or more extensible according to requirements, but obviously the general idea is to keep it simple so that simple uses of the common code don't require a lot of code.

Making multiple web api calls synchronously without nesting in Node.js with Axios

Is there any way I can make the below code run synchronously in a way where I can get all of the productLine ids and then loop through and delete all of them, then once all of this is complete, get all of the productIds and then loop through and delete all of them?
I really want to be able to delete each set of items in batch, but the next section can't run until the first section is complete or there will be referential integrity issues.
// Delete Product Lines
axios.get('https://myapi.com/ProductLine?select=id')
.then(function (response) {
const ids = response.data.value
ids.forEach(id => {
axios.delete('https://myapi.com/ProductLine/' + id)
})
})
.catch(function (error) {
})
// Delete Products (I want to ensure this runs after the above code)
axios.get('https://myapi.com/Product?select=id')
.then(function (response) {
const ids = response.data.value
ids.forEach(id => {
axios.delete('https://myapi.com/Product/' + id)
})
})
.catch(function (error) {
})

There's a lot of duplication in your code. To reduce code duplication, you can create a helper function that can be called with appropriate arguments and this helper function will contain code to delete product lines and products.
async function deleteHelper(getURL, deleteURL) {
const response = await axios.get(getURL);
const ids = response.data.value;
return Promise.all(ids.map(id => (
axios.delete(deleteURL + id)
)));
}
With this helper function, now your code will be simplified and will be without code duplication.
Now to achieve the desired result, you could use one of the following ways:
Instead of two separate promise chains, use only one promise chain that deletes product lines and then deletes products.
const prodLineGetURL = 'https://myapi.com/ProductLine?select=id';
const prodLineDeleteURL = 'https://myapi.com/ProductLine/';
deleteHelper(prodLineGetURL, prodLineDeleteURL)
.then(function() {
const prodGetURL = 'https://myapi.com/Product?select=id';
const prodDeleteURL = 'https://myapi.com/Product/';
deleteHelper(prodGetURL, prodDeleteURL);
})
.catch(function (error) {
// handle error
});
Use async-await syntax.
async function delete() {
try {
const urls = [
[ prodLineGetURL, prodLineDeleteURL ],
[ prodGetURL, prodDeleteURL ]
];
for (const [getURL, deleteURL] of urls) {
await deleteHelper(getURL, deleteURL);
}
} catch (error) {
// handle error
}
}
One other thing that you could improve in your code is to use Promise.all instead of forEach() method to make delete requests, above code uses Promise.all inside deleteHelper function.

Your code (and all other answers) are executing delete requests sequentially, which is huge waste of time. You should use Promise.all() and execute in parallel...
// Delete Product Lines
axios.get('https://myapi.com/ProductLine?select=id')
.then(function (response) {
const ids = response.data.value
// execute all delete requests in parallel
Promise.all(
ids.map(id => axios.delete('https://myapi.com/ProductLine/' + id))
).then(
// all delete request are finished
);
})
.catch(function (error) {
})

All HTTP request are asynchronous but you can make it sync-like. How? Using async-await
Suppose you have a function called retrieveProducts, you need to make that function async and then await for the response to keep processing.
Leaving it to:
const retrieveProducts = async () => {
// Delete Product Lines
const response = await axios.get('https://myapi.com/ProductLine?select=id')
const ids = response.data.value
ids.forEach(id => {
axios.delete('https://myapi.com/ProductLine/' + id)
})
// Delete Products (I want to ensure this runs after the above code)
const otherResponse = await axios.get('https://myapi.com/Product?select=id') // use proper var name
const otherIds = response.data.value //same here
otherIds.forEach(id => {
axios.delete('https://myapi.com/Product/' + id)
})
}
But just keep in mind that it's not synchronous, it keeps being async

Problems with the fetch api on nodejs [duplicate]

This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 2 years ago.
I am creating an array of objects from an endpoint with the fetch api. Each object in the response contains a new url and I scan this array for new objects. The purpose is to access the endpoint, obtain the urls and access these urls to store your objects and attributes. But when accessing the array with objects and from an index it returns undefined.
let url = [];
let pokemon = [];
function getPokemons(){
fetch("https://pokeapi.co/api/v2/type/11/")
.then(async response =>{
await response.json().then(data =>
data.pokemon.forEach(item =>
url.push(item.pokemon.url)
)
)
}).then( response => {
url.map(item =>
fetch(item).then(response =>{
response.json().then(data =>
pokemon.push(data)
)
})
)
})
}
getPokemons()
console.log(pokemon[1])

Solution for web browser :
The node-fetch module is not required, I just clean your code using async/await syntax.
// getPokemon is now an async function for cleaner syntax
async function getPokemon() {
const pokemon = [];
// Then we can use await
const response = await fetch("https://pokeapi.co/api/v2/type/11/"),
json = await response.json(),
urls = json.pokemon.map(item => item.pokemon.url);
for (const url of urls) {
const response = await fetch(url),
json = await response.json();
pokemon.push(json)
}
// We return the pokemon array instead of polluting global scope
return pokemon;
};
getPokemon().then(pokemon => {
console.log(pokemon)
});
Hope it help !
NodeJS solution
You must install node-fetch (see npmjs.org :
npm install -D node-fetch
Then you can use a fetch:
// Importing the node module.
// You can delete this line if your not using a node environment.
const fetch = require('node-fetch');
// getPokemon is now an async function for cleaner syntax
async function getPokemon() {
const pokemon = [];
// Then we can use await
const response = await fetch("https://pokeapi.co/api/v2/type/11/"),
json = await response.json(),
urls = json.pokemon.map(item => item.pokemon.url);
for (const url of urls) {
const response = await fetch(url),
json = await response.json();
pokemon.push(json)
}
// We return the pokemon array instead of polluting global scope
return pokemon;
};
getPokemon().then(pokemon => {
console.log(pokemon)
});
Explanation
fetch is not part of the Javascript language specification but is a Web API. Each browser may or not choose to implement it. Each Implementations may work differently under the hood but the Javascript API provided must match the standard (MDN Web Docs).
This is why you need a module to fetch the data.
EDIT : Adding solution for web browser environment

The problem is that the function getPokemons should be async.
You should await it before accessing:
getPokemons().then(()=> console.log(pokemon[1]))
// or if it inside an async function:
await getPokemons();
console.log(pokemon[1]);
But there's another reason also. You have internal promises outside the parent promise chain. I mean:
.then((response) => {
// this is array of promises
// you should return it to have ability await it on parent promise
url.map((item) =>
fetch(item).then((response) => {
response.json().then((data) => pokemon.push(data));
})
);
});
Your code might look like:
// if you really need global variable
let pokemon = [];
async function getPokemons() {
const result = await fetch("https://pokeapi.co/api/v2/type/11/")
.then((response) => response.json())
.then((data) =>
Promise.all(
data.pokemon.map((item) =>
fetch(item.pokemon.url).then((response) => response.json())
)
)
);
pokemon.push(...result);
}
getPokemons().then(() => {
console.log(pokemon[1]);
});
Or, the same result without global variables:
function getPokemons() {
return fetch("https://pokeapi.co/api/v2/type/11/")
.then(...)
.then(...);
}
getPokemons().then((pokemons) => {
console.log(pokemons[1]);
});

Bluebird Promise Order issue

I am watching videos to learn MongoDB Express.js VueJS Node.js (MEVN) stack.
And I want to create a seed directory and also use promise functions
// const delay = require('delay')
const Promise = require('bluebird')
const songs = require('./songs.json')
const users = require('./users.json')
const bookmarks = require('./bookmarks.json')
const historys = require('./history.json')
sequelize.sync({ force: true })
.then( async function () {
await Promise.all(
users.map( user => {
User.create(user)
})
)
await Promise.all(
songs.map( song => {
Song.create(song)
})
)
//I have to add this line
// ---> await delay(1000)
await Promise.all(
bookmarks.map( bookmark => {
Bookmark.create(bookmark)
})
)
await Promise.all(
historys.map( history => {
History.create(history)
})
)
})
I have four tables with seeds to create, and the last two tables data must be created after the former two tables data. (They are foreign keys)
But every time I run this file, the last two tables data will be created first
The only way I can prevent this is to add delay(1000) between them.
I am wondering if there exists any efficient way to solve this issue~
Thank you.

Race conditions like this one is always caused by that promises weren't properly chained.
A promise should be returned from map callback:
await Promise.all(
users.map( user => User.create(user))
);
etc.
Not returning a value from map is virtually always a mistake. It can be prevented by using array-callback-return ESLint rule.
If User.create(user), etc. were Bluebird promises with default configuration, not chaining them would also result in this warning.

My assumption why your code might fail:
You're not returning the Promises that I guess /(User|Song|Bookmark|History).create/g return to the Promise.all() function, since your map callback is not returning anything.
If you're using Arrow functions with brackets, then you need to explicitly specify the return value (using the familar return keyword).
Otherwise you can just omit the curly brackets.
My suggestion is, that you refactor you're code by utilizing Promise .then()-Chaining.
For you're example, I would suggest something like this:
const Promise = require('bluebird')
const songs = require('./songs.json')
const users = require('./users.json')
const bookmarks = require('./bookmarks.json')
const histories = require('./history.json')
sequelize.sync({
force: true
}).then(() =>
Promise.all(
users.map(user =>
User.create(user)
)
).then(() =>
Promise.all(
songs.map(song =>
Song.create(song)
)
)
).then(() =>
Promise.all(
bookmarks.map(bookmark =>
Bookmark.create(bookmark)
)
)
).then(() =>
Promise.all(
histories.map(history =>
History.create(history)
)
)
)
);

Using Promise.all() to fetch a list of urls with await statements

tl;dr - if you have to filter the promises (say for errored ones) don't use async functions
I'm trying to fetch a list of urls with async and parse them, the problem is that if there's an error with one of the urls when I'm fetching - let's say for some reason the api endpoint doesn't exists - the program crushes on the parsing with the obvious error:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: ext is not iterable
I've tried checking if the res.json() is undefined, but obviously that's not it as it complains about the entire 'ext' array of promises not being iterable.
async function fetchAll() {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
Question 1:
How do I fix the above so it won't crash on an invalid address?
Question 2:
My next step is to write the extracted data to the database.
Assuming the data size of 2-5mgb of content, is my approach of using Promise.all() memory efficient? Or will it be more memory efficient and otherwise to write a for loop which handles each fetch then on the same iteration writes to the database and only then handles the next fetch?

You have several problems with your code on a fundamental basis. We should address those in order and the first is that you're not passing in any URLS!
async function fetchAll(urls) {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
First you have several try catch blocks on DEPENDANT DATA. They should all be in a single try catch block:
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url=>fetch(url)))
let ext = await Promise.all(data.map(res => {
// also fixed the ==! 'undefined'
if (res.json() !== undefined) { return res.json()}
}))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Next is the problem that res.json() returns a promise wrapped around an object if it exists
if (res.json() !== undefined) { return res.json()}
This is not how you should be using the .json() method. It will fail if there is no parsable json. You should be putting a .catch on it
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url => fetch(url).catch(err => err)))
let ext = await Promise.all(data.map(res => res.json ? res.json().catch(err => err) : res))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now when it cannot fetch a URL, or parse a JSON you'll get the error and it will cascade down without throwing. Now your try catch block will ONLY throw if there is a different error that happens.
Of course this means we're putting an error handler on each promise and cascading the error, but that's not exactly a bad thing as it allows ALL of the fetches to happen and for you to distinguish which fetches failed. Which is a lot better than just having a generic handler for all fetches and not knowing which one failed.
But now we have it in a form where we can see that there is some better optimizations that can be performed to the code
async function fetchAll(urls) {
try {
let ext = await Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.catch(error => ({ error, url }))
)
)
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now with a much smaller footprint, better error handling, and readable, maintainable code, we can decide what we eventually want to return. Now the function can live wherever, be reused, and all it takes is a single array of simple GET URLs.
Next step is to do something with them so we probably want to return the array, which will be wrapped in a promise, and realistically we want the error to bubble since we've handled each fetch error, so we should also remove the try catch. At that point making it async no longer helps, and actively harms. Eventually we get a small function that groups all URL resolutions, or errors with their respective URL that we can easily filter over, map over, and chain!
function fetchAll(urls) {
return Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.then(data => ({ data, url }))
.catch(error => ({ error, url }))
)
)
}
Now we get back an array of similar objects, each with the url it fetched, and either data or an error field! This makes chaining and inspecting SUPER easy.

You are getting a TypeError: ext is not iterable - because ext is still undefined when you caught an error and did not assign an array to it. Trying to loop over it will then throw an exception that you do not catch.
I guess you're looking for
async function fetchAll() {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
const ext = await Promise.all(data.map(res => res.json()));
for (let item of ext) {
console.log(item);
}
} catch (err) {
console.log(err);
}
}

Instead of fetch(url) on line 5, make your own function, customFetch, which calls fetch but maybe returns null, or an error object, instead of throwing.
something like
async customFetch(url) {
try {
let result = await fetch(url);
if (result.json) return await result.json();
}
catch(e) {return e}
}

if (res.json()==! 'undefined')
Makes no sense whatsoever and is an asynchronous function. Remove that condition and just return res.json():
try {
ext = await Promise.all(data.map(res => res.json()))
} catch (err) {
console.log(err)
}
Whether or not your approach is "best" or "memory efficient" is up for debate. Ask another question for that.

You can have fetch and json not fail by catching the error and return a special Fail object that you will filter out later:
function Fail(reason){this.reason=reason;};
const isFail = o => (o&&o.constructor)===Fail;
const isNotFail = o => !isFail(o);
const fetchAll = () =>
Promise.all(
urls.map(
url=>
fetch(url)
.then(response=>response.json())
.catch(error=>new Fail([url,error]))
)
);
//how to use:
fetchAll()
.then(
results=>{
const successes = results.filter(isNotFail);
const fails = results.filter(isFail);
fails.forEach(
e=>console.log(`failed url:${e.reason[0]}, error:`,e.reason[1])
)
}
)
As for question 2:
Depending on how many urls you got you may want to throttle your requests and if the urls come from a large file (gigabytes) you can use stream combined with the throttle.

async function fetchAll(url) {
return Promise.all(
url.map(
async (n) => fetch(n).then(r => r.json())
)
);
}
fetchAll([...])
.then(d => console.log(d))
.catch(e => console.error(e));
Will this work for you?

If you don't depend on every resource being a success I would have gone back to basics skipping async/await
I would process each fetch individual so I could catch the error for just the one that fails
function fetchAll() {
const result = []
const que = urls.map(url =>
fetch(url)
.then(res => res.json())
.then(item => {
result.push(item)
})
.catch(err => {
// could't fetch resource or the
// response was not a json response
})
)
return Promise.all(que).then(() => result)
}
Something good #TKoL said:
Promise.all errors whenever one of the internal promises errors, so whatever advice anyone gives you here, it will boil down to -- Make sure that you wrap the promises in an error handler before passing them to Promise.all

Regarding question 1, please refer to this:
Handling errors in Promise.all
Promise.all is all or nothing. It resolves once all promises in the array resolve, or reject as soon as one of them rejects. In other words, it either resolves with an array of all resolved values, or rejects with a single error.