I am trying to write a program that gets the documents from a mongo database with mongoose and process them using an API and then edits each document in the database with the results of the processing. My problem is that I have problems because I don't understand completely nodejs and the asynchronous. This is my code:
Model.find(function (err, tweets) {
if (err) return err;
for (var i = 0; i < tweets.length; i++) {
console.log(tweets[i].tweet);
api.petition(tweets[i].tweet)
.then(function(res) {
TweetModel.findOneAndUpdate({_id: tweets[i]._id}, {result: res}, function (err, tweetFound) {
if (err) throw err;
console.log(tweetFound);
});
})
.catch(function(err) {
console.log(err);
})
}
})
The problem is that in the findOneAndUpdate, tweets is undefined so it can't find that id. Any solution? Thanks
The core thing you are really missing is that the Mongoose API methods also use "Promises", but you seem to just be copying from documentation or old examples using callbacks. The solution to this is to convert to using Promises only.
Working with Promises
Model.find({},{ _id: 1, tweet: 1}).then(tweets =>
Promise.all(
tweets.map(({ _id, tweet }) =>
api.petition(tweet).then(result =>
TweetModel.findOneAndUpdate({ _id }, { result }, { new: true })
.then( updated => { console.log(updated); return updated })
)
)
)
)
.then( updatedDocs => {
// do something with array of updated documents
})
.catch(e => console.error(e))
Aside from the general conversion from callbacks, the main change is using Promise.all() to resolve the ouput from the Array.map() being processed on the results from .find() instead of the for loop. That is actually one of the biggest problems in your attempt, since the for cannot actually control when the async functions resolve. The other issue is "mixing callbacks", but that is what we are generally addressing here by only using Promises.
Within the Array.map() we return the Promise from the API call, chained to the findOneAndUpdate() which is actually updating the document. We also use new: true to actually return the modified document.
Promise.all() allows an "array of Promise" to resolve and return an array of results. These you see as updatedDocs. Another advantage here is that the inner methods will fire in "parallel" and not in series. This usually means a faster resolution, though it takes a few more resources.
Note also that we use the "projection" of { _id: 1, tweet: 1 } to only return those two fields from the Model.find() result because those are the only ones used in the remaining calls. This saves on returning the whole document for each result there when you don't use the other values.
You could simply just return the Promise from the findOneAndUpdate(), but I'm just adding in the console.log() so you can see the output is firing at that point.
Normal production use should do without it:
Model.find({},{ _id: 1, tweet: 1}).then(tweets =>
Promise.all(
tweets.map(({ _id, tweet }) =>
api.petition(tweet).then(result =>
TweetModel.findOneAndUpdate({ _id }, { result }, { new: true })
)
)
)
)
.then( updatedDocs => {
// do something with array of updated documents
})
.catch(e => console.error(e))
Another "tweak" could be to use the "bluebird" implementation of Promise.map(), which both combines the common Array.map() to Promise(s) implementation with the ability to control "concurrency" of running parallel calls:
const Promise = require("bluebird");
Model.find({},{ _id: 1, tweet: 1}).then(tweets =>
Promise.map(tweets, ({ _id, tweet }) =>
api.petition(tweet).then(result =>
TweetModel.findOneAndUpdate({ _id }, { result }, { new: true })
),
{ concurrency: 5 }
)
)
.then( updatedDocs => {
// do something with array of updated documents
})
.catch(e => console.error(e))
An alternate to "parallel" would be executing in sequence. This might be considered if too many results causes too many API calls and calls to write back to the database:
Model.find({},{ _id: 1, tweet: 1}).then(tweets => {
let updatedDocs = [];
return tweets.reduce((o,{ _id, tweet }) =>
o.then(() => api.petition(tweet))
.then(result => TweetModel.findByIdAndUpdate(_id, { result }, { new: true })
.then(updated => updatedDocs.push(updated))
,Promise.resolve()
).then(() => updatedDocs);
})
.then( updatedDocs => {
// do something with array of updated documents
})
.catch(e => console.error(e))
There we can use Array.reduce() to "chain" the promises together allowing them to resolve sequentially. Note the array of results is kept in scope and swapped out with the final .then() appended to the end of the joined chain since you need such a technique to "collect" results from Promises resolving at different points in that "chain".
Async/Await
In modern environments as from NodeJS V8.x which is actually the current LTS release and has been for a while now, you actually have support for async/await. This allows you to more naturally write your flow
try {
let tweets = await Model.find({},{ _id: 1, tweet: 1});
let updatedDocs = await Promise.all(
tweets.map(({ _id, tweet }) =>
api.petition(tweet).then(result =>
TweetModel.findByIdAndUpdate(_id, { result }, { new: true })
)
)
);
// Do something with results
} catch(e) {
console.error(e);
}
Or even possibly process sequentially, if resources are an issue:
try {
let cursor = Model.collection.find().project({ _id: 1, tweet: 1 });
while ( await cursor.hasNext() ) {
let { _id, tweet } = await cursor.next();
let result = await api.petition(tweet);
let updated = await TweetModel.findByIdAndUpdate(_id, { result },{ new: true });
// do something with updated document
}
} catch(e) {
console.error(e)
}
Noting also that findByIdAndUpdate() can also be used as matching the _id is already implied so you don't need a whole query document as a first argument.
BulkWrite
As a final note if you don't actually need the updated documents in response at all, then bulkWrite() is the better option and allows the writes to generally process on the server in a single request:
Model.find({},{ _id: 1, tweet: 1}).then(tweets =>
Promise.all(
tweets.map(({ _id, tweet }) => api.petition(tweet).then(result => ({ _id, result }))
)
).then( results =>
Tweetmodel.bulkWrite(
results.map(({ _id, result }) =>
({ updateOne: { filter: { _id }, update: { $set: { result } } } })
)
)
)
.catch(e => console.error(e))
Or via async/await syntax:
try {
let tweets = await Model.find({},{ _id: 1, tweet: 1});
let writeResult = await Tweetmodel.bulkWrite(
(await Promise.all(
tweets.map(({ _id, tweet }) => api.petition(tweet).then(result => ({ _id, result }))
)).map(({ _id, result }) =>
({ updateOne: { filter: { _id }, update: { $set: { result } } } })
)
);
} catch(e) {
console.error(e);
}
Pretty much all of the combinations shown above can be varied into this as the bulkWrite() method takes an "array" of instructions, so you can construct that array from the processed API calls out of every method above.
Related
Let me first show you what the code looks like
Cart.getCompleteCart((cart)=>{
let products = []; *//this Array has to be filled*
totalPrice = cart.totalPrice;
for(let prod of cart.products){
Product.getProductDetails(prod.productId, productDetails => {
products.push({
product: productDetails,
productCount: prod.productCount
});
});
}
console.log("Products are ->", products); *//this line is running before for loop!*
}
console.log() is running before the for loop completes its work.
How to run "for loop" kind of in sync with the console.log() ?
I assume that Product.getProductDetails() is an async method (it looks like a query to an API).
console.log() is not "running before the for loop", it is just that the tasks performed by Product.getProductDetails() are being handled asynchronously.
Since it also seems that the Product.getProductDetails() method works with callbacks, one way to achieve your goal would be to promisify each call and then condensate all the promises into a single one by using Promise.all() method.
Something like this should do the trick:
Cart.getCompleteCart((cart) => {
const promises = [];
for (let prod of cart.products) {
promises.push(
new Promise((resolve) => {
Product.getProductDetails(prod.productId, (productDetails) => {
resolve({
product: productDetails,
productCount: prod.productCount
});
});
})
);
}
Promise.all(promises).then((products) => {
console.log('Products are ->', products);
});
});
Or:
Cart.getCompleteCart((cart) => {
Promise.all(
cart.products.map((prod) => {
return new Promise((resolve) => {
Product.getProductDetails(prod.productId, (productDetails) => {
resolve({
product: productDetails,
productCount: prod.productCount
});
});
});
})
).then((products) => {
console.log('Products are ->', products);
});
});
Promise.all is designed for pretty much this exact use-case:
// A dummy "Product" with a dummy "getProductDetails" implementation
// so that we can have a working test example
let Product = {
getProductDetails: (productId, callback) => {
setTimeout(() => {
callback({ type: 'productDetails', productId });
}, 100 + Math.floor(Math.random() * 200));
}
};
// This is the function you're looking for:
let getCompleteCart = async (cart) => {
return Promise.all(cart.products.map(async ({ productId, productCount }) => ({
product: await new Promise(resolve => Product.getProductDetails(productId, resolve)),
productCount
})));
}
let exampleCart = {
products: [
{ productId: 982, productCount: 1 },
{ productId: 237, productCount: 2 },
{ productId: 647, productCount: 5 }
]
};
getCompleteCart(exampleCart).then(console.log);
A breakdown of getCompleteCart:
let getCompleteCart = async (cart) => {
return Promise.all(cart.products.map(async ({ productId, productCount }) => ({
product: await new Promise(resolve => Product.getProductDetails(productId, resolve)),
productCount
})));
}
Promise.all (mdn) is purpose-built to wait for every promise in an Array of promises to resolve
We need to supply an Array of Promises to Promise.all. This means we need to convert our Array of data (cart.products) to an Array of Promises; Array.protoype.map is the perfect tool for this.
The function provided to cart.products.map converts every product in the cart to an Object that looks like { product: <product details>, productCount: <###> }.
The tricky thing here is getting the value for the "product" property, since this value is async (returned by a callback). The following line creates a promise that resolves to the value returned by Product.getProductDetails, using the Promise constructor (mdn):
new Promise(resolve => Product.getProductDetails(productId, resolve))
The await keyword allows us to convert this promise into the future value it results in. We can assign this future value to the "product" attribute, whereas the "productCount" attribute is simply copied from the original item in the cart:
{
product: await new Promise(resolve => Product.getProductDetails(productId, resolve)),
productCount
}
In order to run console.log at the right time we need to wait for all product details to finish compiling. Fortunately this is handled internally by getCompleteCart. We can console.log at the appropriate time by waiting for getCompleteCart to resolve.
There are two ways to do this:
Using Promise.prototype.then (mdn):
getCompleteCart(exampleCart).then(console.log);
Or, if we're in an async context we can use await:
let results = await getCompleteCart(exampleCart);
console.log(results);
I want to return results from two database queries in one object.
function route(start, end) {
return new Promise((resolve, reject) => {
const queries = routeQuery(start, end);
var empty_obj = new Array();
for (i=0; i<queries.length; i++) {
query(queries[i], (err, res) => {
if (err) {
reject('query error', err);
console.log(err);
return;
} else {
empty_obj.push(res.rows);
}});
}
console.log(empty_obj);
resolve({coords: empty_obj});
});
}
This is my code right now, the queries are working fine but for some reason, pushing each result into an empty array does not work. When I console log that empty object, it stays empty. The goal is to resolve the promise with the generated object containing the two query results. I'm using node-postgres for the queries.
Output of res is an object:
{
command: 'SELECT',
rowCount: 18,
oid: null,
rows: [
{ ...
I suggest you turn your query function into a Promise so that you can use Promise.all:
// turn the callback-style asynchronous function into a `Promise`
function queryAsPromise(arg) {
return new Promise((resolve, reject) => {
query(arg, (err, res) => {
if (err) {
console.error(err);
reject(err);
return;
}
resolve(res);
});
});
}
Then, you could do the following in your route function:
function route(start, end) {
const queries = routeQuery(start, end);
// use `Promise.all` to resolve with
// an array of results from queries
return Promise.all(
queries.map(query => queryAsPromise(query))
)
// use `Array.reduce` w/ destructing assignment
// to combine results from queries into a single array
.then(results => results.reduce(
(acc, item) => [...acc, ...item.rows],
[]
))
// return an object with the `coords` property
// that contains the final array
.then(coords => {
return { coords };
});
}
route(1, 10)
.then(result => {
// { coords: [...] }
})
.catch(error => {
// handle errors appropriately
console.error(error);
});
References:
Promise.all - MDN
Array.reduce - MDN
Destructing assignment - MDN
Hope this helps.
The issue you currently face is due to the fact that:
resolve({coords: empty_obj});
Is not inside the callback. So the promise resolves before the query callbacks are called and the rows are pushed to empty_obj. You could move this into the query callback in the following manner:
empty_obj.push(res.rows); // already present
if (empty_obj.length == queries.length) resolve({coords: empty_obj});
This would resolve the promises when all rows are pushed, but leaves you with another issue. Callbacks might not be called in order. Meaning that the resulting order might not match the queries order.
The easiest way to solve this issue is to convert each individual callback to a promise. Then use Promise.all to wait until all promises are resolved. The resulting array will have the data in the same order.
function route(start, end)
const toPromise = queryText => new Promise((resolve, reject) => {
query(queryText, (error, response) => error ? reject(error) : resolve(response));
});
return Promise.all(routeQuery(start, end).map(toPromise))
.then(responses => ({coords: responses.map(response => response.rows)}))
.catch(error => {
console.error(error);
throw error;
});
}
I would like to insert products into array and then continue using that array. I don't have an expirience using Promisses. Bellow is my code.
This is the function that I would like to return an array.
const findProduct = async function(request) {
const products = [];
await Promise.all(
request.products.forEach(async product => {
await db
.get()
.collection('products')
.findOne({ _id: product.productId.toString() }, (err, result) => {
if (err) throw new Error('exception!');
products.push(result);
});
})
)
.then(() => {
return products;
})
.catch(e => e);
};
Products is always undefined.
Shoud I maybe return Promise.all i save it into an array, then use that array?
This is how I plan to use array
const purchase = new Purchase(req.body);
const products = await findProduct(req.body);
purchase.products = [...products];
In the context of your findProduct function, using async/await syntax doesn't make a lot of sense if you intend to await the result of findProduct using const products = await findProduct(req.body);
Instead you want to define your function using only Promises.
Firstly, db.collection.findOne() makes use of a callback for asynchronous events, to use this in a Promise chain, you must wrap this in a Promise. Which can be done like so:
new Promise((resolve, reject) => {
db
.get()
.collection('products')
.findOne({ _id: product.productId.toString() }, (err, result) => { err ? reject(err) : resolve(result) })
});
Here (err, result) => { err ? reject(err) : resolve(result) } is just a concise form of
(err, result) => {
if (err) {
reject(err);
} else {
resolve(result);
}
}
Next, you must pass an array to Promise.all(). Because you already have the array request.products, you can convert (or map) each value to it's corresponding Promise to get it's data. This is done using Array.prototype.map. When used this way, Promise.all() will resolve to an array of product data objects in the same order of request.products.
Mixing this in to your code, yields:
const findProduct = function(request) {
return Promise.all(
request.products.map(product => {
return new Promise((resolve, reject) => {
db
.get()
.collection('products')
.findOne({ _id: product.productId.toString() }, (err, result) => err ? reject(err) : resolve(result))
});
})
);
};
Note here that I removed .then(()=>products) (because Promise.all will return products itself thanks to the promise above) and .catch(e=>e) (because you shouldn't be ignoring errors like this).
With these changes, you can now use findProduct as you have used it in your question.
Update:
Seems that the MongoDB API is Promises compliant, which means we can remove the Promise callback wrapper and simplify it down to:
const findProduct = function(request) {
return Promise.all(
request.products.map(product => {
return db
.get()
.collection('products')
.findOne({ _id: product.productId.toString() });
})
);
};
Promise.all() expects an array of Promises and map will return an array. Since db will also return a Promise we just feed those promises into Promise.all()
Edit: forgot to return Promise.all() which returns a promise so it can be consumed with the await findproducts() or findproducts().then()
const findProduct = function (request) {
return Promise.all(
request.products.map(product => {
return db
.get()
.collection('products')
.findOne({ _id: product.productId.toString() }
});
})
)};
Since quite a lot of people ask: "How do i get something from a Promise?" --> use the resolve value
Here a long explanation why this works: wait() will return a Promise that is mapped into an array with map() which returns this array. So when map is done we have an array with 3 Promises returned by the 3 wait() calls. This array is passed into Promise.all() which will wait for all the Promises to resolve and resolves itself to an array with all the resolve values of the Promises from the array. Since findProduct() will return this Promise from Promise.all() when can access it with findProduct().then() or await findProduct()
const start = { products: ['milk', 'butter', 'eggs'] }
const findProduct = function (request) {
return Promise.all(
request.products.map(product => {
return wait(product)
})
)
};
function wait(product) {
return new Promise((resolve) => {
setTimeout(() => { resolve(product) }, 5000)
})
}
findProduct(start).then(res => console.log(res))
// logs [ 'milk', 'butter', 'eggs' ]
Edit: To answer the comment about .then() returning a Promise
findProduct(start)
.then(res => console.log(res))
.then(res => console.log(res))
// logs [ 'milk', 'butter', 'eggs' ] and undefined
findProduct(start)
.then(res => {console.log(res); return res})
.then(res => console.log(res))
// logs 2x [ 'milk', 'butter', 'eggs' ]
I'm trying to have a loop with some db calls, and once their all done ill send the result. - Using a promise, but if i have my promise after the callback it dosent work.
let notuser = [];
let promise = new Promise((resolve, reject) => {
users.forEach((x) => {
User.find({
/* query here */
}, function(err, results) {
if(err) throw err
if(results.length) {
notuser.push(x);
/* resolve(notuser) works here - but were not done yet*/
}
})
});
resolve(notuser); /*not giving me the array */
}).then((notuser) => {
return res.json(notuser)
})
how can i handle this ?
Below is a function called findManyUsers which does what you're looking for. Mongo find will return a promise to you, so just collect those promises in a loop and run them together with Promise.all(). So you can see it in action, I've added a mock User class with a promise-returning find method...
// User class pretends to be the mongo user. The find() method
// returns a promise to 'find" a user with a given id
class User {
static find(id) {
return new Promise(r => {
setTimeout(() => r({ id: `user-${id}` }), 500);
});
}
}
// return a promise to find all of the users with the given ids
async function findManyUsers(ids) {
let promises = ids.map(id => User.find(id));
return Promise.all(promises);
}
findManyUsers(['A', 'B', 'C']).then(result => console.log(result));
I suggest you take a look at async it's a great library for this sort of things and more, I really think you should get used to implement it.
I would solve your problem using the following
const async = require('async')
let notuser = [];
async.forEach(users, (user, callback)=>{
User.find({}, (err, results) => {
if (err) callback(err)
if(results.length) {
notUser.push(x)
callback(null)
}
})
}, (err) => {
err ? throw err : return(notuser)
})
However, if you don't want to use a 3rd party library, you are better off using promise.all and await for it to finish.
EDIT: Remember to install async using npm or yarn something similar to yarn add async -- npm install async
I used #danh solution for the basis of fixing in my scenario (so credit goes there), but thought my code may be relevant to someone else, looking to use standard mongoose without async. I want to gets a summary of how many reports for a certain status and return the last 5 for each, combined into one response.
const { Report } = require('../../models/report');
const Workspace = require('../../models/workspace');
// GET request to return page of items from users report
module.exports = (req, res, next) => {
const workspaceId = req.params.workspaceId || req.workspaceId;
let summary = [];
// returns a mongoose like promise
function addStatusSummary(status) {
let totalItems;
let $regex = `^${status}$`;
let query = {
$and: [{ workspace: workspaceId }, { status: { $regex, $options: 'i' } }],
};
return Report.find(query)
.countDocuments()
.then((numberOfItems) => {
totalItems = numberOfItems;
return Report.find(query)
.sort({ updatedAt: -1 })
.skip(0)
.limit(5);
})
.then((reports) => {
const items = reports.map((r) => r.displayForMember());
summary.push({
status,
items,
totalItems,
});
})
.catch((err) => {
if (!err.statusCode) {
err.statusCode = 500;
}
next(err);
});
}
Workspace.findById(workspaceId)
.then((workspace) => {
let promises = workspace.custom.statusList.map((status) =>
addStatusSummary(status)
);
return Promise.all(promises);
})
.then(() => {
res.status(200).json({
summary,
});
})
.catch((err) => {
if (!err.statusCode) {
err.statusCode = 500;
}
next(err);
});
};
I need to retrieve a list of collections using express and mongodb module.
First, ive retrieved a list of collection names which works, I then retrieve the data of those given collections in a loop. My problem is in getColAsync():
getColAsync() {
return new Promise((resolve, reject) => {
this.connectDB().then((db) => {
var allCols = [];
let dbase = db.db(this.databaseName);
dbase.listCollections().toArray((err, collectionNames) => {
if(err) {
console.log(err);
reject(err);
}
else {
for(let i = 0; i < collectionNames.length; i++) {
dbase.collection(collectionNames[i].name.toString()).find({}).toArray((err, collectionData) => {
console.log("current collection data: " + collectionData);
allCols[i] = collectionData;
})
}
console.log("done getting all data");
resolve(allCols);
}
})
})
})
}
connectDB() {
if(this.dbConnection) {
// if connection exists
return this.dbConnection;
}
else {
this.dbConnection = new Promise((resolve, reject) => {
mongoClient.connect(this.URL, (err, db) => {
if(err) {
console.log("DB Access: Error on mongoClient.connect.");
console.log(err);
reject(err);
}
else {
console.log("DB Access: resolved.");
resolve(db);
}
});
});
console.log("DB Access: db exists. Connected.");
return this.dbConnection;
}
}
In the forloop where i retrieve every collection, the console.log("done getting all data") gets called and the promise gets resolved before the forloop even begins. for example:
done getting all data
current collection data: something
current collection data: something2
current collection data: something3
Please help
The Problem
The problem in your code is this part:
for (let i = 0; i < collectionNames.length; i++) {
dbase.collection(collectionNames[i].name.toString()).find({}).toArray((err, collectionData) => {
console.log("current collection data: " + collectionData);
allCols[i] = collectionData;
})
}
console.log("done getting all data");
resolve(allCols);
You should notice that resolve(allCols); is called right after the for loop ends, but each iteration of the loop doesn't wait for the toArray callback to be called.
The line dbase.collection(collectionNames[i].name.toString()).find({}).toArray(callback) is asynchronous so the loop will end, you'll call resolve(allCols);, but the .find({}).toArray code won't have completed yet.
The Solution Concept
So, basically what you did was:
Initialize an array of results allCols = []
Start a series of async operations
Return the (still empty) array of results
As the async operations complete, fill the now useless results array.
What you should be doing instead is:
Start a series of async operations
Wait for all of them to complete
Get the results from each one
Return the list of results
The key to this is the Promise.all([/* array of promises */]) function which accepts an array of promises and returns a Promise itself passing downstream an array containing all the results, so what we need to obtain is something like this:
const dataPromises = []
for (let i = 0; i < collectionNames.length; i++) {
dataPromises[i] = /* data fetch promise */;
}
return Promise.all(dataPromises);
As you can see, the last line is return Promise.all(dataPromises); instead of resolve(allCols) as in your code, so we can no longer execute this code inside of a new Promise(func) constructor.
Instead, we should chain Promises with .then() like this:
getColAsync() {
return this.connectDB().then((db) => {
let dbase = db.db(this.databaseName);
const dataPromises = []
dbase.listCollections().toArray((err, collectionNames) => {
if (err) {
console.log(err);
return Promise.reject(err);
} else {
for (let i = 0; i < collectionNames.length; i++) {
dataPromises[i] = new Promise((res, rej) => {
dbase.collection(collectionNames[i].name.toString()).find({}).toArray((err, collectionData) => {
console.log("current collection data: " + collectionData);
if (err) {
console.log(err);
reject(err);
} else {
resolve(collectionData);
}
});
});
}
console.log("done getting all data");
return Promise.all(dataPromises);
}
});
})
}
Notice now we return a return this.connectDB().then(...), which in turn returns a Promise.all(dataPromises); this returning new Promises at each step lets us keep alive the Promise chain, thus getColAsync() will itself return a Promise you can then handle with .then() and .catch().
Cleaner Code
You can clean up your code a bit as fallows:
getColAsync() {
return this.connectDB().then((db) => {
let dbase = db.db(this.databaseName);
const dataPromises = []
// dbase.listCollections().toArray() returns a promise itself
return dbase.listCollections().toArray()
.then((collectionsInfo) => {
// collectionsInfo.map converts an array of collection info into an array of selected
// collections
return collectionsInfo.map((info) => {
return dbase.collection(info.name);
});
})
}).then((collections) => {
// collections.map converts an array of selected collections into an array of Promises
// to get each collection data.
return Promise.all(collections.map((collection) => {
return collection.find({}).toArray();
}))
})
}
As you see the main changes are:
Using mondodb functions in their promise form
Using Array.map to easily convert an array of data into a new array
Below I also present a variant of your code using functions with callbacks and a module I'm working on.
Promise-Mix
I'm recently working on this npm module to help get a cleaner and more readable composition of Promises.
In your case I'd use the fCombine function to handle the first steps where you select the db and fetch the list of collection info:
Promise.fCombine({
dbase: (dbURL, done) => mongoClient.connect(dbURL, done),
collInfos: ({ dbase }, done) => getCollectionsInfo(dbase, done),
}, { dbURL: this.URL })
This results in a promise, passing downstream an object {dbase: /* the db instance */, collInfos: [/* the list of collections info */]}. Where getCollectionNames(dbase, done) is a function with callback pattern like this:
getCollectionsInfo = (db, done) => {
let dbase = db.db(this.databaseName);
dbase.listCollections().toArray(done);
}
Now you can chain the previous Promise and convert the list of collections info into selected db collections, like this:
Promise.fCombine({
dbase: ({ dbURL }, done) => mongoClient.connect(dbURL, done),
collInfos: ({ dbase }, done) => getCollectionsInfo(dbase, done),
}, { dbURL: this.URL }).then(({ dbase, collInfos }) => {
return Promise.resolve(collInfos.map((info) => {
return dbase.collection(info.name);
}));
})
Now downstream we have a the list of selected collections from our db and we should fetch data from each one, then merge the results in an array with collection data.
In my module I have a _mux option which creates a PromiseMux which mimics the behaviour and composition patterns of a regular Promise, but it's actually working on several Promises at the same time. Each Promise gets in input one item from the downstream collections array, so you can write the code to fetch data from a generic collection and it will be executed for each collection in the array:
Promise.fCombine({
dbase: ({ dbURL }, done) => mongoClient.connect(dbURL, done),
collInfos: ({ dbase }, done) => getCollectionsInfo(dbase, done),
}, { dbURL: this.URL }).then(({ dbase, collInfos }) => {
return Promise.resolve(collInfos.map((info) => {
return dbase.collection(info.name);
}));
})._mux((mux) => {
return mux._fReduce([
(collection, done) => collection.find({}).toArray(done)
]).deMux((allCollectionsData) => {
return Promise.resolve(allCollectionsData);
})
});
In the code above, _fReduce behaves like _fCombine, but it accepts an array of functions with callbacks instead of an object and it passes downstream only the result of the last function (not a structured object with all the results). Finally deMux executes a Promise.all on each simultaneous Promise of the mux, putting together their results.
Thus the whole code would look like this:
getCollectionsInfo = (db, done) => {
let dbase = db.db(this.databaseName);
dbase.listCollections().toArray(done);
}
getCollAsync = () => {
return Promise.fCombine({
/**
* fCombine uses an object whose fields are functions with callback pattern to
* build consecutive Promises. Each subsequent functions gets as input the results
* from previous functions.
* The second parameter of the fCombine is the initial value, which in our case is
* the db url.
*/
dbase: ({ dbURL }, done) => mongoClient.connect(dbURL, done), // connect to DB, return the connected dbase
collInfos: ({ dbase }, done) => getCollectionsInfo(dbase, done), // fetch collection info from dbase, return the info objects
}, { dbURL: this.URL }).then(({ dbase, collInfos }) => {
return Promise.resolve(collInfos.map((info) => {
/**
* we use Array.map to convert collection info into
* a list of selected db collections
*/
return dbase.collection(info.name);
}));
})._mux((mux) => {
/**
* _mux splits the list of collections returned before into a series of "simultaneous promises"
* which you can manipulate as if they were a single Promise.
*/
return mux._fReduce([ // this fReduce here gets as input a single collection from the retrieved list
(collection, done) => collection.find({}).toArray(done)
]).deMux((allCollectionsData) => {
// finally we can put back together all the results.
return Promise.resolve(allCollectionsData);
})
});
}
In my module I tried to avoid most common anti-pattern thought there still is some Ghost Promise which I'll be working on.
Using the promises from mongodb this would get even cleaner:
getCollAsync = () => {
return Promise.combine({
dbase: ({ dbURL }) => { return mongoClient.connect(dbURL); },
collInfos: ({ dbase }) => {
return dbase.db(this.databaseName)
.listCollections().toArray();
},
}, { dbURL: this.URL }).then(({ dbase, collInfos }) => {
return Promise.resolve(collInfos.map((info) => {
return dbase.collection(info.name);
}));
}).then((collections) => {
return Promise.all(collections.map((collection) => {
return collection.find({}).toArray();
}))
});
}