I'm using Meteor (1.0.3) in general, but for one particular case I'm using a raw server side route to render a file -- so I'm outside a Meteor method.
I'm using node fs.writeFile/fs.readFile and exec commands to call out to Linux command-line utilities too.
My only point in brining this up is that the node calls are async of course. And so I'm opted to use the node Q library in order to manage async callbacks.
This all worked until I added a line to call out to the MongoDB database.
A call like so:
var record_name = Mongo_Collection_Name.findOne({_personId: userId}, {fields: {'_id': 0}});
Produces the following error:
[Error: Can't wait without a fiber]
The error only occurs when I wrap the function in a Promise.
For example, something like this will throw:
getRecordExample = function () {
var deferred = Q.defer();
var record_name = Mongo_Collection_Name.findOne({_personId: userId}, {fields: {'_id': 0}});
// do something
// if no error
deferred.resolve(record_name);
return deferred.promise;
}
If I use the Meteor Fibers library I don't get the error:
getRecordExample = function () {
var deferred = Q.defer();
Fiber = Npm.require('fibers');
var record_name
Fiber(function () {
record_name = Mongo_Collection_Name.findOne({_personId: userId});
}).run()
// do something
// if no error
deferred.resolve(record_name);
return deferred.promise;
}
but, the record_name variable is undefined outside the fiber, so I don't have a way to pass the variable outside of the Fiber scope as far as I can tell.
A More Precise Example
This is a little long, so you have to scroll down to see it all. I'm basically building a workflow here so there are processes and subprocesses.
// both/routes.js
Router.route('/get-route', function(req, res) {
// get the userId then start the workflow below
// using Promises here because these were firing concurrently
Q(userId)
.then(process_1)
.then(process_2)
.done();
}, { name: 'server-side-ir-route', where: 'server' }
// server.js
process_1 = function (userId) {
sub_process_1(userId);
sub_process_2(userId);
return userId;
}
process_2 = function (userId) {
sub_process_3(userId);
sub_process_4(userId);
return userId;
}
sub_process_1 = function (userId) {
var result = get_record_1(userId);
// do stuff with result
// using Q library to call out to async fs.writeFile, return Promise
fs_writeFile_promise(result)
.catch(function (error) {
console.log('error in sub_process_1_write', error);
})
.done(function () {
console.log('done with sub_process_1');
}
return userId;
}.future() // <-- if no future() here, the exception is thrown.
sub_process_2 = function (userId) {
var result = get_record_2(userId);
// do stuff with result
// using Q library to call out to async fs.writeFile, return Promise
fs_writeFile_promise(result)
.catch(function (error) {
console.log('error in sub_process_1_write', error);
})
.done(function () {
console.log('done with sub_process_1');
}
return userId;
}.future()
// async because of I/O operation (I think)
get_record_1 = function (userId) {
var record_1 = Mongo_Collection_Name.findOne({'userId': userId});
// do stuff
return record_1;
}
get_record_2 = function (userId) {
var record_2 = Mongo_Collection_Name.findOne({'userId': userId});
// do stuff
return record_2;
}
// async operation using Q library to return a Promise
fs_writeFile_promise = function (obj) {
var deferred = Q.defer();
fs.writeFile(obj.file, obj.datas, function (err, result) {
if (err) deferred.reject(err);
else deferred.resolve('write data completed');
});
return deferred.promise;
}
For now, lets assume that the process_2 function is exactly like process_1
Also, we should assume I have console.log('step_start') and console.log('step_end') in each function. This is what it would look like on the command line:
start processes
end processes
start processes 1
end processes 1
start processes 2
start sub processes 1
getting record 1
start sub processes 2
getting record 2
returning record 1
end sub processes 1
called writeData in sub process 1
returning record 2
called writeData in sub process 2
end processes 2
ending sub processes 1
The reason I had to place a Fiber (future) on the sub_process_1() function was because when I placed the function process_1() in the Q chain at the top I got the Error: Can't wait without a fiber].
If I remove the process_1() in the Q chain at the top and remove the .future() from sub_process_1() no exception is thrown.
Questions
Why does calling out to a Mongo collection within a Promise cause a
fiber error within a Meteor application?
Does calling a async function within a sync function in general cause the sync function to become a async function?
How do I solve this problem?
The most common way to solve this is wrap your asynchronous callbacks that use Meteor functions in Meteor.bindEnvironment().
If you are using the Meteor core WebApp package to handle your server side route, the code would be like this (also in meteorpad):
WebApp.connectHandlers.use(
'/test',
Meteor.bindEnvironment(function(req, res, next) {
var someSyncData = Players.findOne();
res.write(JSON.stringify(someSyncData));
res.end();
})
);
Working with fibers or promises yourself is unnecessary unless you are trying to get multiple async events to run concurrently.
To deal with file reading or other functions that are not already synchronous, Meteor also provides Meteor.wrapAsync() to make them synchronous.
There are also packages and a help page that give you other high level alternatives.
Related
I'm trying to iterate threw a list of item and do some actions on them by calling an API like this example :
for (i = 0; i < arr.length; i++) {
if (arr[i].id == 42) {
api.requestAction(arr[i].id, function(error, response){ });
}
}
Problem is the loop obviously ended before all the requests are done and the program exits. What should I do to manage it ? I saw the "Promise" method but don't really know how I can use it in this case or maybe there's an other solution.
Thank you by advance !
With node-fetch (a promisify http api) you can together with async/await halt the for loop until it's done but this requires node v6+ with --harmony-async-await flag added
const fetch = require('node-fetch')
async function foo() {
for (let item of arr) {
if (item.id == 42) {
let res = await fetch(url)
let body = await res.text()
console.log(body)
}
}
console.log('done (after request)')
}
now every time you add the async keyword in front of a function it will always return a promise that resolve/rejects when everything is done
foo().then(done, fail)
alternetive you can just wrap you api fn in a promise if you don't want to install node-fetch
await new Promise((rs, rj) => {
api.requestAction(arr[i].id, function(error, response){
error ? rj(error) : rs(response)
})
})
Install bluebird
npm install bluebird --save
Code
//require npm
var Promise = require("bluebird");
//code
//"promisify" converts traditional callback function into a Promise based function
var _requestAction = Promise.promisify(api.requestAction);
//loop over array
Promise.map(arr, function (value) {
if (value.id == 42) {
//async request
return _requestAction(value.id).then(function (_result) {
//success
console.log(_result);
}).catch(function (e) {
//error
console.error(e);
});
}
});
You could use async.js. It's an asyncronous control flow library which provides control flows for things like sequential loops, looping in parralel, and many other common flow control mechanism, check it out.
See code below, the code assumes that you're variable 'arr' is defined somewhere in scope.
npm install async
var async = require("async");
//Loop through each item, waiting for your
//asyncronous function to finish before continuing
//to move onto the next item in the array
//NOTE: This does not loop sequentially, if you want that function with asyncjs then user eachSeries
async.each(arr,
//Item is the current item being iterated over,
//callback is the callback you call to finish the current iteration, it accepts an error and result parameter callback(error, result);
function (item, callback) {
api.requestAction(item.id, function(error, response){
//Check for any errors...
if (error) return callback(error);
callback(null);
});
},
function (err, result) {
//You've now finished the loop
if (err) {
//Do something, you passed an error object to
//in one of the loop's iterations
}
//No errors, move on with your code..
});
Use Bluebird Promises:
var Promise = require('bluebird');
Promise.map(arrayOfIds, function(item){
return api.requestAction(item);
})
.then(function(response){
// all the requests are resolved here
})
if u want sequential execution of the ids then use Promise.mapSeries (is slow as it waits for task to finish)
I have a function that will poll a database ever x seconds. I am using the Q library so the function will return a promise. The function will ultimetly be used in a long chain of .then()s.
The function does give me the results that I expect but the function continues to run for 30-40 seconds after the results are returned. I cannot figure out why it would not exit right after I return.
var _ = require('lodash');
var pg = require('pg');
var Q = require('q');
connString = 'postgres://somedb_info';
var query = "SELECT * FROM job where jobid='somejobid123123'";
exports.run_poller = function () {
var deferred = Q.defer();
function exec_query(callback) {
pg.connect(connString, function(err, client, done) {
if(err) {
deferred.reject(err);
}
client.query(query, function(err, result) {
done();
if(err) {
return deferred.reject(err);
}
callback(result.rows[0]);
});
});
}
function wait_for(res){
if(res.status == 'COMPLETE') {
return deferred.resolve(res);
} else {
setTimeout(function(){
exec_query(wait_for);
}, 1000);
}
}
exec_query(wait_for);
return deferred.promise;
};
Just to test this I call the function from a main.js file like so:
var poller = require('./utils/poller').run_poller;
poller().then(console.log).catch(function(err) {console.log(err,'*');});
Why doesn't main.js exit right after the data is returned? Is there a better way to achieve this?
I see that pg maintains a connection pool. I would assume that it's either taking awhile to time out some shared resources or it just takes a little while to shutdown.
You may be able to just call process.exit(0) in your node app to force it to exit sooner if you know you're done with all your work.
Or, you may be able to find configuration settings that affect how the connection pool works.
On this doc page, there's an example like this that might help:
//disconnect client when all queries are finished
client.on('drain', client.end.bind(client));
You should read the doc for .end() to make sure you're using it correctly as there are some cases where it says it should not be called (though they may not apply if your done with all activity).
I'm trying to get my head around promises in JavaScript. I feel like I know what a promise is. However, I don't understand how to use them. In an attempt to learn, I decided to query a database from Node.js. In my code, I have one JavaScript file called test.js. Test.js looks like this:
Test.js
'use strict';
module.exports = function (app) {
var customerService = require('customer.js')(app);
var getCustomerTest = function() {
customerService.getCustomer(1).then(
function (customer) { console.log(customer); },
function (error) { console.log(error); }
);
};
};
Customer.js
'use strict';
module.exports = function(app) {
var _ = require('lodash');
var db = require('db');
return {
getCustomer: function(customerID) {
try {
console.log('querying the database...');
var database = db.connect(CONNECTION_STRING);
database.query('select * from customers where [ID]="' + customerID + '", function(error, result, response) {
if (error) {
// trigger promise error
} else {
// This throws an exception because displayMessage can't be found.
this.displayMessage('Success');
// trigger promise success
}
});
} catch (ex) {
// trigger promise error
}
},
displayMessage: function(message) {
console.log(new Date() + ' - ' + message);
}
};
};
I'm struggling trying to setup the promise in getCustomer. Especially since the call to the database call happens asynchronously. I feel like my call to customerService.getCustomer is the correct approach. However, once inside of getCustomer, I have two issues:
How do I setup / return my promise?
Why can't I call displayMessage after the database query is done? How do I do this?
Thank you JavaScript whiz!
"Why can't I call displayMessage after the database query is done"
You are getting the error because this is not referencing the object that contains getCustomer and displayMessage. This is because you are in a callback function and the context has changed.
You need to save a reference to the correct context and use that to access displayMessage
getCustomer: function(customerID) {
//saving the context
var that = this;
try {
...
database.query('select * from customers where [ID]="' + customerID + '", function(error, result, response) {
...
// Now use it here to call displayMessage
that.displayMessage('Success');
...
}
});
} catch (ex) {
...
}
},
"How do I setup / return my promise?"
You will need a promise library (unless you plan to make your own). For this I will show the use of the q library
There are a couple ways of doing this, but I will show the use of deferreds.
The basic process is:
Create a deferred object in a function that will call an async function/method.
Return the promise object
Set the appropriate callbacks for then/fail/fin etc on the promise object.
In the callback of the async function resolve or reject the deferred, passing any arguments that will be needed in the callbacks.
The appropriate callbacks that were set in step 2 will then be called in order.
Code
var Q = require("q");
...
getCustomer:{
var deferred = Q.defer(),
database = db.connect(CONNECTION_STRING);
database.query("some query", function(error, result, response) {
if(error){
//Anything passed will be passed to any fail callbacks
deferred.reject(error);
} else {
//Anything passed will be passed to any success callbacks
deferred.resolve(response);
}
});
//Return the promise
return deferred.promise;
}
...
customerService.getCustomer(1).then(function (customer) {
console.log(customer);
}).fail(function (error) {
console.log(error);
}).done();
The q library has quite a few helpful api functions. Some help with getting promises with Node async functions. Read the Adapting Node section of the readme to see how it is done.
JSFiddle Demo demonstrating in browser use of q
I want to call an asynchronous function inside a Meteor method and then return the result from that function to Meteor.call.
(How) is that possible?
Meteor.methods({
my_function: function(arg1, arg2) {
//Call other asynchronous function and return result or throw error
}
});
Use a Future to do so. Like this:
Meteor.methods({
my_function: function(arg1, arg2) {
// Set up a future
var fut = new Future();
// This should work for any async method
setTimeout(function() {
// Return the results
fut.ret(message + " (delayed for 3 seconds)");
}, 3 * 1000);
// Wait for async to finish before returning
// the result
return fut.wait();
}
});
Update:
To use Future starting from Meteor 0.5.1, you have to run the following code in your Meteor.startup method:
Meteor.startup(function () {
var require = __meteor_bootstrap__.require
Future = require('fibers/future');
// use Future here
});
Update 2:
To use Future starting from Meteor 0.6, you have to run the following code in your Meteor.startup method:
Meteor.startup(function () {
Future = Npm.require('fibers/future');
// use Future here
});
and then use the return method instead of the ret method:
Meteor.methods({
my_function: function(arg1, arg2) {
// Set up a future
var fut = new Future();
// This should work for any async method
setTimeout(function() {
// Return the results
fut['return'](message + " (delayed for 3 seconds)");
}, 3 * 1000);
// Wait for async to finish before returning
// the result
return fut.wait();
}
});
See this gist.
Recent versions of Meteor have provided the undocumented Meteor._wrapAsync function which turns a function with a standard (err, res) callback into a synchronous function, meaning that the current Fiber yields until the callback returns, and then uses Meteor.bindEnvironment to ensure that you retain the current Meteor environment variables (such as Meteor.userId()).
A simple use would be as the following:
asyncFunc = function(arg1, arg2, callback) {
// callback has the form function (err, res) {}
};
Meteor.methods({
"callFunc": function() {
syncFunc = Meteor._wrapAsync(asyncFunc);
res = syncFunc("foo", "bar"); // Errors will be thrown
}
});
You may also need to use function#bind to make sure that asyncFunc is called with the right context before wrapping it.
For more information see: https://www.eventedmind.com/tracks/feed-archive/meteor-meteor-wrapasync
Andrew Mao is right. Meteor now has Meteor.wrapAsync() for this kind of situation.
Here's the simplest way to do a charge via stripe and also pass a callback function:
var stripe = StripeAPI("key");
Meteor.methods({
yourMethod: function(callArg) {
var charge = Meteor.wrapAsync(stripe.charges.create, stripe.charges);
charge({
amount: amount,
currency: "usd",
//I passed the stripe token in callArg
card: callArg.stripeToken,
}, function(err, charge) {
if (err && err.type === 'StripeCardError') {
// The card has been declined
throw new Meteor.Error("stripe-charge-error", err.message);
}
//Insert your 'on success' code here
});
}
});
I found this post really helpful:
Meteor: Proper use of Meteor.wrapAsync on server
Another option is this package which achieves the similar goals.
meteor add meteorhacks:async
From the package README:
Async.wrap(function)
Wrap an asynchronous function and allow it to be run inside Meteor without callbacks.
//declare a simple async function
function delayedMessge(delay, message, callback) {
setTimeout(function() {
callback(null, message);
}, delay);
}
//wrapping
var wrappedDelayedMessage = Async.wrap(delayedMessge);
//usage
Meteor.methods({
'delayedEcho': function(message) {
var response = wrappedDelayedMessage(500, message);
return response;
}
});
I'm using the Node.JS driver for MongoDB, and I'd like to perform a synchronous query, like such:
function getAThing()
{
var db = new mongo.Db("mydatabase", server, {});
db.open(function(err, db)
{
db.authenticate("myuser", "mypassword", function(err, success)
{
if (success)
{
db.collection("Things", function(err, collection)
{
collection.findOne({ name : "bob"}, function(err, thing)
{
return thing;
});
});
}
});
});
}
The problem is, db.open is an asychronous call (it doesn't block), so the getAThing returns "undefined" and I want it to return the results of the query. I'm sure I could some sort of blocking mechanism, but I'd like to know the right way to do something like this.
ES 6 (Node 8+)
You can utilize async/await
await operator pauses the execution of asynchronous function until the Promise is resolved and returns the value.
This way your code will work in synchronous way:
const query = MySchema.findOne({ name: /tester/gi });
const userData = await query.exec();
console.log(userData)
Older Solution - June 2013 ;)
Now the Mongo Sync is available, this is the right way to make a synchronous MongoDB query in Node.js.
I am using this for the same. You can just write sync method like below:
var Server = require("mongo-sync").Server;
var server = new Server('127.0.0.1');
var result = server.db("testdb").getCollection("testCollection").find().toArray();
console.log(result);
Note: Its dependent on the node-fiber and some issues are there with it on windows 8.
Happy coding :)
There's no way to make this synchronous w/o some sort of terrible hack. The right way is to have getAThing accept a callback function as a parameter and then call that function once thing is available.
function getAThing(callback)
{
var db = new mongo.Db("mydatabase", server, {});
db.open(function(err, db)
{
db.authenticate("myuser", "mypassword", function(err, success)
{
if (success)
{
db.collection("Things", function(err, collection)
{
collection.findOne({ name : "bob"}, function(err, thing)
{
db.close();
callback(err, thing);
});
});
}
});
});
}
Node 7.6+ Update
async/await now provides a way of coding in a synchronous style when using asynchronous APIs that return promises (like the native MongoDB driver does).
Using this approach, the above method can be written as:
async function getAThing() {
let db = await mongodb.MongoClient.connect('mongodb://server/mydatabase');
if (await db.authenticate("myuser", "mypassword")) {
let thing = await db.collection("Things").findOne({ name: "bob" });
await db.close();
return thing;
}
}
Which you can then call from another async function as let thing = await getAThing();.
However, it's worth noting that MongoClient provides a connection pool, so you shouldn't be opening and closing it within this method. Instead, call MongoClient.connect during your app startup and then simplify your method to:
async function getAThing() {
return db.collection("Things").findOne({ name: "bob" });
}
Note that we don't call await within the method, instead directly returning the promise that's returned by findOne.
While it's not strictly synchronous, a pattern I've repeatedly adopted and found very useful is to use co and promisify yield on asynchronous functions. For mongo, you could rewrite the above:
var query = co( function* () {
var db = new mongo.Db("mydatabase", server, {});
db = promisify.object( db );
db = yield db.open();
yield db.authenticate("myuser", "mypassword");
var collection = yield db.collection("Things");
return yield collection.findOne( { name : "bob"} );
});
query.then( result => {
} ).catch( err => {
} );
This means:
You can write "synchronous"-like code with any asynchronous library
Errors are thrown from the callbacks, meaning you don't need the success check
You can pass the result as a promise to any other piece of code