I am new to node js, and have a method in my code which saves information in the following way:
var saveInformation = function(query, update, options) {
asyncTasks.push(function(done) {
what.findOneAndUpdate(query, update, options, done)
})
}
saveInformation(query, update, options)
I call the saveInformation method on several occasions, and need to write to the log files every time this method is called. What is the easiest way of doing this?
I tried just adding my log (I am using winston module) right after, like this:
saveInformation(query, update, options)
logger.info('Hello')
However this sometimes saves and sometimes not (i.e. it will only log this once, however this method is run several times throughout my program).
So I am wondering, is there a way to add another method to the saveInformation line to ensure that they are both run at the same time, so when saveInformation is saved to the dbs, I get a log created too?
The call to the logging function should be in the callback of the function what.findeOneAndUpdate which is done in this case, this way it will execute after each operation.
// callback for findOneAndUpdate
var done = function(err, result) {
if(err) {
throw err
}
logger.info('Result of findOneAndUpdate: ', result)
}
//
// original code
var saveInformation = function(query, update, options) {
asyncTasks.push(function(done) {
what.findOneAndUpdate(query, update, options, done)
})
}
saveInformation(query, update, options)
This way whenever saveInformation is called the logger will be called if the callback is called with no error, you can also modify that to log errors as well if you like.
Related
I am trying to write a script that will query a local database and send the results to a Google Spreadsheet. Where I am currently stuck and looking for help is how do I get the results of my query.
var myQuery = [];
function runQuery(query, callback) {
client.connect(function (err) {
if (err) throw err;
client.query(query, function (err, result) {
if (err) {
console.log(err);
}
callback({ result: result });
client.end(function (err) {
if (err) throw err;
});
});
});
}
runQuery('SELECT item_number FROM item WHERE item_number LIKE \'11-%\' LIMIT 10', function (resultsObject) {
myQuery = JSON.stringify(resultsObject.result.rows);
console.log('Did stringify work?');
console.log(myQuery);
});
console.log(myQuery);
My output:
Info: Start process (3:46:11 PM)
[]
Did stringify work?
[{"item_number":"11-0011"},{"item_number":"11-0012"},{"item_number":"11-1255"},{"item_number":"11-0052"},{"item_number":"11-0060"},{"item_number":"11-1256"},{"item_number":"11-1281"},{"item_number":"11-0659"},{"item_number":"11-0660"},{"item_number":"11-0054"}]
Info: End process (3:46:12 PM)
I think I understand what's happening, the scope of the variable myQuery is set and printed within the runQuery function just fine, but outside of that function it's not being actually set. How do I get around this?
Very new to JavaScript and NodeJS so I hope I'm using the correct terminology.
Thanks in advance!
The empty myQuery has already been consoled before you actually put values into it. This issue is caused by Javascript's Asynchronous mechanism. If you want to do something with the result, you have to use a callback function. Just like what you did when you accessed result from runQuery.
Another way to do it is promise. You can return a promise in some function and use the promise to access result.
If you decide to use promise, here is the code:
function getresult(parameters) {
return new Promise((resolve, reject) => {
runQuery(parameter, resolve, reject);
});
}
Using this function, you passed resolve as callback to runQuery. You do want to add another err callback function to catch err in runQuery. In promise you use reject as err callback.
When you want to do something with the result:
getresult(parameters)
.then((result) => {put your code here. result variable is your result})
.catch((err) => {do whatever with err});
This is not a scope issue. It is a timing issue. The query takes some time to execute. JavaScript is 'non-blocking' meaning it will start an asynchronous process (like doing a DB query - really virtually all I/O operations) and return the result later. In your code you are using a callback that is run once the data becomes available.
So, with that in mind, the way to think about your code is that you are doing things in this order:
1) declaring your function runQuery - this happens first since in JavaScript function declarations are moved to the top of the script they are in (a process called hoisting)
2) your variable myQuery is established and initialized (to an empty array)
3) your query is kicked off by a call to runQuery; this only starts the process
4) your console.log(myQuery) at the end of your script is run
5) some time later the query completes and your callback is called with the data where your other console.log() shows the returned data
I've looked high and low, and can only find how to write async functions, which I already understand.
What I am trying to do is run an async method in a triggered event [EventEmitter], but such a simple thing seems to be just simply not possible as I can find.
Consider the following...
// Your basic async method..
function doSomething(callback) {
var obj = { title: 'hello' };
// Fire an event for event handlers to alter the object.
// EvenEmitters are called synchronously
eventobj.emit('alter_object', obj);
callback(null, obj);
}
// when this event is fired, I want to manipulate the data
eventobj.on('alter_object', function(obj) {
obj.title += " world!";
// Calling this async function here means that our
// event handler will return before our data is retrieved.
somemodule.asyncFunction(callback(err, data) {
obj.data = data;
});
});
As you can see in the last few lines, the event handler will finish before the object's data property is added.
What I need is something where I can turn the async function into an sync function and get the results there and then. so for example...
obj.data = somemodule.asyncFunction();
I've looked at the wait.for module, the async module, and none of these will not work. I've even looked into the yield method, but it seems not yet fully implemented into the V8 engine.
I've also tried using a while loop too wait for data to populate, but this just brings with it the CPU overload issue.
Has anyone experienced this and found a design pattern to get around this?
You cannot turn an async function into a synchronous one in node.js. It just cannot be done.
If you have an asynchronous result, you cannot return it synchronously or wait for it. You will have to redesign the interface to use an asynchronous interface (which nearly always involves passing in a callback that will be called when the result is ready).
If you're wanting to do something after you .emit() an event that is itself going to do something asynchronously and you want to wait until after the async thing finished, then the event emitter is probably not the right interface. You'd rather have a function call that returns a promise or takes a callback as an argument. You could manipulate an eventEmitter to use this, but you'd have to post back a second event when the async operation finished and have the original caller not do it's second part until it receives the second event (which is really not a good way to go).
Bottom line - you need a different design that works with async responses (e.g. callbacks or promises).
Seems what I wanted to do is just not possible and the two models conflict. To achieve what I wanted in the end, I encapsulated my modules into an array-like object with some event methods that will pass it on to the module's object, which inherited the async-eventemitter class.
So, think of it like so...
My custom app modules may inherit the async-eventemitter module so they have the .on() and .emit(), etc. methods.
I create a customised array item, that will allow me to pass an event on to the module in question that will work asynchronously.
The code I created (and this is by no means complete or perfect)...
// My private indexer for accessing the modules array (below) by name.
var module_dict = {};
// An array of my modules on my (express) app object
app.modules = [];
// Here I extended the array with easier ways to add and find modules.
// Haven't removed some code to trim down this. Let me know if you want the code.
Object.defineProperty(app.modules, 'contains', { enumerable: false, ... });
Object.defineProperty(app.modules, 'find', { enumerable: false, ... });
// Allows us to add a hook/(async)event to a module, if it exists
Object.defineProperty(app.modules, 'on', { enumerable: false, configurable: false, value: function(modulename, action, func) {
if (app.modules.contains(modulename)) {
var modu = app.modules.find(modulename);
if (modu.module && modu.module['on']) {
// This will pass on the event to the module's object that
// will have the async-eventemitter inherited
modu.module.on(action, func);
}
}
} });
Object.defineProperty(app.modules, 'once', { enumerable: false, configurable: false, value: function(modulename, action, func) {
if (app.modules.contains(modulename)) {
var modu = app.modules.find(modulename);
if (modu.on) {
modu.on(action, func);
}
}
} });
This then allows me to bind an event handler to a module by simply calling something like the following... .on(module_name, event_name, callback)
app.modules.on('my_special_module_name', 'loaded', function(err, data, next) {
// ...async stuff, that then calls next to continue to the next event...
if (data.filename.endsWith('.jpg'))
data.dimensions = { width: 100, height: 100 };
next(err, data);
});
And then to execute it I would do something like (express)...
app.get('/foo', function(req, res, next) {
var data = {
filename: 'bar.jpg'
};
// Now have event handlers alter/update our data
// (eg, extend an object about a file with image data if that file is an image file).
my_special_module.emit('loaded', data, function(err, data) {
if (err) next(err);
res.send(data);
next();
});
});
Again, this is just an example of what I did, so i've probably missed something in my copy above, but effectively it's the design I ended up using and it worked like a treat, and I was able to extend data on an object before being pushed out to my HTTP response, without having to replace the main [expressjs] object's standard EventEmitter model.
(eg, I added image data for files that we're loaded that we're image files. If anyone wants the code, let me know, I am more than happy to share what I did)
Whats the proper way to wrap a mongo query/insert in a function to make it reusable? I have an update operation that may take place in various places and want to only write it once and reuse it. I'm using MongoJS to interface w/ the MongoDb API.
When I take something like the following:
mongo.myt.update(
{'_id': req._id},
{
$addToSet: {
"aggregate.clientIds": req.myt.clientIds
},
$inc: {"aggregate.seenCount": 1},
$set: {
"headers": req.myt.headers,
"ip": req.myt.ip
},
$setOnInsert: {
'_id': req.myt._id,
'derived': req.myt.derived
}
},
{upsert: true},
function (err, savedId) {
if (err || !savedId) console.log("failed to save :" + req.myt + " because of " + err);
else console.log("successfully saved :" + req.myt);
});
And wrap it with a simple function like:
function mongoInsert(req) {
//same query as above
}
Then call it using:
mongoInsert(req);
I don't see any impact to speed when profiling. Should I be adding a callback to the wrapper function, is that needed? I was expecting it would have some performance impact and need to be done differently.
So a few questions.
Does the approach above calling mongoInsert() get called synchronously and block until the async mongo update is done?
If it does become sync, I would expect a performance impact which I didnt see. So is the approach I took ok to use?
And if not what would be the correct way to do this?
mongoInsert() is still asynchronous because it's calling an asynchronous function (mongo.myt.update()). Even if you don't add a callback function, it won't magically become synchronous.
The way you wrote it now is "fire-and-forget": somewhere in your code you call mongoInsert(), your code will continue to run while the update is taking place, and since you don't pass a callback the calling code cannot get informed about the result of the update (right now, you're just logging the result to console).
It's a design decision whether or not this is acceptable.
You could make the callback optional for situations where you do want to get informed about the update result:
function mongoInsert(req, callback) {
callback = callback || function() {}; // dummy callback when none is provided
mongo.myt.update(..., function(err, savedId) {
...log status here...
callback(err, savedId);
});
}
It sounds like a pretty typical use case for libraries like Q or async, but I could not really figure out what's the best way to do it.
I want to import a CSV file with 150 lines (using node-csv), and create a mongo document for each line.
However, the stream parsing seems to finish faster than the 'db inserts', and so I run into
the problem that the callback gets called too early.
// importtest.mocha.js
[...]
importer.loadFromCsv (url, function(result) {
result.length.should.be.equal (150); // nope, it's always around 41
}
// importer.js
function loadFromCsv (url, callback){
csv().from.stream(url)
.on ('record', function(record, index){
new Row({data: record}).save(function() {
console.log ('saved a row to db');
});
})
.on ('end', function() {
callback (Row.find({})); // E parser finished, but probably not all Row.save()
});
}
So, could please anybody give me a hint how I could fix that with async/promises so that
while the stream parsing / db inserts are async, the final callback will be done only after all inserts have finished?
As you are inserting many records, you should take care for each of them individually. Here is a not tested code snippet you could try and adapt. Practically you create a list of promises, when all of them get resolved the function passed to then(fn) gets fired. Well, as mentioned in the code, you should take care for records who have errors. Be aware, that the function passed to then(fn) gets executed only when ALL promises get resolved (successfuly). To indicate the promise an error for a record, you should use defer.reject() instead of def.resolve(). Than pass also a function for the onErrorFn placeholder. Its kind of SQL transaction like.
Here goes the code containing comments for you:
var q = require('q');
function loadFromCsv (url, callback){
// create an array holding all promises
var csv_promises = [];
csv().from.stream(url)
.on ('record', function(record, index){
// create new defer object, per row
var row_defer = q.defer();
// make sure, this function gets called, only after the row got saved
new Row({data: record}).save(function() {
console.log ('saved a row to db');
// resolves the promise, per row
row_defer.resolve(record);
// todo: take care for an error, per row
});
csv_promises.push(row_defer.promise); // add promise to promise list, per row
})
.on ('end', function() {
// callback (Row.find({})); // E parser finished, but probably not all Row.save()
// q.all gets resolved and fires passed function as soon as ALL promises in csv_promises array are resolved
// todo: take care for errors
q.all(csv_promises).then(function() {
callback( csv_promises );
} /*, onErrorFn */ );
}
loadFromCsv( "URL", function(rows) {
console.log("Treated rows: ", rows.length);
});
I have a list of tasks that I want to run in parallel using https://github.com/caolan/async.
I want the program to proceed (probably through a callback) after the first of these parallel tasks is complete, not all of them. So I don't think the naive
async.parallel([task1, task2], callback)
works for me.
Alternatively I could spawn two tasks and cancel the incomplete one, but I can't figure out how to do that using async either.
Thanks!
-Charlie
Parallel Race
You can get async to initiate the final callback by returning an error that evaluates as true but isn't actually an error.
I've put together an example that uses -1 as an error code. In the final callback I check the error value and if it's not -1 then it's an actual error. If the error value is -1 then we'll have a valid value in results. At that point, we just need to remove extra elements from results of the other async functions that have not completed yet.
In the below example I've used the request module to pull html pages and the underscore module to filter the results in the final callback.
var request = require('request');
var _ = require('underscore');
exports.parallel = function(req, res) {
async.parallel([
/* Grab Google.jp */
function(callback) {
request("http://google.jp", function(err, response, body) {
if(err) { console.log(err); callback(true); return; }
callback(-1,"google.jp");
});
},
/* Grab Google.com */
function(callback) {
request("http://google.com", function(err, response, body) {
if(err) { console.log(err); callback(true); return; }
callback(-1,"google.com");
});
}
],
/* callback handler */
function(err, results) {
/* Actual error */
if(err && err!=-1) {
console.log(err);
return;
}
/* First data */
if(err===-1) {
/*
* async#parallel returns a list, one element per parallel function.
* Functions that haven't finished yet are in the list as undefined.
* use underscore to easily filter the one result.
*/
var one = _.filter(results, function(x) {
return (x===undefined ? false : true);
})[0];
console.log(results);
console.log(one);
res.send(one);
}
}
);
};
Remaining Function Results
When you setup async#parallel to work like this you won't have access to the results of the other asynchronous functions. If you're only interested in the first one to respond then this isn't a problem. However, you will not be able to cancel the other requests. That's most likely not a problem, but it might be a consideration.
The async.parallel documentation says:
If any of the functions pass an error to its callback, the main callback is immediately called
with the value of the error.
So you could return an error object from all of your parallel functors, and the first one to finish would jump you to the completion callback. Perhaps even your own special error class, so you can tell the difference between an actual error and a "hey I won" error.
Having said that, you would still have your parallel functions running, potentially waiting for callbacks to complete or whatever. Perhaps you could use async.parallelLimit to make sure you're not firing off too many tasks in parallel ?
Having said all that, it's possible you are better served by trying another method from the async library for this task - firing off parallel tasks then having these tasks race each other may not be the best idea.