I'm trying to execute several async methods of redis with the following code
var redis = require("redis");
var client = redis.createClient();
var async = require("asyncjs");
async.list([
client.hincrby("traffic:" + siteId, 'x', 1),
client.hincrby("traffic:" + siteId, 'y', 1),
client.hincrby("traffic:" + siteId, 'z', 1)
]).call().end(function(err, result)
{
console.log(err); // returns error [TypeError: Object true has no method 'apply']
console.log(result); // undefined
if(err) return false;
return result;
});
All the methods execute successfully
but i get the error [TypeError: Object true has no method 'apply']
the method is executed and returns true, and its probably interpreting that as true, but i don't see why it has to use the method apply on it?
I can get the result of the increment by adding a function(err, result) to client.hincrby as last element.. but how do i get all the results in the result variable in the end function?
I suppose the asyncjs module you use is the one documented at:
https://github.com/fjakobs/async.js
In your code:
list() is a generator. It allows the array to be iterated by asyncjs. The array is an array of values.
call() is a mapper which calls each item. The items has therefore to be callable (i.e. they have to be callbacks).
end() is a termination end point, called when the iteration is over. As a parameter, you only get the last value of the sequence (not the whole sequence).
You got the "[TypeError: Object true has no method 'apply']" error because the list you have built is not a list of callbacks. It is a list of values.
Here is some code which should do what you want:
var redis = require("redis");
var client = redis.createClient();
var async = require("asyncjs");
function main() {
var siteId = 1;
async
.list(['x','y','z'])
.map( function (item,next) {
client.hincrby('traffic:' + siteId, item, 1, function (err,res) {
next(err,res)
})
})
.toArray( function(err,res) {
console.log(err);
console.log(res);
});
}
main()
Please note here we use map() instead of call(), and toArray() instead of end().
Related
I'm trying to query a database, then make an array of objects from the results, turn them into a JSON object.
I'm not used to Javascript's asynchronous nature and I'm really confused about how to implement something that needs to work synchronously. I know I probably need to use callbacks, but after looking at many tutorials I'm only left more confused.
This is the code without callbacks:
var foreignTable = (tablename,idArr)=>{
var dataArray = [];
//call a query for each of the ids
var objectToAdd;
for(var id of idArr){
objectToAdd = queryForeignTable(tablename,id);
dataArray.push(objectToAdd);
}
return dataArray;
connection.end();
};
var queryForeignTable = (tablename,id)=>{
connection.query("SELECT * FROM "+tablename+" WHERE id="+id, function (error, results, fields) {
if(error)throw error;
var objectToAddToArray={};
//Go through each field in a result and make the object
for(packet of fields){
var label = packet.name;
objectToAddToArray[label] = results[0][label];
}
return objectToAddToArray;
});
};
var arrayOfDrivers = foreignTable("driver",[1,2]);
outputJson["drive"]=arrayOfDrivers;
console.log(outputJson); // { drive: [ undefined, undefined ] }
I attempted foreignTable(tablename, idArr, callback) with the callback calling queryForeignTable with no luck.
Can someone explain how I can get this piece of code working with callbacks?
A callback function is a function passed into another function as an argument, which is then invoked inside the outer function to complete some kind of routine or action.
from MDN: Callback function
Callbacks are a way of telling a function what to do next, as in "after you're done, run this function".
For example:
first = function (callback) {
console.log("First!");
callback();
}
second = function () {
console.log("Second!");
}
first(second);
Will produce:
First!
Second!
You can also use anonymous function to product the same result:
first(function() {
console.log("Second!")
});
Regarding the specific example from your question, you are correct that you need to use callbacks a bit differently. Rather than using a return statement in each of your two functions, you'll need to use a callback. connection.query is asynchronously coming back with your results as results, but you can't return them to your queryForeignTable function. Instead, give queryForeignTable a callback function to run. The same idea goes for your foreignTable function.
I'm not connected to your database, obviously, so I stubbed a DB connection and simplified what you're trying to do, but it should look something like this:
// Stubbed DB connection
var connection = {};
connection.query = (id, cb) => {
var results = [
{
id: id,
name: 'Name of ' + id,
},
];
cb(null, results);
};
var foreignTable = (ids, cb) => {
var data = [];
for (var i = 0; i < ids.length; i++) {
queryForeignTable(ids[i], (error, obj) => {
data.push(obj);
if (i == ids.length - 1) {
cb(null, data);
}
});
}
};
var queryForeignTable = (id, cb) => {
connection.query(id, (error, results) => {
if (error) {
cb(error, null);
}
cb(null, results[0]);
});
};
foreignTable([1, 2], (error, data) => {
if (error) {
console.error(error);
}
console.log(data);
});
That produces:
[ { id: 1, name: 'Name of 1' }, { id: 2, name: 'Name of 2' } ]
In essence, when you have an urge to return some value(s) from a function in an asynchronous way, give the function a callback parameter, then invoke that callback with your return values.
You can run the code here: https://repl.it/K0YI/3
When you have an asynchronous call, like connection.query(statement, callback), then whatever you want to do with the results of that call needs to be done within the callback.
Bear in mind that async functions don't return the final value that you generally want (usually they return undefined). Instead of using a return value, you pass a callback as a way of saying, "when you're finished, carry on and do this with the results", aka. continuation-passing style.
One of the challenges in your code is that you're issuing separate queries for each ID, and then aggregating the results into an array of responses. This means you'll need to check when all the queries have completed, and only then proceed to display the final results.
Here's your example, re-written, commented, and simplified. Hopefully this helps to explain how the control flow works.
// Read row IDs 1 and 2 from the "driver" table.
readTable("driver", [1, 2], displayData);
// Print the results.
function displayData (arrayOfDrivers) {
console.log(arrayOfDrivers);
}
// Read all rows matching IDs in `idArray` from `tableName`,
// put results into an array, and finally invoke `callback`.
function readTable (tablename, idArray, callback) {
var resultsArray = [];
// Queue up all the async queries.
for (var id of idArray){
queryTable(table, id, handleResponse);
}
// A query finished, so handle the result.
function handleResponse (error, results, fields) {
if (error) {
throw error;
}
// Add the query result to array of results.
resultsArray.push(results[0]);
// Check if all queries are done.
if (resultsArray.length === idArray.length) {
// Invoke the callback with the resultsArray.
// The callback is in fact the `displayData` function.
callback(resultsArray);
}
}
}
// Execute a query, using the `cb` callback to handle the response.
function queryForeignTable (tablename, id, cb) {
var query = "SELECT * FROM " + tablename + " WHERE id=" + id;
connection.query(query, cb);
}
Note that the handleResponse function is defined within the scope of the readTable function, so it can access the variables in readTables scope, such as resultsArray and callback.
Hope that helps.
I am creating a node API using javascript. I have used redis as my key value store.
I created a redis-client in my app and am able to get values for perticular key.
I want to retrieve all keys along with their values.
So Far I have done this :
app.get('/jobs', function (req, res) {
var jobs = [];
client.keys('*', function (err, keys) {
if (err) return console.log(err);
if(keys){
for(var i=0;i<keys.length;i++){
client.get(keys[i], function (error, value) {
if (err) return console.log(err);
var job = {};
job['jobId']=keys[i];
job['data']=value;
jobs.push(job);
});
}
console.log(jobs);
res.json({data:jobs});
}
});
});
but I always get blank array in response.
is there any way to do this in javascript?
Thanks
First of all, the issue in your question is that, inside the for loop, client.get is invoked with an asynchronous callback where the synchronous for loop will not wait for the asynchronous callback and hence the next line res.json({data:jobs}); is getting called immediately after the for loop before the asynchronous callbacks. At the time of the line res.json({data:jobs}); is getting invoked, the array jobs is still empty [] and getting returned with the response.
To mitigate this, you should use any promise modules like async, bluebird, ES6 Promise etc.
Modified code using async module,
app.get('/jobs', function (req, res) {
var jobs = [];
client.keys('*', function (err, keys) {
if (err) return console.log(err);
if(keys){
async.map(keys, function(key, cb) {
client.get(key, function (error, value) {
if (error) return cb(error);
var job = {};
job['jobId']=key;
job['data']=value;
cb(null, job);
});
}, function (error, results) {
if (error) return console.log(error);
console.log(results);
res.json({data:results});
});
}
});
});
But from the Redis documentation, it is observed that usage of
Keys are intended for debugging and special operations, such as
changing your keyspace layout and not advisable to production
environments.
Hence, I would suggest using another module called redisscan as below which uses SCAN instead of KEYS as suggested in the Redis documentation.
Something like,
var redisScan = require('redisscan');
var redis = require('redis').createClient();
redisScan({
redis: redis,
each_callback: function (type, key, subkey, value, cb) {
console.log(type, key, subkey, value);
cb();
},
done_callback: function (err) {
console.log("-=-=-=-=-=--=-=-=-");
redis.quit();
}
});
Combination of 2 requests:
import * as ioredis from 'ioredis';
const redis = new ioredis({
port: redisPort,
host: redisServer,
password: '',
db: 0
});
const keys = await redis.collection.keys('*');
const values = await redis.collection.mget(keys);
Order will be the same for both arrays.
This will get all keys but with no values:
const redis = require('redis');
const client = redis.createClient();
client.keys('*', (err, keys) => {
// ...
});
Now you need to get the values for those keys in a usual way. For example:
Promise.all(keys.map(key => client.getAsync(key))).then(values => {
// ...
});
or with async module or in any way you like.
You should never do this. First off, it is not recommended to use KEYS * in production. Second, this does not scale (cluster).
You can organise your cached entries into SETs and query for the items within the SET, then retrieve the references keys. This also makes invalidation easier.
Have a look at some data storage best practices.
https://redis.io/topics/data-types-intro
how to get all keys and values in redis in javascript?
You may find something useful in this link
https://github.com/NodeRedis/node_redis/tree/master/examples
So, we are trying to rewrite our express server into Rx. It is currently using async for all stream operations. The code looks like the following:
var async = require('async');
function getCountAndChannels(name, cb){
var tasks = [
function(cb) {
//does a mongoDB search and returns count
},
function(cb) {
//does a findOne mongoDB search and returns
}
];
async.parallel(tasks, cb);
}
router.get('data', function(req, res) { //router is the express router
var recorders = req.query.recorders.split(',');
async.map(recorders, function(name, cb) {
getCountAndChannels(name, cb);
}, function(err, countsAndChannels) {
if(err) throw err;
// here countsAndChannels is an array with first element the count
// and second element the document.
// do other async stuff based on the results
res.status(200).json('send some calculations');
});
The thing here I have to do is loop over the array of recorders and for each one calculate the two mongoDB searches. I have tried using Rx.Observable.merge which doesn't return the results in an array but in 2 different calls of the callback. So, then I tried Rx.Observable.zip which I believe is what I'm looking for.
The problem is I don't know how to loop over the recorders and send the result when all operations are finished. Because a simple forEach loop will throw a Cannot set headers after they are sent error.
This is what I have so far:
recorders.forEach(recorder => {
Rx.Observable.zip([
search1,
search2
]).subscribe(
(countsAndChannels) => {
// do stuff
res.send('the results');
},
err => res.status(500).json(err),
() => res.send('OK')
);
});
Haven't used Rx before, so any help is appreciated.
It might be easier to convert your list of recorders to an Observable stream, then flatMap over each recorder (ie perform your async processing), then call toArray to store all the results into an array:
var recorder$ = Rx.Observable.from(recorders);
var countsAndChannels$ = recorder$
.flatMap(performAsyncTask);
// allResults$ will emit once all of the async work is complete
var allResults$= countsAndChannels$.toArray();
allResults$.subscribe(results => {
// Send response to client;
});
I am using nodejs to query data from Mongodb throught Mongoose.
After get the data, I want do something on that data before responding it to client. But I can not get the return-value. After looking on Google, I have learned Node.js functions is asynchronous javascript function (non I/O blocking). I try this tut (http://www.youtube.com/watch?v=xDW9bK-9pNY) but it is not work. Below is my code. The myObject is valued inside "find()" function and undefined outside "find()" function. So what should I do to get the data? Thanks!
var Person = mongoose.model('Person', PersonSchema);
var Product = mongoose.model('Product', ProductSchema);
var myObject = new Object();
Person.find().exec(function (err, docs) {
for (var i=0;i<docs.length;i++)
{
Product.find({ user: docs[i]._id},function (err, pers) {
myObject[i] = pers;
console.log(myObject[i]); //return the value is ok
});
console.log(myObject[i]); //return undefined value
}
console.log(myObject); //return undefined value
});
console.log(myObject); //return undefined value
app.listen(3000);
console.log('Listening on port 3000');
The reason you're getting undefined values is because the find function is asynchronous, and can finish at any time. In your case, it is finishing after you're using console.log(), so the values are undefined when you're accessing them.
To fix this problem, you can only use the values inside the find function's callback. It would look something like this:
var Person = mongoose.model('Person', PersonSchema);
var Product = mongoose.model('Product', ProductSchema);
var myObject = new Object();
function getData(docs, callback) {
function loop(i) {
Product.find({ user: docs[i]._id}, function (err, pers) {
myObject[i] = pers;
if (i < docs.length) {
loop(i + 1);
} else {
callback();
}
});
};
loop(0);
};
Person.find().exec(function(err, docs) {
getData(docs, function() {
// myObject has been populated at this point
});
});
The data processing has been moved to a loop that waits for the previous iteration to complete. This way, we can determine when the last callback has fired in order to fire the callback in the wrapper function.
Keep in mind that by the time the console.log functions are executed, the query has not yet finished, thus will display "undefined". That's the essence of nodeJS's asynchronicity.
For example,
Person.find().exec(function (err, docs) {
for (var i=0;i<docs.length;i++)
{
Product.find({ user: docs[i]._id},function (err, pers) {
myObject[i] = pers;
console.log(myObject[i]); //return the value is ok
});
console.log(myObject[i]); //return undefined value
}
console.log(myObject); //return undefined value
});
console.log(myObject); // <-- Initially, this value will be undefined. After some miliseconds (Or the duration of the .exec function, myObject will contain the results.
If you want to actually wait until the query is finished so you can use the values, I would recommend moving the app.listen(3000); and console.log('Listening on port 3000'); into the function's final callback.
I'd also recommend you to check out this node module. It will help you build asynchronous / synchronous functions more easily, and allow you to execute a callback when all the asynchronous functions are finished.
I want to pass into Seq([644511,340755]) an response from async function getProjects.
So I tried
...
var ids = pivotal.getProjects(function (err, data) {
var ids = data.project.map(function(x) { return parseInt(x.id); });
console.log("IDS_i: ".red + ids);
});
console.log("IDS_e: ".red + ids);
Seq(ids)
.parEach(function(project_id) {
....
Logs:
IDS_e: undefined
GET /stories 200 34ms
GET /favicon.ico 404 2ms
IDS_i: 644511,340755
I am wondering maybe I should put this into Seq:
Seq()
.seq(function() {
pivotal.getProjects(function (err, data) {
data.project.map(function(x) { return parseInt(x.id); });
});
}
but how to return ids as array in that case?
getProjects is also async. A basic rule: You can't return any values from an async function. You have to do all processing in the callback function. Execution will continue before your arrays have been aggregated. So your seq approach is what you need:
Seq()
.seq(function() {
pivotal.getProjects(this);
})
.flatten()
.seqEach(function(project) {
var projectId = project.id;
myService.someOtherAsyncAction(projectId, this);
});
node-seq will take care of passing the result of the callback to the next seq step by passing this as the callback function to your async function. This is how flow and results are passed to the next step. flatten will make sure each project is available as individual elements on the stack so you can do seqEach in the next step.