I am creating a node API using javascript. I have used redis as my key value store.
I created a redis-client in my app and am able to get values for perticular key.
I want to retrieve all keys along with their values.
So Far I have done this :
app.get('/jobs', function (req, res) {
var jobs = [];
client.keys('*', function (err, keys) {
if (err) return console.log(err);
if(keys){
for(var i=0;i<keys.length;i++){
client.get(keys[i], function (error, value) {
if (err) return console.log(err);
var job = {};
job['jobId']=keys[i];
job['data']=value;
jobs.push(job);
});
}
console.log(jobs);
res.json({data:jobs});
}
});
});
but I always get blank array in response.
is there any way to do this in javascript?
Thanks
First of all, the issue in your question is that, inside the for loop, client.get is invoked with an asynchronous callback where the synchronous for loop will not wait for the asynchronous callback and hence the next line res.json({data:jobs}); is getting called immediately after the for loop before the asynchronous callbacks. At the time of the line res.json({data:jobs}); is getting invoked, the array jobs is still empty [] and getting returned with the response.
To mitigate this, you should use any promise modules like async, bluebird, ES6 Promise etc.
Modified code using async module,
app.get('/jobs', function (req, res) {
var jobs = [];
client.keys('*', function (err, keys) {
if (err) return console.log(err);
if(keys){
async.map(keys, function(key, cb) {
client.get(key, function (error, value) {
if (error) return cb(error);
var job = {};
job['jobId']=key;
job['data']=value;
cb(null, job);
});
}, function (error, results) {
if (error) return console.log(error);
console.log(results);
res.json({data:results});
});
}
});
});
But from the Redis documentation, it is observed that usage of
Keys are intended for debugging and special operations, such as
changing your keyspace layout and not advisable to production
environments.
Hence, I would suggest using another module called redisscan as below which uses SCAN instead of KEYS as suggested in the Redis documentation.
Something like,
var redisScan = require('redisscan');
var redis = require('redis').createClient();
redisScan({
redis: redis,
each_callback: function (type, key, subkey, value, cb) {
console.log(type, key, subkey, value);
cb();
},
done_callback: function (err) {
console.log("-=-=-=-=-=--=-=-=-");
redis.quit();
}
});
Combination of 2 requests:
import * as ioredis from 'ioredis';
const redis = new ioredis({
port: redisPort,
host: redisServer,
password: '',
db: 0
});
const keys = await redis.collection.keys('*');
const values = await redis.collection.mget(keys);
Order will be the same for both arrays.
This will get all keys but with no values:
const redis = require('redis');
const client = redis.createClient();
client.keys('*', (err, keys) => {
// ...
});
Now you need to get the values for those keys in a usual way. For example:
Promise.all(keys.map(key => client.getAsync(key))).then(values => {
// ...
});
or with async module or in any way you like.
You should never do this. First off, it is not recommended to use KEYS * in production. Second, this does not scale (cluster).
You can organise your cached entries into SETs and query for the items within the SET, then retrieve the references keys. This also makes invalidation easier.
Have a look at some data storage best practices.
https://redis.io/topics/data-types-intro
how to get all keys and values in redis in javascript?
You may find something useful in this link
https://github.com/NodeRedis/node_redis/tree/master/examples
Related
I have a Redis server with 2 keys. I'm trying to fetch both the keys and combine them into 1 response. I'm using node-redis.
var response = {};
RedisClient.keys('*', function (err, keys) { // gets all keys
async.each( // asynchronously maps over them and parses the values
keys,
function (err, key) {
RedisClient.get(key, function (err, value) {
if (err) throw err;
responsekey] = JSON.parse(value);
});
},
function () {
res.send(response); // finally sends response object
}
);
});
This gets all keys, maps over them asynchronously, and finally parses the values and sends them. The problem is that the redis docs and every resource I've read has said not to use keys. Is there another way to do this in node using SCAN?
I have a function that receives an array of usernames. For each one I need to get the connections IDs and concatenate into another array. But I think the way I did have some problems of race condition that may colide and concatenate less results that it should...
const getConnections = async function (usernames) {
let connections = [];
await Promise.all(usernames.map(async (username) => {
try {
let connsFound = await dynamo.getConnectionsByUsername(username);
if (connsFound && connsFound.length > 0)
connections = connections.concat(connsFound);
} catch (error) {
console.error('ERROR GET CONNECTIONS', error)
}
}));
return connections;
};
the result of the connections its an array, so the result I would like to merge to the var connections...but .concat does not merge, it creates a new array, so I need to do a 'var = var.concat(newArray)'
I am affraid this is not safe and in some operation it will colide and overwrite some results...
Is there a better way to do it?
cheers
JavaScript is single threaded, so there is always at most one function that runs and accesses connections. You shouldn't have any problems.
However, there is no reason to do it that way. Since you are already using async/await, you can just create an array of arrays and flatten that:
const getConnections = async function (usernames) {
const connections = await Promise.all(usernames.map(async (username) => {
try {
return await dynamo.getConnectionsByUsername(username);
} catch (error) {
console.error('ERROR GET CONNECTIONS', error);
return [];
}
}));
return connections.flat();
};
.flat is relatively new, but it should be easy to write a custom helper function to achieve the same.
I'm new to NodeJS and Jade/PUG so I know this can be a really simple question for many of you but for me I can't understeand any of the answers a get from the internet because the term 'Promise' and how it works is a little "confusing" for me.
I am querying a postgre database to get several values from a table (really simple query). If I do this without using a promise everything works fine, console prints the result and everyone is happy, but when I try to store this result into a variable and pass it as a parameter to a Jade template things change.
I have read that, in order to do that, I need to use promises because it is more likely that when the variable is being accessed, the value might not be resolved yet and that's what Promises are for. So here I have my code:
hero.js:
getHeroes: function()
{
//Initialize array
var elem = [];
console.log('Querying heroes');
return new Promise((resolve, reject) =>
{
pg.connect(conString, function (err, client, done)
{
if (err)
{
return console.error('error fetching client from pool', err)
}
//Execute SELECT query
client.query('SELECT name from heroe;', function (err, rows, result)
{
//Iterate over results
for (var i = 0; i < rows.rowCount; i++)
{
//PUSH result into arrays
elem.push(rows.rows[i].name);
}
done()
if (err)
{
return console.error('error happened during query', err)
}
resolve(elem)
})
});
})
}
And this the part of my server.js where I call that function:
app.get('/jade', (request, response) => {
var heroList = [];
heroList = hero.getHeroes();
console.log(heroList);
response.render('test_jade', {param1: 'test'});
})
that console.log shows up "Promise { pending }" and I don't know how to "listen to the resolved event and retrieve the value from it once it has finished".
Would appreciate any advice/solution or even a good Node.js manual where all this mess is explained for total newbies like me.
Thanks in advance!
It's not how you use promise.
Try this,
app.get('/jade', (request, response) => {
var heroList = [];
hero.getHeroes().then(data=>{
heroList = data;
console.log(heroList);
response.render('test_jade', {param1: 'test'});
}).catch(e=>{
//handle error case here when your promise fails
console.log(e)
})
})
You should also catch in case your promise fails.
So, we are trying to rewrite our express server into Rx. It is currently using async for all stream operations. The code looks like the following:
var async = require('async');
function getCountAndChannels(name, cb){
var tasks = [
function(cb) {
//does a mongoDB search and returns count
},
function(cb) {
//does a findOne mongoDB search and returns
}
];
async.parallel(tasks, cb);
}
router.get('data', function(req, res) { //router is the express router
var recorders = req.query.recorders.split(',');
async.map(recorders, function(name, cb) {
getCountAndChannels(name, cb);
}, function(err, countsAndChannels) {
if(err) throw err;
// here countsAndChannels is an array with first element the count
// and second element the document.
// do other async stuff based on the results
res.status(200).json('send some calculations');
});
The thing here I have to do is loop over the array of recorders and for each one calculate the two mongoDB searches. I have tried using Rx.Observable.merge which doesn't return the results in an array but in 2 different calls of the callback. So, then I tried Rx.Observable.zip which I believe is what I'm looking for.
The problem is I don't know how to loop over the recorders and send the result when all operations are finished. Because a simple forEach loop will throw a Cannot set headers after they are sent error.
This is what I have so far:
recorders.forEach(recorder => {
Rx.Observable.zip([
search1,
search2
]).subscribe(
(countsAndChannels) => {
// do stuff
res.send('the results');
},
err => res.status(500).json(err),
() => res.send('OK')
);
});
Haven't used Rx before, so any help is appreciated.
It might be easier to convert your list of recorders to an Observable stream, then flatMap over each recorder (ie perform your async processing), then call toArray to store all the results into an array:
var recorder$ = Rx.Observable.from(recorders);
var countsAndChannels$ = recorder$
.flatMap(performAsyncTask);
// allResults$ will emit once all of the async work is complete
var allResults$= countsAndChannels$.toArray();
allResults$.subscribe(results => {
// Send response to client;
});
I am able to insert and retrieve data from an neDB database in nodejs. But I cannot pass the data outside of the function that retrieves it.
I have read through the neDB documentation and I searched for and tried different combinations of callbacks and returns (see code below) without finding a solution.
I'm new to javascript so I do not know if I am misunderstanding how to use variables in general or if this issue is related to using neDB specifically or both.
Could someone please explain why "x" in my code does not ever contain the docs JSON results from the database? How can I make it work?
var fs = require('fs'),
Datastore = require('nedb')
, db = new Datastore({ filename: 'datastore', autoload: true });
//generate data to add to datafile
var document = { Shift: "Late"
, StartTime: "4:00PM"
, EndTime: "12:00AM"
};
// add the generated data to datafile
db.insert(document, function (err, newDoc) {
});
//test to ensure that this search returns data
db.find({ }, function (err, docs) {
console.log(JSON.stringify(docs)); // logs all of the data in docs
});
//attempt to get a variable "x" that has all
//of the data from the datafile
var x = function(err, callback){
db.find({ }, function (err, docs) {
callback(docs);
});
};
console.log(x); //logs "[Function]"
var x = db.find({ }, function (err, docs) {
return docs;
});
console.log(x); //logs "undefined"
var x = db.find({ }, function (err, docs) {
});
console.log(x); //logs "undefined"*
Callbacks are generally asynchronous in JavaScript meaning you can not use assignment operator, consequently you do not return anything from the callback function.
When you call an async function execution of you programme carries on, passing the 'var x = whatever' statement. The assignment to a variable, the result of whatever callback is received, you need to perform from within the callback itself... what you need is something in the lines of ...
var x = null;
db.find({ }, function (err, docs) {
x = docs;
do_something_when_you_get_your_result();
});
function do_something_when_you_get_your_result() {
console.log(x); // x have docs now
}
EDIT
Here is a nice blog post about asynchronous programming. And there is a lot more resources on this topic that you can pick up.
This is a popular library to help with async flow-control for node.
P.S.
Hope this helps. Please, by all means ask if you need something clarified :)
I ran into the same problem. In the end I used a combination between async-await and a promise with resolve to solve it.
In your example the following would work:
var x = new Promise((resolve,reject) {
db.find({ }, function (err, docs) {
resolve(docs);
});
});
console.log(x);
I had to learn a bit about async functions to get it right. For those who are looking for specific help about getting a return value from nedb, here's a snippet of what worked for me. I was using it in electron.
function findUser(searchParams,callBackFn)
{
db.find({}, function (err, docs))
{
//executes the callback
callBackFn(docs)
};
}
usage
findUser('John Doe',/*this is the callback -> */function(users){
for(i = 0; i < users.length; i++){
//the data will be here now
//eg users.phone will display the user's phone number
}})