We Keep Coding

Awaiting code until filter is finished - Nextjs/Javascript

I am looking on how to make my code after my filter function await the results of my filter function to complete before running. However I am not sure how to do this.
My filter function takes in another function (useLocalCompare) which causes the execution of my filter function to be a little longer than normal, which then leads to my next piece of code (that depends on the results of my filter function) executing before my filter function is complete.....which leads to undefined.
Is there anything similar to a callback I can use to force my subsequent piece of code to wait till the filter is finished?
Relevant code is written below.
if (flatarrayofvalues !== null && genre !== null) {
const filtteredarray = await flatarrayofvalues.filter(
(placeholder) => {
if (useLocalCompare(genre, placeholder.name) == true) {
console.log("HURAY!!!!", placeholder.id, placeholder.name);
placeholder.name == placeholder.name;
}
}
);
console.log("MY FILTERED ARRAY IS", filtteredarray);
console.log("The ID FOR MY MY FILERED ARRAY IS two ID", filtteredarray[0]?.id);
return filtteredarray[0].id;
}
}
}
For those curious, useLocalCompare basically checks to see if the genre parameter pulled down from the URL is the same as a name parameter from the array I am filtering. Reason I have this is due to people having different case sensitivity when putting in URLS. EX: it will pull down "HORrOR" and match it to the object name in the array I am filtering called "horror". I then extract the ID from that object.

you have to return the conditional from filter as it is "explicit return"
const filtteredarray = await flatarrayofvalues.filter(
(placeholder) => {
if (useLocalCompare(genre, placeholder.name) == true) {
console.log("HURAY!!!!", placeholder.id, placeholder.name);
return placeholder.name == placeholder.name; // here
// why not just return true ?? instead of above line
}return false
}
);
Also I'm not sure this makes sense
placeholder.name == placeholder.name; you mean just return true; ?

node.js - how to pass additional parameters to async.map

Background Information
I'm trying to write some javascript / node.js code that will do the following:
Query a redis database and get back a bunch of keys from my hash. This is what the command looks like on the redis-cli:
127.0.0.1:6379> hkeys widgets:1231231234
1) "13:00:00_17:00:00_mtwrf"
2) "00:00:00_00:00:00"
3) "08:00:00_12:00:00_mtwrf"
or
127.0.0.1:6379> hkeys widgets:2222222222
1) "00:00:00_00:00:00"
Each widget will have at least one key... called the default key. Default keys look like this: "00:00:00_00:00:00"
For each HKEYS query, if there's more than one result returned, I need to test each key returned (except the default) against a set of match criteria. Whichever key matches, is what is used to do a subsequent HGET against redis. So for example, in the first result set shown above... If key 3 was a match I would run the following command:
127.0.0.1:6379> hget widgets:2222222222 08:00:00_12:00:00_mtwrf
"some value"
127.0.0.1:6379>
If neither key 1 or 3 matches, then I query for the default key.
Code
I've recently discovered the async module. I'm currently using it in my code to loop through the results from HKEYS.
Please see the code below:
async.map Code
router.get('/:widgetnum', function(req, res, next) {
//validate widgetnum format
var widgetnum = req.params.widgetnum;
if ( !valid_widget(widgetnum) ) {
var retval = {"res":false, "msg":"malformed widgetnum"};
res.send(JSON.stringify(retval));
} else {
var keys = {};
redis.hkeys("widgets:" + widgetnum, function(err, data){
if (err) {
res.send(JSON.stringify(false));
}
if (data) {
current = getCurrentUTC(); // needed by iterator for match criteria. Using a global variable for now.
async.map(data, hash_iterator, function (err, iterator_results) {
if (iterator_results) {
console.log("it are: " + iterator_results);
}
res.send(iterator_results);
}); //end async map
}
}); //end redis.hkeys
}
});
Question / Problem
This is working in the sense that for each key returned by HKEYS, I'm able to run the "hash_iterator" function.
However, once inside the iterator function, after evaluation / running my match criteria on each result, I don't have all the information I need to run the secondary HGET query.
The code:
async.map(data, hash_iterator, function (err, iterator_results)
passes just the values:
"13:00:00_17:00:00_mtwrf"
"00:00:00_00:00:00"
"08:00:00_12:00:00_mtwrf"
But I need the hash name (in this case "widgets:" ) and the widgetnum to make the HGET call.
I guess I can use global variables... but I wanted to make sure that my approach in general is correct here.
Any input would be appreciated.
Here's the hash_iterator logic:
var hash_iterator = function (redis_key, doneCallBack) {
var results = {};
console.log("processing..." + redis_key);
//skip if default rule....
if (redis_key.indexOf('00:00:00_00:00:00') == -1){
//need to write logic here to do HGET and save
//results somewhere... in case no other keys match
} else {
// run some logic to test match criteria.
if (matchfound) {
console.log("bingo. found match in " + redis_key);
redis.hget(hashname + widgetnum + redis_key, function (e, results){
if (results) {
return doneCallBack(null, results);
} else {
return doneCallBack(null, null);
}
});
}else {
console.log ("no match");
}
}
return results;
}

You can solve it in functional style with map. Just generate array of pairs and then pass it to async.map, for example
from [1,2,3] you can generate [['key', '1'], ['key', '2'], ['key', 3]]
So here the code, that generating pairs:
redis.hkeys("widgets:" + widgetnum, function(err, data){
// some code goes here
var pairs = data.map(function(ts) {
return [`widgets:${widgetnum}`, ts];
});
// async.map call goes here
}); //end redis.hkey
Inside async.map iterator callback you can access key with iterator_results[0]
But in your case the variable widgetnums is not global variable. It's just variable from upper scope and you can use it inside nested functions if you define it in propper scope. It's very useful trick that called lexical closure.

Do I ever need to synchronize node.js code like in Java?

I have only recently started developing for node.js, so forgive me if this is a stupid question - I come from Javaland, where objects still live happily sequentially and synchronous. ;)
I have a key generator object that issues keys for database inserts using a variant of the high-low algorithm. Here's my code:
function KeyGenerator() {
var nextKey;
var upperBound;
this.generateKey = function(table, done) {
if (nextKey > upperBound) {
require("../sync/key-series-request").requestKeys(function(err,nextKey,upperBound) {
if (err) { return done(err); }
this.nextKey = nextKey;
this.upperBound = upperBound;
done(nextKey++);
});
} else {
done(nextKey++);
}
}
}
Obviously, when I ask it for a key, I must ensure that it never, ever issues the same key twice. In Java, if I wanted to enable concurrent access, I would make make this synchronized.
In node.js, is there any similar concept, or is it unnecessary? I intend to ask the generator for a bunch of keys for a bulk insert using async.parallel. My expectation is that since node is single-threaded, I need not worry about the same key ever being issued more than once, can someone please confirm this is correct?
Obtaining a new series involves an asynchronous database operation, so if I do 20 simultaneous key requests, but the series has only two keys left, won't I end up with 18 requests for a new series? What can I do to avoid that?
UPDATE
This is the code for requestKeys:
exports.requestKeys = function (done) {
var db = require("../storage/db");
db.query("select next_key, upper_bound from key_generation where type='issue'", function(err,results) {
if (err) { done(err); } else {
if (results.length === 0) {
// Somehow we lost the "issue" row - this should never have happened
done (new Error("Could not find 'issue' row in key generation table"));
} else {
var nextKey = results[0].next_key;
var upperBound = results[0].upper_bound;
db.query("update key_generation set next_key=?, upper_bound=? where type='issue'",
[ nextKey + KEY_SERIES_WIDTH, upperBound + KEY_SERIES_WIDTH],
function (err,results) {
if (err) { done(err); } else {
done(null, nextKey, upperBound);
}
});
}
}
});
}
UPDATE 2
I should probably mention that consuming a key requires db access even if a new series doesn't have to be requested, because the consumed key will have to be marked as used in the database. The code doesn't reflect this because I ran into trouble before I got around to implementing that part.
UPDATE 3
I think I got it using event emitting:
function KeyGenerator() {
var nextKey;
var upperBound;
var emitter = new events.EventEmitter();
var requesting = true;
// Initialize the generator with the stored values
db.query("select * from key_generation where type='use'", function(err, results)
if (err) { throw err; }
if (results.length === 0) {
throw new Error("Could not get key generation parameters: Row is missing");
}
nextKey = results[0].next_key;
upperBound = results[0].upper_bound;
console.log("Setting requesting = false, emitting event");
requesting = false;
emitter.emit("KeysAvailable");
});
this.generateKey = function(table, done) {
console.log("generateKey, state is:\n nextKey: " + nextKey + "\n upperBound:" + upperBound + "\n requesting:" + requesting + " ");
if (nextKey > upperBound) {
if (!requesting) {
requesting = true;
console.log("Requesting new series");
require("../sync/key-series-request").requestSeries(function(err,newNextKey,newUpperBound) {
if (err) { return done(err); }
console.log("New series available:\n nextKey: " + newNextKey + "\n upperBound: " + newUpperBound);
nextKey = newNextKey;
upperBound = newUpperBound;
requesting = false;
emitter.emit("KeysAvailable");
done(null,nextKey++);
});
} else {
console.log("Key request is already underway, deferring");
var that = this;
emitter.once("KeysAvailable", function() { console.log("Executing deferred call"); that.generateKey(table,done); });
}
} else {
done(null,nextKey++);
}
}
}
I've peppered it with logging outputs, and it does do what I want it to.

As another answer mentions, you will potentially end up with results different from what you want. Taking things in order:
function KeyGenerator() {
// at first I was thinking you wanted these as 'class' properties
// and thus would want to proceed them with this. rather than as vars
// but I think you want them as 'private' members variables of the
// class instance. That's dandy, you'll just want to do things differently
// down below
var nextKey;
var upperBound;
this.generateKey = function (table, done) {
if (nextKey > upperBound) {
// truncated the require path below for readability.
// more importantly, renamed parameters to function
require("key-series-request").requestKeys(function(err,nKey,uBound) {
if (err) { return done(err); }
// note that thanks to the miracle of closures, you have access to
// the nextKey and upperBound variables from the enclosing scope
// but I needed to rename the parameters or else they would shadow/
// obscure the variables with the same name.
nextKey = nKey;
upperBound = uBound;
done(nextKey++);
});
} else {
done(nextKey++);
}
}
}
Regarding the .requestKeys function, you will need to somehow introduce some kind of synchronization. This isn't actually terrible in one way because with only one thread of execution, you don't need to sweat the challenge of setting your semaphore in a single operation, but it is challenging to deal with the multiple callers because you will want other callers to effectively (but not really) block waiting for the first call to requestKeys() which is going to the DB to return.
I need to think about this part a bit more. I had a basic solution in mind which involved setting a simple semaphore and queuing the callbacks, but when I was typing it up I realized I was actually introducing a more subtle potential synchronization bug when processing the queued callbacks.
UPDATE:
I was just finishing up one approach as you were writing about your EventEmitter approach, which seems reasonable. See this gist which illustrates the approach. I took. Just run it and you'll see the behavior. It has some console logging to see which calls are getting deferred for a new key block or which can be handled immediately. The primary moving part of the solution is (note that the keyManager provides the stubbed out implementation of your require('key-series-request'):
function KeyGenerator(km) {
this.nextKey = undefined;
this.upperBound = undefined;
this.imWorkingOnIt = false;
this.queuedCallbacks = [];
this.keyManager = km;
this.generateKey = function(table, done) {
if (this.imWorkingOnIt){
this.queuedCallbacks.push(done);
console.log('KG deferred call. Pending CBs: '+this.queuedCallbacks.length);
return;
};
var self=this;
if ((typeof(this.nextKey) ==='undefined') || (this.nextKey > this.upperBound) ){
// set a semaphore & add the callback to the queued callback list
this.imWorkingOnIt = true;
this.queuedCallbacks.push(done);
this.keyManager.requestKeys(function(err,nKey,uBound) {
if (err) { return done(err); }
self.nextKey = nKey;
self.upperBound = uBound;
var theCallbackList = self.queuedCallbacks;
self.queuedCallbacks = [];
self.imWorkingOnIt = false;
theCallbackList.forEach(function(f){
// rather than making the final callback directly,
// call KeyGenerator.generateKey() with the original
// callback
setImmediate(function(){self.generateKey(table,f);});
});
});
} else {
console.log('KG immediate call',self.nextKey);
var z= self.nextKey++;
setImmediate(function(){done(z);});
}
}
};

If your Node.js code to calculate the next key didn't need to execute an async operation then you wouldn't run into synchronization issues because there is only one JavaScript thread executing code. Access to the nextKey/upperBound variables will be done in sequence by only one thread (i.e. request 1 will access first, then request 2, then request 3 et cetera.) In the Java-world you will always need synchronization because multiple threads will be executing even if you didn't make a DB call.
However, in your Node.js code since you are making an async call to get the nextKey you could get strange results. There is still only one JavaScript thread executing your code, but it would be possible for request 1 to make the call to the DB, then Node.js might accept request 2 (while request 1 is getting data from the DB) and this second request will also make a request to the DB to get keys. Let's say that request 2 gets data from the DB quicker than request 1 and update nextKey/upperBound variables with values 100/150. Once request 1 gets its data (say values 50/100) then it will update nextKey/upperBound. This scenario wouldn't result in duplicate keys, but you might see gaps in your keys (for example, not all keys 100 to 150 will be used because request 1 eventually reset the values to 50/100)
This makes me think that you will need a way to sync access, but I am not exactly sure what will be the best way to achieve this.

Inserting multiple values in Mysql using Nodejs and notifying user with a response

I am making a new webservice where i send a curl command with JSON and the JSON contains a array as
[{tempid:1,email:abc#123,address:asd},{tempid:2,email:abc#12345,address:asd45},{tempid:3,email:abc#1234,address:asd4}]
Now when i pass and insert the array in a mysql table tempid is just to show a mapping to the user to the contact id generated in the database as tempid:1 is now inserted and in database it has cid 120 , like this for tempid2 and 3 ,
But when i am trying to show the client the updated values it shows only one value , last last change not the whole updated Array. Its becuase of the async nature of the connection.querry function , so i need help in this , here is my webservice
contactadd webservice -->
for(var i=0;i<=request.body.contact.length-1;i++)
{
if(request.body.contact[i].tempid)
{ var ardata=new Array();
var o=request.body.contact[i];
pair=Object.keys(o).map(function(a){ return [a, o[a]] });
AM.addcontact(pair,request.session.user,request.body.contact.length,function(e,o){
if(!o)
{
response.send('something went wrong'+e);
}
else
{
//response.send(o);
}
});
}
}
Here is the update function in the database.js script -->
//ContactSync-addcontact module for database
exports.addcontact=function (arr,email,addnum,callback)
{
var counter=0;
var uid;
var data=new Array();
var showinsert=new Array();
var values=new Array();
var datatable=new Array();
var inserting=new Array();
var tempid=0;
connection.query('SELECT UID FROM user where email1="'+email.email+'"',function(err,rows,fields){
if(err)
{
throw err;
}
else
{
if(rows[0]!=undefined)
{
uid=rows[0]['UID'];
}
else
{
uid="no id in database";
}
}
});// get the UID of the inserting user
// make array of user provided data
for(var j=0;j<=arr.length-1;j++)
{
if(arr[j][0]!='tempid')
{
data.push(arr[j][0]);
}
else
{
tempid=arr[j][1];
}
}
connection.query('SELECT column_name FROM information_schema.columns where table_schema="webservice" AND table_name="usercontacts"',function(err,rows,fields){
if(err)
{
throw err;
}
else
{
for(var i=0;i<=rows.length-1;i++)
{
datatable.push(rows[i]['column_name']);
}
}
for(var k=0;k<=datatable.length-1;k++)
{
if(inArray(data[k],datatable))
{
inserting.push(data[k]);
}
}
if(inserting.length>0)
{
for(var z=0;z<=arr.length-1;z++)
{
if(inArray(arr[z][0],inserting))
{
values.push('"'+arr[z][1]+'"');
}
}
// Insert tempid values and data in the usercontacts table with inserting and values
connection.query('INSERT INTO usercontacts (cid,uid,'+inserting+') VALUES("","'+uid+'",'+values+')',function(err,rows,fields){
if(err)
{
throw err;
}
else
{
connection.query('SELECT * FROM usercontacts WHERE uid="'+uid+'" ORDER BY cid DESC LIMIT 0,'+addnum+'',function(err,rows,fields){
if(err)
{
throw err;
}
else
{ showinsert.push('temp-id: '+tempid+',cid:'+rows[0].cid+',uid:'+uid);
//for(var i=0;i<=inserting.length-1;i++)
forEach(inserting,function(row,index)
{
showinsert.push(inserting[index]+":"+values[index]);
counter+=1;
});
callback(null,showinsert);
}
});
}
});
//insertion finished
}
else
{
callback("Please Provide atleast one field to enter with tempid");
}
});
}
I just need to insert all the callback in a array which has been inserted and show user that array ,please help , completely stuck and then only i am trying StackOverflow.
Thank you for reading till the end BTW

I'm not sure what the specific problem is, but there are some problems with the code you've shared that will bite you sooner or later. One of these may be causing your problem.
Race conditions
If the query SELECT UID FROM user where email1= for any reason takes longer than the SELECT column_name FROM information_schema.columns just below it then you won't have a value for the variable uuid and your logic will fail. Remember that these calls are non-blocking, so you can't rely on one finishing before the other one unless they're nested or use another flow-control mechanism (As #Tracker points out, async is popular).
Catching edge cases
In the line below you're assigning a string value to the uid variable and then continuing to use that variable even though it now contains an error message.
uid="no id in database";
Doing that means that your code later on will have trouble reacting. Instead use a different variable, leave the uid = undefined or immediately return the callback with an error, e.g.
return callback(new Error("user not found"));
Reporting errors
Don't throw errors in Node unless you want to kill the process, e.g. dependency problems during server startup. It doesn't work like Java, async errors are not caught by try/catch and will kill your process or leave you in a state that's hard to reason about. Instead make the error object your first parameter to the callback and return it immediately, like this:
if ( err ) return callback(err);
Then in your client code you can always check the first parameter to see if there was a problem.
Security problem
As #Tracker mentioned, don't ever do the this:
connection.query('SELECT UID FROM user where email1="'+email.email+'"', ...
If the value of the variable is passed through as "; drop table user; or similar then you're in trouble. Instead you can use node-mysql's build in escaping like this:
connection.query('SELECT UID FROM user where email1=?', [email.email], ...
Whitelist
You're querying information_schema.columns in order to detect which fields are valid then inserting them into usercontacts. This is a clever trick, but increases a 3 query process to 4 queries, and raises questions if there are any fields that a user shouldn't be inserting data into. Using a column whitelist may seem like more code to maintain, but would actually be simpler than all the code required to match columns dynamically.
Arrays
I don't see the source for the function inArray() but it looks like it does the same as Array.prototype.indexOf() so it may be better to use that. e.g.
if ( datatable.indexOf(data[k]) > -1 ) inserting.push(data[k]);
Every line of custom code you can delete is a line of code you don't have to maintain.

Calling asynchronous function in callback

I'm having some trouble understanding asynchronous functions. I've read the chapter in Mixu's Node Book but I still can't wrap my head around it.
Basically I want to request a ressource (using the node package cheerio), parse it for valid URLs and add every match to my redis set setname.
The problem is that in the end it's only adding the first match to the redis set.
function parse(url, setname)
{
request(url, function (error, response, body)
{
if (!error && response.statusCode == 200)
{
$ = cheerio.load(body)
// For every 'a' tag in the body
$('a').each(function()
{
// Add blog URL to redis if not already there.
var blog = $(this).attr('href')
console.log("test [all]: " + blog);
// filter valid URLs
var regex = /http:\/\/[^www]*.example.com\//
var result = blog.match(regex);
if(result != null)
{
console.log("test [filtered]: " + result[0]);
redis.sismember(setname, result[0], function(err, reply)
{
if(!reply)
{
redis.sadd(setname, result[0])
console.log("Added " + result[0])
}
redis.quit()
})
}
})
}
})
}
I'd be very grateful for pointers on how I'd have to restructure this so the redis.sadd method is working with the correct result.
The output of the current implementation looks like:
test [all]: http://test1.example.com/
test [filtered]: http://test1.example.com/
...
Added http://test2.example.com/
So it's adding the test1.example.com but not printing the "added" line, and it's not adding the test2.example.com but it's printing the "added" line for it.
Thank you!

The first issue is caused by redis.sismember() being asynchronous: when its callback is called, you have already overwritten the result variable so it will point to the last value it had, and not the value at the moment at which you called redis.sismember().
One way to solve that is to create a new scoped variable by wrapping the asynchronous function in a closure:
(function(result) {
redis.sismember(setname, result[0], function(err, reply) {
...
});
})(result);
Another option is to create a partial function that's used as callback:
redis.sismember(setname, result[0], function(result, err, reply) {
...
}.bind(this, result));
The second issue is, I think, caused by redis.quit() being called, which closes the Redis connection after the first sadd(). You're not checking err, but if you do it might tell you more.