Related
I'd like to write a feature like this:
Scenario: new Singleton create
When a new, unmatchable identity is received
Then a new tin record should be created
And a new bronze record should be created
And a new gold record should be created
which would tie to steps like this:
defineSupportCode(function ({ Before, Given, Then, When }) {
var expect = require('chai').expect;
var chanceGenerator = require('./helpers/chanceGenerator')
var request = require('./helpers/requestGenerator')
let identMap;
// reset identMap before each scenario
Before(function () {
identMap = [];
});
// should generate a valid identity
// persist it in a local variable so it can be tested in later steps
// and persist to the db via public endpoint
When('a new, unmatchable identity is received', function (callback) {
identMap.push(chanceGenerator.identity());
request.pubPostIdentity(identMap[identMap.length-1], callback);
});
// use the local variable to retrieve Tin that was persisted
// validate the tin persisted all the props that it should have
Then('a new tin record should be created', function (callback) {
request.pubGetIdentity(identMap[identMap.length-1], callback);
// var self = this;
// request.pubGetIdentity(identMap[identMap.length-1], callback, () => {
// console.log('never gets here...');
// self.callback();
// callback();
// });
// request.pubGetIdentity(identMap[identMap.length-1], (callback) => {
// console.log('never gets here...');
// self.callback();
// callback();
// });
});
The issue that I'm having is that I can't do anything in the Then callback. That is where I'd like to be able to verify the response has the right data.
Here are relevant excerpts from the helper files:
var pubPostIdentity = function (ident, callback) {
console.log('pubIdentity');
var options = {
method: 'POST',
url: 'http://cucumber.utu.ai:4020/identity/' + ident.platform + '/' + ident.platformId,
headers: {
'X-Consumer-Custom-Id': ident.botId + '_' + ident.botId
},
body: JSON.stringify(ident)
};
console.log('ident: ', ident);
request(options, (err, response, body) => {
if (err) {
console.log('pubPostIdentity: ', err);
callback(err);
}
console.log('pubPostIdentity: ', response.statusCode);
callback();
});
}
// accept an identity and retrieve from staging via identity public endpoint
var pubGetIdentity = function (ident, callback) {
console.log('pubGetIdentity');
var options = {
method: 'GET',
url: 'http://cucumber.utu.ai:4020/identity/' + ident.platform + '/' + ident.platformId,
headers: {
'X-Consumer-Custom-Id': ident.botId + '_' + ident.botId
}
};
request(options, (err, response) => {
if (err) {
console.log('pubGetIdentity: ', err);
callback(err);
}
console.log('pubGetIdentity: ', response.body);
callback();
});
}
Something that we are considering as an option is to re-write the feature to fit a different step definition structure. If we re-wrote the feature like this:
Scenario: new Singleton create
When a new, unmatchable 'TIN_RECORD' is received
Then the Identity Record should be created successfully
When the Identity Record is retreived for 'tin'
Then a new 'tin' should be created
When the Identity Record is retreived for 'bronze'
Then a new 'bronze' should be created
When the Identity Record is retreived for 'gold'
Then a new 'gold' should be created
I believe it bypasses the instep callback issue we are wrestling with, but I really hate the breakdown of the feature. It makes the feature less readable and comprehensible to the business.
So... my question, the summary feature presented first, is it written wrong? Am I trying to get step definitions to do something that they shouldn't? Or is my lack of Js skills shining bright, and this should be very doable, I'm just screwing up the callbacks?
Firstly, I'd say your rewritten feature is wrong. You should never go back in the progression Given, When, Then. You are going back from the Then to the When, which is wrong.
Given is used for setting up preconditions. When is used for the actual test. Then is used for the assertions. Each scenario should be a single test, so should have very few When clauses. If you want, you can use Scenario Outlines to mix several very similar tests together.
In this case, is recommend to take it back to first principles and see if that works. Then build up slowly to get out working.
I suspect in this case that the problem is in some exception being thrown that isn't handled. You could try rewriting it to use promises instead, which will then be rejected on error. That gives better error reporting.
im very very new to Node.js, javascript in general, and also functional programming (which node is if im not mistaken?)
Im currently on stage of doing learnyounode tutorials.
I know i can find all the solutions and work it out just fine, but im a little curious why wouldnt my code work...
If anyone is familiar with the learnyounode im stuck at "Juggling async".
The code that i wrote:
var http = require("http");
var addriee = [process.argv[2], process.argv[3], process.argv[4]];
function getStuffFromNet(address, callback) {
http.get(address, function getShitDone(response) {
var dataToCallback = "";
response.on("error", function(data) {
callback(data, null);
});
response.on("data", function(data) {
dataToCallback+=data;
});
response.on("end", function(data) {
callback(null, dataToCallback);
});
});
};
function printToConsole(data) {
console.log(data);
}
printToConsole(getStuffFromNet(addriee[0]));
My goal was to reuse function that would get "stuff from net", the error i get is:
learnyounode run http-get3.js
undefined
/home/ubuntu/workspace/learnyounode/http-get3.js:17
callback(null, dataToCallback);
^
TypeError: undefined is not a function
at IncomingMessage.<anonymous> (/home/ubuntu/workspace/learnyounode/http-get3.js:17:7)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:944:16
at process._tickCallback (node.js:442:13)
Why is the last callback null and not data ?
Also it might be handier to not initialize
var dataToCallback = "";
to
var dataToCallback;
because else you can't use data
typeof dataToCallback !== 'undefined'
Not sure about 's atm.
Also try to comment you're code a lot more. Especially when you're learning it.
Example of some debugging level I have (noob or not I quickly find errors this way)
/**
* Divest the desired amount
*/
socket.on("divest", function (amount) {
error.debug(classname + "Divest is called [" + amount + "]");
invest.divest(hash, amount, function (err, callback) {
if (!err) {
error.debug(uid, name + " />divesting [CBACK]" + callback);
} else {
error.debug(uid, name + " />divesting [ERROR]" + err);
}
socket.emit("done", true);
});
});
Hope I helped.
To explain your situation, the data was read to the end and "callback" is invoked,
but the "callback" was not defined at the last line of your script.
If you wonder why the data.on("error" .....) wasn't triggered, It will only be triggered by data error of the http.get(), it means you are "ABLE TO READ DATA" from the URLs, so the http.get() will trigger data.on("data" ....) and data.on("end" .....) only.
I consider myself a very experienced node.js developer.
Yet I still wonder if there is a better way to write the following code so I don't get the pyramid of doom... Now I went easy on you, I have some code that my pyramid gets as high as 20 floors, no kidding; and that's WITH using async.js !!!
The problem is really that I have many dependencies on previews variables so everything must be nested.
The guy that wrote the book "Async Javascript, build more responsive Apps with less code" explains that he would put the functions at the root scope, which sure, would get rid of the pyramid, but now you would have a whole bunch of high scope variables (possibly even global, depending at the scope you declare them at) and this pollution can result in some pretty nasty bugs (this could cause var conflicts with other scripts if set at global space (sure you could use self invoking functions, more yachhh... or even worse, since we are dealing with async, variable overrides...). In fact, the beauty of closure is pretty mush out the door.
What he recommend is doing something like:
function checkPassword(username, passwordGuess, callback) {
var passwordHash;
var queryStr = 'SELECT * FROM user WHERE username = ?';
db.query(selectUser, username, queryCallback);
function queryCallback(err, result) {
if (err) throw err;
passwordHash = result['password_hash'];
hash(passwordGuess, hashCallback);
}
function hashCallback(passwordGuessHash) {
callback(passwordHash === passwordGuessHash);
}
}
again, not a clean approach IMHO.
So, if you look at my code (again, this is just a snippet, I get much bigger nests in other places) you will often see my code getting further and further apart from the left; and that's with using things like waterfall and async forEach...
here is a small example:
ms.async.eachSeries(arrWords, function (key, asyncCallback) {
pg.connect(pgconn.dbserver('galaxy'), function (err, pgClient, pgCB) {
statement = "SELECT * FROM localization_strings WHERE local_id = 10 AND string_key = '" + key[0] + "'";
pgClient.query(statement, function (err, result) {
if (pgconn.handleError(err, pgCB, pgClient)) return;
// if key doesn't exist go ahead and insert it
if (result.rows.length == 0) {
statement = "SELECT nextval('last_resource_bundle_string_id')";
pgClient.query(statement, function (err, result) {
if (pgconn.handleError(err, pgCB, pgClient)) return;
var insertIdOffset = parseInt(result.rows[0].nextval);
statement = "INSERT INTO localization_strings (resource_bundle_string_id, string_key, string_revision, string_text,modified_date,local_id, bundle_id) VALUES ";
statement += " (" + insertIdOffset + ",'" + key[0] + "'," + 0 + ",'" + englishDictionary[key[0]] + "'," + 0 + ",10,20)";
ms.log(statement);
pgClient.query(statement, function (err, result) {
if (pgconn.handleError(err, pgCB, pgClient)) return;
pgCB();
asyncCallback();
});
});
}
pgCB();
asyncCallback();
});
});
});
On my deep scripts I counted over 25 closing parenthesis, CRAZY, and all while remembering where to call my last callBack so async continues to next iteration...
Is there a solution to this problem? Or it is just the natrure of the beast?
As Mithon said in his answer, promises can make this code much clearer and help to reduce duplication. Let's say that you create two wrapper functions that return promises, corresponding to the two database operations you're performing, connectToDb and queryDb. Then your code can be written as something like:
ms.async.eachSeries(arrWords, function (key, asyncCallback) {
var stepState = {};
connectToDb('galaxy').then(function(connection) {
// Store the connection objects in stepState
stepState.pgClient = connection.pgClient;
stepState.pgCB = connection.pgCB;
// Send our first query across the connection
var statement = "SELECT * FROM localization_strings WHERE local_id = 10 AND string_key = '" + key[0] + "'";
return queryDb(stepState.pgClient, statement);
}).then(function (result) {
// If the result is empty, we need to send another 2-query sequence
if (result.rows.length == 0) {
var statement = "SELECT nextval('last_resource_bundle_string_id')";
return queryDb(stepState.pgClient, statement).then(function(result) {
var insertIdOffset = parseInt(result.rows[0].nextval);
var statement = "INSERT INTO localization_strings (resource_bundle_string_id, string_key, string_revision, string_text,modified_date,local_id, bundle_id) VALUES ";
statement += " (" + insertIdOffset + ",'" + key[0] + "'," + 0 + ",'" + englishDictionary[key[0]] + "'," + 0 + ",10,20)";
ms.log(statement);
return queryDb(stepState.pgClient, statement);
});
}
}).then(function (result) {
// Continue to the next step
stepState.pgCB();
asyncCallback();
}).fail(function (error) {
// Handle a database error from any operation in this step...
});
});
It's still complex, but the complexity is more manageable. Adding a new database operation to every "step" no longer requires a new level of indentation. Also notice that all error handling is done in one place, rather than having to add an if (pgconn.handleError(...)) line every time you perform a database operation.
Update: As requested, here's how you might go about defining the two wrapper functions. I'll assume that you're using kriskowal/q as your promise library:
function connectToDb(dbName) {
var deferred = Q.defer();
pg.connect(pgconn.dbserver(dbName), function (err, pgClient, pgCB) {
if (err) {
deferred.reject(err)
} else {
deferred.resolve({pgClient: pgClient, pgCB: pgCB})
}
});
return deferred.promise;
}
You can use this pattern to create a wrapper around any function that takes a single-use callback.
The queryDb is even more straightforward because its callback gives you either a single error value or a single result value, which means that you can use q's built-in makeNodeResolver utility method to resolve or reject the deferred:
function queryDb(pgClient, statement) {
var deferred = Q.defer();
pgClient.query(statement, deferred.makeNodeResolver());
return deferred.promise;
}
For more information on promises, check out my book: Async JavaScript, published by PragProg.
The problem to this sort of thing is promises. If you haven't heard of them, I suggest reading up on kriskowal's q.
Now, I don't know if the db.query returns a promise or not. If it doesn't you might be able to find a db-wrapper that does or a different db library. If that is not an option, you may "promisify" the db-library you're using. See Howto use promises with Node, and especially the section "Wrapping a function that takes a Node-style callback".
Best of luck! :)
The simplest way to combat the async pyramid of hell is to segregate your async callbacks into smaller functions that you can place outside your main loop. Chances are you can at least break some of your callbacks into more maintainable functions that can be used elsewhere in your code base, but the question you're asking is a bit vague and can be solved in a large number of ways.
Also, you should consider what Stuart mentioned in his answer and try to combine some of your queries together. I'm more concerned that you have 20+ nested calls which would indicate something seriously erroneous in your callback structure so I'd look at your code first before anything else.
Consider rewriting your code to have less back-and-forth with the database. The rule of thumb I use to estimate an app's performance under heavy load is that every async call will add two seconds to the response (one for the request, and one for the reply).
For example, is there maybe a way you could offload this logic to the database? Or a way to "SELECT nextval('last_resource_bundle_string_id')" at the same time as you "SELECT * FROM localization_strings WHERE local_id = 10 AND string_key = '" + key[0] + "'" (perhaps a stored procedure)?
I break each level of the pyramid of doom into a function and chain them one to the other. I think it is a lot easier to follow. In the example above i'd do it as follows.
ms.async.eachSeries(arrWords, function (key, asyncCallback) {
var pgCB;
var pgClient;
var connect = function () {
pg.connect(pgconn.dbserver('galaxy'), function (err, _pgClient, _pgCB) {
pgClient = _pgClient;
pgCB = _pgCB;
findKey();
});
};
var findKey = function () {
statement = "SELECT * FROM localization_strings WHERE local_id = 10 AND string_key = '" + key[0] + "'";
pgClient.query(statement, function (err, result) {
if (pgconn.handleError(err, pgCB, pgClient))
return;
// if key doesn't exist go ahead and insert it
if (result.rows.length == 0) {
getId();
return;
}
pgCB();
asyncCallback();
});
};
var getId = function () {
statement = "SELECT nextval('last_resource_bundle_string_id')";
pgClient.query(statement, function (err, result) {
if (pgconn.handleError(err, pgCB, pgClient))
return;
insertKey();
});
};
var insertKey = function () {
var insertIdOffset = parseInt(result.rows[0].nextval);
statement = "INSERT INTO localization_strings (resource_bundle_string_id, string_key, string_revision, string_text,modified_date,local_id, bundle_id) VALUES ";
statement += " (" + insertIdOffset + ",'" + key[0] + "'," + 0 + ",'" + englishDictionary[key[0]] + "'," + 0 + ",10,20)";
ms.log(statement);
pgClient.query(statement, function (err, result) {
if (pgconn.handleError(err, pgCB, pgClient))
return;
pgCB();
asyncCallback();
});
};
connect();
});
I am literally giving my first steps with node and mongodb and I have recently hit this RangeError wall.
Here's what I am trying to do, I have a file that contains a list of countries that I would like to add to my mongo db. This would be part of my "seed" mechanism to get the app running.
I load the json and then I iterate through the collection of objects and add them one by one to the 'Countries' collection.
However, everytime I run the code, I get a "RangeError: Maximum call stack size exceeded".
I have googled around but none of the suggested solutions seem to apply for me.
My guess is there is something wrong with my insertCountry function...
Anyways, here's my code:
var mongoose = require('mongoose');
var countries = require('./seed/countries.json');
// mongodb
var Country = mongoose.Schema({
name: String,
code: String,
extra: [Extra]
});
var Extra = mongoose.Schema({
exampleField: Boolean,
anotherField: Boolean
});
var mCountry = mongoose.model('Countries', Country);
var mExtra = mongoose.model('Extras', Extra);
// do connection
mongoose.connect('...');
var db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error'));
db.once('open', function callback() {
});
// async function
var insertCountry = function(document, callback) {
db.model('Countries').count({code: document.code}, function (err, count) {
if (count < 1) {
db.collection('Countries').insert(document, function (err, result) {
if (!err) {
console.log('country ' + document.name + ' added');
}
else {
console.log('- [' + document.name + '] ' + err);
}
});
}
callback(null,document);
});
};
// doing countries
var Country = mongoose.model('Countries');
var Extras = mongoose.model('Extras');
for(i = 0; i < countries.length; i++)
{
nCountry = new Country();
nCountry.name = countries[i].name;
nCountry.code = countries[i].code;
nCountry.benefits = new Extras();
nCountry.benefits.exampleField = false;
nCountry.benefits.anotherField = false;
insertCountry(nCountry, function (err, value) {
console.log(value.name + ' added to collection (callback)');
});
}
I have been using some guides I have found to build this so this might not be optimal code. Any best pratices, standards, guides or tutorials you can share are most welcome!
Your callback is in the wrong place. It is not waiting for the insert operation to complete before you return from it's own callback. Altering your code:
var insertCountry = function(document, callback) {
db.model('Countries').count({code: document.code}, function (err, count) {
if (count < 1) {
db.collection('Countries').insert(document, function (err, result) {
if (!err) {
console.log('country ' + document.name + ' added');
}
else {
console.log('- [' + document.name + '] ' + err);
}
callback(null,document);
});
}
});
};
That is part of your problem, but it does not completely solve it. The other part is the loop which also does not wait for the wrapping function to complete before moving on. You want something like asyc.eachSeries in order to wait for inserts to complete before performing the next iteration. This is mostly why you are exceeding the call stack:
async.eachSeries(
countries,
function(current,callback) {
// make your nCountry object
insertCountry(nCountry,function(err,value) {
// do something, then
callback(err);
})
},
function(err) {
// called where done, err contains err where set
console.log( "done" );
}
);
There is really still and issue with the array, which must be reasonably large if you are exceeding the call stack limit. You probably should look at using event streams to process that rather that load everything in memory to the array.
Personally, if you were just trying not to insert duplicates for a field and had MongoDB 2.6 available I would just use the Bulk Operations API with "unordered operations" and allow non fatal failures on the duplicate keys. Coupled with the fact that bulk operations are sent in "batches" and not one at a time, this is much more efficient than checking for the presence on every request:
var Country = mongoose.Schema({
name: String,
code: { type: String, unique: true }, // define a unique index
extra: [Extra]
});
var insertCountries = function(countries,callback) {
var bulk = Country.collection.initializeUnorderedBulkOp();
var counter = 0;
async.eachSeries(
countries,
function(current,callback) {
// same object construction
bulk.insert(nCountry);
counter++;
// only send once every 1000
if ( counter % 1000 == 0 ) {
bulk.execute(function(err,result) {
// err should generally not be set
// but result would contain any duplicate errors
// along with other insert responses
// clear to result and callback
bulk = Country.collection.initializeUnorderedBulkOp();
callback();
});
} else {
callback();
}
},
function(err) {
// send anything still queued
if ( counter % 1000 != 0 )
bulk.execute(function(err,result) {
// same as before but no need to reset
callback(err);
});
}
);
};
mongoose.on("open",function(err,conn) {
insertCountries(countries,function(err) {
console.log("done");
});
});
Keeping in mind that unlike the methods implemented directly on the mongoose models, the native driver methods require that a connection is actually established before they can be called. Mongoose "queues" these up for you, but otherwise you need something to be sure the connection is actually open. The example of the "open" event is used here.
Take a look at event streams as well. If you are constructing an array large enough to cause a problem by missing callback execution then you probably should not be loading it all in memory from whatever your source is. Stream processing that source combined with an approach as shown above should provide efficient loading.
I'm using WebSockets to connect to a remote host, and whenever I populate realData and pass it to grapher(), the JavaScript console keeps telling me realDatais undefined. I tried checking the type of the data in the array, but it seems to be fine. I've called grapher() before using an array with random data, and the call went through without any problems. With the data from the WebSocket, however, the call will always give me "error: realData is not defined". I'm not sure why this is happening. Here is the code I used:
current.html:
var command = "Hi Scott"
getData();
function getData()
{
console.log("getData is called");
if("WebSocket" in window)
{
var dataCollector = new WebSocket("ws://console.sb2.orbit-lab.org:6100",'binary');
dataCollector.binaryType = "arraybuffer";
console.log(dataCollector.readyState);
dataCollector.onopen = function()
{
//alert("The WebSocket is now open!");
console.log("Ready state in onopen is: " + dataCollector.readyState);
dataCollector.send(command);
console.log(command + " sent");
}
dataCollector.onmessage = function(evt)
{
console.log("onmessage is being called");
var realData = new Uint8Array(evt.data);
console.log(realData);
grapher(realData); //everything up to this point works perfectly.
}
dataCollector.onclose = function()
{
alert("Connection to Server has been closed");
}
return (dataCollector);
}
else
{
alert("Your browser does not support WebSockets!");
}
}
graphing.js:
function grapher(realData)
{
console.log("grapher is called");
setInterval('myGraph(realData);',1000); //This is where the error is. I always get "realData is not defined".
}
function myGraph(realData)
{
/*
for(var i = 0; i < SAarray.length; i++) // Loop which will load the channel data from the SA objects into the data array for graphing.
{
var data[i] = SAarray[i];
}
*/
console.log("myGraph is called");
var bar = new RGraph.Bar('channelStatus', realData);
bar.Set('labels', ['1','2','3','4','5','6','7','8']);
bar.Set('gutter.left', 50);
bar.Set('gutter.bottom', 40);
bar.Set('ymax',100);
bar.Set('ymin',0);
bar.Set('scale.decimals',1);
bar.Set('title','Channel Status');
bar.Set('title.yaxis','Status (1 is on, 0 is off)');
bar.Set('title.xaxis','Channel Number');
bar.Set('title.xaxis.pos',.1);
bar.Set('background.color','white');
bar.Set('colors', ['Gradient(#a33:red)']);
bar.Set('colors', ['red']);
bar.Set('key',['Occupied','Unoccupied']);
bar.getShapeByX(2).Set('colors',barColor(data[0]));
bar.Draw();
}
Because strings (as code) passed to setInterval execute in the global scope, therefore the realData parameter isn't available. There's rarely a good reason to pass a string to setInterval. Instead, use:
setInterval(function () {
myGraph(realData);
}, 1000);
Reference:
https://developer.mozilla.org/en-US/docs/Web/API/window.setInterval
Try it without it needing to evaluate a string:
setInterval(function() {
myGraph(realData);
},1000);
Any time you are using setTimeout or setInterval, you should opt for passing an actual function instead of a string.