d3.queue never triggering .await function - javascript

I've run into a problem using d3-queue. This is my code:
var dataQueue = d3.queue();
dataQueue.defer(collectData,ISBNs,locations)
.await(processData);
Where collectData is a function that does several API calls (a large number of them to the Google Books API).
Now the problem is that the processData function is never called. I know for a fact that the collectData function runs properly, since I put a print statement just before the return statement, along with several other print statements along the way.

You are not passing your data correctly between the deferred task collectData and the final processData. The documentation has it as follows (emphasis mine):
# queue.defer(task[, arguments…]) <>
Adds the specified asynchronous task callback to the queue, with any optional arguments. The task is a function that will be called when the task should start. It is passed the specified optional arguments and an additional callback as the last argument; the callback must be invoked by the task when it finishes. The task must invoke the callback with two arguments: the error, if any, and the result of the task.
Thus, to pass the result of the deferred task to the function processData, your function collectData() has to be something like this:
function collectData(ISBNs, locations, callback) {
var error = null; // The error, if any
var data = { }; // The actual data to pass on
// ...collect your data...
// Pass the collected data (and the error) by invoking the callback.
callback(error, data);
}

Related

How does the "response" and "data" get passed to then() [duplicate]

I understand the essence of callback functions in that the function is executed again after being passed as the parameter to another function. However, I'm confused as to where the variables inside the callback function come from as shown in the following node.js example:
router.get('/', function(req, res){
res.render('index', {});
});
How do the variables req and res get populated? An example explaining how I can just call res.render(...) without declaring res myself would be greatly appreciated.
They come from the same place they come from when a normal non callback function is invoked, at invocation time.
If you have this function,
function add (a, b) {
return a + b
}
You're fine with knowing that a and b come from when you invoke add,
add(1,2)
and it's the same principle with callbacks, don't let your brain get all twisted just because it's getting invoked later.
At some point the function you pass to router.get is going to be invoked, and when it does, it will receive req and res.
Let's pretend the definition for router.get looks like this
router.get = function(endpoint, cb){
//do something
var request = {}
var response = {}
cb(request, response) // invocation time
}
In the case of your example, it's just up to node to pass your function request and response whenever .get is invoked.
The whole point of the callback is that the invoked function calls it back.
In the case of router.get, it will insert the route (path, method, callback) in a lookup table; when a request comes in, Express will construct the response object, match the request's path and method against all the entries in the lookup table, take the callback from the matching entry and invoke callback(request, response) (passing the detected request and created response).
They get populated by whatever code is calling the callback. In your example, this is something inside the Express framework, though Express uses the node http library under the hood and adds additional functionality to the request and response objects provided by it.
But in code you write you can create a callback function signature that takes whatever params you want.

How this variable gets value [duplicate]

I understand the essence of callback functions in that the function is executed again after being passed as the parameter to another function. However, I'm confused as to where the variables inside the callback function come from as shown in the following node.js example:
router.get('/', function(req, res){
res.render('index', {});
});
How do the variables req and res get populated? An example explaining how I can just call res.render(...) without declaring res myself would be greatly appreciated.
They come from the same place they come from when a normal non callback function is invoked, at invocation time.
If you have this function,
function add (a, b) {
return a + b
}
You're fine with knowing that a and b come from when you invoke add,
add(1,2)
and it's the same principle with callbacks, don't let your brain get all twisted just because it's getting invoked later.
At some point the function you pass to router.get is going to be invoked, and when it does, it will receive req and res.
Let's pretend the definition for router.get looks like this
router.get = function(endpoint, cb){
//do something
var request = {}
var response = {}
cb(request, response) // invocation time
}
In the case of your example, it's just up to node to pass your function request and response whenever .get is invoked.
The whole point of the callback is that the invoked function calls it back.
In the case of router.get, it will insert the route (path, method, callback) in a lookup table; when a request comes in, Express will construct the response object, match the request's path and method against all the entries in the lookup table, take the callback from the matching entry and invoke callback(request, response) (passing the detected request and created response).
They get populated by whatever code is calling the callback. In your example, this is something inside the Express framework, though Express uses the node http library under the hood and adds additional functionality to the request and response objects provided by it.
But in code you write you can create a callback function signature that takes whatever params you want.

How do functions like fs.writeFile() callback arguements work?

I'm learning JavaScript at the moment, specifically Node JS. And during a basic tutorial I notice that some functions have callbacks with predefined arguments. For example
fs.writeFile(
path.join(__dirname, 'test', 'hello.txt'),
"Hello World",
err => {
if (err) throw err;
console.log('File Created...')})
I want to know where was "err" defined as a ErrNoException type.
Thanks!
You can write similar functions yourself. Callback is simply a function (F) which is passed to another function(A) as a parameter. Function (A) executes some tasks and then executes Function F from parameters.
For example:
function getData(url, callback) {
fetch(url)
.then(res => callback(null, res))
.catch(err => callback(err, null))
}
In the sample function, what happens is, after making a fetch request, we wait until a response or error is generated. If a response is there, we simply pass that to callback param 2 and call first param to null and vice versa.
Now, while calling the function,
getData('http://stackoverflow.com', (err, res) => {
if(err)
return console.log(err)
console.log(res)
})
I want to know where was "err" defined as a ErrNoException type.
In this type of interface, you pass a callback function and fs.writeFile() determines what arguments it will pass that callback when it calls it. It is up to you to declare your callback function to have appropriately named arguments that match what the caller will be passing it. So, YOU must declare the callback with an appropriately named argument. Since Javascript is not a typed language, you don't specific what the type of that argument is. The type of the argument is determined by the code that calls the callback. In this case that is the fs.writeFile() implementation. To use the callback appropriately, you have to read the documentation for fs.writeFile() so that you know how to use the argument to the callback.
So, in this specific case, fs.writeFile() requires a callback function and that callback function will be passed one argument and that argument will either by null (if there was no error) or will be an Error object that explains what the error was). You must learn this by either reading the code for fs.writeFile() or reading some documentation for it.
It's important to realize that when you define your callback function, all you are doing is providing a name for the arguments you expect the callback to be called with. You can then use that name in your code to reference those arguments. The actual data that is passed to that callback when it is called will be supplied by the fs.writeFile() function when it calls the callback.
It's unclear exactly what you mean by "defined as a ErrNoException type". Nodejs has an internal function (not available for external use) called errNoException() and you can see the source here. It uses that function internally for creating an Error object with some common properties such as an error code and a stack trace.

Saving a value through JavaScript's request method

I run into an issue when trying to use the request method in javascript where I can't save a value. I'll run a block of code like:
let savedData;
request({
url: url,
json: true
}, function (err, resp, body) {
if (err) {
return;
}
savedData = body.data;
});
console.log(savedData);
I know that request doesn't block or something, so I think it's run after the console.log or something like that? I just need to know how I can save the desired data for use later in the method.
Your code is working correctly, you're just neglecting the fact that the callback provided as a second parameter to request() is executed asynchronously.
At the time when your console.log() is executed, the network request may or may not have successfully returned the value yet.
Further Explanation
Take a look at the documentation for the request() function.
It states that the function call takes the following signature,
request(options, callback);
In JavaScript, a callback performs exactly as the name perscribes; it calls back by executing the provided function, after it has done what it first needs to.
This asynchronous behavior is especially prominent in making networking requests, since you wouldn't want your program to freeze and wait for the network request to retrieve or send what you've requested.
Example
function callback() {
console.log('I finished doing my asynchronous stuff!');
console.log('Just calling you back as I promised.');
}
console.log('Running some asynchronous code!');
request({...options}, callback);
console.log('Hi! I'm being called since I'm the next line of code; and the callback will be called when its ready.');
Output
Running some asynchronous code!
Hi! I'm being called since I'm the next line of code; and the callback
will be called when its ready.
I finished doing my asynchronous stuff!
Just calling you back as I promised.
You'll either need to do the rest of the code in the callback of the request function, or use a promise. Anything outside of that callback is going to execute before saveedData ever shows up.

Async each function

I'm trying to understand async each(coll, iteratee, callback) function to execute a function in parallel for each item of an array. From the async docs I understand that callback will be executed only once (when iteratee function will be executed for each item of the array).
And in the case of an error in the iteratee function, calling callback('some error message') will immediately call the callback function with error message.
Below is an example from the async docs for each function
each(coll, iteratee, callback)
// assuming openFiles is an array of file names
async.each(openFiles, function(file, callback) {
// Perform operation on file here.
console.log('Processing file ' + file);
if( file.length > 32 ) {
console.log('This file name is too long');
callback('File name too long');
} else {
// Do work to process file here
console.log('File processed');
callback();
}
}, function(err) {
// if any of the file processing produced an error, err would equal that error
if( err ) {
// One of the iterations produced an error.
// All processing will now stop.
console.log('A file failed to process');
} else {
console.log('All files have been processed successfully');
}
});
What I'm not able to understand is, what does calling callback() without argument does, it looks very strange to me that we call callback() with no argument when there is no error in the iteratee function. What does calling callback() or callback(null) do in case of no errors.
Can't we just remove those callback() or callback(null), when we actually mean to call the callback only once (when iteratee function is executed for all the elements of the array) rather than for each item of the array.
What does calling callback() or callback(null) do in case of no errors.
Calling callback with no arguments or with null signals to async.each that the iteratee function finished executing on that item (file in the case of the example). When all of the iteratee functions have called their respective callback, or one of them passes an error to it's callback, async.each will call the original callback function passed to it.
To elaborate on that a little, async.js is designed to handle asynchronous functions. The benefit (or issue, depending on how you look at it), of an asynchronous function is that there is no way to tell when it will finish executing. The way to deal with this is to pass the asynchronous function another function, a callback, to execute when it is finished. The asynchronous function could pass any errors it encounters, or any data it retrieves, to the original calling function through the callback function. For example, fs.readFile passes the read file data, and any errors, through the callback function is it passed.
Can't we just remove those callback() or callback(null), when we actually mean to call the callback only once (when iteratee function is executed for all the elements of the array) rather than for each item of the array.
No, because async.js has to assume the iteratee function is asynchorous, and therefore, it has to wait for it to its callback. The callback passed to async.each is only called once.
The confusion may be caused by the variable names. An individual callback function is only meant to be called once. The callback function passed to async.each is not the same callback passed to the iteratee. Each time iteratee is invoked with a value in coll, it is passed a new callback function. That call of iteratee is only supposed to call the passed callback once (async will throw an error otherwise). This allows async to track if a call to the iteratee function has called its callback, and wait for the rest to call their respective callback. Once all of the callback functions are called, async.each knows all of the asynchorous iteratee function calls have finished executing, and that it can call the original callback passed to it.
This is one of the difficult aspects of creating docs. They have to concise enough that a developer can get information from them quickly, and also include enough detail that they can explain the concept, or the function. It's a hard balance to achieve sometimes.
Calling callback with no arguments adds to a counter inside of the .each function. This counter, when full is the thing that actually calls your callback. Without that, it would never know when it completed.

Categories