I run into an issue when trying to use the request method in javascript where I can't save a value. I'll run a block of code like:
let savedData;
request({
url: url,
json: true
}, function (err, resp, body) {
if (err) {
return;
}
savedData = body.data;
});
console.log(savedData);
I know that request doesn't block or something, so I think it's run after the console.log or something like that? I just need to know how I can save the desired data for use later in the method.
Your code is working correctly, you're just neglecting the fact that the callback provided as a second parameter to request() is executed asynchronously.
At the time when your console.log() is executed, the network request may or may not have successfully returned the value yet.
Further Explanation
Take a look at the documentation for the request() function.
It states that the function call takes the following signature,
request(options, callback);
In JavaScript, a callback performs exactly as the name perscribes; it calls back by executing the provided function, after it has done what it first needs to.
This asynchronous behavior is especially prominent in making networking requests, since you wouldn't want your program to freeze and wait for the network request to retrieve or send what you've requested.
Example
function callback() {
console.log('I finished doing my asynchronous stuff!');
console.log('Just calling you back as I promised.');
}
console.log('Running some asynchronous code!');
request({...options}, callback);
console.log('Hi! I'm being called since I'm the next line of code; and the callback will be called when its ready.');
Output
Running some asynchronous code!
Hi! I'm being called since I'm the next line of code; and the callback
will be called when its ready.
I finished doing my asynchronous stuff!
Just calling you back as I promised.
You'll either need to do the rest of the code in the callback of the request function, or use a promise. Anything outside of that callback is going to execute before saveedData ever shows up.
Related
Setup: Because of the time limit on lambdas, I am using a lambda to process some work, then recursively call itself to continue the work, again and again until it is complete. It's just a chron job that I split up to make sure it won't timeout if the data is too large sometimes.
It looks roughly like this:
exports.handler = async (event, context, callback) => {
let response;
let err;
try {
const data = await getSomeAsyncDataToUpdate(); // this won't get previously updated data, so each recursion gets the first X data until there's none left
response = await updateSomeDataAsync(data);
if (!doneConditionIsMet(response)) {
callLambda(); // we do NOT await because the top level lambda would eventually timeout
}
} catch(e) {
err = e;
}
callback(err, response);
};
This is being called from a Cloudwatch Event (chron).
The Problem:
According to the docs, Cloudwatch Events are asynchronous calls and therefore, do not use the callback. https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html
For async handlers, you return a response, error, or promise to the runtime instead of using callback.
Here's the rub: if I omit the callback, the recursive lambda call doesn't happen. I can see that the function gets called, but it does not result in another lambda getting triggered and running.
Why?
I am out of ideas. The only thing I could come up with is that the lambda is exiting before the call actually executes, which would be wild because I assume the actual execution is synchronous. Also, if that were the case, I do not see how calling the callback would keep it alive long enough for that to finish.
If it is working by some fluke, is there a correct way to do this?
Notes on things I've already checked/tried:
the callback does not keep the lambda alive for a long time. It exits immediately as expected
I tried every version I could think of with removing the callback and returning something
As noted in the code snippet, I do NOT want to await the recursive call because that would mean the first lambda has to wait for them all to complete, which defeats the entire purpose of this
So I am new to Node development. My first personal project requires creating a local txt file of the html contents of a page for further manipulation. I don't know why everything else in the program is executed before the request call is made. For example in the below snippet, "Two" will always come before "One". I know it is probably something simple I am missing but I am hoping to learn something from it.
var request = require('request');
var fs = require('fs');
request('https://www.google.com', function (response, body) {
console.log("One")
fs.writeFile("ToParse.txt", body)
});
console.log("Two")
"Two" will always come before "One"
Because Node.js is asynchronous
What it does mean than previous function won't stop the execution process instead the functions are called irrespective of previous function has stopped execution
If you want a sequence use Callbacks or Promises
This is because of the async nature of nodejs/JavaScript because it's single Threaded, Request module is performing asynchronous operation and aftering completion or failure of the operation, it will call the last function which we call a callback function.
Request will take time to perform it's operation while doing a get method to www.google.com
Node will put that operation in queue and execute the other operation, in your case console.log it consoles the result and later from the queue it will execute the queued operation.
Visit this
https://developer.mozilla.org/en/docs/Web/JavaScript/EventLoop
The callback function first parameter should always be error, if no error occurs it will be null.
request('http://www.google.com', function (error, response, body) {
// your code
})
The function you passed as the second argument to the function request is a callback. It will be called once the request processed. The nature of Node.js is, to execute codes continuously without waiting(blocked) for the callback to be called. When the request completes(fails/succeeds) your callback will be called. That's why "Two" printed before "One"
When run, the code shown below logs "[]". I believe this is because the request function is not complete by the time the console.log method is called.
var urlArray = []
request('http://www.example.com', function (error, response, body) {
urlArray.push("example.com")
}
console.log(urlArray)
How do i put a conditional to make sure the request function is complete. the pseudo code would be:
if request function is complete{
console.log(urlArray)
}
If this isn't the best way, what is?
First, if you run it non-async, the line right after the request line will be executed when the request finishes, but thats not recommendable. Second, running async, ussually you get a callback or a promise depending the library you are using in order to perform the request, even if you use pure JS, you have the option to set callbacks for error and success states, that will be executed only after the request has received a response.
I recommend reading a bit about ajax and promises.
http://www.w3schools.com/ajax/ajax_xmlhttprequest_send.asp
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/then
Having a look at the code you've provided, it seems that is just a concept or pseudocode, that will never run and throw all kind of errors and exceptions.
I can't seem to grasp the concept of a callback. I haven't worked with them before so bear with me. To get my hands wet, I'm trying to login to twitter with zombie.js.
Here is an example:
var Browser = require("zombie");
var browser = new Browser({ debug: true})
browser.visit("https://mobile.twitter.com/session/new", function (callback) {
browser.fill("username", "xxxxx");
browser.fill("password", "xxxxx");
browser.pressButton("Sign in", function (err, success) {
if(err){
console.log(browser.text('.message'));
console.log('There has been a error: ' + err);
}
else{
console.log('Worked!');
}
});
});
At the browser.pressButton part, it will determine if I have been able to successfully login or not, depending on if .message contains the text "Typing on your phone stinks, we know! Double-check your username and password and try again."
However, I don't understand how it determines to fire the callback err. If .message isn't present in the html, then I would like to trigger the success callback to move onto the next function.
The convention Zombie seems to use for callbacks comes from node.js where the first argument is an error object, which should be null on success, and any subsequent arguments are for the success case. If you define a callback, the library you are using (Zombie in this case) will execute your callback function when their async operation is complete. When your callback is invoked it means "OK, an operation has completed and you can now process the result as you see fit". Your code needs to look at that first argument to decide if the operation was a success or failure.
When you accept a callback function as an argument and then perform some (possibly asynchronous) operation, the callback is the way for you to tell the calling library you are done, and again use that first argument to distinguish errors from success.
Part of your confusion is probably coming from the fact that your function signature for the callback to browser.visit is wrong. You need to name that first argument to clearly indicate it's an error like this:
browser.visit("https://mobile.twitter.com/session/new", function (error, browser) {
So in the body of that anonymous callback function, if zombie couldn't load that page, the error argument will have info about the error. If the page did load correctly, error will be null and the browser 2nd argument can be used to further tell zombie to do more stuff on the page. This is how Zombie says "I'm done with the visit operation, time for you to handle the results."
visit doesn't pass a callback argument, the anonymous function you pass as an argument to visit IS THE CALLBACK. You could code it like this to clarify (although nobody does)
browser.visit("https://mobile.twitter.com/session/new", function callback(error, browser) {
So that's a callback when a library needs to tell you it is done. You don't invoke it. The library invokes it and you put code inside it.
On the other hand, when your code does async operations, you need to invoke the callback that you received as a function argument appropriately to tell your caller that you are done and whether it was success for failure. In this case, you don't do any of your own async code, so there's no need for you to invoke any callback functions.
Today is my first foray into nodejs and I am particularly stumped trying to understand the way the following piece of logic flows. The logic is as follows:
request({ uri: db.createDbQuery('identifier:abcd1234') },
function(err, response, body) {
response.should.have.status(200);
var search = JSON.parse(body);
search.response.numFound.should.equal(1);
done();
});
});
At a higher level I do understand is that an http request is being made and the function is being called at some juncture that is taking the response and doing something to it. What I am trying to understand is the proper order of the calls and how does the binding of variables take place in the above given logic. How does the compiler know how to bind the return values from the request to the anonymous function? Basically, I want to gain an understanding on how things work under the hood for this snippet.
Thanks
Your question isnt specific to node.js, this is basically a feature of javascript.
Basically you are calling request() which is defined like function request(obj, callback)
Internally, the http request is being called, and once its completed, it calls callback which is actually a function pointer.
function request(obj, callback){
//http request logic...
var err = request_logic_internal_function();
var response = ...
var body = ...
callback(err, response, body)
}
Your code can actually be restructured as :
var options = { uri: db.createDbQuery('identifier:abcd1234') };
var request_callback = function(err, response, body) {
response.should.have.status(200);
var search = JSON.parse(body);
search.response.numFound.should.equal(1);
done();
};
request(options, request_callback);
What you're basically doing is sending in a function pointer as a variable.
I don't know what library(ies) you're using, and it looks like you may have anonymized them by assigning methods into your code's global scope like request, done, and db.
What I can say is this:
That indentation is horrible and initially misled me on what it was doing, please gg=G (vim syntax) your code so it's properly indented.
request takes two arguments, a configuration object and a callback.
db.createDbQuery must be a blocking method or the anonymous object you're creating won't have the proper value.
request uses that configuration value, makes a non-blocking I/O request of some kind, and later will call the callback function you provide. That means that the code immediately after that request call will execute before the callback you provide will execute.
Some time later the request data will come back, Node.js's event loop will provide the data to the library's registered event handler (which may or may not be your callback directly -- it could do something to it and then call your event handler afterwards, you don't know or really care).
Then the function does some checks that will throw errors if they fail, and finally calls a done function in its scope (defined somewhere else) that will execute and continue the logical stream of execution.