I'm somewhat new to javascript. Recently, I've been working on a node.js project, and I came across the problem of asynchronous callbacks in javascript. My code is below:
So ideally, I would like the request method to be caried out, THEN the callback from the res.on('data') event to be executed, and finally have callback for the .each() iterator to be called. Unfortunately, the asynchronous nature of javascript causes the program to exit the main code block even before the first callback is executed.
var req = https.request(options, function(res) {
// console.log("statusCode: ", res.statusCode);
// console.log("headers: ", res.headers);
console.log('1st call back exec');
res.on('data', function(d) {
console.log('2st call back exec');
var $ = cheerio.load(d);
//traversing method
$('.dddefault').each(function(i){
console.log('3st call back exec');
if(i == 3){
remainingSeats = parseInt($(this).text().trim());
console.log('inner0:'+remainingSeats);
return;
}
});
console.log('inner1:'+remainingSeats);
});
console.log('inner2:'+remainingSeats);
});
req.end();
req.on('error', function(e) {
console.error(e);
});
Can anyone point me in the right direction to get my code working in the way I had intended it to work? Thanks.
It shouldn't abruptly stop unless you get an unhandled error. And, chances are, that error would probably be from cheerio as you may not be giving it the entire document to try to parse.
Note: The req.on('error', ...) callback only receives errors that http or req throw. It won't catch errors from cheerio.
res is a Stream, so it won't necessarily arrive in a single chunk of 'data'. You'll want to collect any number of chunks that may arrive and work with the combined result once the stream has come to an 'end'.
var req = https.request(options, function(res) {
var body = [];
res.setEncoding('utf8');
res.on('data', function (chunk) {
body.push(chunk);
});
res.on('end', function () {
body = body.join('');
var $ = cheerio.load(body);
// etc.
});
You may also consider using request, which handles consuming the stream for you:
request(options, function (err, res, body) {
var $ = cheerio.load(body);
// etc.
});
Related
I have written below code to get the response data chunk in the variable called outBody, but it is showing undefined. Any suggestions please?
var options = {
host: 'jsonplaceholder.typicode.com',
port: 80,
path: '/todos/1'
};
var outBody;
http.get(options, function(res) {
console.log("Got response: " + res.statusCode);
res.on("data", function(chunk) {
outBody = chunk;
});
}).on('error', function(e) {
console.log("Got error: " + e.message);
});
console.log(outBody);
I actually want to use the variable outBody outside the http request. How can I use it?
Your code is showing undefined, because your code is a asyncronous code and you are logging data synchronously.
A simple fix is that put your console.log inside the data event listener like this
res.on("data", function(chunk) {
outBody = chunk;
console.log(outBody);
});
and if you want to use the outBody variable outside the http.get then you either has to use promises or a callback function inside another event listener called end , which fires when the response is completed.
Like this
using CallBack :-
//Your code
res.on("data", chunk => {
outBody += chuck
});
res.on("end", () => {
// this event fires when the response body is completely loaded ...
myFunction(outBody); // pass the respone body to your callback function
}
// remaing code
Wanna learn how to deal with this problem again, then learn about async / await and promises on internet
OR
Using promises / async await :-
const options = {...}
var outBody = "";
const req = new Promise( (resolve, reject) => {
http.get(options, res => {
res.on("data", chunk => { outBody += chunk });
res.on("end", () => resolve(outBody));
res.on("error", console.error);
});
});
req.then( data => {
myFunction(data);
})
.catch(err => console.error(err));
// or u can try async / await
(async function (){
const data = await req;
})();
Note : You had done a mistake i.e., inside the data event listener you overwrite the outBody with the new chunk. It is a mistake because the http class loads the response in chuck which means parts. It means you are losing data , I also fixed it by adding += in place of outBody = chunk, it simply appends the new part the older part which gives you the complete response body...
Still confused about the promises and async / await , then them here
The http request is asynchronous, the reason why console.log is logging undefined is because you log the result of the request before the request is done, only after the request is done outbody gets the value of chunk. In order to access the outbody value you should write the console.log inside the callback block which is called right after the data is returned from the request like this:
http.get(options, function(res) {
console.log("Got response: " + res.statusCode);
res.on("data", function(chunk) {
outBody = chunk;
console.log(outbody)
});
res.on("error",e=>{
console.log("Got error: " + e.message);
})
})
I am trying to make a http request with node module http on an azure function with javascript and for some reason http.request is not receiving data(no error printed its like the requested is blocked). There is anything wrong with azure configuration or the code?Am i missing something really obvious?
Code is running fine on local js file
context.log('JavaScript timer trigger function ran!', timeStamp);
printed as expected but context.log("SUCCESS", JSON.parse(data)); not printed at all.
Also tryed different libraries (request, axios) with no success
module.exports = async function (context, myTimer) {
var timeStamp = new Date().toISOString();
if (myTimer.IsPastDue)
{
context.log('JavaScript is running late!');
}
context.log('JavaScript timer trigger function ran!', timeStamp);
const http = require('http');
http.get('http://myAPIurl', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
context.log("SUCCESS", JSON.parse(data));
});
}).on("error", (err) => {
context.log("ERROR: " + err.message);
});
}
I rewrote your code using Axios. It has support for async/await out of the box, and simplifies a lot of the code to function as your would expect since it makes asynchronous code perform like synchronous code.
I think the main issue you may have been running into it that everything in JavaScript is async. As a result, the Azure Function runtime was exiting before your async function finished since it didn't block like synchronous code would when making the HTTP request. If you want to use callbacks, you need to call context.done() inside your callback function so the Azure Function doesn't exit before the callback is completed. By using async/await, you can guarantee that your code will block on the HTTP request until it receives a response or timeout. In the example below, Axios will return a response object that includes data as an element. I'm extracting data with a deconstruction operation, which allows me to log data.
const axios = require('axios');
const url = 'https://google.com';
module.exports = async function (context, myTimer) {
try {
const { data } = await axios.get(url);
context.log(data);
// do something with the data
return data;
} catch (err) {
context.log(err);
// do something with the error
}
context.done();
}
To Install a Package on an Azure Function
Go to the Azure Portal; select your Azure Function App
Select Platform Features. Then Advanced Tools (Kudu) under Development Tools
Using the file explorer, navigate to site/wwwroot/. You should see a package.json file in there, and several folders each with the name of your functions.
From that directory, run npm install --save axios
To confirm it worked, edit your package.json with the pencil icon. axios should be listed under dependencies json element
I've kept your code callback based.
I removed the async moniker from the definition and added a call to context.done (this signals the functions host when your function has ended) in your resp.end handler
module.exports = function (context, myTimer) {
var timeStamp = new Date().toISOString();
if (myTimer.IsPastDue)
{
context.log('JavaScript is running late!');
}
context.log('JavaScript timer trigger function ran!', timeStamp);
const https = require('https');
https.get('https://raw.githubusercontent.com/LearnWebCode/json example/master/animals-1.json', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
context.log("SUCCESS", JSON.parse(data));
context.done(null, data);
});
}).on("error", (err) => {
context.log("ERROR: " + err.message);
});
}
Another option would be to keep the function as async but you'd need to replace the callbacks with promise based calls. In some scenarios this can be achieved by wrapping them using util.promisify and then calling them with the await keyword
I want to make an HTTP GET request and then trigger another one as soon as the last one completes. Each request is close to identical (small change to path).
I am having trouble understanding why the following is not working (simplified version):
var options = { ... } // host, port etc
var req = request(options, function(res) {
res.on('data', function(chunk) {
data+=chunk;
});
res.on('end', function() {
db.insert(data);
options.path = somewhat_new_path;
req.end(); // doesn't do anything
});
});
req.end();
I know there are many libraries and so on for sequencing asynchronous code, but I would really like to understand why I can't achieve an asynchronous loop in this way.
req.end() finishes the request. Response won't start coming until you finish the request. Hence req.end() inside res.on('end',function(){}) doesn't make any sense.
If you want to make another request with some other path you can do something like:
var http = require('http');
var options = { ... } // host, port etc
makeRequest(options, function() {
db.insert(data);
options.path = somewhat_new_path;
makeRequest(options, function() { //this will make a recursive synchronous call
db.insert(data);
});
});
options.path = another_path;
makeRequest(options, function() { //this will make a asynchronous call
db.insert(data);
});
var makeRequest = function(options, callback) {
http.request(options, function(res) {
res.on('data', function(chunk) {
data+=chunk;
});
res.on('end', callback);
}).end();
}
I have a problem when I call my function JavaScript I can not return my Buffer because it's in other function
index.js:
var client = require('./sender');
//envoyer le paquet au seveur blizzard.com
var received =client.sendAndreceive(bufenc);
console.log('received: '+received);
sender.js:
var Http = require('http');
var Url = require('url');
function sendAndreceive(data) {
console.log('##########################################################################');
// connexion au serveur
var options = {
//proxy
host: "proxya.u-pec.fr",
port: 3128,
//hostname : 'http://m.eu.mobileservice.blizzard.com',
//port : 80,
path : 'http://m.eu.mobileservice.blizzard.com/enrollment/enroll.htm',
method : 'POST',
headers: {
Host: "http://m.eu.mobileservice.blizzard.com"
}
};
var req = Http.request(options, callback );
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
req.write(data);
req.end();
}
exports.sendAndreceive = sendAndreceive;
function callback (res) {
res.on('data', function(chunk) {
buf = new Buffer(chunk);
console.log('data length: ' + buf.length);
return buf; // my problem is her !!!!!!!!!!!!!!!!!!!!!!
});
}
output:
##########################################################################
received: undefined
data length: 45
Using the JavaScript callback pattern
You won't get a result as you returning the result from the callback function, which is nothing and wouldn't work either as your request to the server is asynchronously. Take a look at the example below, and see how it can work:
function callback (res, callback) {
res.on('data', function(chunk) {
buf = new Buffer(chunk);
console.log('data length: ' + buf.length);
// Make sure the argument is indeed a callable function
if (typeof callback === 'function') {
callback(buf);
}
});
}
Now what you do is simply implement a callback, that is executed when the result is finished.
var client = require('./sender');
//envoyer le paquet au seveur blizzard.com
client.sendAndreceive(bufenc, function(received) {
console.log('Received', received);
});
This JavaScript pattern is called the callback pattern, and can be very useful in these kinds of situations.
Using a promise
Another more sexy way of doing stuff like this, is through promises. I don't want to fully explain what a promise is or does (you can use search, right?), so I'll just show you an example:
// This is part of the node-promise module: https://github.com/kriszyp/node-promise
var deferred = require("promise").defer;
var client = require('./sender');
function callback (res) {
// Create a deffered object that will be returned
var deferred = defer();
res.on('data', function(chunk) {
buf = new Buffer(chunk);
console.log('data length: ' + buf.length);
// Now let the listeners know that this promise
// has been completed successfully, and return the buf
deffered.resolve(buf);
// Or if something went wrong, you could do the following:
// deffered.reject('Everything is wrong!');
});
return deferred;
}
//envoyer le paquet au seveur blizzard.com
client.sendAndreceive(bufenc)
.then(
// Function ended successfully
function(received) {
console.log('Received', received);
},
// Function returned an error
function(err) {
console.log('Oh noes!', err);
}
)
;
You are calling sendAndreceive(), and sendAndreceive() is starting a HTTP request that asynchronously calls callback(). So the return value of callback() is a return value given to the invoker inside the HTTP request object, while sendAndreceive() returns nothing.
There is no way to force asynchronous calls behave like one step-by-step-flow. Therefore I support the recommendation of Boy, using a callback.
OK, I see the promise pattern by Boy :-) cool. I will take a deeper look. It seems you can make asynchronous steps at least looking like synchronously written with constructs like .then(). Thx!
How could I make the 'request' module in Node.js load things in a synchronous fashion? The best advice I've seen is to somehow use the callback to get the function to not return itself until it is done. I am trying to use the 'request' function inline in code (things need to be processed based on that data that can't be placed in callbacks).
So how could I use the callback of the 'request' module to keep it from returning itself until it is finished with loading the resource?
What I'm doing is running a loop that downloads two values from an API, and then has to do some math based on those values. While the math could be done in callbacks... the loop would advance without the values it needs to perform the next operation. (So stopping the loop from advancing until the data is ready would solve the issue)
/* loop */ {
/* URL Generation */
request( {url: base + u_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
ucomp = split2[1];
});
request( {url: base + v_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
vcomp = split2[1];
});
/* math which needs to be after functions get variables and before loop advances */
}
The short answer is: don't. (...) You really can't. And that's a good thing
I'd like to set the record straight regarding this:
NodeJS does support Synchronous Requests. It wasn't designed to support them out of the box, but there are a few workarounds if you are keen enough, here is an example:
var request = require('sync-request'),
res1, res2, ucomp, vcomp;
try {
res1 = request('GET', base + u_ext);
res2 = request('GET', base + v_ext);
ucomp = res1.split('\n')[1].split(', ')[1];
vcomp = res2.split('\n')[1].split(', ')[1];
doSomething(ucomp, vcomp);
} catch (e) {}
When you pop the hood open on the 'sync-request' library you can see that this runs a synchronous child process in the background. And as is explained in the sync-request README it should be used very judiciously. This approach locks the main thread, and that is bad for performance.
However, in some cases there is little or no advantage to be gained by writing an asynchronous solution (compared to the certain harm you are doing by writing code that is harder to read).
This is the default assumption held by many of the HTTP request libraries in other languages (Python, Java, C# etc), and that philosophy can also be carried to JavaScript. A language is a tool for solving problems after all, and sometimes you may not want to use callbacks if the benefits outweigh the disadvantages.
For JavaScript purists this may rankle of heresy, but I'm a pragmatist so I can clearly see that the simplicity of using synchronous requests helps if you find yourself in some of the following scenarios:
Test Automation (tests are usually synchronous by nature).
Quick API mash-ups (ie hackathon, proof of concept works etc).
Simple examples to help beginners (before and after).
Be warned that the code above should not be used for production. If you are going to run a proper API then use callbacks, use promises, use async/await, or whatever, but avoid synchronous code unless you want to incur a significant cost for wasted CPU time on your server.
In 2018, you can program the "usual" style using async and await in Node.js.
Below is an example, that wraps request callback in a promise and then uses await to get the resolved value.
const request = require('request');
// wrap a request in an promise
function downloadPage(url) {
return new Promise((resolve, reject) => {
request(url, (error, response, body) => {
if (error) reject(error);
if (response.statusCode != 200) {
reject('Invalid status code <' + response.statusCode + '>');
}
resolve(body);
});
});
}
// now to program the "usual" way
// all you need to do is use async functions and await
// for functions returning promises
async function myBackEndLogic() {
try {
const html = await downloadPage('https://microsoft.com')
console.log('SHOULD WORK:');
console.log(html);
// try downloading an invalid url
await downloadPage('http:// .com')
} catch (error) {
console.error('ERROR:');
console.error(error);
}
}
// run your async function
myBackEndLogic();
Though asynchronous style may be the nature of node.js and generally you should not do this, there are some times you want to do this.
I'm writing a handy script to check an API and want not to mess it up with callbacks.
Javascript cannot execute synchronous requests, but C libraries can.
https://github.com/dhruvbird/http-sync
Aredridels answer is relatively good (upvoted it), but I think it lacks the loop equivalent. This should help you:
Sync code equivalent:
while (condition) {
var data = request(url);
<math here>
}
return result;
Async code for serial execution:
function continueLoop() {
if (!condition) return cb(result);
request(url, function(err, res, body) {
<math here>
continueLoop()
})
}
continueLoop()
You should take a look at library called async
and try to use async.series call for your problem.
See sync-request: https://github.com/ForbesLindesay/sync-request
Example:
var request = require('sync-request');
var res = request('GET', 'http://example.com');
console.log(res.getBody());
You can use retus to make cross-platform synchronous HTTP requests. It's a library based on sync-request but with a few comfort features I added:
const retus = require("retus");
const { body } = retus("https://google.com");
//=> "<!doctype html>..."
That's it!
These days if you want to do things sequentially, you'd do something like:
for (const task of tasks) {
const response = await request(...);
}
This code (above) runs the requests in order, but is not synchronous (meaning it does not block the main thread). If you really need synchronous, then most of these libraries boil down to something like:
const child = require('child_process');
const buffer = child.execSync(`curl https://example.com/foo.json`);
const data = JSON.parse(buffer.toString('utf8'));
You can do something exactly similar with the request library, but this is sync using const https = require('https'); or const http = require('http');, which should come with node.
Here is an example,
const https = require('https');
const http_get1 = {
host : 'www.googleapis.com',
port : '443',
path : '/youtube/v3/search?arg=1',
method : 'GET',
headers : {
'Content-Type' : 'application/json'
}
};
const http_get2 = {
host : 'www.googleapis.com',
port : '443',
path : '/youtube/v3/search?arg=2',
method : 'GET',
headers : {
'Content-Type' : 'application/json'
}
};
let data1 = '';
let data2 = '';
function master() {
if(!data1)
return;
if(!data2)
return;
console.log(data1);
console.log(data2);
}
const req1 = https.request(http_get1, (res) => {
console.log(res.headers);
res.on('data', (chunk) => {
data1 += chunk;
});
res.on('end', () => {
console.log('done');
master();
});
});
const req2 = https.request(http_get2, (res) => {
console.log(res.headers);
res.on('data', (chunk) => {
data2 += chunk;
});
res.on('end', () => {
console.log('done');
master();
});
});
req1.end();
req2.end();
The easiest solution I came up for myself was to use the node's native "child_request" and simply call exec with a simple curl command inside.... all dependencies and asynchronity hassle me way too much for the simple rare cases in which I simply want to "save the http response to a variable in node"
As of this writing all answers are either:
Sync but with a control flow (such as using async library)
Sync, but blocking [which means that all other threads on Node are stopped, which is bad performance wise], such as retus or sync-request.
Out of date, such as http-request.
Check out the non-blocking sync-request, based on deasync.
Or see some of the answers in this thread, such as this, which use deasync directly.
The short answer is: don't. If you want code that reads linearly, use a library like seq. But just don't expect synchronous. You really can't. And that's a good thing.
There's little or nothing that can't be put in a callback. If they depend on common variables, create a closure to contain them. What's the actual task at hand?
You'd want to have a counter, and only call the callback when the data is there:
var waiting = 2;
request( {url: base + u_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
ucomp = split2[1];
if(--waiting == 0) callback();
});
request( {url: base + v_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
vcomp = split2[1];
if(--waiting == 0) callback();
});
function callback() {
// do math here.
}
Update 2018: node.js supports async/await keywords in recent editions, and with libraries that represent asynchronous processes as promises, you can await them. You get linear, sequential flow through your program, and other work can progress while you await. It's pretty well built and worth a try.