I am currently buildings proxy using nodejs, which use following syntax for sending and receiving https request and response. However in my project, the response is a liitle bit larger, so typically, req.on('data', callback) will be called 5~7 times before req.on('end', callback) being called.
Here is the simplified code structure:
var http = require("https");
var options = {
hostname: '<WEB SERVICE>',
port: 80,
path: '<WEB PATH>',
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
};
var response = "";
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (body) {
console.log("data");
response += body;
});
res.on('end', function () {
console.log("end");
response = "";
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
// write data to request body
req.write('<SOMETHING>');
req.end();
Ideally, when multiple request comes in, the logging sequence shall be:
data, data, data, data, data, end, data, data, data, data, end
i.e. once one request is done, end will be called once.
However, after doing several tests, when response is big. The sequence becomes:
<response 1 comes>data, data, data ..... data <response 2 comes> data, data, data, ..., data, end
i.e. the end for request 1 is missing.
In short, we need to make sure the callback of 'end' is called exactly once immediate after doing several call back of req.on('data', callback).
I believe there must be some common method for solving this issues (seems a classic bugs in node) and would be appreciated if anyone can indicate how to solve this property.
Thanks for the help!
From the code that you included it is impossible to make two requests. It makes one request with this:
var req = http.request(options, function(res) { ... });
Then binds the error handler here:
req.on('error', function(e) { ... });
And then immediately before even waiting for any response to the request, before even a connection is being made, it calls .write() and .end() methods on the request object:
req.write('<SOMETHING>');
req.end();
Nothing here can possibly cause two requests being made at the same time. But even if the first (and only) request hasn't started yet you already call .write() and .end() methods so maybe there's your problem.
In addition to that you should expect having one request being started before the other one finishes if you are going to do few requests in parallel as you're saying you'd like to.
i have the same issue before, i fixed like this:
res.on('finish', function () {
console.log("res finished");
});
Look here nodejs.org event-finish
Related
I'm new to nodejs and javascript in general. I believe this is an issue with the scope that I'm not understanding.
Given this example:
...
...
if (url == '/'){
var request = require('request');
var body_text = "";
request('http://www.google.com', function (error, response, body) {
console.log('error:', error);
console.log('statusCode:', response && response.statusCode);
console.log('body:', body);
body_text=body;
});
console.log('This is the body:', body_text)
//I need the value of body returned from the request here..
}
//OUTPUT
This is the body: undefined
I need to be able to get the body from response back and then do some manipulation and I do not want to do all the implementation within the request function. Of course, if I move the log line into:
request( function { //here })
It works. But I need to return the body in some way outside the request. Any help would be appreciated.
You can't do that with callbacks because this will works asynchronously.
Work with callbacks is kind of normal in JS. But you can do better with Promises.
You can use the request-promise-native to do what you want with async/await.
async function requestFromClient(req, res) {
const request = require('request-promise-native');
const body_text = await request('http://www.google.com').catch((err) => {
// always use catches to log errors or you will be lost
})
if (!body_text) {
// sometimes you won't have a body and one of this case is when you get a request error
}
console.log('This is the body:', body_text)
//I need the value of body returned from the request here..
}
As you see, you always must be in a function scope to use the async/await in promises.
Recommendations:
JS the right way
ES6 Fetures
JS clean coding
More best practices...
Using promises
I am running this script in node:
var http = require('http');
var server = http.createServer(function (request, response) {
response.writeHead(200, { "Content-Type": "text/plain" });
response.write('Hello World\n');
response.end('Goodbye World', 'utf8', function() {console.log(response.body);
});
server.listen(8000);
console.log('running');
When I load the page (localhost:8000) in Chrome I see:
Hello World
Goodbye World
So far so good, but I'm trying to understand where in the response object the data ('Hello World/nGoodbyeWorld') is. That's why I have 'console.log(response.body)' as the callback in response.end() ( the node http documentation says that the callback will be executed when the response has finished streaming). However the console.log just gives 'undefined'. When I console.log the whole response object it console.logs the response object ok but I can't see any data or body in there even though it has 'hasBody:true'.
So the question is:
a) is there a response.body? I am thinking there has to be one otherwise nothing would show in the browser window.
b) if so how can i access it and why doesn't my way work?
The closest answer i could find was this one: Where is body in a nodejs http.get response? , but I tried adding
response.on('data', function(chunk) {
body += chunk;
});
response.on('end', function() {
console.log(body);
});
, as suggested there and it didn't work. Also people there are just answering HOW you can access the data, not WHY the response.body isn't easily accessible.
Thanks
There is no response body, the data you write to the response stream is just sent to the client as you write it (for the most part). It wouldn't make sense to keep in memory everything ever written to the response.
The same goes for requests. You have to buffer the incoming data yourself if you want that, it is not done behind the scenes, it is merely streamed in.
According to this link: request - Node
The callback argument gets 3 arguments:
An error when applicable (usually from http.ClientRequest object) An
http.IncomingMessage object The third is the response body (String or
Buffer, or JSON object if the json option is supplied)
Code:
var r = require("request");
var options= {
url: "http://www.example.com/"
};
var callback = function (err, res, body) {
if (!err && res.statusCode == 200) {
res.on("data", function(chunk) {
console.log("DATA : "+chunk);
});
res.on("finish", function() {
console.log("FINISHED");
});
console.log(body);
}
};
r(options, callback);
But in the above code, only the console.log(body) works, the event emitters don't.
Also, if the callback would be invoked only when the whole response is body is available, then what's the point of making the second argument as http.IncomingMessage(Readable Stream) when I can't stream it.
When you pass a callback like that, request buffers the entire response for you and that is what is available in body. Because of this, that means you won't see data and such events on res, because they've already been taken care of by request.
It looks like you're mixing two different ways to use the 'request' module. Depending on preference you can use either the callback approach or the streaming approach.
The callback approach involves passing a function as well as the options and when all the data is received it will call the callback function.
The streaming approach allows you to attach listeners to the events such as 'response'. I'm guessing you've mixed this code in from an example from receiving http requests and sending a response with a node server as I can't see any reference to 'data' and 'finish' events in the docs for the request module.
I am using the node.js request module to post a large POST request (~150MB) to a REST service. For any request bigger than about 30MB, it seems to hang. My guess is that it is doing some naive JSON.stringify()ing to the data instead of streaming it, and once it gets large enough to hit swap, it just becomes very slow.
This is what my request looks like:
request({
uri: url
method: 'post',
json: args, // args is a 150MB json object
}, function(err, resp, body) {
// do something
});
The same request made using angularjs's $http from within a browser works in less than a minute, so I know it's the cilent and not the server.
Is there an easy way to fix this?
Streaming is not going to happen automatically. The API to stream something and the API to send a complete buffer are almost always different, and this is no exception. Most libraries out there given a complete javascript object in memory are just going to JSON.stringify it because you've already paid the memory and I/O price to load it in RAM so why bother streaming?
You could try the oboe streaming JSON library, which specializes in this type of thing. Here's a working example:
var oboe = require("oboe");
var args = {foo: "bar"};
var url = "http://localhost:2998";
oboe({url: url, body: args, method: "post"}).on("done", function (response) {
console.log('response:', response);
});
Aside: instead of guessing, you could verify in the source exactly what is happening. It's open source. It's javascript. Go ahead and dig in!
Updating answer:
I'd suggest your try two things :
See how much time superagent is taking to post the same data. Superagent is as simple as request.
var request = require('superagent');
request
.post(url)
.send(args)
.set('Accept', 'application/json')
.end(function(error, res){
});
Compress the data to be posted by using zlip, your will be compressing the data into a zlip buffer and write that as your output
var zlib = require('zlib');
var options = {
hostname: 'www.yourwebsite.com',
port: 80,
path: '/your-post-url',
method: 'POST',
headers: {'Content-Encoding': 'gzip'} // tell the server that the data is compressed
};
//args is your JSON stringified data
zlib.gzip(args, function (err, buffer) {
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
// ... do stuff with returned data
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
req.write(buffer); // send compressed data
req.end();
});
I'm having trouble with something very basic. Going through node.js in Action (great book so far!) and I can't get this simple example to work. Perhaps it's because the stream api was updated after the book came out. I'm not sure. Anyway, here's the code:
var http = require('http');
var server = http.createServer(function (req, res) {
req.on('data', function (chunk) {
console.log("Chuck: ", chunk);
});
req.on('end', function () {
console.log("End of Request");
res.end('yay');
});
}).listen(3000);
The console.log('Chunk: ', chunk) never fires. It's almost as if the data events never fire, but according to the documentation the presence of the data handler should switch the readable stream (req) into flowing mode. Am I missing something?
Any help would rock!
The above code is current initally the request body is undefined you have to pass request data inorder to execute this line.
req.on('data', function (chunk) {
console.log("Chuck: ", chunk);
});
Use postman to send data in the request this line will be executed