I am using the node.js request module to post a large POST request (~150MB) to a REST service. For any request bigger than about 30MB, it seems to hang. My guess is that it is doing some naive JSON.stringify()ing to the data instead of streaming it, and once it gets large enough to hit swap, it just becomes very slow.
This is what my request looks like:
request({
uri: url
method: 'post',
json: args, // args is a 150MB json object
}, function(err, resp, body) {
// do something
});
The same request made using angularjs's $http from within a browser works in less than a minute, so I know it's the cilent and not the server.
Is there an easy way to fix this?
Streaming is not going to happen automatically. The API to stream something and the API to send a complete buffer are almost always different, and this is no exception. Most libraries out there given a complete javascript object in memory are just going to JSON.stringify it because you've already paid the memory and I/O price to load it in RAM so why bother streaming?
You could try the oboe streaming JSON library, which specializes in this type of thing. Here's a working example:
var oboe = require("oboe");
var args = {foo: "bar"};
var url = "http://localhost:2998";
oboe({url: url, body: args, method: "post"}).on("done", function (response) {
console.log('response:', response);
});
Aside: instead of guessing, you could verify in the source exactly what is happening. It's open source. It's javascript. Go ahead and dig in!
Updating answer:
I'd suggest your try two things :
See how much time superagent is taking to post the same data. Superagent is as simple as request.
var request = require('superagent');
request
.post(url)
.send(args)
.set('Accept', 'application/json')
.end(function(error, res){
});
Compress the data to be posted by using zlip, your will be compressing the data into a zlip buffer and write that as your output
var zlib = require('zlib');
var options = {
hostname: 'www.yourwebsite.com',
port: 80,
path: '/your-post-url',
method: 'POST',
headers: {'Content-Encoding': 'gzip'} // tell the server that the data is compressed
};
//args is your JSON stringified data
zlib.gzip(args, function (err, buffer) {
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
// ... do stuff with returned data
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
req.write(buffer); // send compressed data
req.end();
});
Related
I am currently buildings proxy using nodejs, which use following syntax for sending and receiving https request and response. However in my project, the response is a liitle bit larger, so typically, req.on('data', callback) will be called 5~7 times before req.on('end', callback) being called.
Here is the simplified code structure:
var http = require("https");
var options = {
hostname: '<WEB SERVICE>',
port: 80,
path: '<WEB PATH>',
method: 'POST',
headers: {
'Content-Type': 'application/json',
}
};
var response = "";
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (body) {
console.log("data");
response += body;
});
res.on('end', function () {
console.log("end");
response = "";
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
// write data to request body
req.write('<SOMETHING>');
req.end();
Ideally, when multiple request comes in, the logging sequence shall be:
data, data, data, data, data, end, data, data, data, data, end
i.e. once one request is done, end will be called once.
However, after doing several tests, when response is big. The sequence becomes:
<response 1 comes>data, data, data ..... data <response 2 comes> data, data, data, ..., data, end
i.e. the end for request 1 is missing.
In short, we need to make sure the callback of 'end' is called exactly once immediate after doing several call back of req.on('data', callback).
I believe there must be some common method for solving this issues (seems a classic bugs in node) and would be appreciated if anyone can indicate how to solve this property.
Thanks for the help!
From the code that you included it is impossible to make two requests. It makes one request with this:
var req = http.request(options, function(res) { ... });
Then binds the error handler here:
req.on('error', function(e) { ... });
And then immediately before even waiting for any response to the request, before even a connection is being made, it calls .write() and .end() methods on the request object:
req.write('<SOMETHING>');
req.end();
Nothing here can possibly cause two requests being made at the same time. But even if the first (and only) request hasn't started yet you already call .write() and .end() methods so maybe there's your problem.
In addition to that you should expect having one request being started before the other one finishes if you are going to do few requests in parallel as you're saying you'd like to.
i have the same issue before, i fixed like this:
res.on('finish', function () {
console.log("res finished");
});
Look here nodejs.org event-finish
I have an application that uses axios to make requests to a node server which in turn makes requests to another java server.
Call to node server from client:
// here payload is FormData()
axios.post(url, payload).then((response) => {
return callback(null, response);
}).catch((err) => {
return callback(err, null);
});
In the node server, I listen to the request using busboy:
let rawData = '';
const busboy = new Busboy({headers: req.headers});
busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
file.on('data', function (chunk) {
rawData += chunk;
});
});
Now the java server too expects FormData (just like the way I sent it to node). How do I get the FormData from node now? I have been googling hard and trying a lot of stuff in vain. Any solution not involving busboy will help too.
I had finally used the middleware busboy-body-parser which adds support for getting files from the request object as req.files. And once the file is there, I send it as form-data to the java web server using the form-data npm package. The req.files support used to be there by default in Express.js. But from 4.x, it has been deprecated.
Multer is another really good middleware for handling multipart/form-data.
I'm trying to download a tar file (non-compressed) over HTTP and piping it's response to the tar-stream parser for further processing. This works perfect when executed on the terminal without any errors. For the same thing to be utilized on browser, a bundle.js file is generated using browserify and is included in the HTML.
The tar stream contains 3 files. This browserified code when executed on the browser parses 2 entries successfully but raises the following error for the third one:
Error: Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?
Whereas with the same HTTP download and parsing code, the tar file is downloaded and parsed completely without errors on terminal. Why is this happening?!
Code snippet is along these lines:
. . . .
var req = http.request(url, function(res){
res.pipe(tar.extract())
.on('entry', function(header, stream, callback) {
console.log("File found " + header.name);
stream.on('end', function() {
console.log("<<EOF>>");
callback();
})
stream.resume();
})
.on('finish', function(){
console.log("All files parsed");
})
.on('error', function(error){
console.log(error); //Raises the above mentioned error here
})
});
. . . .
Any Suggestions? Headers?
The problem here (and its solution) are tucked away in the http-browserify documentation. First, you need to understand a few things about browserify:
The browser environment is not the same as the node.js environment
Browserify does its best to provide node.js APIs that don't exist in the browser when the code you are browserifying needs them
The replacements don't behave exactly the same as in node.js, and are subject to caveats in the browser
With that in mind, you're using at least three node-specific APIs that have browserify reimplementations/shims: network connections, buffers, and streams. Network connections by necessity are replaced in the browser by XHR calls, which have their own semantics surrounding binary data that don't exist within Node [Node has Buffers]. If you look here, you'll notice an option called responseType; this sets the response type of the XHR call, which must be done to ensure you get binary data back instead of string data. Substack suggested to use ArrayBuffer; since this must be set on the options object of http.request, you need to use the long-form request format instead of the string-url format:
http.request({
method: 'GET',
hostname: 'www.site.com',
path: '/path/to/request',
responseType: 'arraybuffer' // note: lowercase
}, function (res) {
// ...
});
See the xhr spec for valid values for responseType. http-browserify passes it along as-is. In Node, this key will simply be ignored.
When you set the response type to 'arraybuffer', http-browserify will emit chunks as Uint8Array. Once you're getting a Uint8Array back from http.request, another problem presents itself: the Stream API only accepts string and Buffer for input, so when you pipe the response to the tar extractor stream, you'll receive TypeError: Invalid non-string/buffer chunk. This seems to me to be an oversight in stream-browserify, which should accept Uint8Array values to go along nicely with the other parts of the browserified Node API. You can fairly simply work around it yourself, though. The Buffer shim in the browser accepts a typed array in the constructor, so you can pipe the data yourself, converting each chunk to a Buffer manually:
http.request(opts, function (res) {
var tarExtractor = tar.extract();
res.on('data', function (chunk) {
tarExtractor.write(new Buffer(chunk));
});
res.on('end', function () {
tarExtractor.end();
});
res.on('error', function (err) {
// do something with your error
// and clean up the tarExtractor instance if necessary
});
});
Your code, then, should look something like this:
var req = http.request({
method: 'GET',
// Add your request hostname, path, etc. here
responseType: 'arraybuffer'
}, function(res){
var tarExtractor = tar.extract();
res.on('data', function (chunk) {
tarExtractor.write(new Buffer(chunk));
});
res.on('end', tarExtractor.end.bind(tarExtractor));
res.on('error', function (error) {
console.log(error);
});
tarExtractor.on('entry', function(header, stream, callback) {
console.log("File found " + header.name);
stream.on('end', function() {
console.log("<<EOF>>");
callback();
})
stream.resume(); // This won't be necessary once you do something with the data
})
.on('finish', function(){
console.log("All files parsed");
});
});
I have some code that is trying to get a JSON result from the Soundcloud API.
I registered an app, got the client id and such, and I'm trying to make a call like this:
var querystring = require('querystring');
var http = require('http');
var addr = 'http://api.soundcloud.com/resolve.json?url=http://soundcloud.com/matas/hobnotropic&client_id=XXXXX';
var options = {
hostname: "api.soundcloud.com",
path: "/resolve.json?url=http://soundcloud.com/matas/hobnotropic&client_id=XXXXXx",
method: "GET",
headers: {
"Content-Type": "application/json"
}
}
var request = http.get(options, function(response) {
response.setEncoding('utf8');
response.on('data', function(chunk) {
console.log(chunk);
});
});
This produces a result that looks like this:
{"status":"302 - Found","location":"https://api.soundcloud.com/tracks/49931.json?client_id=xxxx"}
When I use the same URL in Chrome, I get the proper JSON info. How do I properly make this call from server side script?
The built-in http client does not handle redirects. However request does and has many other features the built-in client does not support out of the box.
Today I updated my own NodeJS Api-Wrapper Package for Soundcloud, which can be found here: https://www.npmjs.com/package/soundcloud-nodejs-api-wrapper
It does server side API communication, which includes data modification. No user popup window and redirect url is needed.
I did not found yet any other package having support for this in NodeJS.
i'm fetching some binary data over http. My code looks like:
var writeStream = fs.createWriteStream(fileName);
request(url, function(err, res) {
res.socket.pipe(writeStream);
});
now the output file is created but the filesize is 0. The url is correct though, i verified that with wget.
Thanks in advance & best regards
The callback for http.request only supplies one argument, which is a reference to the response of the request. Try
http.request(url, function(res) {
res.pipe(writeStream);
});
Also note that the ClientResponse implements ReadableStream, so you should use .pipe rather than .socket.pipe.
I'm assuming that here request is from mikeal's request library rather than being an instance of http.request. In that case you can simply do request(url).pipe(writeStream);
Remember that for debugging purposes, you can always pipe to process.stdout.
var readStream = fs.createReadStream(fileName);
request(url, function(err, res) {
readStream.pipe(res);
readStream.on('end', function() {
//res.end({"status":"Completed"});
});
});