i'm fetching some binary data over http. My code looks like:
var writeStream = fs.createWriteStream(fileName);
request(url, function(err, res) {
res.socket.pipe(writeStream);
});
now the output file is created but the filesize is 0. The url is correct though, i verified that with wget.
Thanks in advance & best regards
The callback for http.request only supplies one argument, which is a reference to the response of the request. Try
http.request(url, function(res) {
res.pipe(writeStream);
});
Also note that the ClientResponse implements ReadableStream, so you should use .pipe rather than .socket.pipe.
I'm assuming that here request is from mikeal's request library rather than being an instance of http.request. In that case you can simply do request(url).pipe(writeStream);
Remember that for debugging purposes, you can always pipe to process.stdout.
var readStream = fs.createReadStream(fileName);
request(url, function(err, res) {
readStream.pipe(res);
readStream.on('end', function() {
//res.end({"status":"Completed"});
});
});
Related
I Googled this but couldn't find an answer but it must be a common problem. This is the same question as Node request (read image stream - pipe back to response), which is unanswered.
How do I send an image file as an Express .send() response? I need to map RESTful urls to images - but how do I send the binary file with the right headers? E.g.,
<img src='/report/378334e22/e33423222' />
Calls...
app.get('/report/:chart_id/:user_id', function (req, res) {
//authenticate user_id, get chart_id obfuscated url
//send image binary with correct headers
});
There is an api in Express.
res.sendFile
app.get('/report/:chart_id/:user_id', function (req, res) {
// res.sendFile(filepath);
});
http://expressjs.com/en/api.html#res.sendFile
a proper solution with streams and error handling is below:
const fs = require('fs')
const stream = require('stream')
app.get('/report/:chart_id/:user_id',(req, res) => {
const r = fs.createReadStream('path to file') // or any other way to get a readable stream
const ps = new stream.PassThrough() // <---- this makes a trick with stream error handling
stream.pipeline(
r,
ps, // <---- this makes a trick with stream error handling
(err) => {
if (err) {
console.log(err) // No such file or any other kind of error
return res.sendStatus(400);
}
})
ps.pipe(res) // <---- this makes a trick with stream error handling
})
with Node older then 10 you will need to use pump instead of pipeline.
I have this snippet of code:
app.post('/pst', function(req, res) {
var data = req.body.convo;
res.render('waiting.ejs'); //ADDED THIS
myFunc(data).then(result => {
res.render('success.ejs'); //THEN THIS
//---------------------------------
//clever way to send text file to client from the memory of the server
var fileContents = Buffer.from(result, 'ascii');
var readStream = new stream.PassThrough();
readStream.end(fileContents);
res.set('Content-disposition', 'attachment; filename=' + fileName);
res.set('Content-Type', 'text/plain');
readStream.pipe(res);
//--------------------------------------
}).catch( .....
The code i commented as 'clever way to send file from memory of the server' comes from this post:
Node Express.js - Download file from memory - 'filename must be a string'
What this does is is takes a string from the memory and serves it to the client as a .txt file.
This code used to work.
Then i decided to add the res.render('waiting.ejs'); line and i got this error:
Error: Can't set headers after they are sent.
I then experimented with adding another res.render() [in this case res.render('success.ejs');] before and after the code tht sends the .txt file to the client.
The error remained. Also, there is no redirect to success.ejs, in other words the res.render('success.ejs'); never worked, despite whether it is placed before ofr after that piece of code.
app.post('/pst', function(req, res) {
var data = req.body.convo;
myFunc(data).then(result => {
//---------------------------------
//clever way to send text file to client from the memory of the server
var fileContents = Buffer.from(result, 'ascii');
var readStream = new stream.PassThrough();
readStream.end(fileContents);
res.set('Content-disposition', 'attachment; filename=' + fileName);
res.set('Content-Type', 'text/plain');
readStream.pipe(res);
res.redirect(`/success`); //THEN THIS
//--------------------------------------
}).catch( .....
When you add middleware to express (which is built on connect) using the app.use method, you're appending items to Server.prototype.stack in connect.
When the server gets a request, it iterates over the stack, calling the (request, response, next) method.
The problem is, if in one of the middleware items writes to the response body or headers (it looks like it's either/or for some reason), but doesn't call response.end() and you call next() then as the core Server.prototype.handle method completes, it's going to notice that:
there are no more items in the stack, and/or
that response.headerSent is true.
So, it throws an error. But the error it throws is just this basic response (from the connect http.js source code:
res.statusCode = 404;
res.setHeader('Content-Type', 'text/plain');
res.end('Cannot ' + req.method + ' ' + req.url);
The problematic middleware sets the response header without calling response.end() and calls next(), which confuses express's server.
so you set the header through res.render() .Now if you will try to render again it will throw you an error.
app.get('/success',(req,res)=> {
res.render("container/index",{waiting:"waiting",......});
//handle your task then in client side index.ejs with appropriate setTimeout(()=>{},2000) for the waiting div , show waiting div for 2 seconds
});
//then your actual success gets render
You would have to check express.js source code (here):
res.render = function render(view, options, callback) {
var app = this.req.app;
var done = callback;
var opts = options || {};
var req = this.req;
var self = this;
// support callback function as second arg
if (typeof options === 'function') {
done = options;
opts = {};
}
// merge res.locals
opts._locals = self.locals;
// default callback to respond
done = done || function (err, str) {
if (err) return req.next(err);
self.send(str);
};
// render
app.render(view, opts, done);
};
You can see that when You use res.render() method, it will pass the done callback to app.render(...) (source code), it will then pass done to tryInitView etc.
At the end, it will invoke done callback with str in case of success or err in case of failure. It then triggers res.send() inside done callback which simply blocks You from setting headers after that.
res.render() function compiles your template, inserts locals there, and creates html output out of those two things. that's why error comes.
don't use it twice coz it send response.
I'm new to node.js so I'll try my best to explain the problem here. Let me know if any clerification is needed.
In my node.js application I'm trying to take a code (which was received from the response of the 1st call to an API), and use that a code to make a 2nd request(GET request) to another API service. The callback url of the 1st call is /pass. However I got an empty response from the service for this 2nd call.
My understanding is that after the call back from the 1st call, the function in app.get('/pass', function (req, res).. gets invoked and it sends a GET request. What am I doing wrong here? Many thanks in advance!
Here is the part where I try to make a GET request from node.js server and receive an empty response:
app.get('/pass', function (req, res){
var options = {
url: 'https://the url that I make GET request to',
method: 'GET',
headers: {
'authorization_code': code,
'Customer-Id':'someID',
'Customer-Secret':'somePassword'
}
};
request(options, function(err, res, body) {
console.log(res);
});
});
Im a little confused by what you are asking so ill just try to cover what i think you're looking for.
app.get('/pass', (req, res) => {
res.send("hello!"); // localhost:port/pass will return hello
})
Now, if you are trying to call a get request from the request library when the /pass endpoint is called things are still similar. First, i think you can remove the 'method' : 'GET' keys and values as they are not necessary. Now the code will be mostly the same as before except for the response.
app.get('/pass', (req, res) => {
var options = {
url: 'https://the url that I make GET request to',
headers: {
'authorization_code': code,
'Customer-Id':'someID',
'Customer-Secret':'somePassword'
}
};
request(options, function(err, res, body) {
// may need to JSONparse the body before sending depending on what is to be expected.
res.send(body); // this sends the data back
});
});
I've got a simple node.js + Restify backend with standard CORS settings and this endpoint:
var file = '1,5,8,11,12,13,176,567,9483';
server.get('/download', function(req, res, next) {
res.set({"Content-Disposition": "attachment; filename='numbers.csv'"});
res.setHeader("Content-type", "text/csv");
res.send(file);
return next();
}, function(err) {
res.send(err);
});
What it's suppose to do is to is to create CSV file and return it.
It works great when I simply type in the endpoint address to web browser and hit enter. The file gets downloaded properly.
But when I try to do the same thing, but instead of using browser's address bar I use Restangular like that:
Restangular.one('download').get().then(function (res) {
console.log(res);
});
it just writes response to console, but no file is being downloaded.
Is there a way to do this using Restangular? Or maybe I need to use something else for this?
I am not sure if Restangular can do that, but I am using FileSaver script for stuff like that. Add Filesaver to your HTML head and then:
Restangular.one('download').get().then(function (res) {
var file = new Blob([res], { type: 'text/csv' });
saveAs(file, 'something.csv');
});
I am using the node.js request module to post a large POST request (~150MB) to a REST service. For any request bigger than about 30MB, it seems to hang. My guess is that it is doing some naive JSON.stringify()ing to the data instead of streaming it, and once it gets large enough to hit swap, it just becomes very slow.
This is what my request looks like:
request({
uri: url
method: 'post',
json: args, // args is a 150MB json object
}, function(err, resp, body) {
// do something
});
The same request made using angularjs's $http from within a browser works in less than a minute, so I know it's the cilent and not the server.
Is there an easy way to fix this?
Streaming is not going to happen automatically. The API to stream something and the API to send a complete buffer are almost always different, and this is no exception. Most libraries out there given a complete javascript object in memory are just going to JSON.stringify it because you've already paid the memory and I/O price to load it in RAM so why bother streaming?
You could try the oboe streaming JSON library, which specializes in this type of thing. Here's a working example:
var oboe = require("oboe");
var args = {foo: "bar"};
var url = "http://localhost:2998";
oboe({url: url, body: args, method: "post"}).on("done", function (response) {
console.log('response:', response);
});
Aside: instead of guessing, you could verify in the source exactly what is happening. It's open source. It's javascript. Go ahead and dig in!
Updating answer:
I'd suggest your try two things :
See how much time superagent is taking to post the same data. Superagent is as simple as request.
var request = require('superagent');
request
.post(url)
.send(args)
.set('Accept', 'application/json')
.end(function(error, res){
});
Compress the data to be posted by using zlip, your will be compressing the data into a zlip buffer and write that as your output
var zlib = require('zlib');
var options = {
hostname: 'www.yourwebsite.com',
port: 80,
path: '/your-post-url',
method: 'POST',
headers: {'Content-Encoding': 'gzip'} // tell the server that the data is compressed
};
//args is your JSON stringified data
zlib.gzip(args, function (err, buffer) {
var req = http.request(options, function(res) {
res.setEncoding('utf8');
res.on('data', function (chunk) {
// ... do stuff with returned data
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
req.write(buffer); // send compressed data
req.end();
});