I have a question about the progress of sending with Node.js 'request'. I have a Node.js application that acts as a proxy, actually forwarding a body to another server. I want to see the progress of this upload action. Now I am doing this:
BROWSER
-> UPLOAD TO NODE
-> UPLOAD TO 3rd PARTY SERVICE
-> RETURN 3rd PARTY RESPONSE TO BROWSER
If this is possible, I would log it to check in the console.log how much progress is done. But, would it also be possible to return a
res.send(progress)
in the mean time, while waiting for the upload to finish and send the client back the upload has succeeded?
BROWSER
-> UPLOAD TO NODE
-> UPLOAD TO 3rd PARTY SERVICE
-> RETURN progress <-
-> RETURN progress <-
...etc.
-> RETURN 3rd PARTY RESPONSE TO BROWSER
This is the upload code (pretty much straightforward).
var requestOptions = {
timeout: 120000,
url: url, //URL to hit
method: 'post',
headers: headers,
body: payload //Set the body as a string
};
request(requestOptions, function (error, response, body) {
if (error) {
res.send(error);
}
else {
//res.sendStatus(response.statusCode);
console.log("[RETURNING RESPONSE BODY]");
res.send(body);
}
});
Your question contains two parts. One is for getting the progress from request, which can be found here: Upload Progress — Request
The other part would be notifying the browser of the progress. The HTTP protocol does not let you send multiple responses, so using res.send(progress) is not possible.
However, you can keep sending partial responses until it finishes. Writing something to res without closing it is as simple as res.write("string"), but accessing the response can be harder: Accessing partial response using AJAX or WebSockets?
You also need a way to wrap the body (and also errors) from the backing server so that it can fit as the final partial response.
Another solution would be opening another WebSocket request to track the uploading/downloading process. socket.io is a good library for node for this purpose.
Solution for first part was in r.req.connectin.bytesWritten
var r = request(requestOptions, function (error, response, body) {
clearInterval(q);
...
});
var q = setInterval(function () {
console.log("Uploaded: " + r.req.connection.bytesWritten);
}, 250);
Related
There are a lot of similar questions but I think I have a new twist on it.
I'm trying to call a third party url via ajax for tracking purposes, so I don't really care about the response. just that it's sent.
The tracking site has given us a url that they expect the user to click on which then does a 301 redirect to our product page. We want to show a modal window instead of redirecting to the product page, like a quick view button for instance.
The problem is that the browser is sending two requests to the original url which is going to skew our statistics.
Looking at the network panel in chrome I can see one is from jQuery's ajax function and has the call stack associated with it. The second request appears to come from a 'click' event.
If I remove the ajax request from the function then no requests are sent.
I've created a mock server for further testing. When it returned a 200 there was only one request sent. When I changed it to a 301 the two requests were sent.
The weird thing is that both requests are to the original URL. It's not even trying to following the redirect.
I'm a bit stuck and I'm hoping someone can give me some direction on what to try next.
This is the function that calls $.ajax:
function externalRequest(settings) {
return new Promise((resolve, reject) => {
$.ajax(settings)
.done((data, textStatus, jqXHR) => {
const { status, statusText } = jqXHR;
resolve({
data,
status,
statusText,
});
})
.fail(reject);
});
}
Generic third party tagging function:
function thirdPartyTracking(url) {
Ajax.externalRequest({
method: 'GET',
url,
}).catch((error) => {
console.error('Error sending tracking tag: ', url, error);
});
}
This is called like:
thirdPartyTracking('https://tracking.com/click?sku=123&uid=abc456')
Results on the network panel:
Thanks
So my scenario is a user clicks a button on a web app, this triggers a server side POST request to an internal (i.e non public) API sitting on another server in the same network, this should return a PDF to my server which will proxy (pipe) it back to the user.
I want to just proxy the PDF body content directly to the client without creating a tmp file.
I have this code which works using the npm request module but it does not feel right:
var pdfRequest = request(requestOptions);
pdfRequest.on('error', function (err) {
utils.sendErrorResponse(500, 'PROBLEM PIPING PDF DOWNLOAD: ' + err, res);
});
pdfRequest.on('response', function (resp) {
if (resp.statusCode === 200) {
pdfRequest.pipe(res);
} else {
utils.sendErrorResponse(500, 'PROBLEM PIPING PDF DOWNLOAD: RAW RESP: ' + JSON.stringify(resp), res);
}
});
Is the the correct way to pipe the PDF response?
Notes:
I need to check the status code to conditionally handle errors, the payload for the POST is contained in the requestOptions (I know this part is all correct).
I would like to keep using the request module
I defiantly do not want to be creating any temp files
If possible I would also like to modify the content disposition header to set a custom filename, i know how to do this without using pipes
This is something I have no experience with.
On the admin page I'm building it's possible to upload a picture to a service called Cloudinary to have its background removed, a process that takes up to 24 hours, upon which they send a POST request to some url that I send along with the request.
This format doesn't lend itself very well to experimenting, so I need some help.
This is how the picture is uploaded and the request sent:
Cloudinary.upload(image, {
folder: "cutouts",
type: "upload",
notification_url: "someurl.com"
background_removal: "remove_the_background"
}, function(error, result) {
if (!error)
console.dir(result)
}
})
Question: what do I set as the notification_url? Even if it's somewhere on my site, do I need to deploy it before I can check that the code works?
Here's an example of what they will send back, according to their docs:
{
"notification_type": "info",
"info_kind": "remove_the_background",
"info_status": "complete",
"public_id": "wood_chair",
"uploaded_at": "2014-10-26T11:35:22Z",
"version": 1414316122,
"url":
"http://res.cloudinary.com/demo/image/upload/v1393688588/wood_chair.jpg",
"secure_url":
"https://res.cloudinary.com/demo/image/upload/v1393688588/wood_chair.jpg",
"etag": "a56e9e88c2add15cac1775c1f687bf73"
}
So, specifically I need access to the info_status and the url.
Problem is, I can't experiment and I have no idea how to approach this problem since I have no experience with this stuff at all.
Here are their docs if it's any help: http://cloudinary.com/documentation/remove_the_background_image_editing_addon
How do I do this?
On your server side, set up a route (assuming you are using iron-router) to handle the post from cloudinary:
Router.route('/api/cloudinary', { where: 'server' })
.post(function() {
var body = this.request.body; // get the body out of the response
var url = body.url; // based on the JSON you showed
this.response.statusCode = 200; // set the status code to be returned to cloudinary
this.response.end(); // send response
}
);
You can change /api/cloudinary/ to whatever you want. If your site was http://www.example.com, then the cloudinary notification url would be:
http://www.example.com/api/cloudinary
I have a node.js process that uses a large number of client requests to pull information from a website. I am using the request package (https://www.npmjs.com/package/request) since, as it says: "It supports HTTPS and follows redirects by default."
My problem is that after a certain period of time, the requests begin to hang. I haven't been able to determine if this is because the server is returning an infinite data stream, or if something else is going on. I've set the timeout, but after some number of successful requests, some of them eventually get stuck and never complete.
var options = { url: 'some url', timeout: 60000 };
request(options, function (err, response, body) {
// process
});
My questions are, can I shut down a connection after a certain amount of data is received using this library, and can I stop the request from hanging? Do I need to use the http/https libraries and handle the redirects and protocol switching myself in order the get the kind of control I need? If I do, is there a standardized practice for that?
Edit: Also, if I stop the process and restart it, they pick right back up and start working, so I don't think it is related to the server or the machine the code is running on.
Note that in request(options, callback), the callback will be fired when request is completed and there is no way to break the request.
You should listen on data event instead:
var request = require('request')
var stream = request(options);
var len = 0
stream.on('data', function(data) {
// TODO process your data here
// break stream if len > 1000
len += Buffer.byteLength(data)
if (len > 1000) {
stream.abort()
}
})
I want to use nodeJS as tool for website scrapping. I have already implemented a script which logs me in on the system and parse some data from the page.
The steps are defined like:
Open login page
Enter login data
Submit login form
Go to desired page
Grab and parse values from the page
Save data to file
Exit
Obviously, the problem is that every time my script has to login, and I want to eliminate that. I want to implement some kind of cookie management system, where I can save cookies to .txt file, and then during next request I can load cookies from file and send it in request headers.
This kind of cookie management system is not hard to implement, but the problem is how to access cookies in nodejs? The only way I found it is using request response object, where you can use something like this:
request.get({headers:requestHeaders,uri: user.getLoginUrl(),followRedirect: true,jar:jar,maxRedirects: 10,},function(err, res, body) {
if(err) {
console.log('GET request failed here is error');
console.log(res);
}
//Get cookies from response
var responseCookies = res.headers['set-cookie'];
var requestCookies='';
for(var i=0; i<responseCookies.length; i++){
var oneCookie = responseCookies[i];
oneCookie = oneCookie.split(';');
requestCookies= requestCookies + oneCookie[0]+';';
}
}
);
Now content of variable requestCookies can be saved to the .txt file and can loaded next time when script is executed, and this way you can avoid process of logging in user every time when script is executed.
Is this the right way, or there is a method which returns cookies?
NOTE: If you want to setup your request object to automatically resend received cookies on every subsequent request, use the following line during object creation:
var request = require("request");
request = request.defaults({jar: true});//Send cookies on every subsequent requests
In my case, i've used 'http'library like the following:
http.get(url, function(response) {
variable = response.headers['set-cookie'];
})
This function gets a specific cookie value from a server response (in Typescript):
function getResponseCookieValue(res: Response, param: string) {
const setCookieHeader = res.headers.get('Set-Cookie');
const parts = setCookieHeader?.match(new RegExp(`(^|, )${param}=([^;]+); `));
const value = parts ? parts[2] : undefined;
return value;
}
I use Axios personally.
axios.request(options).then(function (response) {
console.log(response.config.headers.Cookie)
}).catch(function (error) {
console.error(error)
});