We Keep Coding

multiple PDF set header

I have this in a loop but I have multiple files that I'd like to generate. I'm receiving Cannot set headers after they are sent to the client.
If I have one key, then it's fine. If I have more than one, it generates the second one, doesn't generate the first one and gives this error.
_.forEach(keys, async function(key) {
try {
const pdf = await myPDFd(key);
res.setHeader('Content-disposition', 'attachment; filename=' + key + '.pdf');
res.setHeader('content-type', 'application/pdf');
res.send(pdf);
res.end();
} catch (error) {
}
});

Your way is not possible.
Each HTTP request responses exactly 1 type of data (one PDF file in your case). Not possible to send 2 files via 1 HTTP request.
You have to create some logic on client (two download links, for example). Client (user or app) must perform two separate actions (run two separate HTTP requests) for 2 files downloading.
Alternatively you can compress this two files to ZIP (or another package type) and send this one ZIP file to user. But in this case user (or app) will have to uncompress it as additional step. You can use node-zip library.

how to use a nodeJS out in javascript file

I would like to use the output of my nodeJS. This is my code
var fs = require('fs'); //File System
var rutaImagen = 'C:/Users/smontesc/Desktop/imagenes1/'; //Location of images
fs.readdir(rutaImagen, function(err, files) {
if (err) { throw err; }
var imageFile = getNewestFile(files, rutaImagen);
//process imageFile here or pass it to a function...
console.log(imageFile);
});
function getNewestFile(files, path) {
var out = [];
files.forEach(function(file) {
var stats = fs.statSync(path + "/" +file);
if(stats.isFile()) {
out.push({"file":file, "mtime": stats.mtime.getTime()});
}
});
out.sort(function(a,b) {
return b.mtime - a.mtime;
})
return (out.length>0) ? out[0].file : "";
}
And the result is console.log(imageFile), I want to call the result of this in my javascript project, like
<script>
document.write(imageFile)
</script>
All this is to get the newest file created in a directory because I can't do it directly on JS.
Thanks a lot

First, there are several fundamental things about how the client/server relationship of the browser and a web server work that we need to establish. That will then offer a framework for discussing solving your problem.
Images are displayed in a browser, not with document.write(), but by inserting an image tag in your document that points to the URL of a specific image.
For a web page to get some result from the server, it has to either have that result embedded in the web page when the web page was originally fetched from the server or the Javascript in the web page has to request information from the server with an Ajax request. An ajax request is an http request where the Javascript in your web page, forms an http request that is sent to your server, your server receives that request and sends back a response which the Javascript in your web page receives and can then do something with.
To implement something where your web page requests some data from your back-end, you will have to have a web server in your back-end that can response to Ajax requests sent from the web page. You cannot just run a script on your server and magically modify a web page displayed in a browser. Without the type of structure described in the previous points, your web page has no connection at all to the displayed server. The web page can't directly reach your server file system and the server can't directly touch the displayed web page.
There are a number of possible schemes for implementing this type of connection. What I would think would work best would be to define an image URL that, when requested by any browser, it returns an image for the newest image in your particular directory on your server. Then, you would just embed that particular URL in your web page and anytime that image was refreshed or displayed, your server would send it the newest version of that image. Your server probably also needs to make sure that the browser does not cache that URL by setting appropriate cache headers so that it won't mistakenly just display the previously cached version of that image.
The web page could look like this:
<img src='http://mycustomdomain.com/dimages/newest'>
Then, you'd set up a web server at mycustomdomain.com that is publicly accessible (from the open internet - you choose your own domain obviously) that has access to the desired images and you'd create a route on that web server that answers to the /dimages/newest request.
Using Express as your web server framework, this could look like this:
const app = require('express')();
const fs = require('fs');
const util = require('util');
const readdir = util.promisify(fs.readdir);
const stat = util.promisify(fs.stat);
// middleware to use in some routes that you don't want any caching on
function nocache(req, res, next) {
res.header('Cache-Control', 'private, no-cache, no-store, must-revalidate, proxy-revalidate');
res.header('Expires', '-1');
res.header('Pragma', 'no-cache');
next();
}
const rutaImagen = 'C:/Users/smontesc/Desktop/imagenes1/'; //Location of images
// function to find newest image
// returns promise that resolves with the full path of the image
// or rejects with an error
async function getNewestImage(root) {
let files = await readdir(root);
let results = [];
for (f of files) {
const fullPath = root + "/" + f;
const stats = await stat(fullPath);
if (stats.isFile()) {
results.push({file: fullPath, mtime: stats.mtime.getTime()});
}
}
results.sort(function(a,b) {
return b.mtime - a.mtime;
});
return (results.length > 0) ? results[0].file : "";
}
// route for fetching that image
app.get(nocache, '/dimages/newest', function(req, res) {
getNewestImage(rutaImagen).then(img => {
res.sendFile(img, {cacheControl: false});
}).catch(err => {
console.log('getNewestImage() error', err);
res.sendStatus(500);
});
});
// start your web server
app.listen(80);

To be able to use that result in your Javascipt project, we definitely have to create an API which has a particular route that responses the imageFile. Then, in your Javascript project, you can use XMLHttpRequest (XHR) objects or the Fetch API to interact with servers to get the result.
The core idea is we definitely need both server-side and client-side programming to perform that functionality.

How can I GET content of a HTTPS webpage?

I want to get the content of a webpage by running javascript code on NodeJs . I want the content to be exactly the same as what I see in the browser.
This is the URL :
https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9
I use the following code but I get 405 in response.
var fs = require('fs');
var link = 'https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9';
var request = require('request');
request(link, function (error, response, body) {
fs.writeFile("realestatedata.html", body, function(err) {
if(err) {
console.log('error in saving the file');
return console.log(err);
}
console.log("The file was saved!");
});
})
The file which is saved is not related to what I can see in the browser.

I think a real answer will be easier to understand since my comment was truncated.
It seems the method of the request you send is not supported by the server (405 Method Not Allowed - The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource.). Do you have more information about the HTTP response.
Have you tried the following code instead of yours ?
request('https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9').pipe(fs.createWriteStream('realestatedata.html'))
You could also have a look at In Node.js / Express, how do I "download" a page and gets its HTML?.
Note that anyway the page will not render the same way when you only open the html since it also requires many other resources (110 requests are done when display the page).
I think the following answer can help you to download the whole page.
https://stackoverflow.com/a/34935427/1630604

Node.js progress indicator feedback

I have a question about the progress of sending with Node.js 'request'. I have a Node.js application that acts as a proxy, actually forwarding a body to another server. I want to see the progress of this upload action. Now I am doing this:
BROWSER
-> UPLOAD TO NODE
-> UPLOAD TO 3rd PARTY SERVICE
-> RETURN 3rd PARTY RESPONSE TO BROWSER
If this is possible, I would log it to check in the console.log how much progress is done. But, would it also be possible to return a
res.send(progress)
in the mean time, while waiting for the upload to finish and send the client back the upload has succeeded?
BROWSER
-> UPLOAD TO NODE
-> UPLOAD TO 3rd PARTY SERVICE
-> RETURN progress <-
-> RETURN progress <-
...etc.
-> RETURN 3rd PARTY RESPONSE TO BROWSER
This is the upload code (pretty much straightforward).
var requestOptions = {
timeout: 120000,
url: url, //URL to hit
method: 'post',
headers: headers,
body: payload //Set the body as a string
};
request(requestOptions, function (error, response, body) {
if (error) {
res.send(error);
}
else {
//res.sendStatus(response.statusCode);
console.log("[RETURNING RESPONSE BODY]");
res.send(body);
}
});

Your question contains two parts. One is for getting the progress from request, which can be found here: Upload Progress — Request
The other part would be notifying the browser of the progress. The HTTP protocol does not let you send multiple responses, so using res.send(progress) is not possible.
However, you can keep sending partial responses until it finishes. Writing something to res without closing it is as simple as res.write("string"), but accessing the response can be harder: Accessing partial response using AJAX or WebSockets?
You also need a way to wrap the body (and also errors) from the backing server so that it can fit as the final partial response.
Another solution would be opening another WebSocket request to track the uploading/downloading process. socket.io is a good library for node for this purpose.

Solution for first part was in r.req.connectin.bytesWritten
var r = request(requestOptions, function (error, response, body) {
clearInterval(q);
...
});
var q = setInterval(function () {
console.log("Uploaded: " + r.req.connection.bytesWritten);
}, 250);

NodeJS - How to get cookies from server response

I want to use nodeJS as tool for website scrapping. I have already implemented a script which logs me in on the system and parse some data from the page.
The steps are defined like:
Open login page
Enter login data
Submit login form
Go to desired page
Grab and parse values from the page
Save data to file
Exit
Obviously, the problem is that every time my script has to login, and I want to eliminate that. I want to implement some kind of cookie management system, where I can save cookies to .txt file, and then during next request I can load cookies from file and send it in request headers.
This kind of cookie management system is not hard to implement, but the problem is how to access cookies in nodejs? The only way I found it is using request response object, where you can use something like this:
request.get({headers:requestHeaders,uri: user.getLoginUrl(),followRedirect: true,jar:jar,maxRedirects: 10,},function(err, res, body) {
if(err) {
console.log('GET request failed here is error');
console.log(res);
}
//Get cookies from response
var responseCookies = res.headers['set-cookie'];
var requestCookies='';
for(var i=0; i<responseCookies.length; i++){
var oneCookie = responseCookies[i];
oneCookie = oneCookie.split(';');
requestCookies= requestCookies + oneCookie[0]+';';
}
}
);
Now content of variable requestCookies can be saved to the .txt file and can loaded next time when script is executed, and this way you can avoid process of logging in user every time when script is executed.
Is this the right way, or there is a method which returns cookies?
NOTE: If you want to setup your request object to automatically resend received cookies on every subsequent request, use the following line during object creation:
var request = require("request");
request = request.defaults({jar: true});//Send cookies on every subsequent requests

In my case, i've used 'http'library like the following:
http.get(url, function(response) {
variable = response.headers['set-cookie'];
})

This function gets a specific cookie value from a server response (in Typescript):
function getResponseCookieValue(res: Response, param: string) {
const setCookieHeader = res.headers.get('Set-Cookie');
const parts = setCookieHeader?.match(new RegExp(`(^|, )${param}=([^;]+); `));
const value = parts ? parts[2] : undefined;
return value;
}

I use Axios personally.
axios.request(options).then(function (response) {
console.log(response.config.headers.Cookie)
}).catch(function (error) {
console.error(error)
});