We Keep Coding

How can i send data through an API to other servers only using Nodejs and Express?

I'm new to NodeJS and I am currently working on getting an API working. Currently running is Express for that purpose and i really would like to stick to express to solve it.
My goal is to let other people send me their data through links (Example would be: http://localhost:1000/api/?product=test) so i can just grab them with a simple 'var productname = req.param('product'); That part works just fine.
But i would like to simply call a method to send data from my server, meaning i would like to trigger sending the data with a function and then send the data as a link to another server. (Example would be to http://google.com/search?q=test)
I can't seem to get it to work even if i directly work with the documentation from express: https://nodejs.org/api/http.html#http_http_get_url_options_callback
Could anyone point me in the right direction?
If i try the code snippet below, I'm not even getting a console.log.
My current code attempt is:
// testing purpose to call the method and get a console log
sendServerUpdates('chair');
function sendServerUpdates(product){
url = 'google.com/';
app.get(url + 'search', (res) => {
const {statusCode} = res;
const contentType = res.headers['content-type'];
let error;
if (statusCode !== 200) {
error = new Error('Request Failed.\n' + `Status Code:
${statusCode}`);
} else if (!/^application\/json/.test(contentType)) {
error = new Error('Invalid content-type.\n' + `Expected
application/json but received ${contentType}`);
}
if (error) {
console.error(error.message);
// Consume response data to free up memory
res.resume();
return;
}
// Information for me that the system is sending a message
console.log('sending update');
// sending (if its working) the parameter product
res.status(200).send(product);
})
}
}

Using JavaScript, is it possible to capture the body payload from an outgoing fetch request?

so I want to add some functionality to an already existing site, this is to make my life easier. One of the things I need that I can't seem to figure out is: how to capture the body payload data a specific outgoing "POST" request. I found the code to do it before but didn't save it and I been searching for that code for 2 days to no avail.
So here is an example of the request the site is making to server.
fetch("https://my.site/api/req", {"credentials":"include","headers":{"accept":"*/*","content-type":"application/json"},"referrerPolicy":"no-referrer-when-downgrade","body":"{\"symbol\":\"mySYM\",\"results\":[{\"data\":{\"id\":\"dataID\"},\"result\":\"signature\"}]}","method":"POST","mode":"cors"});
and the part I need to catch is the "body" portion and then unescape it so it looks like this.
{"symbol":"mySYM","results":[{"data":{"id":"dataID"},"result":"signature"}]}
Also, if possible I would like to have it only catch data when the method = POST and requests going to a specific URL, so it will catch /api/req/ and not pay attention to other URL's and/or when the method is = GET, HEAD.
Currently, I manually get the data from the request using dev tools and clicking on the correct request then scrolling down to find the POST data.
In case you need to know the reason for this. The server signs the data through the websocket connection and I am essentially trying to capture that signature to be able to replay it. I am not trying to catch the websocket data as its incomplete for my needs I need to catch the whole outgoing request body data.
Thanks in advance.

Chosen Solution:
Thanks #thirtydot for your responses. Note that my specific situation involved only fetch requests so that is the reason I went with this route. With your response, a bit of more of my own research, and the help of this post I came up with this solution. Since I don't really care to see the responses (I have other functions taking care of the responses which are important to me.).
const constantMock = window.fetch;
window.fetch = function() {
if (arguments[0] === '/api/req' && arguments[1].method === 'post'){
bodyResults(arguments[1].body)
}
return constantMock.apply(this, arguments)
}
function bodyResults(reqBody){
console.log(reqBody)
}
which put the following in console (Exactly as I wanted).
{"symbol":"NEON","results":[{"data":{"expires_at":"1561273300","id":"2469c8dd"},"signature":"6d712b9fbb22469c8dd240be13a2c261c7af0dfbe3328469eeadbf6cda00475c"}]}
except now I can return this data through that function and continue to run the rest of my script fully automated.
Extra Solution:
In case there are others struggling with similar issues and care to catch the responses of those fetch requests I could have alternatively used:
const constMock = window.fetch;
window.fetch = function() {
if (arguments[0] === '/api/req' && arguments[1].method === 'post'){
bodyResults(arguments[1].body)
}
return new Promise((resolve, reject) => {
constantMock.apply(this, arguments)
.then((response) => {
if(response.url.indexOf("/me") > -1 && response.type != "cors"){
console.log(response);
// do something for specificconditions
}
resolve(response);
})
.catch((error) => {
reject(response);
})
});
}
function bodyResults(reqBody){
console.log(reqBody)
}
Possible XHR Solution
NOTE: this one is untested! An alternative Solution for XHR requests could be done similarly using something along the lines of:
(function(open) {
XMLHttpRequest.prototype.open = function(method, url, async, user, pass) {
alert('Intercept');
open.call(this, method, url+".ua", async, user, pass);
};
})(XMLHttpRequest.prototype.open);
Hope this helps!

So I came across this today and I'm not quite sure if its exactly what you've been looking for over 2 years ago, but solved my problem and I thought I should share it if others needed.
I'm currently using a marketing automation tool which is quite limiting when it comes to landing pages, but I wanted the client to be able to update the content whenever needed and still have access to custom functionality, so I needed the payload which was being sent by the form submission.
Here is what I used to get the form submission payload:
(function() {
var origOpen = XMLHttpRequest.prototype.send;
XMLHttpRequest.prototype.send = function() {
console.log('request started!');
console.log(arguments[0]);
this.addEventListener('load', function() {
console.log('request completed!');
console.log(this.status);
});
origOpen.apply(this, arguments);
};
})();
The arguments[0] piece is actually the JSON sent as the payload, and the status code is the response (200), stating the request was successfull.
I partially used code from this other response here: https://stackoverflow.com/a/27363569/1576797

Node (Express) - Trying to save a PDF from an API call

I've tried all sorts to get this to work. I'm trying to request a PDF from an API on node, then send this back to the client who called it to begin with.
For the minute I just want to successfully save and view the PDF on the node server.
The issue is the PDF file is always empty when I open it (Even though it has a size of 30kb).
The basic flow is like this (removed a few bits, but the below code works and returns me the PDF fine)
// We pass through session ID's, request dates etc through in body
app.post("/getPayslipURL", function(client_request, res) {
// create request, which will simply pass on the data to the database (In order to get the NI number we need for the pay API)
const NI_NUMBER_REQUEST = db_api.createRequestTemplate({
body: JSON.stringify(client_request.body)
});
// Create a chain of HTTPS Requests, Starting with our call to the DB
requestPromise(NI_NUMBER_REQUEST)
.then((db_response) => {
const PAY_API_OPTIONS = /*Code to generate options based on furhter DB info (Includes dates etc)*/
return requestPromise(PAY_API_OPTIONS); // Call pay API
})
.then((pay_pdf_data) => {
console.log(typeof pay_pdf_data); // It's a string
// At this point I can log pay_pdf_data, But if I try to save it to file it's always empty
// No matter how I encode it etc
fs.writeFile("./test.pdf", pay_pdf_data, 'binary', function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
})
.catch(err => `Error caught: ${console.log}`) // Catch any errors on our request chain
});
}
I've tried saving with / without the binary flag as suggested in other posts in both the file save aswell as within the requests itself. Also various types of decoding methods have been tried, I always get an empty PDF saved.
My return data looks like this (is much bigger, when saved as test.pdf I get a 30kb file as before mentioned)
%PDF-1.4
%����
1 0 obj
0 obj
<
I've found a post which says about piping the data all the way through, I have a feeling my pdf_data is corrupted when getting converted to a string
Any ideas how would I go about doing this with the current setup?
e/ RequestPromise is a library, could also use the standards request library if it's easier
https://github.com/request/request-promise -
https://github.com/request/request
Thanks!

Your code doesn't work because the underlying request library (used by request-promise) requires the option encoding set to null for binary data - see https://github.com/request/request#requestoptions-callback.
Here's how you download binary data using that module -
app.post("/getPayslipURL", function(client_request, res) {
const NI_NUMBER_REQUEST = db_api.createRequestTemplate({
body: JSON.stringify(client_request.body),
encoding: null
});
requestPromise(NI_NUMBER_REQUEST)
.then((db_response) => {
const PAY_API_OPTIONS = /*Code to generate options based on furhter DB info (Includes dates etc)*/
return requestPromise(PAY_API_OPTIONS); // Call pay API
})
.then((pay_pdf_data) => {
fs.writeFile("./test.pdf", pay_pdf_data, 'binary', (err) => {
if(err) {
return console.log(err);
}
console.log("The file was saved!");
});
})
.catch(err => `Error caught: ${console.log}`) // Catch any errors on our request chain
});
}

NodeJS - How to get cookies from server response

I want to use nodeJS as tool for website scrapping. I have already implemented a script which logs me in on the system and parse some data from the page.
The steps are defined like:
Open login page
Enter login data
Submit login form
Go to desired page
Grab and parse values from the page
Save data to file
Exit
Obviously, the problem is that every time my script has to login, and I want to eliminate that. I want to implement some kind of cookie management system, where I can save cookies to .txt file, and then during next request I can load cookies from file and send it in request headers.
This kind of cookie management system is not hard to implement, but the problem is how to access cookies in nodejs? The only way I found it is using request response object, where you can use something like this:
request.get({headers:requestHeaders,uri: user.getLoginUrl(),followRedirect: true,jar:jar,maxRedirects: 10,},function(err, res, body) {
if(err) {
console.log('GET request failed here is error');
console.log(res);
}
//Get cookies from response
var responseCookies = res.headers['set-cookie'];
var requestCookies='';
for(var i=0; i<responseCookies.length; i++){
var oneCookie = responseCookies[i];
oneCookie = oneCookie.split(';');
requestCookies= requestCookies + oneCookie[0]+';';
}
}
);
Now content of variable requestCookies can be saved to the .txt file and can loaded next time when script is executed, and this way you can avoid process of logging in user every time when script is executed.
Is this the right way, or there is a method which returns cookies?
NOTE: If you want to setup your request object to automatically resend received cookies on every subsequent request, use the following line during object creation:
var request = require("request");
request = request.defaults({jar: true});//Send cookies on every subsequent requests

In my case, i've used 'http'library like the following:
http.get(url, function(response) {
variable = response.headers['set-cookie'];
})

This function gets a specific cookie value from a server response (in Typescript):
function getResponseCookieValue(res: Response, param: string) {
const setCookieHeader = res.headers.get('Set-Cookie');
const parts = setCookieHeader?.match(new RegExp(`(^|, )${param}=([^;]+); `));
const value = parts ? parts[2] : undefined;
return value;
}

I use Axios personally.
axios.request(options).then(function (response) {
console.log(response.config.headers.Cookie)
}).catch(function (error) {
console.error(error)
});

How do I load the contents of a text file into a javascript variable?

I have a text file in the root of my web app http://localhost/foo.txt and I'd like to load it into a variable in javascript.. in groovy I would do this:
def fileContents = 'http://localhost/foo.txt'.toURL().text;
println fileContents;
How can I get a similar result in javascript?

XMLHttpRequest, i.e. AJAX, without the XML.
The precise manner you do this is dependent on what JavaScript framework you're using, but if we disregard interoperability issues, your code will look something like:
var client = new XMLHttpRequest();
client.open('GET', '/foo.txt');
client.onreadystatechange = function() {
alert(client.responseText);
}
client.send();
Normally speaking, though, XMLHttpRequest isn't available on all platforms, so some fudgery is done. Once again, your best bet is to use an AJAX framework like jQuery.
One extra consideration: this will only work as long as foo.txt is on the same domain. If it's on a different domain, same-origin security policies will prevent you from reading the result.

here is how I did it in jquery:
jQuery.get('http://localhost/foo.txt', function(data) {
alert(data);
});

Update 2019: Using Fetch:
fetch('http://localhost/foo.txt')
.then(response => response.text())
.then((data) => {
console.log(data)
})
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

If you only want a constant string from the text file, you could include it as JavaScript:
// This becomes the content of your foo.txt file
let text = `
My test text goes here!
`;
<script src="foo.txt"></script>
<script>
console.log(text);
</script>
The string loaded from the file becomes accessible to JavaScript after being loaded. The `(backtick) character begins and ends a template literal, allowing for both " and ' characters in your text block.
This approach works well when you're attempting to load a file locally, as Chrome will not allow AJAX on URLs with the file:// scheme.

Update 2020: Using Fetch with async/await
const response = await fetch('http://localhost/foo.txt');
const data = await response.text();
console.log(data);
Note that await can only be used in an async function. A longer example might be
async function loadFileAndPrintToConsole(url) {
try {
const response = await fetch(url);
const data = await response.text();
console.log(data);
} catch (err) {
console.error(err);
}
}
loadFileAndPrintToConsole('https://threejsfundamentals.org/LICENSE');

This should work in almost all browsers:
var xhr=new XMLHttpRequest();
xhr.open("GET","https://12Me21.github.io/test.txt");
xhr.onload=function(){
console.log(xhr.responseText);
}
xhr.send();
Additionally, there's the new Fetch API:
fetch("https://12Me21.github.io/test.txt")
.then( response => response.text() )
.then( text => console.log(text) )

One thing to keep in mind is that Javascript runs on the client, and not on the server. You can't really "load a file" from the server in Javascript. What happens is that Javascript sends a request to the server, and the server sends back the contents of the requested file. How does Javascript receive the contents? That's what the callback function is for. In Edward's case, that is
client.onreadystatechange = function() {
and in danb's case, it is
function(data) {
This function is called whenever the data happen to arrive. The jQuery version implicitly uses Ajax, it just makes the coding easier by encapsulating that code in the library.

When working with jQuery, instead of using jQuery.get, e.g.
jQuery.get("foo.txt", undefined, function(data) {
alert(data);
}, "html").done(function() {
alert("second success");
}).fail(function(jqXHR, textStatus) {
alert(textStatus);
}).always(function() {
alert("finished");
});
you could use .load which gives you a much more condensed form:
$("#myelement").load("foo.txt");
.load gives you also the option to load partial pages which can come in handy, see api.jquery.com/load/.