We Keep Coding

NodeJS - How to get cookies from server response

I want to use nodeJS as tool for website scrapping. I have already implemented a script which logs me in on the system and parse some data from the page.
The steps are defined like:
Open login page
Enter login data
Submit login form
Go to desired page
Grab and parse values from the page
Save data to file
Exit
Obviously, the problem is that every time my script has to login, and I want to eliminate that. I want to implement some kind of cookie management system, where I can save cookies to .txt file, and then during next request I can load cookies from file and send it in request headers.
This kind of cookie management system is not hard to implement, but the problem is how to access cookies in nodejs? The only way I found it is using request response object, where you can use something like this:
request.get({headers:requestHeaders,uri: user.getLoginUrl(),followRedirect: true,jar:jar,maxRedirects: 10,},function(err, res, body) {
if(err) {
console.log('GET request failed here is error');
console.log(res);
}
//Get cookies from response
var responseCookies = res.headers['set-cookie'];
var requestCookies='';
for(var i=0; i<responseCookies.length; i++){
var oneCookie = responseCookies[i];
oneCookie = oneCookie.split(';');
requestCookies= requestCookies + oneCookie[0]+';';
}
}
);
Now content of variable requestCookies can be saved to the .txt file and can loaded next time when script is executed, and this way you can avoid process of logging in user every time when script is executed.
Is this the right way, or there is a method which returns cookies?
NOTE: If you want to setup your request object to automatically resend received cookies on every subsequent request, use the following line during object creation:
var request = require("request");
request = request.defaults({jar: true});//Send cookies on every subsequent requests

In my case, i've used 'http'library like the following:
http.get(url, function(response) {
variable = response.headers['set-cookie'];
})

This function gets a specific cookie value from a server response (in Typescript):
function getResponseCookieValue(res: Response, param: string) {
const setCookieHeader = res.headers.get('Set-Cookie');
const parts = setCookieHeader?.match(new RegExp(`(^|, )${param}=([^;]+); `));
const value = parts ? parts[2] : undefined;
return value;
}

I use Axios personally.
axios.request(options).then(function (response) {
console.log(response.config.headers.Cookie)
}).catch(function (error) {
console.error(error)
});

How to receive Yaml content in an AJAX call?

In my application, there is an endpoint that sends me the raw contents of a Yaml file in response to an AJAX call. I want to display them as they are in UI. The console throws an obvious error, which is for invalid JSON. How would I do it?
Update:
This is the snippet used for reading the file and sending the response.
filename = __file__ # Select your file here.
wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type='text/plain')
response['Content-Length'] = os.path.getsize(filename)
return response
Is there a way I could form a dictionary there with the contents of the file and then send the response?

From server, use jsonify on the raw content, pack it and ship it over to client.
repacked_json = json.dumps(raw_yaml_data)
json_obj = json.loads(repacked_json)
return jsonify(result = json_obj)

Parse.com getting data from callback URL

I'm currently trying to use an API and for the API, the developer console of that app asks the developer to submit a callback URL. Whenever the user of the app does something, it submits a GET request to the callback URL and I can retrieve data from that request. The current url I am using is https://appId:javascript-key=myJavascriptKey#api.parse.com/1/functions/receiveInfo. How can I handle the data, a.k.a the GET parameters, from the GET request? I found an answer on Parse.com that says how to retrieve data from a POST request, but all it says is that data = request.body. Do I do the same for GET requests and if so what do I do after that? Is request.body a json value?
Parse.Cloud.define("receiveInfo", function(request,response){
var params = request.body;//is this right to get the GET parameters they send? if so what do I do next?
});

The documentation has your solution at: https://parse.com/docs/cloud_code_guide#functions
For GET requests you have to use the request.params object which has all your request parameters for a GET are there. POSTS are sent in the request body, GET in the request parameters.
It looks like you are trying to get the params you can use something similar to:
Parse.Cloud.define("myMethod", function(request, response) {
if(request.params.myparam == "moo") {
response.success("Cow!");
}
else {
response.error("Unknown type of animal");
}
});

POST data as json to a Servlet not working in Phantomjs

I am facing very strange issue here. I have a servlet running on my machine which renders my web page based on some input parameters.
Now, my screen capture with PhantomJS is not working if I try to send my data as a JSON object as a POST request type. For e.g. if I try:
Client side
var data = {"op": "get"};
page.open(address, 'POST', data, function (status) {
if (status !== 'success') {
console.log('[ERROR] :: Unable to fetch the address - ' + address + ", with data - " + data);
phantom.exit();
} else {
page.render(output);
}
console.log('processing...');
});
Server side
Now, on the server side, I am using Apache Velocity View templating so I have a single method which handles both get and post like :
public Template handleRequest(HttpServletRequest request, HttpServletResponse response,
Context context){
System.out.println(request.getParameter("op"));
//Always null
}
However, if I try sending my data from phantomjs as:
var data = "op=get&..."
It works
Also, at many places elsewhere in my code..I am doing Ajax POST requests to the same servlet and it works perfectly for all those request.
Would anybody explain why my servlet is not reading the JSON parameters passed from Phantomjs?

Servlets deal with simple request, so they only know how to parse (natively) HTTP parameters, either GET parameters from the URL, or POST parameters sent as application/x-www-form-urlencoded. Newer versions of the Servlet specification can also read multipart/form-data.
But there's nothing about JSON mentioned either in the Servlet or the HTTP specifications. So you must use a library that knows how to parse JSON and make the resulting object available in the Velocity context.

XHR in Chrome Extension with CI

I'm sending a POST from a chrome extension content script to a server I control. I setup the permissions in the manifest ok. Here is my XHR code. (I want to avoid jQuery for this). Its sending an empty responseText
var xhr = new XMLHttpRequest();
xhr.open("POST",'http://mysite.com/make',true);
xhr.onreadystatechange=function() {
if (xhr.readyState == 4) {
var res = JSON.parse(xhr.responseText);
console.log(res);
}
}
xhr.send({'textbox':data[0].user,'from':'extension'});
data[0].user is an object I got directly from the Twitter API
in my CI controller I have
$user = $this->input->get_post('textbox', TRUE);
$from = $this->input->get_post('from', TRUE);
$fullURL = 'http://www.google.com'; //example of a URL from code.
$json = $this->output->set_content_type('application/json');
$json->set_output(json_encode(array('URL' => $fullURL)));
The response text is empty
a jquery call on the other hand works fine
$.post("http://mysite.com/make", { 'textbox': data[0].user, 'from':'jquery' },
function(data) {
console.log(data);
});

Reason is simple, JQuery post method can accept JSON and then convert it to string and send to the server.
What you are trying to do is to directly send JSON here :
xhr.send({'textbox':data[0].user,'from':'extension'}) // Incorrect way
send method should either accept NULL or a string which is generally made up of QueryString Parameters like.
xhr.send("textbox="+ data[0].user + "&from=extension"); // Correct way
This will ensure that your data goes to the appropriate URL with textbox and from as post request parameters.
and queryString will be generated like textbox=username1234&from=extension in the packet's body unlike one goes in Get with the headers along side the URL.
jQuery's post method makes it simpler for you to format data you send using JSON and then internally it converts that to a queryString to send parameters.
You can't directly send Javascript object like that with an XHR object!
Also checkout this example:
http://beradrian.wordpress.com/2007/07/19/passing-post-parameters-with-ajax/