We Keep Coding

CORS fetch authentication using Browser's session cookie

I have a server that stores session cookies and you can log onto it using a page (foo.com/login.html) that runs in the browser. The browser then stores a session cookie for this domain.
Now I want another page (bar.com) upon initialization to make a GET request using JavaScript to the first page (foo.com/authenticate) which should check if a session cookie exists in the browser and validate it, if correct he should respond with the session's username (however this is retrieved from the cookie). Of course I cannot check in bar.com's JavaScript if there exists a session cookie for foo.com.
Trying to solve this I ran into a few problems, one of which is of course CORS. I managed to avoid this problem by placing a reverse proxy in front of foo.com that adds all required CORS headers to the response. besides adding the headers, the proxy only tunnels requests through (eg. rev-proxy.com/authenticate -> foo.com/authenticate)
Now when I call the handler through the rev proxy from just another browser window directly (eg. rev-proxy.com/authenticate), I get the correct response. The handler from foo.com's backend finds the session cookie, reads out the username and passes it back. BUT when I try to make the same call from JavaScript inside bar.com (fetch("rev-proxy.com/authenticate")), I receive null, meaning he did not find the cookie (note that the request itself has status 200, meaning it did reach the backend of foo.com).
I have the feeling I am missing a crucial point in how cookies are used by browsers but I cannot find any useful information on my specific problem since I believe it is a rather unusual one.

See the MDN documentation:
fetch won’t send cookies, unless you set the credentials init option. (Since Aug 25, 2017. The spec changed the default credentials policy to same-origin. Firefox changed since 61.0b13.)

I need a more detailed understanding of precisely how cookies work

I can build a full stack app using Ruby on Rails, JavaScript, React, HTML and CSS. Yet, I feel I don't understand completely how cookies actually work and what they are precisely. Below I write what I think they are, and ask that someone confirm or correct what is written.
An HTTP request contains an HTTP method, a path, the HTTP protocol version, headers, and a body.
An HTTP response contains the HTTP protocol version, a status code, a status message, headers, and a body.
Both are simply text (which means that they are simply sequences of encoded characters), but when this text is parsed it contains useful structure. Is there one single structure that an HTTP request is usually parsed into (an array, a hash)? What about an HTTP response?
Cookies represent some content associated with a specific header in an HTTP request, specifically the "Cookie" header.
When building an HTTP response, the server sets the 'Set-Cookie' header. This header needs the following information: a name for the cookie, a path, and the actual content of the cookie. The path is a description of the range of URLs for which this cookie should be sent from client to server.
Does the browser keep a list of cookies (ie, a list of elements that are each text of some sort), and it only sends the right ones to the right sites (say a google cookie to google.com)?
Let's say I visit site A and then site B and authenticate on both. Session management just adds a specific element in the cookies (perhaps a hash named Session inside another hash that corresponds to the totality of the cookie stored in Cookie), correct? How do sites alter my cookies? Do they append new information, do they ask my browser to append information?

A cookie is a string (with a specific format) that your browser stores. It can be set by a server when it sends a http-response, by the 'Set-Cookie' header. Each http-request that your browser sends that matches the cookie's path will contain that cookie in the 'Cookie' header.
The server cannot tell the browser to append data to the cookie. It can only get the current cookie value, add to it the new information, and then reset it.

Setting a cookie from a remote domain for local development

So I am trying to set up environment for local development to pull data from my dev server at dev.mydomain.com.
The tornado REST server serving data uses a cookie-based authentication.
To obtain the cookie I sent an AJAX post login request to the server (from the website at localhost), and the secure cookie comes back in a response. I can see that in the chrome console (network->cookies). It has the proper name, value, domain (dev.mydomain.com) and everything.
Yet, the cookie doesn't get set and the REST requests that follow fail. It is not cross-origin related. If I go to dev.mydomain.com and log in manually in another tab the cookie gets set correctly and all my subsequent requests sent from local domain work fine (since they grab the now-existent cookie).
All my requests contain this:
xhrFields: {
'withCredentials': true
}
And this is how my tornado server sets the cookie:
self.set_secure_cookie(
COOKIE_NAME, tornado.escape.url_escape(str(COOKIE_VALUE)),
expires_days=1, domain="dev.mydomain.com"
)
Any idea why the cookie doesn't get set if the login request comes from localhost?
I tried mapping 127.0.0.1 to foo.mydomain.com (for whatever that's worth) but this doesn't help.
Also, I cannot grab the cookie with javascript. Tried xhr.getResponseHeader('Set-Cookie');, yields null.

Somehow it makes sense to me that if you set the cookie for dev.mydomain.com that it does neither work for foo.mydomain.com nor for localhost.
What happens if you do something like this:
self.set_secure_cookie(
COOKIE_NAME, tornado.escape.url_escape(str(COOKIE_VALUE)),
expires_days=1, domain=".mydomain.com"
)
*.mydomain.com might work then.
EDIT:
Actually, I checked over and over again, and I can't find an example where people used the argument 'domain' for set_secure_cookie() but instead this argument exists for 'set_cookie()', as stated in the docs:
Additional keyword arguments are set on the Cookie.Morsel directly.
See http://docs.python.org/library/cookie.html#morsel-objects for
available attributes.
If you are sure about using secure cookies, you should first get sure to use a cookie secret in your application settings
class Main(web.Application):
def __init__(self):
settings = dict(
cookie_secret = "xxxx",
)
then try to set the secure cookie, without specifying the domain
self.set_secure_cookie(
COOKIE_NAME, tornado.escape.url_escape(str(COOKIE_VALUE)),
expires_days=1
)

Cookie not stored with Ajax call

I am going crazy with cookies and ajax call.
My configuration is simple. I run a website on 8282 port, (localhost.com:8282). My website calls some webservices on 8080 port (localhost.com:8080). Of course I add a line in my hosts file to avoid localhost trouble :
127.0.0.1 localhost.com
I try to set a cookie when the webservice is called with ajax. Here is my response header that I can see with Chrome debugger :
Set-Cookie:token=Custom eyJ0aW1lc3RhbXAiOiIxNDI0NzE5Mzc5ODY3IiwgImlkIjoiNTRlNzZkZGU2ZDk3ZGM1MjYxZjQzMzFlIiwgInNpZ25hdHVyZSI6Im5tZnFGeEEvYlc0TFJGNFJNb3dBZXJZOUw0aWw0aEorcFh1YUt5b3VFK0k9In0=;domain=.localhost.com;path=/;
The cookie is never stored by Chrome. However, when I use Rest client extension and I call the same webservice, the cookie is stored by Chrome ! So my cookie is well formed but is not stored with ajax call.

It's likely an issue with CORS (Cross Origin Resource Sharing, i.e the fact that the domain of the client and of the target of the AJAX call are not the same). For cookies to work well in a CORS configuration, you need to set the withCredentials flag to true. How to do so varies depending on you AJAX library (if you're using one).
See here: http://www.html5rocks.com/en/tutorials/cors/

In your close reponse of ajax you can set your cookie
document.cookie = "token=Custom eyJ0aW1lc3RhbXAiOiIxNDI0NzE5Mzc5ODY3IiwgImlkIjoiNTRlNzZkZGU2ZDk3ZGM1MjYxZjQzMzFlIiwgInNpZ25hdHVyZSI6Im5tZnFGeEEvYlc0TFJGNFJNb3dBZXJZOUw0aWw0aEorcFh1YUt5b3VFK0k9In0=;domain=.localhost.com;path=/";
Can an AJAX response set a cookie?

Why ajax is not setting HTTP_X_REQUESTED_WITH [duplicate]

All over the Internet, included even here at Stack Overflow, people state that a good way to check if a request is AJAX or not is to do the following:
if (strtolower($_SERVER['HTTP_X_REQUESTED_WITH']) == 'xmlhttprequest' ) {...}
However, I don't see $_SERVER['HTTP_X_REQUESTED_WITH'] in the official PHP documentation
And when I try to do the following:
echo $_SERVER['HTTP_X_REQUESTED_WITH'];
Nothing is outputted.
Am I doing something wrong? Because I'd really like to be able to use $_SERVER['HTTP_X_REQUESTED_WITH'] if it's available.

The variables in $_SERVER are not really part of PHP, which is why you won't find them in the PHP documentation. They are prepared by the Web server which passes them on to the scripting language.
As far as I know, the X-Requested-With is sent by the Ajax functions of most major Frameworks but not all (Dojo, for example, added it only two years ago: #5801). As such, and taking into considerations #bobince' comments, it's safe to say it's not generally a 100% reliable method to determine whether a request is an AJAX request or not.
The only 100% secure way is to send a pre-defined flag (e.g. a GET variable) along with the request and for the receiving page to check for the presence of that flag.

don't forget that you can easily spoof any header with cURL like so
curl_setopt($ch,CURLOPT_HTTPHEADER,array("X-Requested-With : XMLHttpRequest"));

$_SERVER keys that start with HTTP_ are generated from HTTP request headers. In this case, the X-Requested-With header.

This header is a standardization-in-progress from all of the AJAX libraries out there.
It won't be documented in the php documentation per-se, but rather in the different AJAX libraries that set this header. Common libraries do sent this header: jQuery, Mojo, Prototype, ...
Usually these library will set the header using
xhrobj.setRequestHeader("X-Requested-With", "XMLHttpRequest");

Here's a quick function with example usage:
function isXmlHttpRequest()
{
$header = isset($_SERVER['HTTP_X_REQUESTED_WITH']) ? $_SERVER['HTTP_X_REQUESTED_WITH'] : null;
return ($header === 'XMLHttpRequest');
}
// example - checking our active call
if(!isXmlHttpRequest())
{
echo 'Not an ajax request';
}
else
{
echo 'is an ajax request';
}

echo $_SERVER['HTTP_X_REQUESTED_WITH'];
What'd you expect from such a code? Assume you're running it directly from the browser, not using AJAX request. So, how come this header could be set?
Well the Answer to the Ultimate Question of Life, the Universe, and Everything - an HTTP sniffer! Get yourself one and forget of printing $_SERVER variable.
Firebug has one, or you may want to use Fiddler HTTP proxy or LiveHTTPHeaders Mozilla plugin. I'm bored to make links but it easily googled.
So, with HTTP sniffer you can be sure of any HTTP header ever.
Note that you can't prevent any "direct access" by using XHR, as every HTTP request to your server is already "direct".

You have to set it specifically in your ajax request object (that is if you are not using a framework like jQuery), but core Javascript; like so:
xhr.setRequestHeader("X-Requested-With", "XMLHttpRequest");
Where xhr is your request object.
Then, PHP will now receive and set it in the global variable $_SERVER like so:
$_SERVER['HTTP_X_REQUESTED_WITH']
Otherwise $_SERVER['HTTP_X_REQUESTED_WITH'] will always be null.
Note: In your javascript, Please make sure you set headers after the request is open. I mean after xhr.open() method.

You can also blame some browser bugs - see this question and its solution for Firefox
Firefox does not preserve custom headers during Ajax request redirect: an ASP.NET MVC solution
IE also having caching issue which is more serious then detection of request method.
You anyway needs to add cache busters to avoid caching, so why not use another flag to specify the ajax call - or more better you can use different URL like http://ajax.mysite.com/endpoint/sevice?params

I agree Pekka. There is no reliable native method between front side and back side that can auto-detect if a client is really calling an endpoint using AJAX.
For my own use, I have few main ways to check if a client is requesting one of my endpoint:
I can use HTTP_X_REQUESTED_WITH when I'm not in cross domain context.
instead of checking "X-requested-with", I'm checking $_SERVER['HTTP_ORIGIN'] (that is sent from AJAX request) intending to handle cross domain permissions. Most of time, the main reason why I'm checking if a request is an AJAX request, is especially because of cross domain permissions, using this PHP code: header('Access-Control-Allow-Origin: '.$_SERVER['HTTP_ORIGIN']); // If this "HTTP_ORIGIN" is in my white list
my APIs expect from the client to explicit, in few cases, the datatype (JSON, HTML etc.) into a GET or a POST var. For example, I check if $_REQUEST['ajax'] is not empty or equal to an expected value.

The best solution to make sure if an HTTP request is truly sent via AJAX is using SESSION checking , you send session_id in a get parameter and you check this session if it's allowed or not !

$headers = apache_request_headers();
$is_ajax = (isset($headers['X-Requested-With']) && $headers['X-Requested-With'] == 'XMLHttpRequest');