We Keep Coding

Make browser submit additional HTTP-Header if click on hyperlink

Is there a way to make the webbrowser submit additional HTTP header if the user clicks on a link?
Background: In our environment every http-request has a unique ID on the server side. See https://serverfault.com/questions/797609/apache-x-request-id-like-in-heroku
If your web application receives a http-request, I would like to know which page was the page before. The http referrer is not enough, since the user could use several tabs in his browser.
I would like to avoid to put the ugly request-id into every GET request which gets send from the browser to the server. Up to now our URLs are nice.
My prefered solution would be some JavaScript magic which adds the request-id of the current page into the next http request.
Steps in detail:
browser access URL http://example.com/search
web server receives http request with request ID 123
web server sends content of the URL to the browser (a search page). The page includes the request ID 123 somewhere
the user searches for "foobar".
the web browser submits a http request to the server and includes the previous request id somehow.
web server receives second http request (ID 456) and can access the value of the first request (ID 123) somehow.
Web server can store the relation "123 --> 456" in a database for later analysis.
My goal is to track the relations "123 --> 456". Above solution is just a strategy to get to the goal. Other strategies are welcome.
We use the web framework django. But AFAIK this does matter in this context.
the user could use several tabs in his browser
I elaborate what that means for a matching solution. The sequence of requests which come from one user does not solve the issue.
One use with several tabs:
user looks at page A in tab1
user looks at page B in tab2
user follows a link on page A to page C
user follows a link on page C to page D
user follows a link on page B (tab2) to page E.
I want to know see two sequences:
A -> C -> D
And
B -> E

The only modern 'sane' option here is to use a ServiceWorker.
A ServiceWorker can intercept HTTP requests for a domain you control and decorate it with more headers.
A ServiceWorker works 'outside' of a browser tab, and if multiple tabs are open with the same website, the same serviceworker will be used for all of them.
A full tutorial on how to accomplish that is definitely too much for this answer box, but intercepting and doing stuff with HTTP requests is a big use-case, so off-site sources will usually have this as an example.
I would say that this is kind of a bad idea. If you think you need this, maybe you can handle this in a different way. A common way to do this might be using cookies instead.

We can modify request headers using:
.setRequestHeader() method of XMLHttpRequest() object (in same or allowed origins).
Editing the headers in browser console or using some complement (it is not practical).
Performing the request from the server side e.g using CURL, wget, or some library (client->serverProxy->url with custom headers ).
It is not possible (using javascript) to change the headers sent by browser in a request like because at least now, the http content negotiation is a browser's inner capability (except in part using XMLHttpRequest in same or allowed origins).
Then, in my opinion, as #Evert said you have two practical ways (a third in fact) to achieve your goal, performing a server proxy or using cookies. Here you have a very simple way using window.localStorage:
LocalStorage example
if (!localStorage.getItem("ids")) {//<-- the place in which we store the behavior
localStorage.setItem("ids", 'somevalue')
} else {
var ids = JSON.parse(localStorage.getItem("ids"));
ids.ids.push(id);//<-- we add some value
localStorage.setItem("ids", JSON.stringify(ids));
}
Full example here: https://jsfiddle.net/hy4rzob9/ press run several times and you'll see that we store each visit, of course, in your implementation you have to replace the random number for a unique identifier of each page.
LocalStorage example with several tabs
Taking into account the update, we could store the history using also document.referrer with localStorage with something like this:
var session = Math.random();
if(!localStorage.getItem("routes")){//<-- first time
var routes = {};
routes[session] = [document.location.href];
localStorage.setItem("routes", JSON.stringify(routes))
}else{
var routes = JSON.parse(localStorage.getItem("routes"));
if(!document.referrer){
routes[session] = [document.location.href];//<-- new root
}else{
for(let ses in routes){
if(routes[ses].includes(document.referrer)){
routes[ses].push(document.location.href);
}
}
}
localStorage.setItem("routes", JSON.stringify(routes))
}
var r = JSON.parse(localStorage.getItem("routes"));
console.log(r);
Full example here https://codesandbox.io/s/qk99o4vy7q, to emulate your example open this https://qk99o4vy7q.codesandbox.io/a.html (represents A) and open in a new tab https://qk99o4vy7q.codesandbox.io/b.html (represents B), navigate in both tabs and see the console. This example won't work if we share some referrer, because we can't differentiate between referrers if we attach nothing in the URL. A -> C -> D and B -> E will work, but A -> C -> D and B -> E -> A won't.
Ping example
There is other way, that is easy but has a limitation in browser compatibility, that is using ping attribute of <a> like this:
Link to track
ping Contains a space-separated list of URLs to which, when the
hyperlink is followed, POST requests with the body PING will be sent
by the browser (in the background). Typically used for tracking.
Open the console -> network, delete all, run the snippet and click in the link, if your browser supports it, you will see that the browser send a POST request to trackPing.py (I guess doesn't exist in SO), that post is void but you could track the environmental variables such as request.environ['REMOTE_ADDR'] or something.

First of all, sorry for my english.
Edit:
After reading your edit, I realised that my answer didn't fit at all, because of the tabs.
It is not possible to modify directly the way the browser makes a get request. Knowing that, your posibilities are:
Use GET parameters. I know you try to avoid this.
As #Evert said, use ServiceWorkers. It is the cleanest way to modify a request before it leaves the browser.
The last approach (an an easy one) is similar to #Emeeus's, but instead of using localStorage, whose values are shared between tabs, you should use sessionStorage, whose values are tab-independant. Also, instead of store the entire route, you should store just a random ID. This ID will work as the identification of the chain of requests for an specific tab. Then, once your webserver returns each Request-ID for example using <meta name="request_id" content="123" /> you just need to make a request via ajax to an specific tracking endpoint and store:
chain_id (stored in sessionStorage)
request_id (stored in head > meta)
timestamp (generated in webserver)
session_id (accesible from webserver). You can avoid this, but it is still useful for checking purposes.
The request to store the route is made after you page is loaded, instead of before. This approach is quite similar to how Analytics works.
// generate an unique code and store it in sessionStorage.
if (!sessionStorage.getItem('chain_id')) {
sessionStorage.setItem('chain_id', 'a7835e0a-3ee9-e981-...');
}
// Then, if you use JQuery:
$(document).ready(function() {
$.ajax({
type: "POST",
url: 'your/tracking/endpoint/',
data: {
'chain_id': sessionStorage.getItem('chain_id'),
'request_id': document.querySelector("meta[name='request_id']").getAttribute('content'),
}
});
});
Note: It is preferable to don't use JQuery to handle tracking requests neither wait until document is fully loaded. It is just an example.
And that's all. You have the relation between user-agent, the chain, the request and the timestamp of the request, so if you need to know what request was made before or after a given one, you just need to lookup in the database using the Chain-ID and the timestamp as filters.
The django model for your requests could be.
from django.db import models
from django.contrib.sessions.models import Session
class Request(models.Model):
session = models.ForeignKey(Session)
chain_id = models.Charfield(max_length=100)
request_id = models.WhatEverField...
request_url = models.URLField(max_length=200)
created = models.DateTimeField(auto_now_add=True)
I hope it helps.

I don't know if this will help, but I think maybe Ajax will do,
like set additional header inside onclick event listener, as for request id, if it's not something that sensitive then you could use cookie for the container, or maybe something much better ...

Securing PHP scripts - Shopify

Basically my question is similar to this one:
How to secure php scripts?
with one slight difference, the other side is Shopify.
Background info:
Shopify user bought some credits (Audible style), and wants to know how many he has available. He logs in into his account, and it says there that he has X credits.
That number comes from AJAX call to my server (I have full control), where there is a simple php script which checks the value in db (which is updated using webhooks from Shopify, each of which needs to be verified so they are secure, I think).
The script uses customers ID for a look up, and that value needs to be passed to the script somehow, and that allows someone external to just keep running it until he has all IDs and corresponding credits values.
So, my questions is, how do I stop that? How do I ensure that only authenticated users can check it, and only for their IDs.
There is plenty of info on Shopify docs about securing the connections the other way, i.e. to make sure only correct scripts have access to the Shopify db, but nothing about my problem.
As far as I know I only I only have access to JS on Shopify, which creates the problem, because everything I send to my server is visible to all.
Thanks
EDIT: I just read up on CSRF. I can easily implement checks for origin and headers, but these can be faked, right?
EDIT 2: I got around this problem by using metafields. So, instead of storing all that info on my server's db, I just use Customer Metafields to store the available credits. Webhooks are secure so that's brilliant. It still doesn't solve a problem with the next stage though. Customers will still need to be able to use their credits and get digital products, which are generated by my server. So I still need to verify the requests.
EDIT 3: Comment by #deceze and answer by #Jilu got me thinking. Yes, you are correct, I need to do that, but I don't have access to back-end on Shopify, so I cannot create session. However, what I could do (if I figure out how in js) is hash it. PHP login scripts operate on password_hash. That way you do not store a password in the db. Password get's verified again hash (or whatever you call) in the db, and it's either true or false. If true, you are logged in. So I could try to generate a token using a specific string (make it very long) and user id. Send it with the request, and using password_verify or what not, check it against the users. The one that pops positive is logged in user who requested the information. That is assuming I can hide the string in the Shopify...

Step1: Do a session login system.
Step2: Before the Ajax, generate a random token in your form or request page, put it into a input display none, send it with POST.
Verify each time if the token is set and is the same that you got.
You have now verified if the user is really logged in with session.
And you checked that he is from the right page.

You create a token out of shared secret (both Shopify and server have it), and client ID.
On Shopify:
{% assign my_secret_string = "ShopifyIsAwesome!" | hmac_sha256: "secret_key" %}
My encoded string is: {{ my_secret_string }}
On server:
We gonna be checking received hash value against values in our db, so we need to hash local client IDs using the same algo and key (that probably should be done on receiving the Customer Creation webhook).
Hash IDs using: http://php.net/manual/en/function.hash-hmac.php
$hashed_id = hash_hmac('sha256', '$client_id', 'secret_key');
Compare hash using: http://php.net/manual/en/function.hash-equals.php
$check = hash_equals($hashed_id, $received_id);
Then all that's requires is to loop through the db until you find a match. There may be quicker ways of doing it.

How to secure data or row ids that represents something in PHP

In my project a user can make a post (post a photo or some text). Technically I identify each post with a unique id in table posts where I store the user_id (owener of the post) and the id of the post is set to auto increment . Knowing the identification for each post I fetch all rows in the post table and put these post and relevant details( post_id, user_id, data, etc) inside an HTML. There are more things a user can do on that post, like discuss on the post, rate the post, and etc.
These things are done via an ajax post since I store the post_id on the HTML element attribute like ( data-p=52). sometimes I use the php base64_encode function to encrypt the post_id
Most in my application an event is acted on these post_id and user_id that is or are stored in the HTML custom attributes.
Now I am thinking of security issues and my question is : Is there a proper method or way I can hold these info in Javascript or a proper way I can encrypt these information about the post.

It is good you are thinking about the possible security vulnerabilities within your system. However, at the moment, from what I can tell, you are not quite worrying about the right thing. Data, like a user's ID, a post's ID, is not sensitive in itself. If you look at the URL of social networks, etc, it is very likely you will see user ID information, etc. What you need to think about, is how can I make sure that it doesn't matter this data is public? In other words, how can I prevent an attacker from using this data? Because this data on it's own, is just a bunch of numbers. Now, if there is a reason why these IDs are actually sensitive in your system, you should think about a slight structural rearrangement.
Take the common (or less so these days) SQL Injection technique. Where a attacker will input SQL code into a user input, and that input will then be concatenated/substituted right into a SQL query, therefore giving unwanted access to the database (see here http://www.w3schools.com/sql/sql_injection.asp). It does not matter the attacker knows the post ID, meaning oh no! He can delete the post he wants to, instead, it matters that he can delete any post he wants. So the problem is in the SQL vulnerability, not the knowing of the ID. And, to fix the SQL vulnerbabilty, you need to make sure that user input will disallow code-like characters. This is known as sanitization.
Of course I am not saying you shouldn't take care of what data is available to users. But the system itself needs to be robust.

The scenario you're worried about (that an attacker can use these IDs to send requests to your system to manipulate data in a way that you don't want) is independent of whether you expose your IDs to the client or not.
In some fashion, requests from your client will need to be handled by your server. If you have a feature where authors of posts can delete their own posts, hopefully you are validating delete requests that the user initiating the request is actually the owner of that post and not just blindly deleting data whenever someone asks your system to. Even if you introduced a surrogate key so as to prevent the actual primary key from leaking to the public, whatever route endpoint that is used to delete posts will still need to handle that data in a robust fashion that maintains the integrity of your data.
As I stated in a comment, really the only thing you should worry about are people performing maths on your IDs (such as plus and minus 1) in order to see if they can game your system and receive data back that maybe they shouldn't have. But even in that scenario whatever endpoint is responding to the request should validate that request before returning anything back. You can prevent this entirely by a) validating that the user requesting the data actually owns the data and b) by not using auto incremented integers as a primary key and instead relying on UUIDs as your primary keys.

Is it possible to prevent that a popup be opened directly as opposed to being opened by Javascript?

The application I'm working on relies on many popups. Those popups rely themselves on query strings. If someone can just type the url in the browser address bar, the page will throw an error as the query strings values are dynamically constructed.
function myFunction(id)
{
window.open("mypopup.aspx?id=" + id);
}
Is there a why to prevent the page from displaying if the requester of the page is not a Javascript? If someone type something like:
https://mycompanyname.com/path/mypopup.aspx
It shouldn't let the user do so. Or, at least check whether the requester is not javascript so I can display a message or redirect the user to a different page? Otherwise, without all those pieces of data needed to construct a request, the page will throw an exception.
Thanks for helping.

Validate the query string directly in myPopup.aspx, if something is missing just redirect or display a message.
Use the Request.QueryString collection to validate in myPopup.aspx.
There is no easy way to validate if the request came from javascript as far as I know. You could try creating a token to validate that the sender is the one you expect, but if you only need to validate the parameters, no need to worry about who is sending the request.

The page cannot differentiate how it was requested, if both requests come from a same browser.
However, you can include in query string to differentiate them.
For example,
window.open("mypopup.aspx?request=javascript&id=" + id);
If a user intentionally type in https://mycompanyname.com/path/mypopup.aspx?request=javascript, so be it. I won't worry about it.

Popups are browser windows too. So it will be tricky to check if the window requesting the page is normal window or popup.
You should restrict the users to see on what url the popup is being opened you can hide the address bar. So user can not copy or know the what's in the url.
window.open('/pageaddress.html','winname','directories=no,titlebar=no,toolbar=no,location=no,status=no,menubar=no,scrollbars=no,resizable=no,width=400,height=350');
Setup a token based validation. Make request to server(Ajax request) to get a random token(with one time validation mechanism and expire it), You can send the token in the query string and validate it on server if it's same issued token. Identify if the requested page have valid token(popup) otherwise deny the request or show error message. Think of how captcha works, you just need to do it programmatically.
Though it's also not the best solution as token information can be sniffed through network traffic tracker tools like fiddler but it's will work to prevent manual requests.

Get or Pass Sensitive Data via AJAX in PHP

I want to update a row of data in a database using Ajax and PHP; however, I'm struggling with the following issue: the field in the database to update (henceforth the id) is dependent on the page the ajax request is sent from.
I need to get this id to my PHP script that Ajax calls, however:
I don't want to set the id in a data attribute or hidden input on the page because these can both be manipulated by a malicious user.
Similarly, identifying the id using the referring URL is also prone to spoofing as $_SERVER isn't secure.
I can't set the id in a SESSION variable (or COOKIES) because the user could have multiple pages open and the SESSION would only hold the last page id that was opened.
The only solution I can think is to create a map of random tokens to id's in a table in the db and pass that in a SESSION variable (as per #3 above), then check the table for the token and grab the respective id that way. Seems somewhat convoluted though.
Are there any other options or thoughts? Thanks.

This is a problem related to OWASP Top10 A7 (Missing Function Level Access Control).
There might be no issue with putting your ID on the page so the page can send it back - you just need to validate that the actual save request is permitted for the user.
Just think, regardless of whether you put the ID on the page or not, the page does know the base url for performing the action, so they could go ahead and guess IDs anyway.

Simplify your logic. Pass some sort of indicator of what type of id is in use from the client to the server.
If you create overly complex application logic to address a security concern you will probably have more problems with your code than improvements in security.
Use SSL/HTTPS and a WAF (web application firewall - like mod_security).