Is there a way to make the webbrowser submit additional HTTP header if the user clicks on a link?
Background: In our environment every http-request has a unique ID on the server side. See https://serverfault.com/questions/797609/apache-x-request-id-like-in-heroku
If your web application receives a http-request, I would like to know which page was the page before. The http referrer is not enough, since the user could use several tabs in his browser.
I would like to avoid to put the ugly request-id into every GET request which gets send from the browser to the server. Up to now our URLs are nice.
My prefered solution would be some JavaScript magic which adds the request-id of the current page into the next http request.
Steps in detail:
browser access URL http://example.com/search
web server receives http request with request ID 123
web server sends content of the URL to the browser (a search page). The page includes the request ID 123 somewhere
the user searches for "foobar".
the web browser submits a http request to the server and includes the previous request id somehow.
web server receives second http request (ID 456) and can access the value of the first request (ID 123) somehow.
Web server can store the relation "123 --> 456" in a database for later analysis.
My goal is to track the relations "123 --> 456". Above solution is just a strategy to get to the goal. Other strategies are welcome.
We use the web framework django. But AFAIK this does matter in this context.
the user could use several tabs in his browser
I elaborate what that means for a matching solution. The sequence of requests which come from one user does not solve the issue.
One use with several tabs:
user looks at page A in tab1
user looks at page B in tab2
user follows a link on page A to page C
user follows a link on page C to page D
user follows a link on page B (tab2) to page E.
I want to know see two sequences:
A -> C -> D
And
B -> E
The only modern 'sane' option here is to use a ServiceWorker.
A ServiceWorker can intercept HTTP requests for a domain you control and decorate it with more headers.
A ServiceWorker works 'outside' of a browser tab, and if multiple tabs are open with the same website, the same serviceworker will be used for all of them.
A full tutorial on how to accomplish that is definitely too much for this answer box, but intercepting and doing stuff with HTTP requests is a big use-case, so off-site sources will usually have this as an example.
I would say that this is kind of a bad idea. If you think you need this, maybe you can handle this in a different way. A common way to do this might be using cookies instead.
We can modify request headers using:
.setRequestHeader() method of XMLHttpRequest() object (in same or allowed origins).
Editing the headers in browser console or using some complement (it is not practical).
Performing the request from the server side e.g using CURL, wget, or some library (client->serverProxy->url with custom headers ).
It is not possible (using javascript) to change the headers sent by browser in a request like because at least now, the http content negotiation is a browser's inner capability (except in part using XMLHttpRequest in same or allowed origins).
Then, in my opinion, as #Evert said you have two practical ways (a third in fact) to achieve your goal, performing a server proxy or using cookies. Here you have a very simple way using window.localStorage:
LocalStorage example
if (!localStorage.getItem("ids")) {//<-- the place in which we store the behavior
localStorage.setItem("ids", 'somevalue')
} else {
var ids = JSON.parse(localStorage.getItem("ids"));
ids.ids.push(id);//<-- we add some value
localStorage.setItem("ids", JSON.stringify(ids));
}
Full example here: https://jsfiddle.net/hy4rzob9/ press run several times and you'll see that we store each visit, of course, in your implementation you have to replace the random number for a unique identifier of each page.
LocalStorage example with several tabs
Taking into account the update, we could store the history using also document.referrer with localStorage with something like this:
var session = Math.random();
if(!localStorage.getItem("routes")){//<-- first time
var routes = {};
routes[session] = [document.location.href];
localStorage.setItem("routes", JSON.stringify(routes))
}else{
var routes = JSON.parse(localStorage.getItem("routes"));
if(!document.referrer){
routes[session] = [document.location.href];//<-- new root
}else{
for(let ses in routes){
if(routes[ses].includes(document.referrer)){
routes[ses].push(document.location.href);
}
}
}
localStorage.setItem("routes", JSON.stringify(routes))
}
var r = JSON.parse(localStorage.getItem("routes"));
console.log(r);
Full example here https://codesandbox.io/s/qk99o4vy7q, to emulate your example open this https://qk99o4vy7q.codesandbox.io/a.html (represents A) and open in a new tab https://qk99o4vy7q.codesandbox.io/b.html (represents B), navigate in both tabs and see the console. This example won't work if we share some referrer, because we can't differentiate between referrers if we attach nothing in the URL. A -> C -> D and B -> E will work, but A -> C -> D and B -> E -> A won't.
Ping example
There is other way, that is easy but has a limitation in browser compatibility, that is using ping attribute of <a> like this:
Link to track
ping Contains a space-separated list of URLs to which, when the
hyperlink is followed, POST requests with the body PING will be sent
by the browser (in the background). Typically used for tracking.
Open the console -> network, delete all, run the snippet and click in the link, if your browser supports it, you will see that the browser send a POST request to trackPing.py (I guess doesn't exist in SO), that post is void but you could track the environmental variables such as request.environ['REMOTE_ADDR'] or something.
First of all, sorry for my english.
Edit:
After reading your edit, I realised that my answer didn't fit at all, because of the tabs.
It is not possible to modify directly the way the browser makes a get request. Knowing that, your posibilities are:
Use GET parameters. I know you try to avoid this.
As #Evert said, use ServiceWorkers. It is the cleanest way to modify a request before it leaves the browser.
The last approach (an an easy one) is similar to #Emeeus's, but instead of using localStorage, whose values are shared between tabs, you should use sessionStorage, whose values are tab-independant. Also, instead of store the entire route, you should store just a random ID. This ID will work as the identification of the chain of requests for an specific tab. Then, once your webserver returns each Request-ID for example using <meta name="request_id" content="123" /> you just need to make a request via ajax to an specific tracking endpoint and store:
chain_id (stored in sessionStorage)
request_id (stored in head > meta)
timestamp (generated in webserver)
session_id (accesible from webserver). You can avoid this, but it is still useful for checking purposes.
The request to store the route is made after you page is loaded, instead of before. This approach is quite similar to how Analytics works.
// generate an unique code and store it in sessionStorage.
if (!sessionStorage.getItem('chain_id')) {
sessionStorage.setItem('chain_id', 'a7835e0a-3ee9-e981-...');
}
// Then, if you use JQuery:
$(document).ready(function() {
$.ajax({
type: "POST",
url: 'your/tracking/endpoint/',
data: {
'chain_id': sessionStorage.getItem('chain_id'),
'request_id': document.querySelector("meta[name='request_id']").getAttribute('content'),
}
});
});
Note: It is preferable to don't use JQuery to handle tracking requests neither wait until document is fully loaded. It is just an example.
And that's all. You have the relation between user-agent, the chain, the request and the timestamp of the request, so if you need to know what request was made before or after a given one, you just need to lookup in the database using the Chain-ID and the timestamp as filters.
The django model for your requests could be.
from django.db import models
from django.contrib.sessions.models import Session
class Request(models.Model):
session = models.ForeignKey(Session)
chain_id = models.Charfield(max_length=100)
request_id = models.WhatEverField...
request_url = models.URLField(max_length=200)
created = models.DateTimeField(auto_now_add=True)
I hope it helps.
I don't know if this will help, but I think maybe Ajax will do,
like set additional header inside onclick event listener, as for request id, if it's not something that sensitive then you could use cookie for the container, or maybe something much better ...
Related
everyone. I am making a website with t-shirts. I dynamically generate preview cards for products using a JSON file but I also need to generate content for an HTML file when clicking on the card. So, when I click on it, a new HTML page opens like product.html?product_id=id. I do not understand how to check for id or this part ?prodcut_id=id, and based on id it generates content for the page. Can anyone please link some guides or good solutions, I don't understand anything :(.
It sounds like you want the user's browser to ask the server to load a particular page based on the value of a variable called product_id.
The way a browser talks to a server is an HTTP Request, about which you can learn all the basics on javascipt.info and/or MDN.
The ?product_id=id is called the 'query' part of the URL, about which you can learn more on MDN and Wikipedia.
A request that gets a page with this kind of URL from the server is usually a GET request, which is simpler and requires less security than the more common and versatile POST request type.
You may notice some of the resources talking about AJAX requests (which are used to update part of the current page without reloading the whole thing), but you won't need to worry about this since you're just trying to have the browser navigate to a new page.
Your server needs to have some code to handle any such requests, basically saying:
"If anybody sends an HTTP GET request here, look at the value of the product_id variable and compare it to my available HTML files. If there's a match, send a response with the matching file, and if there's no match, send a page that says 'Error 404'."
That's the quick overview anyway. The resources will tell you much more about the details.
There are some solutions, how you can get the parameters from the url:
Get ID from URL with jQuery
It would also makes sense to understand what is a REST Api and how to build a own one, because i think you dont have a backend at the moment.
Here some refs:
https://www.conceptatech.com/blog/difference-front-end-back-end-development
https://www.tutorialspoint.com/nodejs/nodejs_restful_api.htm
I'm new in BackboneJs and I need some help to pass parameters in URL using GET with Jersey.
In fact, I have table of documents like the following :
When I click on a PDF picto, it opens the PDF document using the documentID.
Front side (BackboneJs)
getDatas: function(e){
var clickedEl = $(e.currentTarget);
var id = clickedEl.attr("id");
window.open(window.myApp.baseUrlService.defaultUrl + '/getDocument.json?an200_idDocument=' + id);
},
So here I get the documentId and I use it in the backend to open the document.
Back side (JAVA)
#GET
#Path("/getDocument.json")
public Response getDocumentById(#QueryParam("an200_idDocument") String idDocument){
// my code here
}
It works perfectly but my problem is when I open the document, I can see the documentId which is not secure. I tried with #POST but it didn't work.
So, there is any other solution ?
Anything in the browser should be considered compromised already.
Any encryption in JS is pointless since the data is already available to the malicious party.
So how to hide the document id?
The data is not sensitive
Generally, ids are not sensitive information. Since you're already sending them to the frontend for the user to be able to ask for a specific document, they're already easily available.
Consider not worrying about these ids.
Use GUID
If the ids are currently a guessable sequence (1, 2, 3, etc.) and the user is not logged in meaning that the URL is publicly accessible, consider storing a GUID in the database for each document and use it to identify them.
/getDocument.json?an200_idDocument=123e4567-e89b-12d3-a456-426655440000
It's far less likely for someone to guess a GUID.
Make these chances close to none by namespacing with a category for example, or a user ID if the user is logged in the app.
/getDocument.json?cat=bills&an200_idDocument=123e4567-e89b-12d3-a456-426655440000
Check the permissions by object
If the user is logged in, check if he has access to the document from the backend before returning a response. If he does not have access, return a 403 Forbidden response.
Use short-lived tokens
If the document should be accessible only once for a user, consider short-lived tokens which are unique strings associated to the user or the document and they expires after X amount of times or after X document requests.
Sadly, there's no possibility to completely hide the parameters of an URL using GET, that is in fact one of the main point between GET and POST. Unless you encrypt those datas in order to hide them using GET or use a POST method, you'll have no other solution.
More details between them there : https://www.w3schools.com/tags/ref_httpmethods.asp
What would be the "best" approach to dealing with forms which have to work without and with JavaScript enabled?
Would it be better to create different routes for each, like
AJAX request: route "API/contact" and return res.send("message")
without JavaScript: route "contact"and return a redirect with a query param of "message"
Or in one route and detect xhr and render it depending on this?
Or is there a better way of dealing with the problem of taking the user to the res.send("") when the JavaScript isn't enabled to give the user feedback on the submit?
To clarify:
I have a site which is working with AJAX requests for its forms to avoid full page loads. It lacks the fallback when JavaScript is not enabled and thus when a user clicks submit on a form, he receives the data from the post back with res.send and it replaces the whole page, instead of the desired effect which would be to just update a label with the "success/fail" message. The question then remains as above which would be the neat way of dealing with this?
Probably the best thing to do would be to check the X-Requested-With header and check that it contains XMLHttpRequest (but this might get deprecated as the new fetch API will slowly come into browser.
Based on that value, you might want to return a JSON payload, or eventually trigger a server side rendering, therefore returning an HTML page ready-to-be-consumed.
As an alternative, you can return a redirect response with a particular query string value; once the page is loaded, you will check for that value (using qs for example, or deparam in jquery and manipulate the client side accordingly.
Your server routes have nothing to do with client-side javascript. You don't need javascript to receive a "res.send" message.
Imagine the next scenario: a user wants to register to a webpage and fills a form. While he is filling the form, jQuery keeps checking through a regular expression if fields are valid, etc...
Taking the email as the primary key which the user will use after registering to login, the email field needs to be checked with Ajax to let the user know if that email is registered or not. I want to check it with Ajax to avoid sending the full form and emptying it, refreshing page, etc...
So, when the user has ended filling the email field, the Ajax request is sent to the server, something like the next link:
example.com/check.php?email=abcdefg#gmail.com
When check.php receives the email, it asks the database if it exists or not and returns a message like: User already exists if user exists or null if user does not exist.
The question is: if someone digs through my .js and finds out links similar to that, they could use that link to send a large number of requests to find out if those random emails exist. This could lead to heavy use of the database or in the worst cases even crashing and private information leaks.
Someone could do a huge for loop to check emails like:
//Getting the response of the next links
example.com/check.php?email=aaaaaaa#gmail.com // Returns null
example.com/check.php?email=aaaaaab#gmail.com // Returns null
example.com/check.php?email=aaaaaac#gmail.com // Returns null
example.com/check.php?email=aaaaaad#gmail.com // Returns User already exists
------------------------------------------------------------------------------------------------------------------------------------
Since i last accepted the answer, i kept investigating this and found the solution to avoid this behaviour. The following code is for JAVA but the logic can be applied to any other server-side language.
Before doing ANY ajax request to the server, I request a token to the server. This token looks like this fmf5p81m6e56n4va3nkfu2ns8n it is made by a simple method, it can, however, be more complex, but this is good to go.
public String getToken() throws UnsupportedEncodingException {
return new BigInteger(130, new SecureRandom()).toString(32);
}
When requesting the token, the server does not only return the token, but also a small script that in case someone uses browser to inspect element (and browser navbar) and such the script will run and the token will be cleared. Servlet returns something like this:
_html += "<head>"
+ "<script> "
+ "window.onload=function(){\n"
+ " document.body.innerHTML = \"\";\n"
+ " }"
+ "window.location.href='http://mywebsite.com' "
+ "</script>"
+ "</head>"
+ "<body>"
+ "[" + token+ "]"
+ "</body>"
+ "</html>";
First empties the body then navigates back to wherever we want. javascript/jquery will however, catch the entire content as string, then I simply extract the string between [ and ]. This token is only available for the next request, so every AJAX request will have its unique token. On the 2nd reques the token just used is deleted.
After I get the token I append it as parameter to whatever link i request, something like this:
ajaxRequestObjet = $.ajax({
url: "http://localhost:8084/mywebsite.com/servlet", //<-- local tomcat server
method: "POST",
data: "type=AJAX&page=some-article&token=fmf5p81m6e56n4va3nkfu2ns8n"
});
This method works fine against someone who inspects the website manually and try to use the links, but what about java/php/IIS servers that do this automaticly?
For this ask for header! Something like this:
boolean isAjax = "XMLHttpRequest".equals(request.getHeader("X-Requested-With"));
It will be true only and only if XMLHttpRequest exists....
There is one last thing to keep in mind. Make sure 'Access-Control-Allow-Origin' header is NOT present in your app to make sure that any javascript NOT in your server wont get the server resources. If this header does not exist, chrome will return this:
XMLHttpRequest cannot load http://localhost:8084/mywebsite.com/servlet. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost' is therefore not allowed access.
Java server was in tomcat and I had another apache for this tests, this is the small html present in apache which gave the error above:
<html>
<head>
<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
<script>
ajaxRequestObjet = $.ajax({
url: "http://localhost:8084/mywebsite.com/servlet",
method: "POST",
data: "type=AJAX&page=Token"
});
ajaxRequestObjet.done(function (msg) {
alert(msg);
});
</script>
</head>
<body>
</body>
</html>
While you can not control this 100%... there are a few options..
Try using the same methods that people use with Captcha scripts..
Basically when the user loads the form / page.. You generate a random string/id in their PHP session and store it.. When they send the ajax requests, have your ajax check also append the string/id and require it before allowing a check to perform else return a header of 500 or something..
Using this approach with sessions, you could set a allowed limit of checks (say 5) and once the user has tried more than 5 checks, They are required to reload the page or perform a human check (eg Captcha).. Then it resets their count.. Even allow a total of say 30 within 1 hour / per IP or something.
Also use smart events to trigger when the ajax check is done, eg field/tab change or on a button press.. Or when a valid email is detected.. but say .com.au would trigger twice.
Basically this way, even if someone sniffed your JS files and tried to automate the email checker.. It would require them finding a way to append the string/id that you generate and also limit their amount of requests performed.
Beyond this, there is not to much more you can do easily.. But there are still a few other idea's.
Most of them would work around using a PHP session / cookie.. Say for example if they check and find 3 email addresses.. Then again you set that as a limit and force them to require a manual submission or something.
See how the above suggestion goes for you, any questions do feel free to ask. But may take me a day or two to reply as weekend.. Also research how Captcha scripts work as plenty of source code for them.. As they work on the same idea.
Time Delays will simply look bad / make your site appear slow / bug the user with waiting for a response.
You need to limit the amount of look up's per session / ip address.. Otherwise there is always a way to get past these checks.. Basically once they hit a limit.. Force the user/ip/session to wait a few minutes/hours and verify them with a Captcha script so it can not be scripted...
Javascript Security / Hiding The Source
While you can not do this truly, you can do certain things generate the JS using a PHP page with a JS header.. so <script src='myjscode.php'></script> and this allows PHP to check for a valid session.. So stops external requests to an extent.. But this is mostly useful for allowing JS to be only available behind a membership/login..
Multiple Checks / If Possible In This Case
Depending on your approach, is this for a user to check if they already have an account? If so.. you could combine the email check with something like their name/country/age/dob ... So they would need to select two or three correct matching values before being able to get a check/response from the ajax call?
Maybe not in your case, but just thought would add this as well.
The JavaScript code on your website is executed on the computer of the user, so there is no way you could stop him from digging through your code. Even if you use a code obfuscator (for example, https://www.javascriptobfuscator.com/), the hacker could debug your application and record all requests send to the server.
Everything security-relevant has to happen on the server. You could limit the amount of requests from a specific IP address.
You could protect against brute force attacks with something similar to CSRF tokens:
Assign a server-generated ID to every client session. Each request to check.php should include this ID.
check.php should reject requests that do not include an ID, or include an ID that the server did not generate (to prevent attacks with spoofed IDs). It should also rate limit on ID - if a given ID has made a request in (say) the last second, or a given ID makes more than n requests in a 10 second interval, it should return an error response. This protects against requests from a single session arriving from several IP addresses.
You should also rate limit by IP address to prevent brute-forcing by opening a large number of web application sessions.
There isn't much you can do to prevent an attacker looking up a single, or small number, of specific email addresses - it's an inherent risk with this type of validation.
One approach to resolve this problem could be this:
Suppose you have ajax request calling your server to receive a response from a particular user or client. You can have a table in your database where you provide a unique token for every user or hash value that can be checked every time user makes an ajax request to the server. If the token value matches the user request value than he is a genuine user. You can also record his number of request on the table to ensure he is making legitimate requests. I acknowledge the fact that it may slow down your app performance, but it will be safe option to consider. Note: you need to retrieve the token on your HTML page to send it with ajax.
Please comment to know more. I have been using this approach and there is no problem until now.
Example:
This type of attack can be treated the same as any other brute force attack, where the only effective solution is to use a Captcha. But of course, Captchas are a detriment to UX, so you have to consider if the additional security is worth it, especially for an attack that is very unlikely to happen anyway. That said, you may want to use a Captcha on your registration form anyway, to prevent bots from creating accounts.
This sort of attack has a huge cost for little reward for the attacker. There are billions of possible email addresses to test for. It could only be worth going to great lengths such as this, if the site in question was particularly sensitive, such as some kind of adult site, where the attacker hopes to blackmail users that he finds.
CloudFlare
Not as good as a Captcha solution but the brute force attack might be detected and prevented by CloudFlare's DDoS system. Also, CF can force Tor users to solve a Captcha before accessing your site, which would prevent an attacker from using Tor as a vehicle for the attack.
IP Rate Limiting
Rate limiting on an IP basis has problems because if an attacker decided to undertake a task as huge as this, he will likely be using a Botnet or some other system of multiple machines to launch the attack.
Consider a large organisation such as a University, where all users share the public IP. One of the users launches an attack on your site, and you block his IP, and in the processes blocking everyone else. This countermeasure could actually be used to launch a DoS attack.
Session ID/CRSF Token
Definitely not a solution because the attacker needs to simply make a request to the page first, to obtain the token. It's an additional request to make but only an inconvenience for the attacker.
First of all: I'd URL-encode the mail-address. 'example.com/check.php?email=' . urlencode(abcdefg#gmail.com)
Ad your question: when check.php is called, you can
check, if the user's session and his IP have sent a request during the last seconds
if not, write the user's session, the user's IP plus the current timestamp to a helper-table plus to a cookie and hit your DB
if yes, block the request
But I'm afraid this won't help you from fraud because everyone can check your JavaScript and if someone want's to exploit this, he will find ways.
check.php should depending on the setup either only be accessible internally, or verify from where the connection is made. Take a look at this previous question- I hope it might be what you're looking for. how to verify the requesting server in php?
You could use a CSRF token and exit early from your script if you detect that no or an invalid CSRF token. Almost (if not all) PHP frameworks come with support for this.
Also check this question from the security community: https://security.stackexchange.com/questions/23371/csrf-protection-with-custom-headers-and-without-validating-token
Is there a way to hide the url in the address bar with Grails application. Now users of the web application can see and change the request parameter values from the address bar and they see the record id in the show page.
Is there a way in Javascript or Groovy (URL Mapping) or Grails (.gsp) or HTML or Tomcat (server.xml or conf.xml or in web.xml inside application in the webapps)
ex(http://www.example.com/hide/show /) i want to avoid this url and always see (http://www.example.com) or (http://www.example.com/hide/show) without the record id
Is there a way to prevent this?
No, most browsers doesn't let you hide the address field, even if you open a new window using window.open. This is a security feature, so that one site can't easily pretend to be another.
Your application should have security checks so that one user can't access data that only another user should see. Just hiding the URL would not be safe anyway, you can easily get around that using tools built into the browser, or readily available addons.
It's part of the restful URL pattern implemented by grails.
Your best bet to hide the URL would be using an iframe within the page you want the user to see in their address bar.
Not quite sure what you mean, but I would change the default root URL mapping in UrlMappings.groovy so it looks a bit like this:
static mappings = {
"/$controller/$action?/$id?"{
constraints {
// apply constraints here
}
}
//Change it here!!!!
"/"(controller: 'controllerName', action: 'actionName')
Where 'actionName' and 'controllerName' are what you want them to be - 'hide', 'show' in your example?
Than pass all parameters via a post instead of a get, just change the <g:form> method.
You will still obviously need to implement any security checking required in the controller as stated by other posters.
Thanks,
Jim.
You can probably handle this using a variation of Post/Redirect/Get:
http://en.wikipedia.org/wiki/Post/Redirect/Get
At our Grails site we have a lot of search fields. When a user clicked a pagination link all those search fields ended up in the URL which created ugly URL:s with a higher risk that users bookmarked those addresses which could mean future problems.
We solved this by saving not only all POST but also GET with parameters into the session, redirect to GET without parameters and append those again in the controller. This not only creates nice URL:s but also a memory so that if a user goes back to an earlier menu, then selected details within that menu are redisplayed.
For your specific request to hide the id in "show/42" you can probably handle that likewise or possibly configure Grails to use "show?id=42" instead, but we don't have that requirement so I haven't looked further into that issue. Good luck!
Forgot to mention: this won't add much to security since links will still contain ids, it will only clean up the address bar.
Here's some sample code that should work. If show?id=42 is called, it saves id=42 in the session, then redirects to just show and id=42 is added to params before further processing. It does what you want, but as commented it might not always be a wise thing to do.
def show = {
if (request.method == 'GET' && !request.queryString) {
if (session[controllerName]) {
params.putAll(session[controllerName])
// Add the typical code for show here...
}
} else {
session[controllerName] = extractParams(params)
redirect(action: 'show')
return
}