I am looking for a tool that would run in a browser (any browser will do) and show me where each HTTP request originated (HTML source file and line, Javascript, or whatever else).
A bit of background. There is a third-party Web application that can be accessed either directly or through a content-modifying proxy. In the former case it works, in the latter it doesn't. My task is to figure out why the proxy breaks the app, and fix whatever problem there is (normally the proxy should only modifications that do not affect functionality).
I have narrowed it down to a single HTTP request. When accessed directly, the browser issues a GET to one particular address, say http://example.com/foobar.html. When accessed through the proxy, there's no such request. This foobar.html contains an important part of the application, so it won't function without it. Supposedly the proxy breaks some code that ought to issue this request. The problem is that I cannot find this code, and so cannot figure out what exactly is broken. There's nothing that looks remotely like foobar in the entire application.
The application in question is a jumble of obfuscated Javascript that generates other avascript and/or HTML that may contain more Javascript etc. Somewhere along the road it probably generates, piece by piece, some iframe src=... or whatever via document.write, and this chunk of HTML references the needed http://example.com/foobar.html.
So what I need is the ability to tell the browser: "See this address, http://example.com/foobar.html? Whenever there is a request to this address, stop and show me what are you doing!" Hopefully this will let me narrow down my search somewhat more.
I couldn't find such functionality in firebug or venkman. Am I missing something? Is there any other tool that would let me do it?
I see the Referer header of the request in question, but the referring file is very large and obfuscated. So far, I was not able to make anything meaningful out of it.
FireBug then the Network Tab...
Fiddler - it allows to view and search the HTTP sessions. Be sure to decode sessions when searching.
Related
I try to wrap my head around how to really secure ajax calls of any kind that are publicly available.
Let’s say the JavaScript on a public page (so no user authentication of any kind) contains an AJAX call to a PHP script (REST API or just a script, it doesn’t matter) that does a lot of heavy lifting. So any user can just look into the source code, find the AJAX call, rebuild and execute it, and execute it again a million times in a second and DDoS your site that way - not so great. At first I thought a HTTP_REFERER check could be helpful, but as any header field, also this is manipulable (just use a curl request) so the gain of security wouldn’t be too high.
The next approach was about a combination of using session ids, cookies, etc. to build some kind of access key for every page viewer and when someone exceeds the limit the AJAX call would run into an error. Sounds great so far, but just by cleaning the cookies, etc. everything will be reseted. So also no real solution. But, of course! Use the IP! Great idea! Users in public networks, that use only one IP for internet access will be totally happy, if one miscreant will block the service for all of them by abusing the call... not. So, also no great solution.
So, I’m really stuck here and can’t think of any great answer for my problem.
I also thought about API keys, or something alike. But that is an information that is also extractable from the JavaScript source. So how to prevent other servers using your service in a proxy kind of manner serving your data to their users? (e.g. you implemented the GMaps API in your website (or any other API) and someone uses your script accessing the API with your key)
tl;dr
Is there any good way to really secure your publicly viewable AJAX calls from abusing them for DDoSing your site, presenting your data on other sites, etc.
I think you're overthinking what AJAX is. When your site makes an ajax request, server side, it's the same as any other page request (even if some scripts are more process intensive). You need to protect your entire site, and not just specific scripts. If your server does not have any DDoS protection, it can be attacked through any page. Look into services like CloudFare
As #Sage Mentioned it is similar to normal http request. You can use normal authentication as the http headers/cookie information will be passed on to the server every time you make ajax call. For clear view you can look into developer console on browser. Its the same as exposing you website root url. Just make sure you have authentications checks for ajax calls too.
I am trying to search an outside url for content matching "title" and return the results to my HTML page in the background through Javascript. I have been using Javascript and not found any resources that resolve my query, maybe I'm asking wrong?
but I would basically search the document with :
var title = document.getElementsByName("title");
The hard part is connecting to the page and searching through the HTML source code.
TIA!
You can't generally get the content from an outside URL unless server specifically allows you to do so. But, you can do it from server side. You will be able to get the content of any URL from your server. Server must include an header in response with name access-control-allow-origin which contains patterns/name of your domain.
However, you can do it from server side anyway, unless you are blocked specifically by the server.
You will need to develop a solution in which you grab the content for your outside URL from your server. It can be anything like PHP, Node.js, C# etc. After receiving response from the external server, deliver it in response to the browser using AJAX or anything. Then you can play with it anyway you want using JavaScript or JQuery.
Important Note:
Make sure whatever you are trying to access in anyway, you are allowed to do so. If they (your outside URL) wants to share something with public, they must be providing some APIs or other solutions to allow you access to their content.
Research has led to to a solution, implementing a scraper. There are many in existence,scrapy for instance. Just a head's up for those with the same question.
After experimenting with scripts on facebook.com, I noticed that for a couple of minutes it didn't allow me to connect to facebook.com. This happened more than one time so I suspect that Facebook doesn't like to mess with their code using userscripts.
The Stack Overflow question Can a website know if I am running a userscript? answers the multiple ways a website can detect scripts.
Now I want to know how I can fool a website to not detect any scripts I may be running.
Is there a way to know what my userscript HTML changes trigger on the website?
For Firefox and Chrome there is no magic bullet to prevent detection, it is a process and an art...
The 3 principle ways that a site would detect a script are:
Out of sequence or too-frequent AJAX requests.
Changes to the page's code (HTML, global JS, or even CSS).
Unnatural mouse or keyboard action.
See "Can a website know if I am running a userscript?" for more information.
Some countermeasures:
For AJAX requests that your script makes:
Use Wireshark, or similar, to analyze the traffic of normal AJAX calls. Make sure that you send the same type of calls, in the same order.
Make sure that your requests show the same headers and data as the page's.
Do not send AJAX requests faster than the page does normally or faster than a user could reasonably trigger.
Consider adding random delays to your script's requests, so that they are not uniformly timed.
Beware of constantly changing fields in the page's requests. Be sure that you know how to generate the correct value; don't just cut and paste.
For changes that you make to the page's code:
The page can handle these either entirely locally, with it's own javascript, or it can also send back status or code fragments to the server.
Block and replace the page's javascript. Tools like NoScript, RequestPolicy, firewalls and proxies can be used to block the JS.
Then use Greasemonkey to add your edited version of the page's JS. Greasemonkey will run even if all of a page's JS is disabled.
If the page phones status home, and that status changes in response to your script's changes, Intercept and replace those messages. (This is not needed if you used the previous method.) You will need a custom firewall, proxy, or router to do this.
For events that you trigger (mouse clicks, filling out fields, etc.):
If the page checks for this, give it what it looks for. For example, instead of a click event, you may need a mouseover, mousedown, mouseup, mouseout sequence.
If that is not enough, then the countermeasures are the same as for changes made to the page's code (replace JS, intercept status messages).
Important!
In the rare event that you are battling a site that tries to thwart userscripts, it is like a war. The webmaster can adapt to what you are doing, and the environment can constantly change. So scripting for such a site is an ongoing process.
Scripts are executed on user's side - your side. Therefore site's server code cannot know if you run anything or not.
There's two things to note however:
Site can deliver some script to be run on your computer that will inspect its environment and will report back if necessary. Those are most often targeted to detect some specific script that site owner don't want to have in its environment.
Site author can inspect your requests and analyze if they don't match patterns that would match regular communication from site supplied forms or AJAX. Missing keys or keys not matching hardcoded order in query, headers that identify automation clients or missing browser headers and things like that. Hitting throttling limits if you do requests automatically much faster than human user should fits this category too.
Otherwise if you generate exactly the same responses and don't leave traces in globally accessible environment, it is impossible for site to tell if you're using userscript with either server or client side checks.
I'm a bit embarrassed here because I am trying to get content remotely, by using the client's browser and not the server. But I have specifications which make it look impossible to me, I literally spent all day on it with no success.
The data I need to fetch is on a distant server.
I don't own this server (I can't do any modification to it).
It's a string, and I need to get it and pass it to PHP.
It must be the client's (user browsing the website) browser that actually gets the data (it needs to be it's IP, and not the servers).
And, with the cross-domain policy I don't seem to be able to get around it. I already knew about it, still tried a simple Ajax query, which failed. Then I though 'why not use iFrames', but the same limitation seems to apply to them too. I then read about using YQL (http://developer.yahoo.com/yql/) but I noticed the server I was trying to reach blocked YQL's user-agent making it impossible to use this technique.
So, that's all I could think about or find. But I can't believe it's not possible to achieve such a thing, that doesn't even look hard...
Oh, and my Javascript knowledge is very basic, this mustn't help either.
This is one reason that the same-origin policy exists. You're trying to have your webpage access data on a different server, without the user knowing, and without having "permission" from the other server to do so.
Without establishing a two-way trust system (ie modifying the 'other' server), I believe this is not possible.
Even with new xhr and crossdomain support, two-way trust is still required for the communication to work.
You could consider a fat-client approach, or try #selbie suggestion and require manual user interaction.
The same origin policy prevents document or script loaded from one
origin from getting or setting properties of a document from a different
origin.
-- From http://www.mozilla.org/projects/security/components/same-origin.html
Now if you wish to do some hackery to get it... visit this site
Note: I have never tried any of the methods on the aforementioned site and cannot guarantee their success
I can only see a really ugly solution: iFrames. This article contains some good informations to start with.
You could do it with flash application:
flash with a crossdomain.xml file (won't help though since you don't control the other server)
On new browsers there is CORS - requires Access-Control-Allow-Origin header set on the server side.
You can also try to use JSONP (but I think that won't work since you don't own the other server).
I think you need to bite the bullet and find some other way to get the content (on the server side for example).
I want to have a "control panel" on a website, and when a button is pressed, I want it to run a command on the server (my computer). The panel is to run different python scripts I wrote (one script for each button), and I want to run the panel on my Mac, my iPod touch, and my wii. The best way I see for this is a website, since they all have browsers. Is there a javascript or something to run a command on my computer whenever the button is pressed?
EDIT: I heard AJAX might work for server-based things like this, but I have no idea how to do that. Is there like a 'system' block or something I can use?
Here are three options:
Have each button submit a form with the name of the script in a hidden field. The server will receive the form parameters and can then branch off to run the appropriate script.
Have each button hooked to it's own unique URL and use javascript on the button click to just set window.location to that new URL. Your server will receive that URL and can decide which script to run based on the URL. You could even just use a link on the web page with no javascript.
Use Ajax to issue a unique URL to your server. This is essentially the same (from the server's point of view) as the previous two options. The main difference is that the web browser doesn't change what URL it's pointing to. The ajax call just directs the server to do something and return some data which the host web page can then do whatever it wants with.
On the client side (the browser), you can do it with the simplest approach. Just an html form. javascript would make it nicer for validation and to do ajax calls so the page doesnt have to refresh. But your main focus is handling it on the server. You could receive the form request in the language of your choice. If you are already running python, you could write a super fast cgi python script. Look at the cgi module for python. You would need to put this into the apache server on osx if thats where you will host it.
Unfortunately, your question about exactly how to write it is beyond the scope of a simple answer. But google for how to write and html form, or look at maybe jquery to build a quick form that can make ajax calls easily.
Then search for how to use the python cgi module and receive POST requests.
Javascript is basically for doing work in the browser (usually to render something nice for the end user to look at). What you want (as others have said already) is a way to connect an HTML form action to an action on the webserver "back end". And this is exactly (as RobG has pointed out) what CGI is for. An alternative to CGI which is quite popular with Apache users is mod_python - the difference is basically whether the "back end" operation runs as a standalone process (CGI) or inside a webserver process (mod_python), but for most basic applications your server side scripts don't need to care. And if you're in a shared hosting environment you may not have a choice - ask your sysadmin (or read your hosting service docs) to learn how best to run CGI scripts in this case.
Caveats:
You will probably need fairly elevated webserver admin access & expertise in order to get everything set up the way you want. You will at least need to be able (both in the sense of permissions and technical understanding) to view your webserver logs, edit your webserver configs and bounce (restart) your http service.
Whatever "back end" operations you want done will be done with the permissions/privileges of the webserver, which may not be the same as the permissions/privileges of the user account which you normally use to perform these operations. There are various ways around this (using custom daemons and/or sudo operations), but you really need to have a clear understanding with the webserver sysadmin (if the webserver is exposed to the Big Bad Internet) about how this is going to work before you deploy anything, otherwise you run the very real risk (especially if you are a noob) of making it possible for hackers to exploit your "command gateway" to hack the webserver.
Of course if you're just doing all this for fun on your personal laptop (there is an OSX tag on the question, after all), then you are the webserver sysadmin, and you're free to hack away and happily shoot yourself in the foot repeatedly while learning everything you need to know along the way, which is fine as long as you're not on a network. In this case, you may find this tutorial to be useful.