Tracking http requests sent by the browser using javascript/jquery - javascript

Using javascript or jquery, is there a way to track the http requests(including headers, parameters, etc.), sent by a webpage? What I want to achieve is something similar to the functionality of the 'network' tab of Google Chrome's developer console. All the solutions I found was either tracking Ajax requests or requests made using javascript(using XMLHttpRequest Object). This functionality should also be cross browser compatible.

You have three choices.
Make sure you know all the places where a request can get fired, and attach an event to it, say RequestFired. And bind the onRequestFired event in your JavaScript / jQuery code.
Go through the Network Developers document or each browser and based on the browser, execute it. This feature may not be available in older browsers like Internet Explorer 7 and 8.
Google Developers Doc
Firefox Network Information API
NetworkInformation.connection
If it is for a particular server, read the Server Log using a server side script and access it using an endpoint. You can use long polling method and fetch the contents of the log, may be this way:
// jQuery
$(document).ready(function () {
setInterval (function () {
$("#log").load("/path/to/endpoint.log");
}, 5000);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<h3>Logs</h3>
<div id="log"></div>

You can't track everything.
For example some of the calls in Xmlhttprequest are transparent (301 HTTP codes) and can't be handle by javascript client side.
see the XMLHTTrequest specs: http://www.w3.org/TR/XMLHttpRequest/#infrastructure-for-the-send-method
This among a few other reasons. if you want to track the requests of a "webpage" it's better to use the development tools of that browser or packet capturing.
On the userExperience side you can only do very limited things.

It can be done, if you are implementing a single page web-application with a framework like AngularJS. There you can do that using HTTP interceptors.
Other than that, you can only track Ajax requests, but not JavaScript requests.

Related

Is it possible to prevent an unknown javascript script from being able to make a network call in a browser?

If I'm loading arbitrary external javascript code in a browser setting, is it possible to ensure it can't make the browser run make any ajax calls or network requests?
Can you prevent any resource calls? - No. (haven't explored the 'extension' route though)
Since even an <img src='any valid url'> creates a resource request which your code cannot prevent.
Can you prevent ajax calls? - Yes, to an extent.
Assuming that you want to ensure that any third party libraries shouldn't make any arbitrary ajax calls (cross domain), you will simply ensure that you don't enable CORS in your web server.
Your own application code can make ajax calls since they are in your domain only. However, you can filter those calls on server to check for specific properties like purpose, credentials, etc
It may be worth exploring google caja (haven't tried that myself)

Chrome Extension webRequest synchronous behavior for async calls

I've created a test web server that I'm using to act as a 'web filter' of sorts. I'm trying to create an extension that uses the webRequest API to make sure that my web server allows or blocks all incoming URL's.
To do this, I'm making an AJAX call from within webRequest to my web server and I'd like to use the response to determine whether to block or allow the specified URL. The problem is, the webRequest method is async, and AJAX calls are async, so I can't wait reliably wait for a response from my server.
I also can't store all blocked / allowed URL's in localStorage, because there could potentially be hundreds of thousands. I've tried using jQuery's async: false property in it's ajax implementation, but that makes the browser almost completely unusable when hundreds of requests are happening at the same time. Anyone have any ideas as to how I might be able to work around this?
EDIT: I know similar questions to this have been asked before, but there haven't been any viable solutions to this problem that I've seen.
I see only two good choices:
make that site a webproxy
use unlimitedStorage permission and store the urls in WebSQL database (it's also the fastest). Despite the general concern that it may be deprecated in Chrome after W3C stopped developing the specification in favor of IndexedDB I don't think it'll happen any time soon because all the other available storage options are either [much] slower or less functional.

Why doesn't google analytics code use AJAX

I was wondering how ga collected data and send it to their servers, then I found this answer on SO. Now I'm wondering why does GA uses this method rather than doing an AJAX request, is it cheaper?
It's not cheaper, per se, it is reliable. Unlike AJAX, you can include an image from any domain without running into cross-domain browser restrictions, this is why tracking pixels are used instead of ajax requests.
As Rob said, it's primarily to get around cross-domain issues not supported in older browsers. However, as of recently GA has added support for the navigator.sendBeacon() method, which actually is cheaper, allows for retries on error, and doesn't have the problem of failing when the page is being unloaded (like when trying to send an event when a user clicks on an outbound link). As browser support increases, this will likely become the default method for sending hits to GA.
Here's the documentation on how to use sendBeacon with analytics.js:
https://developers.google.com/analytics/devguides/collection/analyticsjs/field-reference#useBeacon

What is the magic behind Lightstreamer?

I'm gonna develop a framework for comet programming, and I can't use Web Sockets, or Server-Sent Events (because browser support really sucks). So, I need to keep the HTTP connection alive, and send chunked data back to the client.
However, problems show themselves as you get into the work:
Using XMLHttpRequest is not possible, due to the fact that IE doesn't give you xhr.responseText while the xhr.readyState is 3.
A hidden iframe can't be useful, because browser shows the loader while I send data back to the client.
I tried to send a JavaScript file back to the client, sending function execution commands each time, but browsers won't execute JavaScript till it's completely loaded.
However, when I look at Lightstreamer demo page, I see that it sends a JavaScript file back to the client little by little and in each step, it sends a call to the function and that function simply gets executed (I can't do this part). It seems that Lightstreamer uses AJAX, since the request simply shows up in Firebug's console tab, but it works like a charm in IE too.
I tried to use every HTTP header field they've set on their request, and no result. I also tried to use HTTP Post instead of HTTP Get, but still got no result.
I've read almost over 20 articles on how to implement comet, but none of'em appear to solve problems I have:
How to make it cross-browser?
How to get notified when new data is arrived from server (what event should I hook into)?
How to make my page appear as completely loaded to the user (how to implement it, so that browser doesn't show loading activity)?
Can anyone please help? I think there should be a very little tip or trick that I don't know here to glue all the concepts together. Does anyone know what lightstreamer do to overcome these problems?
SockJS author here.
How to make it cross-browser?
This is hard, expect to spend a few months on getting streaming transports on opera and IE.
How to get notified when new data is arrived from server (what event should I hook into)?
There are various techniques, depending on a particular browser. For a good intro take a look at different fallback protocols supported by Socket.IO and SockJS.
How to make my page appear as completely loaded to the user (how to implement it, so that browser doesn't show loading activity)?
Again, there are browser-specific tricks. One is to delay loading AJAX after onload event. Other is to bind-and-unbind an iframe from DOM. ETC. If you still feel interested read SockJS or Socket.io code.
Can anyone please help? I think there should be a very little tip or trick that I don't know here to glue all the concepts together. Does anyone know what lightstreamer do to overcome these problems?
Basically, unless you have a very strong reason to, don't reinvent the wheel. Use SockJS, Socket.io, faye, or any other of dozens projects that do solve this problem already.
The methods you want is the streaming.
How to make it cross-browser?
Considering most browsers, there is no consistent way. You have to choose a proper transport according to the browser. Even worse, you have to rely on the browser sniffing to recognize which browser is being used, and the feature detection counts for nothing about this. You can use XDomainRequest for IE8+, XMLHttpRequest for non-IE and Iframe for IE 6+. Avoid iframe transport if possible.
How to get notified when new data is arrived from server (what event should I hook into)?
This varies according to the transport being used. For example, XDomainRequest fires progress event, XMLHttpRequest fires readystatechange event when chunk is arrived except Opera and IE.
How to make my page appear as completely loaded to the user (how to implement it, so that browser doesn't show loading activity)?
I don't know this issue with iframe, but still occurs in WebKit based browsers such as Chrome and Safari with XMLHttpRequest. The only way to avoid this is to connect after the onload event of window, but, in case of Safari, this does not work.
There are some issues you have to consider besides the above questions.
Event-driven server - The server should be able to process asynchronously.
Transport requirements - The server behaves differently for required transport.
Stream format - If the server is going to send big message or multiple messages in a single chunk, a single chunk does not mean a single data. It could be fragment of a single data or concatenation of multiple data. To recognize what is data, the response should be formatted.
Error handling - Iframe transport does not provide any evidence for disconnection.
...
Last but not least, to implement streaming is pretty tiresome than it looks unlike with long polling. I recommend you use solid framework for doing that such as socketio, sockjs and jquery socket which I've created and managed.
Good luck.
but browsers won't execute JavaScript till it's completely loaded.
Have you tried sending back code wrapped in <script> tags? For example, instead of:
<script type="text/javascript">
f(...data1...);
f(...data2...);
try
<script type="text/javascript">f(...data1...);</script>
<script type="text/javascript">f(...data2...);</script>
The best option in your case would be to use JSONP + Long Pulling on server side. You just have to remember to reconnect any time connection drops (times out) or you receive response.
Example code in jquery:
function myJSONP(){
$.getScript(url, {
success: function(){
myJSONP(); //re-connect
},
failure: function(){
myJSONP(); //re-connect
}
})
}
Obviously your response from server has to be javascript code that will call on of your global functions.
Alternatively you can use some jquery JSONP plugin.
Or take a look on this project http://www.meteor.com/ (really cool, but didn't try it)

JSONP and Backbone.js

I would like to use Backbone.js with a REST api I control. I was hoping to have the REST api and the Backbone scripts live on a different domain but unfortunately this will be blocked, as it is a cross domain request.
Does Backbone.js have an built in functionality to support JSONP requests? Or, alternatively, does anyone have any experience with manually adding JSONP support to Backbone.js sync system?
JSONP support for GET operations can be added via fetch's options.
In the same hash where you configure your success and error handlers, add an object like so:
{dataType: "jsonp"}
This will pass along the jsonp option to JQuery's ajax handler, and automagically, you'll have JSONP support for retrieving models / collections.
You will not be able to use your entire REST API with JSONP. You can only call GET requests with JSONP (it works by writing a new <script> tag on the current document, then calling a javascript callback...).
To use all HTTP verb (POST, DELETE, PUT), you can use the CORS protocol : http://www.w3.org/TR/access-control/.
CORS is a protocol negotiated between a browser and a web-service that tells the browser that it is “OK” to execute Javascript code from a cross-domain call
To use this, you just need to include some custom headers in your server response that tells the browser that it's ok to accept cross domain requests. Here's an blog post that explains how to implement it with RubyOnRails (but it should be quite similar with others framework...) : http://www.tsheffler.com/blog/?p=428
It's the simplest solution, you can use backbone.js as if you where on the same domain, and it works with most current browsers (Internet Explorer 8+, Firefox 3.5+, Safari 4+, and Chrome) !
If you need older browser support, I did manage to make backbone work using easyXDM :
easyXDM is a Javascript library that enables you as a developer to easily work around the limitation set in place by the Same Origin Policy, in turn making it easy to communicate and expose javascript API's across domain boundaries.
It's a little more complicated, and works with a some well known iframe hacks (that are sometimes used in javascript widgets like GMaps, facebook widgets, ...).
Hope this help!

Categories