I would like to know how Gmail (or anyone else) does comet on Opera.
Here is what I know so far from my experiments.
It doesn't use the event-source tag which is broken in Opera 10.51.
It doesn't use iframe which displays a spinning throbber and a busy mouse cursor.
It doesn't use responseText on xmlhttprequest when readyState = 3 which is known to be broken on Opera.
I tried seeing how it was done in mibbit and etherpad, and I found that they both use long-polling.
Bounty
The bounty goes to whoever can tell me a method better than "event-source" for Opera comet streaming, or how gmail does streaming (or long-polling if it does that).
GMail uses BrowserChannel (Docs | Source), which is included in Google's Closure Library.
#fileoverview Definition of the BrowserChannel class. A BrowserChannel
simulates a bidirectional socket over HTTP. It is the basis of the
Gmail Chat IM connections to the server.
I really don't have any idea on what the answer is. But I know Opera supports server-events : http://my.opera.com/WebApplications/blog/show.dml/438711 . Maybe it's a step towards the anwser?
I'm not really sure either, but I think they use it within Opera Unite.
I think that rather cross-browser (including Opera) approach might be to stream data through an Adobe Flash application. Though it would introduce dependence on the Flash plugin and is not very popular because of that.
I am the author of an in progess C++ HTTP server that is compatible with goog.netBrowserChannel. You can find the docs I've written while studying the protocol here:
http://code.google.com/p/libevent-browserchannel-server/wiki/BrowserChannelProtocol
Long story short, BrowserChannel uses forever frames on IE and XHR streaming on all other browsers. The protocol is divided into several phases, the first of which is network testing:
1) test the network to ensure response "streaming" is supported (in other words no buffering proxy exists)
2) check access to a variety of network prefixes (to make sure the network admin has not blocked access to chat)
Then the actual data transmission can start. Data is divided into two channels (forward and back). The back channel is a series of long lived (about 4 mins each) requests used for the server to "stream" content to the client. To do so HTTP chunked encoding is used. The client does it's best to make sure that one backchannel is always open. The server will close it about every 4 mins after which the client will open a new backchannel. The forward channel is used to send data from the client to the server. This pushing of data is done as necessary.
Related
As the title says, how do I get around a browser's 6 connections per domain limit?
I have a web app that needs to download data from the server as quickly as possible. Currently, I have up to about a dozen connections being opened at once, but in the network tab I see that several of the connections are stalled (Chrome) or blocked (Firefox) shown in gray:
After some digging around I believe this is due to Chrome's 6 connections per domain limit. How do I get around this?
After some searching, I have some reason to believe that one of these options may work, but I would like to know which option is the best (not a hack):
Use several subdomains (a.mysite.com, b.mysite.com etc...) and point them to the main server so that Chrome believes that they are different sites. This sounds like the most "hacky" option in my opinion and I don't know how long this will actually work for.
Some answers suggest that HTTP2 isn't restricted by the 6 connection limit, but when I look into using HTTP2 it seems like if the browser supports it then it will use it. So I guess this doesn't actually work since I'm still getting the stalled message in Chrome's most recent version.
I'm currently using fetch to get a stream of data from the server, would using Web Sockets or WebRTC instead not be limited by the 6 connection limit?
What other options are there?
I don't think webRTC is limited to 6 connections per domain, since it is often used for P2P mesh connections where that restriction would make no sense.
But I'd be surprised if you got better performance out of 20 datachannels than one HTTP2 connection since the webRTC datachannel is really not optimized for throughput.
You might also want to look at using Service Workers to work around the problem in a different way.
The RTCDataChannel API does not provide any kind of flow/control or back-pressure, does this mean, that a sender could, theoretically, crash the browser of the receiver ? In my opinion the browser, (Chrome, Firefox, etc. all use SCTP under the hood), reads from the SCTP connection and schedules to run the js-callback consuming the packet. If the event queue cannot not keep up with the sender, the browser basically reads continuously packets while storing the packets in a buffer, which grows indefinitely. So when you connect two browsers, the sender can actually always overwhelm the other one, because there is no barrier like TCP receive windows or something similar.
This applies to the websocket api as well.
Do I just miss something or are these API just broken ? If I'm right, this would be a severe security issue when talking to unauthenticated browsers (in a torrent scenario for instance).
The webrtc data channel used to be based on UDP. During that time there was artificial throttling imposed by the browser in order to prevent network flooding. This was the case until chrome v32, I believe.
Nowadays the data channel is based on SCTP which has build-in flow control (FC) and there is no browser throttling any more (thank God). The parameters that control FC are not exposed through the API but that doesn't mean there is no FC.
I am not familiar with the implementation of webrtc in Chrome/FF but I don't think you can crash the browser with a simple flood attack. The "producer is faster than the consumer" is a pretty old problem.
That said, I have been working with the data channel, for more than an year now and have seen my browser crash almost on a daily basis, so there are probably many bugs in the webrtc implementation. Hopefully they won't pose any threat to security.
Sending big chunks of data useing webrtc data channel is not a particularly pleasant experience. The API doesn't offer a "channel is ready for write" callback or anything of the sort, so, yes!, you have to poll the bufferedamount value and try to keep it inside an optimal window. To add insult to injury bufferedamount used to be broken under Windows versions of Chrome, it was always 0. But I think they fixed this in chrome v37 or around that time.
IMHO the webrtc API is not very well thought through but it does the job and honestly I cannot think of any js API that is well thought through.
I currently have a javascript library that is using a JSON file to print them on the screen in an interactive way. (::We are using D3JS Library)
When we are on a client, we can easily delete, edit and create some nodes, that are updated in the JSON every 5-10 seconds.
The problem comes from two main facts :
First the automatic function that call itself every x seconds could make data corruption if we are doing some stuff on the datas already represented on the screen.
Then the project has been made in order to permit 5 people to interact together. When they are present onto the same session we cannot decently make them refresh every 5 seconds, that cause many overhead and doesn't avoid data corruption.
We have mainly thought about a solution only made with javascript and some AJAX but we realize that it should be reconsidered with a trigger that inform the client that the datas are no longer OK.
We are thinking currently about opening a script onto a server in order to attribute on each client an ID.
The goal would be to detect the modification done on the JSON file (on the server). But the point where we are stuck is :
1) Is there a best scripting language to interact server/web?
2) Which type of things should we use to make the clients update their datas? (socket right?)
About the second point the easiest way would be to call a JS function be we aren't aware of the possibilities given by the shell codes...
Sorry about the fact that we are happy developpers but maybe not enough skilled to solve this problem.
Thanks for your helps !
You can achieve that using pure javascript with the new WebSocket feature.
http://www.html5rocks.com/en/tutorials/websockets/basics/
Edit:
WebSocket is a web technology providing full-duplex communications channels over a single TCP connection. The WebSocket API is being standardized by the W3C, and the WebSocket protocol has been standardized by the IETF as RFC 6455.
WebSocket is designed to be implemented in web browsers and web servers, but it can be used by any client or server application. The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.[1] The WebSocket protocol makes possible more interaction between a browser and a web site, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open.
I am looking at facebook news feed/ticker right now and I am wondering what technology/architecture it uses to pull in data asynchronously when any of my connections make an update. One possibility that I can think of is a javascript setInterval on a function that aggressively polls the server for new data.
I wonder how efficient that is.
Another possible technology that I can think of is something like Comet/NodeJS architecture that pings the client when there is an update on the server. I am not too familiar with this technology.
If I wanted to create something similar to this. What should I be looking into? Is the first approach the preferred way to do this? What technologies are available out there that will allow me to do this?
There are several technologies to achieve this:
polling: the app makes a request every x milliseconds to check for updates
long polling: the app makes a request to the server, but the server only responds when it has new data available (usually if no new data is available in X seconds, an empty response is sent or the connection is killed)
forever frame: a hidden iframe is opened in the page and the request is made for a doc that relies on HTTP 1.1 chunked encoding
XHR streaming: allows successive messages to be sent from the server without requiring a new HTTP request after each response
WebSockets: this is the best option, it keeps the connection alive at all time
Flash WebSockets: if WS are not natively supported by the browser, then you can include a Flash script to enhance that functionality
Usually people use Flash WebSockets or long-polling when WebSockets (the most efficient transport) is not available in the browser.
A perfect example on how to combine many transport techniques and abstract them away is Socket.IO.
Additional resources:
http://en.wikipedia.org/wiki/Push_technology
http://en.wikipedia.org/wiki/Comet_(programming))
http://www.leggetter.co.uk/2011/08/25/what-came-before-websockets.html
Server polling with JavaScript
Is there a difference between long-polling and using Comet
http://techoctave.com/c7/posts/60-simple-long-polling-example-with-javascript-and-jquery
Video discussing different techniques: http://vimeo.com/27771528
The book Even Faster Websites has a full chapter (ch. 8) dedicated to 'Scaling with Comet'.
I could be wrong, but I think that Facebook relies on a "long polling" technique that keeps an http connection open to a server for a fixed amount of time. The data sent from the server triggers an event client side that is acted upon at that time. I would imagine that they use this technique to support the older browsers that do not have websocket support built in.
I, personally, have been working on an application with similar requirements and have opted to use a combination of node.js and socket.io. The socket.io module uses a variety of polling solutions and automatically chooses the best one based on what is available on the client.
Maybe you may have a look to Goliath (non-blocking IO server written in Ruby) : http://postrank-labs.github.com/goliath/
I'm prototyping a realtime notification mechanism using http over port 80. The aim of the project is to allow a Flash application to respond to an event on a remote server quickly (specifically an inbound phone call being connected to a phone next to the computer.) Polling is one approach, but is too slow. Currently I use a socket connection to get low latency notification of the events on the server, which works well but isn't firewall friendly. I don't want to install anything except Flash, or Silverlight on the client. Cross compatibility of browsers isn't a concern - in this application I can specify what browser the client uses but IE is preferred.
I've made a server HttpHandler in .NET which never closes the connection and sends the "events" to the client by writing out bytes to the http response stream (ConnectedClientContext.Response.OutputStream.Write etc) and I have a .NET client application which can read these messages okay.
My Question:
Can I receive the bytes from the server over HTTP as they arrive using JavaScript, Flash or Silverlight? So far I can only find a way to get notified of the "download progress" and don't get the actual bytes until the response is closed - I want them as they arrive.
Best Regards,
Daniel
I don't know about Flash but in Javascript (by which you mean in browser) and Silverlight you are limited pretty much to the http protocol.
You can use the AJAX Http Streaming pattern. The basic ideas which is different from what you are trying is that as soon as data is available outstanding request ends and a new is immediately initiated asychronously, mean while your client process the newly arrived data.
Silverlight gives you more options since is HTTP stack is purely asynchronous but you can get your hands on the stream to you as soon as data starts to arive by setting the HttpWebRequest.AllowReadStreamBuffering to false. (Unlike XmlHttpRequest which always buffers).
it's very easy to use the Comet ideas for notifications. you don't even have to use a comet-specific framework. simply do an ajax request with a callback on answer, wrap this on a loop and you have an event loop, just like a GUI app. on the server side, don't bother answering the request until there's either an event, or a timeout (which is just a 'null' event).
Flex and Flash have several AMF/XML remoting libraries available that support data pushing. I would certainly look into that.
http://raghuonflex.wordpress.com/2008/04/17/data-push-in-flex-with-backend/
These libraries use a Comet - like implementation to achieve this.