We Keep Coding

Is it possible and plausible to implement a WebService over a WebRTC Data Channel?

Is it possible to implement a WebService over a WebRTC Data Channel ?
The idea is:
The client makes one https request to the server for signaling and session establishment
The client and the server start to communicate via a WebRTC DataChannel bidirectionally
Benefits?:
Performance ?
Requests goes over one connection and the standard allows for multiple datachannels over the same connection ( ports )
Flexible networking topologies
UDP
End to end encryption
The server can send events over the same connection
Load balancing could be implemented from a pool of servers client side without a load balancer , or all kinds of different solutions
Currently being debated the addition of DataChannels to Workers/Service Workers/ etc https://github.com/w3c/webrtc-extensions/issues/64
Drawbacks:
Application specific code for implementing request fragmentation and control over buffer limits
[EDIT 3] I don't know how much of a difference in terms of performance and cpu/memory usage will it be against HTTP/2 Stream
Ideas:
Clients could be read replicas of the data for sync, or any other applications that are suitable for orbit-db https://github.com/orbitdb/orbit-db in the public IPFS network, the benefit of using orbit-db is that only allows to the owner to make writes, then the server could additionally sign with his key all the data so that the clients could verify and trust it's from the server, that could offload the main server for reads, just an idea.
[EDIT]
I've found this repo: https://github.com/jsmouret/grpc-over-webrtc
amazing!
[EDIT2]
Changed Orbit-db idea and removed cluster IPFS after investigating a bit
[EDIT3]
After searching Fetch PROS for HTTP/2 i've found Fetch upload streaming with ReadableStreams, i don't know how much of a difference will it be to run GRPC (bidi) over a WebRTC DataChannel or a HTTP/2 Stream
https://www.chromestatus.com/feature/5274139738767360#:~:text=Fetch%20upload%20streaming%20lets%20web,things%20involved%20with%20network%20requests).
Very cool video explaining the feature: https://www.youtube.com/watch?v=G9PpImUEeUA

Lots of different points here, will try to address them all.
The idea is 100% feasible. Check out Pion WebRTC's data-channels example. All it takes a single request/response to establish a connection.
Performance
Data channels are a much better fit if you are doing latency sensitive work.
With data channels you can measure backpressure. You can tell how much data has been delivered, and how much has has been queued. If the queue is getting full you know you are sending too much data. Other APIs in the browser don't give you this. There are some future APIs (WebTransport) but they aren't available yet.
Data channels allow unordered/unreliable delivery. With TCP everything you send will be delivered and in order, this issue is known as head-of-line blocking. That means if you lose a packet all subsequent packets must be delayed. An example would be if you sent 0 1 2 3, if packet 1 hasn't arrived yet 2 and 3 can't be processed yet. Data channels can be configured to give you packets as soon as they arrive.
I can't give you specific numbers on the CPU/Memory costs of running DTLS+SCTP vs TLS+WebSocket server. It depends on hardware/network you have, what the workload is etc...
Multiplexing
You can serve multiple DataChannel streams over a single WebRTC Connection (PeerConnection). You can also serve multiple PeerConnections over a single port.
Network Transport
WebRTC can be run over UDP or TCP
Load Balancing
This is harder (but not intractable) moving DTLS and SCTP sessions between servers isn't easy with existing libraries. With pion/dtls it has the support to export/resume a session. I don't know support in other libraries however.
TLS/Websocket is much easier to load balance.
End to end encryption
WebRTC has mandatory encryption. This is nice over HTTP 1.1 which might accidentally fall back to non-TLS if configured incorrectly.
If you want to route a message through the server (and not have the server see it) I don't think what protocol you use matters.
Topologies
WebRTC can be run in many different topologies. You can do P2P or Client/Server, and lots of things in between. Depending on what you are building you could build a hybrid mesh. You could create a graph of connections, and deploy servers as needed. This flexibility lets you do some interesting things.
Hopefully addressed all your points! Happy to discuss further in the comments/will keep editing the question.

I was also wondering about this HTTP-over-WebRTC DataChannel idea a couple of years ago. The problem at hand was how to securely connect from a web app to an IoT device (raspberry pi) that sits behind a firewall.
Since there was no readily available solution, I ended up building a prototype. It did the job and has been in live deployment since 2019.
See this technical blog post that covers the design and implementation in more detail:
https://webrtchacks.com/private-home-surveillance-with-the-webrtc-datachannel/
High level architecture:
Simplified sequence diagram:
Recently began the process of extracting the code into a standalone repo.
https://github.com/ambianic/peerfetch

If your main use-case exchanges small content, you may have a look at CoAP RFC 7252. A peer may easily implement both roles, client and server, though the exchanged messages for request and response share the same fomat.
For some advanced usage of DTLS 1.2, DTLS Connection ID can do some magic for you.
If you don't stick to javascript and java is an option, you may check the open source project Eclipse/Californium. That's a CoAP/DTLS implementation, which comes with DTLS Connection ID and some prepared advanced examples as built-in-cid-load-balancer-support or DTLS-graceful-restart.

How does Twitter send websockets to it's clients

There are millions of tweets and millions of active users in the Twitter. When a tweet gets like or retweet,how do they send live updates(websockets) of every tweet to its clients?
I think they wouldn't send live updates(websockets) of each tweet to every active user, that would result in (no of active tweets)X(no of active users)=(millions)X(millions)>10^12 live updates in each minute, each user would get millions of updates(of all the tweets) in each minute.
I think the live update of a particular tweet would only be received by the users who are watching that particular tweet.If this assumption is correct,then please tell me, how do they filter clients who are watching a particular tweet and send live updates of that tweet only to those filtered clients?
I was just watching a tweet in the Twitter, I was surprised to see live updates in likes and retweets of that tweet.I haven't seen any social media(like Instagram) giving live updates for every single post of it. I want to implement this method in my social media website.What I had concluded might or might not be correct, but I would request you to explain me, how does Twitter send live updates of every single tweet only to those particular users who are watching it.

To be clear, ONE device has ONE socket connection, to Twitter's cloud.
That ONE socket connection, receives ALL information from Twitter's cloud
new tweets
new likes
new retweets
everything else
all information comes on the ONE socket.
The cloud "figures out" what to send to who.
Is this what you were asking? Hope it clears it up.
The amazing thing is that twitter's cloud can connect to perhaps 100 ? million devices at the same time. (This is an amazing, major engineering achievement which requires an incredible amount of hardware, money and engineers.)
BTW if you're trying to implement something like this for an experiment or client. These days it is inconceivable you'd try to write the server side to achiever this, from scratch. Services exist, which do exactly this - example pusher.com, pubnub.com and so on.
(Indeed, these realtime infrastructure services, are, the basic technology of our era - everything runs on them.)
Here's a glance at the mind-boggling effort involved in Twitter's cloud: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale.html

Realtime communication or what you refer to as 'live updates' is all a play of various low-level networking protocols. Here's a bit of background on the protocols in general just so you know what you are working with:
A regular REST API uses the HTTP as the underlying protocol for communication, which follows the request and response paradigm, meaning the communication involves the client requesting some data or resource from a server, and the server responding back to that client. This is what you usually see in a regular website that isn't really live but shows or does something following a button click or similar trigger from the user.
However, HTTP is a stateless protocol, so every request-response cycle will end up having to repeat the header and metadata information. This incurs additional latency in case of frequently repeated request-response cycles.
With WebSockets, although the communication still starts off as an initial HTTP handshake, it is further upgrades to follow the WebSockets protocol (i.e. if both the server and the client are compliant with the protocol as not all entities support the WebSockets protocol).
Now with WebSockets, it is possible to establish a full-duplex and persistent connection between the client and a server. This means that unlike a request and a response, the connection stays open for as long as the application is running (i.e. it’s persistent), and since it is full-duplex, two-way simultaneous communication is possible. Now the server is capable of initiating communication and 'push' some data to the client when new data (that the client is interested in) becomes available.
The WebSockets protocol is stateful and allows you to implement the Publish-Subscribe (or Pub/Sub) messaging pattern which is the primary concept used in the real-time technologies where you are able to get new updates in the form of server push without the client having to request (refresh the page) repeatedly. Examples of such applications other than Twitter are Uber-like vehicle location tracking, Push Notifications, Stock market prices updating in real-time, chat, multiplayer games, live online collaboration tools, etc.
You can check out a deep dive article on WebSockets which explains the history of this protocol, how it came into being, what it’s used for and how you can implement it yourself.
Another interesting one is SSE or Server-Sent Events which is a subscribe-only version of WebSockets and restricted to the web platform. You can use SSE to receive real-time push updates from servers, but this would be unidirectional as you can only receive updates via SSE and not really publish anything. Here's a video where I explain this in much more detail: https://www.youtube.com/watch?v=Z4ni7GsiIbs
You can implement these various protocols as required from scratch or use a distributed messaging service like Ably which not only provides the messaging infrastructure of these protocols but also offers other add-ons such as scalability, reliability, message ordering, protocol interoperability, etc, out of the box, which is essential for a production-level app.
Full disclaimer: I'm a Dev Advocate for Ably but I hope the info in my answer is useful to you nevertheless.

Express web sockets and a central server

I'm creating an app where the server and the clients will run on the same local network. Is it possible to use web sockets, or rather more specifically, socket.io to have one central server and many clients that are running native apps
? The way I understand socket.io to work is that the clients read the web-pages that are served from the server but what happens when your clients become tablet devices running native apps instead of web pages in a browser?
The scenario I'm working with at the minute will have one central server containing a MEAN app and the clients (iPads) will make GET requests to the data available on the server. However, I'd also like there to be real-time functionality so if someone triggers a POST request on their iPad, the server acknowledges it and displays it in the server's client-side. The iPad apps will (ideally) be running native phonegap applications rather than accessing 192.168.1.1:9000 from their browser.
Is this technically possible to connect to the socket server from the native apps or would the devices have to send POST requests to a central server that's constantly listening for new 'messages'? I'm totally new to the whole real-time stuff so I'm just trying to wrap my head around it all.
Apologies if this isn't totally clear, it's a bit hard to describe with just text but I think you get the idea?

Correct me if I am wrong.
You have multiple iPads running native app. They send a POST request to your node JS server which is running in a computer in the same local network. Whenever the server receives a request from app, you want to display that a request has been received in your computer screen.
If my assumptions about the scenario is correct, then it is fairly easy to do. Here are the steps to do it.
Create a small webpage (front end). Load socket IO in the front end page like this -
<script type="text/javascript" src="YOUR_SERVER_IP/socket.io/socket.io.js"></script>
Then connect to server using var socket = io(). This should trigger connection event in your backend.
Handle all POST request from apps normally. Nothing special. Just add a small snippet in between. socket.emit('new_request', request_data). This sends new_request event to front end.
Handle the new_request in your front end using socket.on('new_request', function(request_data) { ... });. That's it. No need to add anything to your native app for realtime update.
The second step would be a little complicated as it is necessary to make socket variable available inside all POST requests. Since you chose node.js, I don't think you need any help with that.

Not totally clear on your project, but I'll try to give you some pointers.
An effective way to send data between native apps and a server is using a REST server. REST is based on HTTP requests and allows you to modify data on the server, which can connect to your database. The data returned is typically either JSON or XML formatted. See here for a brief intro: http://www.infoq.com/articles/rest-introduction
Android/iOS/etc have built in APIs for making HTTP requests. Your native app would send a request to the server, parse the response, and update your native UI accordingly. The same server can be used from a website using jQuery ajax HTTP requests.
Express.js is more suited to serving web pages and includes things like templating. Look into "restify" (see here: mcavage.me/node-restify/) if you just want to have a REST server that handles requests. Both run on top of node.js (nodejs.org).
As far as real-time communication, if you're developing for iOS look into APNS (Apple Push Notification Service). Apple maintains a persistent connection, and by going through their servers you can easily send messages to your app. The equivalent of this on Android is GCM (Google Cloud Messaging).
You can also do sockets directly if that's easier for you. Be careful with maintaining an open socket on a mobile device though, it can be a huge battery drain. Here's a library for connecting ObjC to Socket.IO using websockets, it may be useful for you: https://github.com/pkyeck/socket.IO-objc
Hope that helps!

To answer your question, it is definitely possible. Socket.io would serve as the central server that can essentially emit messages to all of the client. You can also make Socket.io listen for the messages from any of the clients and serve the emitted message to the rest of the clients.
Here's an example of how socket.io can be used. Simply clone, npm install, and run using 'node app.js'

All you have to do is to provide a valid server address when you connect your socket from the iPad clients:
var socket = io.connect( 'http://my.external.nodejs.server' );
Let us know if you need help with actual sending/receiving of socket events.

It is possible to connect to Websockets from your apps.
If you are using PhoneGap then you need a pluging that gives support to websockets in your app (the client) and then use websocket like normal way using Javascript see this.
If your app is native iOS look into this it could help you.

The primary use of the Sockets in your case is to be a bidirectional "pipe" between an app and server. There is no need of server sending the whole web-page to the native app. All what you need is to send some data from server to the client(app) in response to POST (or GET) request and then using this data on client side to update client's UI in real-time. If you are going to use moderate amount of devices (say tens of them), you may have connected all of them to the server permanently keeping individual socket connection open for every individual link server-to-app. Thus you may deliver data and update client's state in real time.
In fact web browsers also employ sockets to communicate to web servers. However as in general case there is no control on amount of concurrent clients in Internet, for the sake of limited networking resources conservation, servers do not keep sockets open for a long time, closing it just after the web-page was sent to client (or timeout has expired). That's how HTTP protocol works on the low level. The server waiting for the HTTP clients (browsers) by listening the 80 port, responding them by sending the whole web page content, then closing the connection and keep waiting for another requests on the same port.
In your case it's basically a good idea to use socket.io as it's a uniform implementation of sockets (ok WebSockets) on both client and server side. The good starting point is here

Script on server to interact with clients Javascript

I currently have a javascript library that is using a JSON file to print them on the screen in an interactive way. (::We are using D3JS Library)
When we are on a client, we can easily delete, edit and create some nodes, that are updated in the JSON every 5-10 seconds.
The problem comes from two main facts :
First the automatic function that call itself every x seconds could make data corruption if we are doing some stuff on the datas already represented on the screen.
Then the project has been made in order to permit 5 people to interact together. When they are present onto the same session we cannot decently make them refresh every 5 seconds, that cause many overhead and doesn't avoid data corruption.
We have mainly thought about a solution only made with javascript and some AJAX but we realize that it should be reconsidered with a trigger that inform the client that the datas are no longer OK.
We are thinking currently about opening a script onto a server in order to attribute on each client an ID.
The goal would be to detect the modification done on the JSON file (on the server). But the point where we are stuck is :
1) Is there a best scripting language to interact server/web?
2) Which type of things should we use to make the clients update their datas? (socket right?)
About the second point the easiest way would be to call a JS function be we aren't aware of the possibilities given by the shell codes...
Sorry about the fact that we are happy developpers but maybe not enough skilled to solve this problem.
Thanks for your helps !

You can achieve that using pure javascript with the new WebSocket feature.
http://www.html5rocks.com/en/tutorials/websockets/basics/
Edit:
WebSocket is a web technology providing full-duplex communications channels over a single TCP connection. The WebSocket API is being standardized by the W3C, and the WebSocket protocol has been standardized by the IETF as RFC 6455.
WebSocket is designed to be implemented in web browsers and web servers, but it can be used by any client or server application. The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.[1] The WebSocket protocol makes possible more interaction between a browser and a web site, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open.

how to stream video in direct connection between 2 clients

I want to stream video between 2 clients without passing it through the server
Each side sends real time video and also receives the other sides real time video
Is there an open source project that allows that?
Is there an API for that? I'm willing to pay
I want to create it in web app for mobile
Js, html, Ajax, websockets, css...
Thank you so much

VLC has a built in streaming server, as well as the gui it can be use via the comand line so could be scripted to suit your requirements
http://www.videolan.org/doc/streaming-howto/en/

If you stream video directly from one client to another, then you have to understand between two networking models: client-to-server and peer-to-peer.
Server usually is static machine, with networking infrastructure, static ip and many things that allows accessibility by public.
With peer-to-peer you will face many problems, first of them is going through NAT when you creating socket for receiving. One of client might need to create socket to accept connection, and second to accept. They might do both simultaneously and stick to first connected.
Streaming video using web is not possible right now. There is only some beta development happening for Chrome and FireFox that will be publicly available not really soon.
As well you can't establish peer-to-peer connection using WebSockets.
So there is no way doing it using Web technologies.
You might want to have a look into native Mobile development, but there you will face problems with peer-to-peer connections as well.