I'm thinking about building some intranet applications that make use of websockets. I'm currently using Python/Pylons for my web framework on the server, and doing polling to update items in the DOM of the page. Pylons is not well suited to communicate with websockets (IMHO) as it uses a thread per connection. I'm considering using node.js as the server to communicate with the websocket connections from my web application. Here's the "10,000 foot view" of my application:
Pylons delivers the web content (html, css, images, javascript, etc.)
JavasSript on the page application opens up websocket(s) to the node.js server
The node.js server pushes data to the application through the websocket
JavaScript updates the page DOM elements based on the data from the websocket
The data in the case above comes from a MySQL database, which is where my question comes from. I've set up MVC type applications before, and can do the same kind of thing in node.js. However, if I have a long lived websocket open to the node.js server, how does node.js become aware of changes in the Model and push them out to the application? For instance if I want to update totals presented on the web application page, and those totals change due to actions in the system outside of node.js (other web applications), how is node.js notified of those changes? The thing that comes to mind is to have node.js poll the database for various changes and propagate the changes to the various Views. But to me that just sounds like I'm moving my polling from the web application to the node.js server?
Anyone have any ideas, suggestions or pointers on this?
Thanks in advance!
Doug
You can either:
Let the Python scripts notify the node.js application (via a socket or via HTTP)
Or poll from node.js because it is not aware of changes outside it's environment
Polling is considered bad because it doesn't scale. When having a single process that polls does scale because it doesn't need more connections when another user connects. So basically:
// query every second or so
setInterval(function () {
// query database
doSomeDatabaseStuff(function (res) {
// check dirty
if (res.changed) {
// notify all clients
allConnectedSockets.forEach(function (socket) {
socket.send({ msg: "update" });
});
}
})
}, 1000);
This way you have one single process polling the database, and a scalable architecture to notify your connected clients. The database can still be filled from any source.
Sails.js is an MVC framework for node that has the unique distinction of RESTfully routing Socket.io messages in the same way as your Express routes.
Sails currently uses Sequelize, and is configured by default to use mySQL (but also supports SQLite and Postgres). We're switching to a model that lets you choose your own ORM, which will allow us to support JugglingDB (which adds support for Mongo, amongst others)
It also comes bundled with a front-end component, Mast. Mast provides similar functionality to Meteor, in that it allows you to talk directly to the database from the client. If you need to provide more complex functionality, you just add a controller on the backend.
More here: https://github.com/balderdashy/sails
Related
Background:
I am building a reactJS application using AWS cognito, dynamo and S3. The application is based in the recruitment sector where employers and employees can post and view jobs respectively. When an employee applies for a job the employer can view the employees profile and decided whether or not to message them. Employees and employers converse via an on-site messaging service.
The Question:
What is the best method to facilitate user chat?
i.e. what is a nice & efficient way to store messages and notify users when they have a new message.
Our current approach is to have a setTimeout() on the site and check for new messages but this will be very inefficient so i'm looking for some guidance.
I would like to stay inside the amazon infrastructure as much as possible but I am open to all suggestions.
I'm currently building something similar for a startup I'm working at. Our React app is served by node.js server, while the API backend is provided by a django API with drf. As in your user chat case, we need to handle some real time data arriving in the frontend.
Our approach
The solution may be split up into inter server and server-browser realtime communication:
We use redis (aws elasticache to be exact) as a publish/ subscribe message queue to push incoming data from the API backend to the nodejs server. Specifically, whenever an instance of the model in question is created due to an HTTP POST call (i.e. in your case a message, which is send to the server), we publish JSON serialized information on a channel specific to the actors of concern.
On the node.js servers, we subscribe to channels of interest and receive information from the backend in real-time. We then use socket.io to provide a websocket connection to the frontend, which may be easily integrated with React.
Limitations of this approach
You cannot simply server your React app as a static website from S3 and have to rely on a node x React approach. react-boilerplat(by Max Stoiber I think) is a great way to start.
What's more, you can also use websockets end to end. We use this approach as our data source isn't a browser but a constrained device.
Hope that helps!
Case scenario:
I have few 2k concurrent users access to the website with various devices but using their browsers. Once one of them create new topic, all others currently connected should receive a notification (basically I simple update little icon number in app upper right corner).
One way to accomplish this is to have web app keep requesting updates via ajax calls but that overload my slow server with numerous requests.
I use azure to host my web app (written in PHP). There are some services included in my hosting package such as Event Hub, Service Bus etc. What service could I use in order to have my backed talk to a "service" whenever there is a new post, and than to have that "service" talk to my clients (their browsers) and informing them about new notification or any type of data updates?
You're probably looking for websockets. A websocket sets up a connection between the page in the client's browser and your webserver. Through this connection you can push new topics to all connected clients.
It is advisable to decouple the websocket sending process from the request handling of the topic creation. For this you need a background worker which sends websocket notifications when triggered from a processing event.
You can implement this in PHP using ratchet.
The main interface app will return variables based on those initially POSTED by the client, and subsequent database calculations performed in real time by a dedicated engine.
In Sails can we plug the engine into a Controller used for returning the calculated variable?
What would be the best way to implement a real time link between the client and the engine ?
Sails comes with sockets support built in. You can transmit the data out of your controller back to the client via sockets to keep everything in sync.
Reference this page for sockets in sails:
https://gist.github.com/mikermcneil/6598661
As an aside, you could do everything using sockets, including the posting.
What is this 'dedicated engine'? Is this a separate service running somewhere else, or is it just logic for processing this data and handing it back to the controller?
If you want to put the data processing logic in the same app you can create a service which exports whatever data processing functions you need. Then in your controller that is handling the POST requests you can call on those services as needed, process the data, and emit it back to the client. All your sockets logic can go in that same controller since it is for communicating with the client interface. I would consider just moving all of it to sockets. If you look at the sails docs you will see that it has a similar interface with sockets where you can do standard CRUD operations: sockets.PUT, etc.
Sails.js WebSockets
No framework other than Sails, unless necessary for security purpose
Minimalist architecture implemented in Sails
1 POST API
1 POST Model
1 POST Controller
1 Socket per processing method
1 Socket switch within the Controller
Now if a secured framework integrates socket switching within controllers, that would come useful.
We may add middleware to the POST Model for the purpose of filtering text input data...
I'm creating an app where the server and the clients will run on the same local network. Is it possible to use web sockets, or rather more specifically, socket.io to have one central server and many clients that are running native apps
? The way I understand socket.io to work is that the clients read the web-pages that are served from the server but what happens when your clients become tablet devices running native apps instead of web pages in a browser?
The scenario I'm working with at the minute will have one central server containing a MEAN app and the clients (iPads) will make GET requests to the data available on the server. However, I'd also like there to be real-time functionality so if someone triggers a POST request on their iPad, the server acknowledges it and displays it in the server's client-side. The iPad apps will (ideally) be running native phonegap applications rather than accessing 192.168.1.1:9000 from their browser.
Is this technically possible to connect to the socket server from the native apps or would the devices have to send POST requests to a central server that's constantly listening for new 'messages'? I'm totally new to the whole real-time stuff so I'm just trying to wrap my head around it all.
Apologies if this isn't totally clear, it's a bit hard to describe with just text but I think you get the idea?
Correct me if I am wrong.
You have multiple iPads running native app. They send a POST request to your node JS server which is running in a computer in the same local network. Whenever the server receives a request from app, you want to display that a request has been received in your computer screen.
If my assumptions about the scenario is correct, then it is fairly easy to do. Here are the steps to do it.
Create a small webpage (front end). Load socket IO in the front end page like this -
<script type="text/javascript" src="YOUR_SERVER_IP/socket.io/socket.io.js"></script>
Then connect to server using var socket = io(). This should trigger connection event in your backend.
Handle all POST request from apps normally. Nothing special. Just add a small snippet in between. socket.emit('new_request', request_data). This sends new_request event to front end.
Handle the new_request in your front end using socket.on('new_request', function(request_data) { ... });. That's it. No need to add anything to your native app for realtime update.
The second step would be a little complicated as it is necessary to make socket variable available inside all POST requests. Since you chose node.js, I don't think you need any help with that.
Not totally clear on your project, but I'll try to give you some pointers.
An effective way to send data between native apps and a server is using a REST server. REST is based on HTTP requests and allows you to modify data on the server, which can connect to your database. The data returned is typically either JSON or XML formatted. See here for a brief intro: http://www.infoq.com/articles/rest-introduction
Android/iOS/etc have built in APIs for making HTTP requests. Your native app would send a request to the server, parse the response, and update your native UI accordingly. The same server can be used from a website using jQuery ajax HTTP requests.
Express.js is more suited to serving web pages and includes things like templating. Look into "restify" (see here: mcavage.me/node-restify/) if you just want to have a REST server that handles requests. Both run on top of node.js (nodejs.org).
As far as real-time communication, if you're developing for iOS look into APNS (Apple Push Notification Service). Apple maintains a persistent connection, and by going through their servers you can easily send messages to your app. The equivalent of this on Android is GCM (Google Cloud Messaging).
You can also do sockets directly if that's easier for you. Be careful with maintaining an open socket on a mobile device though, it can be a huge battery drain. Here's a library for connecting ObjC to Socket.IO using websockets, it may be useful for you: https://github.com/pkyeck/socket.IO-objc
Hope that helps!
To answer your question, it is definitely possible. Socket.io would serve as the central server that can essentially emit messages to all of the client. You can also make Socket.io listen for the messages from any of the clients and serve the emitted message to the rest of the clients.
Here's an example of how socket.io can be used. Simply clone, npm install, and run using 'node app.js'
All you have to do is to provide a valid server address when you connect your socket from the iPad clients:
var socket = io.connect( 'http://my.external.nodejs.server' );
Let us know if you need help with actual sending/receiving of socket events.
It is possible to connect to Websockets from your apps.
If you are using PhoneGap then you need a pluging that gives support to websockets in your app (the client) and then use websocket like normal way using Javascript see this.
If your app is native iOS look into this it could help you.
The primary use of the Sockets in your case is to be a bidirectional "pipe" between an app and server. There is no need of server sending the whole web-page to the native app. All what you need is to send some data from server to the client(app) in response to POST (or GET) request and then using this data on client side to update client's UI in real-time. If you are going to use moderate amount of devices (say tens of them), you may have connected all of them to the server permanently keeping individual socket connection open for every individual link server-to-app. Thus you may deliver data and update client's state in real time.
In fact web browsers also employ sockets to communicate to web servers. However as in general case there is no control on amount of concurrent clients in Internet, for the sake of limited networking resources conservation, servers do not keep sockets open for a long time, closing it just after the web-page was sent to client (or timeout has expired). That's how HTTP protocol works on the low level. The server waiting for the HTTP clients (browsers) by listening the 80 port, responding them by sending the whole web page content, then closing the connection and keep waiting for another requests on the same port.
In your case it's basically a good idea to use socket.io as it's a uniform implementation of sockets (ok WebSockets) on both client and server side. The good starting point is here
As a web developer, I have developed a chat service and some other real-time collaborative services with the help of third-party services like Redis or Pusher. They provide simple API's that I can use publish/subscribe model to achieve bidirectional communication over the webserver. I want to now implement a simple push notification without the use of any third-party services, but I am not quite sure how to achieve this. The situation is as below:
Backend is in Python (Django)
A user receives a private message from another user.
The recipient should be notified without having to refresh the web browser.
My questions:
From this post, Django seems a bad option to achieve such functionality while Node.js is a good one. Is it true? If it is, why is that?
Is it possible to open a websockets from client to the server, to listen to certain changes to a specific model? (ex. when there's a new message in Message model, update the DOM)
I appreciate any help a lot!
WebSockets are exactly what you want here. They are however, a (comparatively) recent addition to browsers and as a result, support is not ubiquitous. WebSocket Support
The websocket model allows you to connect a socket from a server to a client with a web browser, then send messages from the server and receive them asynchronously at the client and vice versa.
Because Node.js is Javascript and the chances are you will be writing your client in Javascript, it lends itself as a sensible choice for writing coupled components, such as in this scenario.
The most popular WebSocket library is Socket.io which was built for use with Node.js. With Socket.io, your notification model would look something like this:
Server
io.sockets.on('connection', function (socket) {
socket.emit('notification', { name: 'Notification', message: 'It\'s here');
});
Client
var notifications = document.getElementById('nlist');
socket.on('notification', function(notification) {
var div;
// make some DOM changes
document.title = notification.name;
div = document.createElement('div');
div.innerHTML = notification.message;
notifications.appendChild(div);
socket.emit('received-notification');
});
Socket.io allows you to use custom event names, as shown here, which makes designing communication APIs that little bit easier. It also has fallbacks for XHR Long Polling and Flash sockets, in the circumstance that the user does not have Web Sockets.
Node will be faster than Django in this circumstance, but you may find that your codebase is more manageable in Django if this is your first venture into Node. It can be kinda difficult to design an application with a series of callbacks like this. I haven't ever used WebSockets with Django, but my experience with Tornado and WebSockets has been poor.
If you just need unidirectional communication as pushing notifications, you'll better have a look at Server Side Events instead of Websockets.
SSE is much easier to implement, it doesn't need any special protocol (just standard HTTP) and does automatic reconnection.
Creating private messages would be done using good old XMLHttpRequests and should be filtered on client-slide.
Node.js is pretty good for serving SSE: its event loop implementation only uses one thread to manage multiple persistent connections.
Have a look at sse-pubsub which provides an easy way to send notifications.