I want to have sort of live blogging that will allow a reporter to live blog and add updates on-the-fly.
What is the best way to do this? Check for updates every X seconds or use some kind of AJAX push?
Some examples:
http://www.bbc.co.uk/news/world-middle-east-12307698
http://thelede.blogs.nytimes.com/2011/03/11/video-of-the-earthquake-and-tsunami-in-japan/
I would definitely look at using WebSockets or some sort of push technology.
It depends on your level but you can either use a third party service like Pusher, use an open source project like Juggernaut or build your own.
If you're going to bake a solution yourself look at Node.js, EventMachine and Socket.IO. You'll also need some sort of capable pub/sub queuing system. Redis or RabbitMQ are perfect.
I feel Node.js and JavaScript are more suited to building a realtime push server but if you are more comfortable with Ruby EventMachine is also great.
I'd use Socket.IO for all solutions. Socket.IO basically uses WebSockets if your browser/mobile supports them and falls back to Flash, polling etc. It takes all the headache away involved with browsers and their lack of support for HTML5 features.
Personally I would use Juggernaut as that combines Socket.IO, Redis and Node.js which are the technologies I'd choose to build such a system. Juggernaut is readymade, straightforward to setup and easily scalable thanks to Node.js and Redis.
Related
I am a beginner to web development - I am building a website that requires some entries from the user, does some complicated mathematical processing on those entries, and returns the result to the client. I was thinking of implementing the mathematical stuff in another separate application which is better suited for such work (like in Java or C++ where there are good math libraries and the implementation would be more robust and faster). I was wondering what is the best way to do this, architecture-wise.
The "dummy" approach would be spawning a process from the Node.js application and waiting on its output from stdout, parsing it (probably in JSON) and then processing it before sending the result to the client. I have trouble believing that this is the best way to do this (it seems too error-prone, no proper error handling, dependent on the output, and just plain bad practice). A slightly better approach would be to have the Java or C++ application listening on a specific port and waiting for requests from the Node.js application. However this requires more thinking in terms of load-balancing (how would it scale with the number of requests?). Finally, the last approach I found online was to use a queuing system such as RabbitMQ as a way to communicate between the Node.js application and the Java application.
Typically (in a "traditional" software), implementing a separate library that holds all the math magic to which we can make calls would be a good way to go.
What is the best approach to achieve this with a Node.js/web application? There must be good practices/models/architectures/designs for such a problem. Thanks!
The best approach I can think of is writing a native nodejs module, you can use C++ and any library you have to write the code and export some APIs to be called from node's javascript.
This also allows you to package your native application as a module for nodejs, which can then be distributed and compiled/installed with npm.
Here's the official tutorial from node's website.
It isn't the easiest approach, but certainly the more elegant and mantainable one.
IF your problem can be structured in a stateless way -- i.e. you send a request out with the data and the associated task, and then later receive the result as a response, I would probably create one angular app for the front end, and multiple node servers for the back end (IF you needed to handle many requests simultaneously).
As pointed out in the first answer below, you can put whatever parts of the problem that you want to into native code, BUT what might be much easier is to use sockets to connect to a server to do the computation, in any language, etc... In one project I use a socket.service to handle all the communication back and forth to the servers. (The angular-fullstack generator in the next paragraph has such a skeleton socket.service piece.)
In terms of the frontend in combination with a node backend, The Yeoman generator (http://yeoman.io/generators/) for creating MEAN stack applications, using MongoDB, Express, AngularJS, and Node - lets you quickly set up a project following best practices. And the angular-fullstack generator will give you a framework that is light years ahead in so many ways. You can use yeoman to generate a sample application frontend (angular, express) and backend (NodeJs, MongoDB) with support for sockets, etc...
This too is not very simple, but in the end, with a few months of work, you would have a structured problem and solution that you would never outgrow. When I first wanted to work into this space, that was the path I took. And in truth the front-end was/is the harder part and the backend, connected with sockets, http, or whatever has gotten to be much simpler. In terms of servers, AWS, or even just nginx on Linux through DigitalOcean or many others is inexpensive and flexible.
I hope this helps.
I currently have a Qt-based GUI application that simply provides a graphical dashboard (guages, graphs, and such) displaying real-time data provided by another application via a TCP connection. For architectural and security reasons, I have been told that my TCP connection is no longer allowed, and that an HTTPS websocket will be my only conduit. It was suggested that I convert my app to a webapp using HTML5 and Javascript. However, I'm not a web programmer, but may consider this an opportunity to learn a new technology. I want to understand the landscape a little better before making a decision. So, I think my choices are:
Convert my app to a web app, giving me all the advantages/disadvantages of web-based apps. More work for me, as it's a new technology and I already have the Qt/TCP version working fine. If this is the suggested approach, any suggestions on a development environment/tools would be appreciated.
Convert my current Qt app to us a client HTTPS connection rather than a TCP connection. Not sure this is possible. From what I've read, this may not be possible with QtWebKit. Seems strange?
Maybe there's another choice I'm not considering?
You can also use a mixed approach, using WT library. It's c++ (and QT inspired) based.
It has a nice toolkit and supports websockets.
http://www.webtoolkit.eu/wt/examples/
Porting requires a lot of work. On Qt side there are at least two implementations of WebSockets:
QtWebSocket, Qt-based websocket server implementation and alpha-state implementation of websocket client.
WebSocket++, Boost-based websocket server and client implementation.
I just went through a similar exercise for work recently. We settled on the Google Web Toolkit (GWT) framework for doing our web apps. It's completely java-based, meaning you write (almost) everything in Java and the GWT compiler converts it to Javascript for you. We used the GWT-EventService plugin for pushing data from the server to the clients.
I wish I had know about this WT library before we started though, that looks interesting.
You can also have a look at QWebSockets, which is a pure Qt implementation of websockets, both for client and server use.
Update: this library is now an integral part of Qt
im looking forward to build RT web apps with NodeJS. Coming from Rails, I've felt in love with NodeJS and Async JS programming.
Run a few experiments with Node, and then as I search tools and resources to get used with, I got overwhelmed with the lot of stuff over there.
I found lot's of libraries and components over there, and pretty much got confused on how a large-scale well-writen and implemented RT web app should be built.
So the app will run over NodeJS, using Express framework.
I read about knockout.js, a client-side library to provide realtime stuff like automatic UI refresh, and I guess I could conbine it with jQuery.
Also, I found socket.io.
The author says:
Socket.IO aims to make realtime apps possible in every browser and mobile device, blurring the differences between the different transport mechanisms. It's care-free realtime 100% in JavaScript.
So socket.io is about compatibility.
What about backbone.js? Where does it goes to?
With so much stuff, I got shocked. What should I learn? Which modules worth studing?
Im focusing on NodeJS and Express but most books/screencasts covers old versions of nodejs. So im being guided by its official API.
Now im here asking your advice and to organize somehow all the info out there.
Correct me if my assumptions are not precise, please point me to the right direction and feel free to suggest any other module that could help on my learning.
Thanks in advance
It might be useful for you to separate the node.js server side libraries (via npm etc...) from all of the client side (browser) libraries and technologies like jquery, backbone, knockout etc... when you think about it. Even socket.io which exposes a persistent socket connection between the browser and the server (to avoid polling) does not dictate what client side technologies you use.
Focus on exposing a solid web-api ( random example ) from your server and your client technologies can be swapped, augmented etc... with no effect on the server. The only place they intersect is if you're using a view technology like Jade. It's also an option to have a pure separation where the server is just serving up the client files and your client is a thicker javascript application (using knockout, jquery etc...) calling a good server web api.
Some folks try to unify the client and server models - for example, this article using backbone and node. It depends on how much data you work with to say whether that's feasible but it does couple the client and server and makes the server stateful which can have downsides (scale out, requires affinity etc...). Personally, I get wary of that much magic (binding, state, syncing etc...). Node is about keeping things simple, light and fast. It's a fast front end network server.
My 2 cents (and some may disagree). Start with node on the server and pick your storage (mongoDb etc...). Design a solid RESTful (hypermedia) API - a good webapi regardless of the client. Then start with a basic html/css/js, maybe jquery client and add things like knockout etc... as you expand your client skills. That will allow you to replace your client technologies independent of your server as the new technology winds change (and they will).
That's the hallmark of a well designed system - the ability to replace componets/sub-systems without rewriting everything :)
Hope that helps clear up some of the fog :)
You may want to look at Meteor if you are focussing on real-time Javascript apps: http://meteor.com/
I'm wondering if using Node.js (express.js) is the best option to go for a complete website. I've been using Rails + Node.js and am wondering if this is still the best option or if I should move completely to Node.js.
Are there websites built completely with node.js? I imagine it must lack many things Rails offers.
I imagine it must lack many things
Rails offers.
I gave a short list below but I would like to know what you could be missing and I think(almost certain) we can give you some alternatives on node.js.
Modules
node.js is getting pretty complete and I think you can run your entire site using only node.js using for example the following modules. I assume you know about npm. If not I really advice you to google for it and learn that. To search npm you can use http://search.npmjs.org:
web framework:
High performance, high class web
development for Node.js
https://github.com/visionmedia/express
Socket.IO aims to make realtime apps possible in every browser and mobile
device, blurring the differences
between the different transport
mechanisms.
https://github.com/learnboost/socket.io-node
I believe with these two web-frameworks you can create a lot of sites. For example express is a very powerfull web framework and supports a lot of cool things like:
session support.
a lot powerful template engines. I like Jade for example. You could also share these between client and server a lot of the times easily.
excellent routing.
just to name a few.
database:
Redis is an open source, advanced
key-value store. It is often referred
to as a data structure server since
keys can contain strings, hashes,
lists, sets and sorted sets.
https://github.com/mranney/node_redis
MongoDB (from "humongous") is a scalable, high-performance, open source,
document-oriented database.
Mongoose is a MongoDB object modeling tool designed to work in an
asychronous environment.
https://github.com/learnboost/mongoose/
With those two databases I think you should be able to accomplish most of your tasks. The nice thing is that Redis is extremely fast/simple advanced KV-store(dataset in memory/also supports VM) and supports replication while Mongodb is more completely(slower then redis) and also supports sharding.
authentication:
Authentication and authorization (password, facebook, & more) for your node.js Connect and Express apps.
https://github.com/bnoguchi/everyauth
Like I said previously you can get a long way with only these modules.
Express.js is more akin to Sinatra. They're simpler frameworks than rails.
Express's list of sites is fairly small http://expressjs.com/applications.html
So I think it's also good to look at Sinatra's list http://www.sinatrarb.com/wild.html
So to answer your questions in reverse order. Yes it lacks all the features of rails. Yes there are sites written completely in Express. And going to 100% Node.js might be the right decision for your site.
It depends on which features you'd miss and what performance you need.
I'm not sure, but I guess it's perfectly possible. I have built complete sites using plain server side javascript for years without problems. The advantage of node.js seems to be its event driven model and things like socket.io. I just started experimenting with it, I'll probably will try porting an existing site to node.js.
Here you can find a large list of sites built with node.js.
Finally, you may want to read: What it’s like building a real website in Node.js
Yes, as of now node.js lacks many (well, at least a few) things rails offers. Eventually the set of available node.js modules will collectively provide good alternatives to RoR across the board. Or at least different (and often more modern) approaches to the same fundamental problems. There are still some important things missing in the node.js ecosystem including a good ORM for postgresql (rails has ActiveRecord which is great) and a good DB schema management subsystem. Both of these do not exist in the node.js ecosystem as of this moment, but surely they will be there in due time. Rails has these down pat right now.
There are some tricks node.js has that RoR doesn't. Debugging in node.js is more seamless than RoR and express.js is more flexible. Express is the library approach (you tell it what to do) whereas rails is the framework approach (you fill out the boilerplate it creates for you). There are also some fantastic things like stylus and jade, which have equivalents in rails, but when you have a full app written in one language (either javascript or CoffeeScript, which I prefer) and everything is in a modern node.js/TJ Holowaychuck style, you get a level of cohesiveness that is really nice to work within.
The other thing to keep in mind is that while the list of available node.js modules is quite impressive, many of them are not as mature and battle-tested as their rails/django analogs. It's hit or miss, so beware.
I'm thinking about how limiting it is for AJAX apps to have to poll for updates, when what would be ideal is for javascript to be able to set up a real two way connection to the server. I'm wondering if there is some method of integrating javascript with a browser plugin that can make a tcp connection so that I could pass data into and out of the browser plugin.
WebSockets is designed to solve this problem.
Here is an implementation with a similar approach:
socketjs
It uses a Java Applet and bridges its API to JavaScript, interesting...
And here another one:
jSocket
This one is a wrapper of the Actionscript 3 Socket API, bridged to JavaScript...
You can use node.js framework's socket.io package which can can be installed via npm (A node package manager).
More detailed usage.
jSocket and Stream are two options that utilize Flash's built-in XML sockets, though neither appears to be production-ready. I'd lean towards using a Flash-based solution rather than Java, as browser penetration is higher and generally offers a better user experience (load times & stability).