node.js: create app that auto-updates clients by data-driven events - javascript

I have spend almost three days now on learning node.js. I went through an online webcast that cleared the basics. I checked out several samples and tutorials how to build simple apps like webserver, chat program, connecting to mySQL DB, using mongo DB with json...
So far so good. But reaching my goal is still impossible because those simple scenarios always just end when it starts getting intersting.
My destination is to build a demo project that is able to do the following:
1- hold a simple list of data objects serverside
2- show them on webpage of all connected clients
3- allow web-clients to modify the data objects
4- almost immediately updates all the other clients pages as soon an object was modified
(For the first step it would be ok to assume all the objects are once loaded and held in memory of the server (service?) - instead of polling the db for changes or subscrie any DB events - what seems to be another big X-Files topic people fill books about...).
Concerning 1 -3: I already realized using mongodb and json and some addd/edit/delete methods taken from the following walkthrough: http://cwbuecheler.com/web/tutorials/2014/restful-web-app-node-express-mongodb/
Concerning 4: After looking at couple of chat app samples (e..g. http://nodecode.de/chat-nodejs-websocket) I thought this might be a good approach to solve this task, by just broadcasting the object that has been changed through socket (instead of sending a chat text message).
So I tried to combine these samples to one application that match my needs. But I failed. Even before wiring up the functionality I already stuck when I tried to place the two functionalities of showing/modifying the object list and provide the chat function just on one page side by side.
Maybe I am still missing some basics. Maybe the approach is wrong. I can't find samples for a similar task anywhere. So I guess now it's time to ask pros for some help where or how to start.
Thanks in advance.

You are on the right track. Websocket is the right approach.
"already stuck when I tried to place the two functionalities of showing/modifying..."
Without reading your code, I can't really give you a detailed answer to the problem, but here is a vague high level summery of what I would do:
Create an api endpoint on the server-side for the client to read and update the data.
Figure out how to initiate a connection from server to client using websocket.
Send a notification using websocket to the client every time the api receives an "update" request.
Write a set of functions on the client side to fetch and render new data every time when a notification comes from server

Related

Pseudo real-time data stream javascript and php

Update: I am getting the impression that this is not even the right website to post this. If someone can point me in the right direction, I'd be appreciative...
I have an existing PHP+MySQL application that wasn't built to render "real-time" or similarly live-style data. But now I need to build in a way to pull nearly real-time data into the application and keep the data on the page fresh. This live data is only for 1 page in the application.
Looked at things like socket.io and PHP-based websockets libraries, but it seemed like overkill because the data is basically coming from 1 source and being delivered to 1 person (the client). Multiple other users could have this process running, but each one would bring their own data endpoint. That's... like a year down the road. But good to think about. Would ideally have hundreds, or thousands of users on the system, pulling their live-ish data. So I want this to be as streamlined and low-impact as possible.
Users must be authenticated and authorized to consume the data. This is already baked into the current system.
The API to get the data (which has already been built by another vendor) is also NOT streaming. It's set on a 20-second cron, so the new data is available every 20 seconds, which satisfies the client's needs.
My current plan is to do something like this...
Data is pulled on a cron every 20 seconds, organized, and stored into the database (complete)
Adjust #1 so it also does any additional proprietary calculations on data AND compiles + writes a JSON file on the server (unique to the user) which is the exact data needed for the front end (DB data is needed for other pages)
Create small PHP-based service which validates a client-provided JWT and reads the JSON file out
Write AJAX front end to poll endpoint from #3 every X seconds using a JWT for authorization
This all seems sort of like I might be reinventing the wheel, or missing something. The fact that this is an existing PHP based application (LAMP) does have some limiting factors, but I feel like there's got to be a more efficient way to handle this... It's pretty new to me. Also, I'm open to other technologies that'll run on the LAMP stack, if it'll make things better.
I would say go for the API solution in the beginning :) Since it fits the architecture more and is for sure the least amount of work. Also if there will be problem with the "live" feeling of the data you can fix it by polling more often or introducing long polling, assuming you change the cron job time.
I mean in the end it is all about impact for the time spent, don't start implement features that customers don't care about :)
The biggest problem to solve is to implement it in a way that fits your requirements and is somewhat future extendable. You still have to deal with issues like resolution, time outs, reducing server processing when requesting data and so on!
For me, if you need to maintain a global service state because a single client(s) request could affect all other connected client request(s) then most all server-side scripting languages are not the best choice! Also to further add, if you plan on implementing something like this with PHP, you will be setting your self up for a living nightmare! Why, because simply put, PHP(s) socket(s) implementation is that bad!

Connecting microservices with APIs

I am very new to the concept of both microservices and APIs. My only coding experience has been simple programs in a variety of languages. I am trying to understand microservices and REST APIs. I have done a lot of searching, but can't seem to find a clear example of how to connect microservices with APIs to send and receive information between them. Can anybody at least lead me in the right direction? I know there are problem a ton of ways and different for every language, so here an example below.
For example, pretend I have a simple javascript program with HTML and CSS for a simple UI. Let's assume I also have a seperate javascript program with no UI and all it does it calculate some numbers. How would I use an API to GET/POST two numbers to the calculator "microservice" and after calculating, that "microservice" would GET/POST that result back to the UI "microservice"? Can I test this as just two javascript files locally on my laptop by just using localhost for the URL in the API/HTTP Request?
NOTE: I put microservice in quotes because I realize that this is an extremely simplified example of a microservice. I also understand that my question doesn't contain any code or anything, so I apologize for that. I have researched for hours and can't seem to understand how to go about this. I really appreciate anybody's help!
This is super broad, and there are a lot of options.
The simplest approach is that the JS UI just hits the frontend API as needed (either on some event or via polling), and so when a new calculation request comes in and is answered, the new calculation is logged (to some form of persistence, e.g. a database or message queue) and so the current 'state' of that model changes and the next time the client asks for the current state, it gets that update.
A more complex approach is to set up some form of messaging infrastructure (e.g. pub/sub), and perhaps a websocket connection between the client and the frontend server(s). Then the event happening on the frontend microservice would be pushed to the client w/o the client asking for it directly. In that case, you need to account for dropped connections and so on, so that if a client misses an update b/c they're offline they can get the one(s) they missed once they reconnect.

Creating a Node.js dashboard based on a MySQL DB without a poller

I've read a few StackOverflow posts related to this subject but I can't find anything specifically helps me in my scenario.
We have multiple monitoring instances within our network, monitoring different environments (Nagios, Icinga, more...). Currently I have a poller script written in PHP which runs every minute via cron, it asks the instance to return all of its problems in JSON, the script then interprets this and pushes it in to a MySQL database.
There is then an 'overview' page which simply reads the database and does some formatting. There's a bit of AJAX involved, every X seconds (currently use 30) it checks for changes (PHP script call) and if there are changes it requests them via AJAX and updates the page.
There's a few other little bits too (click a problem, another AJAX request goes off to fetch problem details to display in a modal etc).
I've always been a PHP/MySQL dev, so the above methodology seemed logical to me and was quick/easy to write, and it works 'ok'. However, the problems are: database constantly being polled by many users, mesh of javascript on the front end doing half the logic and PHP on the back doing the other half.
Would this use case benefit from switching to NodeJS? I've done a bit of Node.JS before but nothing like this. Can I subscribe to MySQL updates? Or trigger them when a 'data fetcher' pushes data in to the database? I've always been a bit confused as I use PHP to create data and javascript to 'draw' the page, is there still a split of NodeJS doing logic and front end javascript creating all the elements, or does NodeJS do all of this now? Sorry for the lack of knowledge in this area...
This is definitely an area where Node could offer improvements.
The short version: with websockets in the front-end and regular sockets or an API on the back-end you can eliminate the polling for new data across the board.
The long version:
Front-end:
You can remove all need for polling scripts by implementing websockets. That way, as soon as new data arrives on the server, you can broadcast it to all connected clients. I would advise Socket.io or the Primus websocket wrapper. Both are very easy to implement and incredibly powerful for what you want to achieve.
All data processing logic should happen on the server. The data is then sent to the client and should be rendered on the existing page, and that is basically the only logic the client should contain. There are some frameworks that do all of this for you (e.g. Sails) but I don't have experience with any of those frameworks, since they require you to write your entire app according to their rules, which I personally don't like (but I know a lot of developers do).
If you want to render the data in the client without a huge framework, I highly recommend the lightweight but incredibly useful Transparency rendering library. Using this, you can format a Javascript object on the server using Node, JSONify it, send it to the client, and then all the client would have to do is de-JSONify it and call Transparency's .render.
Back-end:
This one depends on how much control you have over the behaviour of the instances you need to check. I assume you have some control, since you can get all their data in a nice JSON format. So, there are multiple options.
You can keep polling every so often. This is the easiest solution since it requires no change to the external services. The Javascript setInterval function is very useful here. Depending on how you connect with the instances, you might be able to use a module like Request to do the actual request, so that takes out a bunch more of the heavy lifting.
The benefit of implementing the polling in your Node app as well, is that you will receive the data in your Node app and that way you can immediately broadcast it to the clients, even before inserting it into a database. This will greatly reduce the number of queries on your database.
An alternative to polling would be to set up a simple Express-based API where the applications can post their 'problems', as you call them. This way your application will get notified the moment a problem occurs, and combined with the websockets connection to the client this would result in practically real-time updates.
To be more redundant, you would have a polling timer alongside the API, so that you can check the instances in case there's something wrong that causes them to not send over any more data.
An alternative to the more high-level API would be to just use direct socket communication, which is basically the same approach only using a different set of functions.
Lastly, you could also keep the PHP-based polling script. This would be the most efficient solution since you wouldn't go and replace everything. Then from the Node app that's connected to the clients with websockets, you could set an interval to query the database every so often and broadcast the updates. This will still greatly reduce the number of queries, since no matter how many clients are connected there will only be one query, the response of which then gets sent to all connected clients.
I hope my post has give you some ideas of how you could implement your application using Node. Keep in mind though that I am just one developer, this is how I would approach building your application in Node. There will definitely be others who have different opinions.

How to get notified about external changes in a mongo collection with sails?

I am developing an application based on mongo and sails, and i am testing how the realtime update in sails works.
I am using sails 0.9.16 now, but i am interested also in answers about sails 0.10.
I want a list to be updated when new documents are created in the corresponding collection. This works when i add documents via sails sockets, sending a post message. In that case i see other clients receiving a message and the list on the client side is updated.
There is an external service writing on the mongo database tough, so the collection is growing all the times. The new elements created directly by the external service in the database are not notified to listening clients, so i have to refresh the web page in order to show those elements.
Questions:
are notifications about database creations supposed to work, when those creations do not come from sails itself?
if yes, does this require some special configuration?
if not, what would be a recommended way to keep client side listing about a collection updated when the database is changing?
Cheers
Very interesting question, though not an unusual one: the guys from Meteor were having the same problem. Basically, without watching the DB you can't even scale your app horizontally, since one server process will have no idea on what data changes were made by another one.
So, at first they sort of patched it by polling the DB every 10 seconds. :) Obviously, it wasn't the best solution, so they ended up with another one (which can also work for Sails): now they are tailing the MongoDB oplog and fire an update whenever there's a change in the corresponding collection.
That said, to answer your questions:
AFAIK, a Sails process has no clue about any external changes made to the DB;
so, nothing to configure;
a way to track external DB (MongoDB) updates would be using one of the oplog watchers you can find in npm (e.g. this or one of these, etc.) to listen to the changes and trigger updates whenever there's a need.
Unfortunately, no ready-to-use solution here, but I hope at least that now you have an idea on how to make it work.

Dynamic Wall updating: NodeJS

Basically, I have created a NODEJS app that uses Jade as its templating engine, along with Express and a MySQL database.
I am looking to create a new page which allows user to share a portion of text, and then a div underneath it named "Wall" will update dynamically with the new status.
Basically, it would ideally be similar to Facebook where something is typed, shared and then the page updates below dynamically. I'm also looking to have the wall page update when a new post have been shared from a users friend. All updates shared by the users would be sent to a database.
I have conducted a lot of searches but seem unable to gather a right answer.
I have narrowed it down to the use of either of the following: JQuery, Ajax, PHP.
Since the site I am building is built in JS - what is my best option?
I'm pretty new to all of this, but I assume when a user clicks share it calls a JS file which then stores the update in the database. But how do I get my "Wall" to refresh upon new content?
Any help greatly appreciated.
You've posed a conceptual question. So I'll do my best to explain some of the conceptual options you can choose to further explore and do your own research on how to best implement it with your project.
You have two paths to go here.
You can have your own wall update (do a refresh / re-render on the UI side) upon a successful AJAX write to your database, this would be something you implement in your AJAX callback function - basically the JS function that gets executed after your write request (submitting the new post) to the database returns successful.
A whole other branch of options you could explore, is implement either of the following options to basically "listen" in for changes server-side, and have the re-rendering react as the callback you use:
Polling - basically issue a request every X number of
seconds to check if there have been updates, or change of state on
server-side.
WebSockets - checkout Socket.io. Through this you can "push" messages from the server-side to your clients. As a note, WebSockets are not universally supported in every browser and from past experience I've found WebSocket protocols even differ by browser versions. So if you need universal support, I'd go with a polling method.
Good luck on your project, hope this helps!
Use...
setTimeout(function(){
/* update wall here */
}, 1000)
to poll your "Wall" backend and updated the content.

Categories