I'm programming a browser application (html5+websockets+css3+js preferred) that enables users to concurrently access (read, write) attributes of the same object. To create a real-time experience I'd like to use optimistic synchronization. I read about Timewarp and Trailing State algorithms and I wonder if there is a javascript library, which already implements these or similar algorithms.
I found this question, but unfortunately it was not answered yet. XSTM only supports pessimistic synchronization as it seems.
Do you have any idea for me?
I am working on a realtime HTML5 web browser application now too. Maybe my choice of weaponry could inspire you...who knows, so I am using:
Frontend:
KnockoutJS -it takes care of displaying data which I send to every connected client in JSON(view models), you can easily subscribe to changes in the client data and push the changes back to server, though I am having problems displaying pages with knockoutjs on mobile browsers
on serverside I run custom made server based on Fleck
Since JSON is my favourite data format, I ditched SQL databases in favour of [RavenDB][2], which stores data almost exactly as they are sent via websocket protocol and also it is pretty quick
Related
Being relatively new to the web world, I encountered a (pretty common?) problem.
Developping a multiplayer game, I'm having trouble handling disconnections. Server and clients communicate through websockets, so whenever a client is disconnected, he misses on the notifications sent by the other clients (routed by the server). Which means that :
whenever a client is disconnected, I need to record all changes happening afterwards.
when he connects back, I need to send him all this info
the client then needs to be able to replay these records to sync with the server state.
I was wondering if there was some kind of design pattern to handle this problem (rather than an implementation specific solution). I believe it's kind of what React, Vue etc... are meant for (?), but I'd rather avoid using any heavy framework.
Any ressource/doc would also be appreciated ;)
Transaction in Distributed Systems is a related technique (but it is not categorized as a design pattern.)
Memento pattern provides the ability to restore an object to its previous state, so it's also somehow related.
The approach you described in the question seems to be good. Maybe you don't need a pattern to fit in.
I have created an application using javascript library D3. Users will constantly click and drag to frequently change graphical elements and I currently save the data in 3-4 local javascript objects and arrays. I want to save the data to the server periodically rather than after each change. Also I want them to be able to work if they are not connected. From twenty years ago, I imagine doing this manually where on the client side records are flagged as “new”, “revised”, and “deleted”. Every 10 seconds client data is saved via AJAX and either an object is updated or a SQL statement is executed. An id is returned from the database and saved on the client side to track each record for future modifications.
Note the data must be organized in a database for ease of separating elements for reuse. When the user is connected, updates every 5-10 seconds are fine. Then I can use an inexpensive and slow server. Of course a tool that deals with records that might not fully update is good, perhaps some transactional functionality.
There will be no separate mobile application. I can modify my javascript objects to be json compliant if need be. I see there are “offline-first” frameworks and javascript "state containers". Redux caught my eye, especially when I saw its use climbing over the years according to Google Trends. I’ve read about so many options and am thoroughly confused by all these. Here is a mish mash of tools I looked at: Store.js, now.js, indexedDB, couchDB, pouchDB, Cloudant, localForage, WebSQL, Polymer App Toolbox, Hoodie framework, Ionic and angular, and Loopback. Not to mention XHR, web sockets.
I have used MVC like Laravel and Zend, both are with PHP and MySql. I wonder if I could integrate the suggested solution. Thanks.
Related: How do I sync data with remote database in case of offline-first applications?
Saving the data locally using PouchDb and then syncing it with a CouchDb database (or IBM's Cloudant service) when a network connection is available is a well-trodden path for this sort of requirement. But your question is asking for an opinion, so there will be many other perfectly valid solutions to this.
I've read a few StackOverflow posts related to this subject but I can't find anything specifically helps me in my scenario.
We have multiple monitoring instances within our network, monitoring different environments (Nagios, Icinga, more...). Currently I have a poller script written in PHP which runs every minute via cron, it asks the instance to return all of its problems in JSON, the script then interprets this and pushes it in to a MySQL database.
There is then an 'overview' page which simply reads the database and does some formatting. There's a bit of AJAX involved, every X seconds (currently use 30) it checks for changes (PHP script call) and if there are changes it requests them via AJAX and updates the page.
There's a few other little bits too (click a problem, another AJAX request goes off to fetch problem details to display in a modal etc).
I've always been a PHP/MySQL dev, so the above methodology seemed logical to me and was quick/easy to write, and it works 'ok'. However, the problems are: database constantly being polled by many users, mesh of javascript on the front end doing half the logic and PHP on the back doing the other half.
Would this use case benefit from switching to NodeJS? I've done a bit of Node.JS before but nothing like this. Can I subscribe to MySQL updates? Or trigger them when a 'data fetcher' pushes data in to the database? I've always been a bit confused as I use PHP to create data and javascript to 'draw' the page, is there still a split of NodeJS doing logic and front end javascript creating all the elements, or does NodeJS do all of this now? Sorry for the lack of knowledge in this area...
This is definitely an area where Node could offer improvements.
The short version: with websockets in the front-end and regular sockets or an API on the back-end you can eliminate the polling for new data across the board.
The long version:
Front-end:
You can remove all need for polling scripts by implementing websockets. That way, as soon as new data arrives on the server, you can broadcast it to all connected clients. I would advise Socket.io or the Primus websocket wrapper. Both are very easy to implement and incredibly powerful for what you want to achieve.
All data processing logic should happen on the server. The data is then sent to the client and should be rendered on the existing page, and that is basically the only logic the client should contain. There are some frameworks that do all of this for you (e.g. Sails) but I don't have experience with any of those frameworks, since they require you to write your entire app according to their rules, which I personally don't like (but I know a lot of developers do).
If you want to render the data in the client without a huge framework, I highly recommend the lightweight but incredibly useful Transparency rendering library. Using this, you can format a Javascript object on the server using Node, JSONify it, send it to the client, and then all the client would have to do is de-JSONify it and call Transparency's .render.
Back-end:
This one depends on how much control you have over the behaviour of the instances you need to check. I assume you have some control, since you can get all their data in a nice JSON format. So, there are multiple options.
You can keep polling every so often. This is the easiest solution since it requires no change to the external services. The Javascript setInterval function is very useful here. Depending on how you connect with the instances, you might be able to use a module like Request to do the actual request, so that takes out a bunch more of the heavy lifting.
The benefit of implementing the polling in your Node app as well, is that you will receive the data in your Node app and that way you can immediately broadcast it to the clients, even before inserting it into a database. This will greatly reduce the number of queries on your database.
An alternative to polling would be to set up a simple Express-based API where the applications can post their 'problems', as you call them. This way your application will get notified the moment a problem occurs, and combined with the websockets connection to the client this would result in practically real-time updates.
To be more redundant, you would have a polling timer alongside the API, so that you can check the instances in case there's something wrong that causes them to not send over any more data.
An alternative to the more high-level API would be to just use direct socket communication, which is basically the same approach only using a different set of functions.
Lastly, you could also keep the PHP-based polling script. This would be the most efficient solution since you wouldn't go and replace everything. Then from the Node app that's connected to the clients with websockets, you could set an interval to query the database every so often and broadcast the updates. This will still greatly reduce the number of queries, since no matter how many clients are connected there will only be one query, the response of which then gets sent to all connected clients.
I hope my post has give you some ideas of how you could implement your application using Node. Keep in mind though that I am just one developer, this is how I would approach building your application in Node. There will definitely be others who have different opinions.
I am developing a webgl application. I am confused in choosing the right server-side technology to use for the application. I an using mongoDB as my database and three.js as my webgl library to ease the development.
There are a lot of things to take into consideration.
first of all, what kind of connection do you need?
There are 3 ways of getting data from a server. Polling, long-polling, and a bidirectional connection.
Polling is the simplest. It's simply sending an AJAX call to a php or asp server, and the server replies.
Long polling is a little more complex. It's sending a request to the server, and the server decides when to answer this call. This can also be done using a simple php/asp script.
A bidirectional connection is when things really become complicated and you need to use websockets. As far as i know neither php or asp support websockets so node.js, for me, is the only option here. Its setting up a connection between client and server so you can pass data back and forth. It doesnt rely on the client requesting data, the server can also decide to sent data to the client, which in turn can make something happen.
As for the databases, i also found mongoDB to work great with node.js, i tried it and havent even looked at SQL since then because quite frankly, i absolutely dispise SQL. So its just a matter of what your personal preference is.
And finally the webGL framework. It also depends on what you want and how complicated you want to make it. If you also need physics and/or collision detection (while three.js does have a small collision detection system) i'd take a look at babylon.js as well.
I'm building a relatively complex and data heavy web application in AngularJS. I'm planning to use php as a RESTful backend (with symfony2 and FOSRESTbundle). I have spent weeks looking around for different solutions to on/off line synchronization solutions and there seem to be many half solutions (see list below for some examples). But non of them seem to fit my situation perfectly. How do I go about deciding which strategy will suite me?
What issues that might determine “best practices” for building an on/off line synchronization system in AngularJS and symfony 2 needs some research, but on the top of my head I want to consider things like speed, ease of implementation, future proof (lasting solution), extensibility, resource usage/requirements on the client side, having multiple offline users editing the same data, how much and what type of data to store.
Some of my requirements that I'm presently aware of are:
The users will be offline often and then needs to synchronize (locally created) data with the database
Multiple users share some of the editable data (potential merging issues needs to be considered).
User's might be logged in from multiple devices at the same time.
Allowing large amount of data to be stored offline(up to a gigabyte)
I probably want the user to be able to decide what he wants to store locally.
Even if the user is online I probably want the user to be able to choose whether he uses all (backend) data or only what's available locally.
Some potential example solutions
PouchDB - Interesting strategies for synchronizing changes from multiple sources
Racer - Node lib for realtime sync, build on ShareJS
Meteor - DDP and strategies for sync
ShareJS - Node.js operational transformation, inspired by Google Wave
Restangular - Alternative to $resource
EmberData - EmberJS’s ORM-like data persistence library
ServiceWorker
IndexedDB Polyfill - Polyfill IndexedDB with browsers that support WebSQL (Safari)
BreezeJS
JayData
Loopback’s ORM
ActiveRecord
BackBone Models
lawnchair - Lightweight client-side DB lib from Brian Leroux
TogetherJS - Mozilla Labs’ multi-client state sync/collaboration lib.
localForage - Mozilla’s DOMStorage improvement library.
Orbit.js - Content synchronization library
(https://docs.google.com/document/d/1DMacL7iwjSMPP0ytZfugpU4v0PWUK0BT6lhyaVEmlBQ/edit#heading=h.864mpiz510wz)
Any help would be much appreciated :)
You seem to want a lot of stuff, the sync stuff is hard... I have a solution to some of this stuff in an OSS library I am developing. The idea is that it does versioning of local data, so you can figure out what has changed and therefore do meaningful sync, which also includes conflict resolution etc. This is sort-of the offline meteor as it is really tuned to offline use (for the London Underground where we have no mobile data signals).
I have also developed an eco system around it which includes a connection manager and server. The main project is at https://github.com/forbesmyester/SyncIt and is very well documented and tested. The test app for the ecosystem will be at https://github.com/forbesmyester/SyncItTodoMvc but I have yet to write virtually any docs for it.
It is currently using LocalStorage but will be easy to move to localForage as it actually is using a wrapper around localStorage to make it an async API... Another one for the list maybe?
To work offline with your requeriments I suggest to divide problem into two scenarios: content (html, js, css) and data (API REST).
The content
Will be stored offline by appcache for small apps or for advanced cases with the awesome serviceworkers. Chrome 40+.
The data
Require solve the storage and synchronization and It becames a more difficult problem.
I suggest a deep reading of the Differential Synchronization algorimth, and take next tips in consideration:
Frontend
Store the resource and shadow (using for example url as key) into the localstorage for small apps or into more advanced alternatives (pouchdb,indexdb,...). With the resource you could work offline and when needs synchronize with the server use jsonpath to get diffs between the resource-shadow and to send it to server the PATCH request.
Backend
At backend take in consideration storage the shadow copies into redis.
The two sides (Frontend/Backend) needs to identify the client node, to do so you could use x- syn-token at HTTP header (send it in all request of the client with angular interceptors).
https://www.firebase.com/
it's reliable and proven, and can be used as a backend and sync library for what you're after. but, it costs, and requires some integration coding.
https://goinstant.com/ is also a good hosted option.
In some of my apps, I prefer to have both: syncing db source AND another main database. (mogno/express, php/mysql, etc..). then each db handles what's its best with, and it's features (real-time vs. security, etc...). This is true regardless to sync-db provider (be it Racer or Firebase or GoInstant ...)
The app I am developing has many of the same requirements and is being built in AngularJS. In terms of future proofing, there are two main concerns that I have found, one is hacking attempts requiring encryption and possible use of one time keys and an backend key manager and the other is support for WebSQL being dropped by the standards consortium in preference to indesedDB. So finding an abstraction layer that can support both is important. The solution set I have come up with is fairly straight forward. Where offline data is is loaded first into the UI and a request goes out to the REST Server if in an online state. As for resolving data conflicts in a multi user environment, that becomes a business rule decision. My decision was to simplify the matter and not delve into data mergers but to use a microtime stamp comparison to determine which version should be kept and pushed out to clients. When in offline mode, store data as a dirty write and the push to server when returning to an online state.
Or use ydn-db, which I am evaluating now as it has built in support for AWS and Google cloud storage built in.
Another suggestion:
Yjs leverages an OT-like algorithm to share a wide range of supported data types, and you have the option to store the shared data in IndexedDB (so it is available for offline editing).