I have a javascript application that I've implemented for mobile apps using react-native and its desktop counterpart using the electron framework. The mobile application uses react-native-sqlite-storage native module to save preferences and data (5 - 6 tables) whereas I use node-sqlite3 for the electron app.
Both, the mobile and desktop apps share a lot of functionality but due to the use of different database plugins, have a lot of differences. Also, for the desktop app, as node-sqlite3 is a native dependency, I have to build the app installers for Windows and macOS separately. That's a pain!
So, what I need is a database solution that is :-
embeddable into the app
efficient and performant compared to sqlite3
supports syncing to a remote database
supports macOS, Windows, and Linux
encrypts the data written within the database
consistent API across JS runtimes (Browser / NodeJS / JavascriptCore)
Here's a list of those that I've come across and that seem appealing:-
NeDB
RxDB
PouchDB
So, what are your suggestions and how have you implemented anything similar for your apps?
I've been testing PouchDB, RxDB (which relies on PouchDB with RxJS streams for queries), Realm-JS (native database like sqlite3), FireStore. NeDB doesn't support remote sync.
I won't go into performance metrics details on each database, but PouchDB has been very slow and heavy on memory when querying more than 20.000 items (tried with indexeddb/websql adapter).
RxDB was generally much faster with that many items, especially when subscribing to query changes (tried with indexeddb/websql adapter too). Also schema and migrations are very handy.
FireStore is a good choice and comes with very easy setup of server and client components, but you'll need to be comfortable to run on a google platform. There is some flexibility with firebase functions if you want some control over server logic and there is customizable ACLs for your collections. Speed has been good and on par with RxDB. Comes with a very good auth module if you want it.
If you want to be able to scale much much more on the client side, which you probably don't, or if you want to make complex queries, I'd really recommend using something like realm. You'll need to compile for each platform like you've experienced with sqlite3, but there is sync, offline persistence, rich queries and great performance. There is just no way that javascript based solutions like PouchDB, RxDB or FireStore can compete, even with sqlite backends, since much of the computation will still happen in your precious JS thread. realm is doing much of its heavy lifting on the native library. I've been able to do LIKE "abc" queries on 100.000 items, returning hundreds of results, within less than hundred milliseconds and without noticeably freezing my UI or pumping up memory usage heavily. Supports client migrations too, which is nice.
In the end, there are multiple answers:
1. Want to host everything yourself and don't need massive scale client side (you can sync against subsets of your server data with filters), RxDB ist very good and comes with nice set of features.
2. Want very easy setup, nice modules like auth, server functions etc, and don't need massive scale on the client (can also sync against server data subsets with filters), FireStore is great.
3. Need lots of lots of data on the client, realm is my preference. I personally really dislike the realm sync platform for its pricing model (though technically it's cool), but the database itself is free and maybe you could try and implement a custom sync.
Take my results with a grain of salt, I've had some very specific challenges like large, non-relational collections and fulltext-search, your use-case will probably differ a lot.
Related
I am creating an app in Electron using React. The first version will be a purely desktop based app using a local database for data storage etc. However, eventually I would like to utilise as much of the same code base as possible and deploy the same app as a cloud-based alternative (by just leaving out the electron component).
I'd like some advice, thoughts, opinions from the community on if I should use axios for creating internal restAPI infrastructure rather than a API using javascript modules/functions etc. for performing requests such as getting data from a database.
My thinking is then that when I port the application to the cloud, I really just need to change the restAPI locations to point to a cloud version which would then get the data from a cloud database rather than a local database etc.
Would exposing such APIs locally within Electron using axios pose any security risks or other issues that I may need to consider?
Perhaps this answer/discussion may also provide useful for the community in the future.
Looking forward to thoughts, or even new suggestions on how this may be best approached.
Thanks.
I'm still new to web development. To learn more about JavaScript(JS) and web development, I am thinking of writing a simple web app which pulls and records time-series data (say, the price of a stock) periodically and draws a live chart showing the historical data. In addition to price data, I would like the app to record/maintain some user-related info such as the ticker of the stock(s) associated to each user.
Ideally, I would like to keep the app light-weight and portable/standalone (meaning, reduce the dependency as much as possible, and the end user hopefully doesn't have to do a lot of configuration/install of dependencies). The issue that I cannot figure out is where to store the historical data. I looked around for database solutions which will allow the app to write data directly from the browser (that is, using JS) to the client's machine. LocalStorage and IndexDB are non-persistent as far as I understand. Some suggested using PouchDB, but upon looking at it closer, it seems like the user need to install CouchDB or some compatible DB (say, SQLite). But that means I cannot share my app with users who aren't technical enough to install and configure CouchDB or SQLite on their machine before using my app.
If anyone could share some insights as to which DB might allow a JS-based app to write persistent data to the client's machine (if such thing even exist), that would be greatly helpful. If there is no such DB solution, please feel free to let me know alternative solutions that would allow the goal of building a simple, portable, JS-based web app. Thank you!
I think the best solution is to use Electron.js. The whole idea of this framework is to create web apps that can reside on client machines. You could package up any DB option you want, or even better, just include an API to your backend through the web app and it will work on your client machine like I think you want it to.
As for DB options, there is a great thread on S.O. that talks about what is possible. It looks like knex.js is your best bet (full disclosure - I haven't used knex).
I'm building a relatively complex and data heavy web application in AngularJS. I'm planning to use php as a RESTful backend (with symfony2 and FOSRESTbundle). I have spent weeks looking around for different solutions to on/off line synchronization solutions and there seem to be many half solutions (see list below for some examples). But non of them seem to fit my situation perfectly. How do I go about deciding which strategy will suite me?
What issues that might determine “best practices” for building an on/off line synchronization system in AngularJS and symfony 2 needs some research, but on the top of my head I want to consider things like speed, ease of implementation, future proof (lasting solution), extensibility, resource usage/requirements on the client side, having multiple offline users editing the same data, how much and what type of data to store.
Some of my requirements that I'm presently aware of are:
The users will be offline often and then needs to synchronize (locally created) data with the database
Multiple users share some of the editable data (potential merging issues needs to be considered).
User's might be logged in from multiple devices at the same time.
Allowing large amount of data to be stored offline(up to a gigabyte)
I probably want the user to be able to decide what he wants to store locally.
Even if the user is online I probably want the user to be able to choose whether he uses all (backend) data or only what's available locally.
Some potential example solutions
PouchDB - Interesting strategies for synchronizing changes from multiple sources
Racer - Node lib for realtime sync, build on ShareJS
Meteor - DDP and strategies for sync
ShareJS - Node.js operational transformation, inspired by Google Wave
Restangular - Alternative to $resource
EmberData - EmberJS’s ORM-like data persistence library
ServiceWorker
IndexedDB Polyfill - Polyfill IndexedDB with browsers that support WebSQL (Safari)
BreezeJS
JayData
Loopback’s ORM
ActiveRecord
BackBone Models
lawnchair - Lightweight client-side DB lib from Brian Leroux
TogetherJS - Mozilla Labs’ multi-client state sync/collaboration lib.
localForage - Mozilla’s DOMStorage improvement library.
Orbit.js - Content synchronization library
(https://docs.google.com/document/d/1DMacL7iwjSMPP0ytZfugpU4v0PWUK0BT6lhyaVEmlBQ/edit#heading=h.864mpiz510wz)
Any help would be much appreciated :)
You seem to want a lot of stuff, the sync stuff is hard... I have a solution to some of this stuff in an OSS library I am developing. The idea is that it does versioning of local data, so you can figure out what has changed and therefore do meaningful sync, which also includes conflict resolution etc. This is sort-of the offline meteor as it is really tuned to offline use (for the London Underground where we have no mobile data signals).
I have also developed an eco system around it which includes a connection manager and server. The main project is at https://github.com/forbesmyester/SyncIt and is very well documented and tested. The test app for the ecosystem will be at https://github.com/forbesmyester/SyncItTodoMvc but I have yet to write virtually any docs for it.
It is currently using LocalStorage but will be easy to move to localForage as it actually is using a wrapper around localStorage to make it an async API... Another one for the list maybe?
To work offline with your requeriments I suggest to divide problem into two scenarios: content (html, js, css) and data (API REST).
The content
Will be stored offline by appcache for small apps or for advanced cases with the awesome serviceworkers. Chrome 40+.
The data
Require solve the storage and synchronization and It becames a more difficult problem.
I suggest a deep reading of the Differential Synchronization algorimth, and take next tips in consideration:
Frontend
Store the resource and shadow (using for example url as key) into the localstorage for small apps or into more advanced alternatives (pouchdb,indexdb,...). With the resource you could work offline and when needs synchronize with the server use jsonpath to get diffs between the resource-shadow and to send it to server the PATCH request.
Backend
At backend take in consideration storage the shadow copies into redis.
The two sides (Frontend/Backend) needs to identify the client node, to do so you could use x- syn-token at HTTP header (send it in all request of the client with angular interceptors).
https://www.firebase.com/
it's reliable and proven, and can be used as a backend and sync library for what you're after. but, it costs, and requires some integration coding.
https://goinstant.com/ is also a good hosted option.
In some of my apps, I prefer to have both: syncing db source AND another main database. (mogno/express, php/mysql, etc..). then each db handles what's its best with, and it's features (real-time vs. security, etc...). This is true regardless to sync-db provider (be it Racer or Firebase or GoInstant ...)
The app I am developing has many of the same requirements and is being built in AngularJS. In terms of future proofing, there are two main concerns that I have found, one is hacking attempts requiring encryption and possible use of one time keys and an backend key manager and the other is support for WebSQL being dropped by the standards consortium in preference to indesedDB. So finding an abstraction layer that can support both is important. The solution set I have come up with is fairly straight forward. Where offline data is is loaded first into the UI and a request goes out to the REST Server if in an online state. As for resolving data conflicts in a multi user environment, that becomes a business rule decision. My decision was to simplify the matter and not delve into data mergers but to use a microtime stamp comparison to determine which version should be kept and pushed out to clients. When in offline mode, store data as a dirty write and the push to server when returning to an online state.
Or use ydn-db, which I am evaluating now as it has built in support for AWS and Google cloud storage built in.
Another suggestion:
Yjs leverages an OT-like algorithm to share a wide range of supported data types, and you have the option to store the shared data in IndexedDB (so it is available for offline editing).
For a couple recent projects on our corporate intranet, I have used a very simple stack of nginx + redis + webdis + client-side javascript to implement some simple data analysis tools. The experience was absolutely wonderful, especially compared to my previous experience with other stacks (including custom c++, apache/mod_perl, ASP.Net MVC, .Net HttpListener, Ruby on Rails, and a bit of Node.js). Given the availability of client-side templating tools and frontend libraries such as jquery-ui, it seems that I could happily implement much more complicated web-apps using such a no-server-side-code stack (perhaps substituting/augmenting redis with couchdb if warranted)...
The major limitation of this stack, of course, is that my database is directly exposed to the network - acceptable in this case on a firewalled corporate network, but not really an option if I wanted to use the same techniques on the internet. I need to have some level of server-side logic to securely handle authentication and user-role management.
Are there any best practices or common development stacks for this? Ideally I'd like something that is lightweight, and gives me a simple framework for filtering the client-side requests through my custom user-role logic before forwarding them on to the database back-end. I'm not interested in any sort of server-side templating, or ActiveRecord-style storage-level abstractions.
I can't comment on a framework.
You've already mentioned the primary weakness of this, especially on the internet, that being security. The problem there is not just authentication. The problem there is essentially the openness of the client, in this case the web browser, and the protocol, notably HTTP using JSON or XML or some other plaintext protocol.
Consider one example. It's quite simple. Imagine an HTTP service that takes an SQL query and returns a collection of JSON representing the rows. This is straightforward to write. You could probably pound out a nascent one in less than an hour from scratch using any tool that gives you SQL access to an RDBMS.
Arguably, back in the Golden Days of Client Server development, this is exactly what folks did, only instead of a some data tunneled over HTTP, folks used a DB specific driver and sent SQL text over to the back in DB directly.
The problem today is that the protocols are too open. If you implemented that SQL service mentioned above, you essentially turn your entire application in a SQL injection vector.
You simply can not secure something like that in the wild. The protocol is open to trivial observation (every browser comes with a built in packet sniffer, effectively today), along with all of the source code for the application. If you try to encrypt the data, that's all done on the client as well -- with the source to the process, as well as any keys involved.
CouchDB, for example, can not be secured this way. If someone has rights to the server, they have rights to all of the data. ALL of the data. The stuff you want them to see, and the stuff you don't.
The solution, naturally, is a service layer. Something that speaks at a higher level than simply raw data streams. Something that can be secured, and can keep secrets from the clients. But that, naturally, takes server side programming to enable, and its a ostensibly more work, more layers, more data conversion, more a pain.
Back in the day, folks would write entire systems using nothing but stored procedures in the DB. The procedures would have rights that the users invoking them did not, thus you could limit at the server what a user could or could not see or change. You could given them unlimited SELECT capability on a restricted view, perhaps, while a stored procedure would have rights to actually change data or access some of the hidden columns.
Stored procedures have mostly been replaced by application layers and application servers, with the DB being more and more relegated to "dumb storage". But the concepts are similar.
There's value for some scenarios to publishing data straight to the web, like you analytics example. That's a specific, read heavy niche. But beyond that, the concept doesn't work well, I fear. Obfuscated JS is hard to read, but not secure.
This is likely why you may have a little difficulty locating such a framework (I haven't looked at all, myself).
I'm wondering if using Node.js (express.js) is the best option to go for a complete website. I've been using Rails + Node.js and am wondering if this is still the best option or if I should move completely to Node.js.
Are there websites built completely with node.js? I imagine it must lack many things Rails offers.
I imagine it must lack many things
Rails offers.
I gave a short list below but I would like to know what you could be missing and I think(almost certain) we can give you some alternatives on node.js.
Modules
node.js is getting pretty complete and I think you can run your entire site using only node.js using for example the following modules. I assume you know about npm. If not I really advice you to google for it and learn that. To search npm you can use http://search.npmjs.org:
web framework:
High performance, high class web
development for Node.js
https://github.com/visionmedia/express
Socket.IO aims to make realtime apps possible in every browser and mobile
device, blurring the differences
between the different transport
mechanisms.
https://github.com/learnboost/socket.io-node
I believe with these two web-frameworks you can create a lot of sites. For example express is a very powerfull web framework and supports a lot of cool things like:
session support.
a lot powerful template engines. I like Jade for example. You could also share these between client and server a lot of the times easily.
excellent routing.
just to name a few.
database:
Redis is an open source, advanced
key-value store. It is often referred
to as a data structure server since
keys can contain strings, hashes,
lists, sets and sorted sets.
https://github.com/mranney/node_redis
MongoDB (from "humongous") is a scalable, high-performance, open source,
document-oriented database.
Mongoose is a MongoDB object modeling tool designed to work in an
asychronous environment.
https://github.com/learnboost/mongoose/
With those two databases I think you should be able to accomplish most of your tasks. The nice thing is that Redis is extremely fast/simple advanced KV-store(dataset in memory/also supports VM) and supports replication while Mongodb is more completely(slower then redis) and also supports sharding.
authentication:
Authentication and authorization (password, facebook, & more) for your node.js Connect and Express apps.
https://github.com/bnoguchi/everyauth
Like I said previously you can get a long way with only these modules.
Express.js is more akin to Sinatra. They're simpler frameworks than rails.
Express's list of sites is fairly small http://expressjs.com/applications.html
So I think it's also good to look at Sinatra's list http://www.sinatrarb.com/wild.html
So to answer your questions in reverse order. Yes it lacks all the features of rails. Yes there are sites written completely in Express. And going to 100% Node.js might be the right decision for your site.
It depends on which features you'd miss and what performance you need.
I'm not sure, but I guess it's perfectly possible. I have built complete sites using plain server side javascript for years without problems. The advantage of node.js seems to be its event driven model and things like socket.io. I just started experimenting with it, I'll probably will try porting an existing site to node.js.
Here you can find a large list of sites built with node.js.
Finally, you may want to read: What it’s like building a real website in Node.js
Yes, as of now node.js lacks many (well, at least a few) things rails offers. Eventually the set of available node.js modules will collectively provide good alternatives to RoR across the board. Or at least different (and often more modern) approaches to the same fundamental problems. There are still some important things missing in the node.js ecosystem including a good ORM for postgresql (rails has ActiveRecord which is great) and a good DB schema management subsystem. Both of these do not exist in the node.js ecosystem as of this moment, but surely they will be there in due time. Rails has these down pat right now.
There are some tricks node.js has that RoR doesn't. Debugging in node.js is more seamless than RoR and express.js is more flexible. Express is the library approach (you tell it what to do) whereas rails is the framework approach (you fill out the boilerplate it creates for you). There are also some fantastic things like stylus and jade, which have equivalents in rails, but when you have a full app written in one language (either javascript or CoffeeScript, which I prefer) and everything is in a modern node.js/TJ Holowaychuck style, you get a level of cohesiveness that is really nice to work within.
The other thing to keep in mind is that while the list of available node.js modules is quite impressive, many of them are not as mature and battle-tested as their rails/django analogs. It's hit or miss, so beware.