im running a multi tenant GAE app where each tenant could have from a few 1000 to 100k documents.
at this moment im trying to make a MVC javascript client app (the admin part of my app with spine.js) and i need CRUD endpoints and the ability to get a big amount of serialized objects at once. for this specific job appengine is way to slow. i tried to store serialized objects in the blobstore but between reading/writing and updating stuff to the blobstore it takes too much time and the app gets really slow.
i thought of using a nosql db on an external machine to do these operations over appengine.
a few options would be mongodb, couchdb or redis. but i am not sure about how good they perform with that much data and concurrent requests/inserts from different tenants.
lets say i have 20 tenants and each tenant has 50k docs. are these dbs capable to handle this load?
is this even the right way to go?
Why not use the much faster regular appengine datastore instead of blobstore? Simply store your documents in regular entities as Blob property. Just make sure the entity size doesn't exceed 1 MB in which case you have to split up your data into more then one entity. I run an application whith millions of large Blobs that way.
To further speed up things use memcache or even in-memory cache. Consider fetching your entites with eventual consistency which is MUCH faster. Run as many database ops in parallel as possible using either bulk operations or the async API.
The overhead of making calls from appengine to these external machines is going to be worse than the performance you're seeing now (I would expect). why not just move everything to a non-appengine machine?
I can't speak for couch, but mongo or redis are definitely capable of handling serious load as long as they are set up correctly and with enough horsepower for your needs.
Related
I was curious to know if there is any limit for data caching in
Single page applications using shared service or ngrx.
Does caching too much data on front end impacts the overall
performance of web Application (DOM).
Lets say I have a very big complex nested object which I am caching in memory
Now assume that I want to use different subsets of object in different modules/components of our
application and for that I may need to do lot of mapping operations(using loops by matching the id's etc) on UI.
I was thinking in other way around that instead of doing so much operations on UI to extract the
relevant data why don't I use a simple API with having id parameter to fetch the relevant information if its not taking much time to get the data from backend.
url = some/url/{id}
So is it worth to cache more complex nested objects if we cant use its subset simply by its properties
obj[prop] and need to do lot of calculations on UI (looping etc) which actually is more time consuming than getting the data from rest API ?
Any help/explanation will be appreciated !!!
Thanks
Caching too much data in memory is not a good idea. It will affect your application performance. Causes performance degradation in a system having less memory.
theoretically cache memory is for keeping less amount of data. The maximum support size is 2GB. I think chrome is also supported up to that limit.
For keeping data of big size in client-side never use memory cache instead you should use client-side database/datastore. It uses disk space instead of memory.
There are number of web technologies that store data on the client-side like
Indexed Database
Web SQL
LocalStorage
Cookies
Depending upon the client application framework it can be decided.
By default browser uses 10% of disk space for these data stores. We also have option to increase that size.
Currently I'm trying to learn nativescript and for this I thought about doing an App like 'Anki'
But while thinking about the data storage I stumpled upon the problem on how to save my flash cards locally for keeping the app offline (for example with SQLite), save the users time when to reflect each card (e.g. to show again in 10 minutes or 1 day) AND have an update functionality to update the database with new cards without deleting the users data.
What's the best way to solve that problem, especially when I want to provide the updates with an App-Update and without fetching everything from an external database?
I don't have any code yet, therefore a recommendation on how to solve that would be nice.
There is several methods in NativeScript you can use:
NativeScript-Sqlite (disclaimer: I'm the author)
This allows full access to Sqlite for saving and loading items; you can have as big of databases as you need and Sqlite is very fast. Sqlite's biggest drawback is speed of writes; if you have a LOT of writing it can be slower than just writing to a file yourself.
NativeScript-LocalStorage (disclaimer again: I'm the author)
This is more geared to smaller data sizes; as when the app starts and saves it has to load the entire json backed data store into memory. This is really fast over all; but not something you want to use for 10's of thousands of records.
NativeScript-Couchbase
This uses sqlite for local storage and can use couchbase for the remote storage; very nice for having syncable storage - couchbase can be your own server or a leased or rented server.
NativeScript-Firebase
This is also very useful for having syncable storage; however Google charges for FireBase at a certain point.
Built in AppSettings.
This is really designed for a few application settings, not designed for lots of data. But useful for the smaller amounts of data.
Role your own to the file system.
I have done this in a couple of my projects; basically a hybrid between my localstorage plugin and a mini-sql type system. One project was very much write dependent so it made more sense to generate the 20 or so separate files on the phone for each table because I could save them much quicker than inserting/replacing > 100,000 records each time the app started up into sqlite. Had minimal searching needs.
Your storage really needs to be dependent upon what you are doing; it is a balancing act. Lots of searchable data; sqlite wins in almost all cases. Lots of frequent writing; something you create might be a lot faster.
I have a javascript application that I've implemented for mobile apps using react-native and its desktop counterpart using the electron framework. The mobile application uses react-native-sqlite-storage native module to save preferences and data (5 - 6 tables) whereas I use node-sqlite3 for the electron app.
Both, the mobile and desktop apps share a lot of functionality but due to the use of different database plugins, have a lot of differences. Also, for the desktop app, as node-sqlite3 is a native dependency, I have to build the app installers for Windows and macOS separately. That's a pain!
So, what I need is a database solution that is :-
embeddable into the app
efficient and performant compared to sqlite3
supports syncing to a remote database
supports macOS, Windows, and Linux
encrypts the data written within the database
consistent API across JS runtimes (Browser / NodeJS / JavascriptCore)
Here's a list of those that I've come across and that seem appealing:-
NeDB
RxDB
PouchDB
So, what are your suggestions and how have you implemented anything similar for your apps?
I've been testing PouchDB, RxDB (which relies on PouchDB with RxJS streams for queries), Realm-JS (native database like sqlite3), FireStore. NeDB doesn't support remote sync.
I won't go into performance metrics details on each database, but PouchDB has been very slow and heavy on memory when querying more than 20.000 items (tried with indexeddb/websql adapter).
RxDB was generally much faster with that many items, especially when subscribing to query changes (tried with indexeddb/websql adapter too). Also schema and migrations are very handy.
FireStore is a good choice and comes with very easy setup of server and client components, but you'll need to be comfortable to run on a google platform. There is some flexibility with firebase functions if you want some control over server logic and there is customizable ACLs for your collections. Speed has been good and on par with RxDB. Comes with a very good auth module if you want it.
If you want to be able to scale much much more on the client side, which you probably don't, or if you want to make complex queries, I'd really recommend using something like realm. You'll need to compile for each platform like you've experienced with sqlite3, but there is sync, offline persistence, rich queries and great performance. There is just no way that javascript based solutions like PouchDB, RxDB or FireStore can compete, even with sqlite backends, since much of the computation will still happen in your precious JS thread. realm is doing much of its heavy lifting on the native library. I've been able to do LIKE "abc" queries on 100.000 items, returning hundreds of results, within less than hundred milliseconds and without noticeably freezing my UI or pumping up memory usage heavily. Supports client migrations too, which is nice.
In the end, there are multiple answers:
1. Want to host everything yourself and don't need massive scale client side (you can sync against subsets of your server data with filters), RxDB ist very good and comes with nice set of features.
2. Want very easy setup, nice modules like auth, server functions etc, and don't need massive scale on the client (can also sync against server data subsets with filters), FireStore is great.
3. Need lots of lots of data on the client, realm is my preference. I personally really dislike the realm sync platform for its pricing model (though technically it's cool), but the database itself is free and maybe you could try and implement a custom sync.
Take my results with a grain of salt, I've had some very specific challenges like large, non-relational collections and fulltext-search, your use-case will probably differ a lot.
I just started programming with node.js and before this I was using PHP and I used to store data on sql database. However, I am confused now. Should I use sql to save data or Json. I would need to save data and show them on the web page later on Or feed those data to draw graphs and charts.
When storing data in a Node.js application (or with any application really), you have three main options...
Memory
You can simply store the data as a variable in the application, this
will most likely be fastest to access.
You may start to encounter problems if you are storing lots of data
as you will use up a large proportion of your machine's memory.
All data will be lost when the script restarts, so the data should be
able to be generated again on startup.
If you use a process manager like PM2 (which you'll want to do if
hosting on a multi-core machine, see https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-16-04), you won't be able to store data across multiple requests as processes will not share a common memory.
I/O
You could also encode data as JSON and store it in a file on your machine.
I can't think of a case where this would be better than using a DB, I/O is usually just used for static files like images.
The obvious drawback of this is that it will probably be slower than database calls but you'd also have to watch out for I/O concurrency issues if you're using miultiple processes.
However, if you really want to store JSON, consider a non-relational DB like Mongo DB instead.
Database
This would be my preferred method for almost every case (apart from static files or very short-lived data, as I mentioned above).
A DB also has helpful ways of storing and finding data - you'll need to choose between a relational or non-relational DB, take a look at https://www.mongodb.com/scale/relational-vs-non-relational-database for some of the differences.
Again, the main thing to be wary of is concurrency when using multiple processes (usually only a problem when updating or inserting records).
I hope that covers your question - I may have missed something but I'm sure others will let me know if I have. Node.js is a great choice for web servers as it is asynchronous so can naturally handle multiple connections at once.
I'm building my first site using this framework, i'm remaking a website i had done in PHP+mySQL and wish to know something about performance... In my website, i have two kinds of content:
Blog Posts (for 2 sections of the site) - these have the tendency to one day sum to thousands of records and, are more often updated
Static (sort of) data: this is information i keep in the database, like site section's data (title, metatags, header image url, fixed html content, javascript and css filenames to include in that section), that is rarely updated and it's very small in size.
While i was learning the basics on nodeJS, i started thinking of a way to improve the performance of the website, in a way i couldn't do with PHP. So, what i'm doing is:
When i run the app, the static content is all loaded into memory, i have a "model" Object for each content that stores the data in an array, has a method to refresh that data, ie, when the administrator updates something, i call refresh() to go get the new data from the database to that array. In this way, for every page load, instead of querying the database, the app queries the object in memory directly.
What i would like to know is if there should be any increase of performance, working with objects directly in memory or if constant queries to the database would work just as good or even better.
Any documentation supporting your answer will be much appreciated.
Thanks
In terms of the general database performance, MongoDB will keep your working set in memory - that's its basic method of operation.
So, as long as there is no memory contention to cause the data to get swapped out, and it is not too large to fit into your physical RAM, then the queries to the database should be extremely fast (in the sub millisecond range once you have your data set paged in initially).
Of course, if the database is on a different host then you have network latency to think about and such, but theoretically you can treat them as the same until you have a reason to question it.
I don't think there will be any performance difference. First thing is that this static data is probably not so big (up to 100 records?) and querying DB for it is not a big deal. Second thing (more important) is that most DB engines (including mongoDB) have caching systems built-in (although I'm not sure how they work in details). Third thing is that holding query results in memory does not scale well (for big websites) unless you use storage engine like Redis. And that's my opinion, although I'm not the expert.