We Keep Coding

pouchdb or alternative where I can control how much data stored locally?

In the design stage for an app that collects large amounts of data...
Ideally, I want it to be an offline-first app and was looking to Pouchdb/Counchdb - However, the data needs to be kept for years for legal reasons, and my concern is that this is going to consume too much local storage over time.
My thoughts were:
handle sync between pouchdb and couchdb myself, allowing me to purge inactive documents from the local store without impacting the couchdb. This feels messy and probably a lot of work
Build a local store using dexie.js and completely write the sync function. It also looks hard work, but may be less as I'm not trying to mess with a sync function
Search harder :)
Conceptually, I guess I'm looking for a 'DB cache' - holding active json document versions and removing documents that have not been touched for X period. It might be that 'offline' mode is handled separate to the DB cache..

Not sure yet if this is the correct answer..
setup a filter on couchdb to screen out old documents (lets say we have a 'date_modified' field in the doc and we filter out any docs with date_modified older than one month)
have a local routine on the client that deletes documents from the local pouchdb that are older than one month ( actually using the remove() method against the local pouchdb, not updating it with _deleted:true) - from https://pouchdb.com/2015/04/05/filtered-replication.html it appears removed documents don't sync.
docs updated on the Pouchdb will replicate normally
there might be a race condition here for replication, we'll see

Optimal way to check out rows in PostgreSQL by multiple scalable worker processes

I have a very large table with 100,000's of rows with a key and a timestamp.
I currently use batch of AWS servers that are using NodeJS and PG to query for the oldest last_updated timestamp, perform the function, and then update the row with the NOW().
My issue is I am trying to find a way scale with more processes and optimize my row checkouts. I started with each process selecting 1000 at a time and each one having a multiple of 1000 for OFFSET, doing the operation, and then updating
I started to read about SELECT using FOR UPDATE and SKIP LOCK but seems like this could have some performance impacts? Also can't find a clear way to do the SELECT/UPDATE in the same query or do I keep doing single updates at a time similar to this post but seems like this may not be good for larger operations like this?
implementing an UPDATE on SELECT in Postgres
Has anyone approached this type of setup? I also have been debating do I need to build my own middleware that manages a pool of work items and then the workers use that table to select/delete?

Meteor: display list of current users for document

I have a list of documents in meteor that only authorised users can access. It's kind of like google docs. Is there an easy way to get a list of users which are currently viewing them?
I thought of including a "currentUsers" field in my mongodb object, and push/remove users whenever users view or stop viewing the component. This feels like a strange way to do it since data is persisted and is probably prone to errors since it doesn't exactly represent users currently viewing the component at a moment in time
This stackoverflow question is kind of what I'm looking for, but the answer is a little old, and I'm not sure how to go about using sockjs either. If someone can provide a working example that will be great.
Any help is greatly appreciated!

There are several options:
DIY solution - storing data in Collection
package meteor-user-status
https://github.com/mizzao/meteor-user-status
package mrt:spy (which seems to be deprecated, but maybe you can build on top of it)
https://atmospherejs.com/mrt/spy
socket.io
https://atmospherejs.com/joncursi/socket-io-client

You can easily find that like this-
Tracker.autorun(function() {
if (Meteor.user()) {
Meteor.subscribe('userList');
}
});
More can be found from here.

Meteor.js - Should you denormalize data?

This question has been driving me crazy and I can't get my head around it. I come from a MySQL relational background and have been using Meteorjs and Mongo. For the purposes of this question take the example of posts and authors. One Author to Many Posts. I have come up with two ways in which to do this:
Have a single collection of posts - Each post has the author information embedded into the document. This of course leads to denormalization and issues such as if the author name changes how do you keep the data correct.
Have two collections: posts and authors - Each post has an author ID which references the authors collection. I then attempt to do a "join" on a non relational database while trying to maintain reactivity.
It seems to me with MongoDB degrees of denormalization is acceptable and I am tempted to embed as implementing joins really does feel like going against the ideals of Mongo.
Can anyone shed any light on what is the right approach especially in terms of wanting my app data to scale well and be manageable?
Thanks

Denormalisation is useful when you're scaling your application and you notice that some queries are taking too much time to complete. I also noticed that most Mongodb developers tend to forget about data normalisation but that's another topic.
Some developers say things like: "Don't use observe and observeChanges because it's slow". We're building real-time applications so that a normal thing to happen, it's a CPU intensive app design.
In my opinion, you should always aim for a normalised database design and then you have to decide, try and test which fields, that duplicated/denormalised, could improve your app's performance. Example: You remove 1 query per user. The UI need an extra field and it's fast to duplicated it, etc.
With the denormalisation you've an extra price to pay. You've to update the denormalised fields according to the main collection.
Example:
Let's say that you Authors and Articles collections. On each article you have the author name. The author might change his name. With a normalised scenario, it works fine. With a denormalised scenario you have to update the Author document name AND every single article, owned by this author, with the new name.
Keeping a normalised design makes you life easier but denormalisation, eventually, becomes necessary.
From a MeteorJs perspective: With the normalised scenario you're sending data from 2 Collections to the client. With the denormalised scenario, you only send 1 collection. You can also reactively join on the server and send 1 collection to the client, although it increases the RAM usage because of MergeBox on the server.
Denormalisation is something that it's very specify for you application needs. You can use Kadira to find ways of making your application faster. The database design is only 1 factor out of many that you play with when trying to improve performance.

Can Firebase/Parse run code at a certain time every day?

Is it possible on Firebase or Parse to set up something kinda like a cron job?
Is there a way to set up some sort of timed operation that runs over the stored user data?
For example, I'm writing a program that allows people to RSVP for lunch everyday. If you have RSVPed by noon, then you get paired up with somebody else who has also RSVPed. Using JavaScript, the user can submit their RSVP in the browser.
The question is, can Firebase/Parse execute the code to match everyone at 12:00pm every day?

Yes, this can be done with Parse. You'll need to write your matching function as a background job in cloud code, and then you'll need to schedule the task in the dashboard. In terms of the flexibility in scheduling, it's not as flexible as cron but you can definitely run a task at the same time every day, or every x minutes/hours.
Tasks can take 15 mins max to execute before they're killed, so depending on the size of your database or the complexity of your task, you may need to break it up into different tasks or make it resumable.

Just to confirm about Firebase:
As #rickerbh said, it can be done with Parse, but currently there is no way for you to run your code on Firebase's server. There are 2 options for you 2 solve this:
You could use Firebase Queue and run your code in Node.js
You could use a different library such as Microsoft Azure (I still haven't tried this yet, I'm not sure if it provides Job Scheduling for Android)
However, Firebase is working on something called Firebase Trigger, which will solve our problem, however it is still not released with no confirmed release date.