I had been wondering about knowing mongoDB changes instantly for a while and best solution that I have ever seen, is tailing mongoDb logs and I just had given wondering up till met Meteor.
When there is a changes on mongoDB somehow(through Meteor or directly), Meteor understands the changes instantly and apply them. We could see it on publishing changes or observing(observe or observeChange) changes.
I guess that, Meteor could understand it by tailing mongoDB logs as the best solution that I have ever seen suggests. But this assumption also make me ask that, Meteor can run with a mongoDB running on different host and if the way that meteor knows MongoDB changes instantly is tailing logs, how could Meteor tails logs of mongoDB running on different machine(different host) and handle changes? Tailing on different host requires painful things and I think Meteor does not follow this way.
I think tailing logs is not my answer because of this question.
I am curious about this knowledge. Cause, I think that, if I have it, I could use it on my other works without Meteor.
So, how could really Meteor knows mongoDB changes instantly? And I wanna ask once again; what would be the best way to know about mongoDB changes instantly?
Thank You!
Yes you are right this is from new new oplog tailing functionality. Meteor pretends that it is mongodb in a replica set and it tails mongodb's operation logs for collections it is monitoring.
Usually databases need multiple servers so that they can stay up in case one fails. MongoDB has this functionality in a replica set.
So this is where meteor comes in. Meteor pretends to be a mongodb replica database so when any data changes on this operations log for other copies to see, Meteor knows that data has changed and can push this new data down to the client instantly.
Usage in production
You can use a different mongo database with this new functionality as you set MONGO_OPLOG_URL. You will need to set mongodb as a replica set first so that the operations log is enabled
This was a very recent introduction put in just last week in version 0.7 you can see the blog post about it below
More links about it:
https://www.meteor.com/blog/2013/12/17/meteor-070-scalable-database-queries-using-mongodb-oplog-instead-of-poll-and-diff
https://www.youtube.com/watch?v=_dzX_LEbZyI
https://github.com/meteor/meteor/wiki/Oplog-Observe-Driver
Related
I'm currently writing a program that is a To-Do list that just runs locally on my machine in the terminal screen(written in node.js, no extra dependencies). I've set up everything and it works totally fine, but I have never actually used any databases other than just session cookies on the browser. What would you all suggest I use when all I want to do is learn how to store some simple data in a database and then be able to manipulate it in the same fashion.
Any help would be greatly appreciated on this one! I'm really just looking for suggestions on a good database, as well as a bit of guidance on the best way to use that particular technology. I'm familiar with SQL and NoSQL databases, but like I said, I have never even used one on an actual application.
Thanks in advance.
In my view, You can use mongodb which is easy to configure and manipulate data. It helps you to work on json objects directly from database. Use express to create your RESTful endpoints and exposed data. Sometime back i use it for angular application, repo link is below for your reference :
AngularJs, ExpressJs, MongoDB application
Hope this helps.
Cheers
You can use sqlite which provides a good SQL API but keeps things very simple by simply using a file on your disk and accessing it through code directly in your application. You don't need to start another server and take care of configuring it.
For a simple To-Do list that should be more than adequate for storing your data whilst keeping your configuration simple and easy.
Here's a tutorial on how to set it up:
http://www.sqlitetutorial.net/sqlite-nodejs/
You'll essentially be using the sqlite3 npm package: https://www.npmjs.com/package/sqlite3
That page also comes with brief usage instructions.
Nodejs has client libraries for database interaction
Ex.
For SQL database:
postgresql : https://github.com/brianc/node-postgres
For NoSQL db:
mongodb : https://github.com/Automattic/mongoose
You can also create pool of db connections
I'm looking for a persistent database -- preferably a document database like MongoDB -- for an Electron application I'm working on. I looked at NeDB, but found out that it keeps a copy of the database in memory at all times -- and this won't work for me because I'm expecting an extremely large number of records, and as far as I know it's not really possible to simply bundle MongoDB with an Electron application.
Do you have any suggestions for me?
According to me you should use LinvoDB because they are best with large objects also they are not effected by large db size and uses mongodb queries for CRUD operation.
I think its best for your application as you were looking for a kind of db that does not persist in the memory.It also does not scan whole db whenever query is fired.
There is also one more db option realm which is the best db according to me and it is released few days back.
please refer: https://github.com/realm/realm-js/issues/262
Cheers :)
We're in the process of rebuilding our current Customer Portal from scratch. Our current portal has a bunch of cron jobs which sync all of our client, contact, and invoice records from our WHMCS billing system, but this is a really horrible way to do it. One of our developers has suggested using MongoDB to cache the data pulled via the WHMCS API, which would work nicely, but he also suggested that he remembered having setup MongoDB so that if a find by ID didn't result in a page, it would trigger a javascript routine on MongoDB to go and make the API call to fetch the required record.
That sounded fantastic, however after many hours of googling and reading through MongoDB documentation, I can't for the life of me find a way to do this on the MongoDB side. We coudl do it fairly easily from the PHP side (Especially since we're using Yii2), but it would be a very clean solution to do it from the MongoDB side..
Hoping perhaps someone has come across such a use case, and can provide an example of doing it from MongoDB?
I am developing an application based on mongo and sails, and i am testing how the realtime update in sails works.
I am using sails 0.9.16 now, but i am interested also in answers about sails 0.10.
I want a list to be updated when new documents are created in the corresponding collection. This works when i add documents via sails sockets, sending a post message. In that case i see other clients receiving a message and the list on the client side is updated.
There is an external service writing on the mongo database tough, so the collection is growing all the times. The new elements created directly by the external service in the database are not notified to listening clients, so i have to refresh the web page in order to show those elements.
Questions:
are notifications about database creations supposed to work, when those creations do not come from sails itself?
if yes, does this require some special configuration?
if not, what would be a recommended way to keep client side listing about a collection updated when the database is changing?
Cheers
Very interesting question, though not an unusual one: the guys from Meteor were having the same problem. Basically, without watching the DB you can't even scale your app horizontally, since one server process will have no idea on what data changes were made by another one.
So, at first they sort of patched it by polling the DB every 10 seconds. :) Obviously, it wasn't the best solution, so they ended up with another one (which can also work for Sails): now they are tailing the MongoDB oplog and fire an update whenever there's a change in the corresponding collection.
That said, to answer your questions:
AFAIK, a Sails process has no clue about any external changes made to the DB;
so, nothing to configure;
a way to track external DB (MongoDB) updates would be using one of the oplog watchers you can find in npm (e.g. this or one of these, etc.) to listen to the changes and trigger updates whenever there's a need.
Unfortunately, no ready-to-use solution here, but I hope at least that now you have an idea on how to make it work.
I have designed a meteor.js application and it works great on localhost and even when deployed to the internet. Now I want create a sign-up site that will spin up new instances of the application for each client who signs up on the back-end. Assuming a meteor.js application and python or javascript for the sign-up site, what high level steps need to be taken to implement this?
I am looking for a more correct and complete answer that takes the form of my poorly imagined version of this:
Use something like node or python to call a shell script that may or may not run as sudo
That script might create a new folder to hold instance specific stuff (like client files, config, and or that instances database).
The script or python code would deploy an instance of the application to that folder and on a specific port
Python might add configuration information to a tool like Pound to forward a subdomain to a port
Other things....!?
I don't really understand the high level steps that need to be taken here so if someone could provide those steps and maybe even some useful tools or tutorials for doing so I'd be extremely grateful.
I have a similar situation to you but ended up solving it in a completely different way. It is now available as a Meteor smart package:
https://github.com/mizzao/meteor-partitioner
The problem we share is that we wanted to write a meteor app as if only one client (or group of clients, in my case) exists, but that it needs to handle multiple sets of clients without them knowing about each other. I am doing the following:
Assume the Meteor app is programmed for just a single instance
Using a smart package, hook the collections on server (and possibly client) so that all operations are 'scoped' only to the instance of the user that is calling them. One way to do this is to automatically attach an 'instance' or 'group' field to each document that is being added.
Doing this correctly requires a lot of knowledge about the internals of Meteor, which I've been learning. However, this approach is a lot cleaner and less resource-intensive than trying to deploy multiple meteor apps at once. It means that you can still code the app as if only one client exists, instead of explicitly doing so for multiple clients. Additionally, it allows you to share resources between the instances that can be shared (i.e. static assets, shared state, etc.)
For more details and discussions, see:
https://groups.google.com/forum/#!topic/meteor-talk/8u2LVk8si_s
https://github.com/matb33/meteor-collection-hooks (the collection-hooks package; read issues for additional discussions)
Let me remark first that I think spinning up multiple instances of the same app is a bad design choice. If it is a stop gap measure, here's what I would suggest:
Create an archive that can be readily deployed. (Bundle the app, reinstall fibers if necessary, rezip). Deploy (unzip) the archive to a new folder when a new instance is created using a script.
Create a template of an init script and use forever or daemonize or jesus etc to start the site on reboot and keep the sites up during normal operation. See Meteor deploying to a VM by installing meteor or How does one start a node.js server as a daemon process? for examples. when a new instance is deployed populate the template with new values (i.e. port number, database name, folder). Copy the filled out template to init.d and link to the runlevel. Alternatively, create one script in init.d that executes other scripts to bring up the site.
Each instance should be listening to its own port, so you'll need a reverse proxy. AFAIK, Apache and Nginx require restarts when you change the configuration, so you'll probably want to look at Hipache https://github.com/dotcloud/hipache. Hipache uses redis to store the configuration information. Adding the new instance requires to add a key to redis. There is an experimental port of Hipache that brings the functionality to Nginx https://github.com/samalba/hipache-nginx
What about DNS updates? Once you create a new instance, do you need to add a new record to your DNS configuration?
I don't really have an answer to your question... but I just want to remind you of another potential problem that you may run into cuz I see you mentioned python, in other words, you may be running another web app on Apache/Nginx, etc... the problem is Meteor is not very friendly when it comes to co-exist with another http server, the project I'm working on was troubled by this issue and we had to move it to a stand alone server after days of hassle with guys from Meteor... I did not work on the issue, so I'm not able to give you more details, but I just looked up online and found something similar: https://serverfault.com/questions/424693/serving-meteor-on-main-domain-and-apache-on-subdomain-independently.
Just something to keep in mind...