I have built an app that connects to an api. For this discussion, let's just call it api X. Api X provides data that is updated on a monthly basis, but the exact date when new data is published varies. My NodeJS application is basically built to check api X twice per day (this is a Cron job) and see if new data has been published. If it has, my app will download the latest data and store it in an NEDB database.
https://github.com/louischatriot/nedb
My app is deployed on Heroku, where I have the basic dyno "web", for $7/month. This app allows me to always check the latest data in a quick and easy way and mostly I have no trouble with it.
But something weird is happening. Sometimes it seems the database "resets". When I uploaded the app to Heroku from my local pc, the database was updated until December. The app quickly self-updates to include january and february, as these are the latest published datas. But sometimes when I check my app to view the data, it only has up until December. It is as if the app has been reset to the state it was in when I uploaded. The app of course quickly gets back the Jan and Feb data, as the Cron job is running. But it's a problem for me that this reset keeps happening. I don't know how often it happens, because I haven't observed many instances of this.
Does anyone know why this might be happening and what I can do about it?
Related
I have made an API using Express.js and scraped a website using cheeriojs. I have deployed this API using heroku. I want my web application to fetch the latest data from the scraped website but my deployed app is not doing so. It is still showing the old data . How to make it fetch live data continuously
Hi I got stuck in a similar problem.The way out was using cron jobs and the database simultaneously.
So I configured cron jobs to visit my website twice a day to activate the server(You don't need this if you have a good number of active users).
So when the server restarts, my app checks if the data stored in my db(i.e. the last data that was scraped from the source when the server was active previously) is the same as the one that is currently present on the target website(from where I scrape data). If it is true then do nothing, else{
update the db with the latest data and display the same on your website
}
The drawbacks with this approach are:
1.It updates data only twice a day
2.If your site has many active users throughout the day, their constant visits won't let your app's server to get idle and hence at the time that you configure for the cron job to visit your site,there are chances that your server may already be online at that moment and it may not update the data.
However , for less active users this way works perfectly fine.
You can configure cron jobs from here: https://cron-job.org/en/members/jobs/
I'm currently working for a startup which ask me to develop an offline app using pouchDB. This pouchdb is used to store the data entered by users.
The offline application works fine. Now I have to add one feature on the online app to sync the dbs. After a login, the online app have to check if data is stored in a pouchdb on the device which is connecting, and, if the check found data, the online app have to pull this data.
I have the folowing problem: the online app can't get the db stored localy on the device (even if I run the both app in the same browser).
I explain my problem in another StackOverflow post, but the formulation was not so good so I think that's a good thing to post another question.
My old post here
I work on this problem during few days and I don't have much time until I have to finish my work, if someone know how to solve this, it could be very nice :)
I think the lack of response is because readers are not very clear what the problem is. In your other post it sounds like you are saying if you write a new entry to the local database you cannot retrieve it again. In this question it sounds like once you have a local database entry you cannot make it replicate to the server database - is that the case?
On the PouchDb front page is a short example of writing to a local database and then replicating it to a server. Like this:
var db = new PouchDB('dbname');
db.put({
_id: 'dave#gmail.com',
name: 'David',
age: 69
});
db.replicate.to('http://example.com/mydb');
(the example assumes the database can be updated by anyone ie no security - otherwise you need a username and password as explained here)
Does this work for you? If not can you say what happens?
Checking to see if there is data locally should be a case of seeing if your local database has any entries in it (db.info would be a start as it returns a document count). Then you could replicate the local database using the db.replicate call.
Does this help?
I am attempting to create a Heroku server that logs a different statistic every day from 6 AM to 5 PM and then displays that data collected throughout the day at midnight in the form of an array on the server output. How do I go about doing this? In order to create events that fire at the top of the hour each hour every day and store that data, should I use node cron or node schedule?
If you want to run some specific code on Heroku at specific time intervals, you should use the Heroku Scheduler addon. This is a free addon Heroku provides which does exactly what you want to do, but in a far more reliable way than using JS directly.
The reason this is better is because when you're running code on Heroku, your web server (dyno) will restart from time to time, messing up your scheduled runs if you're keeping state persistent in the Node process.
I've searched Stack overflow and all I can find is how to use Tornado as a HTTP server.
Now, my question is how do I start doing push notifications using the system?
Let me give you some context...
The database
I have a database on some server far away that I know nothing about, other than its a postgreSQL database and a piece of software on that server updates the database every so often ( maybe every couple of seconds, to couple of days).
Currently
I Have a django app that displays these database rows. it gets these database rows from a different app - an app called api - using an ajax call every 5 seconds. As we all know this method is wasteful.
What I'd like to do
Well I'll bullet point it:
I'd like my Django app to stay the same in structure
The Django app will contain in its view JS code for connecting to a separate server.
this separate server will check the database for changes every 60 seconds. If the database has changed, then notify the clients with a message, such as "new data available"
Hopefully thats not too vague.
Thanks,
Andy.
I found that the django-websocket-redis package suits my needs which are very much comparable to yours as it can easily be implemented on top of your existing project.
Mind that there are a few dependencies (UWGSI and Redis, primarily) and I've had to switch to a Linux development environment to get everything to work properly.
We have a new build requirement that needs to work as follows:
the functionality of the App is that the maintains data related to tasks assigned to the User
the App needs takes data in the form of XML via WebAPI, the XML is a list of tasks for the User has been assigned. This would be a task bundle
once the XML is downloaded the User can work offline, i.e. they need to be able to maintain data related to their tasks, the data needs to persist on the phone, tasks would be completed over time in different sessions
there can be a number of current taks bundles "on the device" for the User
when online the App can submit an XML back up to the server, the purpose of this is to communicate the data entered by the user for the task bundle
If this was an online app it would be fairly straightforward, the data resides on the server. I have done a lot of research on the web with regard to the best way to build the app to satisfy the requirement that the App works on all browsers and platforms.
We're thinking of going with HTML5 & JayData. Haven't made any firm and final decisions, so would welcome any suggestions and constructive comment.
I work for JayData. If you have concerns against JayData just let us know and we'll try to answer.
Probably this article will help you to start your online/offline app - How to create a synchronized Online/Offline data application with EntityFramework, JavaScript and JayData