i dunno how to put it. Hope the title is right for my problem or scenario.
I want to build a REST API, with a data coming from many rssfeed web. Right now, i'm able to fetch the data using a script javascript and saving it in my database. To be able fetch that data, i have to open a page so the script will be able to run and reload every 1 minute. The Rest Api is still in localhost by the way.
The Question is, what if i want to host it, should i have 1 PC to
always running 24 hours which only open a browser and access a REST
API address so the script will keep running and the data will always
be up to date?
Right now this the only method in my head, is there any method that i shouldn't have 1 pc to running 24hours a day seven days a week.
The best solution of your problem is to setup a scheduler that will be running on a predefined period, and fetch the data and store it in DB internally and you don't need to open a page to do that if you are not modifying the response returned from rssfeed.
You can go through this, Post , Tutorial, Node-Schedule, Parse. These are some of the example which you can use based on your requirements
Related
I have the idea to put around 30 charts on a webpage, the charts will be load with data from a SQLite database and a node.js express webserver is sending the data and also getting the data every second from another program.
I think the node.js server will be ok to read and write every seconds this huge amounts of data, but i am not sure what is the best solution to update the charts in the webpage.
I have hear about things like Web Worker or Service Worker or Server Sent Event and i myself would use a setTimeout function in the webpage which would read every second the data again and again and write it into the charts, but in my testing with just around 10 charts now i see already that the browser takes to much cpu and ram.
i am using apexcharts in the page and i am not sure if my setTimeout function ad reading every second with that from the database is the problem or how other profis are doing it, what is the best technique to update such charts every second?
So, I am making a SQL query and processing the data acquired by SQL to a JSON format(so in a program that generates JSON). But the process takes a very long time I am talking about upwards of 6 minutes, so I am trying to display the data shown as the processing of the data is being done, but I am quite unsure as to how to go about that.
The only idea I have had is to split the queries into 10 at a time and display the 10 and run the rest in the background and if load them as required(but again no clue how to go about that)
I am familiar with FLASK,HTML,JS,Jquery(barely). I am open to new frameworks, but if possible I would like to stick with what I know. Any solution is appreciated.
You might want to look into Celery.
https://flask.palletsprojects.com/en/2.0.x/patterns/celery/
Basically, you would offload your sql queries to a celery task. The task could run each query sequentially and keep saving the results to a database, a redis cache, or a plain file.
Then your html/js would set a timer/interval to do a fetch() every few seconds to a flask route that reads from that database until it is done.
I have made an API using Express.js and scraped a website using cheeriojs. I have deployed this API using heroku. I want my web application to fetch the latest data from the scraped website but my deployed app is not doing so. It is still showing the old data . How to make it fetch live data continuously
Hi I got stuck in a similar problem.The way out was using cron jobs and the database simultaneously.
So I configured cron jobs to visit my website twice a day to activate the server(You don't need this if you have a good number of active users).
So when the server restarts, my app checks if the data stored in my db(i.e. the last data that was scraped from the source when the server was active previously) is the same as the one that is currently present on the target website(from where I scrape data). If it is true then do nothing, else{
update the db with the latest data and display the same on your website
}
The drawbacks with this approach are:
1.It updates data only twice a day
2.If your site has many active users throughout the day, their constant visits won't let your app's server to get idle and hence at the time that you configure for the cron job to visit your site,there are chances that your server may already be online at that moment and it may not update the data.
However , for less active users this way works perfectly fine.
You can configure cron jobs from here: https://cron-job.org/en/members/jobs/
I have a news application that's on my desktop, and I am wondering if there is a way to pull data from this app directly using python. I will post a picture of what the app looks like.
Lets say I want to pull the DE GDP results as they change upon news release. Is there any way I can gain access to these values locally, rather than trying to pull it using json or xpath? I tried JSON and got it to pull values, but it takes up to 10 seconds to update, and I want it to update as fast as possible. I figured the fastest way to get the updated values would be to pull them locally on my computer somehow, but I am not very knowledgeable on this sort of thing. Thank you for your help!
you can inspect the page and in network section see the url which is being called to render the data, and you can see the response as well.
like i am done it for sample page.
you can see the screenshot.
click on this link to see the image
No. You need to use a web service. After all, the app you are showing must be pulling its data from some service. You could use a network sniffer to find out where exactly.
[
seeing the network it says the url which is fetching data is.
https://next.newsimpact.com/NewsWidget/GetNextEvents?offset=-330
and when you open it in new tab.
so you can easily use this url to fetch the json data with request library in python
some brief code:
pip install requests
import requests
r = requests.get('https://next.newsimpact.com/NewsWidget/GetNextEvents?offset=-330')
print r.json
So, I'm creating this application that sometime it require pulling the feed and it's always timeout on heroku because of the xml parser takes time. So, I change to be asynchronous load via Ajax every time the page is loaded. I still get H12 error from my Ajax call. Now I'm thinking of using Resque to run the job in background. I can do that no problem but how would I know that the job is finished so I can pull the processed feed on to the html page via AJAX?
Not sure if my question is clear, so how would the web layer knows that the job is done and it should signal e.g (onComplete in javascript) to populate the content on the page?
There are a number of ways to do this
The JavaScript can use AJAX to poll the server asking for the results and the server can respond with 'not yet' or the results. You keep asking until you get the results.
You could take a look at Juggernaut (http://juggernaut.rubyforge.org/) which lets your server push to the client
Web Sockets are the HTML5 way to deal with the problem. There are a few gems around to get you started Best Ruby on Rails WebSocket tool
You have an architecture problem here. The reason for the H12 is so that the user is not sat there for more than 30 seconds.
By moving the long running task into a Resque queue, you are making it disconnected to the front end web process - there is no way that the two can communicate due to process isolation.
Therefore you need to look at what you are doing and how. For instance, if you are pulling a feed, are you able to do this at some point before the user needs to see the output and cache the results in some way - or are you able to take the request for the feed from the user and then email them when you have the data for them to look at etc etc.
The problem you have here is that your users are asking for something which takes longer than a reasonable amount of time to complete, so therefore you need to have a good look at what you are doing and how.