So, I'm creating this application that sometime it require pulling the feed and it's always timeout on heroku because of the xml parser takes time. So, I change to be asynchronous load via Ajax every time the page is loaded. I still get H12 error from my Ajax call. Now I'm thinking of using Resque to run the job in background. I can do that no problem but how would I know that the job is finished so I can pull the processed feed on to the html page via AJAX?
Not sure if my question is clear, so how would the web layer knows that the job is done and it should signal e.g (onComplete in javascript) to populate the content on the page?
There are a number of ways to do this
The JavaScript can use AJAX to poll the server asking for the results and the server can respond with 'not yet' or the results. You keep asking until you get the results.
You could take a look at Juggernaut (http://juggernaut.rubyforge.org/) which lets your server push to the client
Web Sockets are the HTML5 way to deal with the problem. There are a few gems around to get you started Best Ruby on Rails WebSocket tool
You have an architecture problem here. The reason for the H12 is so that the user is not sat there for more than 30 seconds.
By moving the long running task into a Resque queue, you are making it disconnected to the front end web process - there is no way that the two can communicate due to process isolation.
Therefore you need to look at what you are doing and how. For instance, if you are pulling a feed, are you able to do this at some point before the user needs to see the output and cache the results in some way - or are you able to take the request for the feed from the user and then email them when you have the data for them to look at etc etc.
The problem you have here is that your users are asking for something which takes longer than a reasonable amount of time to complete, so therefore you need to have a good look at what you are doing and how.
Related
So, I am making a SQL query and processing the data acquired by SQL to a JSON format(so in a program that generates JSON). But the process takes a very long time I am talking about upwards of 6 minutes, so I am trying to display the data shown as the processing of the data is being done, but I am quite unsure as to how to go about that.
The only idea I have had is to split the queries into 10 at a time and display the 10 and run the rest in the background and if load them as required(but again no clue how to go about that)
I am familiar with FLASK,HTML,JS,Jquery(barely). I am open to new frameworks, but if possible I would like to stick with what I know. Any solution is appreciated.
You might want to look into Celery.
https://flask.palletsprojects.com/en/2.0.x/patterns/celery/
Basically, you would offload your sql queries to a celery task. The task could run each query sequentially and keep saving the results to a database, a redis cache, or a plain file.
Then your html/js would set a timer/interval to do a fetch() every few seconds to a flask route that reads from that database until it is done.
I'm have a very long process in a php script (generate a huge pdf).
I have a button in my HTML page that launches the php script and I'd like to show a kind of progress bar or at least an animated gif and when the php script is over, display the generated pdf.
The generation of the pdf may last 15 minutes so the php engine exits in timeout and the browser too.
I there a way to declare a kind of client-side callback that would be invoked as soon as the server side process is over ?
Thanks for your replies
Edit :
Thanks for your replies :)
If I well understand, I must launch the process on server-side and "detach" my client i.e do not wait untill the process is over. Instead, my client should periodically check the progression of server-side process. Right ?
If so, I may use the following scenario :
The client sends an ajax request to the server. The server launches
the process and returns a guid to the client. This guid identifies
the job.
The client periodically checks the progression of the job
via an Ajax request, from its guid.
Once the job is over, the client can issue a last Ajax query to
download the PDF
That means that the server must save the generated PDF on its disk and wait for the final Ajax request to send the file and delete it, right ?
For something as long as 15 minutes, I wouldn't even use web sockets for this. 15 minutes is a long time and there's really no telling what the user is going to be doing in the meantime. A disconnected notification process is probably going to be more reliable in this case.
Consider something like:
User initiates process, whereby a record is stored in the database "queueing" the process to take place.
User is immediately presented with a page indicating that the process has been queued and that they can continue to use the application.
A separate application which runs periodically (every minute? every few minutes?) checks for "queued" processes in the database, updates their status to "in-progress" (so subsequent runs don't also pick up the same records), and processes them.
As each process completes, it's either removed from the database or updated to a "completed" status.
The user is otherwise notified that the process is complete.
This final notification can be done a number of ways. An email can be sent to the user, for example. Or consider a user experience similar to the Facebook notification bar. Each page in the website can check for "completed" processes when the page loads and present a "notification" in the UI which directs the user to the result of the process. If users spend a lot of time on any given page then this would be a good place to use web sockets or long polling via JavaScript to keep checking for completed processes.
The main thing is to separate the user interface from the long-running process. Web applications by design aren't suited for processes which run for that long. By separating the concerns the web application can focus just on the user interface and the command-line application can focus on the processing. (As an added bonus, this would prevent users from over-loading the server with too many concurrent processes. The command-line application can just run one record at a time, so too many concurrent processes just slows down the response, not the server.)
as #David said, but no one has covered the progress bar, the implantation of this depends on what you know ( you being the application creating the pdf ).
Do you know the size of the pdf when complete.
Do you know how long
it will take to generate.
Do you have code where you can hook into
to update the progress.
The application needs a way to know when to update the completed percentage, and by how much. If you can do that then you can either store the progress in the database with the script that crates the PDF and read it on a user facing page, or store it in a file, etc..
jQuery UI progress bar is easy to use, but you will have to know what percentage is done to be able to tell the end user.
After that it is a pretty simple matter of using ajax (jquery $.post ) and a file, that's how i do it. I just wright a simple text file with a number representing the completion percent. Load it via ajax and feed it to the jquery ui progress widget.
The problem:
I have a jquery ajax (post) based website, where the website doesn't refresh every time user navigates to another page. Which means I have to pull data with ajax and present it to the user. Now for pulling small text data, this system works great. However once the text data is huge (let's say over 200,000 words), the load time is quite high (especially for mobile users). What I mean to say is, ajax tries to load full text information and displays it after it is done loading all text. So the user has to wait quite a bit to get the information.
If you look at a different scenario, let's say wikipedia. There are big pages in wikipedia. However, a user doesn't feel he/she has to wait a lot because the page loads step by step (top to bottom). So even if the page is large, the user is already kept busy with some information. And while the user is processing those, rest of the page keeps loading.
Question:
So is it possible to display, via ajax, information on real time load? Meaning keep showing whatever is loaded and not wait for the full document to be loaded?
Ajax (xmlhttprequest) is a really great feature in html5, for the same thing, ajax is better than socket, by that, I mean non-persistant connection but as soon as the connection is persistant (impossible for xmlhttprequest)socket is fastest.
The simplest way is to use web socket is socket.io but you need a JavaScript server to use this library and there is one host where you can get one for free with admin tools: heroku.
You can use PHP server if you dont want to use JavaScript server with the socketme library but it is a bit more complex.
Also, you can think diferently, you try to send a lot of data.
200 000 words is something like 70ko (I try a lorem ipsum), the upload speed is relative to data and connection speed/ping. You can compress by any way your data before sending and uncompress server-side. There is probably thousand way to do this but I think the simpliest way is to find a JavaScript library to compress/uncompress data and simply use your jquery ajax to send what's compressed.
EDIT — 21/03/14:
I misunderstood the question, you want to display the current upload progress ?
Yes, it is possible by using the onprogress event, in jQuery you must follow this simple exemple: jQuery ajax progress
I'm currently fooling around with AJAX. Right now, I created a Markdown previewer that updates on change of a textarea. (I guess you know that from somewhere... ;-) ).
Now, I'm trying to figure out, how to update a page upon an event is fired from another client. So to say an asynchron message board. A user writes something, an event is called, the post is written.
But on the other clients' pages, the new post is of course not yet available until they reload and get the updated list of posts from the database.
Now, how can you get this to work asynchronously? So in that moment when one client does something, the other clients all get to know that he did something?
I don't think this can be done completely in AJAX, but I also have no idea whatsoever how to implement this on server-side, as it would require a page reload to inform the other clients of the event.
I'm thinking of creating a file or database entry that hashes the current state of data. Whenever a client loads the page, he saves this hash. Then, a timer (does this exist in JavaScript?) checks for the hash every few seconds.
As soon as anyone changes the databse, the hash is recalculated. If the script sees that the hash was changed and is different to the one saved, it reloads the contents form the database and saves the new hash.
Is that even going to work?
Polling that is light as possible is really the best solution here. Even if you did use a socket or something... That's still basically a live connection waiting around that will likely have to poll itself (albeit in a more effecient way).
20 queries in 10 minutes that have responses like {"updates":false} shouldn't even be putting a dent in your application. I mean imagine someone browsing your site requesting 20 pages and the related images/scripts/etc (even if some caching is involved), there could easily be hundreds of requests requiring all sorts of wasted database queries to information to be displayed on the page they don't actually care about.
You could use polling. For example each client might be sending continuous AJAX requests to the server say each 30 seconds to see if new posts are available and if yes, show them:
setInterval(function() {
// TODO: Send an AJAX request here to the server and fetch new posts.
// if new posts are available update the DOM
}, 30 * 1000);
On the other hand when someone decides to write a new post you send an AJAX (or not AJAX) request to the server to store this post in the database.
Another less commonly used approach is the concept of Comet and the HTML 5 WebSockets implementation which allow the clients to be notified by the server of changes using push.
My Django app displays data from a database. This data changes without user intervention, i.e. behind the scenes. Whenever it changes, I would like the webpage to update the changed sections without a full page reload.
Obviously AJAX springs to mind. When the page is loaded initially (or manually, fully re-loaded later on), the rendered template loads a JavaScript that runs window.onload = update("all"), update(...) in turn triggers a number of XMLHTTPRequests which again return data that gets transformed into HTML pieces for the corresponding sections. All works fine. At the initial page load.
Now I find myself in a Python function that saves a new object to the database.
How do I tell the browser to run update(...) ?
Do I need to somehow manually issue a request to a url that is mapped to a view which in turn renders a template that contains the JavaScript code to run update(...) ??? Oh my!
I feel like I'm not following the usual approaches.
Maybe I'm just standing to close in front of the problem.
Can anyone help me ?
2021 update: Use channels: https://channels.readthedocs.io/en/latest/
You have two choices
Have the browser poll using setTimeout()
Look into Comet -- this is a technique for pushing data from the server to the browser.
Here's an article on Comet in Django
two approaches:
just update the database and wait until the next AJAX query. That means it should do the query periodically, you'll have to balance between immediacy and server load. It helps a little if you can do a cheap query to just verify if there has been an update. Maybe make that check rely only on memcached instead of going to the DB
use comet. In short: the client does an AJAX query asking for the update. the server sees there's no update, so it doesn't answer. Instead, the connection is kept open for a long time. Eventually either the update comes and the server finally answers, or the client times out and kill the connection. In that case, the client should immediately reissue the query to keep waiting for the update.
You can also use The Websocket API https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.