I am logging some analytics data based on people clicking on my site. When a person clicks on something, it sends a call to my node application via an ajax call, but if what they clicked is an external link, the original request gets cancelled since they leave the site. Is there a good way for node to process the request even after the client has disconnected?
It definitely wouldn't matter if the running process does not need to write things to the response.
Even then, it probably doesn't matter anyway if you do something like
User submits request, you respond "thanx, the job was started". Before you respond, asynchronously execute the job (not hard given that node.js code should be asynchronous to begin with).
Job puts data in some table or file or nosql store.
User can go to a different url to see a list of jobs and their states (started, running, complete) and view results.
Related
I'm have a very long process in a php script (generate a huge pdf).
I have a button in my HTML page that launches the php script and I'd like to show a kind of progress bar or at least an animated gif and when the php script is over, display the generated pdf.
The generation of the pdf may last 15 minutes so the php engine exits in timeout and the browser too.
I there a way to declare a kind of client-side callback that would be invoked as soon as the server side process is over ?
Thanks for your replies
Edit :
Thanks for your replies :)
If I well understand, I must launch the process on server-side and "detach" my client i.e do not wait untill the process is over. Instead, my client should periodically check the progression of server-side process. Right ?
If so, I may use the following scenario :
The client sends an ajax request to the server. The server launches
the process and returns a guid to the client. This guid identifies
the job.
The client periodically checks the progression of the job
via an Ajax request, from its guid.
Once the job is over, the client can issue a last Ajax query to
download the PDF
That means that the server must save the generated PDF on its disk and wait for the final Ajax request to send the file and delete it, right ?
For something as long as 15 minutes, I wouldn't even use web sockets for this. 15 minutes is a long time and there's really no telling what the user is going to be doing in the meantime. A disconnected notification process is probably going to be more reliable in this case.
Consider something like:
User initiates process, whereby a record is stored in the database "queueing" the process to take place.
User is immediately presented with a page indicating that the process has been queued and that they can continue to use the application.
A separate application which runs periodically (every minute? every few minutes?) checks for "queued" processes in the database, updates their status to "in-progress" (so subsequent runs don't also pick up the same records), and processes them.
As each process completes, it's either removed from the database or updated to a "completed" status.
The user is otherwise notified that the process is complete.
This final notification can be done a number of ways. An email can be sent to the user, for example. Or consider a user experience similar to the Facebook notification bar. Each page in the website can check for "completed" processes when the page loads and present a "notification" in the UI which directs the user to the result of the process. If users spend a lot of time on any given page then this would be a good place to use web sockets or long polling via JavaScript to keep checking for completed processes.
The main thing is to separate the user interface from the long-running process. Web applications by design aren't suited for processes which run for that long. By separating the concerns the web application can focus just on the user interface and the command-line application can focus on the processing. (As an added bonus, this would prevent users from over-loading the server with too many concurrent processes. The command-line application can just run one record at a time, so too many concurrent processes just slows down the response, not the server.)
as #David said, but no one has covered the progress bar, the implantation of this depends on what you know ( you being the application creating the pdf ).
Do you know the size of the pdf when complete.
Do you know how long
it will take to generate.
Do you have code where you can hook into
to update the progress.
The application needs a way to know when to update the completed percentage, and by how much. If you can do that then you can either store the progress in the database with the script that crates the PDF and read it on a user facing page, or store it in a file, etc..
jQuery UI progress bar is easy to use, but you will have to know what percentage is done to be able to tell the end user.
After that it is a pretty simple matter of using ajax (jquery $.post ) and a file, that's how i do it. I just wright a simple text file with a number representing the completion percent. Load it via ajax and feed it to the jquery ui progress widget.
So, I'm creating this application that sometime it require pulling the feed and it's always timeout on heroku because of the xml parser takes time. So, I change to be asynchronous load via Ajax every time the page is loaded. I still get H12 error from my Ajax call. Now I'm thinking of using Resque to run the job in background. I can do that no problem but how would I know that the job is finished so I can pull the processed feed on to the html page via AJAX?
Not sure if my question is clear, so how would the web layer knows that the job is done and it should signal e.g (onComplete in javascript) to populate the content on the page?
There are a number of ways to do this
The JavaScript can use AJAX to poll the server asking for the results and the server can respond with 'not yet' or the results. You keep asking until you get the results.
You could take a look at Juggernaut (http://juggernaut.rubyforge.org/) which lets your server push to the client
Web Sockets are the HTML5 way to deal with the problem. There are a few gems around to get you started Best Ruby on Rails WebSocket tool
You have an architecture problem here. The reason for the H12 is so that the user is not sat there for more than 30 seconds.
By moving the long running task into a Resque queue, you are making it disconnected to the front end web process - there is no way that the two can communicate due to process isolation.
Therefore you need to look at what you are doing and how. For instance, if you are pulling a feed, are you able to do this at some point before the user needs to see the output and cache the results in some way - or are you able to take the request for the feed from the user and then email them when you have the data for them to look at etc etc.
The problem you have here is that your users are asking for something which takes longer than a reasonable amount of time to complete, so therefore you need to have a good look at what you are doing and how.
I'm currently fooling around with AJAX. Right now, I created a Markdown previewer that updates on change of a textarea. (I guess you know that from somewhere... ;-) ).
Now, I'm trying to figure out, how to update a page upon an event is fired from another client. So to say an asynchron message board. A user writes something, an event is called, the post is written.
But on the other clients' pages, the new post is of course not yet available until they reload and get the updated list of posts from the database.
Now, how can you get this to work asynchronously? So in that moment when one client does something, the other clients all get to know that he did something?
I don't think this can be done completely in AJAX, but I also have no idea whatsoever how to implement this on server-side, as it would require a page reload to inform the other clients of the event.
I'm thinking of creating a file or database entry that hashes the current state of data. Whenever a client loads the page, he saves this hash. Then, a timer (does this exist in JavaScript?) checks for the hash every few seconds.
As soon as anyone changes the databse, the hash is recalculated. If the script sees that the hash was changed and is different to the one saved, it reloads the contents form the database and saves the new hash.
Is that even going to work?
Polling that is light as possible is really the best solution here. Even if you did use a socket or something... That's still basically a live connection waiting around that will likely have to poll itself (albeit in a more effecient way).
20 queries in 10 minutes that have responses like {"updates":false} shouldn't even be putting a dent in your application. I mean imagine someone browsing your site requesting 20 pages and the related images/scripts/etc (even if some caching is involved), there could easily be hundreds of requests requiring all sorts of wasted database queries to information to be displayed on the page they don't actually care about.
You could use polling. For example each client might be sending continuous AJAX requests to the server say each 30 seconds to see if new posts are available and if yes, show them:
setInterval(function() {
// TODO: Send an AJAX request here to the server and fetch new posts.
// if new posts are available update the DOM
}, 30 * 1000);
On the other hand when someone decides to write a new post you send an AJAX (or not AJAX) request to the server to store this post in the database.
Another less commonly used approach is the concept of Comet and the HTML 5 WebSockets implementation which allow the clients to be notified by the server of changes using push.
My Django app displays data from a database. This data changes without user intervention, i.e. behind the scenes. Whenever it changes, I would like the webpage to update the changed sections without a full page reload.
Obviously AJAX springs to mind. When the page is loaded initially (or manually, fully re-loaded later on), the rendered template loads a JavaScript that runs window.onload = update("all"), update(...) in turn triggers a number of XMLHTTPRequests which again return data that gets transformed into HTML pieces for the corresponding sections. All works fine. At the initial page load.
Now I find myself in a Python function that saves a new object to the database.
How do I tell the browser to run update(...) ?
Do I need to somehow manually issue a request to a url that is mapped to a view which in turn renders a template that contains the JavaScript code to run update(...) ??? Oh my!
I feel like I'm not following the usual approaches.
Maybe I'm just standing to close in front of the problem.
Can anyone help me ?
2021 update: Use channels: https://channels.readthedocs.io/en/latest/
You have two choices
Have the browser poll using setTimeout()
Look into Comet -- this is a technique for pushing data from the server to the browser.
Here's an article on Comet in Django
two approaches:
just update the database and wait until the next AJAX query. That means it should do the query periodically, you'll have to balance between immediacy and server load. It helps a little if you can do a cheap query to just verify if there has been an update. Maybe make that check rely only on memcached instead of going to the DB
use comet. In short: the client does an AJAX query asking for the update. the server sees there's no update, so it doesn't answer. Instead, the connection is kept open for a long time. Eventually either the update comes and the server finally answers, or the client times out and kill the connection. In that case, the client should immediately reissue the query to keep waiting for the update.
You can also use The Websocket API https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.
We have a POST to a PL/SQL database procedure that (a) does some database operations based on the POST parameters and (b) redirects the user to a page showing the results.
The problem is, when the user does a browser "refresh" of the results page, that still has the original request, so it calls the database procedure and resends the parameters.
There are things we can do with saving state so bad things don't happen if the request gets sent in again. But that got me wondering.
Is there a way to tell the browser to set the url to the redirect call, not the original user request? This would probably be in either the redirect itself, or in Javascript on the target page.
You don't mention what you are using to serve the page, but make sure you perform an EXTERNAL redirect. Some platforms will internally redirect within a site.
For instance, with Apache HTTP Server, you need to specify the force-redirect flag in mod_rewrite: http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteRule
The 4th response here has a decent explanation of this as well.
The canonical solution is described pretty well on Wikipedia. See Post/Redirect/Get. You want the code that's handling the POST to redirect to a GET when it's work is done, as refreshing a GET will not resubmit form data.