I've built an API that delivers live data all at once when a user submits a search for content. I'd like to take this API to the next level by delivering the API content to a user as the content is received instead of waiting for all of the data to be received before displaying.
How does one go about this?
Easiest way to do in Django is using Django Endless Pagination
I think the better way to apply it is setting limit in your query. For example, If you have 1000 of records in your database, then retrieving all data at once takes time. So, if a user search a word 'apple', you initially send the database request with limit 10. And, you can set pagination or scroll feature at your front-end. If the user click next page or scroll your page, you can again send the database request with another limit 10 so that the database read action will not take more time to read the limited data.
From your explanation
We're pulling our data from multiple sources with each user search.
Being directly connected to the scrapers for those sources, we display
the content as each scraper completes content retrieval. I was
originally looking to mimic this in the API, which is obviously quite
different from traditional pagination - hope this clarifies.
So you in your API, you want to
take query from user
initiate live scrapers
get back the data to the user when scrapers finish the job !
(correct me if im wrong)
My Answer
This might feel little complicated, but this is the best one I can think of.
1. When user submits the query:
1. Initiate the live scrapers in to celery queue (take care of the priority).
2. Once the queue is finished, get back to the user with the information you have via sockets(this is how facebook or any website sends users notifications`. But in your case you will send the results html data in the socket.
3. Since you will have the data already, moved into the db as you scraped, you will can paginate it like normal db.
But this approach gives you a lag of a few seconds or a minute to reply back to the user, meanwhile you keep theuser busy with something on the UI front.
Related
The problem:
I have a jquery ajax (post) based website, where the website doesn't refresh every time user navigates to another page. Which means I have to pull data with ajax and present it to the user. Now for pulling small text data, this system works great. However once the text data is huge (let's say over 200,000 words), the load time is quite high (especially for mobile users). What I mean to say is, ajax tries to load full text information and displays it after it is done loading all text. So the user has to wait quite a bit to get the information.
If you look at a different scenario, let's say wikipedia. There are big pages in wikipedia. However, a user doesn't feel he/she has to wait a lot because the page loads step by step (top to bottom). So even if the page is large, the user is already kept busy with some information. And while the user is processing those, rest of the page keeps loading.
Question:
So is it possible to display, via ajax, information on real time load? Meaning keep showing whatever is loaded and not wait for the full document to be loaded?
Ajax (xmlhttprequest) is a really great feature in html5, for the same thing, ajax is better than socket, by that, I mean non-persistant connection but as soon as the connection is persistant (impossible for xmlhttprequest)socket is fastest.
The simplest way is to use web socket is socket.io but you need a JavaScript server to use this library and there is one host where you can get one for free with admin tools: heroku.
You can use PHP server if you dont want to use JavaScript server with the socketme library but it is a bit more complex.
Also, you can think diferently, you try to send a lot of data.
200 000 words is something like 70ko (I try a lorem ipsum), the upload speed is relative to data and connection speed/ping. You can compress by any way your data before sending and uncompress server-side. There is probably thousand way to do this but I think the simpliest way is to find a JavaScript library to compress/uncompress data and simply use your jquery ajax to send what's compressed.
EDIT — 21/03/14:
I misunderstood the question, you want to display the current upload progress ?
Yes, it is possible by using the onprogress event, in jQuery you must follow this simple exemple: jQuery ajax progress
So I was implementing a chat room. I'll start off with the schema that I used.
I have a room table, that basically stores the information regarding the chatroom like the number of participant, the topic etc etc.
I have a users table that stores the users info.
I have a posts table that stores the posts. This has a foreign key from Users and from room tables.
also, I have one final table that is to have a relation between users and rooms. So it just has the roomid and the userid from the users who are a part of the room.
Now, I have three divs on page, one for the chatarea, the other where the people online are shown and then there is a text area to post the message.
What I am doing currently is, to have a javascript function loadChats(), now this method calls a php file that just fetches all the posts in that particular room till now. And the same is dumped into my div ie "chatroom".
Also, similarly, I have a loadParticipants() that load the users every other second.
I am using jquery.post for the purpose and in the end of the method, I do a setTimeout in the end of the function. Now here are my questions
Ofcourse i can make this better. Any suggestions? I was thinking of a few.
On every call to php, I get the entire chathistory and send it back to browser, ofcourse I can check if the count of messages is the same as it is on the client side, and if it is, then I wont send the messages. But is it going to make it any better? How?
Also, making a call to server side every other second seems a bit too much of an overkill. Is there any way to do it like, if some new chat is added to posts table, then that particular chatroom is notified and updated? i.e. instead of constantly pinging the server to ask for new request, just ask it once, and wait if there is anything new or not. When that request is completed, it pings the server again for the next update.
You should look into websockets (I've never used them with PHP but this seems really promising: http://socketo.me/). What you can do is have the server push any new messages to the client whenever they come in, and have each of the clients push to the server, etc. This way you won't have to keep pinging the server over and over every 2 seconds, and loading tons of data to compare. When there's a new message, the server saves it to some database and then pushes that message through all the open sockets. Same thing with logging in/logging off.
edit: Just looked through the page even more and their tutorial even goes through how to get it set up with a basic chatroom-esque functionality.
I am building an analytics system for my rails application and I want to monitor every time I pull a certain object from the database, and I'd like to put the in the model file. I have objects that are being displayed on the page and I need to see the amount of views and clicks that they get. I assume the views can be handled by just figuring out when the object is pulled from the database (if someone could tell me how to do that) and I figured javascript to monitor the clicks. Would you all agree with this? Or is there a better way. I am using Rails 3.1 with MongoMapper and MongoDB
To store the data simply send an ajax request from the browser with the information you want to store in a POST request to a rails resource like :click#create. Be sure to include the relevant data attributes within the request.
You may want to collect the requests and then send them all in a batch based on time or a use clicking a "done" button or something of that sort.
Recording the fact that someone clicked (from javascript) is different than recording when an object is retrieved from the database. You could write a before filter for each of the methods in the class or possibly implement an active record callback for something of that sort.
I'm currently fooling around with AJAX. Right now, I created a Markdown previewer that updates on change of a textarea. (I guess you know that from somewhere... ;-) ).
Now, I'm trying to figure out, how to update a page upon an event is fired from another client. So to say an asynchron message board. A user writes something, an event is called, the post is written.
But on the other clients' pages, the new post is of course not yet available until they reload and get the updated list of posts from the database.
Now, how can you get this to work asynchronously? So in that moment when one client does something, the other clients all get to know that he did something?
I don't think this can be done completely in AJAX, but I also have no idea whatsoever how to implement this on server-side, as it would require a page reload to inform the other clients of the event.
I'm thinking of creating a file or database entry that hashes the current state of data. Whenever a client loads the page, he saves this hash. Then, a timer (does this exist in JavaScript?) checks for the hash every few seconds.
As soon as anyone changes the databse, the hash is recalculated. If the script sees that the hash was changed and is different to the one saved, it reloads the contents form the database and saves the new hash.
Is that even going to work?
Polling that is light as possible is really the best solution here. Even if you did use a socket or something... That's still basically a live connection waiting around that will likely have to poll itself (albeit in a more effecient way).
20 queries in 10 minutes that have responses like {"updates":false} shouldn't even be putting a dent in your application. I mean imagine someone browsing your site requesting 20 pages and the related images/scripts/etc (even if some caching is involved), there could easily be hundreds of requests requiring all sorts of wasted database queries to information to be displayed on the page they don't actually care about.
You could use polling. For example each client might be sending continuous AJAX requests to the server say each 30 seconds to see if new posts are available and if yes, show them:
setInterval(function() {
// TODO: Send an AJAX request here to the server and fetch new posts.
// if new posts are available update the DOM
}, 30 * 1000);
On the other hand when someone decides to write a new post you send an AJAX (or not AJAX) request to the server to store this post in the database.
Another less commonly used approach is the concept of Comet and the HTML 5 WebSockets implementation which allow the clients to be notified by the server of changes using push.
My Django app displays data from a database. This data changes without user intervention, i.e. behind the scenes. Whenever it changes, I would like the webpage to update the changed sections without a full page reload.
Obviously AJAX springs to mind. When the page is loaded initially (or manually, fully re-loaded later on), the rendered template loads a JavaScript that runs window.onload = update("all"), update(...) in turn triggers a number of XMLHTTPRequests which again return data that gets transformed into HTML pieces for the corresponding sections. All works fine. At the initial page load.
Now I find myself in a Python function that saves a new object to the database.
How do I tell the browser to run update(...) ?
Do I need to somehow manually issue a request to a url that is mapped to a view which in turn renders a template that contains the JavaScript code to run update(...) ??? Oh my!
I feel like I'm not following the usual approaches.
Maybe I'm just standing to close in front of the problem.
Can anyone help me ?
2021 update: Use channels: https://channels.readthedocs.io/en/latest/
You have two choices
Have the browser poll using setTimeout()
Look into Comet -- this is a technique for pushing data from the server to the browser.
Here's an article on Comet in Django
two approaches:
just update the database and wait until the next AJAX query. That means it should do the query periodically, you'll have to balance between immediacy and server load. It helps a little if you can do a cheap query to just verify if there has been an update. Maybe make that check rely only on memcached instead of going to the DB
use comet. In short: the client does an AJAX query asking for the update. the server sees there's no update, so it doesn't answer. Instead, the connection is kept open for a long time. Eventually either the update comes and the server finally answers, or the client times out and kill the connection. In that case, the client should immediately reissue the query to keep waiting for the update.
You can also use The Websocket API https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.