I have a web application that I am trying to make more efficient by reducing the number of database queries that it runs. I am inclined to implement some type of Comet style solution but my lack of experience in this department makes me wonder if a more simple solution exists.
For the sake of brevity, let's just say that I have a database that contains a list of systems on a network and their current status (whether they are up or down). A user can sign into the web app and select which systems she is interested in monitoring. After which she can visit the monitoring page which displays the number of systems that are currently down.
As of now the count is refreshed using Ajax... every minute the client sends a request to the server which in turn runs a query against the database to get the current count and returns the result to the client. I know this in inefficient; for every client that logs in, another query is run against the database every minute. O(n) = bad!
I know that I can use some type of caching, such as memcached, but it still means there is a request for every user every minute. Better, but I still feel as if it's not the best solution.
I envision something more like this:
Every minute the server runs a query to pull a count for all the systems that are currently down.
The server then pushes this data to the interested clients.
That way it doesn't matter how many users are logged in and watching the monitoring page, the server only ever runs one query per minute. O(1) = good! The problem is that even after all of the research I've done I can't quite figure out how to implement this. To be honest I don't completely understand what it is that I am looking for, so that makes it very difficult to research a solution. So I am hoping that more enlightened developers can lead me in the right direction.
The solution to this issue can easily be solved with an app called Pusher, which is a hosted publish/subscribe API. In a nut shell, Pusher provides two libraries, one for the client (the subscriber) and one for the server (the publisher).
The publisher can be a single script on your server (there are quite a few languages available) set to run at whatever interval you desire. Each time it runs it will connect to a channel and publish to it whatever data it generates. The client is created via a bit of JavaScript in your web app, and whenever a user navigates to your page, the client subscribes to the same channel your server script is publishing to and receives the data as soon as it becomes available and then can manipulate it however you see fit.
The server:
#!/usr/bin/php
<?php
require('Pusher.php');
$dbh = new PDO("mysql:host=$db_host;dbname=$db_name", $db_user, $db_pass);
foreach($dbh->query('SELECT hostname FROM systems WHERE status = 0') as $row) {
$systems[] = $row['hostname'];
}
$pusher = new Pusher($pusher_key, $pusher_secret, $pusher_app_id);
$pusher->trigger(
'my-channel',
'my-event',
array('message' => implode('<br />', $systems))
);
The client:
<!DOCTYPE html>
<html>
<head>
<title>Pusher Test</title>
<script src="http://code.jquery.com/jquery.min.js" type="text/javascript"></script>
<script src="http://js.pusher.com/1.12/pusher.min.js" type="text/javascript"></script>
<script type="text/javascript">
var pusher = new Pusher(key);
var channel = pusher.subscribe('my-channel');
channel.bind('my-event', function(data) {
$('#systems').html(data.message);
});
</script>
</head>
<body>
<div id="systems"></div>
</body
</html>
So, in this case regardless of how many clients access the page there is only one database query running, and at each interval all the subscribed clients will be updated with the new data.
There is also an open source server implementation of the Pusher protocol written in Ruby called Slanger.
You can do it with Comet. But you can also just have a JavaScript timer that polls the server every minute or so. That depends on how quickly you want the feedback. Its not necessary to keep the TCP connection open the whole time.
Also, the way you find out about the status of the servers is independent from the way the clients get this information. You would not want to update the servers' status every time a client requests it. Instead you would have a timer on the app server which polls the server status and then stores it. Client requests would be fed from this stored status instead of the actual live status.
Related
A web client should only expose some features when a backend API is up and running. Therefor, I'm looking for a clean way to monitor the availability of this backend.
As a quick fix, I made a timer-based function that performs a basic GET on the API root. It's not very clean, generates lots of traffic and pollutes the javascript console with errors (in case of server down).
How should one deal with such situation?
You can trigger something in the lines of this when you need it:
function checkServerStatus()
{
setServerStatus("unknown");
var img = document.body.appendChild(document.createElement("img"));
img.onload = function()
{
setServerStatus("online");
};
img.onerror = function()
{
setServerStatus("offline");
};
img.src = "http://myserver.com/ping.gif";
}
Make ping.gif small (1 pixel) to make it as fast as possible.
Ofc you can do it more smoothly by accessing the API that returns true and keeps a really small response time, but that requires you to do some coding in back-end this simply needs you to place a 1-pixel gif image in a correct directory on a server. You can use any picture already present on the server, but expect more traffic and time as image grows larger.
Now put this in some function that calls it with delay, or simply call this when you need to check status, it's up to you.
If you need a server to send to your app a notification when it's down then you need to implement this:
https://en.wikipedia.org/wiki/Push_technology
Ideally, you would have high-reliability server that has fast response rate and is really reliable to be pinging the desired server in some interval to determine whether it up then use the push to get that information to your app. This way that 3rd server would only send you a push if a status of your app server has changed. Ideally, this server's request has a high priority on your app server queue and servers are well connected and close to each other but not on the same network in case that fails.
Recommendation:
First approach should do you good since it's simple to implement and requires the least amount of knowledge.
Consider second if:
You need a really small interval of checking making your application slower and network traffic higher
You have multiple applications that need the same - making load heavier on both each application, network AND the server. The second approach lets you use single ping to determine truth for all apps.
In order to limit number of request, simple solution can be use of server-sent events. This protocol used on top of HTTP allow server to push multiple updates in response of the same client request.
Client side code (javascript) :
var evtSource = new EventSource("backend.php");
evtSource.onmessage = function(e) {
console.log('status:' + e.data);
}
evtSource.onerror = function(e) {
// add some retry then display error to the user
}
Backend code (PHP, also supported by other languages)
header("Content-Type: text/event-stream\n\n");
while (1) {
// Each 30s, send OK status
echo "OK\n";
ob_flush();
flush();
sleep(30);
}
In both case it will limit number of request (only 1 per "session") but you will have 1 socket per client opened, which can be also to heavy for your server.
If you really want to lower the workload, you should delegate it to external monitoring platform which can expose API to publish backend status.
Maybe it already exists if your backend is hosted on cloud platform.
I am building a simple support chat for my website using Ajax. I would like to check if the user that I am currently chatting with left the browser.
At the moment I have build in that feature by setting interval function at customer side that creates the file with name: userId.txt
In the admin area I have created an interval function that checks if userId.txt exists. If it exists, it deletes it. If the file is not recreated by the custom interval function - next time the admin function will find out that file is not there it mark customer with this userId as inactive.
Abstract representation:
customer -> interval Ajax function -> php [if no file - create a new file]
admin -> interval Ajax function -> php [if file exists - delete the file] -> return state to Ajax function and do something
I was wondering if there is any better way to implement this feature that you can think of?
My solution is to use the jquery ready and beforeunload methods to trigger an ajax post request that will notify when the user arrives and leaves.
This solution is "light" because it only logs twice per user.
support.html
<!DOCTYPE html>
<html>
<head>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
<script>
//log user that just arrived - Page loaded
$(document).ready(function() {
$.ajax({
type: 'POST',
url: 'log.php',
async:false,
data: {userlog:"userid arrived"}
});
});
//log user that is about to leave - window/tab will be closed.
$(window).bind('beforeunload', function(){
$.ajax({
type: 'POST',
url: 'log.php',
async:false,
data: {userlog:"userid left"}
});
});
</script>
</head>
<body>
<h2>Your support html code...</h2>
</body>
</html>
log.php
<?php
//code this script in a way that you get notified in real time
//in this case, I just log to a txt file
$userLog = $_POST['userlog'];
file_put_contents("userlog.txt", $userLog."\n", FILE_APPEND );
//userid arrived
//userid left
Notes:
1 - Tested on Chrome, FF and Opera. I don't have a mac so I couldn't test it on Safari but it should work too.
2 - I've tried the unload method but it wasn't as reliable as beforeunload.
3 - Setting async to false on the ajax request means that the statement you are calling has to complete before the next statement, this ensures that you'll get notified before the window/tab is closed.
#Gonzalon makes a good point but using a normal DB table or the filesystem for constantly updating user movement would be exhaustive to most hard disks. This would be a good reason for using shared memory functions in PHP.
You have to differentiate a bit between the original question "How do i check in real-time, if a user is logged in?" and "How can i make sure, if a user is still on the other side (in my chat)?".
For a "login system" i would suggest to work with PHP sessions.
For the "is user still there" question, i would suggest to update one field of the active session named LAST_ACTIVITY. It is necessary to write a timestamp with the last contact with the client into a store (database) and test whether that is older than X seconds.
I'm suggesting sessions, because you have not mentioned them in your question and it looks like you are creating the userID.txt file manually on each Ajax request, right? Thats not needed, unless working cookie and session-less is a development requirement.
Now, for the PHP sessions i would simply change the session handler (backend) to whatever scales for you and what makes requesting information easy.
By default PHP uses the session temp folder to create session files,
but you might change it, so that the underlying session handler becomes a mariadb database or memcache or rediska.
When the users sessions are stored into a database you can query them: "How many users are now logged in?", "Who is where?".
The answer for "How can I check in real time if a user is logged in?" is, when the user session is created and the user is successfully authenticated.
For real-time chat application there are a lot of technologies out there, from "php comet", "html5 eventsource" + "websockets" / "long polling" to "message queues", like RabbitMq/ActiveMq with publish/subscribe to specific channels.
If this is a simple or restricted environment, maybe a VPS, then you can still stick to your solution of intervalic Ajax requests. Each request might then update $_SESSION['LAST_ACTIVITY'] with a server-side timestamp. Referencing: https://stackoverflow.com/a/1270960/1163786
A modification to this idea would be to stop doing Ajax requests, when the mouse movement stops. If the user doesn't move the mouse on your page for say 10 minutes, you would stop updating the LAST_ACTIVITY timestamp. This would fix the problem of showing users who are idle as being online.
Another modification is to reduce the size of the "iam still here" REQUEST to the server by using small GET or HEADER requests. A short HEADER "ping" is often enough, instead of sending long messages or JSON via POST.
You might find a complete "How to create an Ajax Web Chat with PHP, jQuery" over here. They use a timeout of 15 seconds for the chat.
Part 1 http://tutorialzine.com/2010/10/ajax-web-chat-php-mysql/
Part 2 http://tutorialzine.com/2010/10/ajax-web-chat-css-jquery/
You can do it this way, but it'll be slow, inefficient, and probably highly insecure. Using a database would be a noticeable improvement, but even that wouldn't be particularly scalable, depending on how "real-time" you want this to be and how many conversations you want it to be able to handle simultaneously.
You'd be much better off using a NoSQL solution such as Redis for any actions that you'll need to run frequently (ie: "is user online" checks, storing short-term conversation updates, and checking for conversation updates at short intervals).
Then you'd use the database for more long-term tasks like storing user information and saving active conversations at regular intervals (maybe once per minute, for example).
Why Ajax and not Websockets? Surely a websocket would give you a considerably faster chat system, wouldn't require generating and checking a text file, would not involve a database lookup and you can tell instantly if the connection is dropped.
I would install the https://github.com/nrk/predis library. So at the time the user authenticates, It publishes a message to Redis server.
Then you can set-up a little node server on the back-end - something simple like:
var server = require('http').Server();
var io = require('socket.io')(server);
var Redis = require('ioredis');
var redis = new Redis();
var authenticatedUsers = [];
// Subscribe to the authenticatedUsers channel in Redis
redis.subscribe('authenticatedUsers');
// Logic for what to do when a message is received from Redis
redis.on('message', function(channel, message) {
authenticatedUsers.push(message);
io.emit('userAuthenticated', message);
});
// What happens when a client connects
io.on('connection', function(socket) {
console.log('connection', socket.id);
socket.on('disconnect', function(a) {
console.log('user disconnected', a);
});
});
server.listen(3000);
Far from complete, but something to get you started.
Alternatively, take a look at Firebase. https://www.firebase.com/ if you dont want to bother with the server-side
I would suggest using in built HTML5 session storage for this purpose. This is supported by all modern browsers so we will not face issues for the same.
This will help us to be efficient and quick to recognize if user is online. Whenever user moves mouse or presses keys update session storage with date and time. Check it periodically to see if it is empty or null and decide user left the site.
Depending on your resources you may opt for websockets or the previous method called long pool request. Both ensure a bidirectional communication between the server and the client. But they may be expensive on resources.
Here is an good tutorial on the websocket:
http://www.binarytides.com/websockets-php-tutorial/
I would use a callback that you (admin) can trigger. I use this technique in web app and mobile apps to (All this is set on the user side from the server):
Send a message to user (like: "behave or I ban you").
Update user status/location. (for events to know when attendants is arriving)
Terminate user connections (e.g. force log out if maintenance).
Set user report time (e.g. how often should the user report back)
The callback for the web app is usually in JavaScript, and you define when and how you want the user to call home. Think of it as a service channel.
Instead of creating and deleting files you can do the same thing with cookie benefits of using cookie are
You do not need to hit ajax request to create a file on server as cookies are accessible by javascript/jquery.
Cookies have an option to set the time interval so would automatically delete themselves after a time, so you will not need php script to delete that.
Cookies are accessible by php, so when ever you need to check if user is still active or not, you can simply check if the cookie exist
If it were aspnet I would say signalR... but for php perhaps you could look into Rachet it might help with a lot of what you are trying to accomplish as the messages could be pushed to the clients instead of client polling.
Imo, there is no need for setting up solutions with bidirectional communications. You only want to know if a user is still logged in or attached to the system. If I understand you right, you only need a communication from server to client. So you can try SSE (server sent events) for that. The link gives you an idea, how to implement this with PHP.
The idea is simple. The server knows if user is attached or not. He could send something like "hey, user xyz is still logged in" or "hey, user xzy seems not to be logged in any more" and the client only listens to that messages and can react to the messages (e.g. via JavaScript).
The advantage is: SSE is really good for realtime applications, because the server only has to send data and the client has only to listen, see also the specification for this.
If you really need bidirectional communications or can't go with the two dependencies mentioned in the specs, it's not the best decision to use SSE, of course.
Here is a late Update with a nice chat example (written in Java). Probably it's also good to get an idea how to implement this in PHP.
So to start off, I have a raspberry pi, running a lighttp server on arch. I have a webpage that will have a max of 10 people connected at a time. Each user will be given the tag "Master," or "observer." The webpage will have controls that only the "Master," can use.
Just a heads up: I am just learning most of this so I may be making some mistakes about how to accomplish this.
My original idea is has follows. When a user connects to the database their IP address would be grabbed and inserted into a SQLite database, along with a user tag, and time of connection. From there I would be able to query the database for the users tag when they tried to execute various commands.
Whatever I use needs to be very lightweight and not store cookies on the users device.
Here is the JavaScript I currently have, it probably isn't the most efficient, but I plan on getting it working then making it look nice.
This code is supposed to connect the databases and insert the user.
<script type="text/javascript" src="http://l2.io/ip.js?var=myip"></script>
<script type="application/javascript">
var db = openDatabase('userCon.contbl.sqlite', '1.0', 'contbl', 1024);
db.transaction(function(transaction) {
var ip = myip;
var conStatus = "master"
var date = new Date();
console.log('Inserting into the database ' + ip + ',' + conStatus +',' + date);
transaction.executeSql('INSERT INTO contbl(ipAd, conType, lastActive) VALUES (?,?,?,?)',[ip,conStatus,date], function(transaction, results) {
}, function (transaction, err){
console.log(err.message+":Error"); // here an error
});
});
</script>
<script type="application/javascript" src="http://jsonip.appspot.com/?callback=getip"> </script>
I am unable to connect to the SQLite database I created on the pi, which after my research may be because SQLite is supposed to be run locally and not on a server.
Is there some sort of work around to point to the SQLite database on the pi, or is there a better resource to use for this type of task?
EDIT:
I guess my original post was not specific enough. The basic idea is I need to be able to pass a tiny bit of information from a webpage, back to the server hosting it. i.e. User connect to the server and sends its IP then the server tags that IP as an Observer of Controller. From there the server will treat each person viewing the webpage differently based on how the user was tagged.
My original plan was to use a really light weight database like SQLite, but as I found out, SQLite is local use only. I need to do this on a sever with lower then 10 connections.
My hope is someone has a suggestion, or an example to solve this problem.
The most common way for javascript running on a web page to talk to a server these days is some kind of RESTful web service.
You'll want to find a server-side programming language that runs on the Pi. Maybe node.js? It's like javascript but on a server, see here: http://weworkweplay.com/play/raspberry-pi-nodejs/
You then write whatever methods you want in the server-side language, methods which talk to the database, then have your client-side javascript call those methods. Google "REST services node.js" and you'll find plenty of how-tos.
If the SQLite database is running on the server, and this code runs in the user's browser, it can't connect to the database. The only interaction between the browser and your server is HTTP, unless there's a gigantic security hole in your server.
On my website, I have built a chatroom with support for multiple rooms. When a user joins the room, a session is placed into the database so that if they try to join the room again in another browser window, they are locked out.
It works like this
1. Join the chatroom page
2. Connect to chatroom #main
If the user has a session in the database for #main
--- Block user from joining
else
--- Load chatroom
When the chatroom is closed client side or the user terminates there connection with the /quit command, all of their sessions are deleted, and this works fine.
However
There is a possibility that users will just close the browser window rather than terminating their connection. The problem with this is that their session will stay in the database, meaning when they try to connect to the room, they are blocked.
I'm using this code onbeforeunload to try and prevent that
function disconnect() {
$.ajax({
url: "/remove-chat-sessions.php?global",
async: false
});
};
This is also the function called when the user types the /quit command
The problem
The problem with this is that when I reload the page, 5 times out of 10 the sessions have not been taken out of the database, as if the ajax request failed or the page reloaded before it could finish. This means that when I go back into the chatroom, the database still thinks that I am connected, and blocks me from entering the chatroom
Is there a better way to make sure that this AJAX call will load and if not, is there a better alternative than storing user sessions in an online database?
Edit:
The reason users are blocked from joining rooms more than once is because messages you post do not appear to you when the chatroom updates for new messages. They are appended to the chatroom box when you post them. This means that if users could be in the same chatroom over multiple windows, they would not be able to see the comments that they posted across all of the windows.
In this situation you could add some sort of polling. Basically, you request with javascript a page every X time. That page adds the user session to the database. Then there's a script executing every Y time, where Y > X, that cleans old sessions.
The script that is called every X time
...
// DB call (do as you like)
$All = fetch_all_recent();
foreach ($All as $Session)
{
if ($Session['time'] < time() - $y)
{
delete_session($Session['id']);
}
}
The script that javascript is calling every X time
...
delete_old_session($User->id);
add_user_session($User->id, $Chat->id, time());
The main disadvantage of this method is the increment in requests, something Apache is not so used to (for large request number). There are two non-exclusive alternatives for this, which involve access to the server, are:
Use nginx server. I have no experience in this but I've read it supports many more connections than Apache.
Use some modern form of persistent connection, like socket.io. However, it uses node.js, which can be good or bad, depending on your business.
I am trying to use periodic refresh(ajax)/polling on my site by XMLHttp(XHR) to check if a user has a new message on the database every 10 seconds, then if there is inform him/her by creating a div dynamically like this:
function shownotice() {
var divnotice = document.createElement("div");
var closelink = document.createElement("a");
closelink.onclick = this.close;
closelink.href = "#";
closelink.className = "close";
closelink.appendChild(document.createTextNode("close"));
divnotice.appendChild(closelink);
divnotice.className = "notifier";
divnotice.setAttribute("align", "center");
document.body.appendChild(divnotice);
divnotice.style.top = document.body.scrollTop + "px";
divnotice.style.left = document.body.scrollLeft + "px";
divnotice.style.display = "block";
request(divnotice);
}
Is this a reliable or stable way to check message specifically since when I look under firebug, a lot of request is going on to my database? Can this method make my database down because of too much request? Is there another way to do this since when I login to facebook and check under firebug, no request is happening or going on but I know they are using periodic refresh too... how do they do that?
You can check for new data every 10 seconds, but instead of checking the db, you need to do a lower impact check.
What I would do is modify the db update process so that when it makes a change to some data, it also updates the timestamp on a file to show that there is a recent change.
If you want better granularity than "something changed somewhere in the db" you can break it down by username (or some other identifier). The file(s) to be updated would then be the username for each user who might be interested in the update.
So, when you script asks the server if there is any information for user X newer than time t, instead of making a DB query, the server side script can just compare the timestamp of a file with the time parameter and see if there is anything new in the database.
In the process that is updating the DB, add code that (roughly) does:
foreach username interested in this update
{
touch the file \updates\username
}
Then your function to see if there is new data looks something like:
function NewDataForUser (string username, time t)
{
timestamp ts = GetLastUpdateTime("\updates\username");
return (ts > t);
}
Once you find that there is new data, you can then do a full blown DB query and get whatever information you need.
I left facebook open with firebug running and I'm seeing requests about once a minute, which seems like plenty to me.
The other approach, used by Comet, is to make a request and leave it open, with the server dribbling out data to the client without completing the response. This is a hack, and violates every principle of what HTTP is all about :). But it does work.
This is quite unreliable and probably far too taxing on the server in most cases.
Perhaps you should have a look into a push interface: http://en.wikipedia.org/wiki/Push_technology
I've heard Comet is the most scalable solution.
I suspect Facebook uses a Flash movie (they always download one called SoundPlayerHater.swf) which they use to do some comms with their servers. This does not get caught by Firebug (might be by Fiddler though).
This is not a better approach. Because you ended up querying your server in every 10 seconds even there is no real updates.
Instead of this polling approach, you can simulate the server push (reverrse AJAX or COMET) approach. This will compeletly reduce the server workload and only the client is updated if there is an update in server side.
As per wikipedia
Reverse Ajax refers to an Ajax design
pattern that uses long-lived HTTP
connections to enable low-latency
communication between a web server and
a browser. Basically it is a way of
sending data from client to server and
a mechanism for pushing server data
back to the browser.
For more info, check out my other response to the similar question