We Keep Coding

Save form data in client with unstable internet connection

I have a piece of open source software written in python which uses the bottle web server to display forms in a web browser. The form data are send via "method = post" to the web server. Until now the server process is running on the same (PC) host as the browser, so there is no issue with the internet connection.
Now I have to rewrite this software so that it can be used on mobile devices, with the server somewhere in the internet. The environment in which data entry is to take place will be such that an unstable or lost internet connection is likely. So I have to have provisions for the case that the website containing the form is loaded first (in the office via WLAN, say), then data entry takes place (in the "field") and during data entry, internet connection is lost, so that saving data to the server won't work. In this case it would be great to be able to save the form data locally, in order to send the post-request later on. (Probably it won't be possible to keep the website open all the time until this is possible. The latest when battery goes low, I'd run into problems.)
Probably I'm not the first with this problem, so my question is: is there a "standard" (or well tested) solution for the task to buffer form data on the client side for the case when a post-request is not answered, and send the same request later on? If not, how would you go about to solve this issue? In particular, I see the following (sub-)problems:
How to detect (on the client side) that a post request failed? Probably some kind of timeout mechanism in javascript would have to be employed, but how?
How to save data? My first idea would be to save data to a cookie using javascript. Do I overlook something here?
How to send data back later on?
I'm sufficiently proficient in python to dare this project, but rather new to web technologies, so please excuse if some part of the question is rather stupid. In this case, I'd be grateful to be told so... (... with a hint on how to ask a better question.)
Thanks a lot for any help.

I will try to answer based on (sub-)problems:
How to detect (on the client side) that a post request failed? Probably some kind of timeout mechanism in javascript would have to be employed, but how?
To detect if request failed
Only send status code 200 if you received data and it's saved to backend!
Don't send 200 if there is an error! (use error status code like 5xx or 4xx)
There is a timeout option in jquery to cancel the request if it takes more than given time to complete
When failed, Save data to localStorage
If you are not using jquery, I guess you can do something similar using fetch in vanilla javascript (Click here to know more about fetch)
$.ajax({
timeout: 3000 // sets timeout to 3 seconds
}).done(function () {
console.log("success");
}).fail(function () {
console.log("error");
var _local = localStorage.getItem('data-saved'); //get localStorage data
_local.push({"key": "value"}) // Append JSON based Form data
localStorage.setItem('data-saved', JSON.stringify(_local)); // Update localStorage
});
How to save data? My first idea would be to save data to a cookie using javascript. Do I overlook something here?
Save data using localStorage
In LocalStorage, you can't store JSON however, you can save using JSON.stringify and load back using JSON.parse
// Get data
var get_local_data = JSON.parse(localStorage.getItem('data-saved'));
// Update Data
get_local_data.append({"Name": "value", "age": 10})
// Update localStorage
localStorage.setItem('data-saved', JSON.stringify(get_local_data));
How to send data back later on?
Sending data back using setTimeout method in javascript
Check continuously if there is any data in localStorage's key. If any send an ajax request to back-end!
// Run in each 5 Sec
setTimeout(function () {
// Check if we have any failed data
var get_local_data = JSON.parse(localStorage.getItem('data-saved'));
if(get_local_data.length > 0){
//Make a ajax request
//Update localStorage if success (You need to remove the data from the localStorage),
//Ignored failed case
}
}, 5000);

GAE Channel Api vs Socket

I have a Google app that runs using GO on the server side and Javascript on the client side. The client generate every certain timestep a javascript object (that could be "stringified") that needs to be sent back to the server and saved as json file. Based on the google app engine documentation it seems that this could achieved with both:
channel api
socket service
but I couldn't find any examples for both of these and also it is not clear to me which one is the best to accomplish the goal. Can anyone help me?
Thanks a lot.

Maybe this could be useful to someone so here is the way I did that:
saveData : function saveData() {
var _this = this,
save = this.shadowRoot.querySelector('#save-data'),
subData = JSON.stringify(_this.app.userSession);
save.url="url";
save.body = subData;
save.go();
}

PHP minimal working example of Web Sockets

I'm trying to determine how to setup a web socket for the first time ever so a working minimal example with static variables (IP address for example instead of getservbyname) will help me understand what is flowing where.
I want to do this the right way so no frameworks or addons for both the client and the server. I want to use PHP's native web sockets as described here though without over-complicating things with in-depth classes...
http://www.php.net/manual/en/intro.sockets.php
I've already put together some basic JavaScript...
window.onload = function(e)
{
if ('WebSocket' in window)
{
var socket = new WebSocket('ws://'+path.split('http://')[1]+'mail/');
socket.onopen = function () {alert('Web Socket: connected.');}
socket.onmessage = function (event) {alert('Web Socket: '+event.data);}
}
}
It's the PHP part that I'm not really sure about. Presuming we have a blank PHP file...
If necessary how do I determine if my server's PHP install has this socket functionality already available?
Is the request essentially handled as a GET or POST request in
example?
Do I need to worry about the port numbers? e.g. if
($_SERVER['SERVER_PORT']=='8080')
How do I return a basic message on the initial connection?
How do I return a basic message say, five seconds later?

It's not that simple to create a simple example, I'm afraid.
First of all you need to check in php configuration if the server is configured for sockets with the setting enable-sockets
Then you need to implement (or find) a websocket server that at least follows the Hybi10 specification (https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-10) of websockets. If you find the "magic number" 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 in the code for the header, you can be sure it does follow at least Hybi06 ...
Finally, you need to have access to an admin console on the server in order to execute the PHP websocket server using php -q server.php
EDIT: This is the one I've been using a year ago ... it might still work as expected with current browsers supporting Websockets: http://code.google.com/p/phpwebsocket/source/browse/trunk/+phpwebsocket/?r=5

converting facebook javascript sdk variables into asp variables?

I have a facebook sign up and login button build using js sdk. I want to store user response into a database, but I cant get the userID or User.email with classic asp.
Sometimes it works, sometimes not, can't find pattern.
var email=response.email;
var fbid=response.id;
var fbname=response.name;
var fbbirthday=response.birthday;
var fbhometown=response.hometown.name;
document.getElementById('fbmail').value = email;
document.getElementById('fbid').value = fbid;
document.getElementById('fbname').value = fbname;
document.getElementById('fbbirthday').value = fbbirthday;
document.getElementById('fbhometown').value = fbhometown;

You can use classic ASP to get all of that information server-side. You would need to build your own framework to wrap up the HTTP Get/Post/Delete commands to the Graph API. But to accomplish getting that basic user information should be fairly straight forward once you have nice wrappers around those three HTTP verbs.
A quickie solution would be to FORM post those values back to the server. I assume you have https, so security shouldn't be an issue.

Periodic refresh or polling

I am trying to use periodic refresh(ajax)/polling on my site by XMLHttp(XHR) to check if a user has a new message on the database every 10 seconds, then if there is inform him/her by creating a div dynamically like this:
function shownotice() {
var divnotice = document.createElement("div");
var closelink = document.createElement("a");
closelink.onclick = this.close;
closelink.href = "#";
closelink.className = "close";
closelink.appendChild(document.createTextNode("close"));
divnotice.appendChild(closelink);
divnotice.className = "notifier";
divnotice.setAttribute("align", "center");
document.body.appendChild(divnotice);
divnotice.style.top = document.body.scrollTop + "px";
divnotice.style.left = document.body.scrollLeft + "px";
divnotice.style.display = "block";
request(divnotice);
}
Is this a reliable or stable way to check message specifically since when I look under firebug, a lot of request is going on to my database? Can this method make my database down because of too much request? Is there another way to do this since when I login to facebook and check under firebug, no request is happening or going on but I know they are using periodic refresh too... how do they do that?

You can check for new data every 10 seconds, but instead of checking the db, you need to do a lower impact check.
What I would do is modify the db update process so that when it makes a change to some data, it also updates the timestamp on a file to show that there is a recent change.
If you want better granularity than "something changed somewhere in the db" you can break it down by username (or some other identifier). The file(s) to be updated would then be the username for each user who might be interested in the update.
So, when you script asks the server if there is any information for user X newer than time t, instead of making a DB query, the server side script can just compare the timestamp of a file with the time parameter and see if there is anything new in the database.
In the process that is updating the DB, add code that (roughly) does:
foreach username interested in this update
{
touch the file \updates\username
}
Then your function to see if there is new data looks something like:
function NewDataForUser (string username, time t)
{
timestamp ts = GetLastUpdateTime("\updates\username");
return (ts > t);
}
Once you find that there is new data, you can then do a full blown DB query and get whatever information you need.

I left facebook open with firebug running and I'm seeing requests about once a minute, which seems like plenty to me.
The other approach, used by Comet, is to make a request and leave it open, with the server dribbling out data to the client without completing the response. This is a hack, and violates every principle of what HTTP is all about :). But it does work.

This is quite unreliable and probably far too taxing on the server in most cases.
Perhaps you should have a look into a push interface: http://en.wikipedia.org/wiki/Push_technology
I've heard Comet is the most scalable solution.

I suspect Facebook uses a Flash movie (they always download one called SoundPlayerHater.swf) which they use to do some comms with their servers. This does not get caught by Firebug (might be by Fiddler though).

This is not a better approach. Because you ended up querying your server in every 10 seconds even there is no real updates.
Instead of this polling approach, you can simulate the server push (reverrse AJAX or COMET) approach. This will compeletly reduce the server workload and only the client is updated if there is an update in server side.
As per wikipedia
Reverse Ajax refers to an Ajax design
pattern that uses long-lived HTTP
connections to enable low-latency
communication between a web server and
a browser. Basically it is a way of
sending data from client to server and
a mechanism for pushing server data
back to the browser.
For more info, check out my other response to the similar question