We Keep Coding

File upload from Ajax-Client to Python-Server with progress

There is an excellent tutorial on how to build a Ajax/PHP file upload including a progress handler provided at http://www.developphp.com/video/JavaScript/File-Upload-Progress-Bar-Meter-Tutorial-Ajax-PHP.
The receiver of the transmission (file_upload_parser.php) is implemented in php. However I need to replicate its functionality in Python, which turned out to be significantly more complicated than doing a simple translation.
The main problem seems to be to recreate php's global $_FILES-array (http://php.net/manual/de/reserved.variables.files.php) in Python, probably in the shape of a dictionary. I did not succeed using Pythons cgi.FieldStorage(), neither using the json module. Further, I had trouble keeping the connection established to listen for subsequent posts that might arrive (run_forever and the socket-approach suggested at Connect JS client with Python server turned out to not serve this purpose). There is plenty of documentation on how clients can write the file to the server using post. However I struggled to find information on how accomplish the read, that should occur on the receiving end of the transmission. At least not in a way that I could be implemented to yield the "file_upload_parser.php" functionality, which is key.

Processing file on front-end vs back-end

I am developing a web application with angularjs as the front-end and a CRUD service at the backend. One of the requirements is to allow the user to upload a csv file containing a list of items to be created. This can be implemented on the front-end by parsing the file in javascript and making create API call to the server for each item. However, I am an not sure if this approach is better than passing the file to the server and doing all the processing there. What are advantages/disadvantages of both these approaches? What is the common practice in such a scenario?

There are 4 things that I would use to make this decision:
Do you have very high load. If you parse it on the client you are using the clients CPU. Parsing it on the server could cost you by needing more CPU's.
Access to developer talent, is your team more productive programming it on the client or the server side.
If the answer to the above does not give a clear answer, then I would put it on the server side as it would be easier to test.
Will the "upload TSV" functionality be used by other parties/apps, who consume your API -- or is only the frontend using this functionality ?

Since I have implemented this scenario, couldn't resist from responding. I believe following things would be considered(Addition to points mentioned above) :
The Size of the file, (Huge files freeze UI, no brainer) it can even crash some not so modern browsers.
Does the file need parsing/sanitizing the contents?( you would not want the garbage to make its way to your server)
Does the User need a feedback of the load summary details after the upload?(Aync vs Sync) - This tied back to #1
Regardless, you'll end up using some variation of the bulk copy at the backend.

Well I think its advisable to parse files at the backend. You get so many options like
saving the file for reference
reducing the load on your user's resource (RAM and CPU depending on the size of the file and the operation being done on the file before pushing to backend)
Can re-initiate activity on file if there is an error during batch( if the error is code you can reproduce and help out client because you've got the file😉)
Unless files are alway say some <1mb csv or txt just do stuff backend
I hope this helps 😏.

Browser-based app needing IO control

This is a question about the best way to structure an app that has both server-side and client-side needs. Forgive the length -- I am trying to be as clear as possible with my vague question.
For a standalone non-web-connected art project, I'm creating a simple browser-based app. It could best be compared to a showy semi-complicated calculator.
I want the app to take advantage of the browser presentation abilities and run in a single non-reloading page. While I have lots of experience writing server-side apps in perl, PHP, and Python, I am newer to client-side programming, and neophyte at JavaScript.
The app will be doing a fair bit of math, a fair bit of I/O control on the Raspberry Pi, and lots of display control.
My original thought (and comfort zone) was to write it in Python with some JS hooks, but I might need to rethink that. I'd prefer to separate the logic layer from the presentation layer, but given the whole thing happens on a single non reloading html page, it seems like JavaScript is my most reasonable choice.
I'll be running this on a Raspberry Pi and I need to access the GPIO ports for both input and output. I understand that JavaScript will not be able to do I/O directly, and so I need to turn to something that will be doing AJAX-ish type calls to receive and sent IO, something like nodejs or socket.io.
My principle question is this -- Is there a clear best practice in choosing between these two approaches:
Writing the main logic of the app in client-side JavaScript and using server-side scripting to do I/O, or
Writing the logic of the app in a server-side language such as Python with calls to client-side Javascript to manage the presentation layer?
Both approaches require an intermediary between the client-side and server-side scripting. What would be the simplest platform or library to do this that will serve without being either total overkill or totally overwhelming for a learner?

I have never developed for the Raspberry Pi or had to access GPIO ports. But I have developed stand-alone web apps that acted like showy semi-complicated calculators.
One rather direct approach for your consideration:
Create the app as a single page HTML5 stand-alone web app that uses AJAX to access the GPIO ports via Node.JS or Python. Some thoughts on this approach based on my experience:
jQuery is a wonderful tool for keeping DOM access and manipulation readable and manageable. It streamlines JavaScript for working with the HTML page elements.
Keep your state in the browser local storage - using JavaScript objects and JSON makes this process amazingly simple and powerful. (One line of code can write your whole global state object to the local storage as a JSON string.) Always transfer any persistent application state changes from local variables to local storage - and have a page init routine that pulls the local storage into local variables upon any browser refresh or system reboot. Test by constantly refreshing your app as part of your testing as you develop to make sure state is managed the way you desire. This trick will keep things stable as you progress.
Using AJAX via jQuery for any I/O is very readable and reliable. It's asynchronous approach also keeps the app responsive as you perform any I/O. Error trapping and time-out handling is also easily accomplished.
For a back end, if the platform supports it, do consider Node.JS. It looks like there is at least one module for your specific I/O needs: https://github.com/EnotionZ/GpiO
I have found node to be very well supported and very easy to get started with. Also, it will keep you using JavaScript on both the front and back ends. Where this becomes most powerful is when you rely on JavaScript object literals and JSON - the two become almost interchangeable and allow you to pass complicated data structures to/from the back end via a few (or even one!) single object variable.
You can also keep your options open now on where you want to execute your math functions - since you can execute the exact same JavaScript functions in the browser or in the node back end.
If you do go the route of JavaScript and an HTML5 approach - do invest time in using the browser "developer tools" that offer very powerful debugging tools and dashboards to see exactly what is going on. You can even browse all the local storage key/value pairs with ease. It's quite a nice development platform.

After some consideration, I see the following options for your situation:
Disable browser security and directly communicate with GPIO. No standard libaries?
Use a JavaScript server environment with GPIO access and AJAX. Complexity introduced by AJAX
Use the familiar Python and use an embedded web browser If libraries are around, easy
Don't add too much complexity if you're not familiar with the tooling and language

Oh what a nice question! I'm thinking of it right now. My approach is a little difference:
With old MVC fashion, you consider the V(iew) layer is the rendered HTML page with Javascript CSS and many other things, and M and C will run on the server. And one day, I met Mr.AngularJS, I realized: Wow, some basic things may change:
AngularJS considers the View ( or the thing I believed is view ) is not actually view. AngularJS gave me Controllers, Data resources and even View templates in that "View", in another word: Client side itself can be a real Application. So now my approach is:
Server do the "server job" like: read and write data , sends data to the client, receive data from client ect....
And client do the "client job": interact with user, do the logic processes of data BEFORE IT WAS SENT such as validation, or format the information collected from user ect...
Maybe you can re-think of your approach: Ask your self what logic should run at client, what should at server. Client with javascript do its I/O, Server with server-side script do its I/O. The server will provide the needed resource for client and javascript use that resources as M(odel) of it's MVC. Hope you understand, my bad English :D

Well... it sounds like you've mostly settled on:
Python Server. (Python must manage the GPIO.)
HTML/JavaScript client, to create a beautiful UI. (HTML must present the UI.)
That seems great!
You're just wondering how much work to do on each side of the client/server divide... should be functionally equivalent.
In short: Do most of the work in whichever language you are more productive in.
Other notes come to mind:
Writing the entire server as standalone python is pretty
straightforwad.
You don't have to , but it's nice and
self-contained if you serve the page content itself from it.
If you
keep most of the state on the server/python side, you can make the
whole app a little more robust against page reloads (even though I
know you mentioned, that should never happen).

is json the answer to this: python program will talk and javascript will listen?

the same problem haunting me a month ago is still haunting me now. i know ive asked several questions regarding this on this site and i am truly sorry for that. your suggestions have all been excellent but the answer is still elusive. i now realize that this is a direct result of me not being able to phrase my question properly and for that i am sorry.
to give you guys a generalized view of things, here i go: the situation is like this, i have 2 server side scripts that i want to run.
a python program/script that continuously spouts some numbers
based on the output from that python script, a javascript script will perform some action on a webpage (e.g., change background color, display alert message, change some text)
ive studied the replies to my previous posts and have found that what i want to accomplish is more or less accomplished by json. it is my understanding that json transforms 'program-specific' variables into a format that is more 'standard or general or global'.
two different programs therefore now have the means to 'talk' with each other because they are now speaking the same 'language'.
the problem is then this, how do i actually facilitate their communication? what is the 'cellphone' between these server side scripts? do they even need one?
thank you!

If I understand what you're asking, the "cellphone" is TCP/IP. The javascript is not server-side; it runs on the client side, and alters what the client's browser displays based on json data that it downloads from the server -- data that in this case is generated by Python.
This question provides a relevant example, though it's a bit technical: JSON datetime between Python and JavaScript
Here's a very basic tutorial that explains how to create a dynamic webpage using python and javascript. It doesn't appear to use json, but it should familiarize you with the fundamentals. Once you understand what's there, using json to transport more complicated data should be fairly straightforward.
http://kooneiform.wordpress.com/2010/02/28/python-and-ajax-for-beginners-with-webpy-and-jquery/

I assume you mean: Python is on the web server, and Javascript is running in the client's web browser.
Because browsers are all different (IE6 is terrible, Chrome is great), there are a huge number of ways people found to "hack" this "cellphone" into place. These techniques are called AJAX and COMET techniques. There is no one "cellphone", but a whole bunch of them! Hopefully, you can find a library to select the right technique for the browser, and you just have to worry about the messages.
Comet is harder to do, but lets the server "push" messages to the client.
Ajax can be easier - you just periodically "pull" messages from the server.
Start with Ajax, then look at comet if you really need it. Just start by have the client (javascript) make a "GET" request, to see if the number has changed.

I don't know Javascript or json, but...
if you've ever seen an Unix-like operating system, you know about pipes. Like program1 | program2 | program3 ... Why don't you just connect Python and Javascript programs with pipes? The first one writes to stdout, and the next one reads from stdin.

This probably isn't the answer that you are looking for, and without links to your previous posts, I don't have much to go on, but nonetheless...
javascript is client side. I can interpret your question 2 different ways...
Your python script is running on your computer, and you want a script to actually alter your current browser window.
Not too sure, but writing a browser plugin may be the answer here.
Your python script is running on the server, and as a result of the script running, you want the display of your site to be changed for viewing persons.
In this case, you will could use ajax polling (or similar) on your site. Have your site be polling the server with ajax, call a server method that checks the output of the script (maybe written to a file?), and see if it has changed.

When 2 process need to communicate, they need to decide of a common/shared way to express things and a protocol to exchange those things.
In your case, since one of the processes is a browser, the protocol of choice is http. So the browser needs to do an http request or regular http request to your python process.
This python process Will need in Some way or another to be exposed via http.
There are several ways to build a web server in python. You should read this article : http://fragments.turtlemeat.com/pythonwebserver.php as a jumpstart.
Once you have this, your browser Will be able to issue HTTP GET requests to your server and your server can reply with a string.
This string can be whatever you like. Nevertheless if your answer contains structured data it can be a good start to use the XML notation or the json notation.
Json (stands for Javascript object notation) is very easy to use in javascript and this is why many people advised you to choose this notation.
I hope this will help you
Jérome wagner

Do I need server-end knowledge (e.g. Django, Rails), if I want to do Javascript, AJAX stuff?

I am trying to get into web development, specially interested building the front-end, UI part of websites while learning JavaScript maybe with AJAX technology. (I have a UI, HCI background.)
However, I have absolutely no previous knowledge about server-end web development either. To my understanding, frameworks like Django seem to pretty good at this (correct me if I am misunderstanding).
So the question is: how much Django, or Rails do I need to know, if my interest is primarily the user interface part of web development. Can I just let somebody else do the back-end stuff?
Pardon me for my imprecise choice of terminologies.

You need to know a bit about the server side. Here's what you need to know.
If you have a heavy JavaScript website, you're likely going to want to pass information from the server to clients with JSON (JavaScript Object Notation). This is just a way to format data into strings that JavaScript knows how to convert to objects.
So, each of your server-side functions that send data to the client will return JSON. If you have someone writing the server-side for you, that's all you should have to know. You're JS functions will receive JSON, and then you deal with it.
If you have to write the server-side yourself, then that involves 1) getting data from database 2) formatting the data 3) converting to JSON.
I have open-sourced a commenting widget that accepts JSON messages, and gives examples of how you would set up the Django server code. Maybe it will help you: http://www.trailbehind.com/comment_widget/

You can make a career of front-end user interface development without know a ton about server code. You would do well though to have at least a rudimentary understanding of what happens on the server when you send it a request, where your data comes from, and what the life-cycle of a web page is. This assumes that you have the support of back-end developers. As you mentioned Ajax in your question that implies that you want your web sites to actually do something, which will require things to happen on the back-end (such as storage, manipulation of data, logging in a user, etc.).
As with all things, the more you know, the easier it will be to get what you want from the dedicated professionals. I would suggest that you learn about programming in general, not try an learn a language and framework. In particular, try to understand datatypes, server settings (like timeouts, post versus get, etc.), security and database interactions as they exist beyond JavaScript/ECMAScript. That way when a developer is explaining why they cannot do something you have requested or are offering alternatives, you are speaking the same language.

Yes and no. Typically what people think of AJAX, such as posting a comment on YouTube and seeing the comment appear instantly with a thank you message, for example, requires a server side language handling the requests, looking up data and returning results as html snippets, JSON data, or XML.
However, an AJAX call can be made to static resources as well. You could have an XML file or html snippet stored statically on your web server and have it loaded. The uses for this sort of static loading are generally fewer because if you already have the static html or data in file next to your regular page, why not just put that data directly into the page?

It helps to set up a local server and write a few lines of code to service your AJAX calls. You can do a lot of JavaScript learning with just a little back-end learning.

If you're new in web development you'd rather wait with Ajax and server-side languages until you've learnt the basics with HTML, CSS and JavaScript, especially if you want to mostly work with the user interface and not the funcionality.

As you said you can let somebody else do the back-end and focus on front-end (JavaScript, HTML, CSS).
You would need to communicate with the back-end developer when storing or processing data from the server.
As mentioned before back-end development knowledge would be useful but if you have someone doing it, it's not essential for beginning.