I'm currently working on a project that involves parsing through a user-supplied file, doing computations with that data, and visualizing the results using graphing utilities. Right now, I'm stuck with using Python as the back-end because it has scientific libraries unavailable in JavaScript, but I want to move the entire tool to a web server, where I can do much slicker visualizations using D3.js.
The workflow would be something like: obtain the file contents from the browser, execute the Python script with the contents, return jsonified objects of computed values, and plot those objects using D3. I already have the back-end and front-end working by themselves, but want to know: How can I go about bridging the two? From what I've gathered, I need to do something along the lines of launching a server, sending AJAX requests to the server, and retrieving data from the server. But with the number of frameworks out there (Flask, cgi, apache, websockets, etc.), I'm not really sure where to start. This will likely only be a very simple web app with just a file submit page and a data visualization page. Any help is appreciated!
Apache is a web server, flask is a web framework in python, websockets are a protocol, and cgi is something totally different, and none of them help you on the front end.
You could deploy a simple backend in flask or django or pylons or any other python web framework. I like django, but it may be a little heavy for your purpose, flask is a bit more lightweight. You deploy them to a server with a web server installed and use something like apache to distribute.
Then you need a front end and a way of delivering your front end. Flask / Django are both fully capable of doing so in conjunction with a web server, or you could use a static file server like Amazon S3.
On your front end, you need to load D3 and probably some kind of utility like jQuery to load and parse your data from the back end, then use D3 however you like to present it on screen.
Flask is easy to get up and running and is Python based. It works well with REST APIs and data sent by JSON (or JSON API).
This is one solution with which I have some experience and which seems to work well and is not hard to get up and running (and natural to work with Python). I can't tell you whether it is the best solution for your needs, but it should work.
If you are overwhelmed and don't know where to start, you can pick one of the options and google search for a tutorial. With a decent tutorial, you should have an example up and running by the end of the tutorial, and then you will know if you are comfortable working with it and have an idea whether it will meet your needs.
Then you could do a proof-of-concept; make a small app that just handles one small part (the one you are most concerned about handling, perhaps) and write something which will do it.
By then, you can be pretty sure you have a good tool for the purpose (unless you were convinced otherwise by the proof-of-concept -- in which case, try again with another option :-))
Related
This is more a general question, I'm not asking for implementation details.
I have a Python script running on a machine which monitors an experiment and gathers live data from it. I want to visualise the data on a website and decided to go with d3.js
My current setup however is absolutely basic:
The python script is running in an endless loop and appends new data
to a text-file (CSV file).
There is an HTML-page on the web
server with a javascript, which grabs the file and creates a graph
using d3.js.
I already thought about replacing the text-file with a sqlite database, but I'm sure there is a more sophisticated and easy way to do it via network.
So I am looking for a more solid solution and searched the web for different kind of strategies but most of those involve bigger libraries (Tornado, node.js etc.).
I would be happy with quite basic socket-like solution and tried to figure out how it would work with WebSockets but not sure how to handle this.
So my question is, what kind of design would you recommend or use if your only goal is to have a Javascript running in a browser and waiting/listening for data (let's say every few seconds a float), which is sent from a Python script?
Choosing the "right" web framework is quite challenging task, at least in Java we have a lot of them. But looking at JavaScript frameworks like AngularJS I doubt if we really need something heavy at server. Usually web framework is responsible for routing, templating, building pretty URLs and some other stuff. With AngularJS we can move all these responsibilities to client side. Then the backend becomes nothing more than REST listener and data validator. A thin layer between your application logic and client view. So why do we need web frameworks now if all we want is a REST listener?
At the moment I found two points which must be handled by server side: authentication/authorization and things requiring 'pushing' like Comet. Are these criterias enough to choose the "right" framework?
I'll give you one more capability that I've seen require back-end server support. It's those pages which generate a file. For example, they get a file from a third party and then hand it to the client as though they produced it directly, or they are generating a JPEG/PNG/GIF image on the fly, or perhaps a CSV/XLS dump of data. There may be ways to generate those on the fly from the front end and make them available for download, but sometimes the back-end is just easier for those.
Other than that, your assessment is 100% correct. You literally need less server for apps built with AngularJS than was needed with the previous request/response model of ASP/JSP/PHP/etc.
However, just because you need less doesn't mean you need nothing. Issues like data caching and how user sessions are handled can still come up even for smaller servers as you scale. But it has definitely opened things up for tech like Node.js to be considered that I would not have given much thought to a few years back.
I have the following scenario: I'm doing a publication lookup tool so users can find documents through a search field and filters. Right now we are working with a small budget, so all the data is stored in a json file (~60 records). If the project is successfull, we will have a server with a database and a couple of thousand records.
I want to develop all the lookup solution using breeze, so later I will don't have to make many modifications. The problem is that I can't find information about querying a json file directly (without a server).
Do you think that this is possible?
Actually, it is possible. But I can't think of a way that is as simple as setting up a simple server. That's like following off a log with Visual Studio. Maybe you're coming from a different environment? I'd like to know. Even there, it's usually pretty easy to spin something up with some kind of http API that can return JSON.
If you only have 60 records, I'm guessing this is a prototype that you're trying to stand up in a hurry. You're so much in a hurry that you don't even want to use a server ... which is kind of odd because you need something to serve the HTML, CSS, and JavaScript files, right?
You could do it with node.js / express very easily; almost as simple as setting up an express route that reads and returns the JSON file. But that still involves a server running somewhere (the client's own machine?) and you'd have to learn some elementary node.js
You can do it entirely with HTML and JS script files and no server other than the file system.
Off the top of my head, I think I'd begin by writing a custom Breeze ajax adapter that is actually a mock: no matter what you ask of it, it returns the JSON data in its entirety.
You call this once at application start to load the entities into an EntityManager cache. Then make all subsequent queries be local queries. You can set the EntityManager default query strategy to turn all queries into local queries by default.
No matter what you do, you'll have to define metadata to describe the entity types in your JSON data. I'm guessing you only have one type so that should be simple and quick.
You'll also have to do something to tell Breeze what kind of entity you're querying. Adding .toType('Foo'); to the end of your queries may be sufficient. You can always delve into the JsonResultsAdapter if you need something fancier at a lower level of the stack.
None of this is hard. But none of it is Breeze 101 either. You're not following what we've thought of as a typical application development path. Maybe we're missing something. I'll be curious to see if people can relate to your situation.
This is a question about the best way to structure an app that has both server-side and client-side needs. Forgive the length -- I am trying to be as clear as possible with my vague question.
For a standalone non-web-connected art project, I'm creating a simple browser-based app. It could best be compared to a showy semi-complicated calculator.
I want the app to take advantage of the browser presentation abilities and run in a single non-reloading page. While I have lots of experience writing server-side apps in perl, PHP, and Python, I am newer to client-side programming, and neophyte at JavaScript.
The app will be doing a fair bit of math, a fair bit of I/O control on the Raspberry Pi, and lots of display control.
My original thought (and comfort zone) was to write it in Python with some JS hooks, but I might need to rethink that. I'd prefer to separate the logic layer from the presentation layer, but given the whole thing happens on a single non reloading html page, it seems like JavaScript is my most reasonable choice.
I'll be running this on a Raspberry Pi and I need to access the GPIO ports for both input and output. I understand that JavaScript will not be able to do I/O directly, and so I need to turn to something that will be doing AJAX-ish type calls to receive and sent IO, something like nodejs or socket.io.
My principle question is this -- Is there a clear best practice in choosing between these two approaches:
Writing the main logic of the app in client-side JavaScript and using server-side scripting to do I/O, or
Writing the logic of the app in a server-side language such as Python with calls to client-side Javascript to manage the presentation layer?
Both approaches require an intermediary between the client-side and server-side scripting. What would be the simplest platform or library to do this that will serve without being either total overkill or totally overwhelming for a learner?
I have never developed for the Raspberry Pi or had to access GPIO ports. But I have developed stand-alone web apps that acted like showy semi-complicated calculators.
One rather direct approach for your consideration:
Create the app as a single page HTML5 stand-alone web app that uses AJAX to access the GPIO ports via Node.JS or Python. Some thoughts on this approach based on my experience:
jQuery is a wonderful tool for keeping DOM access and manipulation readable and manageable. It streamlines JavaScript for working with the HTML page elements.
Keep your state in the browser local storage - using JavaScript objects and JSON makes this process amazingly simple and powerful. (One line of code can write your whole global state object to the local storage as a JSON string.) Always transfer any persistent application state changes from local variables to local storage - and have a page init routine that pulls the local storage into local variables upon any browser refresh or system reboot. Test by constantly refreshing your app as part of your testing as you develop to make sure state is managed the way you desire. This trick will keep things stable as you progress.
Using AJAX via jQuery for any I/O is very readable and reliable. It's asynchronous approach also keeps the app responsive as you perform any I/O. Error trapping and time-out handling is also easily accomplished.
For a back end, if the platform supports it, do consider Node.JS. It looks like there is at least one module for your specific I/O needs: https://github.com/EnotionZ/GpiO
I have found node to be very well supported and very easy to get started with. Also, it will keep you using JavaScript on both the front and back ends. Where this becomes most powerful is when you rely on JavaScript object literals and JSON - the two become almost interchangeable and allow you to pass complicated data structures to/from the back end via a few (or even one!) single object variable.
You can also keep your options open now on where you want to execute your math functions - since you can execute the exact same JavaScript functions in the browser or in the node back end.
If you do go the route of JavaScript and an HTML5 approach - do invest time in using the browser "developer tools" that offer very powerful debugging tools and dashboards to see exactly what is going on. You can even browse all the local storage key/value pairs with ease. It's quite a nice development platform.
After some consideration, I see the following options for your situation:
Disable browser security and directly communicate with GPIO. No standard libaries?
Use a JavaScript server environment with GPIO access and AJAX. Complexity introduced by AJAX
Use the familiar Python and use an embedded web browser If libraries are around, easy
Don't add too much complexity if you're not familiar with the tooling and language
Oh what a nice question! I'm thinking of it right now. My approach is a little difference:
With old MVC fashion, you consider the V(iew) layer is the rendered HTML page with Javascript CSS and many other things, and M and C will run on the server. And one day, I met Mr.AngularJS, I realized: Wow, some basic things may change:
AngularJS considers the View ( or the thing I believed is view ) is not actually view. AngularJS gave me Controllers, Data resources and even View templates in that "View", in another word: Client side itself can be a real Application. So now my approach is:
Server do the "server job" like: read and write data , sends data to the client, receive data from client ect....
And client do the "client job": interact with user, do the logic processes of data BEFORE IT WAS SENT such as validation, or format the information collected from user ect...
Maybe you can re-think of your approach: Ask your self what logic should run at client, what should at server. Client with javascript do its I/O, Server with server-side script do its I/O. The server will provide the needed resource for client and javascript use that resources as M(odel) of it's MVC. Hope you understand, my bad English :D
Well... it sounds like you've mostly settled on:
Python Server. (Python must manage the GPIO.)
HTML/JavaScript client, to create a beautiful UI. (HTML must present the UI.)
That seems great!
You're just wondering how much work to do on each side of the client/server divide... should be functionally equivalent.
In short: Do most of the work in whichever language you are more productive in.
Other notes come to mind:
Writing the entire server as standalone python is pretty
straightforwad.
You don't have to , but it's nice and
self-contained if you serve the page content itself from it.
If you
keep most of the state on the server/python side, you can make the
whole app a little more robust against page reloads (even though I
know you mentioned, that should never happen).
I am trying to get into web development, specially interested building the front-end, UI part of websites while learning JavaScript maybe with AJAX technology. (I have a UI, HCI background.)
However, I have absolutely no previous knowledge about server-end web development either. To my understanding, frameworks like Django seem to pretty good at this (correct me if I am misunderstanding).
So the question is: how much Django, or Rails do I need to know, if my interest is primarily the user interface part of web development. Can I just let somebody else do the back-end stuff?
Pardon me for my imprecise choice of terminologies.
You need to know a bit about the server side. Here's what you need to know.
If you have a heavy JavaScript website, you're likely going to want to pass information from the server to clients with JSON (JavaScript Object Notation). This is just a way to format data into strings that JavaScript knows how to convert to objects.
So, each of your server-side functions that send data to the client will return JSON. If you have someone writing the server-side for you, that's all you should have to know. You're JS functions will receive JSON, and then you deal with it.
If you have to write the server-side yourself, then that involves 1) getting data from database 2) formatting the data 3) converting to JSON.
I have open-sourced a commenting widget that accepts JSON messages, and gives examples of how you would set up the Django server code. Maybe it will help you: http://www.trailbehind.com/comment_widget/
You can make a career of front-end user interface development without know a ton about server code. You would do well though to have at least a rudimentary understanding of what happens on the server when you send it a request, where your data comes from, and what the life-cycle of a web page is. This assumes that you have the support of back-end developers. As you mentioned Ajax in your question that implies that you want your web sites to actually do something, which will require things to happen on the back-end (such as storage, manipulation of data, logging in a user, etc.).
As with all things, the more you know, the easier it will be to get what you want from the dedicated professionals. I would suggest that you learn about programming in general, not try an learn a language and framework. In particular, try to understand datatypes, server settings (like timeouts, post versus get, etc.), security and database interactions as they exist beyond JavaScript/ECMAScript. That way when a developer is explaining why they cannot do something you have requested or are offering alternatives, you are speaking the same language.
Yes and no. Typically what people think of AJAX, such as posting a comment on YouTube and seeing the comment appear instantly with a thank you message, for example, requires a server side language handling the requests, looking up data and returning results as html snippets, JSON data, or XML.
However, an AJAX call can be made to static resources as well. You could have an XML file or html snippet stored statically on your web server and have it loaded. The uses for this sort of static loading are generally fewer because if you already have the static html or data in file next to your regular page, why not just put that data directly into the page?
It helps to set up a local server and write a few lines of code to service your AJAX calls. You can do a lot of JavaScript learning with just a little back-end learning.
If you're new in web development you'd rather wait with Ajax and server-side languages until you've learnt the basics with HTML, CSS and JavaScript, especially if you want to mostly work with the user interface and not the funcionality.
As you said you can let somebody else do the back-end and focus on front-end (JavaScript, HTML, CSS).
You would need to communicate with the back-end developer when storing or processing data from the server.
As mentioned before back-end development knowledge would be useful but if you have someone doing it, it's not essential for beginning.