For a research project we are developing a system in python which takes raw data from a real time sensor and applies different DSP filters to it. So far so good. The other part of the system should be the plotting of the data and the access to some controllers for modifying the sample rate and other parameters. The last days I have been trying to use HTML5 + javascript as the technology to do the data plotting; I am using d3.js for the graphics and mimic.js to implement the xmlrpc protocol to communicate the python side with the HTML file. After a lot of tweaking i ALMOST managed to communicate both parts of the system. However, I ran into the 'Cross origin request' error (I am using chrome).
After that I have been thinking of some other strategies in order to communicate python with the GUI:
Using a SimpleHTTPServer. However, I need to have a SimpleXMLRPCServer running so I can register the function (register_function) to pass the value. Is there a way to implement a xmlrpc within a SimpleHTTPServer?
I have been thinking about using json but I am not sure if this is the best way to go as the data streaming will be quite big and fast(4 channels with constant updates).
So after all this testing I really would appreciate a piece of advice:
Is if feasible to have such HTML5 GUI? is it efficient for plotting such amounts of data?
if so, what would be a realistic strategy to implement a communication between Python and HTML/js?
Is xmlrpc the best protocol to use?
Yep. Also you can use some JavaScript charts libraries. Such as amCharts, highcharts, plot.
Really, JSON is better.
XMLRPC isn't very light. See p.2 :)
Related
I'm currently working on a project that involves parsing through a user-supplied file, doing computations with that data, and visualizing the results using graphing utilities. Right now, I'm stuck with using Python as the back-end because it has scientific libraries unavailable in JavaScript, but I want to move the entire tool to a web server, where I can do much slicker visualizations using D3.js.
The workflow would be something like: obtain the file contents from the browser, execute the Python script with the contents, return jsonified objects of computed values, and plot those objects using D3. I already have the back-end and front-end working by themselves, but want to know: How can I go about bridging the two? From what I've gathered, I need to do something along the lines of launching a server, sending AJAX requests to the server, and retrieving data from the server. But with the number of frameworks out there (Flask, cgi, apache, websockets, etc.), I'm not really sure where to start. This will likely only be a very simple web app with just a file submit page and a data visualization page. Any help is appreciated!
Apache is a web server, flask is a web framework in python, websockets are a protocol, and cgi is something totally different, and none of them help you on the front end.
You could deploy a simple backend in flask or django or pylons or any other python web framework. I like django, but it may be a little heavy for your purpose, flask is a bit more lightweight. You deploy them to a server with a web server installed and use something like apache to distribute.
Then you need a front end and a way of delivering your front end. Flask / Django are both fully capable of doing so in conjunction with a web server, or you could use a static file server like Amazon S3.
On your front end, you need to load D3 and probably some kind of utility like jQuery to load and parse your data from the back end, then use D3 however you like to present it on screen.
Flask is easy to get up and running and is Python based. It works well with REST APIs and data sent by JSON (or JSON API).
This is one solution with which I have some experience and which seems to work well and is not hard to get up and running (and natural to work with Python). I can't tell you whether it is the best solution for your needs, but it should work.
If you are overwhelmed and don't know where to start, you can pick one of the options and google search for a tutorial. With a decent tutorial, you should have an example up and running by the end of the tutorial, and then you will know if you are comfortable working with it and have an idea whether it will meet your needs.
Then you could do a proof-of-concept; make a small app that just handles one small part (the one you are most concerned about handling, perhaps) and write something which will do it.
By then, you can be pretty sure you have a good tool for the purpose (unless you were convinced otherwise by the proof-of-concept -- in which case, try again with another option :-))
I have been tasked with researching technology for generating a dynamic report with charts & visuals.
There are countless BI tools available and online solutions that fit the bill, however, the customers have considerable IT restrictions preventing the installation of applications or sharing data across the internet.
One solution I've dreamed up is to serve up an HTML document for offline consumption in a browser. With javaScript bundled in & data added dynamically prior to sending to the client, I believe this would be possible in much the same way a website functioning offline is possible.
How feasible is this? If it's possible, what technologies are recommended? So far I've considered building this using AngularJS since an SPA could mimic a one-page report well & javaScript bundling is readily available.
Are there better solutions? Which?
May be your clients will be okay with providing them files by SFTP. Say, if you are not using a database and are ok with building charts based on files, you can try to provide end users with the files (txt, csv, xml...). Then you will be able to execute client-side code with no server needed. You may even go for some open-source Javascript libraries for data visualisation like D3. All you have to do is to allow local file access for user's browser.
Otherwise you need a local server. I would recommend you to use Python python's SimpleHTTPServer or node.js may fit you.
I'm working on a project to create real-time charts from data in an SQL database. I've done a lot of research on JS charting libraries and have picked out a few good ones.
My problem is that I'm not sure how to get started with linking the SQL data to the web app/site.
My HTML and CSS are fine. I'm coming on with JS and we have software that provides me with necessary SQL query code.
Where is the gap in my knowledge? Is it necessary to learn PHP in order to get this working? How about C#?
My solution will currently be a locally hosted site, running on a networked computer (not currently got server space). The solution I have already uses Excel to query the database and runs a ton of macros to generate static charts every ten minutes. This works, but I'd like to create something far more flexible, robust and interactive.
Could anybody point me to any online resources that may help me? I like ChartJS but any open-source charting library will do. I'm hoping there's a great tutorial somewhere that covers SQL to real-time charts but I just can't find one...
If you want a real time charting you need to receive information from the DB almost every second. So sending a bunch of requests is not the best way to do this. You probably need to read about websockets and how do they work. Here is an example but you may find a lot more in google. If you don't want to do it with PHP, I can assure you that it would be far more easier with node.js
This is more a general question, I'm not asking for implementation details.
I have a Python script running on a machine which monitors an experiment and gathers live data from it. I want to visualise the data on a website and decided to go with d3.js
My current setup however is absolutely basic:
The python script is running in an endless loop and appends new data
to a text-file (CSV file).
There is an HTML-page on the web
server with a javascript, which grabs the file and creates a graph
using d3.js.
I already thought about replacing the text-file with a sqlite database, but I'm sure there is a more sophisticated and easy way to do it via network.
So I am looking for a more solid solution and searched the web for different kind of strategies but most of those involve bigger libraries (Tornado, node.js etc.).
I would be happy with quite basic socket-like solution and tried to figure out how it would work with WebSockets but not sure how to handle this.
So my question is, what kind of design would you recommend or use if your only goal is to have a Javascript running in a browser and waiting/listening for data (let's say every few seconds a float), which is sent from a Python script?
I have a page where i keep getting data from a database periodically. That data is depicted as a graph. At each period, i want to update the graph with new data. Meaning that i dont want to replace the old graph, but extend it. For example,im using a line chart which is currently displaying. When new data comes in, i want the lines in the chart to extend to represent the new data, but the previous lines should remain as-is. Which JavaScript graphing API can provide this functionality ? Is it possible with Google Charts ?
Well, whether you like it or not, you're in the middle of writing a real-time web application process :D
Here are your solutions:
Pure javascript
If you're in the mood of using a library:
I highly recommend using APE library. as I said multiple times before, it's a robust framework for developing real-time web apps.
Another great solution is HTML5's websockets. it's awesome but not cross-browser.
If you're not in the mood of using a library:
You may send an AJAX request to your server periodically to check whether any updates are available for your data or not.
Using third party plugins
The easiest and the most common solution is Adobe Flash's XMLSocket APIs.
You may use Silverlight too, but you know, it doesn't worth it (development time I meant).
EDIT:
Oh, almost forgot to say, this style of programming (updating data without user's interaction/awareness) is called Comet programming, so if you need a keyword to search further on this subject use this one.
I agree with 3p3r, and can elaborate on library implementation.
A topic you may want to keep in mind is Data-Binding.
It is a process commonly employed by MVC systems such as backbone.js.
The notion is that the model(data) is automatically updated into the View when data changes are detected .
Data-Binding is not only available in MVC it is also available in graphing widgets for libraries like ExtJS. Using a library can greatly reduce your maintenance overhead in development and avoid commonly encountered complications.
Other libraries that may easily incorporate a data-binding process are: D3, HighCharts, and Raphael.
All the best!
Nash