I am currently creating a real time online web console that takes output from Bash and displays it on a PHP driven, HTML page. At the current moment, I'm using the method of fetching the entire contents of a page and replacing a DIV's contents with the contents of the fetched page, which is very inefficient, and am wondering how to do this more efficiently, such as with WebSockets, Faye, or long polling.
What is the fastest way I can get Bash output to a client?
If you think I should use something that requires a server-side install, why?
WebSockets will give you significantly lower latency than AJAX/long-polling requests if that is important for your project.
There is a telnet example included with websockify (Disclaimer: I made websockify). Websockify bridges between websockets and TCP sockets. You might be able to use some ideas from that. In particular, you may want to look at the Javascript terminal emulator code that is included with that example if you want to implement anything more than just display of raw text.
Related
I'm working right now on a project that could allow me to generate movies based on the user input. User will upload some samples (photos, movies) to the web app and web server should generate movie based on that input and some predefined movie compositions.
I know that there are plenty of libraries for ffmpeg that could let me connect movies, photos programmatically (for example https://github.com/schaermu/node-fluent-ffmpeg for node.js) , but I was wondering if it's possible to use Aftereffects for that purpose since I have some knowledge in that software. I imagine that there should be set of scripts in Aftereffects that could import user uploaded data, fire the movie renderer and save output to the given location.
Do you think this is achievable using Aftereffects? Or maybe someone had similar problem and solved that differently ?
Cheers!
I have done the exactly same thing.
I DO NOT suggest you use script to do it. I have made the same mistake. Script is fine for a small job, but when you try to use it on a web server and run constantly for days and days it's very unstable. You will be facing a lot of crashing.
I would suggest you use sdk to do the job. It's much more difficult to use sdk than using script, but is more stable and much faster! When you try to create a web service app, you want it to be stable and fast, don't you?
Yes, this is definitely possible. There are existing scripts for rendering and uploading via FTP, and the possibilities are pretty much endless. The part that jumps out at me as needing clarification is "scripts in Aftereffects that could import user uploaded data". This suggests a need for a back-end that "looks for" or "waits for" elements to "appear" in a directory to kick off the ExtendScript script in AE. This is where it gets slightly dicey in that you have to devise a way to do this with a "daemon" in your preferred operating system using any number of languages -- python, Java, AppleScript, shell, batch, etc. The rest of it ("import user uploaded data, fire the movie renderer and save output to the given location") could be done in ExtendScript.
This is more a general question, I'm not asking for implementation details.
I have a Python script running on a machine which monitors an experiment and gathers live data from it. I want to visualise the data on a website and decided to go with d3.js
My current setup however is absolutely basic:
The python script is running in an endless loop and appends new data
to a text-file (CSV file).
There is an HTML-page on the web
server with a javascript, which grabs the file and creates a graph
using d3.js.
I already thought about replacing the text-file with a sqlite database, but I'm sure there is a more sophisticated and easy way to do it via network.
So I am looking for a more solid solution and searched the web for different kind of strategies but most of those involve bigger libraries (Tornado, node.js etc.).
I would be happy with quite basic socket-like solution and tried to figure out how it would work with WebSockets but not sure how to handle this.
So my question is, what kind of design would you recommend or use if your only goal is to have a Javascript running in a browser and waiting/listening for data (let's say every few seconds a float), which is sent from a Python script?
Question regarding javascript debugging:
We have a mobile app, made with JavaScript and HTML. This app is running on the mobile platform (BlackBerry 10, Android, iOS) inside a web container.
There is another part of this application running on the remote server. This part is implemented also with JavaScript running on Node.js.
Assume there is some communication established between the client (JS on mobile) and the server (JS on Node.js) sides using e.g. REST.
The question I would like to know is if it is possible to seamlessly debug both sides at the same time.
E.g. if I have a breakpoint on the mobile app client side, how would I be able to debug all the way to JS on the server side, if it’s possible at all.
Any suggestions would help.
You can use node-inspector on the server, then you'll have TWO instances, but the same debugger toolset.
If you're stepping through code client, and want to debug "into" the server, you must place a breakpoint in the server debugger before making the GET/POST from the client.
TIP: Get a three (at least two) monitor setup.
Using the node inspector is a good strategy for doing debugging, but it would also be good to supplement the debugger with logging.
Debugging will let you step through an event chain and examine values of variables and output of functions in that chain, but in production it won't give you insight into the steps or production conditions that lead to errors users are experiencing (i.e. Why do I have a null variable? Why is my response message wrong?) If you use debugging without logging you'll be trying to replicate the conditions in production that are causing an error, which is an inefficient (and usually futile) way of doing things.
I'd say the best way to implement what you want to do (solve issues taking into account client & server events that happen concurrently) is to implement an open source logging library like Log4j on both your server and your client and configure an appender to send the logs to a log aggregator like Loggly which gives you tools to analyze both client & server logs in the same place (rather than extracting log files from both devices manually).
Once you've done this, you'll be able to distribute your application out to testers and you'll be able to see what actions, application logs, and hardware/network conditions surround certain issues. With this data in hand you'll know a lot better what leads to certain bugs and can use that information to much more effectively use node-inspector to debug them.
I need to get from thousands of online JSON about 300.000 final lines, equal to 30MB.
Being beginner in coding, I prefer to stick to JS to $getJSON data, cut it, append interesting parts to my <body>, and loop on the thousands online JSON. But I wonder :
can my web-browser handles 300.000 $getJSON queries and the resulting 30~50MB webpage without crashing ?
is it possible to use JS to write down a file with this results, so the script's works is constantly saved ?
I expect my script to run about 24 hours. Numbers are estimations.
Edit: I don't have server side knowledge, just JS.
A few things aren't right about your approach for this:
If what you are doing is fetching (and processing) data from another source then displaying it for a visitor, processing of this scale should be done separately and beforehand in a background process. Web browsers should not be used as data processors on the scale you're talking about.
If you try to display a 30-50MB webpage, your user is going to experience lots of frustrating issues - browser crashes, lack of responsiveness, timeouts, long load times, and so on. If you expect any users on older IE browsers, they might as well give up without even trying.
My recommendation is to pull this task out and do it using your backend infrastructure, saving the results in a database which can then be searched, filtered, and accessed by your user. Some options worth looking into:
Cron
Cron will allow you to run a task on a repeated and regular basis, such as daily or hourly. Use this if you want to continually update your dataset.
Worker (Heroku)
If running Heroku, take it out of the dyno and use a separate worker so as not to clog up any existing traffic on your app.
the same problem haunting me a month ago is still haunting me now. i know ive asked several questions regarding this on this site and i am truly sorry for that. your suggestions have all been excellent but the answer is still elusive. i now realize that this is a direct result of me not being able to phrase my question properly and for that i am sorry.
to give you guys a generalized view of things, here i go: the situation is like this, i have 2 server side scripts that i want to run.
a python program/script that continuously spouts some numbers
based on the output from that python script, a javascript script will perform some action on a webpage (e.g., change background color, display alert message, change some text)
ive studied the replies to my previous posts and have found that what i want to accomplish is more or less accomplished by json. it is my understanding that json transforms 'program-specific' variables into a format that is more 'standard or general or global'.
two different programs therefore now have the means to 'talk' with each other because they are now speaking the same 'language'.
the problem is then this, how do i actually facilitate their communication? what is the 'cellphone' between these server side scripts? do they even need one?
thank you!
If I understand what you're asking, the "cellphone" is TCP/IP. The javascript is not server-side; it runs on the client side, and alters what the client's browser displays based on json data that it downloads from the server -- data that in this case is generated by Python.
This question provides a relevant example, though it's a bit technical: JSON datetime between Python and JavaScript
Here's a very basic tutorial that explains how to create a dynamic webpage using python and javascript. It doesn't appear to use json, but it should familiarize you with the fundamentals. Once you understand what's there, using json to transport more complicated data should be fairly straightforward.
http://kooneiform.wordpress.com/2010/02/28/python-and-ajax-for-beginners-with-webpy-and-jquery/
I assume you mean: Python is on the web server, and Javascript is running in the client's web browser.
Because browsers are all different (IE6 is terrible, Chrome is great), there are a huge number of ways people found to "hack" this "cellphone" into place. These techniques are called AJAX and COMET techniques. There is no one "cellphone", but a whole bunch of them! Hopefully, you can find a library to select the right technique for the browser, and you just have to worry about the messages.
Comet is harder to do, but lets the server "push" messages to the client.
Ajax can be easier - you just periodically "pull" messages from the server.
Start with Ajax, then look at comet if you really need it. Just start by have the client (javascript) make a "GET" request, to see if the number has changed.
I don't know Javascript or json, but...
if you've ever seen an Unix-like operating system, you know about pipes. Like program1 | program2 | program3 ... Why don't you just connect Python and Javascript programs with pipes? The first one writes to stdout, and the next one reads from stdin.
This probably isn't the answer that you are looking for, and without links to your previous posts, I don't have much to go on, but nonetheless...
javascript is client side. I can interpret your question 2 different ways...
Your python script is running on your computer, and you want a script to actually alter your current browser window.
Not too sure, but writing a browser plugin may be the answer here.
Your python script is running on the server, and as a result of the script running, you want the display of your site to be changed for viewing persons.
In this case, you will could use ajax polling (or similar) on your site. Have your site be polling the server with ajax, call a server method that checks the output of the script (maybe written to a file?), and see if it has changed.
When 2 process need to communicate, they need to decide of a common/shared way to express things and a protocol to exchange those things.
In your case, since one of the processes is a browser, the protocol of choice is http. So the browser needs to do an http request or regular http request to your python process.
This python process Will need in Some way or another to be exposed via http.
There are several ways to build a web server in python. You should read this article : http://fragments.turtlemeat.com/pythonwebserver.php as a jumpstart.
Once you have this, your browser Will be able to issue HTTP GET requests to your server and your server can reply with a string.
This string can be whatever you like. Nevertheless if your answer contains structured data it can be a good start to use the XML notation or the json notation.
Json (stands for Javascript object notation) is very easy to use in javascript and this is why many people advised you to choose this notation.
I hope this will help you
Jérome wagner