I have external programs such as ffmpeg and gstreamer running in the background and writing to a log file. I want to display the contents of this log with my Flask application, so that the user can watch the log update, like tail -f job.log would do in the terminal.
I tried to use <object data="/out.log" type="text/plain"> to point at the log file, but that failed to show the data, or the browser told me I needed a plugin.
How can I embed and update the log file in an HTML page?
Use a Flask view to continuously read from the file forever and stream the response. Use JavaScript to read from the stream and update the page. This example sends the entire file, you may want to truncate that at some point to save bandwidth and memory. This example sleeps between reads to reduce cpu load from the endless loop and allow other threads more active time.
from time import sleep
from flask import Flask, render_template
app = Flask(__name__)
#app.route('/')
def index():
return render_template('index.html')
#app.route('/stream')
def stream():
def generate():
with open('job.log') as f:
while True:
yield f.read()
sleep(1)
return app.response_class(generate(), mimetype='text/plain')
app.run()
<pre id="output"></pre>
<script>
var output = document.getElementById('output');
var xhr = new XMLHttpRequest();
xhr.open('GET', '{{ url_for('stream') }}');
xhr.send();
setInterval(function() {
output.textContent = xhr.responseText;
}, 1000);
</script>
This is almost the same as this answer, which describes how to stream and parse messages, although reading from an external file forever was novel enough to be it's own answer. The code here is simpler because we don't care about parsing messages or ending the stream, just tailing the file forever.
I am using frontail package from npm.
npm i frontail -g
frontail /var/log/syslog
visit http://127.0.0.1:9001 to view logs
Source: https://github.com/mthenw/frontail
This may not be the exact answer for the question(to embed an html page), but it solves the problem of many users who are looking specifically only for
Display the contents of a log file as it is updated
For me #davidism solution (accepted answer) worked only on Firefox. It didnt work in Chrome, Brave, Vivaldi. Maybe there was some kind of de-sync in backend and frontend loops? I dont know.
Anyway i used far simpler solution, without loop on the backend and javascript loop on frontend. Maybe it's "uglier" and may cause trouble for some very long logs, but at least it works on every browser i use.
#app.route('/stream')
def stream():
with open("job.log", "r") as f:
content = f.read()
# as you see, file is loaded only once, no loop here, (loop is on frontend side)
return app.response_class(content, mimetype='text/plain')
<!DOCTYPE html>
<html>
<head>
<!-- page auto-refresh every 10 seconds -->
<meta http-equiv="refresh" content="10">
<title>Some title</title>
</head>
<body>
<h1>Log file ...</h1>
<script>
// function for adjusting iframe height to log size
function resizeIframe(obj) {
obj.style.height = obj.contentWindow.document.documentElement.scrollHeight + 'px';
}
</script>
<!-- iframe pulls whole file -->
<iframe src="{{ url_for('stream') }}" frameborder="0" style="overflow:hidden;width:100%" width="100%" frameborder="0" scrolling="no" onload="resizeIframe(this)"></iframe>
</body>
</html>
As you see the only javascript code is used to adjust iframe height to current text size.
Related
I'm trying to get python ReactiveX stream (using RxPy library) to be sent to a javascript on Web UI component, but I can't seem to find a way to do so. Also, I might need to get the data stream coming into the Javascript into a RxJS Observable of sorts for further processing.
Could you please help me understand how to achieve this?
I'm still getting a grip on ReactiveX so maybe there are some fundamental concepts I'm missing, but I'm struggling to find anything similar to this around the net.
This issue has come up as I'm working on a desktop app that takes data from a csv or a zeromq endpoint, and streams it to a UI where the data will be plotted dynamically (updated the plot as new data comes in). I'm using Electron to build my app, using python as my backend code. Python is a must as I will be extending the app with some TensorFlow models.
Following fyears really well made example as an initial structure, I have written some sample code to play with but I can't seem to get it to work.
I manage to get from the UI button all the way to the python scripts, but I get stuck in the return of the PricesApi.get_stream(...) method.
index.html
The front end is straight forward.
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Electron Application</title>
</head>
<body>
<button id="super-button">Trigger Python Code</button>
<div id="py-output">
</div>
</body>
<script src="renderer.js" ></script>
</html>
api.py:
The ZeroRPC server file is like the one in the above mentioned link.
import gevent
import json
import signal
import zerorpc
from core_operator import stream
class PricesApi(object):
def get_stream(self, filename):
return stream(filename)
def stop(self):
print('Stopping strategy.')
def echo(self, text):
"""echo any text"""
return text
def load_settings():
with open('settings.json') as json_settings:
settings_dictionary = json.load(json_settings)
return settings_dictionary
def main():
settings = load_settings()
s = zerorpc.Server(PricesApi())
s.bind(settings['address'])
print(f"Initialising server on {settings['address']}")
s.run()
if __name__ == '__main__':
main()
core_operator.py
This is the file were the major logic will sit to get prices from zeroMQ subscription, but currently just creates an Observable from a csv.
import sys
import rx
from csv import DictReader
def prepare_csv_timeseries_stream(filename):
return rx.from_(DictReader(open(filename, 'r')))
def stream(filename):
price_observable = prepare_csv_timeseries_stream(filename)
return price_observable
rendered.js
finally, the javascript that should be receiving the stream:
const zerorpc = require('zerorpc');
const fs = require('fs')
const settings_block = JSON.parse(fs.readFileSync('./settings.json').toString());
let client = new zerorpc.Client();
client.connect(settings_block['address']);
let button = document.querySelector('#super-button');
let pyOutput = document.querySelector('#py-output');
let filename = '%path-to-file%'
button.addEventListener('click', () => {
let line_to_write = '1'
console.log('button click received.')
client.invoke('get_stream', filename, (error, result) => {
var messages = pyOutput;
message = document.createElement('li'),
content = document.createTextNode(error.data);
message.appendChild(content);
messages.appendChild(message);
if(error) {
console.error(error);
} else {
var messages = pyOutput;
message = document.createElement('li'),
content = document.createTextNode(result.data);
message.appendChild(content);
messages.appendChild(message);
}
})
})
I have been looking into using WebSockets, but failed in understanding how to implement it. I did find some examples using Tornado server, however I am trying to keep it as pure as possible and, also, it feels odd that having already a client/server structure from Electron, I'm not able to use that directly.
Also I'm trying to maintain the entire system a PUSH structure as the data requirements don't allow for a PULL type of pattern, with regular pollings etc.
Thank you very much in advance for any time you can dedicate to this, and please let me know if you require any further details or explanations.
I found a solution by using an amazing library called Eel (described as "A little Python library for making simple Electron-like HTML/JS GUI apps"). Its absolute simplicity and intuitiveness allowed me to achieve what I wanted a few simple lines.
Follow the intro to understand the layout.
Then your main python file (which I conveniently named main.py), you expose the stream function to eel, so it can be called from JS file, and pipe the stream into the JavaScript "receive_price" function which is exposed from the JS file!
import sys
import rx
from csv import DictReader
def prepare_csv_timeseries_stream(filename):
return rx.from_(DictReader(open(filename, 'r')))
def process_logic():
return pipe(
ops.map(lambda p: print(p)), # just to view what's flowing through
ops.map(lambda p: eel.receive_price(p)), # KEY FUNCTION in JS file, exposed via eel, is called for each price.
)
#eel.expose # Decorator so this function can get triggered from JavaScript
def stream(filename):
price_observable = prepare_csv_timeseries_stream(filename)
price_observable.pipe(process_logic()).subscribe() # apply the pipe and subscribe to trigger stream
eel.init('web')
eel.start('main.html') # look at how beautiful and elegant this is!
Now we create the price_processing.js file (placed in the 'web' folder as per Eel instructions) to incorporate the exposed functions
let button = document.querySelector('#super-button');
let pyOutput = document.querySelector('#py-output' );
let filename = '%path-to-file%'
console.log("ready to receive data!")
eel.expose(receive_price); // Exposing the function to Python, to process each price
function receive_price(result) {
var messages = pyOutput;
message = document.createElement('li');
content = document.createTextNode(result);
message.appendChild(content);
messages.appendChild(message);
// in here you can add more functions to process data, e.g. logging, charting and so on..
};
button.addEventListener('click', () => {
console.log('Button clicked magnificently! Bloody good job')
eel.stream(filename); // calling the Python function exposed through Eel to start stream.
})
The HTML stays almost the same, apart from the changing the script refs: /eel.js, as per Eel documentation and our price_processing.js file.
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Let's try Eel</title>
</head>
<body>
<h1>Eel-saved-my-life: the App!</h1>
<button id="super-button">Trigger Python Code</button>
<div id="py-output">
</div>
</body>
<script type="text/javascript" src="/eel.js"></script>
<script type="text/javascript" src="price_processing.js"></script>
</html>
I hope this can help anyone struggling with the same problem.
Using Python 3 with Socket.
I'm having trouble linking a javascript file to an HTML file. I have the following 3 files in particular in the same directory:
webserver.py
import socket
def getTextFromFile(filename):
with open(filename, 'r') as myFile:
return myFile.read()
host, port = '192.168.0.7', 11010
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host, port))
s.listen(1)
print('Now serving: ' + host + ' at: ' + str(port))
while True:
try:
c, addr = s.accept()
print('Received connection form: ' + str(addr))
request = c.recv(1024)
response = '\HTTP/1.1 200 OK' + getTextFromFile("index.html")
c.sendall(response.encode())
c.close()
except Exception as e:
print(e)
index.html
<html>
<head>
<title>Title</title>
<script src="indexjs.js"></script>
</head>
<body>
<h1>Hello there.</h1>
<p>This is a webserver test.</p>
<button type="button" onclick="myFunction()">Click me!</button>
<p id="demo"></p>
</body>
</html>
indexjs.js
function myFunction(){
document.getElementById("demo").innerHTML = "Hello";
}
The web server code itself works fine, the router is port-forwarded, can be accessed from an entirely different network, yada-yada; but when the website is visited, the javascript doesn't seem to execute. If I place the same javascript code inside the script tags of index.html and omit indexjs.js then it does work as intended. How do I make it to where indexjs.js can be properly linked with index.html without putting the javascript code in script tags?
I also tried another similar approach by trying to put an image in index.html (png file, same directory as index.html of course) with the img tags, but it did not display either, so it seems like my code is having issues linking any files to index.html whatsoever.
The steps I'm taking:
Run webserver.py
Open browser (in my case it happens to be Chrome)
Type in my public ip xxx.xxx.xxx.xxx:11010 in the address bar, hit enter
The web page shows, but pictures don't show (or any other asset that's linked) and javascript functions aren't executed when, for example, buttons are pressed.
Any help would be appreciated, thanks in advance.
Normally you would have a public folder for your static assets. And you would put your static css and js files into it.
I am working on a web-scraping project. One of the websites I am working with has the data coming from Javascript.
There was a suggestion on one of my earlier questions that I can directly call the Javascript from Python, but I'm not sure how to accomplish this.
For example: If a JavaScript function is defined as: add_2(var,var2)
How would I call that JavaScript function from Python?
Find a JavaScript interpreter that has Python bindings. (Try Rhino? V8? SeaMonkey?). When you have found one, it should come with examples of how to use it from python.
Python itself, however, does not include a JavaScript interpreter.
To interact with JavaScript from Python I use webkit, which is the browser renderer behind Chrome and Safari. There are Python bindings to webkit through Qt. In particular there is a function for executing JavaScript called evaluateJavaScript().
Here is a full example to execute JavaScript and extract the final HTML.
An interesting alternative I discovered recently is the Python bond module, which can be used to communicate with a NodeJs process (v8 engine).
Usage would be very similar to the pyv8 bindings, but you can directly use any NodeJs library without modification, which is a major selling point for me.
Your python code would look like this:
val = js.call('add2', var1, var2)
or even:
add2 = js.callable('add2')
val = add2(var1, var2)
Calling functions though is definitely slower than pyv8, so it greatly depends on your needs. If you need to use an npm package that does a lot of heavy-lifting, bond is great. You can even have more nodejs processes running in parallel.
But if you just need to call a bunch of JS functions (for instance, to have the same validation functions between the browser/backend), pyv8 will definitely be a lot faster.
You can eventually get the JavaScript from the page and execute it through some interpreter (such as v8 or Rhino). However, you can get a good result in a way easier way by using some functional testing tools, such as Selenium or Splinter. These solutions launch a browser and effectively load the page - it can be slow but assures that the expected browser displayed content will be available.
For example, consider the HTML document below:
<html>
<head>
<title>Test</title>
<script type="text/javascript">
function addContent(divId) {
var div = document.getElementById(divId);
div.innerHTML = '<em>My content!</em>';
}
</script>
</head>
<body>
<p>The element below will receive content</p>
<div id="mydiv" />
<script type="text/javascript">addContent('mydiv')</script>
</body>
</html>
The script below will use Splinter. Splinter will launch Firefox and after the complete load of the page it will get the content added to a div by JavaScript:
from splinter.browser import Browser
import os.path
browser = Browser()
browser.visit('file://' + os.path.realpath('test.html'))
elements = browser.find_by_css("#mydiv")
div = elements[0]
print div.value
browser.quit()
The result will be the content printed in the stdout.
You might call node through Popen.
My example how to do it
print execute('''function (args) {
var result = 0;
args.map(function (i) {
result += i;
});
return result;
}''', args=[[1, 2, 3, 4, 5]])
Hi so one possible solution would be to use ajax with flask to comunicate between javascript and python. You would run a server with flask and then open the website in a browser. This way you could run javascript functions when the website is created via pythoncode or with a button how it is done in this example.
HTML code:
<html>
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script>
function pycall() {
$.getJSON('/pycall', {content: "content from js"},function(data) {
alert(data.result);
});
}
</script>
<button type="button" onclick="pycall()">click me</button>
</html>
Python Code:
from flask import Flask, jsonify, render_template, request
app = Flask(__name__)
def load_file(file_name):
data = None
with open(file_name, 'r') as file:
data = file.read()
return data
#app.route('/pycall')
def pycall():
content = request.args.get('content', 0, type=str)
print("call_received",content)
return jsonify(result="data from python")
#app.route('/')
def index():
return load_file("basic.html")
import webbrowser
print("opening localhost")
url = "http://127.0.0.1:5000/"
webbrowser.open(url)
app.run()
output in python:
call_received content from js
alert in browser:
data from python
This worked for me for simple js file, source:
https://www.geeksforgeeks.org/how-to-run-javascript-from-python/
pip install js2py
pip install temp
file.py
import js2py
eval_res, tempfile = js2py.run_file("scripts/dev/test.js")
tempfile.wish("GeeksforGeeks")
scripts/dev/test.js
function wish(name) {
console.log("Hello, " + name + "!")
}
Did a whole run-down of the different methods recently.
PyQt4
node.js/zombie.js
phantomjs
Phantomjs was the winner hands down, very straightforward with lots of examples.
This is probably a simple question but I got somehow confused.
I'm using flask webserver to keep my UI up. This server generates a log file. I want to have, let's say, an iframe below the html page (the one which is rendered via flask render_template('index.html') ) showing the contents of that file.
I'm aware of the questions like this and thanks to #davidism I've learnt nice concepts about serving the static files but that's not exactly what I want to achieve.
So far I can say, again thanks to #davidism, that I have a main.py file like this:
from time import sleep
from flask import Flask, render_template
app = Flask(__name__)
#app.route('/')
def index():
return render_template('index.html')
#app.route('/stream')
def stream():
def generate():
with open('job.log') as f:
while True:
yield f.read()
sleep(1)
return app.response_class(generate(), mimetype='text/plain')
app.run()
and by using this:
<pre id="output"></pre>
<script>
var output = document.getElementById('output');
var xhr = new XMLHttpRequest();
xhr.open('GET', '{{ url_for('stream') }}');
xhr.send();
setInterval(function() {
output.textContent = xhr.responseText;
}, 1000);
</script>
I get that log in a clean full page. How can I get this in that aforementioned way.
p.s: I have multiple html files which will be rendered in the future and I want to implement this in a way to have that iframe view of the log file in all pages.
An iframe is literally just another page contained within an element on your page. So your main page would just do:
<iframe src ="/stream"></iframe>
and that's it.
The title of this question may be slightly misleading, but I'm not sure what the best title would be (since I can't guess at a solution yet).
Basically the system I am developing relies heavily on canvas graphs. These graphs are generated through javascript, and are made using data pulled via ajax from an API server.
The tricky part is, I'd like to be able to email these graphs to users of this system, without them actually having to go to the web page at all. So while I'm aware that it is possible to get the Base64 value of an image generated with javascript in a browser, what about if no one is there to run that javascript?
I'd like to keep the graphs generated in javascript/canvas, rather than making them in a common server-side graphics library (GD, ImageMagick). The Canvas graphs are dynamic, and allow for interaction via javascript. Though I don't want that functionality in the email notification, I do want them to be identical otherwise (at least in appearance).
So the question is, how can I get these graphs into an email?
At this point, my only guess is that I'd need to literally make a website that does AJAX requests for "graphs to render", renders these graphs, and sends the results to the server. Then I'd need a "server" that just sits there on that web page and churns out graphs. Is that the only solution here?
I used phantomJs (like node.js but different) serverside to run exactly the same code as client side, and get the same result. all you need is one single exe-file (like a webkit stand alone web brower)
The following program (in Perl, but should be feasible to translate to you favourite language) takes some data, inserts into a web-page (could be ajax'ed) and either sends that web page to the client, or stores it as a temporary file, and starts PhantomJs on the same page. Then ask PhantomJs to generate a jpg, that is then picked up (and in this case sendt to the client).
#!/usr/bin/perl
use strict;
use File::Temp;
$|=1;
#this script returns a graph, either as html +js web page to render client side,
#or renders the same page server side, and returns the jpg image.
#files needed:
#.\phantom_srv_client.pl #this script
#.\phantomjs.exe #the webkit runtime stand alone file, from http://phantomjs.org/
#.\Scripts\excanvas.min.js #canvas simulator for IE8-
#.\Scripts\jquery.min.js #jQuery as we know it
#.\Scripts\jquery.jqplot.min.js #graph library on top of jQuery from http://www.jqplot.com/ (Flot or any can be used)
#do we want client side rendering (html + js), or server side rendering (jpg)
#jpg seems to render nicer than png on some pages?
use CGI;
my $show_as_jpg = CGI::param("jpg");
#path to javascript libraries (jQuery etc).
#Must be absolute file location for server rendering, relative for web
use FindBin;
my $script_path = $show_as_jpg
? $FindBin::Bin."/Scripts"
: './Scripts';
#data to send to graph (two sets)
my $data = [[2,5,4], [6,4,5]];
#use json to get this as a javascript text
my $json_data;
eval {require JSON; $json_data=JSON::to_json($data)};
#in case JSON is not installed, get the json/javascript data manually (just for demo)
$json_data ||= "[[2,5,4], [6,4,9]]"; #here 9 at the end to see a difference
#The following is the web page that renders the graph, client or server side
#(links to scripts must be abolute to work serverside, as temp-files may go anywhere, $script_path keeps track of that)
#$json_data is the Perl data structure converted to JSON (aka javascript, but not)
my $graph_html =qq|
<!DOCTYPE html>
<html>
<head>
<!--[if lt IE 9]><script language="javascript" type="text/javascript" src="$script_path/excanvas.min.js"></script><![endif]-->
<script class="include" type="text/javascript" src="$script_path/jquery.min.js"></script>
<script class="include" type="text/javascript" src="$script_path/jquery.jqplot.min.js"></script>
<script class="code" type="text/javascript" language="javascript">
jQuery(document).ready(function(){
/*data from perl (\$json_data) inserted here */
var data = $json_data;
jQuery.jqplot("chart1", data );
});
</script>
</head>
<body>
<div id="chart1" style="width:600px; height:400px;"></div>
<a href='?jpg=1'>View as jpg</a>
</body>
</html>
|;
#this is the javascript that tells PhantomJs what to do (ie open a doc and render it to bitmap)
my $phantom_doc_js =qq|
var system = require('system');
//read from commandline which files to open, and write to
var open_doc = system.args[1];
var return_doc = system.args[2];
var page = require('webpage').create();
page.open(open_doc, function () {
page.render(return_doc);
phantom.exit();
});
|;
#see if we shall render this page serverside
if ($show_as_jpg) {
#get temporary filenames with related file handlers
#where to put phantomjs script (generic so could be a static file)
my ($phantom_doc_filehandler, $phantom_doc_filename) = File::Temp::tempfile( SUFFIX => '.js', TMPDIR => 1);
#where to put web page with data to render and ref to javascripts etc
my ($phantom_graph_filehandler, $phantom_graph_filename) = File::Temp::tempfile(SUFFIX => '.html', TMPDIR => 1);
#also get a filename with no handler, so phantomjs can return the jpg file. Extention must be .jpg!
my (undef, $image_filename) = File::Temp::tempfile( SUFFIX => '.jpg',TMPDIR => 1, OPEN => 0);
#store file content and close files
print $phantom_doc_filehandler $phantom_doc_js; close $phantom_doc_filehandler;
print $phantom_graph_filehandler $graph_html; close $phantom_graph_filehandler;
#now call PhantomJs with filenames to read from and write to.
#Next version should support piping, which would simplify a lot
#use absolute path to phantomjs.exe in case web-server does not use current path
system($FindBin::Bin.'\\phantomjs', $phantom_doc_filename, $phantom_graph_filename, $image_filename) == 0
or die "system failed: $?";
#read the entire image file
my $img = slurp_file($image_filename);
print "Content-Type: image/jpeg\nPragma: no-cache\n\n".$img;
#The temp files are no more needed
unlink $phantom_doc_filename, $phantom_graph_filename, $image_filename;
} else { # just render client side
print "Content-Type: text/html\nPragma: no-cache\n\n".$graph_html;
}
#slurp is not always std perl
sub slurp_file{
my $filename = shift;
my $string;
local $/ = undef;
open FILE, $filename or die "Couldn't open file: $!";
binmode FILE;
$string = <FILE>;
close FILE;
return $string;
}