I'm looking to get serial data from Arduino displayed from a web browser.
First, I put the data on a local host using bottle in python in json format
from bottle import route, run
import serial
dic = {}
ser = serial.Serial(port='COM6',baudrate=9600,timeout=None)
#route('/test')
def test():
c = ser.readline()
c = (str(c)[2:-5]) #just to get rid of the extra characters
dic["val"] = c
return(dic)
run(host='localhost', port=8080, debug=True)
Then I proceed to read it using javascript
function getArduinoVals(){
$.getJSON("http://localhost:8080/test", function(data){
$('#vals').html(data.val);
});
t = setTimeout("getArduinoVals()",50);
});
}
getArduinoVals();
however, it doesn't seem to load from local host (I tested other URLs). How should I fix this? Thanks!
You could use p5.serialport of p5.js for getting the serial data on the web browser. But you have to run the node server in the back.
https://github.com/vanevery/p5.serialport
You should check the github repo for getting started with p5.serialport.
p5.js is very similar to arduino code style so it's easier and convenient than python.
Related
So I am trying to access data on a video game stat tracker website. Now when I go to inspect element on the website and look at the code it says:
<div class="trn-defstat__value">Division 7</div>
But when I use requests.get(url).text the same element shows up as:
<div class="trn-defstat__value">{{ activeArena.division.metadata.description }}</div>
I am trying to get the "Division 7" part but keep getting this activeArena thing, I am using python, the code I have tried is
import requests
url = ('https://fortnitetracker.com/profile/all/tl%20starrlol/competitive?season=16')
file = open("myfilename", "w")
r = requests.get(url)
info = r.content
info = str(info)
file.write(info)
file.close()
and I have also tried
import requests
url = ('https://fortnitetracker.com/profile/all/tl%20starrlol/competitive?season=16')
file = open("myfilename", "w")
r = requests.get(url)
info = r.text
file.write(info)
file.close()
I am pretty new to coding so if the answer is obvious I apologize, but I am lost.
The HTML you're receiving contains a template engine code, the javascript on the page is loading and filling it up with values. If you examine the page via the network panel on the browser you'll notice a stats API call. Make the same call from your code to extract the data you need.
import requests
url = "https://fortnitetracker.com/api/v0/profile/863f1c3c-2e61-487e-8987-ceefff2981ad/stats"
querystring = {"season":"16","isCompetitive":"true"}
response = requests.request("GET", url, data="", headers={}, params=querystring)
data = response.json()
print (data[0]['arena']['division']['displayValue'])
# prints "Contender League Division 7"
It's better to check for official APIs instead of this approach. The parameters in the API like the UUID after profile may be a parameter that's valid only for a certain time. It's also worth evaluating the Selenium or Puppeteer approach recommended in the comments(under the question) to see if that fits your overall problem.
I'm trying to create a simple webpage which gets data from a microcontroller (MSP432) and plots it on a graph in real time.
I have researched and I'm currently using plotly.js and html to create the graph. I currently have a graph that updates itself with random data in realtime.
I have attached the code for the page below.
I now want to stream the data in, and I have looked at node.js and in particular the serialport library. I think I have installed the serialport npm.
The part where I'm confused is how does the node.js code fit in with my html/plotlyjs index file?
I have never used javascript before. It's my first time with nodejs especially, so I'm confused. Should I just put it inside the <script> tag right in the getX() getY() function?
Also does anyone know how I can go about starting the nodejs code? I'm kind of lost.
This is my index.html file:
<!DOCTYPE html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<script src="plotly.min.js"></script>
<meta charset="utf-8" />
<title>My Plot Page</title>
</head>
<body>
<p>Plotly plot:</p>
<div id="chart1"></div>
<script>
var counter = 0;
function getX() {
counter++;
return counter;
}
function getY() {
return Math.random();
}
Plotly.plot('chart1', [{
x: [getX()],
y: [getY()],
type: 'line'
}]);
setInterval(function () {
Plotly.extendTraces('chart1', { x: [[getX()]] , y: [[getY()]] }, [0])
}, 200);
</script>
</body>
</html>
Node.js is an open source server environment. It is used to host a webserver and can't be run inside the webbrowser. It can be used to read serial port data and send it to the connected clients.
An example setup would be as below:
---------------------------------------------------------------------------
| MSP432 node.js server web page in browser |
| M <==================> N <===========================> W |
| serial websockets |
---------------------------------------------------------------------------
The nodejs server (N) reads the data from the serial port and manages the MSP432 (M). The server (N) also hosts a webserver (using expressjs) and opens a websocket with the connected webpage (W) to transfer new data. An easy to use websocket library is socket.io.
A simple nodejs webserver can be created by doing:
express --view=pug <your application name>
please note that ejs can also be used instead of pug
ajax can also be used instead of websockets
From experience (I work with data visualization systems), I can tell you Plotly might not be the easiest way to plot real-time data. It can do it for sure, but what it does is re-render the whole plot for every new point, which considering the data structure is not the most efficient method and looks a bit unnatural when updating.
In your use case, I would recommend this plugin instead: http://smoothiecharts.org/ it's lightweight, free, and "smooth". Read the "10 minute tutorial" on their website and you're good to go.
Summarizing, to each smoothiecharts timeseries, you want to use the append() method to add each new data sample. You can call this using a setTimeInterval() JavaScript method to get the value from... a URL (making an AJAX GET request). You want to create a URL that will always return the latest reading from your sensor, and for this you'll need to create a local webserver.
This is where your NodeJS application fits in: Create a simple NodeJS server with a GET method that returns the latest reading from your device. Something like this is enough to return simple content like a number: https://www.tutorialspoint.com/nodejs/nodejs_first_application , you could print a random number first and work on it until it works before continuing.
The hardest part here though is to get your microcontroller's readings into the NodeJS application. I don't know the specifics of the MSP432 or how you're connecting to it, but usually you can write a C script that reads the latest value from it and prints it to the console.
If you manage to do this, you can use the following NodeJS code to execute your C program and read its console output:
var exec = require('child_process').execFile;
var path = 'c:\yourprogram.exe';
var fun = function() {
exec(path, function(err, data) {
console.log(err)
console.log(data.toString());
});
}
fun();
You can easily tweak that code into a function that updates the NodeJS variable that your server returns, and run it in a loop every X milliseconds.
And that would be your full visualization system.
I created a Web Crawler in Python 3.7 that pulls different info and stores them into 4 different arrays. I have now come across an issue that I am not sure how to fix. I want to use the data from those four arrays in my site and place them into a table made from JS and HTML/CSS. How do I go about accessing the info from my Python file in my JavaScript file? I tried searching in other places before creating an account, and came across some things that talk of using Json, but I am not too familiar with these and would appreciate some help if that is the way to do it. I will post my code below which I have stored in the same directory as my other sites files. Thanks in advance!
from requests import get
from bs4 import BeautifulSoup
from flask import Flask
app = Flask(__name__)
#app.route("/")
def main():
# lists to store data
names = []
gp = []
collectionScore = []
arenaRank = []
url = 'https://swgoh.gg/g/21284/gid-1-800-druidia/'
response = get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# username of the guild members:
for users in soup.findAll('strong'):
if users.text.strip().encode("utf-8") != '':
if users.text.strip().encode("utf-8") == '\xe9\x82\x93\xe6\xb5\xb7':
names.append('Deniz')
else:
names.append(users.text.strip().encode("utf-8"))
if users.text.strip().encode("utf-8") == 'Note':
names.remove('Note')
if users.text.strip().encode("utf-8") == 'GP':
names.remove('GP')
if users.text.strip().encode("utf-8") == 'CS':
names.remove('CS')
print(names)
# GP of the guild members:
for galacticPower in soup.find_all('td', class_='text-center'):
gp.append(galacticPower.text.strip().encode("utf-8"))
totLen = len(gp)
i = 0
finGP = []
while i < totLen:
finGP.append(gp[i])
i += 4
print(finGP)
# CS of the guild members:
j = 1
while j < totLen:
collectionScore.append(gp[j])
j += 4
print(collectionScore)
# Arena rank of guild member:
k = 2
while k < totLen:
arenaRank.append(gp[k])
k += 4
print(arenaRank)
if __name__ == "__main__":
app.run()
TLDR: I want to use the four lists - finGP, names, collectionScore, and arenaRank in a JavaScript or HTML file. How do I go about doing this?
Ok, this will be somewhat long but I'm going to try breaking it down into simple steps. The goal of this answer is to:
Have you get a basic webpage being generated and served from python.
Insert the results of your script as javascript into the page.
Do some basic rendering with the data.
What this answer is not:
An in-depth javascript and python tutorial. We don't want to overload you with too many concepts at one time. You should eventually learn about databases and caching, but that's further down the road.
Ok, here's what I want you to do first. Read and implement this tutorial up until the "Creating a Signup Page" section. That starts to get into dealing with Mysql, which isn't something you need to worry about right now.
Next, you need to execute your scraping script when a request for the server. When you get the results back, you output those into the html page template inside a script tag that looks like:
<script>
const data = [];
console.log(data);
</script>
Inside the brackets in data = [] use json.dumps (https://docs.python.org/2/library/json.html) to format your Python array data as json. Json is actually a subset of javascript, so you just output it as a raw javascript string here and it gets loaded into the webpage via the script tag.
The console.log statement in the script tag will show the data in the dev tools in your browser.
For now, lets pause here. Get all of this working first (probably a few hours to a day's work). Getting into doing html rendering with javascript is a different topic and again, I don't want to overload you with too much information right now.
Leave comments on this answer if you need extra help.
I am new to python and am trying to access the db though python and return some results in a JSON array using AJAX.
I test it by returning a JSON list and alerting it using js. it works when I don't use the db connection but as soon as I add it the js alert stops too. the db connection seems to work properly when I run the file getSchedule.py. the db connection is in a separate file webairdb.py
Can someone please try to help me figure out whats wrong?
getSchedule.py
#!D:/Programming/Software/python3.4.4/python
import sys, json,cgi, cgitb, mysql.connector, webairdb
cgitb.enable()
fs = cgi.FieldStorage()
sys.stdout.write("Content-Type: application/json")
sys.stdout.write("\n")
sys.stdout.write("\n")
conn = webairdb.getConnection()
conn.close()
listr = [11111]
sys.stdout.write(json.dumps(listr))
sys.stdout.write("\n")
sys.stdout.close()
webairdb.py
#!D:/Programming/Software/python3.4.4/python
import cgi, cgitb, imp, mysql.connector
host ="localhost"
db = "webair"
user = "root"
password = ""
def getConnection():
conn = mysql.connector.connect(user=user,password=password,host=host,database=db)
if conn.is_connected():
print("aaaqqqq")
return conn
In webairdb.py you write to sys.stdout (that is what print does) - putting effectively breaking the json output. (You might want to have a look at the output by pressing F12 in your browser)
So just remove it and either write to sys.stderr or use logging instead.
You should also consider using wsgi instead of cgi which makes things a bit easier (no need to care about printing at all) or a framework like bottle or cherrypy.
I'm trying to find out how to create a local connection between a Python server and a Javascript client using the JSON format for the data to be retrieved. Particularly, I need to make some queries on the HTML client side, send these queries to the server on JSON format and run them on the Python server side to search for data on a SQLite Database. And after getting the results from the database, send those results back to the client in JSON format too.
By now, I just can run the query on Python and code it on JSON like this:
import sqlite3 as dbapi
import json
connection = dbapi.connect("C:/folder/database.db")
mycursor = connection.cursor()
mycursor.execute("select * from people")
results = []
for information in mycursor.fetchall():
results += information
onFormat = json.dumps(results)
print(onFormat)
I know this code does something alike (in fact it runs), because it calls a service on a server which returns data in JSON format (but the server in this example is NOT Python):
<html>
<head>
<style>img{ height: 100px; float: left; }</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
</head>
<body>
<div id="images"></div>
<script>
$.getJSON("http://api.flickr.com/services/feeds/photos_public.gne?jsoncallback=?",
{
tags: "mount rainier",
tagmode: "any",
format: "json"
},
function(data) {
$.each(data.items, function(i,item){
$("<img/>").attr("src", item.media.m).appendTo("#images");
if ( i == 3 ) return false;
});
});</script>
</body>
</html>
What I need is to know how should I run (locally) the python program to be an available running web-service and how should be the Javascript to retrieve the data from the python server.
I've looking for this on internet everywhere but I didn't find this answer anywhere because the only answers they give are on how to code JSON inside Python or inside Javascript but not connecting both. Hope somebody can help me on this!!!
Here's a "hello world" example of a flask web-application that can serve static html and javascript files, search database using parameter from a javascript request, and return results to javascript as json:
import sqlite3
from flask import Flask, jsonify, g, redirect, request, url_for
app = Flask(__name__)
#app.before_request
def before_request():
g.db = sqlite3.connect('database.db')
#app.teardown_request
def teardown_request(exception):
if hasattr(g, 'db'):
g.db.close()
#app.route('/')
def index():
return redirect(url_for('static', filename='page.html'))
#app.route('/json-data/')
def json_data():
# get number of items from the javascript request
nitems = request.args.get('nitems', 2)
# query database
cursor = g.db.execute('select * from items limit ?', (nitems,))
# return json
return jsonify(dict(('item%d' % i, item)
for i, item in enumerate(cursor.fetchall(), start=1)))
if __name__ == '__main__':
app.run(debug=True, host='localhost', port=5001) # http://localhost:5001/
else:
application = app # for a WSGI server e.g.,
# twistd -n web --wsgi=hello_world.application --port tcp:5001:interface=localhost
The database setup code is from Using SQLite 3 with Flask.
static/page.html and static/json-jquery.js files are from Ajax/jQuery.getJSON Simple Example, where the javascript code is modified slightly to pass a different url and nitems parameter:
$(document).ready(function(){
$('#getdata-button').live('click', function(){
$.getJSON('/json-data', {'nitems': 3}, function(data) {
$('#showdata').html("<p>item1="+data.item1+" item2="+data.item2+" item3="+data.item3+"</p>");
});
});
});
Your question amounts to "how do I make this python into a webservice".
Probably the most lightweight ways to do that are web.py and flask. Check them out.
If this is getting bigger, consider django with tastypie - that's a simple way to make a json-based api.
Update: Apparently, there is also a python-javascript RPC framework called Pico, to which Felix Kling is a contributor. The intro says:
Literally add one line of code (import pico) to your Python module to
turn it into a web service that is accessible through the Javascript
(and Python) Pico client libararies.
I found finally an easier way than Flask. It's a Python framework called Bottle You only need to download the library from the official web site and put all its files in your working directory in order to import the library. You can also install it using the setup python program included to avoid carrying with the sourcecode everywhere. Then, for making your Web Service Server you can code it like this:
from bottle import hook, response, route, run, static_file, request
import json
import socket
import sqlite3
#These lines are needed for avoiding the "Access-Control-Allow-Origin" errors
#hook('after_request')
def enable_cors():
response.headers['Access-Control-Allow-Origin'] = '*'
#Note that the text on the route decorator is the name of the resource
# and the name of the function which answers the request could have any name
#route('/examplePage')
def exPage():
return "<h1>This is an example of web page</h1><hr/><h2>Hope you enjoy it!</h2>"
#If you want to return a JSON you can use a common dict of Python,
# the conversion to JSON is automatically done by the framework
#route('/sampleJSON', method='GET')
def mySample():
return { "first": "This is the first", "second": "the second one here", "third": "and finally the third one!" }
#If you have to send parameters, the right sintax is as calling the resoure
# with a kind of path, with the parameters separed with slash ( / ) and they
# MUST to be written inside the lesser/greater than signs ( <parameter_name> )
#route('/dataQuery/<name>/<age>')
def myQuery(name,age):
connection= sqlite3.connect("C:/folder/data.db")
mycursor = connection.cursor()
mycursor.execute("select * from client where name = ? and age= ?",(name, age))
results = mycursor.fetchall()
theQuery = []
for tuple in results:
theQuery.append({"name":tuple[0],"age":tuple[1]})
return json.dumps(theQuery)
#If you want to send images in jpg format you can use this below
#route('/images/<filename:re:.*\.jpg>')
def send_image(filename):
return static_file(filename, root="C:/folder/images", mimetype="image/jpg")
#To send a favicon to a webpage use this below
#route('/favicon.ico')
def favicon():
return static_file('windowIcon.ico', root="C:/folder/images", mimetype="image/ico")
#And the MOST important line to set this program as a web service provider is this
run(host=socket.gethostname(), port=8000)
Finally, you can call the REST web service of your Bottlepy app on a Javascript client in this way:
var addr = "192.168.1.100"
var port = "8000"
function makeQuery(name, age){
jQuery.get("http://"+addr+":"+port+"/dataQuery/"+ name+ "/" + age, function(result){
myRes = jQuery.parseJSON(result);
toStore= "<table border='2' bordercolor='#397056'><tr><td><strong>name</strong></td><td><strong>age</strong></td></tr>";
$.each(myRes, function(i, element){
toStore= toStore+ "<tr><td>"+element.name+"</td><td>" + element.age+ "</td></td></tr>";
})
toStore= toStore+ "</table>"
$('#theDataDiv').text('');
$('<br/>').appendTo('#theDataDiv');
$(toStore).appendTo('#theDataDiv');
$('<br/>').appendTo('#theDataDiv');
})
}
I hope it could be useful for somebody else