We Keep Coding

Python dictionary from flask returning null

I have created a home prediction model and currently trying to deploy it to heroku. I am using flask, javascript and html.
Everything works fine when I run it locally, but when deployed in heroku the dropdown menu of locations is empty.
On checking my console on the web browser I noticed that the JS actually got the response from the flask server, but the response is null instead of a list of locations.
Here is relevant codes.
import json
with open('columns.json', 'r') as f
__data_columns = json.load(f)['data_columns']
__locations= __data_columns[4:]
def get_location_names():
return __locations
#app.route('/locations', methods=['GET'])
def locations()
response =jsonify({'locations': get_location_names()})
return response
The result i get on print response is: locations: null how do I resolve this as columns.json contains something like this:
{"data_columns": ["New york", "Washigton"]}

The issue can with the path to the file columns.json. You try to build in such way:
import os
with open(os.path.join(os.path.dirname(__file__), 'columns.json'), 'r') as f:
__data_columns = json.load(f)['data_columns']
__locations= __data_columns[4:]

scrapy + selenium: <a> tag has no href, but content is loaded by javascript

I'm almost there with my first try of using scrapy, selenium to collect data from website with javascript loaded content.
Here is my code:
# -*- coding: utf-8 -*-
import scrapy
from selenium import webdriver
from scrapy.selector import Selector
from scrapy.http import Request
from selenium.webdriver.common.by import By
import time
class FreePlayersSpider(scrapy.Spider):
name = 'free_players'
allowed_domains = ['www.forge-db.com']
start_urls = ['https://www.forge-db.com/fr/fr11/players/?server=fr11']
driver = {}
def __init__(self):
self.driver = webdriver.Chrome('/home/alain/Documents/repository/web/foe-python/chromedriver')
self.driver.get('https://forge-db.com/fr/fr11/players/?server=fr11')
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
#time.sleep(1)
sel = Selector(text = self.driver.page_source)
players = sel.xpath('.//table/tbody/tr')
for player in players:
joueur = player.xpath('.//td[3]/a/text()').get()
guilde = player.xpath('.//td[4]/a/text()').get()
yield {
'player' : joueur,
'guild' : guilde
}
next_page_btn = self.driver.find_element_by_xpath('//a[#class="paginate_button next"]')
if next_page_btn:
time.sleep(2)
next_page_btn.click()
yield scrapy.Request(url = self.start_urls, callback=self.parse)
# Close the selenium driver, so in fact it closes the testing browser
self.driver.quit()
def parse_players(self):
pass
I want to collect user names and their relative guild and output to a csv file.
For now my issue is to proceed to NEXT PAGE and to parse again the content loaded by javascript.
if i'm able to simulate click on NEXT tag, i'm not 100% sure that code will proceed all pages and i'm not able to parse the new content using the same function.
Any idea how could i solve this issue ?
thx.

Instead of using selenium, you should try recreate the request to update the table. If you look closely at the HTML under chrometools. You can see that the request is made with parameters and a response is sent back with the data in a nice structured format.
Please see here with regards to dynamic content in scrapy. As it explains the first step to think about is it necessary to recreate browser activity ? Or can I get the information I need from reverse engineering HTTP get requests. Sometimes the information is hidden with <script></script> tags and you can use some regex or some string methods to gain what you want. Rendering the page and then using browser activity should be thought of as a last step.
Now before I go into some background on reverse engineering the requests, this website you're trying to get information from requires only to reverse engineer the HTTP requests.
Reverse Engineering HTTP requests in Scrapy
Now in terms of the actual web itself we can use chrome devtools by right clicking inspect on a page. Clicking the network tab allows you to see all requests the browser makes to render the page. In this case you want to see what happens when you click next.
Image1: here
Here you can see all the requests made when you click next on the page. I always look for the biggest sized response as that'll most likely have your data.
Image2: here
Here you can see the request headers/params etc... things you need to make a proper HTTP request. We can see that the referring URL is actually getplayers.php with all the params to get the next page added on. If you scroll down you can see all the same parameters it sends to getplayers.php. Keep this in mind, sometimes we need to send headers, cookies and parameters.
Image3: here
Here is the preview of the data we would get back from the server if we make the correct request, it's a nice neat format which is great for scraping.
Now You could copy the headers and parameters, cookies here into scrapy, but after a bit of searching and it's always worth checking this first, if just by passing in an HTTP request with the url will you get the data you want then that is the simplest way.
In this case it's true and infact you get in a nice need format with all the data.
Code example
import scrapy
class TestSpider(scrapy.Spider):
name = 'test'
allowed_domains = ['forge-db.com']
def start_requests(self):
url = 'https://www.forge-db.com/fr/fr11/getPlayers.php?'
yield scrapy.Request(url=url)
def parse(self,response):
for row in response.json()['data']:
yield {'name':row[2],'guild':row[3] }
Settings
In settings.py, you need to set ROBOTSTXT_OBEY = False The site doesn't want you to access this data so we need to set it to false. Be careful, you could end getting banned from the server.
I would also suggest a couple of other settings to be respectful and cache the results so if you want to play around with this large dataset you don't hammer the server.
CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 3
HTTPCACHE_ENABLED = True
HTTPCACHE_DIR = 'httpcache'
Comments on the code
We make a request to https://www.forge-db.com/fr/fr11/getPlayers.php? and if you were to print the response you get all the data from the table, it's quite a lot... Now it looks like it's in json format so we use scrapy's new feature to handle json and convert into a python dictionary. response.json() be sure that you have uptodate scrapy to take advantage of this. Otherwise you could use the json library that python provides to do the same thing.
Now you have to look at the preview data abit here but the individual rows are within response.json()['data'][i] where i in the row of data. The name and guild are within response.json()['data'][i][2] and response.json()['data'][i][3]. So looping over every response.json()['data']and grabbing the name and guild.
If the data wasn't so structured as it is here and it needed modifying I would strongly urge you to use Items or ItemLoaders for creating the fields that you can then output the data. You can modifying the extracted data more easily with ItemLoaders and you can interact with duplicates items etc using a pipeline. These are just some thoughts for in the future, I almost never use yielding a dictionary for extracting data particularly large datasets.

Display arduino serial data on a web browser via python

I'm looking to get serial data from Arduino displayed from a web browser.
First, I put the data on a local host using bottle in python in json format
from bottle import route, run
import serial
dic = {}
ser = serial.Serial(port='COM6',baudrate=9600,timeout=None)
#route('/test')
def test():
c = ser.readline()
c = (str(c)[2:-5]) #just to get rid of the extra characters
dic["val"] = c
return(dic)
run(host='localhost', port=8080, debug=True)
Then I proceed to read it using javascript
function getArduinoVals(){
$.getJSON("http://localhost:8080/test", function(data){
$('#vals').html(data.val);
});
t = setTimeout("getArduinoVals()",50);
});
}
getArduinoVals();
however, it doesn't seem to load from local host (I tested other URLs). How should I fix this? Thanks!

You could use p5.serialport of p5.js for getting the serial data on the web browser. But you have to run the node server in the back.
https://github.com/vanevery/p5.serialport
You should check the github repo for getting started with p5.serialport.
p5.js is very similar to arduino code style so it's easier and convenient than python.

Dealing with GET char limit

I have a .js script that sends data to a .py script running on the local host.
To send data from the .js, I have the following work around (to deal with the limitations of XmlHTTPRequest):
var req = document.createElement("img");
req.src = "http://0.0.0.0:8000?var="+data
To recieve it on the python end:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(5)
conn, addr = s.accept()
data = conn.recv(1024)
conn.close()
print data
This works fine, but my problem is that the char limit on a GET prevents me from sending all of the data I need. I tried making to URI GET requests (using the first block of code twice), but my python only recieved the first request. How can I send/recieve mulitple GET requests? I am assuming I will need some sort of loop, but am unsure what steps need to be in the loop.

From your Python code, it doesn't seem like the HTTP request method used makes any difference. In that case, I'd recommend setting up a form element with method="post" and just putting whatever you want in it. There's no theoretical limit on post body length.

How to connect Javascript to Python sharing data with JSON format in both ways?

I'm trying to find out how to create a local connection between a Python server and a Javascript client using the JSON format for the data to be retrieved. Particularly, I need to make some queries on the HTML client side, send these queries to the server on JSON format and run them on the Python server side to search for data on a SQLite Database. And after getting the results from the database, send those results back to the client in JSON format too.
By now, I just can run the query on Python and code it on JSON like this:
import sqlite3 as dbapi
import json
connection = dbapi.connect("C:/folder/database.db")
mycursor = connection.cursor()
mycursor.execute("select * from people")
results = []
for information in mycursor.fetchall():
results += information
onFormat = json.dumps(results)
print(onFormat)
I know this code does something alike (in fact it runs), because it calls a service on a server which returns data in JSON format (but the server in this example is NOT Python):
<html>
<head>
<style>img{ height: 100px; float: left; }</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
</head>
<body>
<div id="images"></div>
<script>
$.getJSON("http://api.flickr.com/services/feeds/photos_public.gne?jsoncallback=?",
{
tags: "mount rainier",
tagmode: "any",
format: "json"
},
function(data) {
$.each(data.items, function(i,item){
$("<img/>").attr("src", item.media.m).appendTo("#images");
if ( i == 3 ) return false;
});
});</script>
</body>
</html>
What I need is to know how should I run (locally) the python program to be an available running web-service and how should be the Javascript to retrieve the data from the python server.
I've looking for this on internet everywhere but I didn't find this answer anywhere because the only answers they give are on how to code JSON inside Python or inside Javascript but not connecting both. Hope somebody can help me on this!!!

Here's a "hello world" example of a flask web-application that can serve static html and javascript files, search database using parameter from a javascript request, and return results to javascript as json:
import sqlite3
from flask import Flask, jsonify, g, redirect, request, url_for
app = Flask(__name__)
#app.before_request
def before_request():
g.db = sqlite3.connect('database.db')
#app.teardown_request
def teardown_request(exception):
if hasattr(g, 'db'):
g.db.close()
#app.route('/')
def index():
return redirect(url_for('static', filename='page.html'))
#app.route('/json-data/')
def json_data():
# get number of items from the javascript request
nitems = request.args.get('nitems', 2)
# query database
cursor = g.db.execute('select * from items limit ?', (nitems,))
# return json
return jsonify(dict(('item%d' % i, item)
for i, item in enumerate(cursor.fetchall(), start=1)))
if __name__ == '__main__':
app.run(debug=True, host='localhost', port=5001) # http://localhost:5001/
else:
application = app # for a WSGI server e.g.,
# twistd -n web --wsgi=hello_world.application --port tcp:5001:interface=localhost
The database setup code is from Using SQLite 3 with Flask.
static/page.html and static/json-jquery.js files are from Ajax/jQuery.getJSON Simple Example, where the javascript code is modified slightly to pass a different url and nitems parameter:
$(document).ready(function(){
$('#getdata-button').live('click', function(){
$.getJSON('/json-data', {'nitems': 3}, function(data) {
$('#showdata').html("<p>item1="+data.item1+" item2="+data.item2+" item3="+data.item3+"</p>");
});
});
});

Your question amounts to "how do I make this python into a webservice".
Probably the most lightweight ways to do that are web.py and flask. Check them out.
If this is getting bigger, consider django with tastypie - that's a simple way to make a json-based api.
Update: Apparently, there is also a python-javascript RPC framework called Pico, to which Felix Kling is a contributor. The intro says:
Literally add one line of code (import pico) to your Python module to
turn it into a web service that is accessible through the Javascript
(and Python) Pico client libararies.

I found finally an easier way than Flask. It's a Python framework called Bottle You only need to download the library from the official web site and put all its files in your working directory in order to import the library. You can also install it using the setup python program included to avoid carrying with the sourcecode everywhere. Then, for making your Web Service Server you can code it like this:
from bottle import hook, response, route, run, static_file, request
import json
import socket
import sqlite3
#These lines are needed for avoiding the "Access-Control-Allow-Origin" errors
#hook('after_request')
def enable_cors():
response.headers['Access-Control-Allow-Origin'] = '*'
#Note that the text on the route decorator is the name of the resource
# and the name of the function which answers the request could have any name
#route('/examplePage')
def exPage():
return "<h1>This is an example of web page</h1><hr/><h2>Hope you enjoy it!</h2>"
#If you want to return a JSON you can use a common dict of Python,
# the conversion to JSON is automatically done by the framework
#route('/sampleJSON', method='GET')
def mySample():
return { "first": "This is the first", "second": "the second one here", "third": "and finally the third one!" }
#If you have to send parameters, the right sintax is as calling the resoure
# with a kind of path, with the parameters separed with slash ( / ) and they
# MUST to be written inside the lesser/greater than signs ( <parameter_name> )
#route('/dataQuery/<name>/<age>')
def myQuery(name,age):
connection= sqlite3.connect("C:/folder/data.db")
mycursor = connection.cursor()
mycursor.execute("select * from client where name = ? and age= ?",(name, age))
results = mycursor.fetchall()
theQuery = []
for tuple in results:
theQuery.append({"name":tuple[0],"age":tuple[1]})
return json.dumps(theQuery)
#If you want to send images in jpg format you can use this below
#route('/images/<filename:re:.*\.jpg>')
def send_image(filename):
return static_file(filename, root="C:/folder/images", mimetype="image/jpg")
#To send a favicon to a webpage use this below
#route('/favicon.ico')
def favicon():
return static_file('windowIcon.ico', root="C:/folder/images", mimetype="image/ico")
#And the MOST important line to set this program as a web service provider is this
run(host=socket.gethostname(), port=8000)
Finally, you can call the REST web service of your Bottlepy app on a Javascript client in this way:
var addr = "192.168.1.100"
var port = "8000"
function makeQuery(name, age){
jQuery.get("http://"+addr+":"+port+"/dataQuery/"+ name+ "/" + age, function(result){
myRes = jQuery.parseJSON(result);
toStore= "<table border='2' bordercolor='#397056'><tr><td><strong>name</strong></td><td><strong>age</strong></td></tr>";
$.each(myRes, function(i, element){
toStore= toStore+ "<tr><td>"+element.name+"</td><td>" + element.age+ "</td></td></tr>";
})
toStore= toStore+ "</table>"
$('#theDataDiv').text('');
$('<br/>').appendTo('#theDataDiv');
$(toStore).appendTo('#theDataDiv');
$('<br/>').appendTo('#theDataDiv');
})
}
I hope it could be useful for somebody else