I see some relevant posts to my query.
Tornado is used in the below link
How to pass variable from python to javascript
I know that it can be done using json but I am not clear how to implement it.
In the web2py default controller I am returning a dictionary which contains the latitudes and longitudes.
def index():
lat_long_list=[]
info1 = {'lat':'1.0032','long':'2.00003','name':'Akash'}
info2 = {'lat':'1.2312','long':'-1.0034','name':'Kalyan'}
lat_long_list.append(info1)
lat_long_list.append(info2)
return dict(lat_long_list=lat_long_list)
In java script I want to iterate through the list of dictionaries and mark the points on the google maps.
I cannot say
<script>
{{ for lat_long_rec in lat_long_list :}}
var name = {{=lat_long_rec['name']}}
{{ pass }}
</script>
This fails. An alternative to handle this is to write the list into an xml and from javascript read the file but I dont want to achieve it this way as writing to file is non performant. Let me know how best this can achieved.
Convert the Python list to JSON and pass that to the view to insert in the Javascript code:
from gluon.serializers import json
return dict(lat_long_list=json(lat_long_list))
In the view:
<script>
...
var latLongList = {{=XML(lat_long_list)}}
...
</script>
Related
I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?
You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions
I want to export a JSON string in python into a JS variable.
<script type="text/javascript">
var data = JSON.parse('{{ dataJSON }}');
console.log(data)
</script>
If I print the content of dataJSON I get: [{"offset":0,"total":1,"units":[{"village_id":37,"village_name":"Glim
But in the JS I get this: JSON.parse('[{"offset":0,"total":1,"units":[{"village_id":37
I use jinja2 template engine: http://jinja.pocoo.org/docs/dev/templates/#if
How can I fix that?
You need to mark the data as safe:
var data = {{ dataJSON|safe }};
This prevents it from being HTML-escaped. There is no need to use JSON.parse() this way; JSON is a valid JavaScript subset (at least insofar that the Python json module produces a valid subset).
Take into account that this doesn't make it JavaScript safe. You may want to adjust your JSON serialisation. If you are using Flask, a tojson filter is provided that ensures you get JavaScript-safe valid JSON:
var data = {{ data|tojson|safe }};
If you are not using Flask, post-process the JSON:
dataJSON = (json.dumps(data)
.replace(u'<', u'\\u003c')
.replace(u'>', u'\\u003e')
.replace(u'&', u'\\u0026')
.replace(u"'", u'\\u0027'))
This is Python code to produce a dataJSON value that can be safely used in HTML (including attribute values) and in JavaScript. Credit here goes to the Flask json.htmlsafe_dumps() function.
I'm trying to pass a string of generated HTML from Python to Javascript. This is the simplified version of my view (using Pyramid framework).
#view_config(route_name='view_page', renderer='templates/page.jinja2', permission='view')
def view_cosmeceutical(request):
gen_html = markdown( ingredient.desc )
return dict(gen_html=gen_html)
From this thread, I saw that I can use {{ variable }} in Javascript too, so in my JS file, I have a function that attempts to change the innerHTML of the element with the id 'content-desc'. But all I'm getting is the string {{ genHTML }} instead of the actual variable containing the generated HTML.
function insertHTML() {
genHTML = '{{gen_html}}';
$('#content-desc').html(genHTML);
}
What am I doing wrong?
One good way to pass data and content from the server-side Python to JavaScript are
JSON embeds in HTML
Separate AJAX calls which serve JSON objects as application/json mime
For the embed approach, I would do somethibng along the lines to export data to the page template as JSON:
import json
gen_html = ...
javascript_data = json.dumps(dict(gen_html=gen_html))
return dict(javascript_data=javascript_data)
Then in the page template pass this to JavaScript global variables:
<script>
window.javascriptData = {{javascript_data}}; // Check need to escape HTML in your case
</script>
And then in JavaScript (keep preferably in a separate static .JS file):
$(document).ready(function() {
$('#content-desc').html(window.javascriptData.gen_html);
});
Also, I would not generate HTML for just passing it to JavaScript in the first place. Instead, I would pass raw data to JavaScript as JSON, and then generate HTML on the client-side from this data using client-side templating. This increases the complexity, but is more "clean" and flexible.
Example microtemplating engines for the client-side JavaScript:
DOM tree based JavaScript template engines
I know that the Jinja2 library allows me to pass datastore models from my python code to html and access this data from inside the html code as shown in this example . However Jinja2 isn't compatible with javascript and I want to access the data inside my Javascript code . What is the simplest templating library which allows to iterate over my datastore entities in Javascript ? I've heard about things like Mustache and Jquery , I think they look a bit too complicated. Is there anything simpler?
You should create a python controller which serves JSON formatted data, which any Javascript library (especially jQuery) can consume from. Then, setup the Jinja2 template to contain some Javascript which calls, loads and displays said data.
One more approach to consider: If the Python object is not dynamic, you may want to use json.dumps() to store it as a TextProperty, and simply JSON.parse(unescape(model_text)) on the JS side. Reduces the overhead, and memory hit which can be important when trying to stay within an F1 limit. For example, I run an instance that very easily runs inside an F1. There is one large dictionary object that we deliver. Were this object to exist as a Python dictionary inside the instance we would kill the instance due to the soft memory limit. Using the TextProperty approach we can pass this large dict to the client without any issues. (Note: we did have to momentarily boost our instance up to an F4 when initially creating this object -- something incredibly easy inside the Admin web page.) With more dynamic objects, answers above apply.
Jinja2 and Javascript play fine together. You need to arrange to have template expansion emit your Python data structures into a JS-friendly form.
https://sites.google.com/a/khanacademy.org/forge/technical/autoescape-in-jinja2-templates covers it fairly well. (Note the use of the escapejs filter.)
It works. I had to serialize(convert) my datastore entities to json format, which Javascript understands well. I created a function which converts every instance of my datastore into a dictionnary then encapsulates all these instances into a list which is then converted to Json using json.dumps. When I pass this result to the Java script , I can then easily access my values as seen below.
import json
import webapp2
from google.appengine.ext import db
import jinja2
JINJA_ENVIRONMENT = jinja2.Environment(
loader=jinja2.FileSystemLoader(os.path.dirname(__file__)),
extensions=['jinja2.ext.autoescape'],
autoescape=True)
# serialize datastore model to JSON format
def serialize(model):
allInstances = model.all() # fetching every instance of model
itemsList = [] #initial empty list
for p in allInstances:
d = db.to_dict(p)
itemsList.append(d)
return json.dumps(itemsList)
class myModel(db.Model):
v = db.FloatProperty()
c = db.FloatProperty()
tdate = db.DateTimeProperty(auto_now_add=True)
class MainPage(webapp2.RequestHandler):
def get(self):
myModel(v=4.5, c=3.0).put()
#creating template variables
template_values = {
'json_data': serialize(myModel)
}
template = JINJA_ENVIRONMENT.get_template('index.html')
self.response.write(template.render(template_values))
Inside my 'index.html' file, I have:
{% autoescape true %}
<!DOCTYPE html>
<html>
<head>
<title> webpage </title>
<script type="text/javascript">
// I retrieve my data here
var results = "{{ json_data }}";
for(var i = 0; i < db_results.length; i++) {
document.write("myModel instance:" + i + results[i] + "<br>");
}
</script>
</head>
<body>
</body>
</html>
{% endautoescape %}
It has nothing to do with compatibility. Jinja is server side templating. You can use javascript for client side coding.
Using Jinja you can create HTML, which can be accessed by javascript like normal HTML.
To send datastore entities to your client you can use Jinja to pass a Python list or use a json webservice.
I want to build a stock website with Django, and I found a Javascript library (tickp) to make charts, but I don't know Javascript, nor do I know how to read json data in a Django template. I use this code to retrieve the stock data from Yahoo Finance.
I put these .py and .js in my folder, and my views like this:
from stock.stockretriever import StockRetriever
def stockretriever(request,number):
data = StockRetriever().get_historical_info('YHOO')
return HttpResponse(simplejson.dumps(data),mimetype='application/json')
But I don't know how I should write the template, can somebody tell me?
thanks.
You have two options:
render a template with data included
dynamically fetch data from the server
If you go for 1. you could add something like this to your template:
<script type="text/javascript">
var my_data = {{ json_plot_data }};
</script>
The template includes also the javascript code that generates the plots from the data. The view function would include the data fetching and return a context object like so:
def my_stock_plot_view(request, number):
# get stock data
json_data = simplejson.dumps(data)
django.shortcuts.render(request, 'template.html', {'json_plot_data':json_data})
If you go for 2. you would need to use something like jQuery.ajax to dynamically load json data using an ajax request. That request would invoke your view, in the jQuery.ajax call you specify that the request returns JSON which automatically makes the data available as an object to Javascript. In the jQuery.ajax success handler you would pass the data to your plot function.
I don't have these libraries installed, but based on the readme of the tickp library, you'll need the following data: [date, open, high, low, close and optionally volume]. The get_historical_info function returns the columns [Date, Open, High, Low, Close, Volume, AdjClose]. The mismatch here is the AdjClose, so you'd need to strip that from the data you get from the StockRetriever:
from django.shortcuts import render
from stock.stockretriever import StockRetriever
def stockretriever(request, number):
data = StockRetriever().get_historical_info('YHOO')
# Assuming data is returned as a list of lists
new_data = [d[:-1] for d in data]
return render(request, 'stock.html', { 'data': simplejson.dumps(new_data) })
Following along with the readme, you need something along the following lines in your template:
<html>
<head><script src="tickp.js"></script><script src="stats.js"></script></head>
<body onload='plot = window.tickp("#chart"); plot.read({{ data }}); plot.plot();'>
<div id="chart"></div>
</body>
</html>
Note that I've cut some corners with respect to possible Ajax calls or proper formatting and usage, but it should give you something to get started with. When you're missing something, please update your question with the specific issues you're having.