I have a Question.
I will use to d3.js using Different chart.
How can I input data using mysql?
Generally, many examples using csv or tsv data.
I know that using this.
But.. I wanna using mysql not csv or tsv dataset.
How can I do this?
data is here
Date Temperature Humidity
2010-01-01 50 40
2010-01-02 55 35
... ... ...
I using web development using flask.
data is getting json.
example)
var data = JSON.parser({{ data | tojson}});
Thanks
2 Options:
Make an XHR requests (e.g. you can use d3.json(url, function(data) {...});) from the browser and on the server side query MySQL and then use Flask's return jsonify(data).
Write out a JSON object from jinja2 directly into a template like the example you show but: var data = {{ data|tojson|safe }};. Note that this has to be in a template file being rendered by Flask's render_template function.
Related
I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?
You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions
Apologies if this seems basic to some, but I'm new to JS/node.js/JSON and still finding my way. I've searched this forum for an hour but cannot find a specific solution.
I have a basic website setup running of a local Node.js server along with 2x JSON data files with information about 32x local suburbs.
An example of an API GET request URL on the site would be:
.../api/b?field=HECTARES
The structure of the JSON files are like:
JSON Structure
In the JSON file there are 32x Features (suburbs), each with it's own list of Properties as shown above. What I am trying to do is use the API 'field' query to push all the HECTARES values each of the 32x Features into a single output variable. The code below is an example of how far I have got:
var fieldStats = [];
var fieldQ = req.query['field'];
for (i in suburbs.features) {
x = suburbs.features[i].properties.HECTARES;
fieldStats.push(x);
}
As you can see in the above "HECTARES" is hard-coded - I need to be able to pass the 'fieldQ' variable to this code but have no idea how to.
Advice appreciated!
Exactly the same syntax you are using just above:
suburbs.features[i].properties[fieldQ];
I have been doing searches for hours upon hours and have tried different snippets of codes and techniques and I simply cannot get any of them to work. I have done research, I am simply a very inexperienced developer trying to learn as I go.
My ultimate goal: Have a live-updating graph of test results data pulled from a MySQL database and rendered using Chart.JS library.
What I have done so far:
I have an HTML file with embedded JavaScript and calls to AJAX function to auto-refresh page every 5 minutes (this has been tested and works).
I have a PHP file which opens a PDO connection to a local MySQL database, runs 3 selection queries, and stores the results into 3 PHP arrays. (I have iterated through the arrays and echoed the results and the data is good).
I have then stored the data inside these PHP arrays into a JSON object using json_encode()
e.g.
$totalTestsPassedArray = array();
$sql = "SELECT totalTestsPassed FROM execution ORDER BY lastDate";
$query = $conn->prepare($sql);
$query->execute(array(':totalTestsPassed' => $totalTestsPassed));
while($row = $query->fetch()){
array_push($totalTestsPassedArray, $row['totalTestsPassed']);
}
$json_totalTestsPassedArray = json_encode($totalTestsPassedArray); //pack data into JSON object
echo $json_totalTestsPassedArray; // for accessibility by AJAX
foreach ($totalTestsPassedArray as $item){ //print out contents of array
echo $item . '<br>';
}
I read that one method to consume this JSON data is to use jQuery's $.getJSON method, and in order to use it, I must 'echo' out the results of my json_encoded object. I realize I am printing the results twice. The echo $json_totalTestsPassedArray is for the aforementioned purpose.
I have included the path of this PHP file in my HTML file from #1 using <?php require '../file_name.php';?>
I have tried several ways of unpacking this JSON encoded object packed with my database data: using AJAX requests, making a new array in the HTML and then flooding it with the JSON object using $.getJSON, and some other things people have suggested to do on various documents online, all met with failure.
Could someone please tell me the proper way to do what I am trying to do? I have a PHP file with an array of data, and I would like to pass that array of data into a dataset for use by Chart.JS to populate a graph and then draw it to the canvas and display my results.
var data = {
labels : ["January","February","March","April","May","June","July"],
datasets : [
{
fillColor : "rgba(220,220,220,0.5)",
strokeColor : "rgba(220,220,220,0.8)",
highlightFill: "rgba(220,220,220,0.75)",
highlightStroke: "rgba(220,220,220,1)",
data : [DATA FROM MY PHP ARRAY IN ANOTHER FILE WOULD GO HERE...]
}
Again, please forgive my ignorance. I have tried my best to figure out the solution on my own and would really appreciate some guidance. It's my first time using these tools. I thought it would be as simple as:
Query DB using PHP script
Store results into array
Pack into JSON object (the goal being to transfer data from PHP to JS file)
Unpack JSON object in JavaScript
Load unpacked data into my JavaScript array
In php print only json and set header content type to application/json then your XHR response will contain javascript object with your data. There is no need for any json parsing.
I am trying to send Javascript Array via AJAX POST to django view.
I store this array in hidden input using JSON.stringify:
$('#id_uuids').val(JSON.stringify(arr));
this is how I try to send it:
$.post("/ajax/generateGallery",{uuids: $('#id_uuids').val()},function(response){
resp = JSON.parse(response);
alert(resp.html);
},"json");
Browser console shows that data which is being send looks like:
uuids:["6ecbe35b-0b77-4810-aa9a-918fecaeef13","e41f52f7-721b-4d44-b7d6-bbb275182d66"]
However, I am not able to use this in my django view. I've tried:
uuids = request.POST.getlist('uuids')
logger.info(uuids)
logger.info(type(uuids))
which returns:
[08/Aug/2014 15:20:00] INFO [app.rest_client:307] [u'["89a26646-6000-4c48-804a-69abcc496fd8"]']
[08/Aug/2014 15:20:00] INFO [app.rest_client:308] <type 'list'>
[08/Aug/2014 15:20:00] INFO [app.rest_client:312] Generate HTML gallery for photo ["89a26646-6000-4c48-804a-69abcc496fd8"]
So, Django treats very list sent as single element. How can I force python code to treat this data as list and be able to iterate on?
try to JSON-decode the posted data
uuids = json.loads(request.POST.get('uuids'))
that is if you loaded some json module before, e.g.
import simplejson as json
I want to build a stock website with Django, and I found a Javascript library (tickp) to make charts, but I don't know Javascript, nor do I know how to read json data in a Django template. I use this code to retrieve the stock data from Yahoo Finance.
I put these .py and .js in my folder, and my views like this:
from stock.stockretriever import StockRetriever
def stockretriever(request,number):
data = StockRetriever().get_historical_info('YHOO')
return HttpResponse(simplejson.dumps(data),mimetype='application/json')
But I don't know how I should write the template, can somebody tell me?
thanks.
You have two options:
render a template with data included
dynamically fetch data from the server
If you go for 1. you could add something like this to your template:
<script type="text/javascript">
var my_data = {{ json_plot_data }};
</script>
The template includes also the javascript code that generates the plots from the data. The view function would include the data fetching and return a context object like so:
def my_stock_plot_view(request, number):
# get stock data
json_data = simplejson.dumps(data)
django.shortcuts.render(request, 'template.html', {'json_plot_data':json_data})
If you go for 2. you would need to use something like jQuery.ajax to dynamically load json data using an ajax request. That request would invoke your view, in the jQuery.ajax call you specify that the request returns JSON which automatically makes the data available as an object to Javascript. In the jQuery.ajax success handler you would pass the data to your plot function.
I don't have these libraries installed, but based on the readme of the tickp library, you'll need the following data: [date, open, high, low, close and optionally volume]. The get_historical_info function returns the columns [Date, Open, High, Low, Close, Volume, AdjClose]. The mismatch here is the AdjClose, so you'd need to strip that from the data you get from the StockRetriever:
from django.shortcuts import render
from stock.stockretriever import StockRetriever
def stockretriever(request, number):
data = StockRetriever().get_historical_info('YHOO')
# Assuming data is returned as a list of lists
new_data = [d[:-1] for d in data]
return render(request, 'stock.html', { 'data': simplejson.dumps(new_data) })
Following along with the readme, you need something along the following lines in your template:
<html>
<head><script src="tickp.js"></script><script src="stats.js"></script></head>
<body onload='plot = window.tickp("#chart"); plot.read({{ data }}); plot.plot();'>
<div id="chart"></div>
</body>
</html>
Note that I've cut some corners with respect to possible Ajax calls or proper formatting and usage, but it should give you something to get started with. When you're missing something, please update your question with the specific issues you're having.