I want to build a stock website with Django, and I found a Javascript library (tickp) to make charts, but I don't know Javascript, nor do I know how to read json data in a Django template. I use this code to retrieve the stock data from Yahoo Finance.
I put these .py and .js in my folder, and my views like this:
from stock.stockretriever import StockRetriever
def stockretriever(request,number):
data = StockRetriever().get_historical_info('YHOO')
return HttpResponse(simplejson.dumps(data),mimetype='application/json')
But I don't know how I should write the template, can somebody tell me?
thanks.
You have two options:
render a template with data included
dynamically fetch data from the server
If you go for 1. you could add something like this to your template:
<script type="text/javascript">
var my_data = {{ json_plot_data }};
</script>
The template includes also the javascript code that generates the plots from the data. The view function would include the data fetching and return a context object like so:
def my_stock_plot_view(request, number):
# get stock data
json_data = simplejson.dumps(data)
django.shortcuts.render(request, 'template.html', {'json_plot_data':json_data})
If you go for 2. you would need to use something like jQuery.ajax to dynamically load json data using an ajax request. That request would invoke your view, in the jQuery.ajax call you specify that the request returns JSON which automatically makes the data available as an object to Javascript. In the jQuery.ajax success handler you would pass the data to your plot function.
I don't have these libraries installed, but based on the readme of the tickp library, you'll need the following data: [date, open, high, low, close and optionally volume]. The get_historical_info function returns the columns [Date, Open, High, Low, Close, Volume, AdjClose]. The mismatch here is the AdjClose, so you'd need to strip that from the data you get from the StockRetriever:
from django.shortcuts import render
from stock.stockretriever import StockRetriever
def stockretriever(request, number):
data = StockRetriever().get_historical_info('YHOO')
# Assuming data is returned as a list of lists
new_data = [d[:-1] for d in data]
return render(request, 'stock.html', { 'data': simplejson.dumps(new_data) })
Following along with the readme, you need something along the following lines in your template:
<html>
<head><script src="tickp.js"></script><script src="stats.js"></script></head>
<body onload='plot = window.tickp("#chart"); plot.read({{ data }}); plot.plot();'>
<div id="chart"></div>
</body>
</html>
Note that I've cut some corners with respect to possible Ajax calls or proper formatting and usage, but it should give you something to get started with. When you're missing something, please update your question with the specific issues you're having.
Related
I've hit a bit of a stumbling block with a Django project having added a chart to a page using Chart.js only to find that it relies on the data inputted being in JSON format. I've researched ways of converting Django object values into JSON, such as serializing and re-writing my views, but for a number of reasons these aren't ideal options for me. Is there a way to convert Django object data to JSON data within the JS script?
I have an 'Accelerator' Django model with five separate decimal fields which ultimately need to be converted to JSON to feature in the chart. For each object created, the value of each field is prone to change (they are determined by a method calculating the mean value of separate values inputted into a separate model). I have an 'accelerator_detail' HTML template which renders and displays the various values of individual objects created using the Accelerator model.
This template is heavily reliant on Django placeholders and Python logic calling on object values, which is one reason why I'm hesitant about attempting to serialize the Django objects as JSON within my views (presumably this would mean I would have to re-write this template).
I've shared some of the relevant code below to provide a better understanding. Currently, the data key in my JS script is populated with dummy data but this is where my Django object values need to be stored:
// One of my model fields which needs converting to JSON
class Accelerator(models.Model):
avg_mentorship = models.DecimalField(decimal_places=2, max_digits=3)
#property
def avg_mentorship(self):
quantity = Review.objects.filter(subject=self)
mentorship_result = Review.objects.filter(subject=self).aggregate(avg_mentorship=Avg('mentorship'))['avg_mentorship']
return mentorship_result if len(quantity) > 0 else 0
// My accelerator_detail view
def accelerator_detail(request, pk):
accelerator = get_object_or_404(Accelerator, pk=pk)
reviews = Review.objects.all()
context = {
'accelerator': accelerator,
'reviews': reviews,
}
return render(request, 'reviews/accelerator_detail.html', context)
// JS script within my HTML template
<script>
var myChart = document.getElementById('accRatings').getContext('2d');
var ratingsChart = new Chart(myChart, {
type:'horizontalBar',
data:{
labels:['Mentorship', 'Hiring', 'Community', 'Fundraising', 'Corporate Development'],
datasets:[{
data:[
4.1,
4.4,
3.9,
3.6,
4.2
],
}]
},
});
</script>
Sorry I can't give this in a comment yet, have you looked into this yet? Django Generic JSON views. Seems to me like it would fit nicely into your project and help with the output you need.
Your data is already output by the view. You can just convert to JSON as you serve it from within the view.
Great example provided by SimpleIsBetterThenComplex:
from django.http import JsonResponse
def profile(request):
data = {
'name': 'Vitor',
'location': 'Finland',
'is_active': True,
'count': 28
}
return JsonResponse(data)
I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?
You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions
My application needs to read in a large dataset and pipe it to the client to manipulate with D3.js. The problem is, on large datasets, the reading/loading of the file contents could take a while. I want to solve this using streams. However, I'm unsure of how to do so in the context of the Sails framework.
What I want to do is read the contents of the file and pipe it to a rendered page. However, I can't figure out how to pipe it through if I use something like res.view('somePage', { data: thePipedData });.
I currently have something like this:
var datastream = fs.createReadStream(path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv'));
datastream.pipe(res);
...
return res.view('analytics', { title: 'Analytics', data: ??? });
What's the best way to approach this?
Based on your example it seems like the best course of action would be to set up a separate endpoint to serve just the data, and include it on the client via a regular <script> tag.
MyDataController.js
getData: function(req, res) {
/* Some code here to determine datatype and dataset based on params */
// Wrap the data in a Javascript string
res.write("var theData = '");
// Open a read stream for the file
var datastream = fs.createReadStream(
path.resolve(DATASET_EXTRACT_PATH, datatype, dataset, dataset + '.csv')
);
// Pipe the file to the response. Set {end: false} so that res isn't closed
// when the file stream ends, allowing us to continue writing to it.
datastream.pipe(res, {end: false});
// When the file is done streaming, finish the Javascript string
datastream.on('end', function() {
res.end("';");
});
}
MyView.ejs
<script language="javascript" src="/mydata/getdata?datatype=<%=datatype%>&etc.."></script>
MyViewController.js
res.view('analytics', {datatype: 'someDataType', etc...});
A slight variation on this strategy would be to use a JSONP-style approach; rather than wrapping the data in a variable in the data controller action, you would wrap it in a function. You could then call the endpoint via AJAX to get the data. Either way you'd have the benefit of a quick page load since the large data set is loaded separately, but with the JSONP variation you'd also be able to easily show a loading indicator while waiting for the data.
I query the db i my model like so
function graphRate($userid, $courseid){
$query = $this->db->get('tblGraph');
return $query->result();
}
My controller gets data back from my model and I json encode it like so
if($query = $this->rate_model->graphRate($userid, $courseid)){
$data['graph_json'] = json_encode($query);
}
$this->load->view('graph', $data);
And thats returns me a json object like so
[
{"id":"1","title":"myTitle","score":"16","date":"2013-08-02"},
{"id":"2","title":"myTitle2","score":"17","date":"2013-09-02"},
{"id":"3","title":"myTitle3","score":"18","date":"2013-10-02"}
]
In my view graph I'm loading an js file
<script type="text/javascript" src="script.js"></script>
Now I want to use $data that is being sent from my controller to my view, to my external script.js to use as labels and data to feed my chart. But How do I get that Json data to my external script.js so I can use it?
1 more thing about the json data, isn't it possible to get the output of the json data as
{
"obj1":{"id":"1","title":"myTitle","score":"16","date":"2013-08-02"},
"obj2":{"id":"2","title":"myTitle2","score":"17","date":"2013-09-02"},
"obj3":{"id":"3","title":"myTitle3","score":"18","date":"2013-10-02"}
}
The problem isn't a Codeigniter problem, it's a javascript scope/file inclusion/where-do-i-get-my-data-from problem.
I run into this all the time and have used these solutions:
naming my php files with .php extensions and loading them as if they're views.
Just putting the script that needs data from a view IN the view file where it's used
Using an ajax request in my included js file to hit a controller and get json data.
I use #2 most frequently (for things like datatables where I WANT the js code right there next to the table it's referencing.
I use #1 occasionally, but try NOT to do that because it means some .js files are in my webroot/js dir and some are in teh application/views directory, making it confusing for me or anyone else who wants to support this project.
#3 is sometimes necessary...but I like to avoid that approach to minimize the number of requests being made and to try to eliminate totally superfluous requests (which that is).
You need to print the result of the output json string to the html generated file.
But you need to parse the string with some script. I would recommend you: http://api.jquery.com/jQuery.parseJSON/
For the second question. It is possible by doing:
$returnValue = json_encode(
array (
"obj1" => array("id"=>"1","title"=>"myTitle","score"=>"16","date"=>"2013-08-02"),
"obj2" => array("id"=>"2","title"=>"myTitle2","score"=>"17","date"=>"2013-09-02"),
"obj3" => array("id"=>"3","title"=>"myTitle3","score"=>"18","date"=>"2013-10-02"),
)
);
Print the output using PHP like:
echo json_encode($query);
Then from the client-side (where JavaScript resides) load that JSON that you printed using PHP. This can be done easily using JQuery.
Like this:
$.get("test.php", function(data) {
alert("Data Loaded: " + data);
});
You can find more information about this here: http://api.jquery.com/jQuery.get/
Now you'll need to parse this data so that JavaScript can understand what you got as text from the server. For that you can use the JSON.parse method on the "data" object in the aforementioned example. Once parsed, you can use the object like any other object in JavaScript. You can find more information about JSON.parse here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse
I hope that helps.
let's say one of my urlpatterns looks like this.
url('^objects/update/(?P<pk>\d+)$', views.UpdateView.as_view(), name = 'update-object'),
I need to redirect user to the update page depending on the selected object (the list of objects is populated using Ajax). So I'd like to pass that named url pattern to the JavaScript, in order to build the actual url on the client side.
Example of what I want to achieve:
pass the name 'update-objects' to the function
get the actual url pattern, replace (?P<pk>..) with {pk}
pass the result to the javascript, resulting in : objects/update/{pk}
any tips?
thanks
to make it more clear: at the moment of rendering, I can't do url reverse because the PK is not known yet. I need to make kind of javascript-urlpattern which will later be converted to the real url (i.e. my JS code will replace {pk} part with the actual pk value)
The actual URL reversing must happen on the server side. There are several ways to do this, and the most elegant of these probably depends on how exactly your script and markup are set up for this. One thing I've done recently is to attach the URL to a logical element using HTML5 data attributes, which are easy to retrieve using jQuery. If you're not using jQuery, I'll leave it up to you to translate to pure JS. You haven't provided any code or specifics for your client-side, so I'm kind of shooting in the dark here, but maybe this will give you the idea:
Django HTML template:
<ul class="object-list">
{% for object in objectList %}
<li data-update-url="{% url update-objects object.pk %}">object.name</li>
{% endfor %}
</ul>
JS:
$('.object-list').on('click', 'li' function () {
var updateUrl = $(this).data('update-url')
...
});
It sounds like you need to make an additional ajax call once the object has actually been selected. Don't try and second guess your url.conf by trying to work out the url on the client side - you'd just be making trouble for yourself later. Wait till you can get a pk, then use django's reverse function to give you your url (doing anything else violates DRY).
How about creating a simple view that returns the url -
from django.core.urlresolvers import reverse
from django.http import HttpResponse, HttpResponseBadRequest
def get_url(request):
if request.is_ajax() and request.method == 'POST':
obj_id = request.POST['obj_id']
url = reverse('object-update', kwargs{'pk': obj_id})
return HttpResponse(obj_id)
return HttpResponseBadRequest()
Then write a javascript function that gets the url using an ajax call to your new view and then redirects. You'd call this function as soon as the object's been selected. I would suggest using JQuery to do this, pure javascript will require you to write more code, and probably write browser specific code (depending on your target). Also it supports dealing with django's csrf protection (you'll need to implement this for ajax calls if you haven't already).
var redirect = function(obj) {
$.ajax({
url: '/your-get-url-view/',
method: 'post',
data: {'obj_id': obj},
success: function(url){
window.location = url;
}
});
}
I'm afraid I don't know how you're getting from the selected object to the pk (For simplicity I've assumed it's available to the redirect function) - you may have to do some processing in the view to get there.
I haven't tested the above code, but it should give you an idea of what I'm suggesting.
Try this one:
Reverse method for generating Django urls
https://github.com/mlouro/django-js-utils
One more
https://github.com/Dimitri-Gnidash/django-js-utils
If you have a URL that only has one PK field in it, you could resolve it with any number (e.g. 0), then substitute the number as required.
In my scenario my URL had a pk then an upload_id, so I had to replace on the right most instance of a 0, with <upload_id>, which the JS would replace this string occurance as required:
detele_url_upload_id_0 = reverse(f'{APP_NAME}:api_upload_delete', args=[pk, 0])
prefix, suffix = detele_url_upload_id_0.rsplit('0', 1)
context['generic_delete_url'] = prefix + '<upload_id>' + suffix
Then in the JS:
const deleteUrl = genericDeleteUrl.replace('<upload_id>', uploadId)