I made a pollution table which it works perfectly with the series data inside the code. Yet I have no clue of how to make it read the same data from a CSV file without data module. You can see the table here: https://jsfiddle.net/Ruloco/58q60968/7/
Understanding the first column as the date, the second as the name of the station, and the third as the pollution value, I assume this is the format the CSV should have:
0,0,30
0,1,38
0,2,12
0,3,65
0,4,220
0,5,115
0,6,80
0,7,90
0,8,160
0,9,220
0,10,50
1,0,31
1,1,42
1,2,13
1,3,58
1,4,240
1,5,123
1,6,85
1,7,40
1,8,69
1,9,90
1,10,53
2,0,33
2,1,44
2,2,11
2,3,53
2,4,234
2,5,130
2,6,123
2,7,180
2,8,50
2,9,98
2,10,55
I hope you could help me. Thanks in advance!
Look at this JQuery plugin, basically it converts CSV files to javascript arrays: https://github.com/evanplaice/jquery-csv/
Once you have installed that you should be able to do
var result = $.csv.toArrays(csvInput);
with the CSV in the format you gave.
This snippet actually does exactly what you want I think.
var csv = $('textarea').val()
var result = $.csv.toArrays(csv)
console.log(result)
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-csv/0.71/jquery.csv-0.71.min.js"></script>
<textarea>
0,0,30
0,1,38
0,2,12
0,3,65
0,4,220
0,5,115
0,6,80
0,7,90
0,8,160
0,9,220
0,10,50
1,0,31
1,1,42
1,2,13
1,3,58
1,4,240
1,5,123
1,6,85
1,7,40
1,8,69
1,9,90
1,10,53
2,0,33
2,1,44
2,2,11
2,3,53
2,4,234
2,5,130
2,6,123
2,7,180
2,8,50
2,9,98
2,10,55
</textarea>
Related
I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?
You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions
I'm just having a little trouble understanding the documentation for CSV Parse with D3. I currently have:
d3.parse("data.csv",function(data){
salesData = data;
});
But I keep on getting the error:
Uncaught TypeError: d3.parse is not a function
What is this supposed to look like? I'm just a little confused, and the only examples that I could find was something like this.
I also tried something like:
d3.dsv.parse("data.csv",function(data){
salesData = data;
});
and got:
Uncaught TypeError: Cannot read property 'parse' of undefined
Why is this happening? Any help would be greatly appreaciated, thanks!!
There is some misunderstanding here: you're confusing d3.csv, which is a request, with d3.csvParse, which parses a string (and also mixing D3 v3 syntax with D3 v4 syntax). This is the difference:
d3.csv (D3 v4)
The d3.csv function, which takes as arguments (url[[, row], callback]):
Returns a new request for the CSV file at the specified url with the default mime type text/csv. (emphasis mine)
So, as you can see, you use d3.csv when you want to request a given CSV file at a given url.
For example, the snippet below gets the CSV at the url between quotes, which looks like this...
name, parent
Level 2: A, Top Level
Top Level, null
Son of A, Level 2: A
Daughter of A, Level 2: A
Level 2: B, Top Level
... and logs the parsed CSV file, check it:
d3.csv("https://gist.githubusercontent.com/d3noob/fa0f16e271cb191ae85f/raw/bf896176236341f56a55b36c8fc40e32c73051ad/treedata.csv", function(data){
console.log(data);
});
<script src="https://d3js.org/d3.v4.min.js"></script>
d3.csvParse
On the other hand, d3.csvParse (or d3.csv.parse in D3 v3), which takes as arguments (string[, row]):
Parses the specified string, which must be in the delimiter-separated values format with the appropriate delimiter, returning an array of objects representing the parsed rows.
So, you use d3.csvParse when you want to parse a string.
Here is a demo, suppose you have this string:
var data = "foo,bar,baz\n42,33,42\n12,76,54\n13,42,17";
If you want to parse it, you'll use d3.csvParse, not d3.csv:
var data = "foo,bar,baz\n42,33,42\n12,76,54\n13,42,17";
var parsed = d3.csvParse(data);
console.log(parsed);
<script src="https://d3js.org/d3.v4.min.js"></script>
You can get csv data into d3 like the following -
// get the data
d3.csv("data.csv", function(error, data) {
if (error) throw error;
console.log(data);
//format data if required...
//draw chart
});
I also could not get the d3.csv("csv_file.csv", function(data) { //modifying code }
to work.
A classmate recommended using the following, which has worked so far:
d3.csv("data.csv").then(function(data){
//modifying code
}
As noted in the comments below, this is a fix if you're running v5 instead of v4.
Use d3.csv("data.csv", function(data){...}) to get CSV from url and parse, or use d3.csv.parse() to parse a CSV-formatted string.
Here is the code you can use to read csv file using d3.js
<script src="https://d3js.org/d3.v4.min.js"></script>
<script>
d3.csv("csv/cars.csv", function(data) {
console.log(data[0]);
});
</script>
Note that, the csv file name is "cars.csv", and the file is saved in a folder called csv.
console.log(data[0])
will help you to see the data output in the browser debug window. Where, you can also find if there is any error as well.
consider this fiddle link FIDDLE. In this example I would like to use a csv file to load data at line no.33-[data.csv] and data at line no.158-[data1.csv]. I want to use two separate csv files. I tried using a csv file for data at line no.33 with this code
d3.csv("data.csv", function(csvData) {
csvData.forEach(function (d,i) {
data[i] = {
first: +d.first,
second: +d.second
}
});
console.log(data);
I was able to get an output but the charts had moved far away from each other with the following errors : Unexpected value NaN parsing cy attribute. How to load the two datasets in an efficient way using two separate csv files ?
Here is hopefully the final plunker in this ever-growing project :) (A lot of the csv work here has been guided by the great Lars...as usual, many kudos to him.)
Updated plunker with data on top chart coming from datam.csv.
I need to extract the data from an excel sheet into csv or json format and then display the content in the form of charts(bar, pie and line) using javascript. I have been working on it since 2 days and not able to find any good source. Any help regarding this would be appreciated. Thank you.
You can also pull data from a google spreadsheet as CSV data like this:
var spreadsheet = "https://docs.google.com/spreadsheet/pub?key=youruniquekeyhere&single=true&gid=2&output=csv"
d3.csv(spreadsheet, function (error, data) {
// use your data here
});
Notice the output=csv at the end of that URL.
gid=0 is worksheet 1, gid=1 is worksheet 2 and so on. You must publish your worksheet to the web first to make it available.
In addition to Christophe Viau's tutorials, here are more resources:
https://github.com/mbostock/d3/wiki/Gallery#basic-charts
http://www.d3noob.org
I'm going to write currency converter with JavaScript. Currency will be updated daily in CSV files. I want to figure out how to get values from CSV and the best (efficient) way to store them in application. Thank you!
A simple way is to do:
<script type="text/javascript" src="conversion_table.js"></script>
<script type="text/javascript" src="converter.js"></script>
The conversion table is just a JavaScript file defining a single object:
var conversion_table =
{
"USD-GBP": 1.2,
:
:
};
It is more efficient to use JavaScript objects instead of having to parse a CSV file whenever your user loads your page.
Use the jQuery-CSV plugin
With jquery-csv, transforming CSV to JavaScript data structure becomes very simple.
Using the data:
val1,"value 2",val3,val4
"val,1",val2,val3,
Do:
var array = $.csv.toArray(csv);
Outputs:
[
["val1","value 2","val3","val4"],
["val,1","val2","val3",""]
]
If you want the entries mapped to key:value pairs, just call the toObjects() function instead.
Using the data:
foo,bar,baz,fuzz
val1,"value 2",val3,val4
"val,1",val2,val3,
Do:
var dict = $.csv.toObjects(csv);
Outputs:
[
{foo:"val1",bar:"value 2",baz:"val3",fuzz:"val4"},
{foo:"val,1",bar:"val2",baz:"val3",fuzz:""}
]
If you need to allow cross-domain requests, this library will also work in Node.js. After the data is converted, assign it to a variable and send a response with the data encoded as 'text/javascript' (Ie JSONP).
Disclaimer: I'm the author of jquery-csv.
I'd suggest using a JQuery plugin to convert your CSV data into an array.
This one seems like it ought to do the trick: http://plugins.jquery.com/project/csv