Scraping Javascript variables into Python - javascript
I want to scrape the following data from http://maps.latimes.com/neighborhoods/population/density/neighborhood/list/:
var hoodFeatures = {
type: "FeatureCollection",
features: [{
type: "Feature",
properties: {
name: "Koreatown",
slug: "koreatown",
url: "/neighborhoods/neighborhood/koreatown/",
has_statistics: true,
label: 'Rank: 1<br>Population per Sqmi: 42,611',
population: "115,070",
stratum: "high"
},
geometry: { "type": "MultiPolygon", "coordinates": [ [ [ [ -118.286908, 34.076510 ], [ -118.289208, 34.052511 ], [ -118.315909, 34.052611 ], [ -118.323009, 34.054810 ], [ -118.319309, 34.061910 ], [ -118.314093, 34.062362 ], [ -118.313709, 34.076310 ], [ -118.286908, 34.076510 ] ] ] ] }
},
From the above html, I want to take each of:
name
population per sqmi
population
geometry
and turn it into a data frame by name
So far I've tried
import requests
import json
from bs4 import BeautifulSoup
response_obj = requests.get('http://maps.latimes.com/neighborhoods/population/density/neighborhood/list/').text
soup = BeautifulSoup(response_obj,'lxml')
The object has the script info, but I don't understand how to use the json module as advised in this thread:
Parsing variable data out of a javascript tag using python
json_text = '{%s}' % (soup.partition('{')[2].rpartition('}')[0],)
value = json.loads(json_text)
value
I get this error
TypeError Traceback (most recent call last)
<ipython-input-12-37c4c0188ed0> in <module>
1 #Splits the text on the first bracket and last bracket of the javascript into JSON format
----> 2 json_text = '{%s}' % (soup.partition('{')[2].rpartition('}')[0],)
3 value = json.loads(json_text)
4 value
5 #import pprint
TypeError: 'NoneType' object is not callable
Any suggestions? Thanks
I'm not really sure how to do that with the beautiful soup, yet another option might be to likely design an expression and extract our desired values:
(?:name|population per sqmi|population)\s*:\s*"?(.*?)\s*["']|(?:geometry)\s*:\s*({.*})
Demo
Test
import re
regex = r"(?:name|population per sqmi|population)\s*:\s*\"?(.*?)\s*[\"']|(?:geometry)\s*:\s*({.*})"
test_str = ("var hoodFeatures = {\n"
" type: \"FeatureCollection\",\n"
" features: [{\n"
" type: \"Feature\",\n"
" properties: {\n"
" name: \"Koreatown\",\n"
" slug: \"koreatown\",\n"
" url: \"/neighborhoods/neighborhood/koreatown/\",\n"
" has_statistics: true,\n"
" label: 'Rank: 1<br>Population per Sqmi: 42,611',\n"
" population: \"115,070\",\n"
" stratum: \"high\"\n"
" },\n"
" geometry: { \"type\": \"MultiPolygon\", \"coordinates\": [ [ [ [ -118.286908, 34.076510 ], [ -118.289208, 34.052511 ], [ -118.315909, 34.052611 ], [ -118.323009, 34.054810 ], [ -118.319309, 34.061910 ], [ -118.314093, 34.062362 ], [ -118.313709, 34.076310 ], [ -118.286908, 34.076510 ] ] ] ] }\n"
" },\n")
matches = re.finditer(regex, test_str, re.MULTILINE | re.IGNORECASE)
for matchNum, match in enumerate(matches, start=1):
print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))
for groupNum in range(0, len(match.groups())):
groupNum = groupNum + 1
print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))
You can't really use json.loads because hoodFeatures object is not really a json. In a proper json, every key is surrounded with double quotes "
You could try adding quotes around keys manually (using regular expressions).
Another option is using Selenium to execute that JS and get the JSON.stringify output of it.
Answer using manual cleanup:
This one cleans up JS code and turns it into a JSON that can be parsed properly. However keep in mind that it is by no means robust, and may break at any sight of unexpected input.
import json
import re
js = '''
var hoodFeatures = {
type: "FeatureCollection",
features: [
{
type: "Feature",
properties: {
name: "Beverlywood",
slug: "beverlywood",
url: "/neighborhoods/neighborhood/beverlywood/",
has_statistics: true,
label: 'Rank: 131<br>Population per Sqmi: 7,654',
population: "6,080",
stratum: "middle"
},
geometry: { }
}]
}
'''
if __name__ == '__main__':
unprefixed = js.split('{', maxsplit=1)[1]
unsuffixed = unprefixed.rsplit('}', maxsplit=1)[0]
quotes_replaced = unsuffixed.replace('\'', '"')
rebraced = f'{{{quotes_replaced}}}'
keys_quoted = []
for line in rebraced.splitlines():
line = re.sub('^\s+([^:]+):', '"\\1":', line)
keys_quoted.append(line)
json_raw = '\n'.join(keys_quoted)
# print(json_raw)
parsed = json.loads(json_raw)
for feat in parsed['features']:
props = feat['properties']
name, pop = props['name'], int(props['population'].replace(',', ''))
geo = feat['geometry']
pop_per_sqm = re.findall('per Sqmi: ([\d,]+)', props['label'])[0].replace(',', '')
pop_per_sqm = int(pop_per_sqm)
print(name, pop, pop_per_sqm, geo)
Related
How to use "Joins" or "sequelize literal" in nested includes in sequelize?
const Contentdata = await question.findAll({
attributes: [`id`, `question`],
include: [{
model: option,
attributes: ['id', 'option','option_es',
[sequelize.literal('(SELECT case when count(*)=0 then "0" else "1" end FROM `answer` WHERE `userid`=' + userid + ' and `optionid`=options.id)'), 'answer'],
],
on: {
col1: sequelize.where(sequelize.col('options.questionid'), '=',
sequelize.col('question.id')),
},
include: [{
model: focuson,
attributes: ['id', 'focuson', 'optionid',
[sequelize.literal('(SELECT case when count(*)=0 then "0" else "1" end FROM
`focuson_answer` WHERE `userid`=' + userid + ' and `focusonid`=focusons.id)'), 'answer'],
],
}]
}]
});
In the above example I not able to use focus on object as it gives the error "column focusons.id" does not exist, whereas if I pull it out of the nested include it does not show this error. Also if I try to apply to join here it also needs to access the column but it's not.
Object with Key:Value where every Value is an Array of Objects - cannot iterate [duplicate]
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 3 years ago.
I am downloading data via d3.json when create an object with each record that holds downloaded data. Each object will be stored in an array. I lost the ability to iterate over the array.
I have tried using multi-dimensional array, and still cannot iterate over the created object.
var datamodel = {
"mkt":[],
"confidence":[],
"homes":[],
"combined":[]
}
var tObject={};
// console.log(datamodel)
const url1 = "https://www.quandl.com/api/v3/datasets/YALE/SPCOMP.json?start_date='2015-01-01'&end_date='2018-09-30'&api_key=4QuxetVDNP8R3sYg9CED";
// Fetch the JSON data and console log it
d3.json(url1).then(function(data) {
data.dataset.data.forEach(element => {
tObject['date']=element[0];
tObject['spx'] = element[1];
tObject['rate']=element[5];
datamodel['mkt'].push(tObject); //creating an array of objects
tObject={};
});
});
I repeat the code above for other datasets to fill "confidence, homes"
I need to iterate after downloading over the datamodel, and update the "combined" so that I can have more stats, like change per date, ratios.
For instance: going to debug console in any browser, I can access the datamodel variable via datamodel.mkt
The debug console allows me to iterate over the objects, not sure why code ran in JS file does not allow me to iterate not get length of the array. I am so lost.
I hit the API in browser and copied the data in my js file. I call this function in HTML onload. Half of this function does what your code did to fill data in datamodel.mkt
Then it loops over the data present in datamodel.mkt and copies the elements in datamodel.combined
Please note that the commented forEach loop also works. You can uncomment the console.log to view the datamodel object in console
Hope this helps.
function setData(){
var data = {"dataset":{"id":539984,"dataset_code":"SPCOMP","database_code":"YALE","name":"S\u0026P Composite","description":"This data set consists of monthly stock price, dividends, and earnings data and the consumer price index (to allow conversion to real values), all starting January 1871. Further info at http://www.econ.yale.edu/~shiller/data.htm.","refreshed_at":"2019-08-24T02:51:26.805Z","newest_available_date":"2019-08-31","oldest_available_date":"1871-01-31","column_names":["Year","S\u0026P Composite","Dividend","Earnings","CPI","Long Interest Rate","Real Price","Real Dividend","Real Earnings","Cyclically Adjusted PE Ratio"],"frequency":"monthly","type":"Time Series","premium":false,"limit":null,"transform":null,"column_index":null,"start_date":"2015-01-01","end_date":"2018-09-30","data":[["2018-09-30",2901.5,52.34,130.39,252.439,3.0,2944.5129194578,53.115907704436,132.32295004932,32.6228911205],["2018-08-31",2857.82,51.89,127.75333333333,252.146,2.89,2903.555479266,52.720428095231,129.79784975768,32.390276880301],["2018-07-31",2793.64,51.44,125.11666666667,252.006,2.89,2839.9251892812,52.29226089855,127.18960684468,31.886366962159],["2018-06-30",2754.35,50.99,122.48,251.989,2.91,2800.1731263567,51.838302217557,124.51765553258,31.630556496455],["2018-05-31",2701.49,50.66,120.13333333333,251.588,2.976,2750.8111875864,51.58490120753,122.32661136461,31.243615074865],["2018-04-30",2653.63,50.33,117.78666666667,250.546,2.87,2713.3151215246,51.462016206605,120.43590997262,30.970179293325],["2018-03-31",2702.77,50.0,115.44,249.554,2.84,2774.5457779178,51.327818828791,118.50566811191,31.808409057643],["2018-02-28",2705.16,49.643333333333,113.58666666667,248.991,2.86,2783.2783925925,51.076911163456,116.86677128892,32.03538233925],["2018-01-31",2789.8,49.286666666667,111.73333333333,247.867,2.58,2883.3787928607,50.939898715844,115.48122581868,33.307343828031],["2017-12-31",2664.34,48.93,109.88,246.524,2.4,2768.7119778399,50.846767708215,114.18440293846,32.086132007706],["2017-11-30",2593.61,48.676666666667,108.94666666667,246.669,2.35,2693.6268919585,50.553775758202,113.14795637068,31.29891333388],["2017-10-31",2557.0,48.423333333333,108.01333333333,246.663,2.36,2655.6697042118,50.291896484272,112.18135979048,30.920393290334],["2017-09-30",2492.84,48.17,107.08,246.819,2.2,2587.3975149806,49.997167205523,111.14172024844,30.168114410679],["2017-08-31",2456.22,47.853333333333,106.06,245.519,2.21,2562.8872302144,49.931478826486,110.66590925753,29.914959397497],["2017-07-31",2454.1,47.536666666667,105.04,244.786,2.32,2568.3429837695,49.749588160679,109.92981011986,30.002220744019],["2017-06-30",2433.99,47.22,104.02,244.955,2.19,2545.5393875916,49.384085342206,108.78722061195,29.748503240633],["2017-05-31",2395.35,46.94,102.77666666667,244.733,2.3,2507.4009520069,49.135784201558,107.58440805899,29.313344980271],["2017-04-30",2359.31,46.66,101.53333333333,244.524,2.3,2471.7859389569,48.884433123129,106.37375574586,28.904245956275],["2017-03-31",2366.82,46.38,100.29,243.801,2.48,2487.0074615157,48.735183100151,105.3827406881,29.086921742465],["2017-02-28",2329.91,46.153333333333,98.376666666667,243.603,2.42,2450.2130769633,48.536424530897,103.4562687549,28.655106525184],["2017-01-31",2275.12,45.926666666667,96.463333333333,242.839,2.43,2400.1214199531,48.45000545629,101.76329713308,28.063573742124],["2016-12-31",2246.63,45.7,94.55,241.432,2.49,2383.8782004353,48.49184501226,100.32612573105,27.865098223924],["2016-11-30",2164.99,45.476666666667,92.73,241.353,2.14,2298.0026949634,48.27066293976,98.427147425141,26.850953531056],["2016-10-31",2143.02,45.253333333333,90.91,241.729,1.76,2271.1447214649,47.958894050776,96.34523552201,26.525143085071],["2016-09-30",2157.69,45.03,89.09,241.428,1.63,2289.5427262476,47.781705881256,94.534136730205,26.727873346479],["2016-08-31",2170.95,44.84,88.366666666667,240.849,1.56,2309.1508982288,47.694477660277,93.992016263302,26.948872433724],["2016-07-31",2148.9,44.65,87.643333333333,240.628,1.5,2287.7964664337,47.53600084986,93.308254598384,26.694003256096],["2016-06-30",2083.89,44.46,86.92,241.018,1.64,2214.9945027446,47.257127579683,92.388428457626,25.840372927671],["2016-05-31",2065.55,44.266666666667,86.76,240.229,1.81,2202.7115000166,47.206165783482,92.521241190697,25.69470992345],["2016-04-30",2075.54,44.073333333333,86.6,239.261,1.81,2222.3196911532,47.190146430049,92.72424778798,25.922337543674],["2016-03-31",2021.95,43.88,86.44,238.132,1.89,2175.2039979402,47.20589106042,92.991732526498,25.372298620188],["2016-02-29",1904.42,43.716666666667,86.47,237.111,1.78,2057.5877800903,47.232689805618,93.42456776573,24.00260677729],["2016-01-31",1918.6,43.553333333333,86.5,236.916,2.09,2074.6144044725,47.094950847558,93.533902838981,24.206167203878],["2015-12-31",2054.08,43.39,86.53,236.525,2.24,2224.7829278089,46.995896575415,93.721017070077,25.965424037124],["2015-11-30",2080.62,43.096666666667,87.906666666667,237.336,2.26,2245.8279922768,46.518682111016,94.886741792227,26.225851890972],["2015-10-31",2024.81,42.803333333333,89.283333333333,237.838,2.07,2180.9734222979,46.104539381848,96.169308258142,25.491441046067],["2015-09-30",1944.41,42.51,90.66,237.945,2.17,2093.4307689277,45.767992340667,97.608237723003,24.496752170486],["2015-08-31",2039.87,42.253333333333,92.076666666667,238.316,2.17,2192.7879220761,45.420835151647,98.979151892865,25.693658417058],["2015-07-31",2094.14,41.996666666667,93.493333333333,238.654,2.32,2247.9380311036,45.080989895413,100.35967970367,26.3811363364],["2015-06-30",2099.29,41.74,94.91,238.638,2.36,2253.6173464096,44.808477170442,101.88722013049,26.495895292785],["2015-05-31",2111.94,41.43,96.356666666667,237.805,2.2,2275.1389967621,44.631480362061,103.80257485124,26.806111379651],["2015-04-30",2094.86,41.12,97.803333333333,236.599,1.94,2268.2422722624,44.523320047845,105.89808151133,26.791371680192],["2015-03-31",2079.99,40.81,99.25,236.119,2.04,2256.7198666245,44.277490640313,107.68294403458,26.728605452928],["2015-02-28",2082.2,40.353333333333,100.27,234.722,1.98,2272.5632823084,44.042600927906,109.43709553216,26.995513699383],["2015-01-31",2028.18,39.896666666667,101.29,233.707,1.88,2223.2183358864,43.733298264494,111.03047325283,26.492295420383]],"collapse":null,"order":null,"database_id":175}}
var datamodel = {
"mkt":[],
"confidence":[],
"homes":[],
"combined":[]
}
var tObject={};
data.dataset.data.forEach(element => {
tObject['date']=element[0];
tObject['spx'] = element[1];
tObject['rate']=element[5];
datamodel['mkt'].push(tObject); //creating an array of objects
tObject={};
});
/*
datamodel.mkt.forEach(element => {
datamodel["combined"].push(element);
});*/
for(var i=0;i<datamodel.mkt.length; i++)
{
datamodel.combined[i] = datamodel.mkt[i];
}
document.getElementById("foo").innerHTML = "Length of datamodel.mkt = " + datamodel.mkt.length + "<br> Length of datamodel.combined = " + datamodel.combined.length;
//console.log(datamodel);
}
<html>
<head></head>
<body onload="setData()">
<div id="foo"></div>
</body>
</html>
You can use map() or forEach() to get your work done. In "forEach(), you are pushing the objects, but the array is getting updated only by the last object. You can do something like this:
let data ={
"dataset": {
"id": 539984,
"dataset_code": "SPCOMP",
"database_code": "YALE",
"name": "S&P Composite",
"description": "This data set consists of monthly stock price, dividends, and earnings data and the consumer price index (to allow conversion to real values), all starting January 1871. Further info at http://www.econ.yale.edu/~shiller/data.htm.",
"refreshed_at": "2019-08-24T02:51:26.805Z",
"newest_available_date": "2019-08-31",
"oldest_available_date": "1871-01-31",
"column_names": [
"Year",
"S&P Composite",
"Dividend",
"Earnings",
"CPI",
"Long Interest Rate",
"Real Price",
"Real Dividend",
"Real Earnings",
"Cyclically Adjusted PE Ratio"
],
"frequency": "monthly",
"type": "Time Series",
"premium": false,
"limit": null,
"transform": null,
"column_index": null,
"start_date": "2015-01-01",
"end_date": "2018-09-30",
"data": [
[
"2018-09-30",
2901.5,
52.34,
130.39,
252.439,
3.0,
2944.5129194578,
53.115907704436,
132.32295004932,
32.6228911205
],
[
"2018-08-31",
2857.82,
51.89,
127.75333333333,
252.146,
2.89,
2903.555479266,
52.720428095231,
129.79784975768,
32.390276880301
],
[
"2018-07-31",
2793.64,
51.44,
125.11666666667,
252.006,
2.89,
2839.9251892812,
52.29226089855,
127.18960684468,
31.886366962159
],
[
"2018-06-30",
2754.35,
50.99,
122.48,
251.989,
2.91,
2800.1731263567,
51.838302217557,
124.51765553258,
31.630556496455
],
[
"2018-05-31",
2701.49,
50.66,
120.13333333333,
251.588,
2.976,
2750.8111875864,
51.58490120753,
122.32661136461,
31.243615074865
],
[
"2018-04-30",
2653.63,
50.33,
117.78666666667,
250.546,
2.87,
2713.3151215246,
51.462016206605,
120.43590997262,
30.970179293325
],
[
"2018-03-31",
2702.77,
50.0,
115.44,
249.554,
2.84,
2774.5457779178,
51.327818828791,
118.50566811191,
31.808409057643
],
[
"2018-02-28",
2705.16,
49.643333333333,
113.58666666667,
248.991,
2.86,
2783.2783925925,
51.076911163456,
116.86677128892,
32.03538233925
],
[
"2018-01-31",
2789.8,
49.286666666667,
111.73333333333,
247.867,
2.58,
2883.3787928607,
50.939898715844,
115.48122581868,
33.307343828031
],
[
"2017-12-31",
2664.34,
48.93,
109.88,
246.524,
2.4,
2768.7119778399,
50.846767708215,
114.18440293846,
32.086132007706
],
[
"2017-11-30",
2593.61,
48.676666666667,
108.94666666667,
246.669,
2.35,
2693.6268919585,
50.553775758202,
113.14795637068,
31.29891333388
],
[
"2017-10-31",
2557.0,
48.423333333333,
108.01333333333,
246.663,
2.36,
2655.6697042118,
50.291896484272,
112.18135979048,
30.920393290334
],
[
"2017-09-30",
2492.84,
48.17,
107.08,
246.819,
2.2,
2587.3975149806,
49.997167205523,
111.14172024844,
30.168114410679
],
[
"2017-08-31",
2456.22,
47.853333333333,
106.06,
245.519,
2.21,
2562.8872302144,
49.931478826486,
110.66590925753,
29.914959397497
],
[
"2017-07-31",
2454.1,
47.536666666667,
105.04,
244.786,
2.32,
2568.3429837695,
49.749588160679,
109.92981011986,
30.002220744019
],
[
"2017-06-30",
2433.99,
47.22,
104.02,
244.955,
2.19,
2545.5393875916,
49.384085342206,
108.78722061195,
29.748503240633
],
[
"2017-05-31",
2395.35,
46.94,
102.77666666667,
244.733,
2.3,
2507.4009520069,
49.135784201558,
107.58440805899,
29.313344980271
],
[
"2017-04-30",
2359.31,
46.66,
101.53333333333,
244.524,
2.3,
2471.7859389569,
48.884433123129,
106.37375574586,
28.904245956275
],
[
"2017-03-31",
2366.82,
46.38,
100.29,
243.801,
2.48,
2487.0074615157,
48.735183100151,
105.3827406881,
29.086921742465
],
[
"2017-02-28",
2329.91,
46.153333333333,
98.376666666667,
243.603,
2.42,
2450.2130769633,
48.536424530897,
103.4562687549,
28.655106525184
],
[
"2017-01-31",
2275.12,
45.926666666667,
96.463333333333,
242.839,
2.43,
2400.1214199531,
48.45000545629,
101.76329713308,
28.063573742124
],
[
"2016-12-31",
2246.63,
45.7,
94.55,
241.432,
2.49,
2383.8782004353,
48.49184501226,
100.32612573105,
27.865098223924
],
[
"2016-11-30",
2164.99,
45.476666666667,
92.73,
241.353,
2.14,
2298.0026949634,
48.27066293976,
98.427147425141,
26.850953531056
],
[
"2016-10-31",
2143.02,
45.253333333333,
90.91,
241.729,
1.76,
2271.1447214649,
47.958894050776,
96.34523552201,
26.525143085071
],
[
"2016-09-30",
2157.69,
45.03,
89.09,
241.428,
1.63,
2289.5427262476,
47.781705881256,
94.534136730205,
26.727873346479
],
[
"2016-08-31",
2170.95,
44.84,
88.366666666667,
240.849,
1.56,
2309.1508982288,
47.694477660277,
93.992016263302,
26.948872433724
],
[
"2016-07-31",
2148.9,
44.65,
87.643333333333,
240.628,
1.5,
2287.7964664337,
47.53600084986,
93.308254598384,
26.694003256096
],
[
"2016-06-30",
2083.89,
44.46,
86.92,
241.018,
1.64,
2214.9945027446,
47.257127579683,
92.388428457626,
25.840372927671
],
[
"2016-05-31",
2065.55,
44.266666666667,
86.76,
240.229,
1.81,
2202.7115000166,
47.206165783482,
92.521241190697,
25.69470992345
],
[
"2016-04-30",
2075.54,
44.073333333333,
86.6,
239.261,
1.81,
2222.3196911532,
47.190146430049,
92.72424778798,
25.922337543674
],
[
"2016-03-31",
2021.95,
43.88,
86.44,
238.132,
1.89,
2175.2039979402,
47.20589106042,
92.991732526498,
25.372298620188
],
[
"2016-02-29",
1904.42,
43.716666666667,
86.47,
237.111,
1.78,
2057.5877800903,
47.232689805618,
93.42456776573,
24.00260677729
],
[
"2016-01-31",
1918.6,
43.553333333333,
86.5,
236.916,
2.09,
2074.6144044725,
47.094950847558,
93.533902838981,
24.206167203878
],
[
"2015-12-31",
2054.08,
43.39,
86.53,
236.525,
2.24,
2224.7829278089,
46.995896575415,
93.721017070077,
25.965424037124
],
[
"2015-11-30",
2080.62,
43.096666666667,
87.906666666667,
237.336,
2.26,
2245.8279922768,
46.518682111016,
94.886741792227,
26.225851890972
],
[
"2015-10-31",
2024.81,
42.803333333333,
89.283333333333,
237.838,
2.07,
2180.9734222979,
46.104539381848,
96.169308258142,
25.491441046067
],
[
"2015-09-30",
1944.41,
42.51,
90.66,
237.945,
2.17,
2093.4307689277,
45.767992340667,
97.608237723003,
24.496752170486
],
[
"2015-08-31",
2039.87,
42.253333333333,
92.076666666667,
238.316,
2.17,
2192.7879220761,
45.420835151647,
98.979151892865,
25.693658417058
],
[
"2015-07-31",
2094.14,
41.996666666667,
93.493333333333,
238.654,
2.32,
2247.9380311036,
45.080989895413,
100.35967970367,
26.3811363364
],
[
"2015-06-30",
2099.29,
41.74,
94.91,
238.638,
2.36,
2253.6173464096,
44.808477170442,
101.88722013049,
26.495895292785
],
[
"2015-05-31",
2111.94,
41.43,
96.356666666667,
237.805,
2.2,
2275.1389967621,
44.631480362061,
103.80257485124,
26.806111379651
],
[
"2015-04-30",
2094.86,
41.12,
97.803333333333,
236.599,
1.94,
2268.2422722624,
44.523320047845,
105.89808151133,
26.791371680192
],
[
"2015-03-31",
2079.99,
40.81,
99.25,
236.119,
2.04,
2256.7198666245,
44.277490640313,
107.68294403458,
26.728605452928
],
[
"2015-02-28",
2082.2,
40.353333333333,
100.27,
234.722,
1.98,
2272.5632823084,
44.042600927906,
109.43709553216,
26.995513699383
],
[
"2015-01-31",
2028.18,
39.896666666667,
101.29,
233.707,
1.88,
2223.2183358864,
43.733298264494,
111.03047325283,
26.492295420383
]
],
"collapse": null,
"order": null,
"database_id": 175
}
}
var datamodel = {
"mkt":[],
"confidence":[],
"homes":[],
"combined":[]
}
var tObject={};
(data['dataset']['data']).map((mkt, index) => {
tObject['date'] = mkt[0];
tObject['spx'] = mkt[1];
tObject['rate'] = mkt[5];
datamodel['mkt'].push(JSON.parse(JSON.stringify(tObject)))
})
console.log(datamodel['mkt'].length)
data is the JSON data which I got after hitting your API. I hope it helps.
ExtJs - How to create instance of Ext.data.Model dynamically using json file containing Model configs
I am using ExtJs Version 5.0.0.970. I have a Json file which has configs related to a model. Following is the structure of the Json file -
{
"model": {
"requires": ["Ext.data.Field"],
"fields": [
{
"name" : "carName",
"mapping" : "carName"
},
{
"name" : "carPrice.currency",
"mapping" : "carPrice.currency"
}
]
}
}
Using above Json i am trying to create a Model dynamically in the following way -
//gridModelJsonData variable contains the content of json.model object
Ext.create("Ext.data.Model",gridModelJsonData);
As soon as the above line of code executes i am getting following error in the console -
Model.js?_dc=1471922882661:2367 Uncaught TypeError: Cannot read property 'length' of undefined
Ext.data.Model#rankFields # Model.js?_dc=1471922882661:2367
Ext.data.Model#makeInitializeFn # Model.js?_dc=1471922882661:2824
Ext.data.Model#constructor # Model.js?_dc=1471922882661:378
Ext.data.Model # Class.js?_dc=1471922882661:29
Ext.create1 # VM11599:3create # ClassManager.js?_dc=1471922882661:1413
Please let me know how to solve this problem!
Thanks!
You should define your model's fields(i.e. the model's metadata) and then create the model:
var jsonData = {
"model": {
"fields": [
{
"name" : "carName",
"mapping" : "carName"
},
{
"name" : "carPrice.currency",
"mapping" : "carPrice.currency"
}
]
}
};
Ext.define('CarModel', { //Defining a model, which is like an object
extend : 'Ext.data.Model',
fields : [
{name: 'name', type: 'string'},
{name: 'mapping', type: 'string'}
]
});
var gridModelJsonData = jsonData.model;
var carModels = [];
for(var i =0; i < gridModelJsonData.fields.length; i++) {
carModels.push(Ext.create('CarModel', gridModelJsonData.fields[i]));
}
for(var i =0; i < carModels.length; i++) {
console.log('CarModel[' + i + '].name=' + carModels[i].get('name'));
console.log('CarModel[' + i + '].mapping=' + carModels[i].get('mapping'));
}
. See this fiddle
Hicharts data API for line graph using datatime and matching x-axis to y-axis
I am having a problem populating and matching the x-axis and the y-axis using highcharts.
Am not sure how highcharts handles data, but my thinking is that on the x-axis i populate the dates for the last week and then on the Y-axis have the total amount collect for that day.
My data will have the coordinate for x-axis and y-axis and the name.
Problem :
My problem is my data is not consistent. Reason being, for a particular date I might not have any transaction. Hence no transactions for that date.
Question 1
Is there a way round this ?
Question 2
can i auto generate datetime using highchart xAxis option and then match these points from my data set ?
I am not limited to change my data structure, any advise on how to go
about is highly appreciated.
The data set format was in reference to
https://jsfiddle.net/f2bnc2ox/
Have a look at my jsfiddle
https://jsfiddle.net/chapskev/ao1m9s9r/3/
[
{
"name":"Open Air Market",
"data":[
[
"2016-06-16",
450
],
[
"2016-06-17",
1980
],
[
"2016-06-18",
1650
],
[
"2016-06-19",
420
],
[
"2016-06-20",
630
],
[
"2016-06-21",
660
]
],
"keys":[
"name",
"y"
]
},
{
"name":"Parking Fee",
"data":[
[
"2016-06-17",
300
],
[
"2016-06-21",
1000
]
],
"keys":[
"name",
"y"
]
},
{
"name":"Fisheries Daily Revenue",
"data":[
[
"2016-06-21",
200
]
],
"keys":[
"name",
"y"
]
}
]
Your question isn't super clear, and I'm not sure this will help anyone else, but I do quite a bit with multidimensional arrays and HighCharts, so I figured I'd take a stab at helping. Making the assumption that your data array is named $aData (for example code).
//INIT VARS
$cSeries = '';
$cXAxis = '';
$bAxis = true;
//LOOP TO BUILD SERIES
foreach ($aData AS $nPos => $aSet) {
//FIRST PASS? BUILD AXIS
if ($bAxis) {
$cXAxis .= "xAxis: {\n" .
" categories: ['" . implode("','", array_keys($aSet['data'])) . "']\n" .
"}, ";
$bAxis = false; //NEED ONLY ONCE
}
$cSeries .= ", {\n" .
" name: '" . $aSet['name'] . "',\n" .
" data: [" . implode(",", $aSet['data']) . "]\n" .
"}";
}
$cSeries = substr($cSeries, 2); //REMOVE LEADING COMMA
$cOutput = "series: [" . $cSeries . "]\n" . $cXAxis;
At the end, $cOutput equals
series: [{
name: 'A revenue',
data: [7700,5000]
}, {
name: 'B revenue',
data: [390,210]
}]
xAxis: {
categories: ['4','5']
},
This matches the two examples of data you are trying to create I believe. You may want to loop through the categories (instead of using implode) so that you can name them something like Apr, May instead.
Sending JSON object using rest
I am trying to create this:
[
{
"id":"1",
},
{
"id":"3",
},
{
"id":"5",
},
{
"id":"6",
},
{
"id":"9",
},
]
Person = {
"id" : Id
};
PersonArray.push(Person);
tempPersonJson['PersonList'] = JSON.stringify(PersonArray);
This is my output:
List = "[{\"id\":\"12\"},{\"id\":\"10\"},{\"id\":\"9\"},{\"id\":\"8\"},{\"id\":\"7\"},{\"id\":\"6\"},{\"id\":\"5\"},{\"id\":\"4\"},{\"id\":\"3\"},{\"id\":\"2\"},{\"id\":\"1\"},{\"id\":\"12\"},{\"id\":\"10\"},{\"id\":\"9\"},{\"id\":\"8\"},{\"id\":\"7\"},{\"id\":\"6\"},{\"id\":\"5\"},{\"id\":\"4\"},{\"id\":\"3\"},{\"id\":\"2\"},{\"id\":\"1\"}]";
API complains with this:
Call Body
Expected body: Content-Type is application/json but body is not a parseable JSONParse error on line 21: ...value": true },] ---------------------^ Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got ']'
Watchout for malformed JSON. Expected data media type ('null') does not match real media type ('application/json').
Update, JSON sent to server:
[{"id":"12"},{"id":"10"},{"id":"9"},{"id":"8"},{"id":"7"},{"id":"6"},{"id":"5"},{"id":"4"},{"id":"3"},{"id":"2"},{"id":"1"}]
So just taking a stab at it...>I created a jsfiddle for you....fixing some missing variables.
var Person = {
"id" : '1'
};
var PersonArray =[];
PersonArray.push(Person);
var tempPersonJson = [];
tempPersonJson['PersonList'] = JSON.stringify(PersonArray);
console.log('tp',tempPersonJson);
Working fiddle click here