my website relies on a database which is a big JSON file like this:
var myjsonData =
[ {
"ID": 0,
"name": "Henry",
"surname": "McLarry",
"...": "...",
}]
I do generate this data every month at high cost to me, therefore I would like to avoid calling it straight in my html <head>, because this will allow any user to download the full database in no time.
I would like to build a "something" that can only call specific items from the json file (just the only one I want to show) without "exposing" the full .json onto client side.
today I use the call
var myvar= myjsonData.ID.Name
to get "Henry" into myvar, I would like to build something like
var myvar = mycallfunction(ID,Name)
I did try with PHP as intermediary but the ajax calls from javacript doesn't allow me to fetch the data.
Can I use JQuery with the JSON Url to get only the item I need?
What you can do is parse your json for an object. So you can get any value you want from json.
Example:
var myjsonData = '{"ID": 0,"name": "Henry","surname": "McLarry"}';
obj = JSON.parse(myjsonData);
console.log(myjsonData.ID); //print the id
console.log(myjsonData.name); //print the name
console.log(myjsonData.surname); //print the surname
So you have a NoSQL Database which has only one kind of Document that is the full JSON element you use in your website. In that scenario you have three options:
Depending on the NoSQL Database you're using you can limit the fields which will be returned(I.e: For MongoDB you can look here: https://docs.mongodb.com/manual/tutorial/project-fields-from-query-results/)
Change the way you store you data into more modular documents and make the logic to connect them in you application. So instead of one big document you'll have modular ones as Users, Products, Transactions and etc and you can use your application to query them individually.
Build a Server Side logic as an API to deal with your data and provide only what you need, so the API(Which can be node.js, php, or any you may like) will get the full JSON it`s endpoints will only the data you want. For example: myapi.com/getUser, myapi.com/getProducts and so on.
If you're able to provide more info on the technologies you're using that would help us. Hope that helped :).
I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?
You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions
I'm trying to fetch data from a MySQL database relevant to the selected date_time.
I have extracted the date_time from the database and displayed it on a page and now the thing I want to do is that "When I select a date the data on that date should be display without refreshing the page".
you have to create an other .php page containing calling your date_time and call it in ajax after ward.
your ajax request should looks like :
$("#Button_get_date").onclick ( function () {
$("#div_to_display").load("/date.php", {"date= (#optiondate#.val));
})
the other page php should contain
$data = $ddb ->query("select * from tableinfodate where date = ".$_POST["date"].");
after i think you know what to do but remember to make your display in your in the loaded page.
have a look at the function load :http://www.w3schools.com/jquery/jquery_ajax_load.asp and also at GET and POST method in php. More over i think it is better to try hard on js before trying to do loaded function as this one.
I hope it helps a bit.
Apologies if this seems basic to some, but I'm new to JS/node.js/JSON and still finding my way. I've searched this forum for an hour but cannot find a specific solution.
I have a basic website setup running of a local Node.js server along with 2x JSON data files with information about 32x local suburbs.
An example of an API GET request URL on the site would be:
.../api/b?field=HECTARES
The structure of the JSON files are like:
JSON Structure
In the JSON file there are 32x Features (suburbs), each with it's own list of Properties as shown above. What I am trying to do is use the API 'field' query to push all the HECTARES values each of the 32x Features into a single output variable. The code below is an example of how far I have got:
var fieldStats = [];
var fieldQ = req.query['field'];
for (i in suburbs.features) {
x = suburbs.features[i].properties.HECTARES;
fieldStats.push(x);
}
As you can see in the above "HECTARES" is hard-coded - I need to be able to pass the 'fieldQ' variable to this code but have no idea how to.
Advice appreciated!
Exactly the same syntax you are using just above:
suburbs.features[i].properties[fieldQ];
So in my controller logic, I build up a large array of data, which I json_encode into a variable which is passed to my view. However, I want this data to be sortable on the client side using JavaScipt--I don't think the details of how the sorting is done are relevant but I'm curious what the best way is to get my array from PHP to Javascript. Currently, I'm thinking of just having a tag in my html like
<script type="text/javascript">var jsonData = <?php echo $myData ?>; </script>
where myData is the encoded array I made in PHP, and then jsonData is available for me to use anywhere else.
However, this would mean the entire ugly array would show up in the source code of my page. This isn't a security concern or anything, but I feel that there must be a better way to do this that's not quite as "ugly".
Any advice is appreciated.
You have two options.
If you don't want 'ugly' HTML source, you can query your application with ajax and have your json array be stored in a variable. This can be accomplished with a simple route and controller method.
In your routes.php
Route::get('api/myData',array('as'=>'api.myData','uses'=>'MyController#getMyData'));
In your MyController.php
public function getMyData() {
return whateverYouWant();
}
You can then do an ajax request to route('api.myData')
The route method is a global function which is available in your view. It will take the named route, api.myData and generate a URL for it.
The other option is as you described, passing the array to your view.