We Keep Coding

Copy data from a dynamic website using scrapy

I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values ​​are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?

You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions

Loading model in tensorflowjs from string

I have a json string representing a model stored in a js variable. I created it using a gui that generates this json. I would like to load it to tensorflow using tf.loadModel, but the API only accepts url or localstorage. Is there any way to do it directly from string?

Unfortunately Tensorflow.js doesn't provide any way to do this currently.
However, if you need to do it with a string, a possible workaround is to save the string directly to localStorage and retrieve it from there with the URI.
Something like this (untested):
var modelJson = { ... }
localStorage.setItem('model', JSON.stringify(modelJson))
const model = await tf.loadModel('localstorage://model');

Print data from json file

I have json file with given data (total.json)
var data = {"trololo":{"info":"61511","path".... }}
I need to get object "info" and then print data "61511" in alert window
I include my json like
var FILE = 'total'
var data_file_names = {};
data_file_names[FILE] = 'total.json';
And then i use it like
var data_trololo = data_file_names[FILE];
Plese, help me print object "info". Maybe there is another way to solve this problem

You need to make an ajax call to the json file. Then you can access the array like the below example.
Note : Your json wasn't properly formatted.
var data = {
"trololo":{
"info": ["61511","path"]
}
};
console.log(data.trololo.info[0]); //this one will print 61511

Usually one can make an ajax call to read the file on the server.
But if you are ok with using HTML5 features then go through the link find out how to read the file on the browser itself. Though File API being part of HTML5 spec is stable across browsers.
http://www.html5rocks.com/en/tutorials/file/dndfiles/

Read JSON String Shown In URL Using Javascript

i am trying to create a simple web app that gets the latitude and longitude stored in a JSON string and uses them to place markers on a google map. Currently, I have a program on a server which retrieves a JSON string with data when a URL is entered into a web browser. The JSON string produced is as follows:-
{"employees":[{"email":"bones93#hotmail.co.uk","lat":"53","lon":"-3","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"}]}
What method could i use in JavaScript that would allow me to get the JSON string that is produced?
P.S I know I will need to parse the text afterwards to make a JSON Object, this is something that can be done afterwards.

Use the Jquery library's get method to request the data from the server. Here is a link to a simple W3 tutorial : http://www.w3schools.com/jquery/ajax_get.asp
Your code will look something like this:
$("button").click(function(){
$.get("/your/server/url",function(data){
var result = JSON.parse(data);
// Process result.employees
});
});

You could use
var x = JSON.parse('{"employees":[{"email":"bones93#hotmail.co.uk","lat":"53","lon":"-3","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"}]}');
and then access it with:
x["employees"][0]["lat"];

try this for normal strings:
JSON.parse(str)
or if you're using AJAX to get that Json you can use as following:
$.get(..,'json')
OR
$.post(..,'json')

Getting all of the .json files from a directory

I'm creating an android app which takes in some json data, is there a way to set up a directory such as;
http://......./jsons/*.json
Alternatively, a way to add into a json file called a.json, and extend its number of containing array data, pretty much add more data into the .json file this increase its size.
It could be by PHP or Javascript.

Look into Parsing JSON, you can use the JSON.parse() function, in addition, I'm not sure about getting all your JSON files from a directory call, maybe someone else will explain that.
var data ='{"name":"Ray Wlison",
"position":"Staff Author",
"courses":[
"JavaScript & Ajax",
"Buildinf Facebook Apps"]}';
var info = JSON.parse(data);
//var infostoring = JSON.stringify(info);

One way to add to a json file is to parse it, add to it, then save it again. This might not be optimal if you have large amounts of data but in that case you'll probably want a proper database anyway (like mongo).
Using PHP:
$json_data = json_decode(file_get_contents('a.json'));
array_push($json_data, 'some value');
file_put_contents('a.json', json_encode($json_data));