We Keep Coding

Copy data from a dynamic website using scrapy

I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values ​​are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?

You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions

How to parse a javascript list containing dictionary in python django

I have a webhook.
data0=[{"senderId":"smsdia","requestId":"******383233353233","report":[{"date":"2017-12-02 17:00:41","number":"12345","status":"1","desc":"DELIVERED"}],"userId":"119385","campaignName":"viaSOCKET"}]
I receive the above data in a POST request to my server.
Content-Type: application/x-www-form-urlencoded
How do I parse it?
I know that if it is a list: data1=['sree','kanth'] I can parse it with request.POST.getlist('data1[]')
But I don't know how to parse when it is a list containing dict.
edit1
Moreover, I get len(data1) is 2. But len(data0) is 0.
edit2
using request.lib:
https://requestb.in/13df2891?inspect

This appears to be JSON sent inside a form field. You can use the json library to parse it:
data = json.loads(request.POST['data'])

Read JSON String Shown In URL Using Javascript

i am trying to create a simple web app that gets the latitude and longitude stored in a JSON string and uses them to place markers on a google map. Currently, I have a program on a server which retrieves a JSON string with data when a URL is entered into a web browser. The JSON string produced is as follows:-
{"employees":[{"email":"bones93#hotmail.co.uk","lat":"53","lon":"-3","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"}]}
What method could i use in JavaScript that would allow me to get the JSON string that is produced?
P.S I know I will need to parse the text afterwards to make a JSON Object, this is something that can be done afterwards.

Use the Jquery library's get method to request the data from the server. Here is a link to a simple W3 tutorial : http://www.w3schools.com/jquery/ajax_get.asp
Your code will look something like this:
$("button").click(function(){
$.get("/your/server/url",function(data){
var result = JSON.parse(data);
// Process result.employees
});
});

You could use
var x = JSON.parse('{"employees":[{"email":"bones93#hotmail.co.uk","lat":"53","lon":"-3","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"},{"email":"unknown","lat":"0","lon":"0","alt":"0","date":"unknown","time":"unknown"}]}');
and then access it with:
x["employees"][0]["lat"];

try this for normal strings:
JSON.parse(str)
or if you're using AJAX to get that Json you can use as following:
$.get(..,'json')
OR
$.post(..,'json')

Get JSON Data from Cherrypy Server with jQuery/AJAX

and python/cherrypy server
#cherrypy.tools.json_out()
#cherrypy.tools.json_in()
def get_data(self):
cherrypy.response.headers['Content-Type'] = 'application/json'
datas = {"ABCDEF"}
return datas
but I get a Internal Server Error (500), where is my mistake?
I get work to post data to server, but with getting data is my problem..

One problem is in your fifth line of your second code block. Change
datas = {"ABCDEF"}
to something like
datas = { "somedata" : "ABCDEF"}
And if this is all of your cherrypy server code, you're not exposing your route. Then you have to add the
#cherrypy.expose
annotation. You can consult the docs for this as well.

Your datas variable is a Python set, and those are not directly serialisable to JSON. Perhaps you meant to create a dictionary or list?

JQuery and JSON

Here's something I want to learn and do. I have a JSON file that contains my product and details (size, color, description). In the website I can't use PHP and MySQL, I can only use Javascript and HTML. Now what I want to happen is using JQuery I can read and write a JSON file (JSON file will serve as my database). I am not sure if it can be done using only JQuery and JSON.
First thing, How to query a JSON file? (Example: I would search for the name and color of the product.)
How to parse the JSON datas that were searched into an HTML?
How to add details, product to the JSON file?
It will also be great if you can point me to a good tutorial about my questions.
I'm new to both JQuery and JSON.
Thanks!

Since Javascript is client side, you won't be able to write to the JSON file on the server using only Javascript. You would need some server side code in order to do that.
Reading and parsing the JSON file is not a problem though. You would use the jQuery.getJSON function. You would supply both a url and a callback parameter (data isn't needed, because you're reading a file, so no need to send data). The url would be the path to your JSON file, and the callback would be a function that uses the data.
Here's an example of what your code might look like. I don't know exactly what your JSON is, but if you have a set called "products" containing a set of objects with the details "name" and "price", this code would print those out:
$.getJSON("getProductJSON.htm",
function(data) {
$.each(data.products, function(i, item) {
var name = item.name;
var price = item.price;
// now display the name and price on the page here!
});
},
);
Basically, the data variable in $.getJSON makes the entire contents of the JSON available to you, very easily. And the $.each is used to loop over a set of JSON objects.