We Keep Coding

Copy data from a dynamic website using scrapy

I started to write a scraper for the site to collect data on cars. As it turned out, the data structure can change, since the sellers do not fill all the fields, because of what there are fields that can change, and during the scraper as a result in the csv file, the values ​​are in different fields.
page example:
https://www.olx.ua/obyavlenie/prodam-voikswagen-touran-2011-goda-IDBzxYq.html#87fcf09cbd
https://www.olx.ua/obyavlenie/fiat-500-1-4-IDBjdOc.html#87fcf09cbd
data example:
Data example
One approach was to check the field name with text () = "Category name", but I'm not sure how to correctly write the result to the correct cells.
Also I use the built-in Google developer tool, and with the help of the command document.getElementsByClassName('margintop5')[0].innerText
I brought out the whole contents of the table, but the results are not structured.
So, if the output can be in json format then it would solve my problem?
innerText result
In addition, when I studied the page code, I came across a javascript script in which all the necessary data is already structured, but I do not know how to get them.
<script type="text/javascript">
var GPT = GPT || {};
GPT.targeting = {"cat_l0":"transport","cat_l1":"legkovye-avtomobili","cat_l2":"volkswagen","cat_l0_id":"1532","cat_l1_id":"108","cat_l2_id":"1109","ad_title":"volkswagen-jetta","ad_img":"https:\/\/img01-olxua.akamaized.net\/img-olxua\/676103437_1_644x461_volkswagen-jetta-kiev.jpg","offer_seek":"offer","private_business":"private","region":"ko","subregion":"kiev","city":"kiev","model":["jetta"],"modification":[],"motor_year":[2006],"car_body":["sedan"],"color":["6"],"fuel_type":["543"],"motor_engine_size":["1751-2000"],"transmission_type":["546"],"motor_mileage":["175001-200000"],"condition":["first-owner"],"car_option":["air_con","climate-control","cruise-control","electric_windows","heated-seats","leather-interior","light-sensor","luke","on-board-computer","park_assist","power-steering","rain-sensor"],"multimedia":["acoustics","aux","cd"],"safety":["abs","airbag","central-locking","esp","immobilizer","servorul"],"other":["glass-tinting"],"cleared_customs":["no"],"price":["3001-5000"],"ad_price":"4500","currency":"USD","safedealads":"","premium_ad":"0","imported":"0","importer_code":"","ad_type_view":"normal","dfp_user_id":"e3db0bed-c3c9-98e5-2476-1492de8f5969-ver2","segment":[],"dfp_segment_test":"76","dfp_segment_test_v2":"46","dfp_segment_test_v3":"46","dfp_segment_test_v4":"32","adx":["bda2p24","bda1p24","bdl2p24","bdl1p24"],"comp":["o12"],"lister_lifecycle":"0","last_pv_imps":"2","user-ad-fq":"2","ses_pv_seq":"1","user-ad-dens":"2","listingview_test":"1","env":"production","url_action":"ad","lang":"ru","con_inf":"transportxxlegkovye-avtomobilixx46"};
data in json dict
How can I get the data from the pages using python and scrapy?

You can do it by extracting the JS code from the <script> block, using a regex to get only the JS object with the data and then loading it using the json module:
query = 'script:contains("GPT.targeting = ")::text'
js_code = response.css(query).re_first('targeting = ({.*});')
data = json.loads(js_code)
This way, data is a python dict containing the data from the JS object.
More about the re_first method here: https://doc.scrapy.org/en/latest/topics/selectors.html#using-selectors-with-regular-expressions

Pass JSON string to PHP and push to array

rookie here. I've searched and searched and still remain ignorant.
I am making an array of markers/info windows for a Google Maps API. Currently, I have succeeded in loading my markers from an external JSON file. JSON data looks like this: https://codedump.io/share/5XUwRzOFvREi/1/json-array
pathway ./json/Markers
{"Markers": [
{
"title" : "Meow Monestary",
"position" : {
"lat" : 40.5178,
"lng" : -122.6438
},
"posterContact" : {
"name" : "Mr Meowser",
"email" : "MrMeow#Couch.com",
"phone" : "(555)-202-3040",
"private" : true
},
"type" : "myResidence",
"ownerContact" : {
"name" : false,
"email" : false,
"phone" : false,
"private" : true
},
"description" : "Meow meow purrrrr. Dogs are not my favorite but they are my second favorite.",
"private" : true
},
I want users to be able to fill out a form containing all of this data and then push the new marker object into the JSON array. So far I have collected the form, created a Javascript object, and converted it to a JSON string like this...
function submitForm(){
//place form data into array
var formData = $("#shelterForm").serializeArray();
//build the javascript object using the values in the array
var shelter = {
title:formData[0].value,
position:{
lat:formData[1].value,
lng:formData[2].value
},
posterContact:{
name:formData[3].value,
email:formData[4].value,
phone:formData[5].value,
private:formData[6].value
},
type:formData[7].value,
ownerContact:{
name:formData[8].value,
email:formData[9].value,
phone:formData[10].value,
private:formData[11].value
},
description:formData[12].value,
private:formData[13].value
};
shelterString = JSON.stringify(shelter);
}
I'm sure there was an easier way to do this. If you feel inclined to go into this... GREAT!! My main issue though is that now I have my JSON string, but can't figure out how to push it into my array. I'm thinking maybe pass the string to PHP and write to file, or possibly ajax allows this?

Whether or not you use AJAX, you will still need a script on the server to save the data server side. Doing an AJAX request is more advanced than just a regular form POST, so I would recommend against it for a beginner.
Once the data is sent to PHP, you will need to store it. The most common way this would be done is with a database, typically MySQL. When the form is posted, you would get the data from the $_POST variable and store it as a new row in the database. Then, for the JSON file, rather than using a static file, you would point the maps to a PHP script for the external JSON file. That script would then query the database, assemble the data into an associative array with code very much like your javascript submitForm() function, and then call json_encode() to convert that array into a JSON string that it would then print. On the client side, you would not need your submitForm() function anymore.
If you don't want to mess around with a database, you can use a regular file on your server and have the PHP script modify that file. It is a little messier, though, and if you have an error in your script or the server loses power while writing the file, you could lose all your data, so I would recommend also setting up a daily backup if you have important data in the file. Also, you will have to take special precaution to not allow two different people to submit their updates at the same time, since having two processes writing to the same file concurrently will cause garbage data. Databases are built to be more resilient to these problems out of the box.
If you want to go the file route, you would probably still want to move the creation of the JSON into PHP. Your javascript relies on the exact indicies of the form elements, and is hard to read and maintain. In PHP, you would have something like:
$shelter = [
'title' => $_POST['shelter_title'],
'position' => [
'lat' => $_POST['shelter_latitude'],
'lng' => $_POST['shelter_latitude']
],
The $_POST keys are the name attributes from your form elements, which makes it much easier to maintain than using index numbers, which would have to be renumbered if you added or removed a form field.
Then, you would need to lock the json file to make sure that two processes don't try to update it at the same time
if (!flock($json_filename,LOCK_EX)) {
die('We are having trouble updating our records. Please try again later.');
}
//Now nobody else can write to the file until the script finishes or calls flock($json_filename, LOCK_UN) to release the lock
Then, we load the old JSON file, and update it with our new record:
$old_json = file_get_contents($json_filename); //get JSON data as string
$old_data = json_decode($old_json); //convert JSON data into PHP array
$new_data = array_push($old_data['Markers'], $shelter); //add the new shelter to PHP array
$new_json = json_encode($new_data); //convert the PHP array back to a JSON string
file_put_contents($json_filename, $new_json); //write the string to the file
//json file is updated, so now you can display a message, or redirect the user to a new page with header('Location: foo')
I haven't tested this code, so back up your JSON before trying this.

What you should be doing here, is to have a local json file to which the google map api would be pointed to and to add to that file using the html form with a php action where you would ingest the form data and add it to the json file.
<form action="addJson.php" method="post">
addJson.php (to get the general idea)
<?php
$localFile = 'json/Markers.json';
$data = [$_POST]; // reformat if required
$existing = json_decode(file_get_contents($localFile));
$all = array_merge($existing,$data);
file_put_contents($localFile,json_encode($all));
echo 'thanks for addition';
?>
And you can totally omit the javascript submitForm function.

Getting all of the .json files from a directory

I'm creating an android app which takes in some json data, is there a way to set up a directory such as;
http://......./jsons/*.json
Alternatively, a way to add into a json file called a.json, and extend its number of containing array data, pretty much add more data into the .json file this increase its size.
It could be by PHP or Javascript.

Look into Parsing JSON, you can use the JSON.parse() function, in addition, I'm not sure about getting all your JSON files from a directory call, maybe someone else will explain that.
var data ='{"name":"Ray Wlison",
"position":"Staff Author",
"courses":[
"JavaScript & Ajax",
"Buildinf Facebook Apps"]}';
var info = JSON.parse(data);
//var infostoring = JSON.stringify(info);

One way to add to a json file is to parse it, add to it, then save it again. This might not be optimal if you have large amounts of data but in that case you'll probably want a proper database anyway (like mongo).
Using PHP:
$json_data = json_decode(file_get_contents('a.json'));
array_push($json_data, 'some value');
file_put_contents('a.json', json_encode($json_data));

Local HTML 5 database usable in Mac Dashboard wigdets?

I'm trying to use HTML 5's local database feature on a Mac Dashboard widget.
I'm programming in Dashcode the following javascript:
if (window.openDatabase)
{
database = openDatabase("MyDB", "1.0", "Sample DB", 1000);
if (database)
{
...database code here...
}
}
Unfortunately the database-variable remains always null after the call to openDatabase-method. I'm starting to think that local databases are not supported in Widgets...
Any ideas?
/pom

No you will not be able to do the above. And even if you could then you would not be able to distribute the widget without distributing the database assuming it was a MySQL or SGLite. (not sure what you mean by HTML 5's local Db.
here are a number of ways round this:-
You can add a data source which can be a JSON file, or an XML file or and RSS feed. So to do this with JSON for example you would write a page on a server in PHP or something that accessed a database so that when the URL was called the result was a JSON string. Take the JSON string and parse it and use it in the Widget. This will let you get data but not save it.
Another way would be to use the user preferences. This allows you to save and retrieve data in the individual widget.
So
var preferenceKey = "key"; // replace with the key for a preference
var preferenceValue = "value"; // replace with a preference to save
// Preference code
widget.setPreferenceForKey(preferenceValue, preferenceKey);
You can then retrieve it with
var preferenceForKey = "key"; // replace with the key for a preference
// Preference code
preferenceForKey = widget.preferenceForKey(preferenceForKey);
The external call, you could also use REST will let you read any amount of data in and the preferences will let you save data for later reuse that will survive log out's and shut downs.
The Apple site has a lot of information about Widgets and tutorials as well thjat are worth working through.
Hope this helps.

JQuery and JSON

Here's something I want to learn and do. I have a JSON file that contains my product and details (size, color, description). In the website I can't use PHP and MySQL, I can only use Javascript and HTML. Now what I want to happen is using JQuery I can read and write a JSON file (JSON file will serve as my database). I am not sure if it can be done using only JQuery and JSON.
First thing, How to query a JSON file? (Example: I would search for the name and color of the product.)
How to parse the JSON datas that were searched into an HTML?
How to add details, product to the JSON file?
It will also be great if you can point me to a good tutorial about my questions.
I'm new to both JQuery and JSON.
Thanks!

Since Javascript is client side, you won't be able to write to the JSON file on the server using only Javascript. You would need some server side code in order to do that.
Reading and parsing the JSON file is not a problem though. You would use the jQuery.getJSON function. You would supply both a url and a callback parameter (data isn't needed, because you're reading a file, so no need to send data). The url would be the path to your JSON file, and the callback would be a function that uses the data.
Here's an example of what your code might look like. I don't know exactly what your JSON is, but if you have a set called "products" containing a set of objects with the details "name" and "price", this code would print those out:
$.getJSON("getProductJSON.htm",
function(data) {
$.each(data.products, function(i, item) {
var name = item.name;
var price = item.price;
// now display the name and price on the page here!
});
},
);
Basically, the data variable in $.getJSON makes the entire contents of the JSON available to you, very easily. And the $.each is used to loop over a set of JSON objects.