I am building a bot to check stock of various Ubiquiti Unifi devices from their shop page (Hey, these things are disappearing FAST) and I need some help. I've been searching all day for something like this, but none of the things I've seen here have quite worked.
I'm using the below code to access the UI.com shop (https://store.ui.com/). They, quite conveniently, have the in-stock product information in the header of every page, and I'm using selenium to get the home-page and need to access:
<script data-ot-ignore type="text/javascript">
window.APP_DATA = {
assets: {...},
cart: {"note":null,"attributes":{"quantity-hdds":"{\"4446782390361\"=\u003e{\"0\"=\u003e{\"sku\"=\u003e\"HDD-1TB\", \"ratio\"=\u003e\"1\"}, \"1\"=\u003e{\"sku\"=\u003e\"HDD-8TB\", \"ratio\"=\u003e\"1\"}}}"},"original_total_price":0,"total_price":0,"total_discount":0,"total_weight":0.0,"item_count":0,"items":[],"requires_shipping":false,"currency":"USD","items_subtotal_price":0,"cart_level_discount_applications":[]},
cartAccessories: [{
"id": 4446782390361,
"title": "Dream Machine Pro",
"handle": "udm-pro",
"url": "\/products\/udm-pro",
"tags": ["#HDD-1TB","#HDD-8TB","ALT","ALT::udm-pro","bestseller","enhanced-wizard","featured","mx29","recommended","redirect-wizard","related","UI::1U","UI::AI","UI::Cloud Key","UI::HDD","UI::Network","UI::SFP+","UI::UniFi","unifi"],
"featured_image": "//cdn.shopify.com/s/files/1/1439/1668/products/UDM-Pro_front-top-angle_53e97c87-61d9-4f3e-acad-6ba113bbf5de_small.png?v=1629983008",
"variants": [{
"id": 32264307703897,
"title": "Default Title",
"price": 37900,
"sku": "UDM-Pro",
"available": true,
"inventory_empty":false,
"inventory_policy": "deny",
"image": "//cdn.shopify.com/s/files/1/1439/1668/products/UDM-Pro_front-top-angle_53e97c87-61d9-4f3e-acad-6ba113bbf5de_small.png?v=1629983008"
},],
"data":{"for":{"product-vendors":[],"product-types":["VoIP","Access","Surveillance"]},"type":"UDM-PRO","view":{"default":["UAP-nanoHD-US","UAP-FlexHD-US","UWB-XG-US","UAP-IW-HD-US","UAP-AC-HD-US","UAP-AC-M-US","UAP-BeaconHD-US","UAP-AC-PRO-US","UAP-AC-LITE-US","UAP-AC-LR-US","UAP-AC-IW-US","UAP-AC-M-PRO-US","UAP-AC-SHD-US","UAP-XG-US","UAP-AC-EDU-US","USW-48-POE","USW-24","USW-Pro-24","USW-48-BETA","USW-Lite-16-PoE-BETA","USW-LEAF-BETA","USW-16-PoE","USW-24-PoE","USW-Pro-48-PoE","USW-Pro-24-PoE","USW-Pro-48o","USW-Industrial","UVC-G4-DoorBell","UVC-G3-FLEX","UP-Sense-BETA","UP-Sense","*","!UT-ATA-BETA","!UT-Conference-BETA"],"checkout":["!UDM-Pro","!UDM-US","UVC-G4-DoorBell","UWB-XG-US","UAP-AC-HD-US","UAP-FlexHD-US","UAP-IW-HD-US","UAP-nanoHD-US","UAP-BeaconHD-US","UAP-XG-US","UAP-AC-SHD-US","UAP-AC-EDU-US","UAP-AC-M-PRO-US","UAP-AC-PRO-US","UAP-AC-LR-US","UAP-AC-M-US","UAP-AC-IW-US","UAP-AC-LITE-US","U6-IW-US-BETA","U6-Extender-US-BETA","U6-Lite-US-BETA"],"bundle":["!UDM-Pro","!UDM-US","UVC-G4-DoorBell","UWB-XG-US","UAP-AC-HD-US","UAP-FlexHD-US","UAP-IW-HD-US","UAP-nanoHD-US","UAP-BeaconHD-US","UAP-XG-US","UAP-AC-SHD-US","UAP-AC-EDU-US","UAP-AC-M-PRO-US","UAP-AC-PRO-US","UAP-AC-LR-US","UAP-AC-M-US","UAP-AC-IW-US","UAP-AC-LITE-US"]},"priority":1,"description":"","countries":[]}},{
Now, I don't have a lot of experience with javascript, but it looks like my data of interest is basically a javascript array of objects inside another object? (The [{}] structure of "cartAccessories" where everything is.) Element inspect of the source code gives me ""/html/head/script[33]" as the XPATH for the script...I think. It seems to return different data almost each time.
I'm using the following basic code to get the page:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
urlpage = 'https://store.ui.com/'
print(urlpage)
driver = webdriver.Firefox()
# get web page
driver.get(urlpage)
time.sleep(1)
print("Getting Results.")
results = driver.find_element(By.XPATH, "/html/head/script[33]")
html = results.get_attribute('innerHTML')
print(f"The results are: {html}")
driver.quit()
But this doesn't seem to be right. I want to get that "cartAccessories" information into a python list so I can work on it. What's the best way to access this information? Am I going about this all wrong?
You can use a regex to grab the overarching JavaScript object containing the array of interest within it, then pass that to hjson to deal with the unquoted keys. Finally, extract the cartAccessories item and do what you want with it.
import requests, re, hjson
r = requests.get('https://store.ui.com/')
data =hjson.loads(re.search(r'window.APP_DATA = (.*?)<', r.text, re.S).group(1))
print(data['cartAccessories'])
I have a JSON file that contains lots of nodes. Each node has a unique id. The following is an example skeleton structure of the JSON file.
{
"node1": {
"id": "1",
"name": "student",
"properties": {
"name": "Ram",
"lastname": "Kumar",
"age": "20"
}
},
"node2": {
"id": "2",
"name": "Teacher",
"properties": {
"name": "Ram",
"subject": "Computers"
}
}
Now, this was just sample data. Suppose, I have to find node 2, provided I have the position of the node in the file. That is it's beginning and the end. I create a read stream from the start position and the end position of the file to get the object within the complete JSON object in the file. Now, using JSON.parse function I am able to create an object within the program after parsing it.
But, now I use a writestream to make the changes in the object at the required positions after computing them.
For example, I have to edit the subject property of the teacher, and the main rule is I cannot completely write the object again. Rather, create a writestream at the position where there is subject property and just edit that value.
So, the question is how to do so? And that to with multiple positions to edit, that is editing multiple properties, without rewriting them all.
I know I might face an issue if I use multi-threading for this to simultaneously edit the multiple properties at once, but I need a work around that too.
I have a web app that accesses data from an API. I need to pull data out, but it's been 'HTMLized' and I'm unsure how to go about rendering the actual data.
I've tried pulling the data from the array using bracket notation, which gives me the data but it includes the HTML tags.
javascript
{
"id": "5c9cd6576ebf251b944ed16d",
"number": 3451,
"user_id": "5b8425c8e694aa3c6a835b67",
"state": "active",
"subject": "Bower Steel 3D",
"label_ids": [
"5c4b0cf4bbddbd48cd69e68a"
],
"customer_id": "5b51be9ee694aa03f8c834be",
"type": "email",
"reply_to": "xxxxxxx.xxxxxx#xxxxxxx.com",
"reply_cc": "",
"group_id": "5a5f65fed5593070120b9779",
"inbox_id": "5a5f65fed5593070120b9779",
"updated_at": "2019-04-03T12:46:50Z",
"created_at": "2019-03-28T14:12:39Z",
"spam": false,
"trash": false,
"summary": "Great! Glad to hear it.",
"rating": null,
"waiting_since": "2019-04-03T12:21:43Z",
"messages": [
{
"id": "5c9cd6576ebf251b944ed16e",
"type": "reply",
"from_name": "Cxxxxxxx Sxxxxxx",
"body": "<div class=\"5cb74b836ebf2578174d567c\">
<style>cb74b836ebf2578174d567c
p.5cb74b836ebf2578174d567cMsoNormal,
</style>\n\n\n
<div class=\"5cb74b836ebf2578174d567cWordSection1\">\n
<p class=\"5cb74b836ebf2578174d567cMsoNormal\">
Actual content I want to render
</p>
<p class=\"5cb74b836ebf2578174d567cMsoNormal\"></p>\n
<p class=\"5cb74b836ebf2578174d567cMsoNormal\"></p>\n
<p class=\"5cb74b836ebf2578174d567cMsoNormal\"></p>
The content I want to render is embedded like this. It's basically an array within the object, called messages, and it contains a message trail - from the 1st message to the last reply.
I've never come across HTMLized data like this. I can get to the data by using bracket notation, but I don't know how to parse the extraneous tags and get to the data.
Any help would be appreciated.
Short of having the API produce a better endpoint for you to consume, you will have to handle the 'HTMLized' data. In cases like this, I have used something to parse the HTML. For example you could throw the body string into the jQuery constructor.
jQuery(messages[i].body)
That will give you something digestible programmatically to search for your data.
I am building a JavaScript/jQuery Plugin which will take a JSON array of objects and generate a popover list item selector/picker. You click an item/input on the page and the popover like the image below opens with your JSON list data.
Key Features:
Search filter
Paging and items per page option
Categories and tab panel/DIVs to organize the list items into different screens on the popover
Clicking an item selects that item by firing a custom DOM event which you can plug into to save the selected value to a DB or do whatever you want with it.
Makes list items from JSON array of objects
Categories can be plucked from the same JSON data and will build separate list objects for each category
JSON Data Format
Right now the JSON array of objects looks like this below:
var mock_data_types = [{
"type": "first_name",
"name": "First Name1",
"description": "Random generated First Name",
"example": "Jason",
"category": "person"
}, {
"type": "address",
"name": "address2",
"description": "Random generated Last Name",
"example": "Davis",
"category": "address"
}, {
"type": "domain",
"name": "Domain Name3",
"description": "Random generated WWW Domain Name",
"example": "google.com",
"category": "it"
}];
Current Problems
1)
When there are multiple popover pickers on the page, when you select a item it updates that selected value for all the input fields that have the popover instead of only the current open one.
I believe the code needs to maybe keep track of multiple instances
I have tried to do this but keep failing. Can anyone hep with this issue?
2)
The complete code base is crap and can be improved as I am a PHP developer and learning JS still!
Demo JSFiddle
https://jsfiddle.net/jasondavis/9fpc3LLo/
Great job with your plugin.
So, perhaps you could use it as a jQuery-UI (plugin) widget. You can get the docs here to use the jQuery-UI widgetFactory. I think this can help you with your first problem and give you some tips to improve the second one...
Hope that can help you.
Regards!
I am trying to learn promises in javascript and I came across this short video tutorial, there is a complete code from it on plunker.
What I wonder about in this code is in this line in promises.js file:
return $.get('tweets.json?id=' + profile.id);
First of all, why do we even need to pass any parameters there, since there is no profile.id field in tweets.json file, and when removing it I get the same results. If there was that field in tweets.json how would we achieve to get only tweets from the users that we pass with get request?
I have tried something like this but it doesn't work:
In promises.json I have changed the get request to:
return $.get('tweets.json?profile=' + profile.id);
And have added a new field to the tweets.json file, but only to the first tweet:
{
"id": 1,
"profile": 12302151,
"tweet": "OMG, worst day ever, my BF #BobbyBoo dumped me",
"usersMentioned": [
{
"id": 10,
"username": "BobbyBoo"
}
]
},
I was then expecting that I will only get that one tweet for that user, but I am still getting the whole list