I am building a bot to check stock of various Ubiquiti Unifi devices from their shop page (Hey, these things are disappearing FAST) and I need some help. I've been searching all day for something like this, but none of the things I've seen here have quite worked.
I'm using the below code to access the UI.com shop (https://store.ui.com/). They, quite conveniently, have the in-stock product information in the header of every page, and I'm using selenium to get the home-page and need to access:
<script data-ot-ignore type="text/javascript">
window.APP_DATA = {
assets: {...},
cart: {"note":null,"attributes":{"quantity-hdds":"{\"4446782390361\"=\u003e{\"0\"=\u003e{\"sku\"=\u003e\"HDD-1TB\", \"ratio\"=\u003e\"1\"}, \"1\"=\u003e{\"sku\"=\u003e\"HDD-8TB\", \"ratio\"=\u003e\"1\"}}}"},"original_total_price":0,"total_price":0,"total_discount":0,"total_weight":0.0,"item_count":0,"items":[],"requires_shipping":false,"currency":"USD","items_subtotal_price":0,"cart_level_discount_applications":[]},
cartAccessories: [{
"id": 4446782390361,
"title": "Dream Machine Pro",
"handle": "udm-pro",
"url": "\/products\/udm-pro",
"tags": ["#HDD-1TB","#HDD-8TB","ALT","ALT::udm-pro","bestseller","enhanced-wizard","featured","mx29","recommended","redirect-wizard","related","UI::1U","UI::AI","UI::Cloud Key","UI::HDD","UI::Network","UI::SFP+","UI::UniFi","unifi"],
"featured_image": "//cdn.shopify.com/s/files/1/1439/1668/products/UDM-Pro_front-top-angle_53e97c87-61d9-4f3e-acad-6ba113bbf5de_small.png?v=1629983008",
"variants": [{
"id": 32264307703897,
"title": "Default Title",
"price": 37900,
"sku": "UDM-Pro",
"available": true,
"inventory_empty":false,
"inventory_policy": "deny",
"image": "//cdn.shopify.com/s/files/1/1439/1668/products/UDM-Pro_front-top-angle_53e97c87-61d9-4f3e-acad-6ba113bbf5de_small.png?v=1629983008"
},],
"data":{"for":{"product-vendors":[],"product-types":["VoIP","Access","Surveillance"]},"type":"UDM-PRO","view":{"default":["UAP-nanoHD-US","UAP-FlexHD-US","UWB-XG-US","UAP-IW-HD-US","UAP-AC-HD-US","UAP-AC-M-US","UAP-BeaconHD-US","UAP-AC-PRO-US","UAP-AC-LITE-US","UAP-AC-LR-US","UAP-AC-IW-US","UAP-AC-M-PRO-US","UAP-AC-SHD-US","UAP-XG-US","UAP-AC-EDU-US","USW-48-POE","USW-24","USW-Pro-24","USW-48-BETA","USW-Lite-16-PoE-BETA","USW-LEAF-BETA","USW-16-PoE","USW-24-PoE","USW-Pro-48-PoE","USW-Pro-24-PoE","USW-Pro-48o","USW-Industrial","UVC-G4-DoorBell","UVC-G3-FLEX","UP-Sense-BETA","UP-Sense","*","!UT-ATA-BETA","!UT-Conference-BETA"],"checkout":["!UDM-Pro","!UDM-US","UVC-G4-DoorBell","UWB-XG-US","UAP-AC-HD-US","UAP-FlexHD-US","UAP-IW-HD-US","UAP-nanoHD-US","UAP-BeaconHD-US","UAP-XG-US","UAP-AC-SHD-US","UAP-AC-EDU-US","UAP-AC-M-PRO-US","UAP-AC-PRO-US","UAP-AC-LR-US","UAP-AC-M-US","UAP-AC-IW-US","UAP-AC-LITE-US","U6-IW-US-BETA","U6-Extender-US-BETA","U6-Lite-US-BETA"],"bundle":["!UDM-Pro","!UDM-US","UVC-G4-DoorBell","UWB-XG-US","UAP-AC-HD-US","UAP-FlexHD-US","UAP-IW-HD-US","UAP-nanoHD-US","UAP-BeaconHD-US","UAP-XG-US","UAP-AC-SHD-US","UAP-AC-EDU-US","UAP-AC-M-PRO-US","UAP-AC-PRO-US","UAP-AC-LR-US","UAP-AC-M-US","UAP-AC-IW-US","UAP-AC-LITE-US"]},"priority":1,"description":"","countries":[]}},{
Now, I don't have a lot of experience with javascript, but it looks like my data of interest is basically a javascript array of objects inside another object? (The [{}] structure of "cartAccessories" where everything is.) Element inspect of the source code gives me ""/html/head/script[33]" as the XPATH for the script...I think. It seems to return different data almost each time.
I'm using the following basic code to get the page:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
urlpage = 'https://store.ui.com/'
print(urlpage)
driver = webdriver.Firefox()
# get web page
driver.get(urlpage)
time.sleep(1)
print("Getting Results.")
results = driver.find_element(By.XPATH, "/html/head/script[33]")
html = results.get_attribute('innerHTML')
print(f"The results are: {html}")
driver.quit()
But this doesn't seem to be right. I want to get that "cartAccessories" information into a python list so I can work on it. What's the best way to access this information? Am I going about this all wrong?
You can use a regex to grab the overarching JavaScript object containing the array of interest within it, then pass that to hjson to deal with the unquoted keys. Finally, extract the cartAccessories item and do what you want with it.
import requests, re, hjson
r = requests.get('https://store.ui.com/')
data =hjson.loads(re.search(r'window.APP_DATA = (.*?)<', r.text, re.S).group(1))
print(data['cartAccessories'])
I am trying to connect to Phabricator conduit API and create a task via a javascript bonded to a google sheet.
The Conduit API Docs linked here doesn't really explain as much. I have seen better API documentations!
Below is what I have in mind but this is a cURL and I have no idea how to make it Javascript or wither this would work or not? I appreciate the help
curl https://secure.phabricator.com/api/maniphest.edit \
-d api.token=api-token \
-d param= [
{
"type": "title",
"value": "A value from a cell on the googlesheet"
},
{
"type": "description",
"value": "A value from a cell on the googlesheet"
},
{
"type": "subscribers.add",
"value": "A value from a cell on the googlesheet"
}
] \
Generally speaking the steps are:
First, generate an API token in:
https://phabricator.yourdomain.com/settings/user/username/page/apitokens/
where phabricator.yourdomain.com must be changed by the subdomain you have Phabricator installed and username must be changed by your administration user name.
Then, let's say you have installed Phabricator in phabricator.yourdomain.com, you can request the API methods with URLs of the following type
https://phabricator.yourdomain.com/api/method_name?parameter1=value1¶meter2=value2...
where method_name must be replaced by the descriptor of a real method from this catalog:
https://secure.phabricator.com/conduit/
For example, if you want to read the contents of task number 125, with a generated API token of value api-svhcp2a3qmgkkjfa5f6sh7cm4joz, use the method maniphest.info to complete a URL like this:
http://phabricator.yourdomain.com/api/maniphest.info?api.token=api-svhcp2a3qmgkkjfa5f6sh7cm4joz&task_id=125&output=json
This URL can be directly tested in your preferred browser to obtain a JSON response with the information about task number 125 (make sure that task ID exists). Firefox will even show the returned JSON in a human-readable fashion.
These working URLs can be then inserted in Javascript as
window.location.href=http://phabricator.yourdomain.com/api/maniphest.info?api.token=api-svhcp2a3qmgkkjfa5f6sh7cm4joz&task_id=125&output=json
or as an asynchronous Ajax call.
I had a similar problem as you (I used HTTParty with Ruby).
To solve it I used the following body (using your example):
"transactions[0][type]=title&transactions[0][value][0]=A value from a cell on the googlesheet&transactions[1][type]=description&transactions[1][value]=A value from a cell on the googlesheet&transactions[2][type]=subscribers.add&transactions[2][value][0]=A value from a cell on the googlesheet"
I am building a JavaScript/jQuery Plugin which will take a JSON array of objects and generate a popover list item selector/picker. You click an item/input on the page and the popover like the image below opens with your JSON list data.
Key Features:
Search filter
Paging and items per page option
Categories and tab panel/DIVs to organize the list items into different screens on the popover
Clicking an item selects that item by firing a custom DOM event which you can plug into to save the selected value to a DB or do whatever you want with it.
Makes list items from JSON array of objects
Categories can be plucked from the same JSON data and will build separate list objects for each category
JSON Data Format
Right now the JSON array of objects looks like this below:
var mock_data_types = [{
"type": "first_name",
"name": "First Name1",
"description": "Random generated First Name",
"example": "Jason",
"category": "person"
}, {
"type": "address",
"name": "address2",
"description": "Random generated Last Name",
"example": "Davis",
"category": "address"
}, {
"type": "domain",
"name": "Domain Name3",
"description": "Random generated WWW Domain Name",
"example": "google.com",
"category": "it"
}];
Current Problems
1)
When there are multiple popover pickers on the page, when you select a item it updates that selected value for all the input fields that have the popover instead of only the current open one.
I believe the code needs to maybe keep track of multiple instances
I have tried to do this but keep failing. Can anyone hep with this issue?
2)
The complete code base is crap and can be improved as I am a PHP developer and learning JS still!
Demo JSFiddle
https://jsfiddle.net/jasondavis/9fpc3LLo/
Great job with your plugin.
So, perhaps you could use it as a jQuery-UI (plugin) widget. You can get the docs here to use the jQuery-UI widgetFactory. I think this can help you with your first problem and give you some tips to improve the second one...
Hope that can help you.
Regards!
I have an API in a Node.js which requires a JSON file, and on some requests maps in content from it. Simple enough, and works perfectly for the first request.
On subsequent requests one of the properties is left off. This is obviously a stripped down version of the code, but is all I'm doing to get the content.
Essentially my JSON looks like:
{"sections": {
"city": {
"title": "Seattle",
"info": "some info",
"tips": [ "tip 1", "tip 2" ]
}
}
}
and I require it:
var Content = require("content");
// some code
return req.status(200).json({ data: data, content: Content.sections.city });
The first request returns the whole content object. Every request after that returns only title and info, but not tips.
Edit -- I'm a dummy. In the //some code section, I work with the tips and eventually call delete on this version of the object. Apparently passing by reference still hasn't sunken in.
I'm trying to decide on a standard for json responses for my applications going forwards.
So I'm thinking something along the lines of
Fake Json
{
Message:"hello, this is a reponse",
action: 0 or 1, or "Good" or "bad",
statusCode: 234,
}
I'm wondering if there is a good standard for this already, and also how I can keep my standards in sync between my front and back end?
Is there a clever way to do this?
I'm using a structure similar to this:
{ status: ("ok", "error" or "redirect"), message: "some message goes here.", fields: "field selectors goes here in case some highlighting is needed", url: "url to open or redirect to if needed" }
The status property is the key to tell what to do next;
The message can be used to show a popup message or an inline message, etc. Can also be a list of messages separated by a '|' character to split;
The field property is used to select (with jQuery or other JS library) the fields or other objects that need an action (think of this for field validation);
The url property is used to change the page, open a popup or post something back, depending on your needs.