We Keep Coding

Scrapy Splash, How to deal with onclick?

I'm trying to scrape the following site
I'm able to receive a response but i don't know how can i access the inner data of the below items in order to scrape it:
I noticed that accessing the items is actually handled by JavaScript and also the pagination.
What should i do in such case?
Below is my code:
import scrapy
from scrapy_splash import SplashRequest
class NmpaSpider(scrapy.Spider):
name = 'nmpa'
http_user = 'hidden' # as am using Cloud Splash
allowed_domains = ['nmpa.gov.cn']
def start_requests(self):
yield SplashRequest('http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=27&tableName=TABLE27&title=%E8%BF%9B%E5%8F%A3%E5%8C%BB%E7%96%97%E5%99%A8%E6%A2%B0%E4%BA%A7%E5%93%81%EF%BC%88%E6%B3%A8%E5%86%8C&bcId=152904442584853439006654836900', args={
'wait': 5}
)
def parse(self, response):
goal = response.xpath("//*[#id='content']//a/#href").getall()
print(goal)

If you use some breakpoints you'll see its a frustrating job that I explain what I understood from my research.
when you are working with this kind of situation you have two ways:
1 - [Easy Way] use selenium and open a browser and click on each link and get the returned contents easily, you can run multiple browsers and get link contents simultaneously.
2 - [Hard Way] simulate what website does (by making similar functions inside python) and do exactly what website does in JS but in the end instead of showing the results, just save it in a variable and use it the way you want.
Now if you choose the HARD WAY this is what I found:
the link JS is like this:
commitForECMA(callbackC,'content.jsp?tableId=26&tableName=TABLE26&tableView=国产医疗器械产品（注册&Id=138150',null)
it calls a function named commitForECMA and get what this function returns and pass it to callBackC function.
well this was obvious, but its important to know what these functions do and how to replicate it.
commitForECMA:
this is the function:
function commitForECMA($_8, $_10, $_12) {
request = createXMLHttp();
request.onreadystatechange = $_8;
if ($_12 == null) {
_$du(request, _$Fe('uM6r2MG'), _$Fe("jp0YV"), $_10);
request.setRequestHeader(_$Fe("XACeXwDYXwcTV8Ur2"), _$Fe("YwDYgwceLwDT7iCYX3Ce9FKyvHKwPFa"));
} else {
var $_9 = "";
var $_19 = $_12.elements;
var $_0 = $_19.length;
for (var $_18 = 0; $_18 < $_0; $_18++) {
var $_14 = _$3P($_19, $_18);
if ($_14.type != _$Fe("yQ6YPMK20") && _$3P($_14, _$Fe('uwbm7wKV')) != "") {
if ($_9.length > 0) {
$_9 += "&" + $_14.name + "=" + _$3P($_14, _$Fe('kwbm7wKV'));
} else {
$_9 += $_14.name + "=" + _$3P($_14, _$Fe('Ewbm7wKV'));
}
$_9 += _$Fe("Jx2J03Up2Hsl");
}
}
_$du(request, _$Fe('uM6r2MG'), _$Fe("HVlesYq"), $_10);
$_9 = encodeURI($_9);
$_9 = encodeURI($_9);
request.setRequestHeader(_$Fe("d3CmOFDVz3CeXwoxBMq"), _$Fe("FMbZz3CmOFDV"));
request.setRequestHeader(_$Fe("yACeXwDYXwcTV8Ur2"), _$Fe("g3UraMD2O3UpNMCgB8cT6w6QzRbenM1TTQbS2MbJBRDY9"));
}
request.send($_9);
if ($_12 != null) {
$_12.reset();
}
}
yes as you can see it just creates a XMLHTTP request which (for the links in question) Posts the $_10 content to the server and get the results in callBackC function which is now in $_8. but the trick here is the $_10 contents goes through ~13000 lines of code to create links like this:
http://app1.nmpa.gov.cn/data_nmpa/face3/content.jsp?6SQk6G2z=GBK-56.it.xmhx8IaDT25ZyaSxljrwULe8AkNw8QjmeNqdT0YqZYbMZ2P6Jgn3ZUIgh3ibPI81bjA6xUCKJmzy1LD.4AZnk4g4G_iMO4tdiebiVDoPPtdVDIkDWw0OnDHek.d_2r.PfBtuIoxDvrbGDL.Lv2AuD6lxiObz_lldDHq6HnEw_irAP1hCH.Dr3KdW33DN2w0X1R75N3f8GXdHinmxXLtYbZNYZEE9K7lk9AGmBWgcTds.XgGVW3gDS5OEwoRat44Ecke8k7ZXoY_2revEbUrD8UpOrGprlPEwVYuAvLoTSZX8WJEWQ_QT2CDjNw0FOwAECzsFJa4hGgUtjCPzG&c1SoYK0a=GBK-4aeKAo74EouxLY.stFwdwvXQQG_hXMGG8gB0Hhe6V2Il9k9c8yiTLqduIXpv2RNt.H.weYXeF5XhV0CR2lATieRmk.cs8.fPhNpfGx7JkG1uacp75kDcmXsNtuKgbzRUHZh8vkj4UEYbPcwIYIOw5gFG_cMi9n1GYq0AXXK9UQn9IsmjCBuI7AOFw.pk91OgjvkJCcg2y0y3yDkGwZPcg5EktfAXi.PjmfaecWg8hodU87q6B3ZuPxhel9K9I3EDBxzCHtZqt_0YFlkJCcK4hLq
the problem is with obfuscation and also the nested variables and functions that can keep you out of track for hours if you try to debug it line by line (which I did) and the code makes the characters after content.jsp? part one by one and that explains why its about 13000 lines!!
this part request.send($_9); should have a body for request because its a POST request and $_9 was always null! it seems there are more protection levels to it as it seems.
callBackC:
well the callbackC is apparently a simple function to get responseText and show it to user:
function callbackC() {
if (request.readyState == 1) {
_$c2(document.getElementById(_$Fe("Y3CeXwDYXwq")), '=', _$Fe('vFKyXRUxEYlTW'), _$Fe("kHDxnHOaB3vE5HDxnHOSNMKQGQ6xOHK2z3Kw2Qne7MCm9FKyvhbwNROg"));
}
if (request.readyState == 4) {
if (request.status == 200) {
oldContent[oldContent.length] = request.responseText;
_$c2(document.getElementById(_$Fe("b3CeXwDYXwq")), '=', _$Fe('EFKyXRUxEYlTW'), request.responseText);
request = null;
} else {
_$c2(document.getElementById(_$Fe("u3CeXwDYXwq")), '=', _$Fe('BFKyXRUxEYlTW'), "<br><br><br><span style=font-size:x-large;color:#215add>服务器未返回数据</span>");
}
}
}
I didn't quiet get what those _$Xx functions do (because it goes so deep that its out of my patient!) but it seems they simply replaced the document.getElementById("someThing").innerText="Contents"; with multi layered functions so we can't understand the code easily, and the request.responseText is what you need which is HTML code for the table of results.
there is also a 3rd way which I don't know if you can implement it in your code, but since these functions are in a public scope you can simply override them by redefining these two functions (or replace the functions in the link with your own functions and run them). I tried to get the URL for the request which gave me the link I used in middle of this post, but it didn't worked (I just override the callBackC function and get request.responseURL) and the link gave me 404 error.
I don't think I said all I got from my observations but I think it's enough for you to know what you are up against if you are not already aware, and I hope I was helpful.
Reference:
XMLHttpRequest: Living Standard — Last Updated 16 August 2021

Grab URL from element on page for Bitly URL shortening

I'm working in shopify - attempting to do this client-side
I have a URL being generated (based on what items are in the cart presently) that adds items to the cart based on their ID#.
I'm building this little thing for our sales team, so they can start an order for a customer and send that arrangement to someone through a URL - right now in shopify if you do it their way it will take the customer to the checkout window and they can't edit that order - This way we're just sending an arrangement in the cart that they can adjust before they actually check out.
So right now, that url gets very very long depending on how many items are in the cart, and I'd like to use bit.ly to create a short url based on that generated url - I have it now so that it can encode the URL so it won't have any strange characters in it - but looking at the bitly api documentation most of the examples seem generic and other cases on stack overflow seemed to be specific to their problem --
Perhaps it can't be done? Thanks for taking the time to read this, if anyone has any suggestions at all - or if you think I just missed a big chunk of something obvious please feel free to tell me so. I can provide code for what I have so far if that makes it easier to understand what I'm trying to do!
screen shot of what that page looks like
---- ADDING CODE BELOW ----
// get the cart
if (typeof Shopify === 'undefined') var Shopify = {};
Shopify.cart = {{ cart | json }};
Shopify.idsInCart = [];
Shopify.quanInCart = [];
//where we gonna put the url
var cartURL = document.getElementById('cart_url');
// for every item in Shopify Cart - push to idsInCart and print the IDs to the cart url
for (var i=0; i<Shopify.cart.items.length; i++) {
Shopify.idsInCart.push(Shopify.cart.items[i].id);
cartURL.innerHTML += 'id[]=' + Shopify.idsInCart[i] + '&';
}
// get the div with cartURLform as an id
var longUrlNode = document.getElementById('cartURLform'),
// grab the .textContent from that div
textContent = longUrlNode.textContent;
//
var uri = longUrlNode.textContent;
var res = encodeURI(uri);
// Copy to clipboard example
document.querySelector("#qlink").onclick = function() {
// Select the content
document.querySelector("#qlink").select();
// Copy to the clipboard
document.execCommand('copy');
};
(function(long_url,callback){
bi = new URL("https://api-ssl.bitly.com/v3/shorten?");
var params = [
"login=__obviously__",
"domain=bit.ly",
"apiKey=__obviously__",
"longUrl="+ encodeURIComponent(long_url)
]
bi.search = "?"+params.join('&')
var xhr = new XMLHttpRequest();
xhr.onreadystatechange=function () {
if (xhr.readyState === 4) {
if (xhr.status === 200) {
var res = JSON.parse(xhr.responseText);
callback(res["data"]["url"]);
// document.getElementById("qlink").value = JSON.parse(xhr.responseText);
} else {
alert('There was a problem with the request.');
}
}
}
xhr.open("GET",bi.toString());
xhr.send(null)
})(res,function(a){
// prompt("hello", a);
document.getElementById("qlink").value = a;
});
--- Edited to add code

Iterate through appended json file

I have this code:
function heatMapRange() {
var script1 = document.createElement('script');
script1.src = 'allCoords.js';
document.getElementsByTagName('head')[0].appendChild(script1);
}
which appends the allCoords.js file above:
allCoords_callback({
"coordinates": [
[50.1729677, 12.6692243, 580000],
[50.001168, 14.4270033, 2895000],
[50.6988037, 13.9384015, 945000],
[50.1218161, 14.4824004, 409900],
[49.470061, 17.0937597, 1499000],
[49.8509959, 18.5593087, 380000]
]
});
What I want is to iterate through this data with something like this:
function allCoords_callback(results1) {
for (var i = 0; i < results1.coordinates.length; i++) {
alert(results1.coordinates[i]);
}
}
Is it possible?

You can iterate an array in javascript with Array.map().
In your example will be something like:
results1.coordinates.map(function(coordinate) { alert(coordinate); })
That's about iterating part.
Then, another topic is how do you get JSON you need to process. In the example given on Google Maps docs they do it using JSONP just because that is the way the real-time earthquake data works. Another method to fetch data are XMLHttpRequests (AKA as AJAX). This is a more common practice and I would recomend using it if possible.
In your case I would re-write your code to look something like this:
function heatMapRange(){
var request = new XMLHttpRequest();
request.open('GET', '/allCoords.js', true);
request.onload = function () {
if (request.status >= 200 && request.status < 400) {
// Success!
var data = JSON.parse(request.responseText);
// process the data in the response, like
// iterating through the list of coordinates
data.coordinates.map(function(coordinate) { alert(coordinate); })
} else {
// We reached our target server, but it returned an error
}
}
request.error = function () {
// There was a connection error of some sort
}
request.send();
}
Which fetch the data from the JSON file allCoords.json:
{
"coordinates": [
[50.1729677,12.6692243,580000],
[50.001168,14.4270033,2895000],
[50.6988037,13.9384015,945000],
[50.1218161,14.4824004,409900],
[49.470061,17.0937597,1499000],
[49.8509959,18.5593087,380000]
]
}
This way of fetching data from a server align more with the best practices used in the industry. This is just the straight forward example using vanillaJS XMLHttpRequest. There are tons of libraries that simplify this action. Even better there is Fetch API coming tackling the topic of fetching resources.

Well the code on the top works, problem was that I disabled the alerts in google chrome. So closing the tab and reopening page did the trick.

How to return data from CRM 2011 javascript odata function?

This seems silly that I haven't been able to accomplish this. I'm using the common code found on the web to do an odata query. The problem is the results stay in getFieldData(retrieveReq) routine. I don't want to immediately set a field on the current form. How can I get my values out of it so the data can be used in other javascript functions? Global variable would be good but nothing I've tried has worked. The below code displays "x".
var var1 = "x"; odataquery(); alert(var1);
The example given here has two alerts that display the data. How can Id and Name get outside of that function to be useful?
Edit1: Below is the main part of the routine that calls getFieldData(this). I want to use OwnerBUID and OwnerBUName in other javascript functions.
var retrieveReq = new XMLHttpRequest();
retrieveReq.open("GET", odataSelect, false);
retrieveReq.setRequestHeader("Accept", "application/json");
retrieveReq.setRequestHeader("Content-Type", "application/json; charset=utf-8");
retrieveReq.onreadystatechange = function () {
getFieldData(this);
};
retrieveReq.send();
function getFieldData(retrieveReq) {
if (retrieveReq.readyState == 4 && retrieveReq.status == 200) {
// 4=request complete, 200=OK
var retrieved = this.parent.JSON.parse(retrieveReq.responseText).d;
var retrievedValue = retrieved.results[0].BusinessUnitId;
OwnerBUID = retrievedValue.Id;
OwnerBUName = retrievedValue.Name;
}
}

I guess you want to put the data as the return value of a javascript function. You could do this:
var returnedData = function getFieldData(retrieveReq)
{
...
return data;
}
BTW, you could consider to use JayData, Breeze and Datajs sources code packages in your client application. They implement the low level APIs for consuming an odata service using javascript.

creating a drop down list in javascript from a text file

Is this possible? If so, I could really use some help on this. I'm very new to JavaScript and thus hardly know any of the syntactical specifications, nor proper methodology.
Some functions I wrote in an external file.
var base = base || {};
base.requestAjax = function () {
try{
return new XMLHttpRequest();
} catch (e) {
try {
return new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
return new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) {
alert("Something is wrong with your browser! Please ensure that you have javascript functionality enabled");
return false;
}
}
}
}
base.onReadyStateChange = function(ajaxRequest, formName, formArray, method, controller) {
var methodVerified = verifyMethod(method);
if (!methodVerified) {
throw new Exception("Error: please make sure the method passed matches that of \'GET\' or \'POST\'.");
}
for (input in formArray) {
document.formName.input.value = ajaxRequest.responseText;
}
ajaxRequest.open(method, controller, true);
file.send(null);
}
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
if (file.readyState != 4) {
throw new Exception("Error: text file ready state not equal to 4!");
}
if (file.status != 200) {
throw new Exception("Error: text file status not equal to 200!");
}
var lines = file.responseText.split("\n");
document.writeln("<select>");
for(line in lines) {
document.writeln("<option>" + line + "</option>");
}
document.writeln("</select>");
file.send(null);
}
base.verifyMethod = function(method) {
var methodLower = method.toString().toLowerCase();
switch(methodLower) {
case "get":
case "post":
return true;
default:
return false;
}
}
The html code which attempts to implement it
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" type="text/css" href="../includes/css/adsmanager.css" />
<script type="text/javascript" src="../includes/js/base.js">
<!--
function createCountriesList() {
var file = base.requestAjax;
var path = "../includes/_notes/countries.txt";
base.writeDropDownList(file, "GET", path);
}
//-->
</script>
</head>
<body>
<form name='adsManager'>
<button type="text" value="test" onclick="createCountriesList()" />
</form>
</body>
</html>
I plan to use the functions for other things, so I figured I'd create a namespace. I found this as a reference and based most of my model off of it.
Thanks to all who can help.

What is your point of failure? Is your Javascript sending the Ajax request and receiving the response?
Is lines getting data in this line of your code?
var lines = file.responseText.split("\n");
If you are getting this far, iterate through lines and add options like this:
var select = document.getElementById('id');
select.options.add(new Option(lines[i]));
In your writeDropDownList method, I made a few changes:
added a method that will be called when your Ajax call has completed. In that method, you should check the response string and add the options to your select box
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
file.onreadystatechange = requestComplete;
file.send(null);
}
requestComplete()
{
if (file.readyState == 4)
{
if(file.readyState == 200)
{
var lines = file.responseText.split("\n");
//foreach loop to populate select
}
}
}
In your code, you are checking and using files.responseText before you have even sent the Ajax request at files.send(null)
EDIT:
Some more comments regarding your code:
In the createCountriesList function, you create file and assign
it a value by calling requestAjax. You then pass it to
writeDropDownList function, where you assign it a value again by
calling requestAjax. You see that this is redundant? There is no
need to create file in createCountriesList and pass it as an
argument. Create it just once in writeDropDownList.
in writeDropDownList you call upload.requestAjax(). What is
upload. I don't see you initializing upload anywhere. Do you mean
to call base.requestAjax()?
you have a function base.OnReadyStateChange but at no point are you
telling your AJAX request to call that function when state changes. See the code I posted above. The function I added called
requestComplete does that, and I tell the AJAX request to call it using file.onreadystatechange = requestComplete;
You set method to GET, yet you are not passing any GET values in your URL
in file.open(method, path, true);, path is supposed to be the URL of the script the AJAX request will call. You have set path to ../includes/_notes/countries.txt. An AJAX request cannot call a .txt file since they do not execute.
I just had a look at the source of your code, and it is all sorts of broken. Please do not use it.
What is countries.txt ? Are you attempting to populate a dropdown with a list of all countries, or some countries depending on user's input?
If the former, there is no need for Javascript / AJAX. You need to add PHP code in your html to populate the select box.
If the latter, your AJAX request should be sending the user input as a GET variable.

Some comments:
Very nice code; it's readable and looks neat.
I'd use a different name than base as namespace - for my liking, the word is too "common" . Chances are that you're going to define a variable base somewhere and it will break.
To create HTML from JavaScript, first create a small example in pure HTML to see how it should look like. After that, create a script which produces the same HTML using document.createElement().
Look at frameworks like jQuery or Prototype because they make many boring tasks much more simple.