We Keep Coding

Scrapy Splash, How to deal with onclick?

I'm trying to scrape the following site
I'm able to receive a response but i don't know how can i access the inner data of the below items in order to scrape it:
I noticed that accessing the items is actually handled by JavaScript and also the pagination.
What should i do in such case?
Below is my code:
import scrapy
from scrapy_splash import SplashRequest
class NmpaSpider(scrapy.Spider):
name = 'nmpa'
http_user = 'hidden' # as am using Cloud Splash
allowed_domains = ['nmpa.gov.cn']
def start_requests(self):
yield SplashRequest('http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=27&tableName=TABLE27&title=%E8%BF%9B%E5%8F%A3%E5%8C%BB%E7%96%97%E5%99%A8%E6%A2%B0%E4%BA%A7%E5%93%81%EF%BC%88%E6%B3%A8%E5%86%8C&bcId=152904442584853439006654836900', args={
'wait': 5}
)
def parse(self, response):
goal = response.xpath("//*[#id='content']//a/#href").getall()
print(goal)

If you use some breakpoints you'll see its a frustrating job that I explain what I understood from my research.
when you are working with this kind of situation you have two ways:
1 - [Easy Way] use selenium and open a browser and click on each link and get the returned contents easily, you can run multiple browsers and get link contents simultaneously.
2 - [Hard Way] simulate what website does (by making similar functions inside python) and do exactly what website does in JS but in the end instead of showing the results, just save it in a variable and use it the way you want.
Now if you choose the HARD WAY this is what I found:
the link JS is like this:
commitForECMA(callbackC,'content.jsp?tableId=26&tableName=TABLE26&tableView=国产医疗器械产品（注册&Id=138150',null)
it calls a function named commitForECMA and get what this function returns and pass it to callBackC function.
well this was obvious, but its important to know what these functions do and how to replicate it.
commitForECMA:
this is the function:
function commitForECMA($_8, $_10, $_12) {
request = createXMLHttp();
request.onreadystatechange = $_8;
if ($_12 == null) {
_$du(request, _$Fe('uM6r2MG'), _$Fe("jp0YV"), $_10);
request.setRequestHeader(_$Fe("XACeXwDYXwcTV8Ur2"), _$Fe("YwDYgwceLwDT7iCYX3Ce9FKyvHKwPFa"));
} else {
var $_9 = "";
var $_19 = $_12.elements;
var $_0 = $_19.length;
for (var $_18 = 0; $_18 < $_0; $_18++) {
var $_14 = _$3P($_19, $_18);
if ($_14.type != _$Fe("yQ6YPMK20") && _$3P($_14, _$Fe('uwbm7wKV')) != "") {
if ($_9.length > 0) {
$_9 += "&" + $_14.name + "=" + _$3P($_14, _$Fe('kwbm7wKV'));
} else {
$_9 += $_14.name + "=" + _$3P($_14, _$Fe('Ewbm7wKV'));
}
$_9 += _$Fe("Jx2J03Up2Hsl");
}
}
_$du(request, _$Fe('uM6r2MG'), _$Fe("HVlesYq"), $_10);
$_9 = encodeURI($_9);
$_9 = encodeURI($_9);
request.setRequestHeader(_$Fe("d3CmOFDVz3CeXwoxBMq"), _$Fe("FMbZz3CmOFDV"));
request.setRequestHeader(_$Fe("yACeXwDYXwcTV8Ur2"), _$Fe("g3UraMD2O3UpNMCgB8cT6w6QzRbenM1TTQbS2MbJBRDY9"));
}
request.send($_9);
if ($_12 != null) {
$_12.reset();
}
}
yes as you can see it just creates a XMLHTTP request which (for the links in question) Posts the $_10 content to the server and get the results in callBackC function which is now in $_8. but the trick here is the $_10 contents goes through ~13000 lines of code to create links like this:
http://app1.nmpa.gov.cn/data_nmpa/face3/content.jsp?6SQk6G2z=GBK-56.it.xmhx8IaDT25ZyaSxljrwULe8AkNw8QjmeNqdT0YqZYbMZ2P6Jgn3ZUIgh3ibPI81bjA6xUCKJmzy1LD.4AZnk4g4G_iMO4tdiebiVDoPPtdVDIkDWw0OnDHek.d_2r.PfBtuIoxDvrbGDL.Lv2AuD6lxiObz_lldDHq6HnEw_irAP1hCH.Dr3KdW33DN2w0X1R75N3f8GXdHinmxXLtYbZNYZEE9K7lk9AGmBWgcTds.XgGVW3gDS5OEwoRat44Ecke8k7ZXoY_2revEbUrD8UpOrGprlPEwVYuAvLoTSZX8WJEWQ_QT2CDjNw0FOwAECzsFJa4hGgUtjCPzG&c1SoYK0a=GBK-4aeKAo74EouxLY.stFwdwvXQQG_hXMGG8gB0Hhe6V2Il9k9c8yiTLqduIXpv2RNt.H.weYXeF5XhV0CR2lATieRmk.cs8.fPhNpfGx7JkG1uacp75kDcmXsNtuKgbzRUHZh8vkj4UEYbPcwIYIOw5gFG_cMi9n1GYq0AXXK9UQn9IsmjCBuI7AOFw.pk91OgjvkJCcg2y0y3yDkGwZPcg5EktfAXi.PjmfaecWg8hodU87q6B3ZuPxhel9K9I3EDBxzCHtZqt_0YFlkJCcK4hLq
the problem is with obfuscation and also the nested variables and functions that can keep you out of track for hours if you try to debug it line by line (which I did) and the code makes the characters after content.jsp? part one by one and that explains why its about 13000 lines!!
this part request.send($_9); should have a body for request because its a POST request and $_9 was always null! it seems there are more protection levels to it as it seems.
callBackC:
well the callbackC is apparently a simple function to get responseText and show it to user:
function callbackC() {
if (request.readyState == 1) {
_$c2(document.getElementById(_$Fe("Y3CeXwDYXwq")), '=', _$Fe('vFKyXRUxEYlTW'), _$Fe("kHDxnHOaB3vE5HDxnHOSNMKQGQ6xOHK2z3Kw2Qne7MCm9FKyvhbwNROg"));
}
if (request.readyState == 4) {
if (request.status == 200) {
oldContent[oldContent.length] = request.responseText;
_$c2(document.getElementById(_$Fe("b3CeXwDYXwq")), '=', _$Fe('EFKyXRUxEYlTW'), request.responseText);
request = null;
} else {
_$c2(document.getElementById(_$Fe("u3CeXwDYXwq")), '=', _$Fe('BFKyXRUxEYlTW'), "<br><br><br><span style=font-size:x-large;color:#215add>服务器未返回数据</span>");
}
}
}
I didn't quiet get what those _$Xx functions do (because it goes so deep that its out of my patient!) but it seems they simply replaced the document.getElementById("someThing").innerText="Contents"; with multi layered functions so we can't understand the code easily, and the request.responseText is what you need which is HTML code for the table of results.
there is also a 3rd way which I don't know if you can implement it in your code, but since these functions are in a public scope you can simply override them by redefining these two functions (or replace the functions in the link with your own functions and run them). I tried to get the URL for the request which gave me the link I used in middle of this post, but it didn't worked (I just override the callBackC function and get request.responseURL) and the link gave me 404 error.
I don't think I said all I got from my observations but I think it's enough for you to know what you are up against if you are not already aware, and I hope I was helpful.
Reference:
XMLHttpRequest: Living Standard — Last Updated 16 August 2021

addPreSearch filter not applying

I am trying to use the addPreSearch function to add a custom filter to a lookup field, but the function does not seem to execute fully before the results of the lookup are displayed. The code for this looks something like this:
function onFieldChange(executionContext) {
var formContext = executionContext.getFormContext();
formContext.getControl("test_code").removePreSearch(testFunctionFilter);
formContext.getControl("test_code").addPreSearch(testFunctionFilter);
}
function testFunctionFilter(executionContext) {
var formContext = executionContext.getFormContext();
var record1 = formContext.getAttribute("test_record1_link").getValue(); //get linked record
var record1FullId, record1Id, stringRecordId, idLength, record1Guid = "0";
if (record1 != null) {
record1Id = record1[0].id;
record1Id = record1FullId.slice(1, -1);
stringRecordId = record1FullId.toString();
idLength = stringRecordId.length;
//Guid when retrieved from tablet does not have parenthesis on each end
if (idLength == 36) {
record1Guid = record1FullId;
} else {
record1Guid = recordId;
}
}
var fieldValue;
Xrm.WebApi.retrieveRecord("test_record1", record1Guid, "?$select=test_field1")
.then(function(result1) {
fieldValue = result1.test_field;
var options = generateOptions(executionContext, fieldValue); //creates option string using retrieved fieldValue
Xrm.WebApi.retrieveMultipleRecords("test_record2", options)
.then(function(result) {
var codes = getCodes(result2, fieldValue);
filter = generateFilter(codes, record1Guid); //creates custom filter using provided parameters
console.log(filter); //displays filter correctly
formContext.getControl("test_codelookup").addCustomFilter(filter, "test_coderecord"); //not working?
});
});
}
The filter is generated correctly using the functions used above whose definitions aren't shown. That isn't the issue. I've tried creating a separate test function where I hard coded one of the filters that the function above generated, and the lookup displayed the correct results. The testFunctionFilter should run to completion before the results of the lookup are displayed, correct? Because the filter is logged to the console after the results of the lookup appear. Are the nested asynchronous Xrm.WebApi calls somehow causing the issue? I'm not quite sure what is wrong. Please advise.

You are right. Xrm.WebApi calls are always Asynchronous, which is unusable in this case of adding dynamic filter using addCustomFilter.
You have to use XMLHttpRequest and make that call as Synchronous by setting third parameter as false like below:
var req = new XMLHttpRequest();
req.open("GET", Xrm.Utility.getGlobalContext().getClientUrl() +
"/api/data/v9.0/test_record1?$select=test_field1", false);

In order to work around the async delay, I think you're going to have to reorganise your code:
Add a form OnLoad event and execute the query to retrieve test_field1 and cache the results in a parameter
In the OnChange event, remove the presearch filter, re-execute the query to retrieve test_field1 and update the same parameter (from onload)
In testFunctionFilter use the cached results rather than building the presearch filter from scratch

accessing object values that are inside an array

I'm hoping someone can help with what is likely a simple answer - but I'm ready to bash my head against the wall....again.
I have a function which makes a JSON call to an API, and then pushes the results into an array. The function appears to work just fine as my console.log is showing that the array is populated correctly.
I'm struggling with how to access the values of the modified twichResult object (after the function has run), so that I can do 'stuff' with it. e.g. display the value of the 'status' property onscreen etc... I give some examples of what I've tried in the in the large commented out section.
I'd really appreciate some intelligence weighing in on this as I've exhausted my resources. Thanks in advance.
<script type="text/javascript">
$(document).ready(function() {
var twitchResult = {results:[]};
var channel = { logo:"", display_name:"", status:"", url:"" };
var finalUrl = "https://api.twitch.tv/kraken/streams/freecodecamp?callback=?"
getTwitchers (finalUrl, "freecodecamp");
console.log(twitchResult);
// How do I access the individual values in the object TwitchResult?
// I get "undefined" in the console if I try to access the object's property values
// I've tried every way I can think of to get 'into' the returned object :
// console.log(twitchResult.results);
// console.log(twitchResult["results"])
// console.log(twitchResult.results.status)
// console.log(twitchResult[0])
// console.log(twitchResult[0][0])
// etc etc
function getTwitchers (url, item) {
$.getJSON(url, function(data) {
var obj = data.stream;
// Check if the object is not valid using (obj == null) which is shorthand for both null and undefined
if (obj == null) {
if (obj === undefined) {
channel.display_name = item;
channel.status = "closed";
console.log ("this is undefined");
}
else {
channel.display_name = item;
channel.status = "offline";
console.log("this is null");
}
}
else {
channel.logo = obj.channel.logo;
channel.display_name = obj.channel.display_name;
channel.status = obj.channel.status;
channel.url = obj.channel.url;
console.log("valid entry");
}
twitchResult["results"].push(channel);
// twitchResult.results.push(channel);
// console.log(twitchResult);
});
}
});
</script>

$.getJSON is making an ajax-request. You must handle this request from within the request handler. When getTwichers returns, twichResults is not yet set.
There are methods to delay Program execution, until twichResults is done, but You should not think of using them, since they would delay program execution. The idea of ajax is to execute things asynchronously, without disturbing the rest of the execution flow. If the code You want to execute depends on the json, then You should add it to the handle in $.getJSON. Just write a new function (e.g. continue_execution(twichResult)) and invoke it right after twitchResult["results"].push(channel);. Just don't do anything after getTwitchers(...).
By the way: It is a good habit to define functions, before they are used, because it follows the flow the human eye reads the code and there are programming languages, which depend on this style of declaring function.
If this is unclear to You, then add a comment.

creating a drop down list in javascript from a text file

Is this possible? If so, I could really use some help on this. I'm very new to JavaScript and thus hardly know any of the syntactical specifications, nor proper methodology.
Some functions I wrote in an external file.
var base = base || {};
base.requestAjax = function () {
try{
return new XMLHttpRequest();
} catch (e) {
try {
return new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
return new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) {
alert("Something is wrong with your browser! Please ensure that you have javascript functionality enabled");
return false;
}
}
}
}
base.onReadyStateChange = function(ajaxRequest, formName, formArray, method, controller) {
var methodVerified = verifyMethod(method);
if (!methodVerified) {
throw new Exception("Error: please make sure the method passed matches that of \'GET\' or \'POST\'.");
}
for (input in formArray) {
document.formName.input.value = ajaxRequest.responseText;
}
ajaxRequest.open(method, controller, true);
file.send(null);
}
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
if (file.readyState != 4) {
throw new Exception("Error: text file ready state not equal to 4!");
}
if (file.status != 200) {
throw new Exception("Error: text file status not equal to 200!");
}
var lines = file.responseText.split("\n");
document.writeln("<select>");
for(line in lines) {
document.writeln("<option>" + line + "</option>");
}
document.writeln("</select>");
file.send(null);
}
base.verifyMethod = function(method) {
var methodLower = method.toString().toLowerCase();
switch(methodLower) {
case "get":
case "post":
return true;
default:
return false;
}
}
The html code which attempts to implement it
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" type="text/css" href="../includes/css/adsmanager.css" />
<script type="text/javascript" src="../includes/js/base.js">
<!--
function createCountriesList() {
var file = base.requestAjax;
var path = "../includes/_notes/countries.txt";
base.writeDropDownList(file, "GET", path);
}
//-->
</script>
</head>
<body>
<form name='adsManager'>
<button type="text" value="test" onclick="createCountriesList()" />
</form>
</body>
</html>
I plan to use the functions for other things, so I figured I'd create a namespace. I found this as a reference and based most of my model off of it.
Thanks to all who can help.

What is your point of failure? Is your Javascript sending the Ajax request and receiving the response?
Is lines getting data in this line of your code?
var lines = file.responseText.split("\n");
If you are getting this far, iterate through lines and add options like this:
var select = document.getElementById('id');
select.options.add(new Option(lines[i]));
In your writeDropDownList method, I made a few changes:
added a method that will be called when your Ajax call has completed. In that method, you should check the response string and add the options to your select box
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
file.onreadystatechange = requestComplete;
file.send(null);
}
requestComplete()
{
if (file.readyState == 4)
{
if(file.readyState == 200)
{
var lines = file.responseText.split("\n");
//foreach loop to populate select
}
}
}
In your code, you are checking and using files.responseText before you have even sent the Ajax request at files.send(null)
EDIT:
Some more comments regarding your code:
In the createCountriesList function, you create file and assign
it a value by calling requestAjax. You then pass it to
writeDropDownList function, where you assign it a value again by
calling requestAjax. You see that this is redundant? There is no
need to create file in createCountriesList and pass it as an
argument. Create it just once in writeDropDownList.
in writeDropDownList you call upload.requestAjax(). What is
upload. I don't see you initializing upload anywhere. Do you mean
to call base.requestAjax()?
you have a function base.OnReadyStateChange but at no point are you
telling your AJAX request to call that function when state changes. See the code I posted above. The function I added called
requestComplete does that, and I tell the AJAX request to call it using file.onreadystatechange = requestComplete;
You set method to GET, yet you are not passing any GET values in your URL
in file.open(method, path, true);, path is supposed to be the URL of the script the AJAX request will call. You have set path to ../includes/_notes/countries.txt. An AJAX request cannot call a .txt file since they do not execute.
I just had a look at the source of your code, and it is all sorts of broken. Please do not use it.
What is countries.txt ? Are you attempting to populate a dropdown with a list of all countries, or some countries depending on user's input?
If the former, there is no need for Javascript / AJAX. You need to add PHP code in your html to populate the select box.
If the latter, your AJAX request should be sending the user input as a GET variable.

Some comments:
Very nice code; it's readable and looks neat.
I'd use a different name than base as namespace - for my liking, the word is too "common" . Chances are that you're going to define a variable base somewhere and it will break.
To create HTML from JavaScript, first create a small example in pure HTML to see how it should look like. After that, create a script which produces the same HTML using document.createElement().
Look at frameworks like jQuery or Prototype because they make many boring tasks much more simple.

Trying to call JS Code that is passed back from AJAX call

Okay, so I have an javascript function that retrieves some HTML...
function updateQuestions(i){
var url = 'getQuestions.php?sys=' + i;
if (receiveReq.readyState == 4 || receiveReq.readyState == 0) {
receiveReq.open("GET", url, true);
receiveReq.onreadystatechange = handleQuestionsUpdate;
receiveReq.send(null);
}
}
function handleQuestionsUpdate() {
if (receiveReq.readyState == 4) {
var a=receiveReq.responseText;
document.getElementById('questions').innerHTML=a;
checkSpeakers(); //Error Occurs Here, even though checkSpeakers() is a function in the returned HTML chunk.
}
}
This HTML is not just HTML, but it is more specifically a form and a chunk of javascript. The javascript is hard-coded into the HTML and not referenced by <script src="..">
Is it normal that this retrieved JS code isn't recognized upon call-time? If so, what is my alternative if I need the JS to change every time the div is update?
This is the text being returned to the javascript function.
function checkPillowSpeakers()
{
var pillowSpeakers = document.getElementById('Q9').value + document.getElementById('Q10').value;
document.getElementById('PS1').style.display = ((pillowSpeakers > 0)? '' : 'none');
document.getElementById('PS2').style.display = ((pillowSpeakers > 0)? '' : 'none');
}~ARRAYSEPERATOR~<html>....</html>
The JS Code is seperated from the HTML code by an ~ARRAYSEPERATOR~ tag. The issue is that I don't want to EXECUTE the code at this time, I just want it queued so I can call it on command.

You should first try to get the JavaScript part from the HTML content.
Then you can easily execute it using eval() function from JavaScript;

My answer from How To Call Javascript In Ajax Response? IE: Close a form div upon success
// response is the data returned from the server
var response = "html\<script type=\"text/javascript\">alert(\"foo\");<\/script>html";
var reScript = /\<script.*?>(.*)<\/script>/mg;
response = response.replace(reScript, function(m,m1) {
var fn = new Function(m1); // this will make the eval run in the global scope
fn(); //will run alert("foo");
return "";
});
alert(response); // will alert "htmlhtml"

After doing a whole lot of research, it seems Eval() has some memory issues... so I actually came across this solution:
//Firstly, create a div called codeHolder
var javascriptCode="function test(){.....}";
var JSONCode=document.createElement("script");
JSONCode.setAttribute("type","text/javascript");
JSONCode.text=javascriptCode;
var cell=document.getElementById("codeHolder");
if ( cell.hasChildNodes() )
while ( cell.childNodes.length >= 1 )
cell.removeChild( cell.firstChild );
cell.appendChild(JSONCode);

you should realy think of using an js-lib for ajax for browser-compatibilty and less memory leaks - but if you want to do this by yourself, you have to eval() the passed back javascript before you can use it.

There is also responseXML:
receiveReq.responseXML.getElementsByTagName('input')
etc., etc.