We Keep Coding

Scrapy Splash, How to deal with onclick?

I'm trying to scrape the following site
I'm able to receive a response but i don't know how can i access the inner data of the below items in order to scrape it:
I noticed that accessing the items is actually handled by JavaScript and also the pagination.
What should i do in such case?
Below is my code:
import scrapy
from scrapy_splash import SplashRequest
class NmpaSpider(scrapy.Spider):
name = 'nmpa'
http_user = 'hidden' # as am using Cloud Splash
allowed_domains = ['nmpa.gov.cn']
def start_requests(self):
yield SplashRequest('http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=27&tableName=TABLE27&title=%E8%BF%9B%E5%8F%A3%E5%8C%BB%E7%96%97%E5%99%A8%E6%A2%B0%E4%BA%A7%E5%93%81%EF%BC%88%E6%B3%A8%E5%86%8C&bcId=152904442584853439006654836900', args={
'wait': 5}
)
def parse(self, response):
goal = response.xpath("//*[#id='content']//a/#href").getall()
print(goal)

If you use some breakpoints you'll see its a frustrating job that I explain what I understood from my research.
when you are working with this kind of situation you have two ways:
1 - [Easy Way] use selenium and open a browser and click on each link and get the returned contents easily, you can run multiple browsers and get link contents simultaneously.
2 - [Hard Way] simulate what website does (by making similar functions inside python) and do exactly what website does in JS but in the end instead of showing the results, just save it in a variable and use it the way you want.
Now if you choose the HARD WAY this is what I found:
the link JS is like this:
commitForECMA(callbackC,'content.jsp?tableId=26&tableName=TABLE26&tableView=国产医疗器械产品（注册&Id=138150',null)
it calls a function named commitForECMA and get what this function returns and pass it to callBackC function.
well this was obvious, but its important to know what these functions do and how to replicate it.
commitForECMA:
this is the function:
function commitForECMA($_8, $_10, $_12) {
request = createXMLHttp();
request.onreadystatechange = $_8;
if ($_12 == null) {
_$du(request, _$Fe('uM6r2MG'), _$Fe("jp0YV"), $_10);
request.setRequestHeader(_$Fe("XACeXwDYXwcTV8Ur2"), _$Fe("YwDYgwceLwDT7iCYX3Ce9FKyvHKwPFa"));
} else {
var $_9 = "";
var $_19 = $_12.elements;
var $_0 = $_19.length;
for (var $_18 = 0; $_18 < $_0; $_18++) {
var $_14 = _$3P($_19, $_18);
if ($_14.type != _$Fe("yQ6YPMK20") && _$3P($_14, _$Fe('uwbm7wKV')) != "") {
if ($_9.length > 0) {
$_9 += "&" + $_14.name + "=" + _$3P($_14, _$Fe('kwbm7wKV'));
} else {
$_9 += $_14.name + "=" + _$3P($_14, _$Fe('Ewbm7wKV'));
}
$_9 += _$Fe("Jx2J03Up2Hsl");
}
}
_$du(request, _$Fe('uM6r2MG'), _$Fe("HVlesYq"), $_10);
$_9 = encodeURI($_9);
$_9 = encodeURI($_9);
request.setRequestHeader(_$Fe("d3CmOFDVz3CeXwoxBMq"), _$Fe("FMbZz3CmOFDV"));
request.setRequestHeader(_$Fe("yACeXwDYXwcTV8Ur2"), _$Fe("g3UraMD2O3UpNMCgB8cT6w6QzRbenM1TTQbS2MbJBRDY9"));
}
request.send($_9);
if ($_12 != null) {
$_12.reset();
}
}
yes as you can see it just creates a XMLHTTP request which (for the links in question) Posts the $_10 content to the server and get the results in callBackC function which is now in $_8. but the trick here is the $_10 contents goes through ~13000 lines of code to create links like this:
http://app1.nmpa.gov.cn/data_nmpa/face3/content.jsp?6SQk6G2z=GBK-56.it.xmhx8IaDT25ZyaSxljrwULe8AkNw8QjmeNqdT0YqZYbMZ2P6Jgn3ZUIgh3ibPI81bjA6xUCKJmzy1LD.4AZnk4g4G_iMO4tdiebiVDoPPtdVDIkDWw0OnDHek.d_2r.PfBtuIoxDvrbGDL.Lv2AuD6lxiObz_lldDHq6HnEw_irAP1hCH.Dr3KdW33DN2w0X1R75N3f8GXdHinmxXLtYbZNYZEE9K7lk9AGmBWgcTds.XgGVW3gDS5OEwoRat44Ecke8k7ZXoY_2revEbUrD8UpOrGprlPEwVYuAvLoTSZX8WJEWQ_QT2CDjNw0FOwAECzsFJa4hGgUtjCPzG&c1SoYK0a=GBK-4aeKAo74EouxLY.stFwdwvXQQG_hXMGG8gB0Hhe6V2Il9k9c8yiTLqduIXpv2RNt.H.weYXeF5XhV0CR2lATieRmk.cs8.fPhNpfGx7JkG1uacp75kDcmXsNtuKgbzRUHZh8vkj4UEYbPcwIYIOw5gFG_cMi9n1GYq0AXXK9UQn9IsmjCBuI7AOFw.pk91OgjvkJCcg2y0y3yDkGwZPcg5EktfAXi.PjmfaecWg8hodU87q6B3ZuPxhel9K9I3EDBxzCHtZqt_0YFlkJCcK4hLq
the problem is with obfuscation and also the nested variables and functions that can keep you out of track for hours if you try to debug it line by line (which I did) and the code makes the characters after content.jsp? part one by one and that explains why its about 13000 lines!!
this part request.send($_9); should have a body for request because its a POST request and $_9 was always null! it seems there are more protection levels to it as it seems.
callBackC:
well the callbackC is apparently a simple function to get responseText and show it to user:
function callbackC() {
if (request.readyState == 1) {
_$c2(document.getElementById(_$Fe("Y3CeXwDYXwq")), '=', _$Fe('vFKyXRUxEYlTW'), _$Fe("kHDxnHOaB3vE5HDxnHOSNMKQGQ6xOHK2z3Kw2Qne7MCm9FKyvhbwNROg"));
}
if (request.readyState == 4) {
if (request.status == 200) {
oldContent[oldContent.length] = request.responseText;
_$c2(document.getElementById(_$Fe("b3CeXwDYXwq")), '=', _$Fe('EFKyXRUxEYlTW'), request.responseText);
request = null;
} else {
_$c2(document.getElementById(_$Fe("u3CeXwDYXwq")), '=', _$Fe('BFKyXRUxEYlTW'), "<br><br><br><span style=font-size:x-large;color:#215add>服务器未返回数据</span>");
}
}
}
I didn't quiet get what those _$Xx functions do (because it goes so deep that its out of my patient!) but it seems they simply replaced the document.getElementById("someThing").innerText="Contents"; with multi layered functions so we can't understand the code easily, and the request.responseText is what you need which is HTML code for the table of results.
there is also a 3rd way which I don't know if you can implement it in your code, but since these functions are in a public scope you can simply override them by redefining these two functions (or replace the functions in the link with your own functions and run them). I tried to get the URL for the request which gave me the link I used in middle of this post, but it didn't worked (I just override the callBackC function and get request.responseURL) and the link gave me 404 error.
I don't think I said all I got from my observations but I think it's enough for you to know what you are up against if you are not already aware, and I hope I was helpful.
Reference:
XMLHttpRequest: Living Standard — Last Updated 16 August 2021

How to slow down an Ajax call?

I have a function in JS contains a loop, that calls an AJAX call every iteration. The call to inserts checked elements into a DB and returns the results of those elements in the same page in the next section.
The problem I have is that when I check for e.g. 4 checkboxes out of 3 groupes, the only checkboxes of the last group gets added to the page. However, when I use alert(), I can see all elements.
I used setTimeout, but I got error in the code. I also added lines to give more time to AJX call, but the problem remains. So I wonder if there is a solution to slow down the code without using alert().
This is my script:
addAptitudeField : function(currentAutocompleteField, idChamp) {
var currentAutocompleteFieldBind = currentAutocompleteField;
var idChampBind = idChamp;
window.setTimeout(function() {
// Code ...
var paramDwr = {};
var newDivName = "div" + idChamp + lastValueId;
paramDwr[attributs.r_divId] = newDivName;
paramDwr[attributs.r_currentValue] = currentValue;
paramDwr[attributs.r_hiddenIdsField] = hiddenIdsField.id;
paramDwr[attributs.r_lastValueId] = lastValueId;
paramDwr[attributs.r_itemmod] = nbAptitudesCat % 2 == 0;
// setTimeout ( RepertoireDwr.ligneSuppEtSpanMessage, 1000 ) doesn't work
RepertoireDwr.ligneSuppEtSpanMessage(paramDwr, function(ajaxPage) {
divCategorie.update(divCategorie.innerHTML + ajaxPage.texte);
aptitudeAvecDetail.remetsValeursStockees();
var btnSuppression = $(newDivName).getElementsByTagName('img')[0];
btnSuppression.setAttribute("onclick", "formulaireFiche.updateCSS('" + newDivName + "');" + btnSuppression.getAttribute("onclick") + "fiche.updateCategorieSuppressionAptLieeUo(\'divCat" + currentCategorie + "\');"); });
}
//
// alert() : It works in this case.
//
// for (var i=0; i<5000000; i++) ; it doesn't work
}, 400);
}
Thank you in advance for your help and time.

I will likely be downvoted for mentioning this, because it is not a recommended procedure, but I believe every coder should have all facts.
In jQuery AJAX construct, there is option async:false, which will delay the script from continuing UNTIL the AJAX has completed processing. Needless to say, if things go wrong in the AJAX the browser could freeze. A lot depends on who your users are, and amount of traffic -- on a few of my ten-user in-house projects it was an acceptable solution.
$.ajax({
async: false,
type: 'post',
url: 'ajax/ax.php',
data: 'request=',
success: function(d){
if (d.length) alert(d);
}
});
Ref:
What does "async: false" do in jQuery.ajax()?
The better idea, however, is to look into the Promises interface, with methods like .when() and .then()
References:
https://jsfiddle.net/v86bc028/2/
http://jqfundamentals.com/chapter/ajax-deferreds#
http://digitizor.com/jquery-html-callback-function-using-promise/#
how does jquery's promise method really work?

The problem you're running into deals with asynchronous functions, or the A in AJAX. If you don't know what an asynchronous function is, there are many others who can explain it better than I can, so give that a google.
What's happening without the alert() in there is your code makes 4 sever calls, but all 4 get sent out before you get a response to any of them. With the alert() (or setTimeout), you're giving the code time to received each response to a call before the next one is made.
There are several ways you can approach this, the first way is by calling the next call after the first receives a response. The second way is to use an async function to call all 4 at once on different chains(?). I'm not the best at explaining this part, but there's plenty of code to be found on SO and online.

I think you have a more generic problem in your code, since you seem to need to delay your executions to wait till sth. else is finished, instead of getting anounced when it is done.
The line that annoys me most is this one
divCategorie.update(divCategorie.innerHTML + ajaxPage.texte);
what exactly is update doing? How is it implemented?
I assume it does sth. like divCategorie.innerHTML += ajaxPage.texte;
Wich is highly unfavorable, since the browser has to parse and rebuild, whatever there already is in divCategorie.innerHTML.
Just appending the new Markup would be better.
long way short: maybe a good hack would be to insert some hidden node as a placeholder (so you kan keep order, although the AJAX-requests may return in a different order) and replace that node with the real content, as soon as it arrives.
Kind of like this:
addAptitudeField : function(currentAutocompleteField, idChamp) {
var currentAutocompleteFieldBind = currentAutocompleteField;
var idChampBind = idChamp;
//this is done immediately, and therefore preserves the order of the loop,
//without any delays/timeouts
var placeholder = document.createElement("div");
placeholder.className = "placeholder";
placeholder.style.display = "none";
divCategorie.appendChild(placeholder);
window.setTimeout(function() {
// Code ...
var paramDwr = {};
var newDivName = "div" + idChamp + lastValueId;
paramDwr[attributs.r_divId] = newDivName;
paramDwr[attributs.r_currentValue] = currentValue;
paramDwr[attributs.r_hiddenIdsField] = hiddenIdsField.id;
paramDwr[attributs.r_lastValueId] = lastValueId;
paramDwr[attributs.r_itemmod] = nbAptitudesCat % 2 == 0;
// setTimeout ( RepertoireDwr.ligneSuppEtSpanMessage, 1000 ) doesn't work
RepertoireDwr.ligneSuppEtSpanMessage(paramDwr, function(ajaxPage) {
//convert the passed text into a DocumentFragment
var frag = fragment(ajaxPage.texte);
//replacing the placeholder with the fragment
divCategorie.insertBefore(frag, placeholder);
divCategorie.removeChild(placeholder);
aptitudeAvecDetail.remetsValeursStockees();
var btnSuppression = $(newDivName).getElementsByTagName('img')[0];
//this is also pretty horrible to me:
btnSuppression.setAttribute("onclick", "formulaireFiche.updateCSS('" + newDivName + "');" + btnSuppression.getAttribute("onclick") + "fiche.updateCategorieSuppressionAptLieeUo(\'divCat" + currentCategorie + "\');"); });
}
}, 400);
}
I think you should do some major refactoring. And take a look into Promises.
// * -> DocumentFragment
//strings/primitives are parsed as HTML-markup,
//null / undefined is ignored
//Arraylike structures are parsed recursively
var fragment = (function(container){
return function(src){
return reducer(document.createDocumentFragment(), src);
}
function reducer(frag, node){
var i, len, fc, c, r;
if(node === Object(node)){
if("nodeType" in node){
//dom nodes
frag.appendChild(node);
}else{
//Arraylike structures, like NodeLists or jQuery-Objects, or just plain Arrays
for(i = 0, len = ("length" in node && node.length)|0, r = reducer; i < len; (i in node) && r(frag, node[i]));
}
}else if(node != null) {
//strings (all primitives)
for((c=container).innerHTML = node; fc = c.firstChild; frag.appendChild(fc));
}
return frag;
}
})(document.createElement("div"));

accessing object values that are inside an array

I'm hoping someone can help with what is likely a simple answer - but I'm ready to bash my head against the wall....again.
I have a function which makes a JSON call to an API, and then pushes the results into an array. The function appears to work just fine as my console.log is showing that the array is populated correctly.
I'm struggling with how to access the values of the modified twichResult object (after the function has run), so that I can do 'stuff' with it. e.g. display the value of the 'status' property onscreen etc... I give some examples of what I've tried in the in the large commented out section.
I'd really appreciate some intelligence weighing in on this as I've exhausted my resources. Thanks in advance.
<script type="text/javascript">
$(document).ready(function() {
var twitchResult = {results:[]};
var channel = { logo:"", display_name:"", status:"", url:"" };
var finalUrl = "https://api.twitch.tv/kraken/streams/freecodecamp?callback=?"
getTwitchers (finalUrl, "freecodecamp");
console.log(twitchResult);
// How do I access the individual values in the object TwitchResult?
// I get "undefined" in the console if I try to access the object's property values
// I've tried every way I can think of to get 'into' the returned object :
// console.log(twitchResult.results);
// console.log(twitchResult["results"])
// console.log(twitchResult.results.status)
// console.log(twitchResult[0])
// console.log(twitchResult[0][0])
// etc etc
function getTwitchers (url, item) {
$.getJSON(url, function(data) {
var obj = data.stream;
// Check if the object is not valid using (obj == null) which is shorthand for both null and undefined
if (obj == null) {
if (obj === undefined) {
channel.display_name = item;
channel.status = "closed";
console.log ("this is undefined");
}
else {
channel.display_name = item;
channel.status = "offline";
console.log("this is null");
}
}
else {
channel.logo = obj.channel.logo;
channel.display_name = obj.channel.display_name;
channel.status = obj.channel.status;
channel.url = obj.channel.url;
console.log("valid entry");
}
twitchResult["results"].push(channel);
// twitchResult.results.push(channel);
// console.log(twitchResult);
});
}
});
</script>

$.getJSON is making an ajax-request. You must handle this request from within the request handler. When getTwichers returns, twichResults is not yet set.
There are methods to delay Program execution, until twichResults is done, but You should not think of using them, since they would delay program execution. The idea of ajax is to execute things asynchronously, without disturbing the rest of the execution flow. If the code You want to execute depends on the json, then You should add it to the handle in $.getJSON. Just write a new function (e.g. continue_execution(twichResult)) and invoke it right after twitchResult["results"].push(channel);. Just don't do anything after getTwitchers(...).
By the way: It is a good habit to define functions, before they are used, because it follows the flow the human eye reads the code and there are programming languages, which depend on this style of declaring function.
If this is unclear to You, then add a comment.

How to return data from CRM 2011 javascript odata function?

This seems silly that I haven't been able to accomplish this. I'm using the common code found on the web to do an odata query. The problem is the results stay in getFieldData(retrieveReq) routine. I don't want to immediately set a field on the current form. How can I get my values out of it so the data can be used in other javascript functions? Global variable would be good but nothing I've tried has worked. The below code displays "x".
var var1 = "x"; odataquery(); alert(var1);
The example given here has two alerts that display the data. How can Id and Name get outside of that function to be useful?
Edit1: Below is the main part of the routine that calls getFieldData(this). I want to use OwnerBUID and OwnerBUName in other javascript functions.
var retrieveReq = new XMLHttpRequest();
retrieveReq.open("GET", odataSelect, false);
retrieveReq.setRequestHeader("Accept", "application/json");
retrieveReq.setRequestHeader("Content-Type", "application/json; charset=utf-8");
retrieveReq.onreadystatechange = function () {
getFieldData(this);
};
retrieveReq.send();
function getFieldData(retrieveReq) {
if (retrieveReq.readyState == 4 && retrieveReq.status == 200) {
// 4=request complete, 200=OK
var retrieved = this.parent.JSON.parse(retrieveReq.responseText).d;
var retrievedValue = retrieved.results[0].BusinessUnitId;
OwnerBUID = retrievedValue.Id;
OwnerBUName = retrievedValue.Name;
}
}

I guess you want to put the data as the return value of a javascript function. You could do this:
var returnedData = function getFieldData(retrieveReq)
{
...
return data;
}
BTW, you could consider to use JayData, Breeze and Datajs sources code packages in your client application. They implement the low level APIs for consuming an odata service using javascript.

creating a drop down list in javascript from a text file

Is this possible? If so, I could really use some help on this. I'm very new to JavaScript and thus hardly know any of the syntactical specifications, nor proper methodology.
Some functions I wrote in an external file.
var base = base || {};
base.requestAjax = function () {
try{
return new XMLHttpRequest();
} catch (e) {
try {
return new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
return new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) {
alert("Something is wrong with your browser! Please ensure that you have javascript functionality enabled");
return false;
}
}
}
}
base.onReadyStateChange = function(ajaxRequest, formName, formArray, method, controller) {
var methodVerified = verifyMethod(method);
if (!methodVerified) {
throw new Exception("Error: please make sure the method passed matches that of \'GET\' or \'POST\'.");
}
for (input in formArray) {
document.formName.input.value = ajaxRequest.responseText;
}
ajaxRequest.open(method, controller, true);
file.send(null);
}
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
if (file.readyState != 4) {
throw new Exception("Error: text file ready state not equal to 4!");
}
if (file.status != 200) {
throw new Exception("Error: text file status not equal to 200!");
}
var lines = file.responseText.split("\n");
document.writeln("<select>");
for(line in lines) {
document.writeln("<option>" + line + "</option>");
}
document.writeln("</select>");
file.send(null);
}
base.verifyMethod = function(method) {
var methodLower = method.toString().toLowerCase();
switch(methodLower) {
case "get":
case "post":
return true;
default:
return false;
}
}
The html code which attempts to implement it
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" type="text/css" href="../includes/css/adsmanager.css" />
<script type="text/javascript" src="../includes/js/base.js">
<!--
function createCountriesList() {
var file = base.requestAjax;
var path = "../includes/_notes/countries.txt";
base.writeDropDownList(file, "GET", path);
}
//-->
</script>
</head>
<body>
<form name='adsManager'>
<button type="text" value="test" onclick="createCountriesList()" />
</form>
</body>
</html>
I plan to use the functions for other things, so I figured I'd create a namespace. I found this as a reference and based most of my model off of it.
Thanks to all who can help.

What is your point of failure? Is your Javascript sending the Ajax request and receiving the response?
Is lines getting data in this line of your code?
var lines = file.responseText.split("\n");
If you are getting this far, iterate through lines and add options like this:
var select = document.getElementById('id');
select.options.add(new Option(lines[i]));
In your writeDropDownList method, I made a few changes:
added a method that will be called when your Ajax call has completed. In that method, you should check the response string and add the options to your select box
base.writeDropDownList = function(file, method, path) {
var file = upload.requestAjax();
file.open(method, path, true);
file.onreadystatechange = requestComplete;
file.send(null);
}
requestComplete()
{
if (file.readyState == 4)
{
if(file.readyState == 200)
{
var lines = file.responseText.split("\n");
//foreach loop to populate select
}
}
}
In your code, you are checking and using files.responseText before you have even sent the Ajax request at files.send(null)
EDIT:
Some more comments regarding your code:
In the createCountriesList function, you create file and assign
it a value by calling requestAjax. You then pass it to
writeDropDownList function, where you assign it a value again by
calling requestAjax. You see that this is redundant? There is no
need to create file in createCountriesList and pass it as an
argument. Create it just once in writeDropDownList.
in writeDropDownList you call upload.requestAjax(). What is
upload. I don't see you initializing upload anywhere. Do you mean
to call base.requestAjax()?
you have a function base.OnReadyStateChange but at no point are you
telling your AJAX request to call that function when state changes. See the code I posted above. The function I added called
requestComplete does that, and I tell the AJAX request to call it using file.onreadystatechange = requestComplete;
You set method to GET, yet you are not passing any GET values in your URL
in file.open(method, path, true);, path is supposed to be the URL of the script the AJAX request will call. You have set path to ../includes/_notes/countries.txt. An AJAX request cannot call a .txt file since they do not execute.
I just had a look at the source of your code, and it is all sorts of broken. Please do not use it.
What is countries.txt ? Are you attempting to populate a dropdown with a list of all countries, or some countries depending on user's input?
If the former, there is no need for Javascript / AJAX. You need to add PHP code in your html to populate the select box.
If the latter, your AJAX request should be sending the user input as a GET variable.

Some comments:
Very nice code; it's readable and looks neat.
I'd use a different name than base as namespace - for my liking, the word is too "common" . Chances are that you're going to define a variable base somewhere and it will break.
To create HTML from JavaScript, first create a small example in pure HTML to see how it should look like. After that, create a script which produces the same HTML using document.createElement().
Look at frameworks like jQuery or Prototype because they make many boring tasks much more simple.