We Keep Coding

Set cookies using js-cookies to array

newbie here regarding Javascript. I am following this thread to set cookies to array by clicking button. Product compare session. Its working but the problem is, when i reload or open new page, when i click the button on new page or refreshed page, the cookies doesn't add new value, it replace all cookies which has been set from previous page. Here is the script.
`
cookie_data_load = Cookies.get('compare_data');
$('.view__compare').attr("href", "https://shop.local/compare/?id=" + cookie_data_load);
var fieldArray = [];
$( ".product__actions-item--compare" ).click(function(){
fieldArray.push($(this).data("compare"));
var unique=fieldArray.filter(function(itm,i){
return i==fieldArray.indexOf(itm);
});
var str = unique.join('-');
Cookies.set('compare_data', str, { expires: 7, path: '/' });
cookie_data = Cookies.get('compare_data');
console.log(str);
console.log(unique);
alert(unique);
$('.view__compare').attr("href", "https://shop.local/compare/?id=" + cookie_data);
return false;
});
`
And second question is how to limit the number of cookies value (array) from above code? Many thanks
I have read the js-cookies github but cant understand single thing.
*** Updated code from https://stackoverflow.com/users/8422082/uladzimir
`
var fieldArray = (Cookies.get('compare_data') || '').split('-');
$(".product__actions-item--compare").click(function () { if
(fieldArray.length >= 3) {
alert("unfortunately limit exceeded :("); } else {
fieldArray.push($(this).data("compare"));
var unique = fieldArray.filter(function (itm, i) {
return i == fieldArray.indexOf(itm);
});
var str = unique.join('-');
Cookies.set("compare_data", str, { expires: 7, path: "/" });
cookie_data = Cookies.get("compare_data");
console.log(str);
console.log(unique);
alert(unique);
$(".view__compare").attr(
"href",
"https://shop.local/compare/?id=" + cookie_data
);
return false; } });
`

Ivan, whenever you reload a page, the array of data "fieldArray" is ALWAYS empty (despite there is data in "compare_data" cookie from previous browser session)
What you have to do is to initialize "fieldArray" with it's initial value taken from cookie:
var fieldArray = (Cookies.get('compare_data') || '').split('-')
Cookie stores string data with maximum size of 4kb. More over, cookie have no idea, if it stores serialized array, object, or anything else... It just keeps a string of text and that's it. So (as far as I know), there is no way to limit array length using cookie settings.
So, the only workaround here is to do this length-check programmatically, like following:
$('.product__actions-item--compare').click(function () {
if (fieldArray.length >= 3) {
alert('unfortunately limit exceeded :(');
} else {
// do your actions
}
});

Scrapy Splash, How to deal with onclick?

I'm trying to scrape the following site
I'm able to receive a response but i don't know how can i access the inner data of the below items in order to scrape it:
I noticed that accessing the items is actually handled by JavaScript and also the pagination.
What should i do in such case?
Below is my code:
import scrapy
from scrapy_splash import SplashRequest
class NmpaSpider(scrapy.Spider):
name = 'nmpa'
http_user = 'hidden' # as am using Cloud Splash
allowed_domains = ['nmpa.gov.cn']
def start_requests(self):
yield SplashRequest('http://app1.nmpa.gov.cn/data_nmpa/face3/base.jsp?tableId=27&tableName=TABLE27&title=%E8%BF%9B%E5%8F%A3%E5%8C%BB%E7%96%97%E5%99%A8%E6%A2%B0%E4%BA%A7%E5%93%81%EF%BC%88%E6%B3%A8%E5%86%8C&bcId=152904442584853439006654836900', args={
'wait': 5}
)
def parse(self, response):
goal = response.xpath("//*[#id='content']//a/#href").getall()
print(goal)

If you use some breakpoints you'll see its a frustrating job that I explain what I understood from my research.
when you are working with this kind of situation you have two ways:
1 - [Easy Way] use selenium and open a browser and click on each link and get the returned contents easily, you can run multiple browsers and get link contents simultaneously.
2 - [Hard Way] simulate what website does (by making similar functions inside python) and do exactly what website does in JS but in the end instead of showing the results, just save it in a variable and use it the way you want.
Now if you choose the HARD WAY this is what I found:
the link JS is like this:
commitForECMA(callbackC,'content.jsp?tableId=26&tableName=TABLE26&tableView=国产医疗器械产品（注册&Id=138150',null)
it calls a function named commitForECMA and get what this function returns and pass it to callBackC function.
well this was obvious, but its important to know what these functions do and how to replicate it.
commitForECMA:
this is the function:
function commitForECMA($_8, $_10, $_12) {
request = createXMLHttp();
request.onreadystatechange = $_8;
if ($_12 == null) {
_$du(request, _$Fe('uM6r2MG'), _$Fe("jp0YV"), $_10);
request.setRequestHeader(_$Fe("XACeXwDYXwcTV8Ur2"), _$Fe("YwDYgwceLwDT7iCYX3Ce9FKyvHKwPFa"));
} else {
var $_9 = "";
var $_19 = $_12.elements;
var $_0 = $_19.length;
for (var $_18 = 0; $_18 < $_0; $_18++) {
var $_14 = _$3P($_19, $_18);
if ($_14.type != _$Fe("yQ6YPMK20") && _$3P($_14, _$Fe('uwbm7wKV')) != "") {
if ($_9.length > 0) {
$_9 += "&" + $_14.name + "=" + _$3P($_14, _$Fe('kwbm7wKV'));
} else {
$_9 += $_14.name + "=" + _$3P($_14, _$Fe('Ewbm7wKV'));
}
$_9 += _$Fe("Jx2J03Up2Hsl");
}
}
_$du(request, _$Fe('uM6r2MG'), _$Fe("HVlesYq"), $_10);
$_9 = encodeURI($_9);
$_9 = encodeURI($_9);
request.setRequestHeader(_$Fe("d3CmOFDVz3CeXwoxBMq"), _$Fe("FMbZz3CmOFDV"));
request.setRequestHeader(_$Fe("yACeXwDYXwcTV8Ur2"), _$Fe("g3UraMD2O3UpNMCgB8cT6w6QzRbenM1TTQbS2MbJBRDY9"));
}
request.send($_9);
if ($_12 != null) {
$_12.reset();
}
}
yes as you can see it just creates a XMLHTTP request which (for the links in question) Posts the $_10 content to the server and get the results in callBackC function which is now in $_8. but the trick here is the $_10 contents goes through ~13000 lines of code to create links like this:
http://app1.nmpa.gov.cn/data_nmpa/face3/content.jsp?6SQk6G2z=GBK-56.it.xmhx8IaDT25ZyaSxljrwULe8AkNw8QjmeNqdT0YqZYbMZ2P6Jgn3ZUIgh3ibPI81bjA6xUCKJmzy1LD.4AZnk4g4G_iMO4tdiebiVDoPPtdVDIkDWw0OnDHek.d_2r.PfBtuIoxDvrbGDL.Lv2AuD6lxiObz_lldDHq6HnEw_irAP1hCH.Dr3KdW33DN2w0X1R75N3f8GXdHinmxXLtYbZNYZEE9K7lk9AGmBWgcTds.XgGVW3gDS5OEwoRat44Ecke8k7ZXoY_2revEbUrD8UpOrGprlPEwVYuAvLoTSZX8WJEWQ_QT2CDjNw0FOwAECzsFJa4hGgUtjCPzG&c1SoYK0a=GBK-4aeKAo74EouxLY.stFwdwvXQQG_hXMGG8gB0Hhe6V2Il9k9c8yiTLqduIXpv2RNt.H.weYXeF5XhV0CR2lATieRmk.cs8.fPhNpfGx7JkG1uacp75kDcmXsNtuKgbzRUHZh8vkj4UEYbPcwIYIOw5gFG_cMi9n1GYq0AXXK9UQn9IsmjCBuI7AOFw.pk91OgjvkJCcg2y0y3yDkGwZPcg5EktfAXi.PjmfaecWg8hodU87q6B3ZuPxhel9K9I3EDBxzCHtZqt_0YFlkJCcK4hLq
the problem is with obfuscation and also the nested variables and functions that can keep you out of track for hours if you try to debug it line by line (which I did) and the code makes the characters after content.jsp? part one by one and that explains why its about 13000 lines!!
this part request.send($_9); should have a body for request because its a POST request and $_9 was always null! it seems there are more protection levels to it as it seems.
callBackC:
well the callbackC is apparently a simple function to get responseText and show it to user:
function callbackC() {
if (request.readyState == 1) {
_$c2(document.getElementById(_$Fe("Y3CeXwDYXwq")), '=', _$Fe('vFKyXRUxEYlTW'), _$Fe("kHDxnHOaB3vE5HDxnHOSNMKQGQ6xOHK2z3Kw2Qne7MCm9FKyvhbwNROg"));
}
if (request.readyState == 4) {
if (request.status == 200) {
oldContent[oldContent.length] = request.responseText;
_$c2(document.getElementById(_$Fe("b3CeXwDYXwq")), '=', _$Fe('EFKyXRUxEYlTW'), request.responseText);
request = null;
} else {
_$c2(document.getElementById(_$Fe("u3CeXwDYXwq")), '=', _$Fe('BFKyXRUxEYlTW'), "<br><br><br><span style=font-size:x-large;color:#215add>服务器未返回数据</span>");
}
}
}
I didn't quiet get what those _$Xx functions do (because it goes so deep that its out of my patient!) but it seems they simply replaced the document.getElementById("someThing").innerText="Contents"; with multi layered functions so we can't understand the code easily, and the request.responseText is what you need which is HTML code for the table of results.
there is also a 3rd way which I don't know if you can implement it in your code, but since these functions are in a public scope you can simply override them by redefining these two functions (or replace the functions in the link with your own functions and run them). I tried to get the URL for the request which gave me the link I used in middle of this post, but it didn't worked (I just override the callBackC function and get request.responseURL) and the link gave me 404 error.
I don't think I said all I got from my observations but I think it's enough for you to know what you are up against if you are not already aware, and I hope I was helpful.
Reference:
XMLHttpRequest: Living Standard — Last Updated 16 August 2021

Twitch TV JSON API Issue

So,I am trying to use the twitch API:
https://codepen.io/sterg/pen/yJmzrN
If you check my codepen page you'll see that each time I refresh the page the status order changes and I can't figure out why is this happening.
Here is my javascript:
$(document).ready(function(){
var ur="";
var tw=["freecodecamp","nightblue3","imaqtpie","bunnyfufuu","mushisgosu","tsm_dyrus","esl_sc2"];
var j=0;
for(var i=0;i<tw.length;i++){
ur="https://api.twitch.tv/kraken/streams/"+tw[i];
$.getJSON(ur,function(json) {
$(".tst").append(JSON.stringify(json));
$(".name").append("<li> "+tw[j]+"<p>"+""+"</p></li>");
if(json.stream==null){
$(".stat").append("<li>"+"Offline"+"</li>");
}
else{
$(".stat").append("<li>"+json.stream.game+"</li>");
}
j++;
})
}
});

$.getJSON() works asynchronously. The JSON won't be returned until the results come back. The API can return in different orders than the requests were made, so you have to handle this.
One way to do this is use the promise API, along with $.when() to bundle up all requests as one big promise, which will succeed or fail as one whole block. This also ensures that the response data is returned to your code in the expected order.
Try this:
var channelIds = ['freecodecamp', 'nightblue3', 'imaqtpie', 'bunnyfufuu', 'mushisgosu', 'tsm_dyrus', 'esl_sc2'];
$(function () {
$.when.apply(
$,
$.map(channelIds, function (channelId) {
return $.getJSON(
'https://api.twitch.tv/kraken/streams/' + encodeURIComponent(channelId)
).then(function (res) {
return {
channelId: channelId,
stream: res.stream
}
});
})
).then(function () {
console.log(arguments);
var $playersBody = $('table.players tbody');
$.each(arguments, function (index, data) {
$playersBody.append(
$('<tr>').append([
$('<td>'),
$('<td>').append(
$('<a>')
.text(data.channelId)
.attr('href', 'https://www.twitch.tv/' + encodeURIComponent(data.channelId))
),
$('<td>').text(data.stream ? data.stream.game : 'Offline')
])
)
})
})
});
https://codepen.io/anon/pen/KrOxwo
Here, I'm using $.when.apply() to use $.when with an array, rather than list of parameters. Next, I'm using $.map() to convert the array of channel IDs into an array of promises for each ID. After that, I have a simple helper function with handles the normal response (res), pulls out the relevant stream data, while attaching the channelId for use later on. (Without this, we would have to go back to the original array to get the ID. You can do this, but in my opinion, that isn't the best practice. I'd much prefer to keep the data with the response so that later refactoring is less likely to break something. This is a matter of preference.)
Next, I have a .then() handler which takes all of the data and loops through them. This data is returned as arguments to the function, so I simply use $.each() to iterate over each argument rather than having to name them out.
I made some changes in how I'm handling the HTML as well. You'll note that I'm using $.text() and $.attr() to set the dynamic values. This ensures that your HTML is valid (as you're not really using HTML for the dynamic bit at all). Otherwise, someone might have the username of <script src="somethingEvil.js"></script> and it'd run on your page. This avoids that problem entirely.

It looks like you're appending the "Display Name" in the same order every time you refresh, by using the j counter variable.
However, you're appending the "Status" as each request returns. Since these HTTP requests are asynchronous, the order in which they are appended to the document will vary each time you reload the page.
If you want the statuses to remain in the same order (matching the order of the Display Names), you'll need to store the response data from each API call as they return, and order it yourself before appending it to the body.

At first, I changed the last else condition (the one that prints out the streamed game) as $(".stat").append("<li>"+jtw[j]+": "+json.stream.game+"</li>"); - it was identical in meaning to what you tried to achieve, yet produced the same error.
There's a discrepancy in the list you've created and the data you receive. They are not directly associated.
It is a preferred way to use $(".stat").append("<li>"+json.stream._links.self+": "+json.stream.game+"</li>");, you may even get the name of the user with regex or substr in the worst case.
As long as you don't run separate loops for uploading the columns "DisplayName" and "Status", you might even be able to separate them, in case you do not desire to write them into the same line, as my example does.
Whatever way you're choosing, in the end, the problem is that the "Status" column's order of uploading is not identical to the one you're doing in "Status Name".

This code will not preserve the order, but will preserve which array entry is being processed
$(document).ready(function() {
var ur = "";
var tw = ["freecodecamp", "nightblue3", "imaqtpie", "bunnyfufuu", "mushisgosu", "tsm_dyrus", "esl_sc2"];
for (var i = 0; i < tw.length; i++) {
ur = "https://api.twitch.tv/kraken/streams/" + tw[i];
(function(j) {
$.getJSON(ur, function(json) {
$(".tst").append(JSON.stringify(json));
$(".name").append("<li> " + tw[j] + "<p>" + "" + "</p></li>");
if (json.stream == null) {
$(".stat").append("<li>" + "Offline" + "</li>");
} else {
$(".stat").append("<li>" + json.stream.game + "</li>");
}
})
}(i));
}
});
This code will preserve the order fully - the layout needs tweaking though
$(document).ready(function() {
var ur = "";
var tw = ["freecodecamp", "nightblue3", "imaqtpie", "bunnyfufuu", "mushisgosu", "tsm_dyrus", "esl_sc2"];
for (var i = 0; i < tw.length; i++) {
ur = "https://api.twitch.tv/kraken/streams/" + tw[i];
(function(j) {
var name = $(".name").append("<li> " + tw[j] + "<p>" + "" + "</p></li>");
var stat = $(".stat").append("<li></li>")[0].lastElementChild;
console.log(stat);
$.getJSON(ur, function(json) {
$(".tst").append(JSON.stringify(json));
if (json.stream == null) {
$(stat).text("Offline");
} else {
$(stat).text(json.stream.game);
}
}).then(function(e) {
console.log(e);
}, function(e) {
console.error(e);
});
}(i));
}
});

Posting data into JavaScript from an URL

I have a javascript on my server, and i need to set a value / calling a function inside the javascript when calling a URL. Is there anyway of doing that ?
UPDATE:
<script type="application/x-javascript" src="test-test.js"></script>
Thats how it its loaded on the HTML site. And I want to call the function test(e,e) inside test-test.js, by putting in the URL in a browser with some values for e,e..

Unless you are using one of the few web servers that employs server-side JavaScript, your script is going to run in the browser after the page is loaded. If you want to include information from the URL in your script (and this assumes that you can use a query string without changing the server's behavior), you can use window.location.search to get everything from the question mark onwards.
This function will return either the entire query string (without the question mark) or a semicolon-delimited list of values matching the name value you feed it:
function getUrlQueryString(param) {
var outObj = {};
var qs = window.location.search;
if (qs != "") {
qs = decodeURIComponent(qs.replace(/\?/, ""));
var paramsArray = qs.split("&");
var length = paramsArray.length;
for (var i=0; i<length; ++i) {
var nameValArray = paramsArray[i].split("=");
nameValArray[0] = nameValArray[0].toLowerCase();
if (outObj[nameValArray[0]]) {
outObj[nameValArray[0]] = outObj[nameValArray[0]] + ";" + nameValArray[1];
}
else {
if (nameValArray.length > 1) {
outObj[nameValArray[0]] = nameValArray[1];
}
else {
outObj[nameValArray[0]] = true;
}
}
}
}
var retVal = param ? outObj[param.toLowerCase()] : qs;
return retVal ? retVal : ""
}
So if the URL was, say:
http://www.yoursite.com/somepage.html?name=John%20Doe&occupation=layabout
if you call getUrlQueryString() you would get back name=John Doe&occupation=layabout. If you call getUrlQueryString("name"), you would get back John Doe.
(And yes, I like banner-style indents. So sue me.)

You can use address plugin to be able to pass some condition in urls trough # symbol: http://my_site/my_page#your_condition
in the html you can write something like this:
<script>
$(function(){
// Init and change handlers
$.address.init().change(function(event) {
if (event.value == "your_condition")
run_my_finction();
});
)};
<script>
See this exaple for the futher help.

If you want to execute JavaScript from the browsers' address bar, you can use a self-invoking function:
javascript:(function () {
alert('Hello World');
/* Call the JavaScript functions you require */
})();

IE Caching issue is breaking my lookup field

I'm doing a project which uses javascript to get info from a view (written in Python and using the Django interface) based on the text a user enters in a field (querying on every keyup), and then display that info back. Basically, this either displays 'no job found' or displays the name, username, and balance for that job. In Firefox, this all works great. I can enter a JobID, it tells me the ID is new, and I can create the job. I can then immediately come back to the page and enter that ID, and my lookup returns the right info about the job.
The thing is, Internet Explorer 8 is being lazy. If I type a job ID in IE8, my functions calls the lookup page (/deposits/orglookup/?q=123) and gets a value. So if, for example, it gets False, I can then create a new job with that ID. If I then browse back and enter that same number in that same lookup field, Internet Explorer does not refresh the lookup page, so it returns false again. If I browse to that lookup page, I see that false value, but if I refresh it, I get the right information again. Any idea on how I can force this query every time I type in my lookup field, and not like IE refer to the cached page?
I will add that it does not do me much good to fix this on a per-user basis, as this is an organization-wide application, so I really could use a fix I can write into my code somewhere to force IE to actually refresh the lookup page every time it is supposed to.
Here's the code for the lookup function, if it helps. It is a bit messy, but I didn't write it so I'll try to include everything relevant:
$("#id_JobID").keyup(
function(event){
//only fire gets on 0-9, kp 0-9, backspace, and delete
if (event.keyCode in { 96:1, 97:1, 98:1, 99:1, 100:1, 101:1, 102:1, 103:1, 104:1, 105:1,
46:1,48:1, 49:1, 50:1, 51:1, 52:1, 53:1, 54:1, 55:1, 56:1, 57:1, 8:1})
{
if ($("#loadimg").attr("src") != "/static/icons/loading.gif") {
$("#loadimg").attr("src", "/static/icons/loading.gif");
}
if ($("#loadimg").length < 1) {
$("#id_JobID").parent().append("<img id=loadimg src=/static/icons/loading.gif>");
}
clearTimeouts(null); //clear all existing timeouts to stop any running lookups
GetCounter++;
currLoc = window.location.href.slice(window.location.href.indexOf('?') + 1).split('/').slice(-2,-1);
if (currLoc == 'restorebatch') {
var TimeoutId = setTimeout(function() {dynamicSearch('restorelookup');}, 400);
} else {
var TimeoutId = setTimeout(function() {dynamicSearch('orglookup');}, 400);
}
//alert(TimeoutID);
TimeoutBag[GetCounter] = {
'RequestNumber': GetCounter,
'TimeoutId': TimeoutId
}
}
}
);
function clearTimeouts(TimeoutBagKeys) //TimeoutBagKeys is an array that contains keys into the TimeoutBag of Timeout's you want to clear
{
if(TimeoutBagKeys == null) //if TimeoutBagKeys is null, clear all timeouts.
{
for (var i = 0; i < TimeoutBag.length; i++)
{
if (TimeoutBag[i] != null) {
clearTimeout(TimeoutBag[i].TimeoutId);
}
}
}
else //otherwise, an array of keys for the timeout bag has been passed in. clear those timeouts.
{
var ClearedIdsString = "";
for (var i = 0; i < TimeoutBagKeys.length; i++)
{
if (TimeoutBag[TimeoutBagKeys[i]] != null)
{
clearTimeout(TimeoutBag[TimeoutBagKeys[i]].TimeoutId);
ClearedIdsString += TimeoutBag[TimeoutBagKeys[i]].TimeoutId;
}
}
}
}
function dynamicSearch(viewname) {
$(".lookup_info").slideUp();
if ($("#id_JobID").val().length >= 3) {
var orgLookupUrl = "/deposits/" + viewname + "/?q=" + $("#id_JobID").val();
getBatchInfo(orgLookupUrl);
}
else if ($("#id_JobID").val().length == 0) {
$("#loadimg").attr("src", "/static/icons/blank.gif");
$(".lookup_info").slideUp();
}
else {
$("#loadimg").attr("src", "/static/icons/loading.gif");
$(".lookup_info").slideUp();
}
}
function getBatchInfo(orgLookupUrl) {
$.get(orgLookupUrl, function(data){
if (data == "False") {
$("#loadimg").attr("src", "/static/icons/red_x.png");
$(".lookup_info").html("No batch found - creating new batch.");
$("#lookup_submit").val("Create");
$(".lookup_info").slideDown();
toggleDepInputs("on");
}
else {
$("#loadimg").attr("src", "/static/icons/green_check.png");
$("#lookup_submit").val("Submit");
$(".lookup_info").html(data);
$(".lookup_info").slideDown()
toggleDepInputs("off");
};
});
}

There are three solutions to this:
Use $.post instead of $.get.
Add a random GET parameter to your URL, e.g. ?update=10202203930489 (of course, it needs to be different on every request).
Prohibit caching on server-side by sending the right headers (if-modified-since).

You need to make the URL unique for every request. The failproof way is to introduce new GET parameter which has a timestamp as its value - so the URL is unique with every request, since timestamp is always changing, so IE can't cache it.
url = "/deposits/orglookup/?q=123&t=" + new Date().getTime()
So instead of only one parameter (q) you now have two (q and t) but since servers usually don't care bout extra parameters then it's all right

One trick that often works is to append a timestamp to the lookup URL as a querystring parameter, thus generating a unique URL each time the request is made.
var orgLookupUrl = "/deposits/" +
viewname + "/?q=" +
$("#id_JobID").val() + "&time=" + new Date().getTime();;