We Keep Coding

Scraping Trophy Data from The Official Playstation Website

Im trying to use PhantomJS to scrape the trophy data from http://my.playstation.com/logged-in/trophies/public-trophies/
The page requires you enter a valid username and then click 'go' and the page will load the data. Ive gotten this to work somewhat, but it never loads the trophy data into the div. Im hoping im missing something ajax related thats causing this?
var fullpagehtml = page.evaluate(function()
{
document.getElementById("trophiesId").value = "<<valid user id>>";
//checkPTrophies(); btn click calls this function
$('#btn_publictrophy').click().delay( 6000 );
console.log("\nWaiting for trophy list to load...");
var trophylist = document.getElementById("trophyTrophyList").innerHtml; // all the data i want ends up inside this div
var counter = 0; //delay andset timeout wont work here so this is the best i coukld think of
while (trophylist == null)
{
//presumably the ajax query should kick in on the page and populate this div, but it doesnt.
trophylist = document.getElementById("trophyTrophyList").innerHtml;
counter ++;
if(counter == 1000000)
{
console.log($('#trophyTrophyList').html());
counter = 0;
}
}
return document.all[0].outerHTML;
});

The delay( 6000 ) does absolutely nothing as the documentation says:
The .delay() method is best for delaying between queued jQuery effects. Because it is limited—it doesn't, for example, offer a way to cancel the delay—.delay() is not a replacement for JavaScript's native setTimeout function, which may be more appropriate for certain use cases.
To wait you have to do this outside of the page context (busy waiting doesn't work in JavaScript because it is single threaded):
page.evaluate(function() {
document.getElementById("trophiesId").value = "<<valid user id>>";
//checkPTrophies(); btn click calls this function
$('#btn_publictrophy').click();
});
console.log("\nWaiting for trophy list to load...");
setTimeout(function(){
var fullpagehtml = page.evaluate(function() {
var trophylist = document.getElementById("trophyTrophyList").innerHTML;
return trophylist;
});
}, 20000);
You also might want to use waitFor to wait until #trophyTrophyList is populated instead of using setTimeout:
waitFor(function(){
return page.evaluate(function(){
var e = document.getElementById("trophyTrophyList");
return e && e.innerHTML;
});
}, function(){
// TODO: get trophies
});
This won't get you far, because just because #trophyTrophyList is loaded, doesn't mean that the descendent elements are already in the DOM. You have to find some selector which signalizes that the page is sufficiently loaded for example by waiting until a .trophy-image exists in the page. It works for me with a 20 second timeout of the waitFor function.
waitFor(function(){
return page.evaluate(function(){
var e = document.querySelector("#trophyTrophyList .trophy-image");
return e;
});
}, function(){
setTimeout(function(){
var trophiesDiv = page.evaluate(function(){
return document.getElementById("trophyTrophyList").innerHTML;
});
console.log(trophiesDiv);
}, 1000); // wait a little longer
}, 20000);
Don't forget that you need page.evaluate to actually access the DOM. Btw, it is innerHTML not innerHtml.

How to use mouse event to do AJAX call only if user is idle for X seconds

Apologies if this is a repost. I have seen many examples. But I can't seem to put together my needs.
I have a "today" page which displays all groups. Throughout the day more and more groups will appear. I want to be able to dynamically update these groups if the user has the page open and hasn't moved the mouse for X seconds. I have this chunk of code:
var timeout = null;
j$(document).on('mousemove', function() {
if (timeout !== null) {
clearTimeout(timeout);
}
timeout = setTimeout(function() {
timeout = null;
//calls another page to check if there's new data to display. if so, wipe existing data and update
j$.ajax({
url: "/include/new_Groups.php",
cache: false,
success: function(data){
j$( ".group_Container_Main" ).append( data ).fadeIn('slow');
}
})
.done(function( html ) {
});
}, 3000);
});
What this is doing is if the user hasn't moved the mouse after 3 seconds, do an AJAX call to update the group. This semi works. If you don't move the mouse, it will update. But it won't update again unless the mouse is moved and idle again for 3 seconds which is not good user experience.
I'm trying to find a way to just continually update the page every 3 seconds (for this example) if the user is idle. But if he's moving the mouse, there is to be no updating. Please ask questions if I'm unclear! Thanks in advance.

Should be straigh forward, use an interval and a function call instead
jQuery(function($) {
var timer;
$(window).on('mousemove', function() {
clearInterval(timer);
timer = setInterval(update, 3000);
}).trigger('mousemove');
function update() {
$.ajax({
url : "/include/new_Groups.php",
}).done(function (html) {
$(".group_Container_Main").append(html).fadeIn('slow')
});
}
});
FIDDLE
EDIT:
To solve the issue of stacking ajax requests if for some reason they take more than three seconds to complete, we can just check the state of the previous ajax call before starting a new one, if the state is pending it's still running.
jQuery(function($) {
var timer, xhr;
$(window).on('mousemove', function() {
clearInterval(timer);
timer = setInterval(update, 1000);
}).trigger('mousemove');
function update() {
if ( ! (xhr && xhr.state && xhr.state == 'pending' ) ) {
xhr = $.ajax({
url : "/include/new_Groups.php",
}).done(function (html) {
$(".group_Container_Main").append(data).fadeIn('slow')
});
}
}
});

On the AJAX parameter, use the complete option to trigger a mouse move :
j$(document).on('mousemove', function() {
if (timeout !== null) {
clearTimeout(timeout);
}
timeout = setTimeout(function() {
timeout = null;
//calls another page to check if there's new data to display. if so, wipe existing data and update
j$.ajax({
url: "/include/new_Groups.php",
cache: false,
success: function(data){
j$( ".group_Container_Main" ).append( data ).fadeIn('slow');
},
complete: function(data){
j$(document).trigger('mousemove');
}
})
.done(function( html ) {
});
}, 3000);
});

You can invert your timer idea to this logical connection...
Set a timer for 3 seconds after which you will do the AJAX call
If the mouse is moved, reset the timer for 3 seconds
You now have a three second timer running whether or not the mouse is moved and you reset it on mouse move to get the behaviour you want in respect of only updating on idle.
var timeout = setTimeout(update, 3000);
function update() {
clearTimeout(timeout);
//calls another page to check if there's new data to display. if so, wipe existing data and update
j$.ajax({
url: "/include/new_Groups.php",
cache: false,
success: function(data){
j$( ".group_Container_Main" ).append( data ).fadeIn('slow');
}
}).done(function(html) {
}).always(function() {
timeout = setTimeout(update, 3000);
});
}
j$(document).on('mousemove', function() {
clearTimeout(timeout);
timeout = setTimeout(update, 3000);
});

You should use setInterval() instead of setTimeout() to make a repeating event.
I would call setInterval() outside of your event handler code, and then make your event handler code update a lastTimeMouseMoved (or something) timestamp, which would be checked by the code passed to your setInterval() call.
So, your code might look like this:
const IDLE_TIME = 3000;
var lastTimeMouseMoved = Date.now();
timer = setInterval(function() {
if(Date.now() - lastTimeMouseMoved >= IDLE_TIME) {
//calls another page to check if there's new data to display. if so, wipe existing data and update
j$.ajax({
url: "/include/new_Groups.php",
cache: false,
success: function(data){
j$( ".group_Container_Main" ).append( data ).fadeIn('slow');
}
})
.done(function( html ) { });
} // end idle if
}, IDLE_TIME);
j$(document).on('mousemove', function() {
lastTimeMouseMoved = Date.now();
});

Multiple ajax Allow only latest call

I have an input box on which there is an ajax request on every key press. so if i enter word "name" there will be 4 successful request. So i actually want only the latest request of executed. so if i enter word "name" there will be only one request which will be the last one.
and i also have a solution for this (this is a simple example with click method)
JS script
var callid = 1;
function ajaxCall (checkval ){
if(checkval == callid){
$.ajax({
type: 'post',
url: baseurl + "test/call_ajax",
data: {
val: "1"
},
success: function(data) {
console.log(data)
}
});
}
}
function call(){
var send = callid+=1;
setTimeout( function(){ ajaxCall(send) } , 500);
}
html script
<a href="#" onclick="call()" > Call ajax </a>
This is working perfectly. But i was think if there is way to refine it a little bit more.
Any ideas :)

I am sure you are looking some better intent technique for event dispatching.
var eventDispatcher = null;
$('.textbox').keyup(function(){
if(eventDispatcher) clearTimeout(eventDispatcher);
eventDispatcher = setTimeout(function(){
$.ajax({ ... });
}, 300);
});

You could do your ajax inside of a setTimeout. So you don't need to declare and check an additional variable or write another function like call()
$(document).ready(function () {
var timer;
$('#fillMe').keypress(function () {
clearTimeout(timer);
timer = setTimeout(function () {
//replace this with your ajax call
var content = $('#fillMe').val();
$('#result').text('You will only see this if the user stopped typing: ' + content);
}, 1000); // waits 1s before getting executed
});
});
<input type="text" id="fillMe">
<div id="result"></div>
On every keypress event this clears the timeout and immediately creates a new timeout. This means the content of the setTimeout function only gets executed if the user stopped typing for at least 1 second.
Of course 1 second is just the value for the example purpose. You can change it to whatever you want or think is a good time (like 500ms)
See my jsfiddle

setTimeout returns an id that you can store and use to clear the previously set timer:
var timerId;
function call() {
if (timerId !== undefined) {
clearTimeout(timerId);
}
timerId = setTimeout( function() { ajaxCall(send) }, 500);
}
The result of this should be that the ajaxCall method will be called 500ms after the last letter is entered.

Preventing error 503 on JQuery load

This is my code on shoutbox update :
function updateShoutbox(){
$("#shoutdiv").load("shoutbox.php", { 'action': 'update' } ,
function (responseText, textStatus, req) {
if (textStatus == "error") {
return false;
}
});
}
$(document).ready(function(){
updateShoutbox();
var auto_refresh = setInterval(
function ()
{
updateShoutbox();
$("#shoutdiv").scrollTop($("#shoutdiv")[0].scrollHeight);
}, 6000);
It returns error each some minutes :
shoutbox.php returned error:
Service Unavailable
Is there anyway to handle this error and hide it somehow ?
I edited my code so to stop showing any error on shoutbox update, but it still shows this error each minutes.

Ok, so let's take this for example:
$(document).ready(function(){
(function iterate(i) {
if (!!i) {
console.log('iteration #', i--);
setTimeout(function next(){
iterate(i);
}, 1000);
}
})(10);
});​
http://jsfiddle.net/userdude/6C8yp/
If you look at the console, you'll see it counts down until i is equal to 0, or i is not given (that's what the !! is for there). What I'm doing here is looping each second, but only after the last loop has finished. I'm feeding my loop.
Looking at what you have here, I might do this instead:
$(document).ready(function($){
var $shoutbox = $("#shoutdiv"),
timer;
(function update(){
var opts = {
url: 'shoutbox.php',
action: 'update',
complete: wait
};
$.ajax(opts);
function wait(res, status, req){
if (status == 200) {
$shoutbox
.append(res)
.scrollTop($shoutbox[0].scrollHeight);
timer = setTimeout(update, 6000);
}
}
})();
});​
http://jsfiddle.net/userdude/whsPn/
http://jsfiddle.net/userdude/whsPn/1/
Ok, so what we have above should mostly emulate the code you have in the question. You'll note that I have the complete: wait part in there, and the setTimeout() is in that callback. And.. it's only called if the status returned is 200 (success).
Now, there you could turn complete: wait to success: wait, and take out the status == 200 if statement altogether. Of course, if you do want to run the update again, but maybe do something different, this is your chance.
Also note, in the fiddle linked I've got some dummy code in there. So don't just copy/page what's in the fiddle, or you'll have errors and it won't run at all.
EDIT: Oops, found an error with url =. Fixed.

If you want to "hide" your error instead of looking for the cause of the error in the first place, try this in your callback function in the $.load:
function (responseText, textStatus, req) {
if(req.status!=200&&req.status!=302) {
return false;
}
//update the shoutbox
}
At least to me this is what seems to be the most reliable way to prevent random errors from getting through your checks.

How can I stop a javascript function (poll) from being executed multiple times?

I have a poll() function:
$("document").ready(function(){
poll(10);
$("#refresh").click(function(){ poll(10); });
});
function poll(timeout){
return setTimeout(function(){
if (timeout == 10){
timeout = 10000;
}
$.ajax({ url: "/ajax/livedata.php", success: function(data){
$("#result").html(data);
poll(timeout);
}, dataType: "json"});
}, timeout);
}
So, what happens there:
1) on page load, poll() is called and the loop starts being executed.
2) on #refresh click, poll() is called right at that moment to request new data immediately.
The problem: whenever I click #refresh to request new data, the poll() is called again, but fired multiple times (first time the initial + every new click). Thus it fires not every 10 seconds as expected, but multiple times every 10 seconds (depending on how many times I clicked on #refresh).
How can I fix this, so that whenever I click #refresh, only 1 instance will be left looping?

first store your setTimeout Object to a variable somewhat like this
var a = poll(10);
then stop it upon clicking refresh by using this code
clearTimeout(a);
So it will be like this
$("document").ready(function(){
var a = null;
$("#refresh").click(function(){
if(a != null){
clearTimeout(a);
a = null;
}
a = poll(10);
});
});
reference:
http://www.w3schools.com/jsref/met_win_cleartimeout.asp
http://www.w3schools.com/js/tryit.asp?filename=tryjs_timing_stop