AJAX parse + Yahoo YQL returning no results? - javascript

I'm working on a script that gets all the <table> elements from an external website by going through Yahoo's YQL. This has worked fine recently, but it stopped working as of today. I'm not entirely sure why, all websites used to work with this code:
<script type="text/javascript">
$(document).ready(function () {
var container = $('#target');
function doAjax(url) {
if (url.match('^http')) {
$.getJSON("http://query.yahooapis.com/v1/public/yql?"
+ "q=select%20*%20from%20html%20where%20url%3D%22"
+ encodeURIComponent(url)
+ "%22&format=xml'&callback=?",
function (data) {
if (data.results[0]) {
var fullResponse = $(filterData(data.results[0])),
justTable = fullResponse.find("body");
container.append(justTable);
} else {
var errormsg = '<p>Error: could not load the page.</p>';
container.html(errormsg);
}
});
} else {
$('#target').load(url);
}
}
function filterData(data) {
data = data.replace(/<?\/body[^>]*>/g, '');
data = data.replace(/[\r|\n]+/g, '');
data = data.replace(/<--[\S\s]*?-->/g, '');
data = data.replace(/<noscript[^>]*>[\S\s]*?<\/noscript>/g, '');
data = data.replace(/<script[^>]*>[\S\s]*?<\/script>/g, '');
data = data.replace(/<script.*\/>/, '');
data = data.replace(/<img[^>]*>/g, '');
return data;
}
doAjax('http://www.google.com');
});
</script>
I changed the url to google and changed it to find the <body> tag instead of <table> tags to better show its not working. I looked at the URL that it's requesting and it's not showing any content. Not sure what the problem is though.

Have you checked if the "external website" you have crawled has structural changes?
When it has worked before and now not anymore, then my tip is that the site structure has changed.

It looks like the problem was that YQL was down? I just tested it again and it worked out fine. I wish they would tell us in the future if an outage occurred.

Related

How to ping ip address from java script [duplicate]

I'm making a web app that requires that I check to see if remote servers are online or not. When I run it from the command line, my page load goes up to a full 60s (for 8 entries, it will scale linearly with more).
I decided to go the route of pinging on the user's end. This way, I can load the page and just have them wait for the "server is online" data while browsing my content.
If anyone has the answer to the above question, or if they know a solution to keep my page loads fast, I'd definitely appreciate it.
I have found someone that accomplishes this with a very clever usage of the native Image object.
From their source, this is the main function (it has dependences on other parts of the source but you get the idea).
function Pinger_ping(ip, callback) {
if(!this.inUse) {
this.inUse = true;
this.callback = callback
this.ip = ip;
var _that = this;
this.img = new Image();
this.img.onload = function() {_that.good();};
this.img.onerror = function() {_that.good();};
this.start = new Date().getTime();
this.img.src = "http://" + ip;
this.timer = setTimeout(function() { _that.bad();}, 1500);
}
}
This works on all types of servers that I've tested (web servers, ftp servers, and game servers). It also works with ports. If anyone encounters a use case that fails, please post in the comments and I will update my answer.
Update: Previous link has been removed. If anyone finds or implements the above, please comment and I'll add it into the answer.
Update 2: #trante was nice enough to provide a jsFiddle.
http://jsfiddle.net/GSSCD/203/
Update 3: #Jonathon created a GitHub repo with the implementation.
https://github.com/jdfreder/pingjs
Update 4: It looks as if this implementation is no longer reliable. People are also reporting that Chrome no longer supports it all, throwing a net::ERR_NAME_NOT_RESOLVED error. If someone can verify an alternate solution I will put that as the accepted answer.
Ping is ICMP, but if there is any open TCP port on the remote server it could be achieved like this:
function ping(host, port, pong) {
var started = new Date().getTime();
var http = new XMLHttpRequest();
http.open("GET", "http://" + host + ":" + port, /*async*/true);
http.onreadystatechange = function() {
if (http.readyState == 4) {
var ended = new Date().getTime();
var milliseconds = ended - started;
if (pong != null) {
pong(milliseconds);
}
}
};
try {
http.send(null);
} catch(exception) {
// this is expected
}
}
you can try this:
put ping.html on the server with or without any content, on the javascript do same as below:
<script>
function ping(){
$.ajax({
url: 'ping.html',
success: function(result){
alert('reply');
},
error: function(result){
alert('timeout/error');
}
});
}
</script>
You can't directly "ping" in javascript.
There may be a few other ways:
Ajax
Using a java applet with isReachable
Writing a serverside script which pings and using AJAX to communicate to your serversidescript
You might also be able to ping in flash (actionscript)
You can't do regular ping in browser Javascript, but you can find out if remote server is alive by for example loading an image from the remote server. If loading fails -> server down.
You can even calculate the loading time by using onload-event. Here's an example how to use onload event.
Pitching in with a websocket solution...
function ping(ip, isUp, isDown) {
var ws = new WebSocket("ws://" + ip);
ws.onerror = function(e){
isUp();
ws = null;
};
setTimeout(function() {
if(ws != null) {
ws.close();
ws = null;
isDown();
}
},2000);
}
Update: this solution does not work anymore on major browsers, since the onerror callback is executed even if the host is a non-existent IP address.
To keep your requests fast, cache the server side results of the ping and update the ping file or database every couple of minutes(or however accurate you want it to be). You can use cron to run a shell command with your 8 pings and write the output into a file, the webserver will include this file into your view.
The problem with standard pings is they're ICMP, which a lot of places don't let through for security and traffic reasons. That might explain the failure.
Ruby prior to 1.9 had a TCP-based ping.rb, which will run with Ruby 1.9+. All you have to do is copy it from the 1.8.7 installation to somewhere else. I just confirmed that it would run by pinging my home router.
There are many crazy answers here and especially about CORS -
You could do an http HEAD request (like GET but without payload).
See https://ochronus.com/http-head-request-good-uses/
It does NOT need a preflight check, the confusion is because of an old version of the specification, see
Why does a cross-origin HEAD request need a preflight check?
So you could use the answer above which is using the jQuery library (didn't say it) but with
type: 'HEAD'
--->
<script>
function ping(){
$.ajax({
url: 'ping.html',
type: 'HEAD',
success: function(result){
alert('reply');
},
error: function(result){
alert('timeout/error');
}
});
}
</script>
Off course you can also use vanilla js or dojo or whatever ...
If what you are trying to see is whether the server "exists", you can use the following:
function isValidURL(url) {
var encodedURL = encodeURIComponent(url);
var isValid = false;
$.ajax({
url: "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22" + encodedURL + "%22&format=json",
type: "get",
async: false,
dataType: "json",
success: function(data) {
isValid = data.query.results != null;
},
error: function(){
isValid = false;
}
});
return isValid;
}
This will return a true/false indication whether the server exists.
If you want response time, a slight modification will do:
function ping(url) {
var encodedURL = encodeURIComponent(url);
var startDate = new Date();
var endDate = null;
$.ajax({
url: "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22" + encodedURL + "%22&format=json",
type: "get",
async: false,
dataType: "json",
success: function(data) {
if (data.query.results != null) {
endDate = new Date();
} else {
endDate = null;
}
},
error: function(){
endDate = null;
}
});
if (endDate == null) {
throw "Not responsive...";
}
return endDate.getTime() - startDate.getTime();
}
The usage is then trivial:
var isValid = isValidURL("http://example.com");
alert(isValid ? "Valid URL!!!" : "Damn...");
Or:
var responseInMillis = ping("example.com");
alert(responseInMillis);
const ping = (url, timeout = 6000) => {
return new Promise((resolve, reject) => {
const urlRule = new RegExp('(https?|ftp|file)://[-A-Za-z0-9+&##/%?=~_|!:,.;]+[-A-Za-z0-9+&##/%=~_|]');
if (!urlRule.test(url)) reject('invalid url');
try {
fetch(url)
.then(() => resolve(true))
.catch(() => resolve(false));
setTimeout(() => {
resolve(false);
}, timeout);
} catch (e) {
reject(e);
}
});
};
use like this:
ping('https://stackoverflow.com/')
.then(res=>console.log(res))
.catch(e=>console.log(e))
I don't know what version of Ruby you're running, but have you tried implementing ping for ruby instead of javascript? http://raa.ruby-lang.org/project/net-ping/
let webSite = 'https://google.com/'
https.get(webSite, function (res) {
// If you get here, you have a response.
// If you want, you can check the status code here to verify that it's `200` or some other `2xx`.
console.log(webSite + ' ' + res.statusCode)
}).on('error', function(e) {
// Here, an error occurred. Check `e` for the error.
console.log(e.code)
});;
if you run this with node it would console log 200 as long as google is not down.
You can run the DOS ping.exe command from javaScript using the folowing:
function ping(ip)
{
var input = "";
var WshShell = new ActiveXObject("WScript.Shell");
var oExec = WshShell.Exec("c:/windows/system32/ping.exe " + ip);
while (!oExec.StdOut.AtEndOfStream)
{
input += oExec.StdOut.ReadLine() + "<br />";
}
return input;
}
Is this what was asked for, or am i missing something?
just replace
file_get_contents
with
$ip = $_SERVER['xxx.xxx.xxx.xxx'];
exec("ping -n 4 $ip 2>&1", $output, $retval);
if ($retval != 0) {
echo "no!";
}
else{
echo "yes!";
}
It might be a lot easier than all that. If you want your page to load then check on the availability or content of some foreign page to trigger other web page activity, you could do it using only javascript and php like this.
yourpage.php
<?php
if (isset($_GET['urlget'])){
if ($_GET['urlget']!=''){
$foreignpage= file_get_contents('http://www.foreignpage.html');
// you could also use curl for more fancy internet queries or if http wrappers aren't active in your php.ini
// parse $foreignpage for data that indicates your page should proceed
echo $foreignpage; // or a portion of it as you parsed
exit(); // this is very important otherwise you'll get the contents of your own page returned back to you on each call
}
}
?>
<html>
mypage html content
...
<script>
var stopmelater= setInterval("getforeignurl('?urlget=doesntmatter')", 2000);
function getforeignurl(url){
var handle= browserspec();
handle.open('GET', url, false);
handle.send();
var returnedPageContents= handle.responseText;
// parse page contents for what your looking and trigger javascript events accordingly.
// use handle.open('GET', url, true) to allow javascript to continue executing. must provide a callback function to accept the page contents with handle.onreadystatechange()
}
function browserspec(){
if (window.XMLHttpRequest){
return new XMLHttpRequest();
}else{
return new ActiveXObject("Microsoft.XMLHTTP");
}
}
</script>
That should do it.
The triggered javascript should include clearInterval(stopmelater)
Let me know if that works for you
Jerry
You could try using PHP in your web page...something like this:
<html><body>
<form method="post" name="pingform" action="<?php echo $_SERVER['PHP_SELF']; ?>">
<h1>Host to ping:</h1>
<input type="text" name="tgt_host" value='<?php echo $_POST['tgt_host']; ?>'><br>
<input type="submit" name="submit" value="Submit" >
</form></body>
</html>
<?php
$tgt_host = $_POST['tgt_host'];
$output = shell_exec('ping -c 10 '. $tgt_host.');
echo "<html><body style=\"background-color:#0080c0\">
<script type=\"text/javascript\" language=\"javascript\">alert(\"Ping Results: " . $output . ".\");</script>
</body></html>";
?>
This is not tested so it may have typos etc...but I am confident it would work. Could be improved too...

Web crawler encounter javascript

I'm new to web crawling. I am trying to crawl a webpage using java and I encounter a problem. I need to get the link in a 'HTML Tag' whose href is a javascript function. I have no idea how to get the link in the javascript function. Here is the html source and javascript source.
HTML
<a href='javascript:ShowPostGridUnique(205316,0);'>link</a>
JSShowPostGridUnique
function ShowPostGridUnique(parentpostid, pageShow) {
//alert(parentpostid);
var divid;
divid = 'divPostContent' + parentpostid;
if (document.getElementById(divid).className == 'divGridShow') {
document.getElementById(divid).className = 'divGridHide';
document.getElementById(divid).innerHTML = '';
}
else {
document.getElementById(divid).className = 'divGridShow';
// call server side method
PageMethods.divParentInnerHtml( parentpostid, pageShow, CallSuccessShowPost, CallFailedAlert, parentpostid);
try {
divid = 'TDtitle' + parentpostid;
document.getElementById(divid).className = 'TDtitle';
divid = 'TDPage' + parentpostid;
document.getElementById(divid).className = 'TDtitle';
}
catch (err) {
//Handle errors here
}
}
}
How can i get the link of href? Thanks.
Use Headless Browser like Phantomjs.
http://phantomjs.org/
Use ghostdriver/selenium to control Phantomjs
https://github.com/SeleniumHQ/selenium
https://github.com/detro/ghostdriver

Remove multiple records serverside with ajax and jquery

I'm trying to delete multiple tables with a same class name, my js code snippet works once and it doesn't loop to the next table unless the page is refreshed again.
when I comment my ajax call and just run it plain jquery it works fine. , I think there's an issue with my ajax call somewhere....
This jsfiddle url http://jsfiddle.net/ehsansajjad465/ExnkV/ has the snippet without the ajax....how do I make sure the ajax call works properly rather refreshing my page every time?
$(".closeprod").live("click",function(e){
e.preventDefault();
elem = $(this).parents('.tbl');
//get serial number
prodsn = $(".tbl").find(".prodsn:eq(0)");
sn = $(prodsn[0]).html().substr(5);
tpl = "anything";
url = "delprod.asp?email=<%=email%>&sn=" + sn + "&t=" + tpl + "&nf=notfeatured";
//remove product from xml file
$.get(url, function(data,status){
if (data == "OK") {
//remove product from template
elem.remove();
}else{
alert("opps something is wrong")
}
});
});
I figured it out this line prodsn = $(".tbl").find(".prodsn:eq(0)"); should be prodsn = elem.find(".prodsn:eq(0)"); and it worked like this

Internet Explorer jQuery getJSON not working

im making a small script which gets some data from a database through php using jquery's getJSON method.
the code as shown below:
$(document).ready(function(){
var id = userid;
$('#a-div').after('<div id="data"></div><input type="button" id="getdata" value="Get Data">');
$('#getdata').click(function(){
$.getJSON('http://mysite.com/data.php?id=' + id, function(data) {
var notfound = data['notfound'];
var user = data['user'];
if(notfound == '1'){
$('#data').html("Not found");
}
else{
$('#data').html("Found , user is "+ user);
}
});//end of getJSON
});//end of click
}); //end of document ready
My php script returns JSON data something like this :
If the data is found in database-
{"notfound":"0","user":"john"}
If the data is NOT found in database-
{"notfound":"1","user":"none"}
This works perfectly on Firefox , Google Chrome and Safari , just dosent work in Internet Explorers(7,8,9)
can anyone help me out.
P/S i have tried a few techniques in other posts similar to this one , is not working.
Like changing the the META content-type
Thanks.
Try putting the button inside of the <body> like so:
$("body").append( /* markup for div and btn here */ );
...instead of using after().
Maybe internet explorer is taking long time to add the html, try with live
$('#getdata').live('click', function(){
you are using very ugly code, after body incorrect to write any HTML content, for that you must add in body any div and append your cod to it, and then this code will work in IEs
$(document).ready(function () {
$('#anydiv').html('<div id="data"></div><input type="button" id="getdata" value="Get Data">');
$('#getdata').click(function () {
alert("asd");
$.getJSON('/HowItWork/Index?id=' + 23, function (data) {
var notfound = data['notfound'];
var user = data['user'];
if (notfound == '1') {
$('#data').html("Not found");
}
else {
$('#data').html("Found , user is " + user);
}
}); //end of getJSON
}); //end of click
}); //end of document ready
This works for me- jQuery.support.cors = true;
Find more discussion http://jquery.10927.n7.nabble.com/jQuery-getJSON-response-not-working-with-IE-works-perfectly-in-firefox-safari-td81254.html

Populating A Page with XML and Javascript/Jquery, checking if the url is the same as an xml tag

I need to create a javascript function that will write a page based on the url, so basically I am trying to create a javascript function that will check the url, and find the corresponding xml item from there.
The reason behind this is so that the html page can just be duplicated, renamed, and the xml updated, and the page will fill in everything else from the xml sheet.
please let me know whether this is the completely incorrect way to do it, of it there is a better way. thanks!!!
XML CODE::
<!--XML INFORMATION-->
<channel>
<design>
<motion><!--design content-->
<item><!--//////////POST//////////POST//////////POST//////////-->
<tag>/portfolio_dec.html</tag>
<!--RSS INFORMATION-->
<title>Decoze</title>
<link>http://payamrajabi.com/portfoliotest.html</link>
<description><img src="http://payamrajabi.com/thumbs/small_jump.png" title="JUMP!." /></description>
<!--PROJECT CONTENT-->
<project><!--project start-->
<titl>TITLE</titl><!--project title-->
<dsc>PROJECT DESCRIPTION</dsc><!--project description-->
</project><!--project end-->
</item><!--//////////END//////////END//////////END//////////-->
</motion>
</design>
</channel>
JAVASCRIPT:
$(document).ready(function(){
$.ajax({
type: "GET",
url: "code/content9.xml",
dataType: "xml",
success: function(xml) {
var xpathname = window.location.pathname;
var xproject = $(xml).find('tag').text();
if (xpathname == xproject) {
$(xml).find('item').children('tag').text(xpathname).each(function(){
var ttl = $(this).find('titl').text();
$('<p>'+ttl+'</p>').appendTo('h1#ttl');
});
$(xml).find('item').children('tag').text(xpathname).each(function(){
var dsc = $(this).find('dsc').text();
$('<p>'+dsc+'</p>').appendTo('h1#ttl');
});
} else {
PUT ERROR MESSAGE HERE
}
}
});
});
and THE HTML:
<html>
<head>
<script type="text/javascript" src="code/jquery-1.3.1.js"></script>
<script type="text/javascript" src="code/project/project_design.js"></script>
</head>
<body>
<h1 id="ttl"></h1>
<p id="dsc"></p>
</body>
</html>
any help would really be appreciated, i am frairly new to javascript/jquery/xml, and am really having trouble with this. The primary thing I want to do is have an xml file that populates a site, with each item being the content for a new page, in this case of a portfolio item.
cheers!
willem
Hmm... I'm afraid you don't quite understand how jquery works.
Your code should look something like this:
var xpathname = window.location.pathname;
var xitem = $(xml).find('tag:contains(' + xpathname + ')').parent();
if (xproject.length != 0) {
$('#ttl').append('<p>' + xitem.find('titl').text() + '</p>');
$('#dsc').append('<p>' + xitem.find('dsc').text() + '</p>');
}
else {
$('#err').text('The page you requested does not exist');
}
Demo 1
Here's a quick demo. Take a look at the source to see the XML and the JavaScript.
http://jsbin.com/ujiho#itemOne
http://jsbin.com/ujiho#itemTwo
http://jsbin.com/ujiho#itemThree
Demo 2
I've created another demo that uses $.get to retrieve the XML from a separate URL.
http://jsbin.com/aqefo#nov
http://jsbin.com/aqefo#dec
The XML: http://jsbin.com/afiwa
Here's the JavaScript. Let me know if you need help understanding anything.
$(function(){
$.get(
'http://jsbin.com/afiwa',
function(xml){
var hash = window.location.hash.substring(1);
if ($.trim(hash) === '') {
showError();
return;
}
var xitem = $(xml).find('urlname:contains(' + hash + ')').parent();
if (xitem.length != 0) {
$('#ttl').append(xitem.find('titl').text());
$('#dsc').append( xitem.find('dsc').text());
}
else {
showError();
}
function showError() {
$('#err').text('The page you requested does not exist');
}
}
);
});

Categories