We Keep Coding

Loading a specific div from another website into own website

Ive tried using the js load function but as the external site does not allow CORS requests, my original GET request gets blocked.
<div id="test"></div>
<script>
$(document).ready(function () {
$("#test").load("https://mywebsite.com");
});
</script>
So it seems that my only approach is to use iframes?! Is there a way to only crawl a specific div with iframes? I dont want to display the whole website.
EDIT: Since I am using Django I was able to crawl the website with python in a view and then push the crawled and cleaned up code snippet in the html template. Nevertheless to answer my question -> There is no correct way of doing it as long as the website you are trying to access is blocking the content.

Work with the owner of the site you want to take content from.
They can set you up with an API. That avoids having to use hackey methods or risking copyright-related legal trouble.

Unable to load script from another server - Content Security Policy issue?

I have been trying to turn a bookmarklet into a small development environment that I can use for testing some javascript and sending commands easily on the fly and updating the code on my server quickly to see the result. This has half way worked using method's I have found in this site and google however it doesn't seem to work very well and sometimes randomly doesn't work. The end goal is to have a bookmarklet that I can click on from any page and it loads a javascript file I have saved on my server. I have created the following two bookmarklets to try and get this working:
Failed Method 1:
javascript:
var s = document.createElement('script');
s.type='text/javascript';
document.body.appendChild(s);
s.src='//smewth.com/test.js';
void(0);
Method 1 in one line bookmarklet form: javascript: var s = document.createElement('script'); s.type='text/javascript'; document.body.appendChild(s); s.src='//smewth.com/test.js'; void(0);
Failed Method 2:
javascript:(
function(){
var imported = document.createElement('script');
imported.type='text/javascript';
imported.src = 'https://smewth.com/test.js';
document.head.appendChild(imported);
})();
Method 2 in one line bookmarklet form: javascript:( function(){ var imported = document.createElement('script'); imported.type='text/javascript'; imported.src = 'https://smewth.com/test.js'; document.head.appendChild(imported); })();
I got method 1 by decomposing the kickass bookmarklet from (http://kickassapp.com/). The actual one I got from their site works fine on my browser no problems. I even did a direct substitution from the URL they were using to load with my URL. The second method I found while searching on this site and this actually worked for a small while and stopped working for some unknown reason (maybe different browsers). I tried appending this script object to the head and the body on each of them with no improved results.
I created the test.js script just for this post and it contains a simple alert box statement:
$$ [/]# cat test.js
alert("hi");
$$ [/]#
NOTE: When I do this with the code embedded within the the bookmarklet itself without appending it to a head/body object then it works fine such as this:
javascript:%20alert("hi");
I did notice that with both of these methods, the code is actually getting injected into the page however I am not seeing the code is ever executed when I click the bookmark. Does anyone know which method is the best or something similar to do this so I can have javascript load through a page which I update on a remote server (reliably)? Maybe I need to attach the to a different object?
Thank you for your help.
-Jeff
UPDATE: I am showing this works while this site is loaded but it doesn't work when your at a site like google.com. Not sure what the difference is or how to accomodate this, google.com has a head and a body object too. I am showing this works in some sites and in some it doesn't.

I figured this out. There were two things occurring which accounts for the intermittent symptom of this issue. The first issue was that the site which was hosting the code was on a self-signed certificate. I began to notice the issue was occurring only when trying to run this from within secure sites. Then in Chrome I saw a error show up in the console. It would be nice if Firefox gave me a error on the console or something as this was the root of the issue. The second thing I had to do was disable OCSP in Firefox as I used a free certificate for testing purposes.
I also had to use method 1 as described above. Firefox and Chrome both did not like the anonymous function call for some reason. From now on I will refer to Chrome to look for errors in the console as Firefox has proven itself not very useful for this.

how should my site handle ocassionally missing javascript files gracefully?

Say I've got this script tag on my site (borrowed from SO).
<script type="text/javascript" async=""
src="http://edge.quantserve.com/quant.js"></script>
If edge.quantserve.com goes down or stops responding without returning a 404, won't SO have to wait for the timeout before the rest of the page loads? I'm thinking Chaos Monkey shows up and blasts a server that my site is depending on, a server that isn't part of a CDN and has a poor failover.
What's the industry standard way to handle this issue? I couldn't find a dupe on SO, maybe I'm searching for the wrong terms.
Update: I should have looked a bit more closely at the SO code, there's this at the bottom:
<script type="text/javascript">var _gaq=_gaq||[];_gaq.push(['_setAccount','UA-5620270-1']);
_gaq.push(['_setCustomVar', 2, 'accountid', '14882',2]);
_gaq.push(['_trackPageview']);
var _qevents = _qevents || [];
(function(){
var s=document.getElementsByTagName('script')[0];
var ga=document.createElement('script');
ga.type='text/javascript';
ga.async=true;
ga.src='http://www.google-analytics.com/ga.js';
s.parentNode.insertBefore(ga,s);
var sc=document.createElement('script');
sc.type='text/javascript';
sc.async=true;
sc.src='http://edge.quantserve.com/quant.js';
s.parentNode.insertBefore(sc,s);
})();
</script>
OK, so if the quant.js file fails to load, it's creating a script tag with ga.async=true;. Maybe that's the trick.
Possible answer: https://stackoverflow.com/a/1834129/30946

Generally, it's tricky to do it well and cross-browser.
Some proposals:
Move the script to the very bottom of the HTML page (so that almost everything is displayed before you request that script)
Move it to the bottom and wrap it in <script>document.write("<scr"+"ipt src='http://example.org/script.js'></scr"+"ipt>")</script> or the way you added after update (document.createElement('script'))
A last option is to load it via XHR (but this works only for same-domain, or cross-domain only if the CORS is enabled on a third-party server); you can then use timeout property of the XHR (for IE and Fx12+), and in the other browsers, use setTimeout and check the XHR's readyState. It's kind of convoluted and very non-cross-browser for now, so the option 2 looks the best.

Make a copy of the file on your server and use this. it will load your copy only if the one from the server has failed to load
<script src="http://edge.quantserve.com/quant.js"></script>
<script>window.quant || document.write('<script src="js/quant.js"><\/script>')</script>

To answer your question about the browser having to wait for the script to load before the rest of the page loads, the answer to that would typically be no. Typical browsers will have multiple threads processing the download of the page and linked content (CSS, images, js). So the rest of the page should be loaded, though the user's browser indicator will still show the page trying to load until the final request is fulfilled or timed out.
Depending on the nature of the resource you are trying to load, this will obviously effect your page differently. Typically, if you are worried about this, you can host all your files on a common CDN (or your website if it is not that highly trafficked), that way at least if one thing fails, chances are everything is failing and you have a bigger issue to contend with :)

Is there a way to mitigate downloading of resources (images/css and js files) with Javascript?

I have a html page on my localhost - get_description.html.
The snippet below is part of the code:
<input type="text" id="url"/>
<button id="get_description_button">Get description</button>
<iframe id="description_container" src="#"/>
When the button is clicked the src of the iframe is set to the url entered in the textbox. The pages fetched this way are very big with lots of linked files. What I am interested in the page is a block of text contained in a <div id="description"> element.
Is there a way to mitigate downloading of resources linked in the page that loads into the iframe?
I don't want to use curl because the data is only available to logged in users and the steps to take with curl to get the content is too complicated. The iframe is simple as I use this on a box which sends the right cookies to identify the request as coming from a logged in user, but the problem is that it is very wasteful to get nearly 1 MB of data to keep 1 KB of it and throw out the rest.
Edit
If the proposed method just works in Firefox it is fine, so I added Firefox tag. Also, it is possible that the answer actually is from the realm of Firefox add-on techniques, so I added that tag as well.
The problem is not that I cannot get at what I'm looking for, rather, the problem is the easy iframe method is wasteful.
I know that Firefox does allow loading only the text of a page. If you open a page and press Ctrl+U you are taken to 'view page source' window, There links behave as normal and are clickable, if you click on a link in source view, the source of the new page is loaded into the view source window, without the linked resources being downloaded, exactly what I'm trying to get. But I don't know how to access this behaviour.
Another example is the Adblock add-on. It somehow kills elements before they get loaded. With plain Javascript this is not possible. Because it only is triggered too late to intervene in good time.

The Same Origin Policy forbids any web page to access contents of any other web page in a different domain so basically you cannot do that.
However it seems that with some browsers it is allowed to access web pages content if you are trying to access it from a local web page which seems to be your case.
Safari, IE 6/7/8 are browser that allow a local web page to do so via XMLHttpRequest (source: Google Browser Security Handbook) so you may want to choose to use one of those browsers to do what you need (note that future versions of those browsers may not allow to do so anymore).
A part from this solution I only see two possibities:
If the web pages you need to fetch content from are somehow controlled by you, you can create a simpler interface to let other web pages to get the content you need (for example allowing JSONP requests).
If the web pages you need to fetch content from are not controlled by you the only solution I see is to fetch content server side logging in from the server directly (I know that you don't want to do so, but I don't see any other possibility if the previous I mentioned are not practicable)
Hope it helps.

Actually I've seen Cross Domain jQuery .load request before, here: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/
The author claims that codes like these found on that page
$('#container').load('http://google.com'); // SERIOUSLY!
$.ajax({
url: 'http://news.bbc.co.uk',
type: 'GET',
success: function(res) {
var headline = $(res.responseText).find('a.tsh').text();
alert(headline);
}
});
// Works with $.get too!
would work. (The BBC code might not work because of the recent redesign, but you get the idea)
Apparently it is using YQL wrapped into a jQuery plugin to do the trick. Now I cannot say I fully understand what he is doing there but it appears to work, and fits the bill. Once you load the data I suppose it is a simple matter of filtering out the data that you need.
If you prefer something that works at the browser level, may I suggest Mozilla's Jetpack framework for lightweight extensions. I've not yet read the documentations in its entirety but it should contain the APIs needed for this to work.

There are various ways to go about this in AJAX, I'm going to show the jQuery way for brevity as one option, though you could do this in vanilla JavaScript as well.
Instead of an <iframe> you can just use a container, let's say a <div> like this:
<div id="description_container"></div>
Then to load it:
$(function() {
$("#get_description_button").click(function() {
$("#description_container").load($("input").val() + " #description");
});
});
This uses the .load() method which takes a string in this format: .load("url selector"), then takes that element in the page and places it's content inside the container you're loading, in this case #description_container.
This is just the jQuery route, mainly to illustrate that yes, you can do what you want, but you don't have to do it exactly like this, just showing the concept is getting what you want from an AJAX request, rather than in an <iframe>.

Your description sounds like you are fetching pages from the same domain (you said that you need to be logged in and have session credentials) so have you tried to use async request via XMLHttpRequest? It might complain if the html on a page is particularly messed up but you chould still be able to get raw text via .responseText and extract what you need with a regex.

w3schools AJAX example

What would I need to do to get this example running on my machine?
http://www.w3schools.com/ajax/tryit.asp?filename=tryajax_httprequest_js (page no longer available)
I'm looking to access the XML file hosted on w3schools (and not move it to my machine), but run the HTML and Javascript code on my machine. I tried changing the third to last line from:
<button onclick="loadXMLDoc('note.xml')">Get XML</button>
to:
<button onclick="loadXMLDoc('http://www.w3schools.com/ajax/note.xml')">Get XML</button>
thinking this would make it work, but it didn't seem to help. Any suggestions?

Just put the full URL into your browser window which will let your browser get it, then copy/paste and save locally. Javascript won't fetch stuff from outside the domain it's served from (without a fair bit of extra work), due to the Same Origin policy ( a security feature).

You can't go cross domain using AJAX. You should move the XML file to the same server that you have the site files stored on and call it that way.

https://developer.mozilla.org/en/Same_origin_policy_for_JavaScript

You need to use the following code in the function that does the AJAX:
try {
netscape.security.PrivilegeManager.enablePrivilege("UniversalPreferencesRead");
} catch (e) {
alert("error");
}
This only works for Firefox! There are other options which can be passed to enablePrivilege that may be useful.