We Keep Coding

How to scrape a certain text regardless of which tags it is contained in using scrapy

I am trying to scrape a number of sites to find if a certain code snippet is present. Most of the time the scraper works perfectly as intended.
I am using the following method to find the bit of code I am looking for:
...
item["foo"] = response.xpath("//script[contains(text(), 'fooscript')]")
...
if len(item["foo"]) != 0:
doStuff()
However, my issue is the following: sometimes the thing I want to find is not in the script itself but as the source for the script (I know how to scrape this as well), and sometimes when JQuery is used, I can't get the correct scrape results.
So my question is, is there an easier way to look through the raw HTML/JS text to find a match for what I am looking for? Trying to look through all alternatives to scrapes will quickly bloat up the code, and I only need to see if this certain text is present. I have not found a suitable method from the official scrapy documentation (though I am still somewhat inexperienced with the tool, so I might have missed it), so if anyone has a solution for this it would be greatly appreciated.

Maybe simple regex search through the HTML source is what you are looking for? Something like
if re.search(r'fooscript', response.text):
doStuff()
Or, if you just know it's wrapped inside some element and just don't know which, you can do
item["foo"] = response.xpath("//*[contains(text(), 'fooscript')]")
Also, you don't need to use len to check the result, simply
if item["foo"]:
doStuff()
is enough.

What does IE use //# in javascript for?

So, a bug in a piece of javascript revolved around code similar to :
<script>
(function() {
if (true) {
//#todo: do we need to set total or -- ?
alert('hello?');
}
})();
</script>
In the larger system IE complained "Expected ';' ". In the small scale example IE simply caused a warning about blocking ActiveX controls.
Obviously, "//#" has some context to activeX controls in IE. I was unable to find this as searching for the symbols was useless, and any search about special comments in IE result in the conditional html comments. I am just curious how the //# are supposed to be used in IE.

The IE JScript engine supports conditional comments which turn comments written in a particular way into code (partially). However, you are not using those.
In your case it seems to be a way to tell e.g. an IDE that there is a TODO item. The error you got is most likely unrelated.

Unless there's some quirk about IE that I don't know, the //#todo is just commenting fluff that some programmers use when they are too lazy/don't know how to implement something.

JavaScript DOM on IE

I want to make a JS function that switch visible/hidden.
var foo = function(n){
var hidden_elements = document.getElementsByName('hidden');
for(var i=0;i<hidden_elements.length;i++){
hidden_elements[i].style.visibility = 'hidden';
}
hidden_elements[n].style.visibility = 'visible';
};
It works on Firefox and Chrome, but it doesn't on IE. Why?
Thanks in advance.

I would recommend saving yourself the horror and going with:
http://jquery.com/
http://mootools.net/docs/core
http://dojotoolkit.org/
etc....
The libraries do a lot to smooth over the surprises of different browsers. If you are being super minimalist you can always check the source for how they are handling the differences. Also have a look at quirksmode's compatibility listing.
I know I didn't give a solid answer but you are going to run into these troubles all the time and these are some good tools for hammering them out.

IE up to IE8 does not follow W3C specs. Microsoft has their own standards. Many scripting methods that work on Firefox or Chrome (which are W3C standards) may not work properly in various builds on IE.
Why don't you try something from scratch? Either that, or do some ease of access. You can do this by making a pattern for ids and dynamically building those ids (may be incremental). Then, access those tags from their id.
Access by name is not preferred. Id is most appropriate.

Your html is invalid. the "name" property needs to be unique. Use "class" instead.
Internet Explorer might give some issues, so DOM polyfills like flowjs could be used.

Copy javascript error text to clipboard?

How do you copy javascript errors in IE to clipboard? CTRL + C doesn't work and I don't want to take screenshots. Are there any tools out there that lets you copy the text?
(ANSWER) EDIT: I finally found a machine which has IE8 on it and was able to copy the error message. It's a hassle, though.

If you were using IE8+ there is a new button called Copy error details to clipboard.
IE7 or older, have done everything that is "easy", "de facto", "cultural" to something else, just to get your job harder.

IE allows you to use the method window.clipboardData.getData() to programatically copy data. I guess you could use this in a try/catch statement. But you will have to set permissions to use it, and it will only work in IE.

These error messages in IE6/7 are rarely useful anyways, it's better to describe what you were doing when you experienced the error on which page. Because IE gives vague error messages and likes to concatenate all the javascript files together any line number that would have pointed to actual source of the problem is practically lost. Whomever is fixing the javascript is going to have to go through some grueling line by line medival debugging.
Edit: That is if the problem doesn't present in IE8, if it's present in IE8 as well, you have much better options.

Try `Ctrl + C' to copy. It works in windows' alerts.

javascript equivalent of html_entity_decode that doesn't rely on innerHTML?

I'm looking for a javascript version of PHP's html_entity_decode. I found this:
function html_entity_decode(str){
var tarea=document.createElement('textarea');
tarea.innerHTML = str; return tarea.value;
tarea.parentNode.removeChild(tarea);
}
However, I can't use this because I need to write this code for an FBML/FBJS Facebook canvas app, and they have disabled innerHTML and anything similar (insanity, I know).
Is there any other way to do this that doesn't resort to sticking the string into an element and pulling it back out again? Please make sure to only use functions that would be allowed in FBJS

I guess you'll have to do it manually. A quick Google search brought up this library that does what you want.