I would like to scrape a web page but I can't be detected as a bot. I am using js to fetch data, fill inputs or click buttons.
I have read that I have to take into account some "diplayed: none" attributes as they seems to be honeypots. In my webe there is a div which shows this:
// When mouse is not used
<div style="position: absolute; top: 0px; left: 0px; display: none; z-index: 10000;"><div class="tip">
// When mouse has moved but not on any clickable point
<div style="position: absolute; top: 264px; left: 272px; z-index: 10000; display: none;"><div class="tip">
// When mouse on a clickabel point
<div style="position: absolute; top: 264px; left: 272px; z-index: 10000;"><div class="tip">
This make me think the web check if the click has been done programmatically or if it is from a real user.
Thereby
Is this something I have to work with or it is insignificant??
If so, how could I bypass it??
Thank you very much
display:none; isn't your only problem, one could also use negative left:-100px; or top:-100px; values, and/or color:white; on a white background, and so forth.
But they are all foolhardy attempts to prevent bots when all they had to do is this...
Luckily, JavaScript already provides a mechanism for determining if an event was user-initiated via the "isTrusted" boolean property. Usage is as follows...
<button id="logon" onclick="if(event.isTrusted){SomeFunction();}">Logon</button>
Any JavaScript attempt to click this button such as logon.click(); would fail this test and the function would never run, and no there's no way you can simulate a human gesture.
Hackers exploit bad programmers who use foolish tricks like the ones you and I mentioned.
Good point raised by Dave, but the version posted was for clarity purposes.
See if you can fool this little modification...
<!DOCTYPE html>
<head>
</head>
<body>
<script type="text/javascript">
function SomeFunction(event){
if(event.isTrusted){
alert('This came from a human');
} else {alert('This came from a bot');}
}
</script>
<button id="logon" onclick="SomeFunction(event);">Logon</button>
<button onclick="logon.click();">BOT clicking button</button>
<button onclick="SomeFunction(true);">BOT running function directly</button>
</body>
</html>
NB: For the last button, we are simulating a BOT here through a button so we have the luxury of passing the human onclick event to get through, but that won't be the case for a BOT and hence the boolean variable I added.
Related
Is there any way of freezing the browser window intentionally like alert, confirm and prompt (in short: "acp") does?
I want to show a modal dialog with custom css instead of the native popups for "acp" but also want to have the ability to freeze the browser until I have users feedback just like "acp".
But man, why? This is bad practice (uh I have to downvote)!
So when it is bad practice - then why does "acp" actually offer this synchronous behavior? Because in some particular scenarios its just exactly the right tool for an appropriate UX. Those native modals do look so ugly and are also very limited at the same time.
Here's just one quick and dirty example where it would be totally fine to freeze the browser until the user gives feedback. Lets say we have a form with an input[type="reset"]-element. So before we really let the form reset we ask the user something like: "Are you sure you want to reset (data will be lost)?".
If I would be fine with the native modal look (which I'm not) I could do:
document.querySelector('form').addEventListener(
'reset', e => confirm('Are you sure you want to reset (data will be lost)?') || e.preventDefault()
);
<form>
<input placeholder="type something, reset and cancel to keep input value" style="width:100%">
<input type="reset">
</form>
So "everybody" should agree (edit: or at least somebody could) this isn't bad practice, right?
But how can we achieve the same with a custom styled modal/dialog/popup?
I'm -of course- not asking for the HTML/CSS part but for the capability of freezing the browser window via JavaScript!
To be honest, I actually expect some downvotes for this but maybe there is this one Killer JavaScript Ninja, who have this one special hack to make it possible...
You can't freeze the browser application like the native dialogs do. Those are not built with JavaScript, they are built with native code and can affect the browser application in any way. JavaScript is prohibited from affecting the client application in such ways.
But, you can freeze interactions with your page content within the browser window....
Just place a window sized div above the page with fixed positioning. That will prevent the user from being able to interact with anything on the main page (behind it). Then, display your modal dialog on top of that. When the user clears the modal, hide it and the window sized div, thus making the main page interactive again.
let modal = document.querySelector(".modal");
let pageCover = document.querySelector(".pageCover");
let main = document.querySelector("main");
document.getElementById("open").addEventListener("click", function(){
modal.classList.remove("hidden");
pageCover.classList.remove("hidden");
main.addEventListener("focus", preventFocus);
});
document.getElementById("close").addEventListener("click", function(){
modal.classList.add("hidden");
pageCover.classList.add("hidden");
main.removeEventListener("focus", preventFocus);
});
function preventFocus (evt){
evt.preventDefault();
}
.hidden { display:none; }
.pageCover {
position:fixed;
z-index:10;
background-color:rgba(0,0,0,.25);
width:100vw;
height:100vh;
top:0;
left:0;
}
.modal {
position:absolute;
z-index:100;
background-color:aliceblue;
border:2px double grey;
width:200px;
height:200px;
top:30%;
left:30%;
text-align:center;
}
.modalTitle {
margin:0;
background-color:#00a;
color:#ff0;
padding:5px;
font-weight:bold;
font-size:1.1em;
}
#close {
background-color:#800080;
color:#ff0;
border:1px solid #e0e0e0;
padding:5px 10px;
}
#close:hover { background-color:#c000c0; }
<main>
<h1>This is some page content</h1>
<p>And more content</p>
<button id="open">Click Me To Show Modal</button>
<div>More content</div>
</main>
<div class="hidden pageCover"></div>
<div class="hidden modal">
<div class="modalTitle">Modal Title Here</div>
<p>
Now, you can only interact with the contents of this "modal".
</p>
<button id="close" >Close</button>
</div>
You can place everything within a div except for your popup. You can then turn on/off pointer events via css.
Everything within the div will no longer be interactable. This will give the appearance of freezing the browser window.
.freeze { pointer-events: none; }
Note: When adding pointer-events to an element this affects all child elements as well, so placing it on the body would also lock the popup.
Once the popup has been closed, we can remove the freeze class from the div and everything will start working again.
const content = document.getElementById('content')
const overlay = document.querySelector('.overlay')
const a = document.querySelector('.open-overlay')
const close = document.querySelector('.overlay .close')
// Open a popup and freeze the browser content
a.addEventListener('click', e => {
e.preventDefault()
content.classList.add('freeze')
overlay.classList.remove('hidden')
})
// Close the popup window and unfreeze the browser content
close.addEventListener('click', e => {
e.preventDefault()
content.classList.remove('freeze')
overlay.classList.add('hidden')
})
.freeze { pointer-events: none; }
.hidden { display: none; }
.overlay {
position: fixed;
z-index: 100;
padding: 20px;
background: white;
border: solid 1px #ccc;
width: 300px;
top: 20px;
left: 0;
right: 0;
margin: auto;
}
<div class="overlay hidden">
<h1>Overlay</h1>
Close
</div>
<div id="content">
Open Overlay
<p>Hello World</p>
<p>Hello World</p>
<p>Hello World</p>
<p>Hello World</p>
<p>Hello World</p>
<form>
<input type="text">
</form>
</div>
I am new to JavaScript. Currently, I am working on a small toggle for my website.
The goal is to have three buttons that open up different sections with information. I have this working on my website. Now, what I want to achieve is to make other divs close when the others are opened up. Furthermore, I would like the first div to be open when the page is loaded, including an indicator (for example orange image) on the button. Can you please help me with this?
For some reason, the script works on my website, but not on the JSfiddle:
https://jsfiddle.net/q7evaLsn/1/
Current code:
$('.button1').click(function(){
$('.product').slideToggle('slow');
});
$('.button2').click(function(){
$('.lockedin').slideToggle('slow');
});
$('.button3').click(function(){
$('.developers').slideToggle('slow');
});
.button2
{
padding-top: 10px;
}
.button3
{
padding-top: 15px;
}
<h3>
<img src="http://www.mindaffect.nl/wp-content/uploads/2017/12/product-holder.png" class="button1" alt="Expand"/>
</h3>
<h3>
<img src="http://www.mindaffect.nl/wp-content/uploads/2017/12/lockedin-holder.png" class="button2" alt="Expand"/>
</h3>
<h3>
<img src="http://www.mindaffect.nl/wp-content/uploads/2017/12/developers-holder.png" class="button3" alt="Expand"/>
</h3>
<div class="product">
Testdiv1
</div>
<div class="lockedin">
Testdiv2
</div>
<div class="developers">
Testdiv3
</div>
Your help is greatly appreciated!
You can simply slide up everything before you start toggling.
For ex
$('.button3').click(function(){
$('.product').slideUp();
$('.lockedin').slideUp();
$('.developers').slideToggle('slow');
});
Your JSfiddle isn't working because you haven't included the jQuery library required for some of your functions. For future reference, jQuery is a popular javascript library which simplifies and extends some basic javascript functions, you can use both interchangeably however if you do want the extra features of jQuery then you'll have to include it like so in your HTML:
<script src="https://code.jquery.com/jquery-2.2.4.min.js"></script>
As mentioned by #SURESH you'll likely want to slide the other areas up where you are toggling the target area:
$('.example-button').click(function(){
$('.section-to-hide-1').slideUp();
$('.section-to-hide-2').slideUp();
$('.section-to-toggle-1').slideToggle();
});
Just as further formatting advice, you have your images (that are acting as buttons) within header tags.
It's generally bad practice to use these header tags for anything
other than headings/titles
I'd recommend using A tags or even BUTTON tags to do the same job
I'd try not to use IMG tags as essentially text buttons, you will be able to style a button similarly like so:
<button class="button1">Products</button>
<style>
.button1 { text-align: center; padding: 10px; text-transform: uppercase: border-radius: 100%; border: 3px solid orange; background: white; color: #000; }
</style>
This will allow search engines/screen readers to read your button element, and you can make hover effects etc.
I want to check if our advertisement(picture or flash) has successfully displayed on website page.But the element can not be found in page source.
I get the element in chrome Developer tools.
<div id="QQcom_all_Rectangle:1" data-loc="QQcom_all_Rectangle" data-index="1" style="height: 250px; display: block; width: 300px; position: relative;" class="l_qq_com" adconfig_lview="l.qq.com" adconfig_charset="gbk" adconfig_lview_template="http://l.qq.com/lview?c=www&loc={loc}" oid="1800716433" display="banner">
<a class="absolute a_cover" href="http://c.l.qq.com/lclick?loc=QQcom_all_Rectangle&click_data=dXNlcl9pbmZvPW9CM2pnVGd4RnhHNyZhZHhfZXh0PSZwY3RyPTUwMSZhdmVyPTUwMTIwMSZwcmk9eHRiQkZXc0ovclloYXdzdWZqRmpSTkhyZWFuL3pQU2omYnRwcmk9R3VBL25heHhnY3JyNTdrdVNCNW4yWis3TlJqM01nTmw=&oid=1800716433&soid=gmLndBibVj/1bQtQRjKVNkVKAV1Q&dtype=0&pctr=501&aver=501201&btoid=100418428&pri=xtbBFWsJ/rYhawsufjFjRNHrean/zPSj&btpri=GuA/naxxgcrr57kuSB5n2Z+7NRj3MgNl&index=1&page_type=2&chl=703&k=%E6%9B%9D%E7%81%AB%E7%AE%AD%E5%B7%B2%E8%A3%81%E6%8E%89%E6%B5%B7%E8%80%B6%E6%96%AF%20%E5%9B%9E%E5%BD%92%E4%BC%91%E5%9F%8E%E4%BB%85%E4%B8%80%E5%91%A8%E5%BE%81%E6%88%982%E5%9C%BA%2C%E6%B5%B7%E8%80%B6%E6%96%AF%2C%E7%81%AB%E7%AE%AD%2CNBA&t=%E6%9B%9D%E7%81%AB%E7%AE%AD%E5%B7%B2%E8%A3%81%E6%8E%89%E6%B5%B7%E8%80%B6%E6%96%AF%20%E5%9B%9E%E5%BD%92%E4%BC%91%E5%9F%8E%E4%BB%85%E4%B8%80%E5%91%A8%E5%BE%81%E6%88%982%E5%9C%BA_&r=&s=" target="_blank" rel="nofollow" style="position:absolute;width:300px;height:250px;left:0px;top:0px;cursor:pointer;z-index:10;background-color:#fff;filter:alpha(opacity=0);opacity:0;"></a>
<div class="absolute" style="position: absolute; width: 24px; height: 16px; left: 26px; bottom: 0px; cursor: pointer; display: none; z-index: 20; background: url(http://ra.gtimg.com/web/res/icon/report_default_new.png) 50% 0% no-repeat;"></div>
<div style="position: absolute; left: 0px; bottom: 0px; width: 26px; height: 16px; z-index: 12; background: url(http://ra.gtimg.com/web/res/icon/leftbottom_new.png) 100% 0% no-repeat;"></div>
<div class="absolute" style="position:absolute;width:18px;height:18px;right:0px;bottom:0px;cursor:pointer;z-index:20;background:url(http://ra.gtimg.com/web/privacy/white_icon.png) no-repeat;"></div>
</div>
but in page source,it only shows
<!--$loc$_div AD begin...."l=$loc$&log=off"--><div id="QQcom_all_Rectangle:1" data-loc="QQcom_all_Rectangle" data-index="1" style="height:0;" class="l_qq_com"></div><!--$loc$ AD end --><!--[if !IE]>|xGv00|c5668531d36ed7899852180841ca2aa2<![endif]-->
how can I get the 300px_250px image url?
anyone knows?
Yeah! That's simply because of the fact that there are DOM manipulation libraries which alter the Document-Object Markup and not the source.
You have to understand the difference between the DOM and the physical page source. The physical page source helps render the DOM, after which it can be modified by using libraries; since you can't change the source on the server side as it needs to be re-rendered for different people, only the DOM is changed. In very simple words, DOM is what the browser has rendered: it's like a copy of the source which has been, then, modified by the libraries.
Using a library like jQuery can do this easily.
var $element = $( "#QQcom_all_Rectangle:1" );
if ( !$element || $element === null ) {
console.log( "The element hasn't been rendered. Not found.");
}
But if it is not YOUR page, and you can't really modify it and/or get the data. You can try doing this with a web-kit emulator like PhantomJS or Selenium web-kit. Since these are just testing frameworks, you can't create a fully fledged JavaScript applications.
Theoretically, you can:
Render this in a WebKit (Browser Rendering Component)
Get the source of the DOM
Use it
But that's just in theory because you need to use some language to create an application like that and since it's outside the scope of your question, you can't.
Using selenium webdriver can handle this.
WebElement QQcom_all_Rectangle=driver.findElement(By.id("QQcom_all_Rectangle:1"));
List<WebElement> links=QQcom_all_Rectangle.findElements(By.tagName("a"));
String style=links.get(0).getAttribute("style");
I have a situation where clicking on an image will direct the user to a certain link, but pressing a button that is shown within an image will run a javascript method instead. However, I cannot prevent the page from redirecting to the certain link when the button is pressed (the javascript method is also run when the button is clicked).
I have found out that button cannot be nested within an anchor element, and tried to wrap the button within a form as well, but no luck.
Does anyone know a way around such problem?
the basic logic in code looks like this
<a href="an item description link">
<img src="an item image"/>
<form style="display: inline" action="html_form_action.asp" method="get">
<button type="button" id="add-btn" class="add-cart" onclick="quick_add()">+</button>
</form>
</a>
Thanks in advance for any help!
A straightforward way that validates would be just superimposing the button over the link. This requires the link and the button to be in the same containing element, and for both of them to use position: absolute:
HTML
<div class="box">
<a href="http://example.com">
<img src="http://placehold.it/300x200">
</a>
<button>AAAAA</button>
</div>
CSS
.box {
box-sizing: content-box;
width: 300px;
height: 200px;
border: thin solid black;
}
.box > a {
display: block;
position: absolute;
}
.box > button {
position: absolute;
}
See it in action on CodePen: http://codepen.io/millimoose/pen/avYLjQ
The button will automatically be stacked over the preceding link. (This is specified behaviour.) And it will handle clicks before they can be passed to elements underneath is.
That said, this solution has a few downsides. You'll have to give a fixed size to the container; it can't be sized automatically to fit its contents, because its contents are outside of the rendering flow. This also means they won't automatically fill their parent box unless you set their size explicitly again.
I am developing a web application for a router, and I have two buttons, Scan and Reboot in two different pages . The function for scan is to scan all the connected devices and displaying ( using php ), and that for Reboot is to reboot the whole system. I am using openwrt. I am planning to use a scanning spinner and a rebooting spinner in these pages while it is doing these processes. I got this reference and I tried this but I guess it is not working for the scan and reboot. So Is there a way to make this work using pure JS. (I got images from this link)
EDIT
Here is my code for reboot page
JS
function spin(){
window.onload=document.getElementById("loading").style.display = "none";
}
CSS
#loading {width: 100%;height: 100%;top: 0px;left: 0px;position: fixed;display: block; z-index: 99}
#loading-image {position: absolute;top: 40%;left: 45%;z-index: 100}
HTML PHP
<body onload="spin();">
<div id="loading">
<img id="loading-image" src="http://jimpunk.net/Loading/wp-content/uploads/loading1.gif" alt="Loading..." />
</div>
<form name="reboot" style="height: 56px;" action="reboot.php" method="post">
<input type="submit" name="reboot" value="Reboot">
</form>
<?php
if (isset($_POST['reboot']))
shell_exec("reboot");
?>
Okay then, lets see now. Where to start?
A number of issues seem apparent to me when I review your method of achieving the task.
1. - you never actually do anything to make the image visible.
2. - you use both window.onload and body.onload
3. - you use the CSS to make the image visible, yet by default, use Javascript to then immediately hide it again. A better option would be to hide it with the css, using the javascript to show it only if that's the appropriate action.
4. - When you click the button, you ask for the same file again - passing Reboot=reboot as a POST parameter. This is all fine and good. But begs the question - how will you then indiate that the reboot has completed? You just get left with a page that says "loading" - it seems rather confusing.
I'm too tired to even consider thinking about a better way to approah the problem. There's likely several solutions to the problem already available.
I suspect that using ajax to periodically poll the server, waiting for any response would be a reasonable way to see when the router is online again. I've only provided a means by which you can show the image if the page is loaded as a result of pressing the button.
reboot.php
<!DOCTYPE html>
<html>
<head>
<script>
</script>
<style>
#loadingDiv
{
width: 100%;
height: 100%;
top: 0px;
left: 0px;
position: fixed;
display: none;
z-index: 99
}
#loading-image
{
position: absolute;
top: 40%;
left: 45%;
z-index: 100
}
</style>
</head>
<body>
<div id="loadingDiv">
<img id="loading-image" src="http://jimpunk.net/Loading/wp-content/uploads/loading1.gif" alt="Loading..." />
</div>
<form name="rebootForm" style="height: 56px;" action="reboot.php" method="post">
<input type="submit" name="reboot" value="Reboot"/>
</form>
<?php
if (isset($_POST['reboot']))
{
shell_exec("reboot");
echo "<script>document.getElementById('loadingDiv').style.display = 'block';</script>";
}
?>
</body>
</html>