We have own form in our webpage. We want to show google(www.google.com/scholar) results into our webpage. We tried
jQuery, Ajax
HTML framing (iFrame, frameset)
and all of these have SAMEORIGIN issues.
We also tried using a new tab and found it has same problem here . All these solutions seem to work only with http://www.google.com/custom and no other google websites.
Is there any alternative to do so?
Google prevents you from doing this. You could set up some server side code to scrape the results but this would almost definitely be blocked by Google. If this suits your needs you can create a form that submits your search to that page and opens it in a new tab like so:
<form action="http://scholar.google.co.uk/scholar" method="get" target="_blank">
<input type="text" name="q">
<input type="submit" value="Search">
</form>
Live example here: http://jsfiddle.net/v8qG8/
Update:
I mentioned that Google would block the scraping of its services, however I just found the third party library which does that. Not sure on Google's opinion on it though. http://www.icir.org/christian/scholar.html
I answer it myself. There are no solutions so far.
You may also want to look at Google's custom API. You can get 100 free searches a day and then you pay $5 / 1000 search after that. If you cache the results or do low volume this may work for you.
Search API: https://developers.google.com/custom-search/json-api/v1/overview
All Google API's: https://developers.google.com/apis-explorer/#p/
Related
in the name of GOD. I have a js code in a page in website 1, for showing content.
<script id="mdval" src="http://web1.com/api/pc.php" type="text/javascript" dval="sourceval">
with this method I received articles and news from database in website 2 in other server.
I dont have any problem about receiving content from website 2 and News and articles showing is very good in website 1 page. But problem is here that Google not showing website 1 page in search results. I test it in Google search after a month, but not shows the page.
Please note that I can use just client side methods with AJAX.
UPDATE:
I tested page in Fetch as Google tool. I selected fetch and render for a page that uses ajax. googlebot saw the page just like visitors with all contents and images. I click on "submit to index" for indexing page. Now after one day I searched URL in google and google listed it.
But Now problem is here that google just shows url and meta description in search results and not shows any content of the page.
please search this link in google. www.neginkoodebasir.ir/more?naapi= گیاهان مورد استفاده در تهیه کود سبز (بخش دوم)
image 1
image 2
The google bot crawler does not parse your data that come from Ajax, it's only crawl the source of the page (CTRL + U on Chrome Windows).
Try to see with https://developers.google.com/webmasters/ajax-crawling/docs/learn-more
You can also try to put the data from your PHP script on cache and load the cache.
And every X hours, reload the cache.
Try to see with this tool : support.google.com/webmasters/answer/6066468?hl=en
it seems that Google CAN crawl your scripts: "Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers. To reflect this improvement, we recently updated our technical Webmaster Guidelines to recommend against disallowing Googlebot from crawling your site's CSS or JS files." https://developers.google.com/webmasters/ajax-crawling/docs/learn-more
Try to see with this tool: support.google.com/webmasters/answer/6066468?hl=en
I want to make an iframe in which i load a web page that i have on other server and i would like to fill some inputs via jquery, i know i cant do this since both apps are on different servers and have different ips, but i tried the next thing.
I made an URL rewrite on my IIS so my Host app matches 192.168.25.17 and made an url for the child app match the next ip 192.168.25.17/mysecondapp so i tried to run a jquery and it failed, so i checked the document.domain and both says its 192.168.25.17 what am doing wrong? or how can i check what is troubling me for working around the policy?
<button class="btn btn-success" id="botondeMagia">Set UserName</button>
<div id="magicContainer">
<iframe id="foo" src="http://192.168.25.17/apex/anapp.faces" height="500px">
</iframe>
</div>
<script>
$("#botondeMagia").on('click', function() {
$("#magicContainer input").val("TEST!!");
$("#magicContainer input").text("TEST!!");
});</script>
I don't know if the next info is useful but, the second app doesnt have jquery, just the hosting app.
The guest app is hosted in tomcat
Thank you very much
Actually Quentin is right, i had to use
$("#magicContainer").contents()
for viewing the contents on it, URL rewrite actually is great workaround for this tho
thank you so much guys
I am trying to use jQuery/javascript with an iframe containing google form.
Here is the code snippet:
<body>
<iframe id="myFormFrame" src="https://docs.google.com/forms/d/smfjkafj809890dfafhfdfd/viewform?embedded=true" width="760" height="1750" frameborder="0" marginheight="0" marginwidth="0">Loading...</iframe>
<button id="mybtn">Click</button>
<script type="text/javascript">
$('#mybtn').click(function(){
var m=document.getElementById("ss-form");
alert(m.innerHTML);
});
</script>
The button on click should give the innerHTML of the form(which on loading inside iFrame has an id of "ss-form"). But its not working. So what could be the reason(cross domain iframe?)?
And what is the way to make jQuery work with google forms?
If it helps, I am trying to put custom ajax validation in place(for example, the email id field must contain emails from a specific list obtained from different database).
EDIT
Well it looks like a cross domain issue. I had this doubt in mind. But after looking at this article, it looks like possible.
document.getElementById searches the current document, not other documents loaded into frames.
If the framed document was on your own site then you could access the frame and then access the document (via contentDocument) first. Then call getElementById on that.
Since it is cross-origin, that will fail. The reasons for this are obvious if you consider what would happen if Mallory's evil website were to stick your bank's online banking site into a frame while you were logged in.
If you had the cooperation of Google (you don't) then you could pass messages between the frames using postmessage.
Well it looks like a cross domain issue. I had this doubt in mind. But after looking at this article, it looks like possible.
They aren't using a form hosted by Google. They've used Google tools to design the form and then copied the HTML to their own site. If Google ever implement CSRF protection, that will break.
I am scraping profiles on ask.fm for a research question. The problem is that only the top most recent questions are viewable and I have to click "view more" to see the next 15.
The source code for clicking view more looks like this:
<input class="submit-button-more submit-button-more-active" name="commit" onclick="return Forms.More.allowSubmit(this)" type="submit" value="View more" />
What is an easy way of calling this 4 times before scraping it. I want the most recent 60 posts on the site. Python is preferable.
Without using Headless Browser
Open the chrome debugger tools -> Network Tab.
Now click on the View More button.
Inspect the request being fired in the Network dialog when you click on View More.
The data would be loaded from an external API in most cases so check whether the request to the API is a get or a post request and the kind of response from it.
There could be a limit or any similar query parameter which could be passed to that url so as to limit the no of response objects. In your case, it would be 15.
Try to make the request to the same URL from your script increasing the limit to 60 let's say and check the response.
The above technique works in most cases. But if this doesn't work for you try the following steps.
Using Headless Browser
Try to use a headless browser which loads the dynamic content, you have the methods to scroll down, click etc available.
Examples of Headless browsers are Selenium, Splash, PhantomJS, SlimmerJS etc.
You could probably use selenium to browse to the website and click on the button/link a few times. You can get that here:
https://pypi.python.org/pypi/selenium
Or you might be able to do it with mechanize:
http://wwwsearch.sourceforge.net/mechanize/
I have also heard good things about twill, but never used it myself:
http://twill.idyll.org/
I saw some guy had a file (I guess a batch file). On clicking of the batch file he was able to log in to multiple sites. (Perhaps it was done using VB.)
I looked for such a script on Google but didn't find anything useful.
I know a bit of C++ and UNIX (also some HTML and JavaScript). I don't know if it can be done on a windows machine using these languages, but even if it could be done I think it would be difficult compared to VB or C## or some other high level languages.
I learned how to open multiple sites using basic windows batch commands enclosed in a batch file like:
start http://www.gmail.com
start http://stackoverflow.com
But still I can't figure out how actually clicking on the batch file would help me to log in to the sites without even typing the username and password.
Do I need to start learning Visual Basic, .NET, or windows batch programming to do this?
One more thing: can I also use it to log in to remote desktops?
From the term "automatic login" I suppose security (password protection) is not of key importance here.
The guidelines for solution could be to use a JavaScript bookmark (idea borrowed form a nice game published on M&M's DK site).
The idea is to create a javascript file and store it locally. It should do the login data entering depending on current site address. Just an example using jQuery:
// dont forget to include jQuery code
// preferably with .noConflict() in order not to break the site scripts
if (window.location.indexOf("mail.google.com") > -1) {
// Lets login to Gmail
jQuery("#Email").val("youremail#gmail.com");
jQuery("#Passwd").val("superSecretPassowrd");
jQuery("#gaia_loginform").submit();
}
Now save this as say login.js
Then create a bookmark (in any browser) with this (as an) url:
javascript:document.write("<script type='text/javascript' src='file:///path/to/login.js'></script>");
Now when you go to Gmail and click this bookmark you will get automatically logged in by your script.
Multiply the code blocks in your script, to add more sites in the similar manner. You could even combine it with window.open(...) functionality to open more sites, but that may get the script inclusion more complicated.
Note: This only illustrates an idea and needs lots of further work, it's not a complete solution.
The code below does just that. The below is a working example to log into a game. I made a similar file to log in into Yahoo and a kurzweilai.net forum.
Just copy the login form from any webpage's source code. Add value= "your user name" and value = "your password". Normally the -input- elements in the source code do not have the value attribute, and sometime, you will see something like that: value=""
Save the file as a html on a local machine double click it, or make a bat/cmd file to launch and close them as required.
<!doctype html>
<!-- saved from url=(0014)about:internet -->
<html>
<title>Ikariam Autologin</title>
</head>
<body>
<form id="loginForm" name="loginForm" method="post" action="http://s666.en.ikariam.com/index.php?action=loginAvatar&function=login">
<select name="uni_url" id="logServer" class="validate[required]">
<option class="" value="s666.en.ikariam.com" fbUrl="" cookieName="" >
Test_en
</option>
</select>
<input id="loginName" name="name" type="text" value="PlayersName" class="" />
<input id="loginPassword" name="password" type="password" value="examplepassword" class="" />
<input type="hidden" id="loginKid" name="kid" value=""/>
</form>
<script>document.loginForm.submit();</script>
</body></html>
Note that -script- is just -script-. I found there is no need to specify that is is JavaScript. It works anyway. I also found out that a bare-bones version that contains just two input filds: userName and password also work. But I left a hidded input field etc. just in case. Yahoo mail has a lot of hidden fields. Some are to do with password encryption, and it counts login attempts.
Security warnings and other staff, like Mark of the Web to make it work smoothly in IE are explained here:
http://happy-snail.webs.com/autologinintogames.htm
I used #qwertyjones's answer to automate logging into Oracle Agile with a public password.
I saved the login page as index.html, edited all the href= and action= fields to have the full URL to the Agile server.
The key <form> line needed to change from
<form autocomplete="off" name="MainForm" method="POST"
action="j_security_check"
onsubmit="return false;" target="_top">
to
<form autocomplete="off" name="MainForm" method="POST"
action="http://my.company.com:7001/Agile/default/j_security_check"
onsubmit="return false;" target="_top">
I also added this snippet to the end of the <body>
<script>
function checkCookiesEnabled(){ return true; }
document.MainForm.j_username.value = "joeuser";
document.MainForm.j_password.value = "abcdef";
submitLoginForm();
</script>
I had to disable the cookie check by redefining the function that did the check, because I was hosting this from XAMPP and I didn't want to deal with it. The submitLoginForm() call was inspired by inspecting the keyPressEvent() function.
You can use Autohotkey, download it from: http://ahkscript.org/download/
After the installation, if you want to open Gmail website when you press Alt+g, you can do something like this:
!g::
Run www.gmail.com
return
Further reference: Hotkeys (Mouse, Joystick and Keyboard Shortcuts)
Well, its true that we can use Vb Script for what you intended to do.
We can open an application through the code like Internet Explorer. We can navigate to site you intend for. Later we can check the element names of Text Boxes which require username and password; can set then and then Login. It works fine all of using code.
No manual interaction with the website. And eventually you will end up signing in by just double clicking the file.
To get you started :
Set objIE = CreateObject("InternetExplorer.Application")
Call objIE.Navigate("https://gmail.com")
This will open an instance of internet explore and navigate to gmail.
Rest you can learn and apply.