I'm sorry if the title is confusing.
I'm trying to scrape my job's website to updates on my schedule. I'm using Ruby with Mechanize. I've successfully logged into my job's website, though, once logged in, all of the links going forward are JavaScript links, which Mechanize doesn't play well with.
After doing some research, I saw some suggestions for similar problems; Instead of clicking the link, I can emulate the request the link is making, though I'm not sure at all how I could do this.
The code behind the link is:
My Work Schedule
How might I tackle this in Ruby? Any suggestions?
When the link is clicked, it submits a form.
To accomplish the same thing with mechanize:
page.form(:id => 'schedForm').submit
As far I know, Mechanize can't make JavaScript calls. If you need to interact with JavaScript, I would recommend Watir.
Related
First off, go easy on me. While I have been doing extremely basic coding since I was a child, it's only been over the last few months that I started getting into the real stuff like Java, Javascript, PHP/MySQL, etc. So please bear with me if I use wrong terminology or write something that sounds completely wrong. I'd love to get some insight from all you tireless helpers here.
Here is what I need:
I have a website at myserver.com.
I want to use the API at randomsite.com to automate a process.
If I manually go to the URL randomsite.com?do=this, my process will be triggered.
What I want to be able to do is write an app on myserver.com that will trigger randomsite.com?do=this. Obviously I can have a link on my page, but the user will see that they have been directed to randomsite.com.
I do not need to capture anything about the randomsite.com link having been triggered. I don't care who clicked it, etc. All I want is to simulate the effects of going to randomsite.com?do=this but without actually opening that site on the front end.
Thank you so much for your help!
Consider using AJAX for doing stuff behind the scenes...
https://api.jquery.com/jQuery.ajax/
EDIT: You would need to use JSONP for cross domain.
I have to create a web page, where all activity of some user will be seen by second user.
Simply, the second user see everything what the first user is doing at his page.
I know, that It is possible using AJAX or Websockets, but I' m interested in simpler solution.
Do you know any simple solution, which enables to intercept frame of another user?
I had browsed many pages, but I haven't found satisfactory solution.
I assume that I control the source code of this application.
I greatly appreciate your help.
Check out Mozillas TogetherJS.
It is a solution for users interacting with a page together in realtime; it might do what you are trying to achieve.
Im new to ajax, jquery and javascript. When i learnt ajax and jquery, i also learned about back button issues and of course i learnt it too. But this solution uses the # anchors. But I want to change the url to the actual page just like to facebook. If you use facebook, you may have noticed that when you click an image, the url gets changed to the url of image but the page doesnt navigates away.
If Any one know about facebook like solution then please answer it.
I know you don't want to hear this, and I'm also new to AJAX and I'm also working on this right now, but i can give you the advise to use hashes first. Why?
Because hashes are accpted in any browser. If i were you I would now work out a solution that can be used with or without ajax (so there's a 'normal' navigation if ajax is not available, because web crawlers don't understand javascript).
If you have successfully worked out your solution, THEN you can add some functions to use the javascript pushState() (or replaceState()) method to manipulate the url.
Don't underestimate this and use hashes at the beginning.
You are looking for Hash Navigation, also known as Fragment Navigation. Google has a very good article on this.
Here is a demo from them: http://gtv-resources.googlecode.com/svn/trunk/examples/location-hash-html.html
I'm going to be accessing a number of accounts on Amazon's KDP - http://kdp.amazon.com/
My task is to login to each account and check the account's earnings. Mechanize works great for logging in and dealing with the cookies and such but the page which displays the account earnings uses javascript to dynamically populate the page.
I did a little bit of digging and found that the javascripts sends out the following request:
https://kdp.amazon.com/self-publishing/reports/transactionSummary?_=1326419839161&marketplaceID=ATVPDKIKX0DER
Along with a cookie which contains a session ID, a token, and some random stuff. Every time I click a link to display the results, the numerical part of the above GET url is different, even if it's the same link.
In response to the request, the browser then receives this (cut out a bunch of it so it doesn't take up the whole page):
{"iTotalDisplayRecords":13,"iTotalRecords":13,"aaData":[["12/03/2011","<span
title=\"Booktitle\">Hold That ...<\/span>","<span
title=\"Author\">Amy
....
<\/span>","B004PGMHEM","1","1","0","70%","4.47","0.06","4.47","0.01","0.00",""],["","","","","","","","","","","","","<div
class='grandtotal'>Total: $ 39.53<\/div>","Junk"]]}
I think I can use mechanize's cookie container to extract the cookies which are a part of that request but how do I figure out what that number is and how it's generated? The javascripts in the source code of the page seem cryptic on the best of days. Here's one of them:
http://kdp.amazon.com/DTPUIFramework/js/all-signin-thin.js
Is there a way to really track down what javascripts are running "behind the scenes" so to speak after I click on something on the page so that I can emulate that request in conjunction with mechanize?
Danke..
PS: I can't (or, rather, I don't want to) use watir for this task, because in theory I might be handling more than just a handful of accounts so this's gotta be pretty snappy.
It's just a timestamp and it's only used for cache busting. Try this:
Time.now.to_i.to_s
Mechanize doesn't run JavaScript that is embedded in the page. It only retrieves the HTML.
If the page contains JavaScript, Mechanize can see it and you can use Nokogiri, which Mechanize uses internally, to retrieve the <script> tags' content. But, anything that would be loaded as a result of the JavaScript being executed in a browser will not run in Mechanize. Watir is the solution for that, because it drives the browser itself, which will interpret and run the JavaScript in the page.
You can step through the pages in a browser and look at the source code to get an idea what is running using FireBug. From that information you can get an understanding of what the JavaScript is doing, and then use Mechanize and Nokogiri to extract the needed information from a page that lets you build up your next URLs, but it can be a lot of work.
What you ask is similar to many other's questions regarding Mechanize and JavaScript. I'd recommend you look at these SO links to get alternate ideas:
Mechanize and JavaScript
Ruby Mechanize not returning Javascript built page correctly
Or search Stack Overflow for questions about Ruby, JavaScript and Mechanize.
I'm specifically looking, how is the hover on done? and what did they use? JavaScript? Jquery, DHTML? and does anyone have example?
Now, I'm new to web programming. (but not new to programming, I'm c++ and c# prg).
I'm looking into getting web dev. for personal use. I saw this amazon feature and thought to myself that it would be nice to do something like this (i.e. menu) on my own website and inline editing, but I'm not sure where and how to start.
I'm looking for resources, books and general examples. I know that I could google, but I'm not sure what to google, I'm not sure what's it called and what to look for.
Thanks guys.
EDIT: Example of it:
http://www.amazon.com/wishlist/ref=topnav_lists
Go to amazon wishlist -> click on any of your wishlists.
Then in the middle of the screen you should see "Manage this list"
get yourself a firefox browser and install add on firebug.
when you enter the page open firebug(F12)
and there you will be able to follow what is happening on the page in most times
usually you will se something as ajax requests loading pages
after scripts and loads more. I recomend to read up a bit about javascript and html for beginning.
the menu what do you see there is javascript action, or jquery (i did not looked too close)
the problem as david wrote is, you have to be more specific what you want to achieve:)
good luck