I am making a webscraper using perl WWW::Mechanize. My problem is the site that I am scraping is using javascript a bit too much. I am logging in using credentials, Then traversing to custom search using $mech->follow_link(url).
The problem starts from here. I am landing on to a page where I have to select one checkbox and one radiobutton from a javascript enabled dropdownlist. I am stuck at this point.
The part of html is as below. When I am using $mech->tick('cs-MajorIndustryGroup'), I am getting an error
Can't call method "find_input" on an undefined value
WWW::Mechanize doesn't support JavaScript. You could try some of these modules:
Gtk2::WebKit::Mechanize
Win32::IE::Mechanize
WWW::Mechanize::Firefox
WWW::Scripter
WWW::Selenium.
For more information see WWW::Mechanize::FAQ.
Related
I already use Laravel 8 with Yajra DataTable et wanted to test Infyom Laravel-Generator that seems to be a great package.
I created my first test app and generated a Yajra DataTable.
I see the datatable page, I was able to create a first record, the table in the database is updated, but the page shows always no record even if I refresh it.
And there is no Search input on the page.
I see nothing in the Laravel logs.
When I open my browser (firefox) in dev mode, I see no XHR request that seems to be abnormal since I see such a request on a website with datatable build without Infyom package.
Having a look in the source of the html page, no datable script is present!
So the pb should come from the genaration script, I used this cde :
php artisan infyom:scaffold Es5 --fromTable --tableName=es5 --datatables=true
I'll appreciate any help to debug/explai this.
Thanks in advance.
PS : Let me know if you want me to share some code, but now I don't know which part should be useful.
this could happen if the number of columns you defined inside your HTML table is not equal to the number of fields you are trying to fill inside your javascript code.
How did you implement it?
Did you use service implementation? I suggest to handle it in this way
I forgot to publish datatable add_on html & buttons
and found the solution here
https://github.com/InfyOmLabs/laravel-generator/issues/84
If you are using the adminlte-generator scaffold, there is an issue with the javascript is not being imported. Here is the solution:
https://github.com/InfyOmLabs/adminlte-generator/issues/87
Find #yield('third_party_scripts') in views/layouts/app and change it to #stack('third_party_scripts').
I had put my 2 days on that but the answer was so simple. make sure the yajra/datatable JS script is loading while you trying to render table
Simply change in your layout/app.blade.php
from
#yield('third_party_scripts')
to
#stack('third_party_scripts')
I understand autocomplete works like bellow:
Get all list on script in advance
Search item every time the input text changes
I don't like how this works, because the whole list exist on script so anyone can see the list using web browser's Dev Tool.
I'm wondering if there's a way to querying for every typing instead of having all list on script in advance.
Here's how I expect it to work:
Query every time the input text changes
Get the result on script.
It does not matter if the queries slow down the web site,
because the list would be not that large and security is more important here.
Any comments will be appreciated. Thanks.
Use ajax.
Create a servlet which returns the html code you need.
Send an Ajax request to the servlet.
Assign the response text to the element.
Read more,
How to use Servlets and Ajax?
I am writing an application to list or update users using Javascript. I have enabled the relevant API's in dev console. This all works ok if I use AdminDirectory.Users.list, however when I try to get a single user via AdminDirectory.Users.get and pass the google ID as the key, I get Login Required.
I have tried to add option .viewType = "domain_public" but to no avail.
For now, Im going to re-write the local function that uses .get to get the user via list, but would like to know if anyone else has this problem or know how to fix it.
I ran into the same issue when I tried to write a custom function to retrieve the users in our directory. After reading Paul's comment, I checked and found that custom functions operate with same minimal permissions, so I also added a custom menu to load the users.
At risk of giving advice without testing it myself: for triggers, you'll need to use an event with an authMode value of FULL.
See the Google Apps Script Event Objects guide.
I've been trying to teach myself how to crawl and scrape different websites. I got a good feeling about crawling/scraping, but only with websites which mainly use HTML. Now I'm working with this link https://intel.taleo.net/careersection/10000/jobsearch.ftl
I'm using Perl (with mechanize) to do the following task : I want to write a crawler/scraper to click the "United States" checkbox on the left (filtering the results) and then collect the titles of all jobs. However, I couldn't find a way to navigate to this radio button using Perl. Can someone get me started on this? (an example code would be helpful).
you need to analyise the page and see how this radio button impelented in order to use WWW-Mechanize to eumulate the Javascript code if there JavaScript code there .
also on Perl you have more easy options to handle JavaScript below some of crawling modules that handle javascript out of the box :
1.WWW-Mechanize-Firefox which automate FireFox
2.WWW-Mechanize-PhantomJS which based on PhatonJS Broweser and can handle javascript
3.WWW::Selenium which use Selenium
4.WWW::HtmlUnit which based on Java HtmlUnit and can handle javascript
I'm trying to retrieve some information from Gmail but have been unsuccessful after many attempts. This is the line of code that I'm trying to extract using javascript.
Inbox (182)
Im trying to get the text "Inbox (182)," to do that, I'm using this piece of code
NSString *js_result = [webview1 stringByEvaluatingJavaScriptFromString:#"document.getElementsByClassName('J-Ke n0').innerText"];
This however does not work, my result being nothing at all, and I've tried many alternatives but none have worked. All I need to do here is extract the "Inbox (182)" text in any way possible. Thanks.
I think your javascript is incorrect, since there are multiple elements with that class. If I login to gmail, this works:
document.getElementsByClassName('J-Ke n0')[0].innerText
I would be weary of using this in a production environment, though. It seems very brittle; that class or order of elements could be changed by Google at any time.
You also need to make sure that the page has loaded before trying to execute javascript. Typically this is implemented in a webViewDidFinishLoad: callback. If you're not getting a result and your JS is valid, this is probably the issue.