How to pull dynamic data from a javascript-based website? - javascript

I would like to pull the data from a site like this. The idea is to build a Google Chrome extension that can pull the data and maybe inject some javascript into the browser with statistics of each players action frequencies while the table is open.
I have noticed that if you use the Google devtools and 'inspect' a table while it is running, that the data changes with every players action. This is the data that would be useful to retrieve. What is the best way to do accomplish this task? Would it be to build a Javascript Chrome extension? Is this project too complicated for someone beginning in Javascript but has a few months in Python?
Thanks in advance!

Related

Using speech recognition for easy access to website functions(menu, options, etc.)

So, I'm working on a pretty simple project right now where I need to use speech recognition to access the functions of a website.
I made a simple website with just 6 buttons(menu, options, settings, etc.) and I trained a model to recognize these 6 words. Now I need to link the Pycharm script to the website and because I'm very limited in JavaScript I'm stuck at this step.
I created a button that works exactly like an audio recorder, after I press the button it records what I say and then I can download the output manually. But I want it to download automatically and then send the recording as an input to my script.
If anyone has any ideas on how to do it or if you think that I approached it the wrong way please tell me.
Thank you!

Is it possible to make a script to automate that web scraping scenario?

I need a script to automate the following web scraping process:
Use search filter on the webpage by entering a pre-given ID in a TextInput field.
Mark one of the listed results by clicking an icon next to it.
Press a button and wait for a data fetching process to complete
Extract HTML embedded text from the fetched data.
For Python I would recommend:
Selenium: Selenium is a web testing library. It is used to automate browser activities.
BeautifulSoup: Beautiful Soup is a Python package for parsing HTML and XML documents. It creates parse-trees that are helpful in extracting data easily.
I would recommend the Selenium library for Python or Java to accomplish this task.
All of your points should be possible with this library as far as I know.

Pdf files editing and saving in browser

The scenario is that I have several editable pdfs that need to be displayed edited and saved all in a web based application.
I know this question has been asked before but I'd like to review the options. I work as a junior developer and I am being asked to research this. I don't think its possible as the pdfs are opened and viewed in different browser plugins that my code doesn't have access to but I still would like to know for sure before I report back to my superior. Is there any way I can tap into some API that allows for editing and saving on a webpage?
you could try this :-
A general-purpose, web standards-based platform for parsing and rendering PDFs
https://mozilla.github.io/pdf.js/

Load and parse URL via JS in the background

Currently I am trying to develop a little Firefox extension.
In detail: i want to display users from the site dota2lounge.com the current prize of their steam items on the steam community market. My idea was to do this via a Firefox extension which reads the item names from the HTML code on dota2lounge.com . Via JS i would like to search the steam community market for the item names and parse the current prize. This should happen without any further action from the user and without opening extra tabs/windows.
In java i would just load the site into a variable and work with it. How could i do this with JS (or Jquery)? Or maybe there is an even better way in the addon-sdk from firefox which could solve this issue.
Any thoughts and hints are welcome.
This should be pretty simple to do using the Add-on SDK. Here is a list of modules you should look at:
the request module will allow you to make requests to other sites: https://developer.mozilla.org/en-US/Add-ons/SDK/High-Level_APIs/request
while the request module is fine, what you may want to do instead to get info from the steam site is use the page-worker module to load the site and easily extract info from it using jQuery. This is much nicer than using regex. The code would look something like this gist:
https://gist.github.com/canuckistani/6c299c812bbe582d9efb

Screen scrape a web page that uses javaScript and frames

I want to scrape data from www.marktplaats.nl . I want to analyze the scraped description, price, date and views in Excel/Access.
I tried to scrape data with Ruby (nokogiri, scrapi) but nothing worked. (on other sites it worked well) The main problem is that for example selectorgadget and the add-on firebug (Firefox) don’t find any css I can use to scrape the page. On other sites I can extract the css with selectorgadget or firebug and use it with nokogiri or scrapi.
Due to lack of experience it is difficult to identify the problem and therefore searching for a solution isn’t easy.
Can you tell me where to start solving this problem and where I maybe can find more info about a similar scraping process?
Thanks in advance!
I used excel web query and it works perfect. You can find a lot about scraping with excel on youtube if you search for mrexcel.
Thanks, Mello
You can try IRobotSoft web scraper. It has good frame support and is free.
Iframes aren't a problem - just access the embedded iframe URL directly. You will find that it redirects in the browser unless you disable JavaScript.
Description and date can be extracted straight from HTML source. However prices are images which will make scraping them more cumbersome.

Categories