Hello I need to extract Feature table from protein page in NCBI webpage. Example of page is here:
https://www.ncbi.nlm.nih.gov/protein/Q8TB36.3
I am writing script in Python 3.x, using BeautifulSoup package for getting html of page. Problem is, that if I download html source of this page, I am not getting Feature table - Feature table is generated by javascript?
Is there any way, how to call that script and generate Feature table on my own? Or any other way how to extract desired data from page?
Thank you for advice
The JavaScript fetch data from this url:
https://www.ncbi.nlm.nih.gov/sviewer/viewer.fcgi?id=269849682&db=protein&report=genpept&extrafeat=984&fmt_mask=0&retmode=html&withmarkup=on&tool=portal&log$=seqview&maxplex=3&maxdownloadsize=1000000
You can make requests to this url and get the data you need.
You can find this url in Chrome Dev tools.
Related
Is it possible for a static website to fetch Open Graph data using plain/vanilla (no library dependencies) JavaScript on the front end (such as by XMLHttpRequest)?
For example, if a webpage were to have a list of URLs, could a script be written that upon page load would loop through the URLs, submit a request for the title, description, and other og: meta data found in each URL’s HTML content, and upon receipt of those data, then create or edit elements in the DOM to display them to the user?
All I’m finding online are libraries and packages designed for back-end usage with NPM or the like. Thank you.
I'd like to create a browser extension or a javascript to retrieve pieces of information from a currently open webpage and use that info to prefill a form on another webpage.
So, for example, if I'm on a StackOverflow page, I'd like a script that takes info from that page (title, question,...) and prefill that data in a new webpage (eg: https://stackoverflow.com/questions/ask).
I'm not an expert in coding, but I created some scripts using Python and Selenium (nothing too fancy though). I looked for a similar question, but I didn't find anything. Does anyone have an idea on how I could accomplish something like that?
You can do it with the help of content script. fetch the required data from any webpage using a content script and store the data on extension storage. Then you can inject the data to required input on required webpage with executeScript()
for more reference: https://developer.chrome.com/extensions/content_scripts
I'm trying to scrape https://stats.nba.com/players/traditional/?sort=NBA_FANTASY_PTS&dir=-1&Season=2018-19&SeasonType=Regular%20Season&LastNGames=3 with nodejs
but I don't get the same html that the one displayed on the site, I want data from the table but it is only present when you inspect element on site, I tried with request+cheerio, jquery, scrape-table but they always scrape the source html.
Yes, it's because the site is built with Angular. The page is generated with Javascript. All Request gives you is the base HTML, but it stops here. It won't fetch the scripts, won't build the DOM.
You need a more powerful tool : a headless browser like PhantomJS or Puppeteer.
These will not only load the base HTML, but also the scripts and do everything like a real browser. Then, you can extract the fully built HTML page from them.
Dears, my question is as follow:
please see my table
https://docs.google.com/spreadsheets/d/1c-oyXZHjmt69FpZI2d8NEAPh5ZAAlVljTNdeypUd38g/edit?usp=sharing
I'm trying to download list of exhibitors by import xml
And I even can download first page of it. (you could see in table).
But the matter is I can't understand how to download other pages, because table made by Javascript.
Google Sheets import functions only can import content that is available without user interaction on the source. You should check if the source allow the use of URL parameters. If so, you should make a formula for each table page.
I am working on an interactive display with sensors on a raspberry pi. My display is showing a webpage. I am using a python script to interact with the sensors. I want it so that when a user walks by, a different web page is displayed in the web browser. I have already tried changing to a different web page doing it straight from the python script. Unfortunately, it does not work the way I want it to. JavaScript works just fine for changing to a different page. I am thinking it might be good to have my python script send a message to a javascript file to change the page using JSON. Does this sound like a practical solution? Thanks.
Create your html file as a string in python. This string can be modified in response to sensor input. Write it to a file on your system where the html is source file exists; rewrite each time the sensor inputs change. Set up a timer in Javascript, e.g. setTimeout() to refresh the page often.