I'm trying to get the https://www.cleverbot.com page content with puppeteer. But when I try:
const puppeteer = require("puppeteer");
(async () => {
// Launches browser
let browser = await puppeteer.launch();
console.log("browser opened");
let page = await browser.newPage();
console.log("page opened");
// Go to cleverbot
await page.goto("https://www.cleverbot.com/");
console.log("cleverbot.com loaded");
})();
it always displays this error:
The website works when I use my browser application but not with puppeteer... Does anyone have any idea why?
OS: Macos 10.12.6
Puppeteer: 2.0.0
Maybe your internet connection is slow.
Do this instead for disabling timeout.
const puppeteer = require("puppeteer")
;(async () => {
// Launches browser
let browser = await puppeteer.launch()
console.log("browser opened")
let [page] = await browser.pages()
console.log("page opened")
page.setDefaultNavigationTimeout(0)
// Go to cleverbot
await page.goto("https://www.cleverbot.com/", {timeout: 0, waitUntil: 'networkidle0'})
console.log("cleverbot.com loaded")
})()
Related
on raspberry pi 3A+ chromium always shows "about:blank" and stops working unless I close the tab manually and enable new tab
const puppeteer = require("puppeteer");
async function test() {
const browser = await puppeteer.launch({
headless: false,
executablePath: "chromium-browser",
});
const [page] = await browser.pages();
await page.evaluate(() => window.open("https://www.example.com/"));
const page1 = await browser.newPage();
page.close();
await page1.goto("https://allegro.pl/");
await page1.screenshot({ path: "hello.png" });
await browser.close();
}
test();
enter image description here
try to working code on my raspberry pi 3A+
I am trying to make a scraper that gets the reviews for hotel on tripadvisor.com. I was just working with pagination and testing if the browser would go all the way to the end, where there is no more pages.
Here is my code so far:
const puppeteer = require("puppeteer");
const cheerio = require("cheerio");
async function main() {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.tripadvisor.com/Hotels-g298656-Ankara-Hotels.html');
while(true) {
await page.click('a[class="nav next ui_button primary"]');
await page.waitForNavigation({waitUntil: 'networkidle0'});
}
}
main();
However, this stops when the 'accept cookies' popup appears. How can I handle this?
I'm trying to open a new window page from a button in Puppeteer.
An example given: I'm logging to a website and the moment I click the button for the login a new fresh window page will pop-up, redirecting to the site the button is meant to be going. How can I do it?
You can do that by simply pressing Shift button while doing page.click
And to catch the newly opened window you can use waitForTarget.
const puppeteer = require('puppeteer')
;(async () => {
const browser = await puppeteer.launch({
headless: false,
defaultViewport: null,
})
const context = browser.defaultBrowserContext()
const page = (await context.pages())[0]
await page.goto('https://www.amazon.com/gp/product/B093GQSVPX/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1', {waitUntil: 'load'})
await page.waitForSelector('a[title="Add to List"]', {visible: true})
await page.keyboard.down('Shift')
await page.click('a[title="Add to List"]')
await page.keyboard.up('Shift')
const popup = await browser.waitForTarget(
(target) => target.url().includes('www.amazon.com/ap/signin')
)
const popupPage = await popup.page()
await popupPage.waitForSelector('a.a-link-expander[role="button"]')
await popupPage.click('a.a-link-expander[role="button"]')
await popupPage.click('input#continue[type="submit"]')
await browser.close()
})()
I am using puppeteer for scraping a page (load test application) and I cannot add username and password into this page. Does anyone of you know puppeteer and may help me? This is the code:
(async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto(“https://d22syekf1i694k.cloudfront.net/”, {waitUntil: ‘networkidle2’});
await page.waitForSelector(‘input[name=username]’);
await page.type(‘input[name=username]’, ‘Adenosine’);
await page.$eval(‘input[name=username]’, el => el.value = ‘Adenosine’);
await browser.close();
})(); ```
I'm writing a simple script to just check if all the resources load correctly (i check status codes of the responses). I've decided to use puppeteer for this and i wrote
(async () => {
const browser = await puppeteer.launch({headless: false});
const [page] = await browser.pages();
page.on("response", (res) => {
const url = res.url(), status=res.status();
// my functionality goes here
if (url.includes("favicon")) console.log(status, url); // not logging in headless
});
await page.goto("https://stackoverflow.com/", {waitUntil: 'networkidle2'});
await browser.close();
})();
the issue is that if i run my application in headless mode the favicon is missing from the responses, i assume it has something to do with puppeteer not loading a favicon in headless. Any built in functionality or workarounds?
From the lack of a better solution right now I'm evaluating favicons url and manually visiting it
async function checkFavicon(page){
const iconUrl = await page.$eval("link[rel*='icon']", ({href}) => href);
await page.goto(iconUrl);
await page.goBack();
}