I'm writing a simple script to just check if all the resources load correctly (i check status codes of the responses). I've decided to use puppeteer for this and i wrote
(async () => {
const browser = await puppeteer.launch({headless: false});
const [page] = await browser.pages();
page.on("response", (res) => {
const url = res.url(), status=res.status();
// my functionality goes here
if (url.includes("favicon")) console.log(status, url); // not logging in headless
});
await page.goto("https://stackoverflow.com/", {waitUntil: 'networkidle2'});
await browser.close();
})();
the issue is that if i run my application in headless mode the favicon is missing from the responses, i assume it has something to do with puppeteer not loading a favicon in headless. Any built in functionality or workarounds?
From the lack of a better solution right now I'm evaluating favicons url and manually visiting it
async function checkFavicon(page){
const iconUrl = await page.$eval("link[rel*='icon']", ({href}) => href);
await page.goto(iconUrl);
await page.goBack();
}
Related
I am using puppeteer for scraping a page (load test application) and I cannot add username and password into this page. Does anyone of you know puppeteer and may help me? This is the code:
(async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto(“https://d22syekf1i694k.cloudfront.net/”, {waitUntil: ‘networkidle2’});
await page.waitForSelector(‘input[name=username]’);
await page.type(‘input[name=username]’, ‘Adenosine’);
await page.$eval(‘input[name=username]’, el => el.value = ‘Adenosine’);
await browser.close();
})(); ```
I use the following blog to use the playright
login and I need something similar to use for my app, when I use the headless:flase
I see it opens the UI with the user password in however, it doesnt click on the logon button, I adopt the code I try with the following , am I missing something?
(async () => {
const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://app.com');
await page.fill('input[type="text"]', 'user#test.com');
await page.fill('input[type="password"]', 'Abcd1234!');
// page.click('div[data-testid="LoginForm_Login_Button"]');
page.click('div[id="logOnFormSubmit"]');
}
)();
You are currently using
page.click('div[id="logOnFormSubmit"]');
There is no div in your given code example with that ID, but instead there is a button. You'd need to change that line to reflect this. The final code would look like below.
(async () => {
const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://app.com');
await page.fill('input[type="text"]', 'user#test.com');
await page.fill('input[type="password"]', 'Abcd1234!');
// page.click('div[data-testid="LoginForm_Login_Button"]');
page.click('button[id="logOnFormSubmit"]');
}
)();
I am trying to scrape github contributor insights using NodeJS, puppeteer, and cheerio.
const cheerio = require('cheerio');
const puppeteer = require('puppeteer');
const url = 'https://github.com/grey-software/grey.software/graphs/contributors';
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.waitForSelector('#contributors', {
visible: true,
})
await page.goto(url);
const pageContent = await page.content()
const $ = cheerio.load(pageContent);
$('.contrib-person').each(function (i, elem) {
console.log(elem)
});
await browser.close();
})();
I get the following error when I run the code above
UnhandledPromiseRejectionWarning: TimeoutError: waiting for selector "#contributors" failed: timeout 30000ms exceeded
The #contributors div should load within the 30 seconds, but I always get this timeout.
Note: page.waitForNavigation() gives the same timeout error
Your issue is you are waiting for the selector to exist... Before you go to the website in the first place. use page.waitForSelector() after you use page.goto()
I'm trying to get the https://www.cleverbot.com page content with puppeteer. But when I try:
const puppeteer = require("puppeteer");
(async () => {
// Launches browser
let browser = await puppeteer.launch();
console.log("browser opened");
let page = await browser.newPage();
console.log("page opened");
// Go to cleverbot
await page.goto("https://www.cleverbot.com/");
console.log("cleverbot.com loaded");
})();
it always displays this error:
The website works when I use my browser application but not with puppeteer... Does anyone have any idea why?
OS: Macos 10.12.6
Puppeteer: 2.0.0
Maybe your internet connection is slow.
Do this instead for disabling timeout.
const puppeteer = require("puppeteer")
;(async () => {
// Launches browser
let browser = await puppeteer.launch()
console.log("browser opened")
let [page] = await browser.pages()
console.log("page opened")
page.setDefaultNavigationTimeout(0)
// Go to cleverbot
await page.goto("https://www.cleverbot.com/", {timeout: 0, waitUntil: 'networkidle0'})
console.log("cleverbot.com loaded")
})()
I'm trying to make a InstagramBot that logs in and then go to some profile, my code worked yesterday for awhile and than it just stopped working .
I've tried to clone my repository from github, but it does'n work either, sometimes it works again, but if I try to create another function, the code just ignore the line of the code that changes the page.
I've also tried to create a new page and then in this new page use the goto function and it worked, but the account doesn keep logged in
The version of puppeteer that I'm using: 1.16.0
The version of node.js that I'm using: v10.15.3
const puppeteer = require('puppeteer');
const BASE_URL = "https://www.instagram.com/accounts/login/?hl=en&source=auth_switcher";
const instagram = {
browser: null,
page: null,
profile_url: null,
initialize: async (profile) => {
instagram.browser = await puppeteer.launch({
headless: false
})
instagram.profile_url = await "https://www.instagram.com/" + profile;
instagram.page = await instagram.browser.newPage();
await instagram.page.goto(BASE_URL, {waitUntil: 'networkidle2'});
},
login: async(username, password) =>{
await instagram.page.waitFor(1000);
await instagram.page.type('input[name="username"]', username);
await instagram.page.type('input[name="password"', password);
await instagram.page.click('button[type="submit"]');
await instagram.page.waitFor(1500);
await console.log(instagram.profile_url);
await instagram.page.goto(instagram.profile_url, {timeout: 0, waitUntil: 'domcontentloaded'}); // the code just ignore this line
await instagram.page.waitFor(1000);
},
getPhotosLinks: async() => {
console.log("Do something here");
}
}
module.exports = instagram;
It doesn't give any error message, just doesn't work
Replace
await instagram.page.click('button[type="submit"]');
await instagram.page.waitFor(1500);
with
await Promise.all([
instagram.page.click('button[type="submit"]');,
instagram.page.waitForNavigation()
]);
and see if it works