Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 11 hours ago.
Improve this question
I'm trying to scrape YouTube Shorts from a specific YouTube Channel, using Puppeteer running on MeteorJs Galaxy.
Here's the code that I've done so far:
import puppeteer from 'puppeteer';
import { YouTubeShorts } from '../imports/api/youTubeShorts'; //meteor mongo local instance
let URL = 'https://www.youtube.com/#ummahtoday1513/shorts'
const processShortsData = (iteratedData) => {
let documentExist = YouTubeShorts.findOne({ videoId:iteratedData.videoId })
if(documentExist === undefined) { //undefined meaning this incoming shorts in a new one
YouTubeShorts.insert({
videoId: iteratedData.videoId,
title: iteratedData.title,
thumbnail: iteratedData.thumbnail,
height: iteratedData.height,
width: iteratedData.width
})
}
}
const fetchShorts = () => {
puppeteer.launch({
headless:true,
args:[
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--single-process'
]
})
.then( async function(browser){
async function fetchingData(){
new Promise(async function(resolve, reject){
const page = await browser.newPage();
await Promise.all([
await page.setDefaultNavigationTimeout(0),
await page.waitForNavigation({waitUntil: "domcontentloaded"}),
await page.goto(URL, {waitUntil:["domcontentloaded", "networkidle2"]}),
await page.waitForSelector('ytd-rich-grid-slim-media', { visible:true }),
new Promise(async function(resolve,reject){
page.evaluate(()=>{
const trialData = document.getElementsByTagName('ytd-rich-grid-slim-media');
const titles = Array.from(trialData).map(i => {
const singleData = {
videoId: i.data.videoId,
title: i.data.headline.simpleText,
thumbnail: i.data.thumbnail.thumbnails[0].url,
height: i.data.thumbnail.thumbnails[0].height,
width: i.data.thumbnail.thumbnails[0].width,
}
return singleData
})
resolve(titles);
})
}),
])
await page.close()
})
await browser.close()
}
async function fetchAndProcessData(){
const datum = await fetchingData()
console.log('DATUM:', datum)
}
await fetchAndProcessData()
})
}
fetchShorts();
I am struggling with two things here:
Async, await, and promises, and
Finding reason behind why Puppeteer output the ProtocolError: Protocol error (Target.createTarget): Target closed. error in the console.
I'm new to puppeteer and trying to learn from various examples on StackOverflow and Google in general, but I'm still having trouble getting it right.
A general word of advice: code slowly and test frequently, especially when you're in an unfamiliar domain. Try to minimize problems so you can understand what's failing. There are many issues here, giving the impression that the code was written in one fell swoop without incremental validation. There's no obvious entry point to debugging this.
Let's examine some failing patterns.
First, basically never use new Promise() when you're working with a promise-based API like Puppeteer. This is discussed in the canonical What is the explicit promise construction antipattern and how do I avoid it? so I'll avoid repeating the answers there.
Second, don't mix async/await and then. The point of promises is to flatten code and avoid pyramids of doom. If you find you have 5-6 deeply nested functions, you're misusing promises. In Puppeteer, there's basically no need for then.
Third, setting timeouts to infinity with page.setDefaultNavigationTimeout(0) suppresses errors. It's fine if you want a long delay, but if a navigation is taking more than a few minutes, something is wrong and you want an error so you can understand and debug it rather than having the script wait silently until you kill it, with no clear diagnostics as to what went wrong or where it failed.
Fourth, watch out for pointless calls to waitForNavigation. Code like this doesn't make much sense:
await page.waitForNavigation(...);
await page.goto(...);
What navigation are you waiting for? This seems ripe for triggering timeouts, or worse yet, infinite hangs after you've set navs to never timeout.
Fifth, avoid premature abstractions. You have various helper functions but you haven't established functionally correct code, so these just add to the confused state of affairs. Start with correctness, then add abstractions once the cut points become obvious.
Sixth, avoid Promise.all() when all of the contents of the array are sequentially awaited. In other words:
await Promise.all([
await foo(),
await bar(),
await baz(),
await quux(),
garply(),
]);
is identical to:
await foo();
await bar();
await baz();
await quux();
await garply();
Seventh, always return promises if you have them:
const fetchShorts = () => {
puppeteer.launch({
// ..
should be:
const fetchShorts = () => {
return puppeteer.launch({
// ..
This way, the caller can await the function's completion. Without it, it gets launched into the void and can never be connected with the caller's flow.
Eighth, evaluate doesn't have access to variables in Node, so this pattern doesn't work:
new Promise(resolve => {
page.evaluate(() => resolve());
});
Instead, avoid the new promise antipattern and use the promise that Puppeteer already returns to you:
await page.evaluate(() => {});
Better yet, use $$eval here since it's an abstraction of the common pattern of selecting elements first thing in evaluate.
Putting all of this together, here's a rewrite:
const puppeteer = require("puppeteer"); // ^19.6.3
const url = "<Your URL>";
let browser;
(async () => {
browser = await puppeteer.launch();
const [page] = await browser.pages();
await page.goto(url, {waitUntil: "domcontentloaded"});
await page.waitForSelector("ytd-rich-grid-slim-media");
const result = await page.$$eval("ytd-rich-grid-slim-media", els =>
els.map(({data: {videoId, headline, thumbnail: {thumbnails}}}) => ({
videoId,
title: headline.simpleText,
thumbnail: thumbnails[0].url,
height: thumbnails[0].height,
width: thumbnails[0].width,
}))
);
console.log(result);
})()
.catch(err => console.error(err))
.finally(() => browser?.close());
Note that I ensure browser cleanup with finally so the process doesn't hang in case the code throws.
Now, all we want is a bit of text, so there's no sense in loading much of the extra stuff YouTube downloads. You can speed up the script by blocking anything unnecessary to your goal:
const [page] = await browser.pages();
await page.setRequestInterception(true);
page.on("request", req => {
if (
req.url().startsWith("https://www.youtube.com") &&
["document", "script"].includes(req.resourceType())
) {
req.continue();
}
else {
req.abort();
}
});
// ...
Note that ["domcontentloaded", "networkidle2"] is basically the same as "networkidle2" since "domcontentloaded" will happen long before "networkidle2". But please avoid "networkidle2" here since all you need is some text, which doesn't depend on all network resources.
Once you've established correctness, if you're ready to factor this to a function, you can do so:
const fetchShorts = async () => {
const url = "<Your URL>";
let browser;
try {
browser = await puppeteer.launch();
const [page] = await browser.pages();
await page.goto(url, {waitUntil: "domcontentloaded"});
await page.waitForSelector("ytd-rich-grid-slim-media");
return await page.$$eval("ytd-rich-grid-slim-media", els =>
els.map(({data: {videoId, headline, thumbnail: {thumbnails}}}) => ({
videoId,
title: headline.simpleText,
thumbnail: thumbnails[0].url,
height: thumbnails[0].height,
width: thumbnails[0].width,
}))
);
}
finally {
await browser?.close();
}
};
fetchShorts()
.then(shorts => console.log(shorts))
.catch(err => console.error(err));
But keep in mind, making the function responsible for managing the browser resource hampers its reusability and slows it down considerably. I usually let the caller handle the browser and make all of my scraping helpers accept a page argument:
const fetchShorts = async page => {
const url = "<Your URL>";
await page.goto(url, {waitUntil: "domcontentloaded"});
await page.waitForSelector("ytd-rich-grid-slim-media");
return await page.$$eval("ytd-rich-grid-slim-media", els =>
els.map(({data: {videoId, headline, thumbnail: {thumbnails}}}) => ({
videoId,
title: headline.simpleText,
thumbnail: thumbnails[0].url,
height: thumbnails[0].height,
width: thumbnails[0].width,
}))
);
};
(async () => {
let browser;
try {
browser = await puppeteer.launch();
const [page] = await browser.pages();
console.log(await fetchShorts(page));
}
catch (err) {
console.error(err);
}
finally {
await browser?.close();
}
})();
I'm newer to JavaScript and struggling to understand why this function just returns right away, even though I added an await. The code below is the minimal example to replicate what I am attempting to do. When I call add_user() from add(), id is undefined because add_user() is returning right away instead of waiting for completion of the query. I added the console log to verify that row.id is the value I expected (i.e. 1).
'use strict';
import sqlite3 from 'sqlite3'
import { open } from 'sqlite';
async function add_user(value) {
await (async () => {
const db = await open({
filename: dbFile,
driver: sqlite3.Database
})
const row = await db.get(`SELECT id FROM Users WHERE name LIKE "${value}"`)
console.log(row.id)
return row.id
})()
}
async function add(req, res) {
var id = await add_value(req.body.value)
console.log(id)
}
I'm pretty sure the code is running asynchronously as desired - it's just that you aren't returning anything from add_user. For a smaller example:
async function someFn() {
await somePromise;
}
Above, someFn will always return a Promise that resolves to undefined, because nothing was returned from someFn. You're running into the same issue.
Use instead
async function add_user(value) {
const db = await open({
filename: dbFile,
driver: sqlite3.Database
})
const row = await db.get(`SELECT id FROM Users WHERE name LIKE "${value}"`)
console.log(row.id)
return row.id
}
I am uploading an initial data to firebase-firestore. The data is to be used as a fixture to develop my frontend further. I do not have errors, at least haven't caught one yet. The following two functions, are being used to upload data:
Edited to explain better
async function uploadData() {
const userId = 'AKFROjo1isTPRfjYYDsSehwZdIi1';
const itemsListRef = await db.collection('users').doc(userId).collection('userData').doc('itemsList');
await itemsListRef.set({ created: firebase.firestore.FieldValue.serverTimestamp() });
await summaryList.forEach(async summary => {
await itemsListRef.collection('summaryList').add(summary);
});
await coreCompetencyList.forEach(async coreCompetency => {
await itemsListRef.collection('coreCompetencyList').add(coreCompetency);
});
}
and a second function as -
async function uploadData2() {
const userId = 'AKFROjo1isTPRfjYYDsSehwZdIi1';
const itemsListRef = await db.collection('users').doc(userId).collection('userData').doc('itemsList');
await itemsListRef.set({ created: firebase.firestore.FieldValue.serverTimestamp() });
await educationList.forEach(async education => {
await itemsListRef.collection('educationList').add(education);
});
await categoryList.forEach(async category => {
await itemsListRef.collection('categoryList').add(category);
});
}
It is being called as :
async function main() {
try {
await uploadData();
await app.delete();
} catch (e) {
console.log('Data upload failed, reason:', e, '\n\n');
}
}
main().then(r => console.log('Done.'));
I am surprised that I cannot put all 4 calls in a single function
It's not a bug. This is the way it was intended to work.
In actuality, collections are not really entities that require creation or deletion like folders in a filesystem. They are just virtual containers for documents that spring into existence immediately when the first document in created under it, and automatically disappear when the last document in deleted. You can think of them more like units of organization that make indexing and security rules possible to implement.
I'm trying to setup tests for my database crawler program and I can't manage to replace what the class method I'm testing imports.
So as not to write down too much code I'll just lay out the general form of the problem. In my test function I have:
describe("test",()=>{
let result1;
beforeAll(async ()=>{
await createConnection();
})
afterAll(async ()=>{
getConnection().close();
})
test("setup test",async () => {
result1 = await WeatherController.startForecastAPI();
expect(result1.status).toBe(Status.SUCCESS);
})
})
The WeatherController.ts file (... where code was taken out):
...
import AccessTokenService from '../services/AccessTokenService';
export default class WeatherController{
...
static async startForecastAPI(){
...
const accessToken = AccessTokenService.getAccessToken();//get and validate token
...
}
}
Inside the WeatherController class, startForecastAPI is defined as a static async method. The class imports multiple other classes, among them the AccessTokenService class which is used to get valid access tokens. AccessTokenService.getAccessToken() should return an object with several properties that it gets through a http request.
I want to mock the results of calling AccessTokenService but I'm not calling it directly in my test function, I'm calling WeatherController and WeatherController is calling AccessTokenService. How can I replace what WeatherController calls when I test it but without touching the WeatherController code? I've tried going through the jest docs but I'm fairly new to all of this and they're confusing. I'm not entirely clear how scoping works here either (I tried defining a function in the test code and calling it in the tested function but it's out of scope).
The await WeatherController.startForecastAPI() call in the test function returns undefined but the code works fine when I hard-code accessToken to be a valid object, I just can't find a way to inject that object into the code through the test function.
Assuming AccessTokenService.getAccessToken returns a promise or is an async function, then you can use jest.spyOn(...).mockResolvedValue() to prevent calling the server
describe("test",()=>{
let result1;
beforeAll(async ()=>{
await createConnection();
})
afterAll(async ()=>{
getConnection().close();
})
test("setup test",async () => {
const expectedResultFromGetToken = {property: 'property 1'};
const getTokenSpy = jest.spyOn(AccessTokenService, 'getAccessToken')
.mockResolvedValue(expectedResultFromGetToken)
result1 = await WeatherController.startForecastAPI();
expect(result1.status).toBe(Status.SUCCESS);
expect(getTokenSpy).toHaveBeenCalled()
})
})
if the AccessTokenService.getAccessToken is not an async function then you have to use jest.spyOn(...).mockReturnValue()
If inside your class you have
const AccessToken = require('access-token');
you can mock it with
jest.mock('access-token', () => {
function getToken() {
return 'fakeToken'
}
);
const WeatherController = require('weather-controller');
describe("test",()=>{
let result1;
beforeAll(async ()=>{
await createConnection();
})
afterAll(async ()=>{
getConnection().close();
})
test("setup test",async () => {
result1 = await WeatherController.startForecastAPI();
expect(result1.status).toBe(Status.SUCCESS);
})
})
so, I'm using Puppeteer with Jest. After adding
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
My tests does not perform any actions. It doesn't matter if I'm using headless mode or let's call it "normal" mode. Anybody can help me?
homepage.test.js
const puppeteer = require('puppeteer');
const HomePage = require('./page_objects/HomePage');
const homePage = new HomePage();
describe('Homepage', () => {
beforeAll(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto(homePage.path);
await page.waitForSelector(homePage.loginPanel);
});
it('Log into your account', async () => {
await homePage.fillLoginForm();
await expect(page).toMatchElement(homePage.productList);
await page.screenshot({ path: 'example.png' });
});
HomePage.js
module.exports = class HomePage {
constructor() {
this.path = 'https://www.saucedemo.com/index.html';
this.loginPanel = '#login_button_container';
this.productList = 'div[class="inventory_container"]';
this.loginForm = {
fields: {
usernameInput: 'input[id="user-name"]',
passwordInput: 'input[id="password"]',
logInButton: 'input[class="btn_action"]',
},
};
}
async fillLoginForm() {
await page.type(this.loginForm.fields.usernameInput, 'standard_user');
await page.type(this.loginForm.fields.passwordInput, 'secret_sauce');
await page.click(this.loginForm.fields.logInButton);
}
};
The answer has two parts, one with normal jest and another with jest-puppeteer. You can skip to the jest-puppeteer if you want.
Problem (with jest):
The browser and page inside beforeAll block has no relation to the it blocks. It also does not have any relation with the page inside HomePage class as well.
You did not mention if you were using jest-puppeteer or not.
Solution:
Create block scoped variables for the describe block, and pass the page object to the modules.
Refining the HomePage class
Consider the following HomePage class.
// HomePage.js
class HomePage {
constructor(page) {
this.page = page;
}
async getScreenshot() {
await this.page.screenshot({ path: "example.png" });
}
async getTitle(page) {
return page.title();
}
}
As you can see, there are two ways to access to the page inside the class. Either pass inside the constructor, or use with the method directly.
The method getScreenshot has a this.page, while getTitle has access to a page.
Refining the test
You cannot use this inside the jest tests due to this issue, but you can declare a variable on top of a block, then access it later.
describe("Example", () => {
// define them up here inside the parent block
let browser;
let page;
let homepage;
beforeAll(async () => {
// it has access to the browser, page and homepage
browser = await puppeteer.launch({ headless: true });
page = await browser.newPage();
homepage = new HomePage(page); // <-- pass the page to HomePage here
await page.goto("http://example.com");
await page.waitForSelector("h1");
return true;
});
});
Now all other blocks can access to the page. According to our previous example HomePage class, we can do either of following depending on how we defined the methods.
it("Gets the screenshot", async () => {
await homepage.getScreenshot(); // <-- will use this.page
});
it("Gets the title", async () => {
await homepage.getTitle(page); // <-- will use the page we are passing on
});
Finally we cleanup the tests,
afterAll(async () => {
await page.close();
await browser.close();
return true;
});
We probably need to run the jest tests with detectOpenHandles for headfull mode.
jest . --detectOpenHandles
Result:
Problem (with jest-puppeteer):
jest-puppeteer already gives you a global browser and page object. You do not need define anything.
However if you want to use jest-puppeteer and expect-puppeteer, you have to use a custom config file.
Solution:
Create a jest-config.json file and put the contents,
{
"preset": "jest-puppeteer",
"setupFilesAfterEnv": ["expect-puppeteer"]
}
Now, get rid of browser and page creation code, as well as any afterAll hooks for page.close as well.
Here is a working test file,
class HomePage {
async getTitle() {
return page.$("h1");
}
}
describe("Example", () => {
const homepage = new HomePage();
beforeAll(async () => {
// it has access to a global browser, page and scoped homepage
await page.goto("http://example.com");
await page.waitForSelector("h1");
});
it("Gets the screenshot", async () => {
const element = await homepage.getTitle();
await expect(element).toMatch("Example");
});
});
And let's run this,
jest . --detectOpenHandles --config jest-config.json
Result: