We Keep Coding

Trying to use puppeteer inside async function inside async function which already has puppeteer

I'm trying to build telegram bot to parse page on use request. My parsing code works fine inside one async function, but completeky falls on its face if I try to put it inside another async function.
Here is the relevant code I have:
const puppeteer = require('puppeteer');
const fs = require('fs/promises');
const { Console } = require('console');
async function start(){
async function searcher(input) {
const browser = await puppeteer.launch();
const page = await browser.newPage();
const url = ; //here is a long url combining logic, that works fine
await page.goto(url);
const currentUrl = requestPage.url();
console.log(currentUrl); //returns nothing.
//here is some long parsing logic
await browser.close();
return combinedResult;
}
//here is a bot code
const { Telegraf } = require('telegraf');
const bot = new Telegraf('my bot ID');
bot.command('start', ctx => {
console.log(ctx.from);
bot.telegram.sendMessage(ctx.chat.id, 'Greatings message', {});
bot.telegram.sendMessage(ctx.chat.id, 'request prompt ', {});
})
bot.on('text', (ctx) => {
console.log(ctx.message.text);
const queryOutput = searcher(ctx.message.text);
bot.telegram.sendMessage(ctx.chat.id, queryOutput, {});
});
bot.launch()
}
start();
Here is an error message:
/Users/a.rassanov/Desktop/Fetch/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:218
return Promise.reject(new Error(`Protocol error (${method}): Session closed. Most likely the ${this._targetType} has been closed.`));
^
Error: Protocol error (Page.navigate): Session closed. Most likely the page has been closed.
I'm very new to this, and your help is really appriciated.

How to mock a function using Frisby and Jest to return custom response?

I'm trying to mock a function using Frisby and Jest.
Here are some details about my code:
dependencies
axios: "^0.26.0",
dotenv: "^16.0.0",
express: "^4.17.2"
devDependencies
frisby: "^2.1.3",
jest: "^27.5.1"
When I mock using Jest, the correct response from API is returned, but I don't want it. I want to return a fake result like this: { a: 'b' }.
How to solve it?
I have the following code:
// (API Fetch file) backend/api/fetchBtcCurrency.js
const axios = require('axios');
const URL = 'https://api.coindesk.com/v1/bpi/currentprice/BTC.json';
const getCurrency = async () => {
const response = await axios.get(URL);
return response.data;
};
module.exports = {
getCurrency,
};
// (Model using fetch file) backend/model/cryptoModel.js
const fetchBtcCurrency = require('../api/fetchBtcCurrency');
const getBtcCurrency = async () => {
const responseFromApi = await fetchBtcCurrency.getCurrency();
return responseFromApi;
};
module.exports = {
getBtcCurrency,
};
// (My test file) /backend/__tests__/cryptoBtc.test.js
require("dotenv").config();
const frisby = require("frisby");
const URL = "http://localhost:4000/";
describe("Testing GET /api/crypto/btc", () => {
beforeEach(() => {
jest.mock('../api/fetchBtcCurrency');
});
it('Verify if returns correct response with status code 200', async () => {
const fetchBtcCurrency = require('../api/fetchBtcCurrency').getCurrency;
fetchBtcCurrency.mockImplementation(() => (JSON.stringify({ a: 'b'})));
const defaultExport = await fetchBtcCurrency();
expect(defaultExport).toBe(JSON.stringify({ a: 'b'})); // This assert works
await frisby
.get(`${URL}api/crypto/btc`)
.expect('status', 200)
.expect('json', { a: 'b'}); // Integration test with Frisby does not work correctly.
});
});
Response[
{
I hid the lines to save screen space.
}
->>>>>>> does not contain provided JSON [ {"a":"b"} ]
];

This is a classic lost reference problem.
Since you're using Frisby, by looking at your test, it seems you're starting the server in parallel, correct? You first start your server with, say npm start, then you run your test with npm test.
The problem with that is: by the time your test starts, your server is already running. Since you started your server with the real fetchBtcCurrency.getCurrency, jest can't do anything from this point on. Your server will continue to point towards the real module, not the mocked one.
Check this illustration: https://gist.githubusercontent.com/heyset/a554f9fe4f34101430e1ec0d53f52fa3/raw/9556a9dbd767def0ac9dc2b54662b455cc4bd01d/illustration.svg
The reason the assertion on the import inside the test works is because that import is made after the mock replaces the real file.
You didn't share your app or server file, but if you are creating the server and listening on the same module, and those are "hanging on global" (i.e: being called from the body of the script, and not part of a function), you'll have to split them. You'll need a file that creates the server (appending any route/middleware/etc to it), and you'll need a separate file just to import that first one and start listening.
For example:
app.js
const express = require('express');
const { getCurrency } = require('./fetchBtcCurrency');
const app = express()
app.get('/api/crypto/btc', async (req, res) => {
const currency = await getCurrency();
res.status(200).json(currency);
});
module.exports = { app }
server.js
const { app } = require('./app');
app.listen(4000, () => {
console.log('server is up on port 4000');
});
Then, on your start script, you run the server file. But, on your test, you import the app file. You don't start the server in parallel. You'll start and stop it as part of the test setup/teardown.
This will give jest the chance of replacing the real module with the mocked one before the server starts listening (at which point it loses control over it)
With that, your test could be:
cryptoBtc.test.js
require("dotenv").config();
const frisby = require("frisby");
const URL = "http://localhost:4000/";
const fetchBtcCurrency = require('./fetchBtcCurrency');
const { app } = require('./app');
jest.mock('./fetchBtcCurrency')
describe("Testing GET /api/crypto/btc", () => {
let server;
beforeAll((done) => {
server = app.listen(4000, () => {
done();
});
});
afterAll(() => {
server.close();
});
it('Verify if returns correct response with status code 200', async () => {
fetchBtcCurrency.getCurrency.mockImplementation(() => ({ a: 'b' }));
await frisby
.get(`${URL}api/crypto/btc`)
.expect('status', 200)
.expect('json', { a: 'b'});
});
});
Note that the order of imports don't matter. You can do the "mock" below the real import. Jest is smart enough to know that mocks should come first.

Trying to run multiple selenium tests but getting ''cannot read property 'get' of undefined'

I am relatively new to selenium and javascript. I am trying to run multiple seleniumjs test files sequentially. To do this I have created another js file (testAll) in which I call all the exported test functions I have created in separate files. I am running into an issue with where I am defining the driver and feel like I'm in a bit of a catch 22. When I define the driver within the test async function itself it works fine but when I then transfer the driver definition to my testAll file to avoid multiple browser windows opening up then I receive a 'cannot read property 'get' of undefined' message. This must be referring to my driver as an undefined variable but I can't see a way to get around this. I have included an example test.js file and my testAll.js file code below:
testAll.js:
const servicesPage = require('./servicesPage');
const organisations = require('./organisations');
const {By , Builder, until} = require('selenium-webdriver');
const allServices = async () => {
const driver = await new Builder().forBrowser('chrome').build();
try {
await servicesPage(driver);
await organisations(driver);
}
finally{
await driver.quit();
}
}
allServices();
test.js:
//Setup
const {By , Builder, until} = require('selenium-webdriver');
const properties = require('../test_Properties')
const authentication = require('../mainAuth');
const assert = require('assert');
const organisations = async (driver) => {
try {
//Execution
await driver.get(properties.servicesUrls.orgsPage);
await authentication(driver);
await driver.wait(until.elementLocated(By.linkText('Request an organisation')), 7000);
await driver.findElement(By.linkText('Request an organisation')).click();
//Assert organisations request page and click back
let rqstOrg = await driver.findElement(By.tagName('h1')).getText();
assert.equal(rqstOrg , 'Request an organisation' , 'Request an organisation heading does not
match');
await driver.findElement(By.className('link-back')).click();
//Assert organisations page
let orgTitle = await driver.getTitle();
assert.equal(orgTitle , 'Organisations' , 'Organisations title does not match');
await driver.findElement(By.linkText('Sign out')).click();
} catch (e) {
throw e;
}
};
module.exports = organisations;

How to use puppeteer-core with electron?

I got this code from another Stackoverflow Question:
import electron from "electron";
import puppeteer from "puppeteer-core";
const delay = (ms: number) =>
new Promise(resolve => {
setTimeout(() => {
resolve();
}, ms);
});
(async () => {
try {
const app = await puppeteer.launch({
executablePath: electron,
args: ["."],
headless: false,
});
const pages = await app.pages();
const [page] = pages;
await page.setViewport({ width: 1200, height: 700 });
await delay(5000);
const image = await page.screenshot();
console.log(image);
await page.close();
await delay(2000);
await app.close();
} catch (error) {
console.error(error);
}
})();
Typescript compiler complains about executablePath property of launch method options object cause it needs to be of type string and not Electron. So how to pass electron chromium executable path to puppeteer?

You cannot use electron executable with Puppeteer directly without some workarounds and flag changes. They have tons of differences in the API. Specially electron doesn't have all of the chrome.* API which is needed for chromium browser to work properly, many flags still doesn't have proper replacements such as the headless flag.
Below you will see two ways to do it. However you need to make sure of two points,
Make sure the puppeteer is connected before the app is initiated.
Make sure you get the correct version puppeteer or puppeteer-core for the version of Chrome that is running in Electron!
Use puppeteer-in-electron
There are lots of workarounds, but most recently there is a puppeteer-in-electron package which allows you to run puppeteer within electron app using the electron.
First, install the dependencies,
npm install puppeteer-in-electron puppeteer-core electron
Then run it.
import {BrowserWindow, app} from "electron";
import pie from "puppeteer-in-electron";
import puppeteer from "puppeteer-core";
const main = async () => {
const browser = await pie.connect(app, puppeteer);
const window = new BrowserWindow();
const url = "https://example.com/";
await window.loadURL(url);
const page = await pie.getPage(browser, window);
console.log(page.url());
window.destroy();
};
main();
Get the debugging port and connect to it
The another way is to get the remote-debugging-port of the electron app and connect to it. This solution is shared by trusktr on electron forum.
import {app, BrowserWindow, ...} from "electron"
import fetch from 'node-fetch'
import * as puppeteer from 'puppeteer'
app.commandLine.appendSwitch('remote-debugging-port', '8315')
async function test() {
const response = await fetch(`http://localhost:8315/json/versions/list?t=${Math.random()}`)
const debugEndpoints = await response.json()
let webSocketDebuggerUrl = debugEndpoints['webSocketDebuggerUrl ']
const browser = await puppeteer.connect({
browserWSEndpoint: webSocketDebuggerUrl
})
// use puppeteer APIs now!
}
// ... make your window, etc, the usual, and then: ...
// wait for the window to open/load, then connect Puppeteer to it:
mainWindow.webContents.on("did-finish-load", () => {
test()
})
Both solution above uses webSocketDebuggerUrl to resolve the issue.
Extra
Adding this note because most people uses electron to bundle the app.
If you want to build the puppeteer-core and puppeteer-in-electron, you need to use hazardous and electron-builder to make sure get-port-cli works.
Add hazardous on top of main.js
// main.js
require ('hazardous');
Make sure the get-port-cli script is unpacked, add the following on package.json
"build": {
"asarUnpack": "node_modules/get-port-cli"
}
Result after building:

the toppest answer dones't work for me use electron 11 and puppeteer-core 8.
but start puppeteer in main process other then in the renderer process works for me.you can use ipcMain and ipcRenderer to comunicate each other.the code below
main.ts(main process code)
import { app, BrowserWindow, ipcMain } from 'electron';
import puppeteer from 'puppeteer-core';
async function newGrabBrowser({ url }) {
const browser = await puppeteer.launch({
headless: false,
executablePath:
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
});
const page = await browser.newPage();
page.goto(url);
}
ipcMain.on('grab', (event, props) => {
newGrabBrowser(JSON.parse(props));
});
home.ts (renderer process code)
const { ipcRenderer } = require('electron');
ipcRenderer.send('grab',JSON.stringify({url: 'https://www.google.com'}));

There is also another option, which works for electron 5.x.y and up (currently up to 7.x.y, I did not test it on 8.x.y beta yet):
// const assert = require("assert");
const electron = require("electron");
const kill = require("tree-kill");
const puppeteer = require("puppeteer-core");
const { spawn } = require("child_process");
let pid;
const run = async () => {
const port = 9200; // Debugging port
const startTime = Date.now();
const timeout = 20000; // Timeout in miliseconds
let app;
// Start Electron with custom debugging port
pid = spawn(electron, [".", `--remote-debugging-port=${port}`], {
shell: true
}).pid;
// Wait for Puppeteer to connect
while (!app) {
try {
app = await puppeteer.connect({
browserURL: `http://localhost:${port}`,
defaultViewport: { width: 1000, height: 600 } // Optional I think
});
} catch (error) {
if (Date.now() > startTime + timeout) {
throw error;
}
}
}
// Do something, e.g.:
// const [page] = await app.pages();
// await page.waitForSelector("#someid")//
// const text = await page.$eval("#someid", element => element.innerText);
// assert(text === "Your expected text");
// await page.close();
};
run()
.then(() => {
// Do something
})
.catch(error => {
// Do something
kill(pid, () => {
process.exit(1);
});
});
Getting the pid and using kill is optional. For running the script on some CI platform it does not matter, but for local environment you would have to close the electron app manually after each failed try.
Please see this sample repo.

How to get passed or failed test case name in the puppeteer

I need to integrate the puppeteer-jest test framework with TestRail using TestRail API. But for that, I need to know what tests are failed and what of the tests are passed
I Search some information in the official GitHub Repository and in the Jest site. But nothing about it.
Test:
describe('Single company page Tests:', () => {
let homePage;
beforeAll(async () => {
homePage = await addTokenToBrowser(browser);
}, LOGIN_FLOW_MAX_TIME);
it('Open the company page from the list', async done => {
await goto(homePage, LIST_PAGE_RELATIVE_PATH);
await listPage.clickSearchByCompanyName(homePage);
await addCompanyNamePopup.isPopupDisplayed(homePage);
await addCompanyNamePopup.fillCompanyName(homePage, companies.century.link);
await addCompanyNamePopup.clickNext(homePage);
await addCompanyNamePopup.fillListName(homePage, listNames[0]);
await addCompanyNamePopup.clickSave(homePage);
await addCompanyNamePopup.clickViewList(homePage);
const nextPage = await clickCompanyName(homePage, browser, companies.century.name);
await companyPage.isOverviewTabPresent(nextPage);
await companyPage.isPeopleTabPresent(nextPage);
await companyPage.isSocialTabPresent(nextPage);
await companyPage.isFinanceTabPresent(nextPage);
await companyPage.isLeaseTabPresent(nextPage);
await homePage.close();
done();
});
}
I expected to get all passed and failed test cases name and write it to JSON with the name of test cases and the result of them.
Actually, I have nothing of this.

You can use true/false assertion approach I like I do in my github project.
for example, try anchor case to some final selector with simple assert:
describe('E2E testing', () => {
it('[Random Color Picker] color button clickable', async () => {
// Setup
let expected = true;
let expectedCssLocator = '#color-button';
let actual;
// Execute
let actualPromise = await page.waitForSelector(expectedCssLocator);
if (actualPromise != null) {
await page.click(expectedCssLocator);
actual = true;
}
else
actual = false;
// Verify
assert.equal(actual, expected);
});