We Keep Coding

To set data in attribute "VALUE" using puppeteer

I have a use case where I need to set VALUE for a input tag.
'''
module.exports = async function(selectorId, inputString) {
const randomDate = inputString;
console.log(rdate)
await this.page.evaluate(((el) => {el.value = ""}), selector[0]);
await this.page.evaluate(((el,randomDate ) => {el.value = randomDate },randomDate ), selectorId[0]);
await this.page.waitForTimeout(1000);
};
'''
Here this code is working fine but it is not doing any action on UI, like not setting the selectorId.value as per definition, I am using xPath for the object. Can any one help on same.

testcafe higher order selector

I am trying to create a higher order selector which I can reuse by changing a text parameter. In the snippet below, when I am trying to perform t.click using actualSelector, it's not working as expected. When I console log the value for the actualSelector, i see whole item values are printed.
Seems I am not using it right way. Can you please help me with this.
e.g
const testItemNameGenericSelector = (itemName) =>
Selector(
".ms-Callout-container .ms-Callout-main div div"
)
.withText(itemName);
...
const itemNameToSelect = "Test Item-8ab1ec12-e719-4ab6-a0a3-ed538143d6d3";
const actualSelector = testItemNameGenericSelector(itemNameToSelect)
...
console.log(`selecting ${await actualSelector().textContent}`)
Below is full testcafe code for this test.
fixture`Getting Started`
.page // declare the fixture
`https://hasans30.github.io/testpage/dropdown.html`; // specify the start page
//then create a test and place your code there
test("My first test", async (t) => {
const testItemNameGenericSelector = (itemName) =>
Selector(
".ms-Callout-container .ms-Callout-main div div"
)
.withText(itemName);
const buttonSelector = Selector('.ms-Button-label');
const selectedValue = Selector('.ms-Dropdown-title');
const itemNameToSelect = "Test Item-8ab1ec12-e719-4ab6-a0a3-ed538143d6d3";
const actualSelector = testItemNameGenericSelector(itemNameToSelect);
await t.click(buttonSelector,{speed:0.51})
console.log(`selecting ${await actualSelector().textContent}`)
await t.click(actualSelector,{speed:0.51})
await t.expect(await selectedValue().textContent).eql(itemNameToSelect);
});

Puppeteer: Converting circular structure to JSON Are you passing a nested JSHandle?

I am trying to scrape a one-page website. There are multiple selection combinations that would result in different search redirects. I wrote a for loop in the page.evaluate's call back function to click the different selections and did the click search in every button. However, I got error: Converting circular structure to JSON Are you passing a nested JSHandle?
Please help!
My current version of code looks like this:
const res = await page.evaluate(async (i, courseCountArr, page) => {
for (let j = 1; j < courseCountArr[i]; j++) {
await document.querySelectorAll('.btn-group > button, .bootstrap-select > button')['1'].click() // click on school drop down
await document.querySelectorAll('div.bs-container > div.dropdown-menu > ul > li > a')[`${j}`].click() // click on each school option
await document.querySelectorAll('.btn-group > button, .bootstrap-select > button')['2'].click() // click on subject drop down
const subjectLen = document.querySelectorAll('div.bs-container > div.dropdown-menu > ul > li > a').length // length of the subject drop down
for (let k = 1; k < subjectLen; k++) {
await document.querySelectorAll('div.bs-container > div.dropdown-menu > ul > li > a')[`${k}`].click() // click on each subject option
document.getElementById('buttonSearch').click() //click on search button
page.waitForSelector('.strong, .section-body')
return document.querySelectorAll('.strong, .section-body').length
}
}
}, i, courseCountArr, page);

Why the error happens
While you haven't shown enough code to reproduce the problem (is courseCountArr an array of ElementHandles? Passing page to evaluate won't work either, that's a Node object), here's a minimal reproduction that shows the likely pattern:
const puppeteer = require("puppeteer");
let browser;
(async () => {
const html = `<ul><li>a</li><li>b</li><li>c</li></ul>`;
browser = await puppeteer.launch();
const [page] = await browser.pages();
await page.setContent(html);
// ...
const nestedHandle = await page.$$("li"); // $$ selects all matches
await page.evaluate(els => {}, nestedHandle); // throws
// ...
})()
.catch(err => console.error(err))
.finally(() => browser?.close())
;
The output is
TypeError: Converting circular structure to JSON
--> starting at object with constructor 'BrowserContext'
| property '_browser' -> object with constructor 'Browser'
--- property '_defaultContext' closes the circle Are you passing a nested JSHandle?
at JSON.stringify (<anonymous>)
Why is this happening? All code inside of the callback to page.evaluate (and family: evaluateHandle, $eval, $$eval) is executed inside the browser console programmatically by Puppeteer. The browser console is a distinct environment from Node, where Puppeteer and the ElementHandles live. To bridge the inter-process gap, the callback to evaluate, parameters and return value are serialized and deserialized.
The consequence of this is that you can't access any Node state like you're attempting with page.waitForSelector('.strong, .section-body') inside the browser. page is in a totally different process from the browser. (As an aside, document.querySelectorAll is purely synchronous, so there's no point in awaiting it.)
Puppeteer ElementHandles are complex structures used to hook into the page's DOM that can't be serialized and passed to the page as you're trying to do. Puppeteer has to perform the translation under the hood. Any ElementHandles passed to evaluate (or have .evaluate() called on them) are followed to the DOM node in the browser that they represent, and that DOM node is what your evaluate's callback is invoked with. Puppeteer can't do this with nested ElementHandles, as of the time of writing.
Possible fixes
In the above code, if you change .$$ to .$, you'll retrieve only the first <li>. This singular, non-nested ElementHandle can be converted to an element:
// ...
const handle = await page.$("li");
const val = await page.evaluate(el => el.innerText, handle);
console.log(val); // => a
// ...
Or:
const handle = await page.$("li");
const val = await handle.evaluate(el => el.innerText);
console.log(val); // => a
Making this work on your example is a matter of either swapping the loop and the evaluate call so that you access courseCountArr[i] in Puppeteer land, unpacking the nested ElementHandles into separate parameters to evaluate, or moving most of your console browser calls to click on things back to Puppeteer (depending on your use case and goals with the code).
You could apply the evaluate call to each ElementHandle:
const nestedHandles = await page.$$("li");
for (const handle of nestedHandles) {
const val = await handle.evaluate(el => el.innerText);
console.log(val); // a b c
}
To get an array of results, you could do:
const nestedHandles = await page.$$("li");
const vals = await Promise.all(
nestedHandles.map(el => el.evaluate(el => el.innerText))
);
console.log(vals); // [ 'a', 'b', 'c' ]
You can also unpack the ElementHandles into arguments for evaluate and use the (...els) parameter list in the callback:
const nestedHandles = await page.$$("li");
const vals = await page.evaluate((...els) =>
els.map(e => e.innerText),
...nestedHandles
);
console.log(vals); // => [ 'a', 'b', 'c' ]
If you have other arguments in addition to the handles you can do:
const nestedHandle = await page.$$("li");
const vals = await page.evaluate((foo, bar, ...els) =>
els.map(e => e.innerText + foo + bar)
, 1, 2, ...nestedHandle);
console.log(vals); // => [ 'a12', 'b12', 'c12' ]
or:
const nestedHandle = await page.$$("li");
const vals = await page.evaluate(({foo, bar}, ...els) =>
els.map(e => e.innerText + foo + bar)
, {foo: 1, bar: 2}, ...nestedHandle);
console.log(vals); // => [ 'a12', 'b12', 'c12' ]
Another option may be to use $$eval, which selects multiple handles, then runs a callback in browser context with the array of selected elements as its parameter:
const vals = await page.$$eval("li", els =>
els.map(e => e.innerText)
);
console.log(vals); // => [ 'a', 'b', 'c' ]
This is probably cleanest if you're not doing anything else with the handles in Node.
Similarly, you can totally bypass Puppeteer and do the entire selection and manipulation in browser context:
const vals = await page.evaluate(() =>
[...document.querySelectorAll("li")].map(e => e.innerText)
);
console.log(vals); // => [ 'a', 'b', 'c' ]
(note that getting the inner text throughout is just a placeholder for whatever browser code of arbitrary complexity you might have)

I wrote a little utility to solve this problem
const jsHandleToJSON = (jsHandle) => {
if (jsHandle.length > 0) {
let json = []
for (let i = 0; i < jsHandle.length; i++) {
json.push(jsHandleToJSON(jsHandle[i]))
}
return json
} else {
let json = {}
const keys = Object.keys(jsHandle)
for (let i = 0; i < keys.length; i++) {
if (typeof jsHandle[keys[i]] !== 'object') {
json[keys[i]] = jsHandle[keys[i]]
} else if (['elements', 'element'].includes(keys[i])) {
json[keys[i]] = jsHandleToJSON(jsHandle[keys[i]])
} else {
console.log(`skipping field ${keys[i]}`)
}
}
return json
}
}
It will create a new object with all the primitive fields of the jsHandle (recursively) and parse some extra jsHandle properties ['elements', 'element'], skips the others.
You could add more properties in there if you need them (but adding all of them will result in a infinite loop).
To make the log into puppeteer working you need to add the following line before the evaluate
page.on('console', message => console.log(`${message.type()}: ${message.text()}`))

Get all links with XPath in Puppeteer (pausing or not working)?

I am required to use XPaths to select all links on a page, for then my Puppeteer app to click into and perform some actions. I am finding that the method (code below) is getting stuck sometimes and my crawler will be paused. Is there a better/different way of getting all links from an XPath? Or is there something in my code that is incorrect and could be pausing my app's progress?
try {
links = await this.getLinksFromXPathSelector(state);
} catch (e) {
console.log("error getting links");
return {...state, error: e};
}
Which calls:
async getLinksFromXPathSelector(state) {
const newPage = state.page
// console.log('links selector');
const links = await newPage.evaluate((mySelector) => {
let results = [];
let query = document.evaluate(mySelector,
document,
null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
for (let i=0, length=query.snapshotLength; i<length; ++i) {
results.push(query.snapshotItem(i).href);
}
return results;
}, state.linksSelector);
return links;
}
The XPath is in state.linksSelector.

You can use page.$x() to evaluate an XPath expression and obtain an ElementHandle array. It may be appropriate to use page.waitForXPath() beforehand to ensure that the elements specified by XPath string are added to the DOM.
Then you can pass the ElementHandle array elements to the page context via page.evaluate() and return an array containing the href attribute values for each element.
const xpath_expression = '//a[#href]';
await page.waitForXPath(xpath_expression);
const links = await page.$x(xpath_expression);
const link_urls = await page.evaluate((...links) => {
return links.map(e => e.href);
}, ...links);
console.log(link_urls);

How to delete existing text from input using Puppeteer?

I'm trying to test amending text in an editable input which contains the title of the current record - and I want to able to test editing such text, replacing it with something else.
I know I can use await page.type('#inputID', 'blah'); to insert "blah" into the textbox (which in my case, having existing text, only appends "blah"), however, I cannot find any page methods1 that allow deleting or replacing existing text.

You can use page.evaluate to manipulate DOM as you see fit:
await page.evaluate( () => document.getElementById("inputID").value = "")
However sometimes just manipulating a given field might not be enough (a target page could be an SPA with event listeners), so emulating real keypresses is preferable. The examples below are from the informative issue in puppeteer's Github concerning this task.
Here we press Backspace as many times as there are characters in that field:
const inputValue = await page.$eval('#inputID', el => el.value);
// focus on the input field
await page.click('#inputID');
for (let i = 0; i < inputValue.length; i++) {
await page.keyboard.press('Backspace');
}
Another interesting solution is to click the target field 3 times so that the browser would select all the text in it and then you could just type what you want:
const input = await page.$('#inputID');
await input.click({ clickCount: 3 })
await input.type("Blah");

You can use the page.keyboard methods to change input values, or you can use page.evaluate().
Replace All Text:
// Using page.keyboard:
await page.focus('#example');
await page.keyboard.down('Control');
await page.keyboard.press('A');
await page.keyboard.up('Control');
await page.keyboard.press('Backspace');
await page.keyboard.type('foo');
// Using page.evaluate:
await page.evaluate(() => {
const example = document.getElementById('example');
example.value = 'foo';
});
Append Text:
// Using page.keyboard:
await page.focus('#example');
await page.keyboard.press('End');
await page.keyboard.type(' bar qux');
// Using page.evaluate:
await page.evaluate(() => {
const example = document.getElementById('example');
example.value += ' bar qux';
});
Backspace Last Character:
// Using page.keyboard:
await page.focus('#example');
await page.keyboard.press('End');
await page.keyboard.press('Backspace');
// Using page.evaluate:
await page.evaluate(() => {
const example = document.getElementById('example');
example.value = example.value.slice(0, -1);
});
Delete First Character:
// Using page.keyboard:
await page.focus('#example');
await page.keyboard.press('Home');
await page.keyboard.press('Delete');
// Using page.evaluate:
await page.evaluate(() => {
const example = document.getElementById('example');
example.value = example.value.slice(1);
});

If you are not interested in simulating any key events, you could also use puppeteer's page.$eval method as a concise means to remove the textarea's value...
await page.$eval('#inputID', el => el.value = '');
await page.type('#inputID', 'blah');
...or even completely replace the value in one step, without simulating the subsequent typing:
await page.$eval('#inputID', el => el.value = 'blah');

This works perfect for "clear only" method:
const input = await page.$('#inputID');
await input.click({ clickCount: 3 })
await page.keyboard.press('Backspace')

above answers has an ESLint issues.
the following solution passing ESLint varification:
await page.evaluate(
(selector) => { (document.querySelector(selector).value = ''); },
inputBoxSelector,
);

Use the Keyboard API which simulates keystrokes:
await page.focus(css); // CSS selector of the input element
await page.keyboard.down('Shift');
await page.keyboard.press('Home');
await page.keyboard.up('Shift'); // Release the pressed 'Shift' key
await page.keyboard.press('Backspace');
This keystroke is cross-platform as opposed to using ctrl + A(does not work in Mac to select all characters in a input field)

The most clean way for me is:
Setup
const clearInput = async (page, { selector }) => {
const input = await page.$(selector)
await input.click({ clickCount: 3 })
await page.keyboard.press('Backspace')
}
Usage
const page = await context.newPage()
await clearInput(page, { selector: 'input[name="session[username_or_email]"]' })
await clearInput(page, { selector: 'input[name="session[password]"]' })

Well, the reason you want to delete existing text generally may be want to replace it.
You can use page.evalute
let title = getTitle()
let selector = getSelector()
await page.evaluate(
({selector, title}) => {
let el = document.querySelector(selector)
if ('value' in el) el.value = title
else el.innerText = title
},
{selector, title}
)

someField.type("");
Pass the empty string before typing your content.
This worked for me.