How I can calculate page fully load time with Pupppeteer? - javascript

I'm trying to get the page fully load time in seconds with puppeteer in Node, for this I do some research on the API and other questions and create the following code:
/* First Configuration */
puppeteer.launch({
defaultViewport: { width: 1600, height: 800 }
}).then(async browser => {
const page = await browser.newPage();
await page.setCacheEnabled(false);
await page.goto('https://stackoverflow.com', {waitUntil: 'networkidle0'});
/* Get Page Metrics */
const perf = await page.metrics();
console.log(JSON.stringify(perf));
/* Get Page Evaluate */
const timing = await page.evaluate(() => {
const result = {};
for (const key of Object.keys(window.performance.timing.__proto__))
result[key] = window.performance.timing[key];
return result;
});
console.log(JSON.stringify(timing));
/* Show Results on Browser Close */
await browser.close().then(() => {
var fullyLoadEvaluate = (timing.loadEventEnd - timing.navigationStart);
console.log('Fully Load Time (Page Evaluate): ' + fullyLoadEvaluate);
var fullyLoadMetrics = (perf.LayoutDuration + perf.RecalcStyleDuration + perf.ScriptDuration + perf.TaskDuration);
console.log('Fully Load Time (Page Metrics): ' + fullyLoadMetrics);
/* Send Response to Server */
res.send('Check The Console');
});
});
Basically I use two codes to return metrics, One of them is page.metrics() that return the following data:
{"Timestamp":961736.600171,"Documents":8,"Frames":4,"JSEventListeners":375,"Nodes":8654,"LayoutCount":27,"RecalcStyleCount":31,"LayoutDuration":0.705517,"RecalcStyleDuration":0.144379,"ScriptDuration":0.527385,"TaskDuration":1.812213,"JSHeapUsedSize":11082496,"JSHeapTotalSize":20344832}
And the last one page.evaluate(), return the following:
{"navigationStart":1556722407938,"unloadEventStart":0,"unloadEventEnd":0,"redirectStart":0,"redirectEnd":0,"fetchStart":1556722407938,"domainLookupStart":1556722408247,"domainLookupEnd":1556722408548,"connectStart":1556722408548,"connectEnd":1556722408737,"secureConnectionStart":1556722408574,"requestStart":1556722408738,"responseStart":1556722408940,"responseEnd":1556722409087,"domLoading":1556722408957,"domInteractive":1556722409995,"domContentLoadedEventStart":1556722409995,"domContentLoadedEventEnd":1556722410190,"domComplete":1556722412584,"loadEventStart":1556722412584,"loadEventEnd":1556722412589,"toJSON":{}}
In my example I'm testing the site https://stackoverflow.com. Like webpagetest.org and getmetrix.com, I'm trying to get Page Fully Load Time.
I know this kind of value is inconsistent, but I wonder if the values ​​I'm calculating are right, and which of the two results seems to be more correct ? Fully Load Time (Page Evaluate) or Fully Load Time (Page Metrics) ?

You can use page.metrics() to compare two points in time (e.g. before and after page.goto). The page.evaluate approach to read the data from the performance API is also a good alternative. As I already pointed out in the comment, it is not defined what should be considered a "full page load". Both approaches are valid.
It's even more complex
There are a number of thing which people might consider a page to be loaded:
DOMContentLoaded event fired
Load event fired
Time it takes from navigation start until all resources embedded in the document (like images are loaded)
Time it takes from navigation start until all resources are loaded
Time until there are not more ongoing network requests.
...
You also have to consider whether whether you want network related phases (like DNS) to be part of the measurement. Here is an example request (generated with the Chrome DevTools Network tab) showing how complex a single request might be:
There is also a document explaining each of these phases.
Simple approach
The simplest way to measure the load time would just to start measuring when the navigaiton starts and stop measuring after the page is loaded. This could be done like this:
const t1 = Date.now();
await page.goto('https://example.com');
const diff1 = Date.now() - t1;
console.log(`Time: ${diff1}ms`);
Note that there are also other APIs (page.metrics, process.hrtime, perf_hooks) to get more precise timestamps.
You can also pass options to the page.goto function to change the resolving of the promise to something like this (quoted from the docs):
Consider navigation to be finished when there are no more than 0 network connections for at least 500ms
For that, you would have to use the setting networkidle0:
await page.goto('https://example.com', { waitUntil: 'networkidle0' });
There are also other events in the docs linked above you could use.
More complex: Use the Performance API
To get more precise results, you can use the Performance API as you already did in your code. Instead of going through the prototype of window.performance you can also use the functions performance.getEntries() or performance.toJSON() like this:
const perfData = await page.evaluate(() =>
JSON.stringify(performance.toJSON(), null, 2)
);
That way, you get data that looks like this:
{
"timeOrigin": 1556727036740.113,
"timing": {
"navigationStart": 1556727036740,
"unloadEventStart": 0,
"unloadEventEnd": 0,
"redirectStart": 0,
"redirectEnd": 0,
"fetchStart": 1556727037227,
"domainLookupStart": 1556727037230,
"domainLookupEnd": 1556727037280,
"connectStart": 1556727037280,
"connectEnd": 1556727037348,
"secureConnectionStart": 1556727037295,
"requestStart": 1556727037349,
"responseStart": 1556727037548,
"responseEnd": 1556727037805,
"domLoading": 1556727037566,
"domInteractive": 1556727038555,
"domContentLoadedEventStart": 1556727038555,
"domContentLoadedEventEnd": 1556727038570,
"domComplete": 1556727039073,
"loadEventStart": 1556727039073,
"loadEventEnd": 1556727039085
},
"navigation": {
"type": 0,
"redirectCount": 0
}
}
So if you want to know how long it took from navigationStart to loadEventStart you subtract one value from the other one (e.g. 1556727039073 - 1556727036740 = 2333 ms).
So which one to take?
This is up to your decision. In general, it is a good idea to use the Load event as a starting point. Waiting until all requests are finished might actually never happen because there are constantly resources being loaded in the background. Using networkidle2 as waitUntil option might be an alternative in case you don't want to use the load event.
In the end, however, it comes down to your use case which metric to use.

Related

How to target the selector on new tab after clicking a[target="_blank"] - Failing to activate the new tab created

After clicking a[target="_blank"] new tab opens. How to get code to get new page object so I can access password input field?
Using NodeJS, JavaScript, Puppeteer.
Navigation is working up to the point included below.
EDIT: I used the page.url() method to retrieve current URL and the URL of the newly created tab does not log to console, previous page logs.
I tried adjusting the script and received following errors
Cannot read properties of undefined (reading 'page') - I thought adding a time delay would solve this but no go.
I was having this error but as the code is below I do not get this error: No node found for selector: #Password
I have looked at related issues
I came across dheerajbhaskar GitHub issue and read up on related issues
#386
#3535
#978
and more
I tried to implement code from an accepted answer without any success.
Using Puppeteer to get a handle to the new page after "_blank" click?
try {
await sleep(2300)
// This block creates a new tab
// I was previously using a selector and not mouse click API
await Promise.all([
page.mouse.click(xToolsBtn, yToolsBtn, { delay: 2000 }),
])
// NEW TARGET CREATED
// Below is a snippet from an accepted answer but the the type method
// does not work
// Seems like page is still not activated
const [newTarget] = await Promise.all([
// Await new target to be created with the proper opener
new Promise((x) =>
browser.on("targetcreated", (target) => {
if (target.opener() !== page.target()) return
browser.removeListener("targetcreated", arguments.callee)
x()
})
),
// page.click('link')
])
// Trying to input password without success
const newPage = await newTarget.newPage()
await newPage.type("#Password", process.env.PASSWORD, {
delay: randomGenerator,
})
} catch (err) {
console.error(
"LOGIN BUTTON FAIL",
err.message
)
}
Alternatively atempt#1: I tried to select the input via mouse x, y co-ordinates which activates the input field but this returns the following error"
No node found for selector: #Password
Alternatively atempt#2:
//* WAIT FOR TARGET
try {
await sleep(2300)
await Promise.all([
page.mouse.click(xToolsBtn, yToolsBtn, { delay: 2000 }),
])
sleep(5000)
await page.evaluate(() => window.open(`${loginUrl3}`))
const newWindowTarget = await browser.waitForTarget(
(target) => target.url() === `${loginUrl3}`
)
console.log("GOT TARGET")
await newWindowTarget.type("#Password", process.env.PASSWORD, {
delay: randomGenerator,
})
} catch (err) {
console.log("WAIT FOR TARGET FAILED")
}
Note: URLS are randomly generated so I would be curious what if any work around there is to use current URL. I would assume the new tab created would still need to be activated...
managed to solve this together (Linker :)
Process
First, we mapped the target being created to check for focus
browser.on('targetcreated', function (target) {
console.log('New tab:');
console.log(target);
});
We saw the URL is trying to open - for some reason the URLs in the target were empty. We re-installed stuff to rule out weird dependency bugs, then figured there's a focus issue.
Workaround
To solve it, we need to wait for the .newPage() to open before goto'ing to the URL, calling bringToFront() and then waiting for it to load (short sleep is the easy way). Once we did that, we had a working POC to start from.
Relevant part from the solution:
let mappedURL = tabs
.map((e, index) => e.url())
.filter((e, idx) => idx == 2)
console.log("MAPPED URL ", mappedURL)
sleep(2500)
const page3 = await browser.newPage()
await page3.goto(`${mappedURL}`)
await page3.bringToFront()
Ref
Here's a cool SO Answer showing how to use once syntax to test the event. Happy we were able to solve it, and I hope the process helps other people out.
Addressing just the question in the title, "How to target the selector on new tab after clicking a[target="_blank"]" -
Handling newly opened tabs in Playwright is far from intuitive if you're not used to it. A summary of how they work:
If you click a link in your test with target="_blank", which opens a new tab, the page object you're working with still refers to the original page/tab you opened the link on.
To get ahold of the new page, you have to:
const [newPage] = await Promise.all([
context.waitForEvent('page'), // get `context` by destructuring with `page` in the test params; 'page' is a built-in event, and **you must wait for this like this,**, or `newPage` will just be the response object, rather than an actual Playwright page object.
page.locator('text=Click me').click() // note that, like all waiting in Playwright, this is somewhat unintuitive. This is the action which is *causing the navigation*; you have to set up the wait *before* it happens, hence the use of Promise.all().
])
await newPage.waitForLoadState(); // wait for the new tab to fully load
// now, use `newPage` to access the newly opened tab, rather than `page`, which will still refer to the original page/tab.
await expect(newPage).toHaveURL('http://www.someURL.com');
await newPage.locator('text=someText');

Update HTML data without creating more elements on the webpage

Have an interval that loads the html every 3 secs, but want to refresh it and not keep adding more under the already made code.
async function Products(){
setInterval(async function() {
const response = await fetch('http://localhost:3000/api/products');
const data = await response.json();
const mainContainer = document.getElementById("myData");
for (let obj of data) {
const div = document.createElement("div");
div.innerHTML = `${obj["sku"]}: ${obj["name"]}`;
mainContainer.appendChild(div);
}
}, 10000)
}
When I click a start button everything works, but how do i make it refresh the already made HTML rather than repeatedly recreating it with the interval. Trying to figure out a good approach to this. Thanks
Create and append a <div> immediately, and in the interval, assign to its innerHTML:
function Products() {
const container = document.getElementById("myData").appendChild(document.createElement("div"));
setInterval(async function () {
const response = await fetch('http://localhost:3000/api/products');
const data = await response.json();
container.innerHTML = '';
for (let obj of data) {
container.innerHTML += `${obj["sku"]}: ${obj["name"]}`;
}
}, 10000)
}
I think what you are trying to do is implement a Serie of observable elements that you can look only for those who have changed instead of all the data, something like React does with the virtual DOM.
Considering the code tout already posted, refreshing elements at a set interval is a bad idea. What if you have 1000 user refreshing at the same time? What if it cause your response tone to be more then 3 seconds?
That said, if you really want to work on creating something like that, you have to find a way to load not all the products from the api, but only the ones that are different. Again, if you want to keep it that way, here are, in my opinion, what you could do:
Start by loading all the product on the page, but set an interval to check a new Endpoint which would tell you what products have been added after the last one.
Use either an index or a key to identify which product is which, so tout know the ones you have.
You need a way to know quick product was updated since the last load.
That's a start. You can implement these in different way. I would suggest having a timestamp for the time created and updated, so you can then query only those who fall after this timestamp.
Add a dynamic ID to your product elements, (e.g. <div id=${sku} >**Product Here**<\div>
That way, you can track your product and recreate only the ones who changed/are new.
That's obviously a complicated way of implementing an open connection, if you want another solution, you could also open a Socket with your api, which would send event for data updated and created and would ultimately make your 3 second ping obsolete, resulting in a better scalability in my opinion.

Passing object in Electron IPC yields old object version

I'm experiencing a behavior of Electron's IPC that seems weird. The setting is an app that, when clicking on a button, performs a download in the main process via IPC, like so (in renderer.js):
$('#Download').click(async function () {
await app.download();
updateUI();
}
The download function in main performs the download, then updates a main.js closure-scoped object named items, like so:
app.download = async function() {
const response = await fetch('https://...');
const result = await response.json();
await result.items.forEach(async element => {
items[element.id] = element.content;
}
console.log('Nr of items: ' + Object.keys(items).length);
}
After the download is complete, the UI should be updated. For this, there's a getter in main.js:
app.getItems = function() { return items; }
The renderer process uses it to update the UI:
function updateUI() {
var items = app.getItems();
console.log('Received items: ' + Object.keys(items).length);
}
However, given this flow, app.getItems always returns the items state before the download, i.e. missing the new items and showing a lower result for Object.keys(items).length). Only after I refresh the Electron app window, it returns the state including the downloaded items. I tried several methods to get the updated item list back from main to renderer:
Have the app.download function return the items: Same result (!) - the item count written to the console differs between main and renderer. I even output timestamps on both sides to assure the renderer is called after main.
Trigger the UI update from main using webContents.send('UpdateUI') and calling updateUI in an ipc.on('UpdateUI') handler: same result.
Add items to app.global in main, and retrieve it via remote.getGlobal() in renderer: same result.
It was only the final try that succeeded:
Pass items as a parameter with webContents.send('UpdateUI', items)
Now before refactoring the code to use this pattern I'd really be curious if anyone can explain this :-)
The only idea I have is that passing variables between main and renderer is not "by reference" (given they're two isolated processes), and that Electron's IPC does some kind of caching when deciding when to actually copy data. I carefully added await on every reasonable place (and the timestamps indicated I succeeded), so I'd be surprised if it's caused by any async side effects.

evernote findNotesMetadata - using offset and maxnotes to loop to retrieve all notes

Per Evernote documentation for findNotesMetadata the maxnotes returned from server in 1 response is 250. I am trying to understand how to make multiple requests to retrieve entire array if more then 250. Below is current code.
const Evernote = require('evernote');
const developerToken = "...";
const client = new Evernote.Client({token: developerToken, sandbox: false});
const noteStore = client.getNoteStore();
const spec = {}
spec.includeTitle = true;
spec.includeTagGuids = true;
spec.includeAttributes = true;
spec.includeNotebookGuid = true;
const filter = new Evernote.NoteStore.NoteFilter({
words: '*',
});
noteStore.findNotesMetadata(filter, 0, 250, spec)
.then(noteobj => {
...
})
.catch( e => console.error(e));
Current code doenst incorporate any loop yet but works up to 250 notes. Due to Evernote SDK and dealing with promises Im not positive even where to start. I have searched online quite a bit to find a solution directly (looking at Evernote examples) and indirectly (looking at other rest API examples). Not having any luck. Any help is appreciated.
The offset param to findNotesMetadata is how you indicate the start index into the actual result set you want. In the case of the code you've shown, you're passing in 0 (it's the second param). That is telling the API that you want your results to begin with item 0 in the actual result set, up to a maximum of 250 results.
If you want to "page" through the result set in windows of 250 results, you can call the method again using 250 as the offset, and ask for the next 250 results. This is a fairly common design pattern for paging through result sets via a remote API or anything that has a resource constraint on retrieving data. You'll want to handle the cases when no more results are available-- either because you get fewer back than the maxNotes that you ask for or the corner case where you get exactly the max number but then zero on the following request. That's how you know to break out of your loop.
The Evernote API seems to offer a findNoteCounts method, which should give you an idea of how many actual results there would be, but as with all async systems, there's a theoretical race where that number changes between API calls.

Load list of array via an in web-browser

I need to load URL from multiple array in browser
So for example-: api url is www.example.com/dst=ad. Here ad is one set of input and I have number of list that needs to be executed one after another like 5-10 secs time interval.
I'm familiar with doing this via bash using for loop this kind of tasks but I needed to get this particular task only via browser.
so here is exactly what I needed
for dst in ad ef gh mn; do www.example.com/dst=$dst; done;
I wanted the browser to load the set of given lists for the api url one after another in browser as API gets authentication with signal signon. I heard this can be done via javascript but I'm not familiar with this language.
Can someone please provide any input with this.
Try this:
const loadURL = async (interval, array) => {
array.forEach(async (element) => {
await fetch(`https://www.example.com/dst=${element}`)
})
await setTimeout(() => loadURL(interval, array), interval)
}
loadURL(yourIntervalTime, yourArray);

Categories