I created a helper to manage my code in my other module. I wrote this piece of code:
Scraper: async function(page, selector1, selector2, selector3) {
let list_of_items = await page.evaluate(() => {
let items = [];
let elemend = selector1
let items_present = document.querySelectorAll(elemend);
items_present.forEach((element) => {
let itemJson = {};
try {
itemJson.name = element.querySelector(selector2).innerText;
if(element.querySelector(selector3)){
itemJson.price = element.querySelector(selector3).innerText;
}
}
catch (exception){
}
items.push(itemJson);
});
return items;
});
console.dir(list_of_items);
Every time I try to execute my code it fails and says evaluate is not a function
await Scraper.Scraper(hunt.div_container, hunt.name_selector, hunt.price_selector);
You are calling it wrong, do not do call it as await Scraper.Scraper() call it like a normal function Scraper.Scraper() also maybe don't use the same name for the class and function as that can get confusing and will become a nightmare to debug in a not so distant time and lastly ensure that the class as been instantiated first, cause it seems to me that your automation tool is inside a class, so ensure you instantiate it first before calling functions inside of it
Related
In angular project and learning to build a feature to run a custom script using the new function method.
I am able to access a custom function within the string. However is there a way to access the nested functions? The reasoning is I intend to have quite a number of functions accessible so it makes sense to pass it through a parent function.
Here is the code + stackblitz. Current I get an error 'nestFunc' undefined. Where nestFunc is the nested function.
https://stackblitz.com/edit/angular-bx5kia?file=src/main.ts
test() {
const script = `
var value = 1 +4;\n
console.log('value',value);\n
testFunc('Success');\n
testFunc().nestFunc2();\n
console.log("End Script");\n
`;
var testFunc = (msg?: string) => {
console.log('Test Function', msg);
var nestFunc1 = () => {
alert('hi');
};
var nestFunc2 = () => {
alert('hi');
};
};
const userFunction = new Function('testFunc', script);
userFunction(testFunc);
}
I have a JS module that is a function that fetches Issue data from our company Github, that I then want assigned to a variable:
import { request } from "https://cdn.skypack.dev/#octokit/request";
let issueData
async function getIssues() {
const issueAPI = await request("GET repos/{owner}/{repo}/issues", {
baseUrl: "https://company.com/api/v3/",
owner: "me",
repo: "exampleRepo",
});
window.issueData = issueAPI.data
return issueAPI.data
}
export { getIssues, issueData }
I can't seem to get the issueAPI.data to assign to my variable issueData? What's going wrong here? I have a feeling it's something to do with the promises/async or scope but just can't work it out.
You are assigning issueAPI.data to a window variable called issueData, which is not the same variable that you have defined with let on your snippet. If you want to assign to your declared variable, you just need to do this.
let issueData = await getIssues()
If there are no other errors in the function, this should correctly assign what you want to your declared variable.
Also you can drop the window.issueData = issueAPI.data unless you really want to have that accesible from the window object.
EDIT: As Jeremy points out, this would require top-level await. A workaround also provided by Jeremy:
let issueData
( async () => { issueData = await getIssues() })()
Like in the subject. I have a function like below and I have quite a bit of helping functions declared within a function (twice as much than in the example) because it's the only one using them.
My question is: should I extract those helping functions outside the function to maintain rule "Function should do one job and do it well" or it should be within? I also read about that higher level functions should be higher for better readability, but it somehow doesn't work (shouldn't hoisting make it work?).
const queryThings = async (body = defaultBody) => {
try {
(...)
// helping functions
const isNonTestDeal = obj => (...)
const isNonEmpty = obj => (...)
const replaceHTMLEntity = obj => (...)
const extractCountries = o => (...)
const queried = await query(...) // that one is outside this function
const cases = queriedCases
.filter(isNonTestDeal)
.map(obj => {
let countries = [(...)]
.filter(isNonEmpty)
.map(replaceHTMLEntity)
.map(extractCountries)
let data = {
(...)
}
return data
})
.filter(obj => (...))
.sort((a,b) => a.d - b.d)
.slice(0, 45) // node has problem with sending data of more than 8KB
return cases
} catch (error) {
console.log(error)
}
}
If you declare the function outside, and only use it in one function, then you cause namespace pollution. (What is namespace pollution?) Thus, I would recommend keeping it inside. Also, if you do so, it is easier to read as well, since it will be closer to the code where it is used.
To address your question about hoisting, it only works if you declare your function without assigning it to a variable.
i think when you write function in other function the memory use is better than write out of function
but you can't use in another function it is local function and it isn't public function
I know
await page.evaluateOnNewDocument(fs.readFileSync('./helperFunctions.js', 'utf8'));
to add functions to evaluate() context, that is very handy.
But is anyone can provide any example to have the same on the main context ? By example, said I want to add a
page.existsText()
or
existsText()
function with this code from a file to be included/sourced :
existsText = (string) => {
// code goes here
}
What is the way to go ?
If I understand your intention correctly, you would like to import functions from a file into global namespace, without having to assign them to an intermediary variable. Here's a simple way to do this in node.
helperFunctions.js
(function(){
// Assign the function to the scope from where it's called
this.existsText = (string) => {
console.log("Do stuff with " + string);
}
})()
Then when your require it in node.js/puppeteer script it executes at once assigning functions from it to global scope:
require("./helperFunctions");
existsText("global scope");
Result:
Do stuff with global scope
Bonus: using the same functions in page.evaluate
If needed the same helper file can then be used in browser scope:
await page.evaluateOnNewDocument(fs.readFileSync('./helperFunctions.js', 'utf8'));
await page.evaluate(() => {
existsText("browser scope");
});
Do stuff with browser scope
You can add some methods of the Page class from the page object for quick hacking.
const puppeteer = require("puppeteer");
puppeteer.launch().then(async browser => {
const page = await browser.newPage();
// page constructor is Page,
// and we can add some other methods to it's prototype
page.constructor.prototype.myCustomMethod = function() {
return this._frameManager.mainFrame().url();
};
await page.goto("https://example.com");
const customUrl = await page.myCustomMethod();
console.log({ customUrl }); // Returns { customUrl: 'https://example.com/' }
await browser.close();
});
It's as simple as that for some quick hacks. So, let's add some exists method to that.
// get all arguments
page.constructor.prototype.existsText = function(...args) {
// let's assume the argument as a variable called string
return this._frameManager.mainFrame().evaluate(string => {
// create a new regex to find global multiline insensitive results
const searchRegex = new RegExp(string, "gmi");
// find all matchs
return document.querySelector("body").outerHTML.match(searchRegex);
}, args); // pass all arguments
};
const foundText = await page.existsText("more");
console.log({ foundText });
// Result: { foundText: [ 'More' ] }
Again, these are all quick hacks and have their own limitation. Feel free to explore.
I have the following code:
ngOnInit(): void
{
const source = this.categoryService.getCategories();
console.log(source instanceof Observable);
const example = source.map((categor) => categor.map((categories) => {
const links = this.categoryService.countCategoryLinks(categories.id);
const aff = example.merge(links).subscribe(linke => console.log(linke));
return categories.id
}));
}
where getCategories() returns an observable.
On each item of this Observable, I get categories.id field to run another method called countCategoryLinks(categories.id) that also returns an Observable().
My problem is that : I only have 3 categories (ok), the countCategoryLinks() returns 3 items (ok) but the code above shows an infinite loop in the console.
Both methods started "manually" for testing purpose do not show any loop.
The problem really comes from the code above.
Any idea ?
Thanks in advance and Regards
example.merge(links) <= you are using an observable created by the map callback in the map callback which would cause recursion (ie. loop back into itself). This is where it pays to use proper indention as it is easier to see.
ngOnInit(): void {
const source = this.categoryService.getCategories();
console.log(source instanceof Observable);
const example = source.map((categor) => categor.map((categories) => {
const links = this.categoryService.countCategoryLinks(categories.id);
// this line is the issue. You are using variable example which is being created by the callback this line of code is in
const aff = example.merge(links).subscribe(linke => console.log(linke));
return categories.id
}));
}
I am thinking maybe you did not mean to still be inside of map at this point?