Employee moved on and left me with this code that was once working to generate PDFs. I haven't had any luck trying to debug - with breakpoints or even console.logs - the script listed here at the bottom; is there a way to search the huge list of loaded scripts in Visual Studio?
C# error:
{System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The server returned an invalid or unrecognized response.
at System.Net.Http.HttpConnection.FillAsync()
Client Side error: (is this because the server never returns anything?)
ERROR Error: Uncaught (in promise): DataCloneError: Failed to execute 'postMessage' on 'Worker': TypeError: Failed to fetch could not be cloned.
Error: Failed to execute 'postMessage' on 'Worker': TypeError: Failed to fetch could not be cloned.
at MessageHandler.postMessage (pdf.js:12334)
at sendStreamRequest (pdf.js:12151)
at Object.error (pdf.js:12194)
at eval (pdf.js:8419)
at ZoneDelegate.invoke (zone.js:392)
Controller method
public async Task<IActionResult> Generate(string id)
{
try
{
var stream = await _reportService.GenerateReportAsync(id);
return new FileStreamResult(stream, "application/pdf");
}
catch(Exception ex)
{
throw;
}
}
Service method:
public async Task<Stream> GenerateReportAsync(string id)
{
return await Services.InvokeAsync<Stream>("./Node/generate-pdf.js", Configuration.Url, id, new { format = "A4" });
}
generate-pdf.js:
const pdf = require('html-pdf');
const puppeteer = require('puppeteer');
module.exports = async function (result, url, id, options) {
const browser = await createBrowser();
const page = await browser.newPage();
const css = await browser.newPage();
await page.goto(`${url}/reports/${id}`, {
waitUntil: 'networkidle2'
});
await css.goto(`${url}/styles.bundle.css`, {
waitUntil: 'networkidle2'
});
await page.waitForSelector('.report-loaded');
let cssBody = await css.evaluate(() => `<style>${document.documentElement.innerHTML}</style>`);
let bodyHtml = await page.evaluate(() => document.documentElement.innerHTML);
bodyHtml = bodyHtml.replace('<link href="styles.bundle.css" rel="stylesheet">', cssBody);
browser.close();
pdf.create(cssBody + bodyHtml, options).toStream((error, stream) => stream.pipe(result.stream));
}
async function createBrowser() {
return await puppeteer.launch({
args: ['--no-sandbox', '--disable-setuid-sandbox']
});
}
Looks like the generate-pdf.js script is using "html-pdf". This can be found on npm:
https://www.npmjs.com/package/html-pdf
And it has a github page:
https://github.com/marcbachmann/node-html-pdf
So the problem is going to be with the usage of that package, or some kind of bug in it. (well, that's an assumption on my part, I don't know this package at all and have no experience working with it)
At this point I'd try to figure out which version of that package is being used, check out the source code and try to find a hint in there.
This structure seems rather convoluted though. Why not just generate the PDF in the client in the first place, or generate it in the C# code? That it was working at some point shouldn't be an argument as you are now noticing this is proving difficult to maintain.
Related
import puppeteer from 'puppeteer'
async function scrapeProduct(url) {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto(url)
const element = await page.waitForSelector('input[type="hidden"][name="abuseID"]', {
hidden: true,
timeout: 15000
})
const abuseID = await page.evaluate(element => element.value, element)
console.log(abuseID)
}
scrapeProduct('https://steamcommunity.com/id/lupusRe')
I don't know where I am going wrong but I am getting an error of
file:///C:/Users/charl/Desktop/Code/abg/steam-search/node_modules/puppeteer/lib/esm/puppeteer/common/ExecutionContext.js:282
throw new Error('Evaluation failed: ' + getExceptionMessage(exceptionDetails));
^
Error: Evaluation failed: TypeError: Cannot read properties of null (reading 'value')
at pptr://__puppeteer_evaluation_script__:1:21
at ExecutionContext._ExecutionContext_evaluate (file:///C:/Users/charl/Desktop/Code/abg/steam-search/node_modules/puppeteer/lib/esm/puppeteer/common/ExecutionContext.js:282:15)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async ExecutionContext.evaluate (file:///C:/Users/charl/Desktop/Code/abg/steam-search/node_modules/puppeteer/lib/esm/puppeteer/common/ExecutionContext.js:114:16)
at async scrapeProduct (file:///C:/Users/charl/Desktop/Code/abg/steam-search/test.js:13:21)
My goal is to be able to get the steam ID in the below tag
<input type="hidden" name="abuseID" value="76561198036553525>"
I am open to using other things beside puppeteer but its just the main thing that I have been using for web scraping.
This is on the official steam community website if that matters.
Copy Xpath of an element
and use page.$x(Xpath)
Edit: i've tried couple of things and it seems like you can get persons steamID through a <script> tag
import puppeteer from 'puppeteer'
async function scrapeProduct(url) {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto(url);
const text = await page.evaluate(() => Array.from(document.querySelectorAll('script'), element => element.textContent));
let spl = text[38].split(',');
console.log(spl[1])
browser.close()
}
scrapeProduct('https://steamcommunity.com/id/DaYanack')
try this and tell me if it works
So I'm trying to build a weather app by using data from a weather API.
import fetch from 'node-fetch'
//fetch weather API
let weather
let getWeather = async() => {
let url = \https://api.openweathermap.org/data/2.5/weather?q=auckland&appid=c156947e2c7f0ccb0e2a20fde1d2c577`try {let res = await fetch(url)weather = await res.json() } catch (error) {console.log("error") } let weatherMain = weather.weather.map( el => el.description)if(weatherMain ="Rain"){console.log(weatherMain)// weatherImg = "[https://icon-library.com/images/raining-icon/raining-icon-1.jpg](https://icon-library.com/images/raining-icon/raining-icon-1.jpg)" } }console.log(getWeather())`
My problem is that I'm getting this error when running in vscode:
SyntaxError: Cannot use import statement outside a module
and this error when running in browser:
Uncaught TypeError: Failed to resolve module specifier "node-fetch". Relative references must start with either "/", "./", or "../".`
Not sure what exactly is going on, Can someone please explain what's happening?
I've tried fetch API once before and that time I didn't need to import fetch, so I'm pretty confused.
SS
Edit - Understood now, running in browser and in vscode is 2 different things. What works in the browser won't necessarily work in Node.js
When running in browser, there's no need to import fetch.
Thanks everyone.
let weather;
let getWeather = async () => {
let url = `https://api.openweathermap.org/data/2.5/weather?q=auckland&appid=c156947e2c7f0ccb0e2a20fde1d2c577`;
try {
let res = await fetch(url);
weather = await res.json();
console.log('weather', weather);
} catch (error) {
console.log(error);
}
let weatherMain = weather.weather.map((el) => el.description);
if ((weatherMain = 'Rain')) {
console.log('weatherMain', weatherMain);
let weatherImg =
'[https://icon-library.com/images/raining-icon/raining-icon-1.jpg](https://icon-library.com/images/raining-icon/raining-icon-1.jpg)';
return weatherImg;
}
};
const main = async () => {
const data = await getWeather();
console.log('data', data);
};
main();
Yes, you are right about no need to import fetch if you are running the js in the browser. But I see that you are importing node-fetch, this package is used to bring the fetch (window.fetch) for the node system.
But If you want to run it in the node, then you should know that the node doesn't support ES6 module. But you can user the experimental flag to run the code. e.g.
node --experimental-modules app.mjs
Before I start the question, I am new in JavaScript, and I have very basic knowledge of async js, but i need to solve this so i can have my first project functional.
I am trying to build a scraping app using Node and Puppeteer. Basically, the user enters a URL ("link" in the code below), puppeteer goes trough the website code, tries to find the specific piece and returns the data. That part I got working so far.
The problem is when a user enters a URL of a site that doesn't have that piece of code. In that case, I get UnhandledPromiseRejectionWarning: Error: Evaluation failed theme is not defined
What do I do so when there is an error like that, I can catch it and redirect the page instead of Getting Internal Server error.
app.post("/results", function(req, res) {
var link = req.body.link;
(async link => {
const browser = await puppeteer.launch({ args: ['--no-sandbox'] })
const page = await browser.newPage()
await page.goto(link, { waitUntil: 'networkidle2'})
const data = await page.evaluate('theme.name');
await browser.close()
return data
})(link)
.then(data => {
res.render("index", {data: data, siteUrl: link});
})
})
You can extend the async part to the whole route handler and do whatever you want on catch:
app.post('/results', async (req, res) => {
try {
const link = req.body.link
const browser = await puppeteer.launch({ args: ['--no-sandbox'] })
const page = await browser.newPage()
await page.goto(link, { waitUntil: 'networkidle2'})
const data = await page.evaluate('theme.name')
await browser.close()
res.render("index", {data: data, siteUrl: link})
} catch(e) {
// redirect or whatever
res.redirect('/')
}
});
I am relatively new to selenium and javascript. I am trying to run multiple seleniumjs test files sequentially. To do this I have created another js file (testAll) in which I call all the exported test functions I have created in separate files. I am running into an issue with where I am defining the driver and feel like I'm in a bit of a catch 22. When I define the driver within the test async function itself it works fine but when I then transfer the driver definition to my testAll file to avoid multiple browser windows opening up then I receive a 'cannot read property 'get' of undefined' message. This must be referring to my driver as an undefined variable but I can't see a way to get around this. I have included an example test.js file and my testAll.js file code below:
testAll.js:
const servicesPage = require('./servicesPage');
const organisations = require('./organisations');
const {By , Builder, until} = require('selenium-webdriver');
const allServices = async () => {
const driver = await new Builder().forBrowser('chrome').build();
try {
await servicesPage(driver);
await organisations(driver);
}
finally{
await driver.quit();
}
}
allServices();
test.js:
//Setup
const {By , Builder, until} = require('selenium-webdriver');
const properties = require('../test_Properties')
const authentication = require('../mainAuth');
const assert = require('assert');
const organisations = async (driver) => {
try {
//Execution
await driver.get(properties.servicesUrls.orgsPage);
await authentication(driver);
await driver.wait(until.elementLocated(By.linkText('Request an organisation')), 7000);
await driver.findElement(By.linkText('Request an organisation')).click();
//Assert organisations request page and click back
let rqstOrg = await driver.findElement(By.tagName('h1')).getText();
assert.equal(rqstOrg , 'Request an organisation' , 'Request an organisation heading does not
match');
await driver.findElement(By.className('link-back')).click();
//Assert organisations page
let orgTitle = await driver.getTitle();
assert.equal(orgTitle , 'Organisations' , 'Organisations title does not match');
await driver.findElement(By.linkText('Sign out')).click();
} catch (e) {
throw e;
}
};
module.exports = organisations;
I'm trying to use google play scraper to scrape 100 applications, and show the results of those applications with all the permissions.
I've written multiple .js files to run via node, and all of them have returned errors.
I am not sure how to use promises here, so this code will not return any results, but does not give me a 404
var scraper = require("google-play-scraper")
// get a list of apps
var apps = scraper.search({
term: "foo",
num: 100,
})
var meta = []
// function to get meta from the app
function getMeta(item) {
var appId = item.appId
var metadata = scraper.permissions({
appId: appId,
})
// append metadata to meta somehow idk js that well anymore
}
// console.log(apps)
// await apps.forEach(getMeta)
async function printFiles () {
const files = await apps.forEach(getMeta)
for await (const file of fs.readFile(file, 'utf8')) {
console.log(contents)
}
}
the code sample below gives a 404 pointing to the location of the installed module
var gplay = require('google-play-scraper');
gplay.app({appId: 'com.dxco.pandavszombies'})
.then(console.log, console.log);
Here is the error for the second code sample.
Error: App not found (404)
at C:\Users\Matt\Documents\node_modules\google-play-scraper\lib\utils\request.js:44:19
at processTicksAndRejections (internal/process/task_queues.js:93:5)
{
status: 404
}