Web Scraping With Puppeteer In React JS [duplicate] - javascript

This question already has an answer here:
How to make Puppeteer work with a ReactJS application on the client-side
(1 answer)
Closed 2 years ago.
I am trying to use the web scraper from https://github.com/BlakeStevenson/populartimes.js in my React JS project for school.
I keep getting an error that says:
../node_modules/populartimes.js/node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in ...
I did some research and I believe its an issue with how puppeteer interacts with React JS. Is there anyway I can fix this?
This is all the code that I am using that is associated with the scraper:
import React, { Component } from 'react';
import { Map, GoogleApiWrapper, InfoWindow, Marker } from 'google-maps-react';
let temp;
const populartimes = require('populartimes.js');
populartimes('ChIJKznozuF0hlQRbfbe7fJJ1rM').then(out => {temp=out.now.percent});
I have installed both websocket and puppeteer.

The problem is that you are trying to use populartimes.js/puppeteer in your React.Js app (on the client/frontend side) which is not possible at the moment. Unpackaged versions of puppeteer neither won't work even if you've added them in your HTML source.
You need to use populartimes.js/puppeteer on the server (backend) side. So if your app is React.Js-only so far: you will need a Node.Js server as well to run the scrapers and to make their output data available to your React.Js app via an endpoint (most probably you would choose Express for this purpose).
Edit: To get some ideas about the relationship between the Node.Js backend, Express and React.Js client app I can recommend this tutorial on dev.to.

Related

Why do I need to put http://localhost:3000 for fetching from API routes in Next.js [duplicate]

This question already has an answer here:
Fetch error when building Next.js static website in production
(1 answer)
Closed 5 months ago.
I made an API route in the pages folder on a NextJS project. It works fine, I can fetch data from it by going to the URL directly such as http://localhost:3000/api/tv/popular. I want to fetch the data in getStaticProps and then forward to the components. It works fine if I do it this way
export const getStaticProps: GetStaticProps = async (ctx) => {
const { data } = await axios.get("http://localhost:3000/api/tv/popular");
return {
props: {
popularContent: data,
},
};
};
But if I remove local host and send request to axios.get("/api/tv/popular") it fails.
Why is this, I thought this should be possible in NextJs to not include localhost
As per API concept, it has to come from a server correct?
Hence Nextjs gives the same server where it runs. Hence you have to use localhost:3000 or any port you give
In future it will be a problem for you while production.
Hence use .env file
NEXT_PUBLIC_BASE_URL = http://localhost:3000
and use it in your api calls whenever you go for production you can change the same.
Hope it clarifies something.

How to use google calendar api for JavaScript with npm install?? Or is it possible to use google calendar api for nodejs in browser in Nextjs?

I want to use google calendar api as library in Next.js without using _document.tsx. In that case, I have come up with 2 ways which might be possible as below;
Use google calendar api for JavaScript with npm install
Use google calendar api for node.js in browser in Next.js
Regarding 1, there is seemingly no ways to npm install xxx in the official site here, but <script src="https://apis.google.com/js/api.js"></script> is said to use. This is not what I wanted because I use Next.js so I wanna use this library using npm install.
As for method 2, npm install googleapi is introduced on the official site here, but I'm still concerned that it may only work on the server side. Since I am creating a dashboard-like site, I want to call the API from the client side (browser) when users request their dashboard, so if I can only run it on the server side, it will be a problem for me!
Anyone can answer that??
Method 1
This approach is viable. You mentioned not wanting to use _document.tsx which makes me think you're not sure where to place the <script> tag that imports 3rd-party JS or if that's even allowed in Next.js. It is — you would use Next.js' custom <Script> component instead which can be used in other pages and components. You'd just have to port this example over to Next.js/React.
Method 2
The googleapis (or #googleapis/calendar) package will only work on the server side, but you can still call/access them from the client side (browser) if you import them into API pages. For example, pages/api/gcal.ts would import #googleapis/calendar then pages/index.tsx would make a client-side data request to http://localhost:3000/api/gcal.
So for your case, I think method 2 is what you'd want.
In case people are still looking for a definitive working solution event after reading #Mark G's response above here is the way I implemented it.
Package used:
npm i googleapis --save
Step 1: Create a googleService.js file under the /pages/api
Step 2: initialized the google javascript object over here as this runs on server side this will work fine
// Import the Google Calendar API
const { google } = require('googleapis')
export default function handler(req, res) {
// Fetch Calendar Service Instance
if (req.method === 'GET') {
res.status(200).json({
googleApiInstance: google.calendar({
version: 'v3',
auth: process.env.NEXT_PUBLIC_G_API_KEY,
}),
})
}
}
Step 3: Used this res object to set the Google API instance object in my service file using the useEffect hook on my main application landing page. As I didn't want to create multiple API handling in the pages/api way
useEffect(() => {
fetch(`api/googleService`).then((res) => {
if (res.status !== 200) {
throw new Error(data.message)
}
res.json().then((data) => {
if (data?.googleApiInstance) {
GoogleService.initService(data.googleApiInstance)
}
})
})
}, [])
Not the ideal way I would have loved to do this, but hope this helps for others.

Avoiding Net.createConnection errors in Node

I am currently building an app with React and Node.js
In this app, I need to query a database on my own server with the following function, located in a separate file called "database.js"
const fetchQuery = util.promisify(con.query).bind(con)
// Get all the tracks for a given date from the
const fetchTracks = async (date) => {
const rows = await fetchQuery("SELECT * FROM tracks WHERE playlistDate = '"+date+"'");
}
This works perfectly when I run the file with Node from the command line. However, when I attempt to import it into my react app with
import { fetchTracks, addTracks } from '../scripts/database'
I begin to get errors in the database file, specifically Unhandled Rejection (TypeError): Net.createConnection is not a function on my fetchQuery call.
From what I've read, this happens when attempting to call the function from the browser, as that would pose a security risk. However, as I understand it, all node operations are performed on the server side, right? Why would I be getting this flag when the database is supposedly queried before the page is served? What do I need to do amend this?
React runs in the browser, so as soon as you include database.js in your React app, it's running in the browser with the rest of React, so of course you'll get the error. Making this work gives two options:
expose an API endpoint from Node for React to call, and that API endpoint calls database.js, or,
investigate server-side rendering for React, where some of your React app -- particular the more static parts of it like your main menu -- are created on the Node side and only the final HTML is sent to the browser. (This is a large topic all by itself though.)
By your example which takes a date parameter for the SQL, I'm guessing #2 isn't an option.

Application Insights log to wrong target

I have two web apps. Both have App Insights JS SDK added to their views.
One is big web app and the second one is small microservice with one page. They have different instrumentation keys.
The problem is that we find logs from the "big app" in the microservice AI resource. This probably happens when something wrong happens in the "big app", user redirects the the microservice page and then exists the site or closes the window. All logs in the buffer are sent from microservice page using AI JS SDK, even though they did not happen there.
The above are just my assumptions, but I do not see any other explenation.
I do not want to use one AI resource, because apps have to be monitored separately.
Do you have any ideas how to solve the problem? Or maybe there might be different reason for it?
If you are using the latest version of the Application Insights JS SDK, you can specify a namePrefix as part of your configuration. This will prefix all cookie/sessionstorage/localstorage keys with this string, so that they do not conflict with other instances of the SDK.
import { ApplicationInsights } from '#microsoft/applicationinsights-web'
const appInsights = new ApplicationInsights({ config: {
instrumentationKey: 'YOUR_INSTRUMENTATION_KEY_GOES_HERE',
namePrefix: 'my_app'
} });
appInsights.loadAppInsights();
appInsights.trackPageView();

Nodejs website + react

I'm new in react world.
So I want to make an website with express + react. My question is, I have to make 2 servers 1 to my cliente-side(react) and another to my server-side(express)?? or I can create just 1 server with both, like express + ejs, blade?
(Sorry for my english)
Thanks,
You could use only one server if you wanted to. Setup your express server to return your react bundle on one url, all your api requests on other dedicated urls (same goes for static content such as images), and fall back to returning your index.html on all other urls.
You can use one server by setting up react to use a proxy.
Read this awesome article. Link

Categories