We Keep Coding

web scraping for html page but need for repeat on lots link?

I wrote the following code for parse some part of HTML for one URL. I means parse page const URL= 'https://www.example.com/1'
Now I want to parse the next page 'https://www.example.com/2' and so on. so I want to implement a For-Loop manner here.
what is the easiest way that I can use the iteration manner here to
change URL (cover page 1,2,3, ...) automatically and run this code in repeat to parse other pages? How I can use for-loop manner here?
const PORT = 8000
const axios = require('axios')
const cheerio = require('cheerio')
const express = require('express')
const app = express()
const cors = require('cors')
app.use(cors())
const url = 'https://www.example.com/1'
app.get('/', function (req, res) {
res.json('This is my parser')
})
app.get('/results', (req, res) => {
axios(url)
.then(response => {
const html = response.data
const $ = cheerio.load(html)
const articles = []
$('.fc-item__title', html).each(function () {
const title = $(this).text()
const url = $(this).find('a').attr('href')
articles.push({
title,
url
})
})
res.json(articles)
}).catch(err => console.log(err))
})
app.listen(PORT, () => console.log(`server running on PORT ${PORT}`))

Some considerations, if you added CORS to your app, so that you can GET the data, it's useless, you add CORS when you want to SEND data, when your app is going to receive requests, CORS enable other people to use your app, it's useless then trying to use other people's app. And CORS problems happen only in the browser, as node is on the server, it will never get CORS error.
The first problem with your code, is that https://www.example.com/1, even working on the browser, returns 404 Not Found Error to axios, because this page really doesn't exist, only https://www.example.com would work.
I added an example using the comic site https://xkcd.com/ that accepts pages.
I added each axios request to an array of promises, then used Promise.all to wait for all of them:
The code is to get the image link:
const PORT = 8000;
const axios = require("axios");
const cheerio = require("cheerio");
const express = require("express");
const app = express();
const url = "https://xkcd.com/";
app.get("/", function (req, res) {
res.json("This is my parser");
});
let pagesToScrap = 50;
app.get("/results", (req, res) => {
const promisesArray = [];
for (let pageNumber = 1; pageNumber <= pagesToScrap; pageNumber++) {
let promise = new Promise((resolve, reject) => {
axios(url + pageNumber)
.then((response) => {
const $ = cheerio.load(response.data);
let result = $("#transcript").prev().html();
resolve(result);
})
.catch((err) => reject(err));
});
promisesArray.push(promise);
}
Promise.all(promisesArray)
.then((result) => res.json(result))
.catch((err) => {
res.json(err);
});
});
app.listen(PORT, () => console.log(`server running on PORT ${PORT}`));

What does it mean by this async Javascript server app error?

I was setting a weather app website that is connected to another site using a server, and asynchronous javascript was used, but after trying to run the code, an error reading "uncaught syntax error: unexpected end of input" in the last line in the app file...I don't understand what it means and therefore I don't know how to solve it
here's my app file code
/* Global Variables */
const apiKey = "726f360f99f8ed5ce834f19b2f632fd3"
// Create a new date instance dynamically with JS
let d = new Date();
let newDate = +d.getMonth()+1+'.'+ d.getDate()+'.'+ d.getFullYear();
const gen = document.querySelector("#generate");
gen.addEventListener("click", async() =>{
const Zcode = document.querySelector("#zip").value;
const feel = document.querySelector("#feelings").value;
try {
getTemp()
.then(temp =>{
const object = {
date: newDate,
temp: temp,
}
return DealingWithServer()
})
.then(data =>{
UpdateSite(data)
})
}catch(error){
console.log(error);
}
});
async function getTemp (){
const res = await fetch (`https://api.openweathermap.org/data/2.5/weather=?zip=${zipCode}&appid=${apiKey}&units=metric`);
const data= await res.json;
const temp = data.main.temp
return temp
}
async function DealingWithServer (){
await fetch('/recieve', {
method: "POST",
credentials: "same-origin",
headers: {"Content-Type": "application/json"},
body:JSON.stringfy({
date: newDate,
temp: temp,
feel: feel
})
});
const Sres = await fetch('/get', {credentials: "same-origin"});
const Sdata = await Sres.json()
return (Sdata);
}
function UpdateSite (data)
and my server file code
// Setup empty JS object to act as endpoint for all routes
projectData = {};
const port = 3000;
// Require Express to run server and routes
const express= require("express");
const bodyParser = require("body-parser");
const cors = require("cors");
// Start up an instance of app
const app=express()
/* Middleware*/
//Here we are configuring express to use body-parser as middle-ware.
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
// Cors for cross origin allowance
app.use(cors())
// Initialize the main project folder
app.use(express.static('website'));
app.get("/get",(req, res) => {
res.send(projectData)
})
app.post("/recieve", (req, res) => {
projectData =req.body
res.status(200)
})
// Setup Server
app.listen(3000,() =>{
console.log("Server running");
})

It looks to me like you've not finished your UpdateSite function at the bottom. It should read something like this:
function UpdateSite (data) {
// Do things to update the site
}
Since there is no function definition, the input (JavaScript code in this case) has ended unexpectedly - i.e. the parser was not expecting the input to end with function UpdateSite (data)

Express not rendering my React Front End?

I have two repos for the Front End and Back End portions of my project.
The Front End is a simple create-react-app project that hits my Express Back End and received responses from API calls.
I ran npm run build in my Front End project and moved that build folder to the root of my express backend repo.
However, when I try to reach the root page (i.e. localhost:3001), for some reason the response only returns the static html from index.html and doesn't actually render anything.
But if I go to something that has a path like localhost:3001/pokedex/1 then at least I see a correct response coming from the API.
I have a feeling that there is something wrong with the way I'm declaring my paths.
Here is the code on the Front End that is reaching out to the Back End:
import axios from 'axios'
const baseUrl = '/'
const getAll = () => {
const request = axios.get(baseUrl)
return request.then(response => response.data)
}
const getPkm = (id) => {
const request = axios.get(`${baseUrl}pokedex/${id}`)
return request.then(response => response.data)
}
export default { getAll, getPkm }
This is my Express Back End entry index.js:
const express = require('express')
const app = express()
const cors = require('cors')
const axios = require('axios')
//Middleware
app.use(cors())
app.use(express.json())
app.use(express.static('build'))
const unknownEndpoint = (request, response) => {
response.status(404).send({ error: 'unknown endpoint' })
}
let fullPkmList = require('./fullPkmList.json')
function ignoreFavicon(req, res, next) {
if (req.originalUrl.includes('favicon.ico')) {
res.status(204).end()
}
next();
}
app.get('/', (req, res) => {
axios.get(`https://pokeapi.co/api/v2/pokemon/?limit=100`)
.then((list) => res.json(list.data.results))
})
app.get('/pokedex/:id', (request, response) => {
const id = Number(request.params.id)
const pokemon = fullPkmList[id - 1]
if (pokemon) {
axios.all([
axios.get(`https://pokeapi.co/api/v2/pokemon/${id}`),
axios.get(`https://pokeapi.co/api/v2/pokemon-species/${id}`)
])
.then(axios.spread((pokemonResponse, speciesReponse) => {
let pkmResponse = pokemonResponse.data
let speciesResponse = speciesReponse.data
response.json({pkm: pkmResponse, species: speciesResponse })
}))
} else {
response.status(404).end()
}
})
app.use(unknownEndpoint)
const PORT = process.env.PORT || 3001
app.listen(PORT, () => {
console.log(`this is a test ${PORT}`)
})
Code for the Front End: https://github.com/rohithpalagiri/pocketdex
Code for the Back End: https://github.com/rohithpalagiri/pocketdex-backend
To see the issue, you only need to run the backend. I console log the response and in that, you will see the index.html file markup being returned. My goal is to have all of the paths relative so that the root url doesn't really matter. I think that is the part I'm getting stuck on.
I'd appreciate any help!

Getting cannot POST / error in Express

I have a RESTful API that I am using postman to make a call to my route /websites. Whenever I make the call, postman says "Cannot POST /websites". I am trying to implement a job queue and I'm using Express, Kue(Redis) and MongoDB.
Here is my routes file:
'use strict';
module.exports = function(app) {
// Create a new website
const websites = require('./controllers/website.controller.js');
app.post('/websites', function(req, res) {
const content = req.body;
websites.create(content, (err) => {
if (err) {
return res.json({
error: err,
success: false,
message: 'Could not create content',
});
} else {
return res.json({
error: null,
success: true,
message: 'Created a website!', content
});
}
})
});
}
Here is the server file:
const express = require('express');
const bodyParser = require('body-parser');
const kue = require('kue');
const websites = require('./app/routes/website.routes.js')
kue.app.listen(3000);
var app = express();
const redis = require('redis');
const client = redis.createClient();
client.on('connect', () =>{
console.log('Redis connection established');
})
app.use('/websites', websites);
I've never used Express and I have no idea what is going on here. Any amount of help would be great!!
Thank you!

The problem is how you are using the app.use and the app.post. You have.
app.use('/websites', websites);
And inside websites you have:
app.post('/websites', function....
So to reach that code you need to make a post to localhost:3000/websites/websites. What you need to do is simply remove the /websites from your routes.
//to reach here post to localhost:3000/websites
app.post('/' , function(req, res) {
});

Parse request in my simple Node Js server

I'm new to Node and am trying to build a simple server in Node using Express. The requests are in the form of say /input00001/1/output00001. What I need to do is to parse this request and if the flag is 1 (middle value), I need to replace the file \home\inputfiles\input00001.txt with file \home\outputfiles\output00001.txt. How is it possible to do that?
Here is my simple server so far. I'm OK with not using the Express and pure NodeJs if that makes things easier.
const express = require('express');
const app = express();
const port = 8000;
app.get('/', (request, response) => {
response.send('Hello from Express!');
request.param
});
app.get('/*', (request, response) => {
response.send('Start!');
var url = request.originalUrl;
});
app.listen(port, (err) => {
if (err) {
return console.log('something bad happened', err);
}
console.log(`server is listening on ${port} for incoming messages`);
});

You should set up a route that expects these items as url parameters and then use those parameters to do what you want. For example if you're url is /input00001/1/output00001 then you could set up a route like this:
app.get('/:input/:flag/:output', (req, res) => {
var params = req.params
var input = params.input //input0001
var flag = params.flag // 1
var output = params.output //output0001
// now do what you need to with input, flag, and output
if(typeof flag!=='undefined' && flag==1){
var file_name_string = '\home\inputfiles\input00001.txt';
var res = file_name_string.replace("input", "output");
}
console.log(input, flag, output)
res.send("done")
})