I am trying to fetch data from an API in Nodejs and then send the data to Reactjs. I can't directly fetch data in Reactjs as I need to provide the API key so for this, I am using Nodejs. I am using Axios to fetch the data. The problem is that since Axios sends the data only after it has completely fetched all the data, it takes more than 3 4 seconds to display the data which is not very good. I want to know how can I display data after every interval or as soon as Axios fetches some data, keep displaying that and loading the rest part simultaneously. My code at the backend part is
const express = require('express');
const axios = require('axios');
const bodyParser = require('body-parser');
const cors = require('cors');
const app = express();
app.use(cors());
app.use(express.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.get('/getdata', function (req, res) {
const fetchData = async () => {
const data = await axios(`${APIurl}?${APIKey}`)
res.send(data.data);
}
fetchData();
})
app.listen(5000);
The code for the Reactjs part is like this.
import React, { useState, useEffect } from 'react'
import axios from 'axios';
const Projects = () => {
const [data, updateData] = useState([]);
useEffect(() => {
axios.get('http://localhost:5000/getdata')
.then(res => updateData(res.data))
.catch(error => console.log("Error"))
}, [])
return (
{data.map(project => (
<div key={project["id"]}>
<div>{project["name"]}</div>
</div>
))}
)
}
export default Projects
So how can I send data from backend code so that it displays data after every some particular interval?
Seems like a good use case for WebSocket or maybe WebSocketStream API.
The implementation details will depend greatly on whether you want to implement any of these yourself or if you want to use some library/service (for instance, socket.io) and which of these 2 APIs suits best to your project.
Related
I am trying to make a webscraper, that outputs certain data from node js into the javascript, or html file im working on. Its important that the data of multiple sub pages can be scraped (that I have no code access to) and be displayed in the same html or js file. The problem is that I cant output the results I get from the axios function into global. If i could my problem would be solved.
So far I have been trying to use axios to get the data I need and cheerio to modify it. I created a const named "articles" where I pushed in every title I needed from the website im scraping.
const axios = require('axios')
const cheerio = require('cheerio')
const express = require('express')
const hbs = require('hbs')
const url = 'https://www.google.com/'
const articles = []
axios(url)
.then(response => {
const html = response.data
const $ = cheerio.load(html)
$('.sprite', html).parent().children('a').each(function() {
const text = $(this).attr('title')
articles.push({
text
})
})
console.log(articles)
const finalArray = articles.map(a => a.text);
console.log(finalArray)
}).catch(err => console.log(err))
That works well so far. If I ouput the finalArray I get the array I want to. But once im outside of the axios function the array is empty. Only way it worked for me is when I put the following code inside the axios function, but in this case I wont be able to scrape multiple websides.
console.log(finalArray) //outputs empty array
// with this function I want to get the array displayed in my home.hbs file.
app.get('/', function(req, res){
res.render('views/home', {
array: finalArray
})
})
Basicly all I need is to get the finalArray into global so I can use it in the app.get function to render the Website with the scraped data.
There are two cases here. Either you want to re-run your scraping code on each request, or you want to run the scraping code once when the app starts and re-use the cached result.
New request per request:
const axios = require("axios");
const cheerio = require("cheerio");
const express = require("express");
const scrape = () =>
axios
.get("https://www.example.com")
.then(({data}) => cheerio.load(data)("h1").text());
express()
.get("/", (req, res) => {
scrape().then(text => res.json({text}));
})
.listen(3000);
Up-front, one-off request:
const scrapingResultP = axios
.get("https://www.example.com")
.then(({data}) => cheerio.load(data)("h1").text());
express()
.get("/", (req, res) => {
scrapingResultP.then(text => res.json({text}));
})
.listen(3000);
Result:
$ curl localhost:3000
{"text":"Example Domain"}
It's also possible to do a one-off request without a callback or promise that uses a race condition to populate a variable in scope of the request handlers as well as the scraping response handler. Realistically, the server should be up by the time the request resolves, though, so it's common to see this:
let result;
axios
.get("https://www.example.com")
.then(({data}) => (result = cheerio.load(data)("h1").text()));
express()
.get("/", (req, res) => {
res.json({text: result});
})
.listen(3000);
Eliminating the race by chaining your Express routes and listener from the axios response handler:
axios.get("https://www.example.com").then(({data}) => {
const text = cheerio.load(data)("h1").text();
express()
.get("/", (req, res) => {
res.json({text});
})
.listen(3000);
});
If you have multiple requests you need to complete before you start the server, try Promise.all. Top-level await or an async IIFE can work too.
Error handling has been left as an exercise.
Problem has been resolved. I used this code, instead of the normal axios.get(url) function:
axios.all(urls.map((endpoint) => axios.get(endpoint))).then(
axios.spread(({data:user}, {data:repos}) => {
with "user", and "repos" I am now able to enter both URL data and can execute code regarding the URL i like to chose in that one function.
I am making API calls to Squarespace for an inventory management system. I've set up a Node.js and Express server, and have been using http-proxy-middleware to set up a proxy and make GET requests.
I am able to generate the GET requests successfully on my localhost - an HTML pre-tag is filled with all of the JSON data of the request. However, I am completely clueless on how to handle and use the data further that was returned to me. When I make a call and receive Pending orders, I want to pull JSON data from the returned request body of the orders, such as SKU numbers for products purchased.
const { response } = require('express');
const express = require('express');
require("dotenv").config();
const { createProxyMiddleware, responseInterceptor } = require('http-proxy-middleware');
const router = express.Router();
const PORT = 3000;
const HOST = "localhost";
const GET_ORDERS_URL = process.env.SS_GET_ORDERS_URL;
const API_KEY = process.env.SS_AUTH;
const app = express();
const proxy = app.use("/testing", createProxyMiddleware({
target: GET_ORDERS_URL,
headers: {
'Authorization': API_KEY,
'User-Agent': 'halp me'
},
changeOrigin: true,
pathRewrite: {
[`^/testing`]: '',
},
selfHandleResponse: true, //
onProxyRes: responseInterceptor(async (responseBuffer, proxyRes, req, res) => {
var orderResponse = responseBuffer.toString('utf-8');
return orderResponse;
}),
}));
// Start Proxy
app.listen(PORT, HOST, () => {
console.log(`Starting Proxy at ${HOST}:${PORT}`);
});
The API request returns JSON data, which I would love to use and process for the next part of my inventory management. I'm trying to figure out why I can't get return orderResponse; to output anything at all.
I have tried every variation of returning a variable I can imagine, console.logged a million things - any guidance to what I'm missing here would be greatly appreciated!
I am currently having an issue with my React Application not working on any other browser other than Chrome. The Javascript loads just fine with no errors on Chrome and the application is currently fully deployed on Heroku (link: https://weathrd.herokuapp.com/).
In regards to my application, I have a search query set up in the "overview.js" component that creates a "get" request, with a parameter passed in that gets fed into the weather api I am using. Then, I retrieve the json information from the "/forecast" page and feed that back into "overview.js" to display on the screen.
I do not have any regex notation within any of my code, so I don't think that would be an issue here. I also have fully updated my Heroku deploy code and I do not think there is some sort of confusion on Heroku? Regardless, here is my server code, overview component code, and the error I am receiving on Safari:
server code:
const PORT = process.env.PORT || 8000;
const path = require('path');
const express = require('express');
const cors = require('cors');
const axios = require('axios');
require('dotenv').config();
const app = express();
app.use(cors());
app.use(express.static("public"))
app.get('/', (req, res) => {
res.json('hi');
});
app.get('/forecast', (req, res) => {
const options = {
method: 'GET',
url: `http://api.weatherapi.com/v1/forecast.json?`,
params: {
q: req.query.city,
key : process.env.REACT_APP_API_KEY,
days: '3',
api: 'no',
alerts: 'no',
},
};
axios.request(options).then((response) => {
res.json(response.data);
}).catch((error) => {
console.log(error);
});
});
app.listen(PORT, () => console.log(`Server running on http://localhost:${PORT} `))
Safari Error:
The error also mentions the component from which I am making the API request from "overview.js", so here is that code also:
overview.js
import React, {useState} from 'react';
import './overview.css';
import { RecentSearches } from '../Recent Searches/recentSearches';
import { Hourly } from '../Hourly/hourly';
import { Fiveday } from '../5 Day Forecast/fiveday';
import 'animate.css';
const axios = require('axios');
export function Overview() {
const [forecast, setForecast] = useState(null);
// this callback function receives the searched city entered from recentSearches and applies it to fetchForecast
const getSearch = (searchedCity) => {
fetchForecast(searchedCity);
};
async function fetchForecast(searchedCity) {
const options = {
method: 'GET',
url: 'https://weathrd.herokuapp.com/forecast',
params: {city: searchedCity}
};
axios.request(options).then((response) => {
console.log(response.data);
setForecast(response.data);
}).catch((error) => {
console.log(error);
})
};
return (
<div>
<div className='jumbotron' id='heading-title'>
<h1>Welcome to <strong>Weathered</strong>!</h1>
<h3>A Simple Weather Dashboard </h3>
</div>
<div className='container-fluid' id='homepage-skeleton'>
<div className='d-flex' id='center-page'>
<RecentSearches getSearch={getSearch}/>
<Hourly forecast={forecast}/>
</div>
</div>
<Fiveday forecast={forecast}/>
</div>
)
};
Thanks for any assistance!
I'm a beginner, and I'm trying to figure out how to send a variable generated from a function in react to the backend server side.
The user clicks on a button and a json object called rowObject is generated in home.jsx . I want to send it to backend to post.js to save it to the database. How do I achieve this?
Your front end would make a request to your server, perhaps with something like the browsers built in fetch() function.
For example:
function MyComponent() {
function onClick() {
fetch(
'/some/path/here',
{
method: 'POST',
body: JSON.stringify({ myData: 123 })
}
)
}
return <div onClick={onClick}>Click Me</div>
}
Then on the backend in express you would have something like:
const express = require('express')
const app = express()
const port = 3000
app.post('/some/path/here', (req, res) => {
dbOrSomething.saveSomewhere(req.body) // your implementation here
res.send('Saved!')
})
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
I'm reading the book introducing NodeJS with a simple web application example. The requirement in the example is that there are several data store classes in its own module, and we need to adopt the data store dynamically by setting environment variable. The code snippets of the example is something like following:
// memory-store.mjs
// The data store for storing data in memory
export default class MemoryStore {
// Some CRUD operation
}
// fs-store.mjs
// The data store for storing data into file system
export default class FSStore {
// Some CRUD operation
}
// store.mjs
// Provide a async function to import data store dynamically and
// set the instance to variable store, which is exported
let store;
async function load() {
try {
const moduleName = process.env.MODULE_NAME ?? 'memory';
const storeModule = await import(`./${moduleName}-store.mjs`);
const storeClass = storeModule.default;
store = new storeClass();
return store;
} catch(err) {
throw new Error('Something goes wrong...');
}
}
export { load, store };
// app.mjs
// Import the function to load the data store dynamically and
// the exported store for fetching data list
import express from 'express';
import { load, store } from './store.mjs';
const app = express();
load()
.then(store => {})
.catch(err => console.error(`Exception with error: ${err}`));
app.use('/', (req, res, next) => {
const dataList = store.retrieveAll();
res.send(dataList);
});
The code snippets above is not same as the one in the book overall. But the concept is same. It works fine in my local environment, but I'm wondering isn't there any problem if the request is coming and handled before the data store is imported due that the import function is async operation? Are there other solutions which can fulfill the requirement? Or I'm just missing something that the example from the book is just masterpiece? Thanks in advance!
If you want to guarantee that store has been initialized before any requests are handled by your express app, you could set up the express listener after the load promise has resolved. This would be as simple as the following:
import express from 'express';
import { load, store } from './store.mjs';
const app = express();
app.use('/', (req, res, next) => {
const dataList = store.retrieveAll();
res.send(dataList);
});
load()
.then(() => {
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`);
});
})
.catch(err => console.error(`Exception with error: ${err}`));