I'm using express and I have the following end-point:
router.get('/', async (_req, res) => {
const customers = await Customer.find().sort({ name: 1 });
res.status(200).json(customers);
});
I would like to set a debugger here:
router.get('/', async (_req, res) => {
debugger // <--- Adding this here
const customers = await Customer.find().sort({ name: 1 });
res.status(200).json(customers);
});
Now I run my app like so:
node inspect app.js
Once in the debugger, I try to use the repl and/or use the inspect console in Chrome... but when I try to interact with my Customer model with the await keyword, I get:
SyntaxError: await is only valid in async function
I would like to play with this query in real-time while debugging the current state of the app. Is there a way to achieve this in node/js?
Related
I am trying to make a webscraper, that outputs certain data from node js into the javascript, or html file im working on. Its important that the data of multiple sub pages can be scraped (that I have no code access to) and be displayed in the same html or js file. The problem is that I cant output the results I get from the axios function into global. If i could my problem would be solved.
So far I have been trying to use axios to get the data I need and cheerio to modify it. I created a const named "articles" where I pushed in every title I needed from the website im scraping.
const axios = require('axios')
const cheerio = require('cheerio')
const express = require('express')
const hbs = require('hbs')
const url = 'https://www.google.com/'
const articles = []
axios(url)
.then(response => {
const html = response.data
const $ = cheerio.load(html)
$('.sprite', html).parent().children('a').each(function() {
const text = $(this).attr('title')
articles.push({
text
})
})
console.log(articles)
const finalArray = articles.map(a => a.text);
console.log(finalArray)
}).catch(err => console.log(err))
That works well so far. If I ouput the finalArray I get the array I want to. But once im outside of the axios function the array is empty. Only way it worked for me is when I put the following code inside the axios function, but in this case I wont be able to scrape multiple websides.
console.log(finalArray) //outputs empty array
// with this function I want to get the array displayed in my home.hbs file.
app.get('/', function(req, res){
res.render('views/home', {
array: finalArray
})
})
Basicly all I need is to get the finalArray into global so I can use it in the app.get function to render the Website with the scraped data.
There are two cases here. Either you want to re-run your scraping code on each request, or you want to run the scraping code once when the app starts and re-use the cached result.
New request per request:
const axios = require("axios");
const cheerio = require("cheerio");
const express = require("express");
const scrape = () =>
axios
.get("https://www.example.com")
.then(({data}) => cheerio.load(data)("h1").text());
express()
.get("/", (req, res) => {
scrape().then(text => res.json({text}));
})
.listen(3000);
Up-front, one-off request:
const scrapingResultP = axios
.get("https://www.example.com")
.then(({data}) => cheerio.load(data)("h1").text());
express()
.get("/", (req, res) => {
scrapingResultP.then(text => res.json({text}));
})
.listen(3000);
Result:
$ curl localhost:3000
{"text":"Example Domain"}
It's also possible to do a one-off request without a callback or promise that uses a race condition to populate a variable in scope of the request handlers as well as the scraping response handler. Realistically, the server should be up by the time the request resolves, though, so it's common to see this:
let result;
axios
.get("https://www.example.com")
.then(({data}) => (result = cheerio.load(data)("h1").text()));
express()
.get("/", (req, res) => {
res.json({text: result});
})
.listen(3000);
Eliminating the race by chaining your Express routes and listener from the axios response handler:
axios.get("https://www.example.com").then(({data}) => {
const text = cheerio.load(data)("h1").text();
express()
.get("/", (req, res) => {
res.json({text});
})
.listen(3000);
});
If you have multiple requests you need to complete before you start the server, try Promise.all. Top-level await or an async IIFE can work too.
Error handling has been left as an exercise.
Problem has been resolved. I used this code, instead of the normal axios.get(url) function:
axios.all(urls.map((endpoint) => axios.get(endpoint))).then(
axios.spread(({data:user}, {data:repos}) => {
with "user", and "repos" I am now able to enter both URL data and can execute code regarding the URL i like to chose in that one function.
router.get("/(A|B)/account/", async (req, res) => {});
How to do I reference the (A|B) inside of the async function?
I guess your route responsibility is getting account information of only A or B. So let's change your router path to /account/:name(A|B), then your express router will look like this:
router.get("/account/:name(A|B)", async (req, res) => {
const name = req.params; // A or B
});
Only 2 kinds of requests are handled by this router:
GET /account/A
or
GET /account/B
I am in a programming bootcamp and we are currently learning about fullstack JavaScript. We are using the Express framework and I have opted to use a SQLite3 database. This is a basic app for maintaining books at a local community center. I am required to write up to 3 unit tests and the one that I am having difficulty with is the GET / route. This route is intended to pull data from the database and render them to create a list of books currently in the database when you visit the root route.
I want to test whether the bookList variable containing the awaited Book.findAll() Sequelize function is actually sending an array of objects containing row data to the Pug template engine to render the list of books.
When I host locally, this is working as intended. I am just puzzled on how to have that reflected in my unit tests. I've attached the routes/index.js, views/index.pug & the test/readOp_test.js containing all of my tests including the one for the GET / route. Please let me know if I need to share more of the code and will reply with what is needed for anyone willing to help me.
readOp_test.js
let chai = require('chai');
let chaiHTTP = require('chai-http');
let chaiPromised = require('chai-as-promised');
let application = require('../app');
let expect = require('chai').expect;
chai.use(chaiHTTP);
chai.use(chaiPromised);
chai.should();
// Unit Test Suite
describe('Unit Tests for Endpoints to meet Code Louisville Project Requirements', () => {
//Test Spec for READ ALL operation
/* it('should pull data from all rows in the database', (done) => {
chai.request(application)
.get('/')
.end((err, res) => {
res.body.should.be.a('object');
done();
})
}); */
//Test Spec for READ-by-id operation
it('should pull data from the below id', (done) => {
const id = 2;
chai.request(application)
.get(`/${id}`)
.end((err, res) => {
res.body.should.be.a('object');
done();
})
});
it('should return status 404 when READING the below id which is not in the database', (done) => {
const id = 12;
chai.request(application)
.get(`/${id}`)
.end((err, res) => {
res.should.have.status(404);
done();
})
});
//Test Spec for CREATE operation
it('shouldnt POST the below book entry since the title value is an empty string.', async() => {
let res = await chai
.request(application)
.post('/')
.type('form')
.send({title: '', author: "al-Gibber", genre: "Math", language: "Arabic"})
expect(res.status).to.equal(500)
});
it('should POST the below book entry since all properties contain actual string values.', async() => {
let res = await chai
.request(application)
.post('/')
.type('form')
.send({title: 'Gibberish', author: "al-Gibber", genre: "Math", language: "Arabic"})
expect(res.status).to.equal(200)
});
});
routes/index.js
// Requiring Express to enable access to the framework's methods, properties, & other tools.
const express = require('express');
// Router handles HTTP requests.
const router = express.Router();
// These two variables are ensuring `routes/index.js` has access to the database & its models.
const db = require('../db');
const { Book } = db.models;
//This is a convenient function that handles async/await.
function asyncHandler(cb){
return async(req, res, next) => {
try {
await cb(req, res, next)
} catch(error){
// Forward error to the global error handler
next(error);
}
}
}
//This route handles the initial load of the app, the root route, & will draw data from each row in the database & display it on the homepage.
//This route is a READ operation distinctly a READ ALL operation.
router.get('/', asyncHandler(async (req, res) => {
const bookList = await Book.findAll();
//The use of .render() method ensures that the `index.pug` template is rendered when user visits the root directory.
//{ bookList } is an object containing data from each row in the database to be used in the Pug template.
res.render('index', {bookList});
}));
//This route handles adding a book to the app's database by rendering the newBook.pug template which contains a form to collect the column data for the database.
//This route is the beginning of the CREATE operation for this app.
router.get('/new', (req, res) => {
res.render('newBook', { book: {}});
});
//This route handles the data collected by newBook.pug's form. It creates the database entry and posts the data.
//This route is the closure for the CREATE operation for this app.
router.post('/', asyncHandler(async (req, res) => {
console.log(req.body);
let book;
try {
book = await Book.create(req.body);
console.log(book);
res.redirect('/');
} catch (error) {
throw error;
}
}));
//This route handles rendering found data for each book that would be clicked on in the index.pug template.
//This route is another distinct READ operation that is reading an entry's data based on the primary key ID selected on index.pug.
router.get("/:id", asyncHandler(async (req, res) => {
const book = await Book.findByPk(req.params.id);
if(book) {
res.render("bookViewing", { book });
} else {
res.sendStatus(404);
}
}));
//Ensures that all routes written in this folder can be used in the root's `app.js` file.
module.exports = router;
index.pug
doctype html
html(lang="en")
head
title Library
link(rel='stylesheet', href='/static/stylesheets/style.css')
body
div#app
div#container
h1 Library
h2 Our Book Selection
each book in bookList
book
p
span= book.title
|
a(href=`/${book.id}`) Learn More
p This library is open to all in our local community!
a(href="/new") Add New Book
Screenshot of the correctly rendered index.pug template
Screenshot of rendered PUG template
I am using Node and Express with the the mssql npm package to connect to an SQL Server database. I do this in my app.js file which sets up a global variable to create a connectionPool to the database like so (I omitted some boilerplate stuff for brevity):
const mssql = require('mssql/msnodesqlv8'); // needs to use msnodesqlv8 driver for Windows Authentication to work
const express = require('express');
const app = express();
const DB_MYDB_Config = {
server: "localhost",
database: "MYDB",
options: {
trustedConnection: true // Windows auth enabled hence no username/password required
}
};
global.MSSQL_MYDB = new mssql.ConnectionPool(DB_MYDB_Config); //connectionPool available to whole app
I have a Model file called offer.js which just does this:
async function getOffersAll() {
await MSSQL_MYDB.connect(); // connects to the database
try {
var result = await MSSQL_MYDB.request(MSSQL_MYDB).query('SELECT Top(1) * FROM [dbo].[Offer]');
return result; // returns the recordset of data
} catch (error) {
console.log(error);
} finally {
if (MSSQL_MYDB){
try {
await MSSQL_MYDB.close(); // closes the DB connection
}
catch (error) {
console.log('Error closing connection');
}
}
}
}
exports.getOffersAll = getOffersAll;
So far so good. I then have a Controller file index.js which doesn't really work (explained with comments):
var router = require('express').Router();
const Offer = require('../models/offer'); // the `offer.js` file
/* the function below works perfectly */
(async function () {
var rsOffersAll = await Offer.getOffersAll();
console.dir(rsOffersAll); // works great! recordset rsOffersAll is printed to console
})();
/* this is where it goes wrong even after commenting out the test function above */
router.get('/', async function(req, res) {
var rsOffersAll = await Offer.getOffersAll(); // this just hangs and eventually I get a login failed error for SQL Server.
console.dir(rsOffersAll);
res.render('index', { pagetitle: 'Homepage'}); // never renders
});
module.exports = router;
My question is why does the first async function() that awaits a result from Offer.getOffersAll() not fail, but the same async function placed within the router.get('/') handler fails with a login error? If I remove the var rsOffersAll = await Offer.getOffersAll(); from the router.get('/') handler then the page renders, but of course I have no data to pass to it.
The exact same thing happens even if I store the test function's value in a variable and try to put it in the router.get() handler like this:
async function getOffersTest() {
return await Offer.getOffersAll();
}
router.get('/', async function(req, res) {
var rsOffersAll = await getOffersTest(); // still breaks
console.dir(rsOffersAll);
res.render('index', { pagetitle: 'Homepage'}); // still never renders
});
My ultimate question how do I fix this so it just works the way it should in that when the homepage is visited, the router waits for the data to be returned from the database and then I can pass it to my view or just print to the console if I want?
because of this line global.MSSQL_MYDB = new mssql.ConnectionPool(DB_MYDB_Config);,
when you execute this code outside of router,
(async function () {
var rsOffersAll = await Offer.getOffersAll();
console.dir(rsOffersAll); // works great! recordset rsOffersAll is printed to console
})();
getOffersAll has access to global variable,
and you can successfully connect with db in line await MSSQL_MYDB.connect(); //
but as for router, global scope is the current module.exports object, not the global object.
Solution
you can set MSSQL_MYDB in app like this,
app.set('MSSQL_MYDB')
then you can get this same variable in following function like this
router.get('/', async function(req, res) {
const MSSQL_MYDB = req.app.get('MSSQL_MYDB')
var rsOffersAll = await getOffersTest(MSSQL_MYDB );
console.dir(rsOffersAll);
res.render('index', { pagetitle: 'Homepage'}); // still never renders
});
This whole problem was just solved and it is a bug or something in node mssql package. It only works if you provide a username and password. If you use options: {trustedConnection: true } to enable windows authentication, then the router can never log in. I don't know why this is the case, but if you just supply a username and password (I used sa for testing) then it works fine.
I'm kinda new to module creation and was wondering about module.exports and waiting for async functions (like a mongo connect function for example) to complete and exporting the result. The variables get properly defined using async/await in the module, but when trying to log them by requiring the module, they show up as undefined. If someone could point me in the right direction, that'd be great. Here's the code I've got so far:
// module.js
const MongoClient = require('mongodb').MongoClient
const mongo_host = '127.0.0.1'
const mongo_db = 'test'
const mongo_port = '27017';
(async module => {
var client, db
var url = `mongodb://${mongo_host}:${mongo_port}/${mongo_db}`
try {
// Use connect method to connect to the Server
client = await MongoClient.connect(url, {
useNewUrlParser: true
})
db = client.db(mongo_db)
} catch (err) {
console.error(err)
} finally {
// Exporting mongo just to test things
console.log(client) // Just to test things I tried logging the client here and it works. It doesn't show 'undefined' like test.js does when trying to console.log it from there
module.exports = {
client,
db
}
}
})(module)
And here's the js that requires the module
// test.js
const {client} = require('./module')
console.log(client) // Logs 'undefined'
I'm fairly familiar with js and am still actively learning and looking into things like async/await and like features, but yeah... I can't really figure that one out
You have to export synchronously, so its impossible to export client and db directly. However you could export a Promise that resolves to client and db:
module.exports = (async function() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true
});
const db = client.db(mongo_db);
return { client, db };
})();
So then you can import it as:
const {client, db} = await require("yourmodule");
(that has to be in an async function itself)
PS: console.error(err) is not a proper error handler, if you cant handle the error just crash
the solution provided above by #Jonas Wilms is working but requires to call requires in an async function each time we want to reuse the connection. an alternative way is to use a callback function to return the mongoDB client object.
mongo.js:
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb+srv://<user>:<pwd>#<host and port>?retryWrites=true";
const mongoClient = async function(cb) {
const client = await MongoClient.connect(uri, {
useNewUrlParser: true
});
cb(client);
};
module.exports = {mongoClient}
then we can use mongoClient method in a diffrent file(express route or any other js file).
app.js:
var client;
const mongo = require('path to mongo.js');
mongo.mongoClient((connection) => {
client = connection;
});
//declare express app and listen....
//simple post reuest to store a student..
app.post('/', async (req, res, next) => {
const newStudent = {
name: req.body.name,
description: req.body.description,
studentId: req.body.studetId,
image: req.body.image
};
try
{
await client.db('university').collection('students').insertOne({newStudent});
}
catch(err)
{
console.log(err);
return res.status(500).json({ error: err});
}
return res.status(201).json({ message: 'Student added'});
};