I have been bashing my head for days as I cannot find a valid example of async configuration in Sequelize
So as you may know, you can simply config a Sequelize instance like that
const sequelize = new Sequelize('postgres://user:pass#example.com:5432/dbname')
and then declare your Model
const User = sequelize.define('User', {
// Model attributes are defined here
firstName: {
type: DataTypes.STRING,
allowNull: false
},
lastName: {
type: DataTypes.STRING
// allowNull defaults to true
}
}, {
// Other model options go here
});
However what happens when the db credentials comes from an external service?
const credentials = await getDbCredentials();
const sequelize = new Sequelize({credentials})
since sequelize models creation are coupled with the instance creation (unlike many others ORMs) this becomes a big problem.
My current solution is the following:
const Sequelize = require("sequelize");
// Models
const { User } = require("./User");
const env = process.env.NODE_ENV || "development";
const db = {};
let sequelize = null;
const initSequelize = async () => {
if (!sequelize) {
let configWithCredentials = {};
if (env === "production") {
const credentials = await getDbCredentials();
const { password, username, dbname, engine, host, port } = credentials;
configWithCredentials = {
username,
password,
database: dbname,
host,
port,
dialect: engine,
operatorsAliases: 0
};
}
const config = {
development: {
// Dev config
},
production: configWithCredentials,
};
sequelize = new Sequelize(config[env]);
sequelize.authenticate().then(() => {
console.log("db authenticated")
});
});
}
db.User = User;
db.sequelize = sequelize;
db.Sequelize = Sequelize;
};
initSequelize().then(() => {
console.log("done");
});
module.exports = db;
However I feel that this is not a good approach because of the asynchronous nature of the initialization and sometimes the db is undefined.
Is there a better way to approach this thing?
Thanks
You can achieve this with beforeConnect hook, something like this:
sequelize = new Sequelize(config.database, '', '', config);
sequelize.beforeConnect(async (config) => {
config.username = await getSecretUsername();
config.password = await getSecretPassword();
});
Leave the initial credentials empty, then use beforeConnect to mutate the config. Not sure if this is the cleanest way to use it but seems to be working.
https://sequelize.org/master/manual/hooks.html
I think your db is sometimes undefined, because in your async function you're not "waiting" for the resolution of sequelize.authenticate(). Change this:
sequelize.authenticate().then(() => {
console.log("db authenticated")
});
To this:
await sequelize.authenticate()
console.log("db authenticated")
What was happening, is that your initSequelize async function would resolve, before sequelize.authenticate promise would. This is a common pitfall in JS. I think this adjustment will solve your problem. Regarding "the best approach", i don't see much that can be done here, but of course i don't have the entire picture.
I found a 'pure' sequelize way to do this through lifecycle hooks:
Basically a generic setup in a db.js file would look like this:
const { Sequelize } = require('sequelize');
const asyncFetch = require('../util/async-fetch');
const sequelize = new Sequelize({
dialect: 'mysql',
database: 'db_name',
host: '127.0.0.1'
});
sequelize.beforeConnect(async (config) => {
const [username, password] = await Promise.all([
asyncFetch('username'),
asyncFetch('password')
]);
config.username = username;
config.password = password;
});
module.exports = sequelize;
The sequelize model definition is really just a plain object so that can be set early. Model initialisation does require a sequelize instance to be passed in.
The setup was a bit clearer to me when using ES6 class definitions for the models. sequelize.define is replaced with a call to Model.init, and this can all be done in an async setup function.
const Sequelize = require('sequelize')
const { Model } = Sequelize
class User extends Model {
static get modelFields(){
return {
id: {
type: Sequelize.UUID,
primaryKey: true,
defaultValue: Sequelize.UUIDV4,
},
name: {
type: Sequelize.STRING,
allowNull: false,
unique: true,
}
}
}
static get modelOptions(){
return {
version: true,
}
}
static init(sequelize){
const options = { ...this.modelOptions, sequelize }
return super.init(this.modelFields, options)
}
static associate(models) {
this.hasMany(models.Task)
}
}
module.exports = User
const User = require('./User')
class Database {
async static setup(){
const credentials = await getCredentials()
this.sequelize = new Sequelize(credentials)
User.init(this.sequelize)
this.User = User
// When you have multiple models to associate add:
this.User.associate(this)
}
}
module.exports = Database
Due to the async credentials requirement, the rest of your app will just need to cope with the delay until the DB is setup. If this is a koa/express app for example, you could delay the server .listen() until the Database.setup() promise has resolved.
As this would have changed a lot of my code. I have ended up by creating a script in golang that gets my credential "asynchronously" before running my server.
I have use some code from this package: https://github.com/telia-oss/aws-env
And then pass my starting script as a command argument so I could "inherit" the environmental variables
./getEnv exec -- node index.js
Related
I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.
You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.
I am writing a node/express rest api.
Hitting,
http://localhost:5000/api/news
and
http://localhost:5000/api/news/?id=c5f69d56be40e3b56e55d80
both give me all the news objects because it enters the same .getNews function on for both the urls.
My controller:
const NewsController = {};
const News = require('../models/news.model');
// This implementation of getNews is using Promises
NewsController.getNews = function(req, res) {
console.log('Inside getNews');
sendResponse = function(arg) {
res.json(arg);
}
const allnews = News.find({}, function(err, ns) {
sendResponse(ns);
});
};
// ES6 style
NewsController.getSingleNews = async (req, res) => {
console.log("Inside getSingleNews");
const news = await News.findById(req.params.id);
res.json[news];
};
NewsController.createNews = async (req, res) => {
const news = new News(req.body);
await news.save();
res.json[{
'status': 'item saved successfully'
}];
};
NewsController.deleteNews = async (req, res) => {
await News.findByIdAndRemove(req.params.id);
res.json[{
'status': 'item deleted successfully'
}]
};
module.exports = NewsController;
My routes.js (I am using the router at /api. So app.js has // use Router
app.use('/api', newsRoutes);
)
const express = require('express');
const router = express.Router();
var newsController = require('../controllers/NewsController')
router.get('/news', newsController.getNews);
router.get('/news/:id', newsController.getSingleNews);
router.post('/news', newsController.createNews);
router.delete('news/:id', newsController.deleteNews);
module.exports = router;
My Model
const mongoose = require('mongoose');
const { Schema } = mongoose;
const newsSchema = new Schema({
title: { type: String, required: true },
content: { type: String, required: true },
author: { type: String },
image: { type: String },
source: { type: String }
});
module.exports = mongoose.model('news', newsSchema);
The issue with your code is the way you are trying to call your endpoint. Express routes don't match query string parameters.
Having said that, your call to the news endpoint that looks like this:
http://localhost:5000/api/news/?id=c5f69d56be40e3b56e55d80
Should look like this instead:
http://localhost:5000/api/news/c5f69d56be40e3b56e55d80
That way the id parameter will get mapped to the req.params.id property inside your getSingleNews controller.
Being that the expected behavior for the way you declared your route:
router.get('/news/:id', newsController.getSingleNews);
For more information on how express routes work, check the documentation here: https://expressjs.com/en/guide/routing.html
Use /news/:id first. Your request will be redirected to the first matched url following the declaration order.
So /api/news satisfies app.get(/news)? Yep, gets redirected to that controller.
/api/news/?id=c5f69d56be40e3b56e55d80 satisfies app.get(/news)? Yep, also gets redirected to /news controller.
By the way, as your getting the id from req.params you should use /news/c5f69d56be40e3b56e55d80. If you were to get it from req.query you wouldn't need another route. /news?id=c5f69d56be40e3b56e55d80 would be perfect, you'd just need to check the existence of req.query.
ORM OpenRecord
I'm getting an error ( TypeError: User.create is not a function ) when I do the following:
await User.create(req.body.data)
When I log User I get this: [Function: User]
Database config:
//config/database/openRecord.js
/** more code above */
let store = new Store({
database,
user,
password,
host,
autoConnect: true,
autoAttributes: true,
models: require("../../app/models/models")
});
Models:
//app/models/models.js
module.exports = [
require("./user")
]
Model:
//app/models/user.js
const Store = require('openrecord/store/mysql');
class User extends Store.BaseModel {
static definition(){
this.validatePresenceOf(
'first_name', 'last_name', 'email', 'password'
);
this.validatesConfirmationOf('password');
this.validateFormatOf('email', 'email');
}
fullName(){
return `${this.first_name} ${this.last_name}`
}
}
module.exports = User;
Why is it throwing an error when I've clearly defined my object the correct way?
openrecord github link
openrecord website documentation
Two small typos are validatePresenceOf (should be validatesPresenceOf) and validateFormatOf (should be validatesFormatOf).
However, this is not the error you experience!
I'm pretty sure that you forget to wait until the store is ready.
If you use e.g. express I recommend to start listening after the store has loaded:
const express = require('express')
const store = require('./store')
const app = express()
async function start() {
// add middleware ...
await store.ready()
// start express server
app.listen()
}
start()
You have not exported the module. For example, we write:
module.exports = mongoose.model("user", userobj)
This is missing
I'm kinda new to module creation and was wondering about module.exports and waiting for async functions (like a mongo connect function for example) to complete and exporting the result. The variables get properly defined using async/await in the module, but when trying to log them by requiring the module, they show up as undefined. If someone could point me in the right direction, that'd be great. Here's the code I've got so far:
// module.js
const MongoClient = require('mongodb').MongoClient
const mongo_host = '127.0.0.1'
const mongo_db = 'test'
const mongo_port = '27017';
(async module => {
var client, db
var url = `mongodb://${mongo_host}:${mongo_port}/${mongo_db}`
try {
// Use connect method to connect to the Server
client = await MongoClient.connect(url, {
useNewUrlParser: true
})
db = client.db(mongo_db)
} catch (err) {
console.error(err)
} finally {
// Exporting mongo just to test things
console.log(client) // Just to test things I tried logging the client here and it works. It doesn't show 'undefined' like test.js does when trying to console.log it from there
module.exports = {
client,
db
}
}
})(module)
And here's the js that requires the module
// test.js
const {client} = require('./module')
console.log(client) // Logs 'undefined'
I'm fairly familiar with js and am still actively learning and looking into things like async/await and like features, but yeah... I can't really figure that one out
You have to export synchronously, so its impossible to export client and db directly. However you could export a Promise that resolves to client and db:
module.exports = (async function() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true
});
const db = client.db(mongo_db);
return { client, db };
})();
So then you can import it as:
const {client, db} = await require("yourmodule");
(that has to be in an async function itself)
PS: console.error(err) is not a proper error handler, if you cant handle the error just crash
the solution provided above by #Jonas Wilms is working but requires to call requires in an async function each time we want to reuse the connection. an alternative way is to use a callback function to return the mongoDB client object.
mongo.js:
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb+srv://<user>:<pwd>#<host and port>?retryWrites=true";
const mongoClient = async function(cb) {
const client = await MongoClient.connect(uri, {
useNewUrlParser: true
});
cb(client);
};
module.exports = {mongoClient}
then we can use mongoClient method in a diffrent file(express route or any other js file).
app.js:
var client;
const mongo = require('path to mongo.js');
mongo.mongoClient((connection) => {
client = connection;
});
//declare express app and listen....
//simple post reuest to store a student..
app.post('/', async (req, res, next) => {
const newStudent = {
name: req.body.name,
description: req.body.description,
studentId: req.body.studetId,
image: req.body.image
};
try
{
await client.db('university').collection('students').insertOne({newStudent});
}
catch(err)
{
console.log(err);
return res.status(500).json({ error: err});
}
return res.status(201).json({ message: 'Student added'});
};
I am attempting to seed a database using mongoose-seeder, and I keep getting a MissingSchemaError. I am sure that I am setting up the schema properly, so I am lost as to why this is happening.
The file where I set up the schema looks like this:
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const UserSchema = new Schema({
fullName: {
type: String,
required: true,
trim: true
},
emailAddress: {
type: String,
unique: true,
required: true,
match: /^(([^<>()\[\]\\.,;:\s#"]+(\.[^<>()\[\]\\.,;:\s#"]+)*)| .
(".+"))#((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-
Z\-0-9]+\.)+[a-zA-Z]{2,}))$/
},
password: {
type: String,
required: true
}
});
const User = mongoose.model('User', UserSchema);
module.exports.User = User;
and in my main file:
'use strict';
// load modules
const morgan = require('morgan');
const mongoose = require('mongoose');
const seeder = require('mongoose-seeder');
const data = require('./data/data.json');
const express = require('express');
const app = express();
//set up database connection
mongoose.connect("mongodb://localhost:27017/courseapi");
const db = mongoose.connection;
//handle err connecting to db
db.on("error", (err) => console.error("Error connecting to database: ",
err));
//success
db.once("open", () => {
console.log("Connected to database");
seeder.seed(data, {dropDatabase: false}).then(function(dbData){
console.log("Database seeded!");
}).catch(function(err){
console.error("Error seeding database", err);
})
});
any help would be much appreciated!
The mongoose-seeder package is not maintained. Hence suggesting an alternative to import data. You can populate MongoDB in the CLI (command line interface) using mongoimport.It will load a JSON file into a specified MongoDB Instance & Collection. All you need is a mongod instance to be running before execution.
Please go through the walkthrough.
thank you for your help! The project required using a module to seed the data, so I ended up using mongoose-seed instead. (Required some reformatting of the json, but thankfully the file was relatively small)
it's better to use the actively maintained Seedgoose. It's the ultimate mongoose seeder with smart reference support.