I'm working on a MERN webapp (using MySQL instead of MongoDB) and we are having an issue where at some point after querying to add an entry, the backend is somehow re-querying the last query. In the meantime, between the original entry and the duplicate one, the backend runs continuously to keep the frontend displaying up-to-date data.
I've read on old threads that MySQL for Node may have some issues with memory leaking and thus running sleep queries. I don't believe this is a frontend issue, and the backend isn't running the queries as if they were called from the function that handles the frontend button click (to submit).
I'm a noob to full-stack JS, so I don't really know where to look for an answer to this issue. I'm not sure if we should band-aid the solution and just make sure a query isn't sent without the submit button being clicked within a 1 second time span.
EDIT: Here's code for the query to add the entry (data for a music album).
db.query(insertAlbumQuery, insertAlbumValues, function(err, result) {
if (err) throw err;
console.log("1 Album added.");
});
We don't close the connection to the db after querying, one connection runs for the duration of the app's execution.
Here's the code for the db connection:
let mysql = require("mysql");
var db;
function connectDB() {
if (!db) {
db = connection = mysql.createConnection({
host: "******",
user: "******",
password: "******",
database: "******"
});
connection.connect(function(err) {
if (err) {
return console.error("error: " + err.message);
} else {
console.log("Connected to the MySQL server.");
}
});
}
return db;
}
module.exports = connectDB();
From the main, backend app.js file we have the router post
router.post("/addAlbum", (req, res) => {
const {
Album_title,
Artist,
Release_date,
Category,
Description,
Rotation
} = req.body;
album.add(Album_title, Artist, Release_date, Category, Description, Rotation, null);
});
From the front end we post with
addAlbum = currentState => {
this.setState({ Rotation: +this.state.Rotation });
axios.post("http://localhost:3001/api/addAlbum", this.state);
};
Not sure if this is enough background on the app, but I can add more snippets if needed.
Related
Github repo. I am trying to use MongoDB Atlas database with my node JS Login & Signup app for storing data. The problem is that the data is not saving to the database or in other words the request isn't going through even if my app is connected to Atlas. Full code available on www.github.com/tahseen09/login
// Connection to mongodb atlas
const uri = "mongodb+srv://tahseen09:<PASSWORD>#cluster0-pirty.mongodb.net/userdb"
MongoClient.connect(uri, function(err, client) {
if(err) {
console.log('Error occurred while connecting to MongoDB Atlas...\n',err);
}
console.log('Connected to Atlas');
const collection = client.db("userdb").collection("credentials");
client.close();
});
//New User Registration
app.post('/register', function(req,res){
var cred= new credential();
cred.uname=req.body.uname;
const hash = bcrypt.hashSync(req.body.password, 10);
cred.password=hash;
collection.save(function(err,newuser){
if(err){
res.status(500).send("Username exists");
}
else{
res.status(200).send("New User Created");
}
})
})
The code that is important is attached as a snippet and the rest of the code is available on www.github.com/tahseen09/login
Note: I am running this app on localhost.
Let me describe your flow so you can understand wrong point there :)
Connect to MongoDB
Create reference to the collection
Close connection
When someone tries to access /register route, you already have closed connection by that time. Thus, any operation attempt to the database will end up with connection error.
From the documentation it's recommended calling MongoClient.connect once and reusing the database variable returned by the callback, i.e. do not close connection manually, driver will just create and use pool of connections, so don't worry about closing connection. Check out example code in the documentation.
Lets step through the code to see what happens:
MongoClient.connect(uri, function(err, client) {
A connection to mongodb is created, then somewhen the connection is established or it fails, then the callback gets called back. Now you create a local variable holding the database reference:
const collection = client.db("userdb").collection("credentials");
And then you close the connection:
client.close();
Then the callback ends:
});
which means that a variables inside (connection) can't be accessed anymore and get therefore recycled.
Now somewhen (that might even happen before the db connection gets established), someone requests the webpage and you try to do:
collection.save(/*...*/);
That won't work for various reasons:
1) The db might not even be opened
2) If it was opened already, it was also closed already.
3) Even if it is open at the moment, you still cannot access connection as it is not in scope.
Now to resolve that we have to:
1) only start the webserver when the db connection is establishee
2) don't close the connection
3) expose the connection so that it can be used elsewhere
For that it makes sense to create a function that establishes the connection and calls back with the db:
function withCredentials(callback) {
const uri = "mongodb+srv://tahseen09:<PASSWORD>#cluster0-pirty.mongodb.net/userdb"
MongoClient.connect(uri, function(err, client) {
if(err) {
console.log('Error occurred while connecting to MongoDB Atlas...\n',err);
} else {
console.log('Connected to Atlas');
const collection = client.db("userdb").collection("credentials");
callback(collection);
}
});
}
So now you can use that:
withCredentials(function(credentials) {
app.post('/register', function(req,res){
const cred = { };
cred.uname = req.body.uname;
cred.password = bcrypt.hashSync(req.body.password, 10);
credentials.insertOne(cred, function(err,newuser){
if(err){
res.status(500).send("Username exists");
} else {
res.status(200).send("New User Created");
}
})
});
});
I started working on a MERN App today and am trying to write a restful api. First I am using mlab to store my mongodb database. I have succesfully connected to this database after creating a user. I can manually create a collection and inject some data into this collection. From my server.js file I can then get the data stored in here.
MongoClient.connect(db_url, (err, database) => {
if (err) return console.log(err);
var collection = database.collection('memories'); // Collection called memories
app.listen(3000, () => {
console.log("Listening on 3000");
});
});
Thats all fine and dandy but I want to take it to the next level. I want to write a CRUD api for the collection Memory. Coming from django, I would like to create my model first. Therefore, in my models/memory.js:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var MemorySchema = new Schema({
name: String,
description: String
});
module.exports = mongoose.model('Memory', MemorySchema);
Then I went ahead and started working on my routes/api/api.js:
let router = require('express').Router();
let Memory = require('../../../models/memories');
router.use(function (req, res, next) {
console.log("Something is happening");
next(); // Request stops at middleware without next()
});
router.route('/memory')
.post(function (req, res) {
let memory = new Memory();
memory.name = req.body.name;
memory.description = req.body.description;
memory.save(function (err) {
if (err) {
res.send(err);
}
res.json({message: 'Memory Created'});
});
})
.get(function (req, res) {
res.json({message: 'First memory'});
});
module.exports = router;
And in my server.js I call this module:
const apiRoutes = require('./routes/api/api');
app.use('/api/', apiRoutes);
However, after testing the post api with postman, it the POST request just takes forever before showing up as Could not get any response. However, the GET request works. What am I missing?
EDIT: So the post function is having trouble saving the model instance...
Try adding results as the first parameter in the callback of the save function, then res.json(results, { message: "Memory Created" }) to see if you are returned anything.
The main difference between the post and the get method is that the post method uses Mongoose, while the get doesn't. If you fail to connect to the database then the response can time out due to memory.save(...) not working as it should. And there are no responses sent outside the callback to save, so if your program never enter it, you will never send a response. The request will time out eventually.
In your model file you register a model on the following line:
module.exports = mongoose.model('Memory', MemorySchema);
Mongoose will then look for data in the memorys collection. If you change it to
module.exports = mongoose.model('Memory', MemorySchema, 'memories');
it will use the memories collection instead. This will make it consistent with the connection-to-db snippet you posted. I don't know if that will fix your issue though. I would suggest changing the connection code to
mongoose.connect(dburl, {
useMongoClient: true
});
instead of the native mongo client. You can add these lines too
mongoose.connection.on('connected', function () {
console.log('Mongoose connected');
});
mongoose.connection.on('error', function (err) {
console.log('Mongoose connection error: ' + err);
});
mongoose.connection.on('disconnected', function () {
console.log('Mongoose disconnected');
});
right after the connection code to help with debugging. Make sure you get connected when starting the app.
If you see an error similar to this Error: Can't set headers after they are sent. in the node terminal window, it might be because you are sending two responses in the post function. If an error occurs while saving it will enter the if(err) block, send a response async then go to the res.json(...) response and send that too.
So you have to return after sending the response to exit the function. Either like this
res.send(err);
return;
or like this
return res.send(err);
Same for the json response.
If that doesn't fix the problem you should either fire up the debugger (node --inspect or nodemon --inspect), or insert a console.log('inside post'); inside the post function to see that you're actually entering it.
I'm creating a web server that stores a user's data in a MongoDB database. The code behind the web requests uses asynchronous functions to insert a document into the database, but because these functions are asynchronous it means that for every request a new connection is made with the server.
exports.create_user = function(username, password, callback) {
mongo.connect(url, function(err, db) {
db.collection('users').insertOne({username: username, password: password}, function(err, result) {
callback(result)
db.close()
})
})
}
I'm under the impression that doing it this way is not the best practise, but I can't think a way to do it using the module model that I'm using above. Any suggestions or advice would be appreciated.
I stumbled upon this on my own research whether to use a new connection for mongodb on each query is the best practice or to use connection pooling. Turns out, that mongodb suggests connection pooling for most use-cases.
Citing from the docs:
A Connection Pool is a cache of database connections maintained by the driver so that connections can be re-used when new connections to the database are required. To reduce the number of connection pools created by your application, we recommend calling MongoClient.connect once and reusing the database variable returned by the callback
I am usually using the following form to establish and reuse a connection while firing queries:
// db.js
import { MongoClient } from 'mongodb';
// this will hold our cached database connection, which will itself hold multiple connections in a pool to be used
let connection,
database;
export {
connect: (next) => {
// already established? => return connection
if (database) return next(undefined, database);
// establish connection
MongoClient.connect('http://localhost:27017/admin', (err, db) => {
if (err) return next(err);
// save connection
connection = db;
// connect to database
database = db.db('myDatabase');
// call callback
next(undefined, database);
});
},
disconnect: (next) => {
if (!connection) return next();
// close connection
connection.close();
next();
}
};
Firing queries:
import db from './db';
db.connect((err, db) => {
if (err) return next(err);
db.collection('myUsers').insertOne({name: 'test'}, (err) => {
if (err) throw err;
db.disconnect((err) => {
if (err) throw err;
console.log('Everything finished, database connection closed');
});
});
});
Note: It is possible to determine the maximum amount of pooled connections manually (afaik the default is 5?). Refer to the docs about how to set the amount of opened connections via the mongodb url.
By doing db.close() you can close the connection, If you don't close your connection, event loop will keep the connection open and your process will not exit. If you are building a web server where your process will not be terminated, it's not necessary for you to close the connection.
For a reference node-mongodb-native
Getting started with Node.js and Heroku; I am trying to make sense of the following code, in order to build something of my own:
app.get('/db', function (request, response) {
pg.connect(process.env.DATABASE_URL, function(err, client, done) {
client.query('SELECT * FROM test_table', function(err, result) {
done();
if (err)
{ console.error(err); response.send("Error " + err); }
else
{ response.render('pages/db', {results: result.rows} ); }
});
});
});
Where can I find a tutorial or some comments or explanations for that?
Even though I can do some guessing, a good deal of this code is pretty mysterious.
Currently my main concerns are:
What happens if I change the SQL query, replacing it by 'SELECT
count(*) FROM test_table'? How do I then render the result?
What does "done();" do? Is it something I can modify or make use
of?
The parameter "request" is never used. Can it be used for
something at some point?
Before handling heroku, you should first look at tutorials about web application in node.js which will answers your last question.
You can see how works express.js, a web framework.
Then look at node-postgre documentation. You will find your answers about the second question here :
//this initializes a connection pool
//it will keep idle connections open for a 30 seconds
//and set a limit of maximum 10 idle clients
var pool = new pg.Pool(config);
// to run a query we can acquire a client from the pool,
// run a query on the client, and then return the client to the pool
pool.connect(function(err, client, done) {
if(err) {
return console.error('error fetching client from pool', err);
}
client.query('SELECT $1::int AS number', ['1'], function(err, result) {
//call `done()` to release the client back to the pool
done();
if(err) {
return console.error('error running query', err);
}
console.log(result.rows[0].number);
//output: 1
});
});
And finanlly, why don't you just log result output after changing the SQL query and look what you get ?
I have declared mysql, and my connection in app.js like so:
var mysql = require('mysql');
var connection = mysql.createConnection({
host: '192.168.1.75',
user: 'dev',
password: 'devaccount',
database: 'PugIt'
});
app.set('connection', connection);
And in my User.js for registration I have:
router.route('/register/steam/finish')
.get(function(req, res) {
res.render('user/register_steam');
})
.post(function(req, res) {
var connection = req.app.get('connection');
connection.connect();
// Look For Users
connection.query("SELECT * FROM Users", function(err, rows, fields) {
console.log('We Found Something!');
});
connection.end();
});
When the page first loads and I hit register, it works fine, but if I hit the button a second time I get a 500 error on my server.
But if I manually declare var connection inside each route file, this does not happen.
How come I cannot use req.app.get with MySQL, I used this method when I used to use MongoDB which worked great that way I had one main config in app.js I could alter to change in all route files if I needed. Not sure why I'm getting a 500 error on second POST
I think the connection.connect() and connection.end() on every POST request is causing problems. Drop those two lines and you should be good to go. This way the connection is only established once and all requests can re-use the same connection without constantly trying to tear it down and bring it back up again.
You can also create a pool of mysql connections if you find yourself needing greater concurrency with your database queries.