We Keep Coding

Model.find({}) query using mongoose with node.js and express combination fails to fetch large amount of datasets of more than 200,000+ documents

I am using a simple get request using mongoose coupled with express and node to fetch all the documents in a particular collection I have defined in a MongoDB instance. It works fine for small amounts of data but is failing for large datasets. I am able to run the same query on Mongo Shell and after a decent amount of time, it is able to return the data.
I have tried to modify the query to use lean() function along with the find({}) function of mongoose but the problem still persists.
/*
Fetch all the players
GET - /
*/
getPlayerRouter.route('/')
.get((req, res, next) => {
Player.find({}).lean()
.then((players) => {
res.status(200).json({
success: true,
totalPlayers: players.length,
players
});
})
.catch((err) => console.log(err));
});
I expect the query to fetch all the documents on the collection.

var query=Player.find({}).stream();
query.on('data', (players)=> {
res.status(200).json({
success: true,
totalPlayers: players.length,
players
});
}).on('error',(err)=>{
}).on('close',()=>{
console.log('connection closed');
});
You can use stream in mongoose to process large records.Comment on whether it works.

authentication conflict between sessions in Express server

I am playing around with this library and I am experiencing an annoying scenario which I believe comes from some sort of conflict in cookies or headers authentication.
When I login to one account everything works great. But then when trying to login to another account, it simply ignore the new data provided and move through the authentication with the old data and connecting to the old account. No matter if the email or the password even exist. (Tried also with fake data).
The library doesn't have proper logout method which make sense, you dont really need one because when you run it simply using node on your machine without any server involved and there is no cookies or any kind of data in memory, everything work great. I can login to as many account as I want.
The problem is when running it on an Express server.
CODE:
// api.js
const OKCupid = require("./okc_lib");
const Promise = require("bluebird");
const okc = Promise.promisifyAll(new OKCupid());
async function start(req, res, next) {
const {
body: {
username,
password
}
} = req;
try {
await okc.loginAsync(username, password);
okc.search(
searchOpt,
(err, resp, body) => {
if (err) return console.log({ err });
const results = body.data;
// do dsomething with results
return res.status(200).json({ message: "OK" });
});
}
);
} catch (error) {
res.status(400).json({ message: "Something went wrong", error });
}
}
module.exports = { start };
// routes.js
const express = require("express");
const router = express.Router();
const { start, login } = require("../actions/okc");
router.post("/login", login);
router.post("/start", start);
module.exports = router;
So when trying first to post to url/login it works fine. But when you try to do it again with different username and password it simply go through and ignore the new data and connect to the old one.
As part of my investigation I looked at the source code of the library and found a method clearOAuthToken which clear the token from the header. However it didn't really do anything. So I tried to remove the jar initialisation from the requester helper and it was the only thing that helped me to move on and login to another account. BUT it was only for experimenting and cant be a solution as you do need those cookies for other parts of the library. It was only a proof the problem isn't in the headers but in the cookies.
Any idea how can I "reset" state of server between each call?

"when trying to login to another account, it simply ignore the new data provided and move through the authentication with the old data and connecting to the old account."
As OP mentioned in the comment, this is not an authorization header issue, but a cookie issue.
To implement the logout interface, you can manually clear the cookies:
OKCupid.prototype.logout = function(){
request = request.defaults({jar: request.jar()}) // reset the cookie jar
headers.clearOAuthToken(); // just in case
}

Express JS/ MongoDB Post not working

I started working on a MERN App today and am trying to write a restful api. First I am using mlab to store my mongodb database. I have succesfully connected to this database after creating a user. I can manually create a collection and inject some data into this collection. From my server.js file I can then get the data stored in here.
MongoClient.connect(db_url, (err, database) => {
if (err) return console.log(err);
var collection = database.collection('memories'); // Collection called memories
app.listen(3000, () => {
console.log("Listening on 3000");
});
});
Thats all fine and dandy but I want to take it to the next level. I want to write a CRUD api for the collection Memory. Coming from django, I would like to create my model first. Therefore, in my models/memory.js:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var MemorySchema = new Schema({
name: String,
description: String
});
module.exports = mongoose.model('Memory', MemorySchema);
Then I went ahead and started working on my routes/api/api.js:
let router = require('express').Router();
let Memory = require('../../../models/memories');
router.use(function (req, res, next) {
console.log("Something is happening");
next(); // Request stops at middleware without next()
});
router.route('/memory')
.post(function (req, res) {
let memory = new Memory();
memory.name = req.body.name;
memory.description = req.body.description;
memory.save(function (err) {
if (err) {
res.send(err);
}
res.json({message: 'Memory Created'});
});
})
.get(function (req, res) {
res.json({message: 'First memory'});
});
module.exports = router;
And in my server.js I call this module:
const apiRoutes = require('./routes/api/api');
app.use('/api/', apiRoutes);
However, after testing the post api with postman, it the POST request just takes forever before showing up as Could not get any response. However, the GET request works. What am I missing?
EDIT: So the post function is having trouble saving the model instance...

Try adding results as the first parameter in the callback of the save function, then res.json(results, { message: "Memory Created" }) to see if you are returned anything.

The main difference between the post and the get method is that the post method uses Mongoose, while the get doesn't. If you fail to connect to the database then the response can time out due to memory.save(...) not working as it should. And there are no responses sent outside the callback to save, so if your program never enter it, you will never send a response. The request will time out eventually.
In your model file you register a model on the following line:
module.exports = mongoose.model('Memory', MemorySchema);
Mongoose will then look for data in the memorys collection. If you change it to
module.exports = mongoose.model('Memory', MemorySchema, 'memories');
it will use the memories collection instead. This will make it consistent with the connection-to-db snippet you posted. I don't know if that will fix your issue though. I would suggest changing the connection code to
mongoose.connect(dburl, {
useMongoClient: true
});
instead of the native mongo client. You can add these lines too
mongoose.connection.on('connected', function () {
console.log('Mongoose connected');
});
mongoose.connection.on('error', function (err) {
console.log('Mongoose connection error: ' + err);
});
mongoose.connection.on('disconnected', function () {
console.log('Mongoose disconnected');
});
right after the connection code to help with debugging. Make sure you get connected when starting the app.
If you see an error similar to this Error: Can't set headers after they are sent. in the node terminal window, it might be because you are sending two responses in the post function. If an error occurs while saving it will enter the if(err) block, send a response async then go to the res.json(...) response and send that too.
So you have to return after sending the response to exit the function. Either like this
res.send(err);
return;
or like this
return res.send(err);
Same for the json response.
If that doesn't fix the problem you should either fire up the debugger (node --inspect or nodemon --inspect), or insert a console.log('inside post'); inside the post function to see that you're actually entering it.

Angular-Fullstack: Using $resourse query with params

I'm using Angular-Fullstack generator, and I'm not able to get a list of drivers depending on a companyID, through $resource query. This is what I have:
server/api/driver/index.js:
router.get('/:company', controller.index);
server/api/driver/driver.controller.js:
export function index(req, res) {
return Driver.find({company: req.params.company}).exec()
.then(function(res){
console.log(res); /* I get here the result correctly */
respondWithResult(res)
})
.catch(handleError(res));
}
client/services/driver.service.js:
export function DriverResource($resource) {
'ngInject';
return $resource('/api/drivers/:id/:company', {company: '#_id'});
}
client/app/driver/driver.controller.js:
this.driverList = Driver.query({company: Auth.getCurrentUser()._id}});
console.log(this.driverList); /* Empty array */
I'd be grateful if someone could help me getting the response from the server...
Thank you in advance.

I just realised that I was duplicating the 'res' variable:
server/api/driver/driver.controller.js:
export function index(req, res) {
return Driver.find({company: req.params.company}).exec()
.then(function(**res**){
/* Should be the result, not the response */
console.log(**res**);
respondWithResult(**res**)
})
.catch(handleError(res));
}

You were close.
Driver.query({company: 'foo'}).$promise.then(function(results) {
console.log(results) //here
}, function(err) {
//here is your 404 error, but you should create an http interceptor
});
It's async, do you don't get your results right away.
This will work of course, assuming your backend responds properly.
EDIT: Your backend is missing some endpoints. You should be able to respond to requests to /api/drivers/ with a list of drivers
EDIT 2:
Angular's resource will give you access to some methods:
Driver.get(1) Will make a request to /api/drivers/:id and will be expecting the backend to respond with an object representing the driver with said ID. This should be used when you want to fetch only 1 record
Driver.query({foo: 'bar', some_id: 1}) Will make a request to /api/drivers?foo=bar&some_id=1 and will be expecting the backend to respond with an array of objects, each representing a driver. This should be used when you want to fetch several records, for example in an index.
Driver.query() will make a request to /api/drivers and will be expecting the backend to respond with an array
Driver.create(data) will make a POST request to /api/drivers and will expect an object (the created driver) in the response. Used to create a new record
There are some others, this is the ones I use.
So, your backend, considering you are using this three methods, needs to handle:
router.get('/drivers/:id', function(req, res) {
let id = req.params.id
})
router.get('/drivers', function(req, res) {
//if request was /drivers?foo=bar
let foo = req.query.foo
})
router.post('/drivers', function(req, res) {
let body = req.body
})
As I said, there are several things in play here. If you are at a lost, break the problem into pieces. Get the backend working before going to Angular.

Why is req.params returning undefined?

I've checked two similar questions here and neither of the things suggested in the comments are working for me.
app.get('/:id', function(req,res) {
console.log(req.params.id);
});
app.get('/:id', function(req, res) {
db.query("SELECT * FROM entries WHERE id = $1", [req.params.id], function(err, dbRes) {
if (!err) {
res.render('show', { entry: dbRes.rows[0] });
}
});
});
As you can see, I've tried logging the result to the console to see what's going on. Visiting the URL in question just makes the page load until it times out. In the console, I get "undefined".
How do I define req.params? Or where is it's definition being pulled and why isn't it returning the values?
Full context: http://pastebin.com/DhWrPvjP

Just tested your code and it works fine. I think you might be missing your url parameter. It should be http://localhost:3000/1 - or whatever ID you're trying to retrieve. Try it out.
Also, you should pass the extended option to your bodyParser.urlencode method: express throws error as `body-parser deprecated undefined extended`
Edit: To specifically answer your question about defining request parameters. You don't have to do anything to define request parameters other than make sure that you're passing in the correct URL. Express takes care of parsing the URL and defining the request parameters for you. So, if you go to the URL http://localhost/jimbob on your server then the value passed in for the id parameter will be available as req.params.id. See this link on request parameters for more info.
Edit 2: You could try debugging your app to see what you get. Here's a link on how to enable debugging in Express and how to use node-inspector for debugging. I saw that your running this on Ubuntu. So, there may be something weird there that I'm not aware of. (I'm running it on a Mac.)
I would also check the version of Node that you're running on the computer(s) that the app works on and check the version of Node on your Ubuntu environment (or whatever computers the app doesn't work on).

app.get('/:id', function(req, res) {
db.query("SELECT * FROM entries WHERE id = $1", [req.params.id], function(err, dbRes) {
if (!err) {
res.render('show', { entry: dbRes.rows[0] });
}
});
});
in your code the url would be localhost/some-id req.params.id would equal some-id, params are pulls straight from the url string, if you are trying to send info with post or get methods you want to use req.body and req.query respectively. I dont see any reason you wouldnt be able to get the id unless the url is wrong
or if you need to do it manually
app.get('/:id', function(req, res) {
//if no req.params and assuming the id is the last item in the url
var urlArray = req.url.split('/'),
id = urlArray[urlArray.length-1];
db.query("SELECT * FROM entries WHERE id = $1", [req.params.id], function(err, dbRes) {
if (!err) {
res.render('show', { entry: dbRes.rows[0] });
}
});
});

try this req.param('id') :D. It may be working for you

I know I'm late to the party but this post helped me debug my issue so I figured I'll add my suggestion in hopes it will help someone else.
If you are using mysql2 with promises
const mysql = require("mysql2");
const pool = mysql.createPool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_DATABASE
});
module.exports = pool.promise();
Then you need to use promises in your request.
router.get("/:id", (req, res) => {
mysql
.execute("SELECT * FROM entries WHERE id = $1", [req.params.id])
.then(result => {
res.send(result[0]);
})
.catch(err => {
console.log(err);
});
});
I spent hours debugging my code only to realize I was using promise() in my connection. Hope this helps as this post helped me debug.