I am new to Node and MongoDB, so please forgive me if I sound too naive.
I have a DB class that creates a connection to MongoDB
db.service.js
const mongoose = require("mongoose");
const fs = require("fs");
const dbConfigs = JSON.parse(fs.readFileSync("/configs.json"));
const dbHost = dbConfigs.DB_HOST;
const dbName = dbConfigs.DB_NAME;
const dbPort = dbConfigs.DB_PORT;
const dbUrl = "mongodb://" + dbHost + ":" + dbPort + "/" + dbName;
const dbOptions = {
useNewUrlParser: true
};
let dbConnection = mongoose.createConnection(dbUrl, dbOptions);
exports.getConnection = () => {
if (dbConnection)
return dbConnection;
}
exports.closeConnection = () => {
if (dbConnection)
return dbConnection.close();
}
Next I have another module that creates a Schema for MongoDB
schema.js
const connection = require("./db.service").getConnection();
const Schema = require("mongoose").Schema;
const SampleSchema = new Schema({...})
exports.Sample = connection.model("Sample", SampleSchema);
Then I have another module that makes use of this Schema to save objects
logger.js
const Sample = require("./schema").Sample;
exports.log = () => {
let body = {....};
let sampleObj = new Sample(body);
return sampleObj.save(sampleObj);
The Main module
Main.js
const logger = require("./logger");
for (let i=0; i < 100; i++) {
logger.log();
}
When I run node Main.js, everything is saved, but when I check the MongoDB with the command db.serverStatus.connections, I see 6 connections open.
I don't know how it's getting there. I know the command mongo itself keep a connection open, but where are the other 5 connections coming from?
I have checked this, which seems to suggest that Mongoose opens 5 connections for an application, but my application is firing just one transaction, where is the need to open 5 connection then? Couldn't it be done with just one?
Mongoose is managing the connections itself; it tries to optimize the speed of the request.
You are seeing it like you are using only one transaction, but behind the scene mongoose is probably doing much more than you expect.
You can modify the number of connection mongoose can open using the parameter poolSize.
Example :
const dbOptions = {
useNewUrlParser: true,
poolSize: 1,
};
Try it with poolSize equals to 1 and see how much time it takes to execute your transaction, you'll have your answer.
Related
The goal is to call a function from my main script that connects to a database, reads a document from it, stores pieces of that document in a new object, and returns that object to my main script. The problem is I cannot get it all to work together. If I try one thing, I get the results but my program locks up. If I try something else I get undefined results.
Long story short, how do I open a database and retrieve something from it to another script.
The program is a quiz site and I want to return the quiz name and the questions.
const myDb = require('./app.js');
var myData = myDb.fun((myData) => {
console.log(myData.quizName);
});
Here is the script that tries to open the database and find the data
const { MongoClient } = require("mongodb");
const {mongoClient} = require("mongodb");
const uri = connection uri goes here but my name is hard coded into it at the moment so I removed for privacy
const client = new MongoClient(uri);
const fun = async (cback) => {
try {
await client.connect();
const database = client.db('Quiz-Capstone');
const quizzes = database.collection('Quiz');
const query = {quizName: "CIS01"};
const options = {
sort: {},
projection: {}
};
const quiz = await quizzes.findOne(query, options);
var quizObject = {
quizName: quiz.quizName,
quizQuestions: quiz.quizQuestions
}
//console.log(testOb);
} finally {
await client.close();
cback(quizObject);
}
}
fun().catch(console.dir);
module.exports = {
fun: fun
}
UPDATE: Still stuck. I have read several different threads here about asynchronous calls and callbacks but I cannot get my function located in one file to return a value to the caller located in another file.
I'm using the azure file storage, and using express JS to write a backend to render the contents stored in the azure file storage.
I am writing the code based on https://learn.microsoft.com/en-us/javascript/api/#azure/storage-file-share/shareserviceclient?view=azure-node-latest
const { ShareServiceClient, StorageSharedKeyCredential } = require("#azure/storage-file-share");
const account = "<account>";
const accountKey = "<accountkey>";
const credential = new StorageSharedKeyCredential(account, accountKey);
const serviceClient = new ShareServiceClient(
`https://${account}.file.core.windows.net`,
credential
);
const shareName = "<share name>";
const fileName = "<file name>";
// [Node.js only] A helper method used to read a Node.js readable stream into a Buffer
async function streamToBuffer(readableStream) {
return new Promise((resolve, reject) => {
const chunks = [];
readableStream.on("data", (data) => {
chunks.push(data instanceof Buffer ? data : Buffer.from(data));
});
readableStream.on("end", () => {
resolve(Buffer.concat(chunks));
});
readableStream.on("error", reject);
});
}
And you can view the contents through
const downloadFileResponse = await fileClient.download();
const output = await streamToBuffer(downloadFileResponse.readableStreamBody)).toString()
Thing is, I only want to find if the file exists and not spend time downloading the entire file, how could I do this?
I looked at https://learn.microsoft.com/en-us/javascript/api/#azure/storage-file-share/shareserviceclient?view=azure-node-latest
to see if the file client class has what I want, but it doesn't seem to have methods useful for this.
If you are using #azure/storage-file-share (version 12.x) Node package, there's an exists method in ShareFileClient. You can use that to find if a file exists or not. Something like:
const fileExists = await fileClient.exists();//returns true or false.
I am trying to show Nodejs is not good for CPU Intensive tasks.
In this code, I created the async function mySlowFunction that accepts baseNumber.
When I hit http://localhost:8080 with query params 12 it takes 5 sec.
What If 5 subsequent request is being made to this api for query params 11,89,23,12,23? It would be very slow.
In this kind of scenario is it really good to use NodeJs?[i.e for CPU Intensive Task]. If yes, then how can I make this fast?
async function mySlowFunction(baseNumber) {
let result = 0;
for (var i = Math.pow(baseNumber, 7); i >= 0; i--) {
result += Math.atan(i) * Math.tan(i);
};
return result;
}
const express = require('express');
const cookieparser = require('cookie-parser');
const csrf = require('csurf');
const dotenv = require("dotenv").config();
const app = express();
app.use(express.urlencoded({extended:true}));
app.use(express.json());
app.use(cookieparser());
app.get("/",async(req,res)=>{
let data = await mySlowFunction(req.query.base);
res.json({data})
})
app.listen(process.env.PORT, () => {
console.log(`Listening on http://localhost:${process.env.PORT}`);
});
I have a very general answer.
You can precalculate the results if query params are limited.
You can store results in files or database query them.
You can cache results.
If there is no result for query param in db, calculate it and save it to db.
I tried setting up parse live query, but for some reason it returns no results.
I can pull data from it, so I know the server and db are running fine.
The 'open' connection works when I run on localhost, but even then, the subscription events are never called.
Client
var Parse = require('parse/node');
Parse.initialize("key", "", "pass");
Parse.serverURL = parseServer;
Parse.liveQueryServerURL = 'ws://localhost:1337/';
let query = new Parse.Query('groups');
query.equalTo('name', 'name');
let subscription = query.subscribe();
subscription.on('update', (people) => {
console.log("YEAY");
// console.log(people.get('score')); // This should output 100
});
subscription.on('open', () => {
console.log('subscription opened');
});
Server
var app = new ParseServer({
startLiveQueryServer: true,
liveQuery: {
classNames: ["groups", "comments"] // List of classes to support for query subscriptions
},
..}
var port = process.env.PORT || 1337;
var httpServer = require('http').createServer(app);
httpServer.listen(port, function() {
console.log('parse-server-example running on port ' + port + '.');
});
// This will enable the Live Query real-time server
var parseLiveQueryServer = ParseServer.createLiveQueryServer(httpServer);
You can try changing this
let subscription = query.subscribe();
to this
let subscription = await query.subscribe();
Hope it helps.
We have different clients and the idea is to keep their data separate from each other in the same application. We are using node.js with mongodb and mongoose is being used for querying.
This is 'index.js' file in models directory
var mongoose = require('mongoose');
var fs = require('fs');
var connectionUrl = 'mongodbserverlink/';
var companies = [{ db: 'comp1_db', comp_id: 'com1' }, { db: 'com2_db', comp_id: 'com2' }, { db: 'com3_db', compa_id: 'com3'}];
var connections = {};
var models = {};
fs.readdirSync(__dirname)
.forEach(function (file) {
var Schema = file.split('.js')[0];
if (Schema === 'index') return;
models[Schema] = require('./' + Schema);
});
companies.forEach(function (company) {
var conn = mongoose.createConnection(connectionUrl + company.db);
connections[company.company_id] = {};
Object.keys(models).forEach(function (Schema) {
connections[company.company_id][Schema] = conn.model(Schema, models[Schema]);
});
conn.on('error', console.error.bind(console, company.db + ' connection error in mongodb in the first step!'));
conn.once('open', function() {
console.log(company.db + " mongodb connected");
});
});
module.exports = connections;
Here the connection is being made with different databases. The models directory has this index file.
Now in the controller where application logic is being done, this is what we are doing.
var models = require('../models');
var comp_id = req.body.comp_id;
db.collectionname.find...(This is not the syntax for find, I just cut it short to keep it simple) // -> this is not working now
when we tried logging models object this is what we got
models object is: {"com1":{},"com2":{},"com3":{}}
and only db when logged gives {}
We are facing issues in grasping the complete work... it is because the person who wrote the major chunk is not with us and there is no documentation.
What are we doing wrong here?
It looks like you already have your models exported from the index file. So, in the controller you can do a var models = require('index');. From the connections object, you may be able to retrieve the corresponding model: var companyModel = models[comp_id];.
Hope it helps.