I use the MEAN stack and I am writing some REST API functions to manipulate data with routing functions...
I just have the problem that I am quite new to REST Api and the MEAN stack and also Javascript...
so I was wondering, that there are many different styles of writing those javascript functions, e.g. to get some data (and even how to do data manipulation, e.g by mongoose /mongoDB driver api function like aggregate / find, or doing the selection or filtering of data manually with javascript).
Just to show some examples:
exports.getTestbyIDRange = function(req, res) {
Test.aggregate([{"$match": {"$and": [{ "_id": {"$gte": +req.params.start, "$lte": +req.params.end}}]}}])
//Test.find({"_id": {"$gte": Number(req.params.start), "$lte": Number(req.params.end)}})
.exec(function(err, demo) {
console.log("Anzahl gefundener Dokumente: ", demo.length)
res.json(demo)
for (let elem of demo) {
console.log("ID: ", elem["_id"])
} } ) }
here, after the query I am using .exec(function..., but I also have seen ..,function(err, Test) {...} or a then-clause:
}).then( doc => {
// do something with matched document
}).catch(e => { console.err(e); res.send(e); })
So I am quite confused, is there a "best practice" style approach / design pattern for writing those functions? Or are all those styles equivalent? Which is best in terms of error handling? Or in case of errors / error handling, which is best to continue your app in case of some minor errors? I don't want my backend to get stuck or hang up.... Is there maybe some good literature to find online about styles?
Second question: When writing queries or data aggregation operations, should I do it with mongodb operations like find and aggregate, or can I also do it with plain javascript code ? (I find that sometimes easier because I am quite new to MongoDB)
The thing here is that my backend functions are all running on the same server like where my MongoDB is sitting, so its basically localhost, that means it shouldnt be too bad performance when retrieving complete documents from the server and then doing javascript code on it, otherwise if it would be by network, then it's understandable that I should do database operations with db drivers first...
Some tips from my side
1.Try to break big codes into smaller functions like some function to get data from db and another to process data
Make a common function for querying the database
There should be only one res.send or res.json etc in function and no code should be below that as the code below that will also exexute
Use promises for avoiding callback hell plus it will also help you to write a good maintainable code and will also help with better error handling
For generating a structure you can use swagger create project it will provide you a nice project structure
Use one middle ware with first param as err to cath unhadeled errors while executing api for example
app.use((err, req, res, next) => {
console.log('-----Something broke!---', err)
res.status(500).send('Something broke!')
})
Use process.on in index file to handle errors
For example
process.on('unhandledRejection', (reason, p) =>{console.log('Unhandled Rejection at:', p, 'reason:', reason)})
process.on('uncaughtException', err => {console.log('-----uncaughtException----- ', err)})
Related
Can someone please explain what’s the use of this pattern with example?
All I'm confused is that I can have database instance wherever I want and I have flexibility to do anything by it, am I wrong? specially is
The repository pattern is a strategy for abstracting data access.
it’s like putting a universal adapter in between your application and your data so it doesn’t matter what data storage technology you use. All your app wants is having defined operations on items, it shouldn’t have to care about how it’s stored or where it comes from.
Also, there's no need to mention that all impacts of changes will be handled from one place instead of cascading all through your code!
Personally, I love this design pattern because it allows me to only have concerns about my business logics at the first steps instead of dealing with variant databases, on top of that, it solves a huge amount of my headaches when it comes to writing tests! So instead of stubbing or spying databases, which can be a headache, I can simply enjoy a mock version of operations
Now let’s implement a sample in js, it can be just as simple as below code (it's a simplified sample of course)
// userRepository.js
const userDb = [];
module.exports = {
insert: async (user) => userDb.push(user),
findAll: async () => userDb,
};
here is how I use this pattern, first I write something like below code in a 5 minute
// userRepository.js
const userDb = new Map();
module.exports = Object.freeze({
findById: async (id) => userDb.get(id),
insert: async (user) => userDb.set(user.id, user),
findAll: async () => Array.from(userDb.values()),
removeById: async (id) => userDb.delete(id),
update: async (updatedUser) => {
if (!userDb.has(updatedUser.id)) throw new Error('user not found');
userDb.set(updatedUser.id, { ...userDb.get(updatedUser.id), ...updatedUser });
},
});
Then I start to write my unit tests for repository that I’ve just written and business use-cases and so on…
anytime I’m satisfied with everything I can simply use a real database, because it’s just an IO mechanism, isn’t it? :)
So in above code I’ll replace userDb with a real database and write real data access methods, and of course expect all my tests to be passed.
I can't seem to get the article duplicates out of my web scraper results, this is my code:
app.get("/scrape", function (req, res) {
request("https://www.nytimes.com/", function (error, response, html) {
// Load the HTML into cheerio and save it to a variable
// '$' becomes a shorthand for cheerio's selector commands, much like jQuery's '$'
var $ = cheerio.load(html);
var uniqueResults = [];
// With cheerio, find each p-tag with the "title" class
// (i: iterator. element: the current element)
$("div.collection").each(function (i, element) {
// An empty array to save the data that we'll scrape
var results = [];
// store scraped data in appropriate variables
results.link = $(element).find("a").attr("href");
results.title = $(element).find("a").text();
results.summary = $(element).find("p.summary").text().trim();
// Log the results once you've looped through each of the elements found with cheerio
db.Article.create(results)
.then(function (dbArticle) {
res.json(dbArticle);
}).catch(function (err) {
return res.json(err);
});
});
res.send("You scraped the data successfully.");
});
});
// Route for getting all Articles from the db
app.get("/articles", function (req, res) {
// Grab every document in the Articles collection
db.Article.find()
.then(function (dbArticle) {
res.json(dbArticle);
})
.catch(function (err) {
res.json(err);
});
});
Right now I am getting five copies of each article sent to the user. I have tried db.Article.distinct and various versions of this to filter the results down to only unique articles. Any tips?
In Short:
Switching the var results = [] from an Array to an Object var results = {} did the trick for me. Still haven't figured out the exact reason for the duplicate insertion of documents in database, will update as soon I find out.
Long Story:
You have multiple mistakes and points of improvement there in your code. I will try pointing them out:
Let's follow them first to make your code error free.
Mistakes
1. Although mongoose's model.create, new mongoose() does seem to work fine with Arrays but I haven't seen such a use before and it does not even look appropriate.
If you intend to create documents one after another then represent your documents using an object instead of an Array. Using an array is more mainstream when you intend to create multiple documents at once.
So switch -
var results = [];
to
var results = {};
2. Sending response headers after they are already sent will create for you an error. I don't know if you have already noticed it or not but its pretty much clear upfront as once the error is popped up the remaining documents won't get stored because of PromiseRejection Error if you haven't setup a try/catch block.
The block inside $("div.collection").each(function (i, element) runs asynchronously so your process control won't wait for each document to get processed, instead it would immediately execute res.send("You scraped the data successfully.");.
This will effectively terminate the Http connection between the client and the server and any further issue of response termination calls like res.json(dbArticle) or res.json(err) will throw an error.
So, just comment the res.json statements inside the .create's then and catch methods. This will although terminate the response even before the whole articles are saved in the DB but you need not to worry as your code would still work behind the scene saving articles in database for you (asynchronously).
If you want your response to be terminated only after you have successfully saved the data then change your middleware implementation to -
request('https://www.nytimes.com', (err, response, html) => {
var $ = cheerio.load(html);
var results = [];
$("div.collection").each(function (i, element) {
var ob = {};
ob.link = $(element).find("a").attr("href");
ob.title = $(element).find("a").text();
ob.summary = $(element).find("p.summary").text().trim();
results.push(ob);
});
db.Article.create(results)
.then(function (dbArticles) {
res.json(dbArticles);
}).catch(function (err) {
return res.json(err);
});
});
After making above changes and even after the first one, my version of your code ran fine. So if you want you can continue on with your current version, or you may try reading some points of improvement.
Points of Improvements
1. Era of callbacks is long gone:
Convert your implementation to utilise Promises as they are more maintainable and easier to reason about. Here are the things you can do -
Change request library from request to axios or any one which supports Promises by default.
2. Make effective use of mongoose methods for insertion. You can perform bulk inserts of multiple statements in just one query. You may find docs on creating documents in mongodb quite helpful.
3. Start using some frontend task automation library such as puppeteer or nightmare.js for data scraping related task. Trust me, they make life a hell lot easier than using cheerio or any other library for the same. Their docs are really good and well maintained so you won't have have hard time picking these up.
PREFACE
So it seems I've coded myself into a corner again. I've been teaching myself how to code for about 6 months now and have a really bad habit of starting over and changing projects so please, if my code reveals other bad practices let me know, but help me figure out how to effectively use promises/callbacks in this project first before my brain explodes. I currently want to delete it all and start over but I'm trying to break that habit. I did not anticipate the brain melting difficulty spike of asynchronicity (is that the word? spellcheck hates it)
The Project
WorldBuilder - simply a CLI that will talk to a MySQL database on my machine and mimic basic text game features to allow building out an environment quickly in my spare time. It should allow me to MOVE [DIRECTION] LOOK at and CREATE [ rooms / objects ].
Currently it works like this:
I use Inquirer to handle the prompt...
request_command: function(){
var command = {type:"input",name:"command",message:"CMD:>"}
inquirer.prompt(command).then(answers=>{
parser.parse(answers.command);
}).catch((error)=>{
console.log(error);
});
}`
The prompt passed whatever the user types to the parser
parse: function(text) {
player_verbs = [
'look',
'walk',
'build',
'test'
]
words = text.split(' ');
found_verb = this.find(player_verbs, words)[0];
if (found_verb == undefined) {
console.log('no verb found');
} else {
this.verbs.execute(found_verb, words)
}
}
The parser breaks the string into words and checks those words against possible verbs. Once it finds the verb in the command it accesses the verbs object...
(ignore scope mouthwash, that is for another post)
verbs: {
look: function(cmds) {
// ToDo: Check for objects in the room that match a word in cmds
player.look();
},
walk: function(cmds) {
possible_directions = ['north','south','east','west'];
direction = module.exports.find(possible_directions, cmds);
player.walk(direction[0]);
},
build: function(cmds) {
// NOTE scope_mouthwash exists because for some
// reason I cannot access the global constant
// from within this particular nested function
// scope_mouthwash == menus
const scope_mouthwash = require('./menus.js');
scope_mouthwash.room_creation_menu();
},
execute: function(verb, args) {
try{
this[verb](args);
}catch(e){
throw new Error(e);
}
}
}
Those verbs then access other objects and do various things but essentially it breaks down to querying the database an unknown number of times to either retrieve info about an object and make a calculation or store info about an object. Currently my database query function returns a promise.
query: function (sql) {
return new Promise(function(resolve, reject){
db.query(sql, (err, rows)=>{
if (err) {
reject(err);
}
else {
resolve(rows);
}
});
});
},
The Problem
Unfortunately I jumped the gun and started using promises before I fully understood when to use them and I believe I should have used callbacks in some of these situations but I just don't quite get it yet. I solved 'callback hell' before I had to experience 'callback hell' and now am trying to avoid 'promise hell'. My prompt used to call itself in a loop after it triggered the required verbs but this whole approach broke down when I realized I'd get prompt messages in the middle of other prompt cycles for room building and such.
Should my queries be returning promises or should I rewrite them to use callback functions handled by whichever verb calls the query? How should I determine when to prompt the user again in the situation of having an unknown number of asynchronous processes?
So, put in a different way, my question is..
Given the parameters of my program how should I be visualizing and managing the asynchronous flow of commands, each of which may chain to an unknown number of database queries?
Possible Solution directions that have occurred to me..
Create an object that keeps track of when there are pending promises and
simply prompts the user when all promises are resolved or failed.
Rewrite my program to use callback functions where possible and force a known number of promises. If I can get all verbs to work off callback functions I might be able to say something like Prompt.then(resolve verb).then(Prompt)...
Thank you for bearing with me, I know that was a long post. I know enough to get myself in trouble and Im pretty lost in the woods right now.
This question already has answers here:
How to avoid long nesting of asynchronous functions in Node.js
(23 answers)
Closed 9 years ago.
I just started experimenting with node (using Express to build a simple website with a MySql database).
I have basically ran with the application structure Express provides (which doesn't matter for the sake of this question). I have a file routes/index.js which exports the index function that is hit whenever a request is made for my home page. The contents of index.js are:
var db = require('../db');
exports.index = function(req, res){
db.getConnection(function(err, connection) {
connection.query('SELECT * FROM test_table', function (err, rows) {
var templateVariables = {
title: 'Index Page',
response: rows[0].text
};
res.render('index', templateVariables);
});
connection.end();
});
};
This is obviously a very preliminary and lightweight example, however on this particular GET request for the Index page, there is already a 3-deep set of callback functions. Each callbuck must live within the callback of the "parent", for it depends on the result (in a sequentially executed language/environment this would be obvious and trivial).
My question is that when building more complex and potentially very large applications, how does one avoid the issue of having massive nestings of callback functions? This will of course be the case when you have sequential dependency on logic. I know the philosophy of Node is to be asynchronous, but when it comes to waiting for data from the database(s) and say we're running 5 separate queries, what then? Would we just write a single multi-statement query as an atomic unit? Though this question isn't exclusive to databases.
There's a nice general discussion on the issue here:
http://callbackhell.com/
Also, many people use modules like Async to manage the flow control issues.
Since you mention by using Express you can use the next() as an alternative to callbacks.
app.get('/',function first(req,res,next){
res.write('Hello ');
res.locals.user = req.user;
next();
//Execute next handler in queue
});
app.get('/',function second(req,res,next){
res.write('World!');
//Use res.locals.user
res.end();
});
//response shows Hello World!
The route handlers use extra parameter next and are executed in the order they are given, until one of them returns a response. next takes either no parameters at all or an error as a parameter. You can set the variable you want to pass into next function in the res.locals
Use a Promise or Future library, such as Q (available on npm).
Quoting from Q's readme, promises let you turn this:
step1(function (value1) {
step2(value1, function(value2) {
step3(value2, function(value3) {
step4(value3, function(value4) {
// Do something with value4
});
});
});
});
into this:
Q.fcall(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
// Do something with value4
}, function (error) {
// Handle any error from step1 through step4
})
.done();
Every other solution I've seen to callback hell introduces trade-offs that simply seem a step backward. Asynchronous operations do not form the natural logical boundaries between larger operations, so if you factor functions or otherwise modularize along these boundaries you will get microfactored code.
One way that I am fond of doing is like this...
exports.index = function(req, res) {
var connection
db.getConnection(gotConnection)
function gotConnection(err, _c) {
connection = _c
connection.query('SELECT * FROM test_table', gotData)
}
function gotData(err, rows) {
connection.end();
var templateVariables = {
title: 'Index Page',
response: rows[0].text
}
res.render('index', templateVariables);
}
});
You should also always handle errors as well in your code, I'm assuming you left them out to make the code easier to read here.
I found the accepted answer in this question to answer an error handling question I had.
Note the comment though - a http.ServerResponse needs to be finished so that the application doesn't run out of memory eventually - this is a concern.
How would one implement something like that though? how can the callback function have access to said object?
As is mentioned in the answer to this post - if you have an un-caught exception your best approach is to let the process die and recover from there - otherwise you have no idea what the current state of the application is.
Because you did not catch the Exception, it suggests that the application does not know what went on. To get around this - you design your application in lots of little chunks rather than in one process.
This takes a slight hit on performance (RPC vs in process communication) but it means you get to sleep easy whilst tiny chunks go wrong, restart and everything is fine because they never maintain state within themselves (use Redis for that)...
If you really want to catch an exception within a request and then call end() on the request you must use a closure with access to the response like so:
// a method that will catch an error and close the response
var runFunctionSafely = function(res, tryFunction){
try{
tryFunction();
}
catch(error){
console.log('there was an error, closing the response...');
console.log(error);
res.end();
}
}
app.get('/test', function(req, res, next) {
runFunctionSafely(res, function(){
throw new Error('this is a problem');
})
});
The 'uncaughtException' handler doesn't have access to the response object, so it can't end the response. The presumed point of the comment is that you shouldn't rely on 'uncaughtException' for all your error handling needs because in many cases (like the one mentioned) you're not in the right scope to properly handle the error.