I have an express app and one of the functionalities is "moving" files. For example, you can drag a folder to another folder and have it's contents move. Once all the contents are moved, the server responds accordingly.
What I've found is that when doing this with folders that contain many files (possibly additional folders), this can cause the event loop to get blocked for up to 3+ seconds, which I've found using blocked.
Would anyone have any suggestions on how I could prevent this? One option I though is to use child_process.fork, I'm not sure how that will tie into an http route, but I'll test it. Would there be any other ways to improve something like this?
Code example: This is one part I'm testing now, that basically is building a "tree" from a materialized path pattern in Mongo:
var items = []
Items.find({parentId: null, user_id: '123', type: 'directory'})
.exec(function(err, docs) {
if (err) return res.send(err);
async.each(docs, function(doc, cb) {
doc.getArrayTree({
condition: {type: 'directory'}
}, function(err, childDocs) {
if (_.isEmpty(childDocs)) return cb();
items.push(childDocs[0]);
cb();
});
}, function(err) {
if (err) return res.send(err);
res.send(items);
});
});
This is using mongoose materialized, specifically I believe the issue/delay is with this. Which is being called for every item returned. There must be a more efficient way, perhaps with an aggregate. This is showing about a +-500ms delay, of course this degrades as there are more documents.
Related
Firstly to be mentioned, I'm absolutely new to Node.Js and MongoDB.
I'm coding a back end API with Node.Js and MongoDB which will deal with GET, POST, DELETE requests from the front end, quite simple stuff.
I'm stuck while working with DELETE functionality.
Here is my posts.service.ts file contains this deletePost() function which sends the postId to the back end app.js file.
`
deletePost(postId: string) {
this.http.delete('http://localhost:3000/api/posts/' + postId)
.subscribe(() => {
console.log(postId);
console.log('Deleted');
});
}
`
I have added this console.log(postId) to check whether it is actually containing the actual postId and found that it does. I have also matched it with the real Id in MongoDB by checking through mongo shell.
Here is the delete() function in the back end app.js file that should do the actual task.
`
app.delete("/api/posts/:id", (req, res, next) => {
Post.deleteOne({ _id: req.params.id }).then(result => {
console.log(result);
res.status(200).json({message: "Post deleted"});
});
});
`
The console.log(result) line should print some result in the terminal, but it does not, so does not it delete the collection in the DB.
I`m running this on an Ubuntu 16.04 LTS pc.
Some clue would mean great help. Thank you very much for your kind effort.
deleteOne doesn't return the deleted document. It always deletes the first matching document and return the number of documents deleted with the boolean value.
From the mongodb docs deleteOne:
Returns:
A document containing: A boolean acknowledged as true if the operation ran with write concern or false if write concern was
disabled
deletedCount containing the number of deleted documents
From the mongoose docs
Deletes the first document that matches conditions from the
collection. Behaves like remove(), but deletes at most one document
regardless of the single option.
I was facing exactly the same, i solved returning the deleteOne promise object and then using the .then property of the promise.
Something like this:
Model.js
...
bicicleSchema.statics.deleteById= function(id, cb){
return this.deleteOne({code: id}, cb);
};
...
module.exports = mongoose.model('Bicicle', bicicleSchema);
Service.js
var Bicicle = require('../../../model/bicicle');
...
const id = req.params.id;
Bicicle.deleteById(id).then(()=>{
//your code
});
...
Steps:
pull data from mongodb to populate a link on search page. Be able to click the link to take end user to the generic business page which is auto filled by mongodb. render the page with the JSON data from mongodb.
Issues:
res.render gets stuck in Infinite Loop (no error is thrown also)
I've tried adding an if statement so res.render would not render again, yet still stuck in a loop.
routes/business.js
router.get('/:business', (req, res) => {
console.log(req.params.business);
Business.getBusinessByUrl(req.params.business, (err, business) => {
if (err) throw err;
res.render('business', {
found: true,
business_name: business.business_name,
business_profile_image: business.business_profile_image,
business_url: business.business_url
});
});
});
module.exports = router;
models/business.js
module.exports.getBusinessByUrl = function (businessUrl, callback) {
const query = { business_url: businessUrl };
Business.find(query, callback);
};
Here is what it looks like in the browser :
Imgur
Let me know if you need any other code.
After a while of searching I couldn't find anything helpful. Eventually I figured it out. It happened to be that when I had commented out the line <div w3-include-html="file-included.html"> it stopped infinitely loading. A thank you too everyone who tried to help me.
I've been working on a server and a push notification daemon that will both run simultaneously and interact with the same database. The idea behind this is that if one goes down, the other will still function.
I normally use Swift but for this project I'm writing it in Node, using Mongoose as my database. I've created a helper class that I import in both my server.js file and my notifier.js file.
const Mongoose = require('mongoose');
const Device = require('./device'); // This is a Schema
var uri = 'mongodb://localhost/devices';
function Database() {
Mongoose.connect(uri, { useMongoClient: true }, function(err) {
console.log('connected: ' + err);
});
}
Database.prototype.findDevice = function(params, callback) {
Device.findOne(params, function(err, device) {
// etc...
});
};
module.exports = Database;
Then separately from both server.js and notifier.js I create objects and query the database:
const Database = require('./db');
const db = new Database();
db.findDevice(params, function(err, device) {
// Simplified, but I edit and save things back to the database via db
device.token = 'blah';
device.save();
});
Is this safe to do? When working with Swift (and Objective-C) I'm always concerned about making things thread safe. Is this a concern? Should I be worried about race conditions and modifying the same files at the same time?
Also, bonus question: How does Mongoose share a connection between files (or processes?). For example Mongoose.connection.readyState returns the same thing from different files.
The short answer is "safe enough."
The long answer has to do with understanding what sort of consistency guarantees your system needs, how you've configured MongoDB, and whether there's any sharding or replication going on.
For the latter, you'll want to read about atomicity and consistency and perhaps also peek at write concern.
A good way to answer these questions, even when you think you've figured it out, is to test scenarios: Hammer a duplicate of your system with fake data and events and see if what happen is OK or not.
I have got the following code to automatically load my handelbar templates from the views/static folder without having to manually setup a route for each page.
app.get("/:template", function(req,res){
var template = req.params.template; // Is this safe?
res.render("static/" + template, function(err, html) {
if (err) {
res.send(404, 'Sorry cant find that!');
} else {
res.send(html);
}
});
});
It works fine, however I am worried that this potentially exposes my app to security problems. Any suggestions how I could do this better. I am using Express. Thanks so much for your help.
I think it's pretty safe.
Usually, you have to worry about paths being passed that contain stuff like ../ (to go back a directory level), but those won't match your route. Also, the route you declare will stop matching at a /, so requests like /foo/../bar won't match either.
An issue that may occur is when the static directory contains files that you don't want to expose: a request for /secret.js will at least try to render a file called static/secret.js.
I have just started a large project with node.js and I have stumbled upon some callback nightmare issues. I have done some node.js development before, but only small stuff based on tutorials.
I have a user model and a owner model, who is the owner for that user...basically the part I am building in node.js will be a REST service so I need to send a json containing a user and it's owner name. My problem is trying to get the owner name like I would do it in ruby or php and setting it as a property..but it doesn't work.
How do people handle this kind of logic in node.js where you need to change objects in callback functions? Also I do not want it to affect performance.
I am using mysql as the database because this was a requirement. So the code is this:
//find all
req.models.users.find({
'id' : req.query.id
}, function(err, users) {
if (err) console.log(err); //do something better here
else {
var json = [];
users.forEach(function(user) {
user.getOwner(function(err, owner) {
user.accountOwner = owner.name;
json.push(user;
});
});
res.type('application/json');
res.send(json);
}
});
I want to send something like this:
[{
'id': user.id,
'username": user.username,
'email': user.email,
'owner': user.owner.name'
},
{...}]
The problem is you are not understanding the control flow of node code. It doesn't go line by line top to bottom in chronological order. So your issue is your res.send(json) happens BEFORE (in time) your user.getOwner callback executes, so you send your empty json, then you stuff things into the json array after it's already been sent. You need to use something like async.each to do your joins, wait for all of the user`s owners to be populated, and then send the response. Or you could actually let the database join the data by writing a SQL join instead of doing N+1 queries against your database.