I am developing an app with Mongo, Node.JS and Angular
Every time the object is delivered and handled in the front-end, all objectId's are converted to strings (this happens automatically when I send it as json), but when when I save objects back into mongo, I need to convert _id and any other manual references to other collections back to ObjectID objects. If I want to nicely separate database layer from the rest of my backend, it becomes even more messy, lets assume my database layer has the following signature:
database.getItem(itemId, callback)
I want my backend business treat itemId as opaque type (i.e no require'ing mongo or knowing anything about ObjectId outside of this database layer), yet at the same time I want to be able to take the result of this function and send it directly to
the frontend with express js.
exports.getItem = function(req, res) {
database.getItem(req.params.id, function(err, item) {
res.json(item);
});
};
What I end up doing now is:
exports.getItem = function(itemId, callback) {
if (typeof itemId == 'string') {
itemId = new ObjectID(itemId);
}
var query = {_id: itemId};
items.findOne(query, callback);
};
This way it can handle both calls which come from within the backend, where itemId reference might be coming from another object and thus might already be in the right binary format, as well as requests with string itemId's.
As I already mentioned above, when I am saving an object that came from front-end and which contains many manual references to other collections that is even more painful, since I need to go over the object and change all id strings to ObjectIds.
This all feels very wrong, there must be a better way to do it. What is it?
Thanks!
Related
I have my current Discord Bot project using JavaScript however i cannot find a way to store the value itself when done through a command such as !setname ___ hoping for a response of that being repeated back when I use !myname.
I know how to do it by storing it temporarily through:
bot.on('message', function (user, userID, channelID, message, evt) {
if(message == "!myname") {
bot.sendMessage({
to: channelID,
message: 'Your name is ' + user.username;
});
}
}
However i cant figure out how to do it through the ability to call for it whenever you want rather than getting an immediate response. For example setting my name now but being able to have it output when i use the command instead of having command that sets the value then outputs it in an immediate string
Would recommend a simple key-value dictionary style in the current instance.
(Though you could store it to a text-file or database later if you prefer the data to be persistent across different sessions.)
Instance based:
Based on this post on stackoverflow...
You can create a global dictionary at the top var usernameChangeDict = {};
Whenever the user invokes !setname, just do:
usernameChangeDict[user.id] = Their_new_name_here
Fetch it back using usernameChangeDict[user.id], should return null or empty if they did not set their name before.
Persistent data:
Could either store it in a text file on your local computer, or do a database as Zooly mentioned.(Though personally, a database would be an overkill if your bot is not going to be used by a wide range of people.)
Text file
Read from this post, it contains details on how to write onto a text file.
This post talks about reading from a text file.
I would recommend storing it in a JSON format to make your life easier, since JSON.parse exists.
Most likely you would end up storing in a format like this:
{ "23125167744": "someone", "USER_ID": "username" }
Access it by parsedJson["USER_ID"].
SQLite
This tutorial would probably explain how to set your SQLite for javascript better than I do.
Once its all setup, just store the userID and username, you can just do:
SELECT username FROM yourTable WHERE userID = ?;
and
INSERT INTO yourTable (username, userID) VALUES (?, ?);
For inserting and selecting into database respectively.
Thanks for looking at my question. It should be easy for anyone who has used Meteor in production, I am still at the learning stage.
So my meteor setup is I have a bunch of documents with ownedBy _id's reflecting which user owns each document (https://github.com/rgstephens/base/tree/extendDoc is the full github, note that it is the extendDoc branch and not the master branch).
I now want to modify my API such that I can display the real name of each owner of the document. On the server side I can access this with Meteor.users.findOne({ownedBy}) but on the client side I have discovered that I cannot do this due to Meteor security protocols (a user doesnt have access to another user's data).
So I have two options:
somehow modify the result of what I am publishing to include the user's real name on the server side
somehow push the full user data to the clientside and do the mapping of the _id to the real names on the clientside
what is the best practice here? I have tried both and here are my results so far:
I have failed here. This is very 'Node' thinking I know. I can access user data on clientside but Meteor insists that my publications must return cursors and not JSON objects. How do I transform JSON objects into cursors or otherwise circumvent this publish restriction? Google is strangely silent on this topic.
Meteor.publish('documents.listAll', function docPub() {
let documents = Documents.find({}).fetch();
documents = documents.map((x) => {
const userobject = Meteor.users.findOne({ _id: x.ownedBy });
const x2 = x;
if (userobject) {
x2.userobject = userobject.profile;
}
return x2;
});
return documents; //this causes error due to not being a cursor
}
I have succeeded here but I suspect at the cost of a massive security hole. I simply modified my publish to be an array of cursors, as below:
Meteor.publish('documents.listAll', function docPub() {
return [Documents.find({}),
Meteor.users.find({}),
];
});
I would really like to do 1 because I sense there is a big security hole in 2, but please advise on how I should do it? thanks very much.
yes, you are right to not want to publish full user objects to the client. but you can certainly publish a subset of the full user object, using the "fields" on the options, which is the 2nd argument of find(). on my project, i created a "public profile" area on each user; that makes it easy to know what things about a user we can publish to other users.
there are several ways to approach getting this data to the client. you've already found one: returning multiple cursors from a publish.
in the example below, i'm returning all the documents, and a subset of all the user object who own those documents. this example assumes that the user's name, and whatever other info you decide is "public," is in a field called publicInfo that's part of the Meteor.user object:
Meteor.publish('documents.listAll', function() {
let documentCursor = Documents.find({});
let ownerIds = documentCursor.map(function(d) {
return d.ownedBy;
});
let uniqueOwnerIds = _.uniq(ownerIds);
let profileCursor = Meteor.users.find(
{
_id: {$in: uniqueOwnerIds}
},
{
fields: {publicInfo: 1}
});
return [documentCursor, profileCursor];
});
In the MeteorChef slack channel, #distalx responded thusly:
Hi, you are using fetch and fetch return all matching documents as an Array.
I think if you just use find - w/o fetch it will do it.
Meteor.publish('documents.listAll', function docPub() {
let cursor = Documents.find({});
let DocsWithUserObject = cursor.filter((doc) => {
const userobject = Meteor.users.findOne({ _id: doc.ownedBy });
if (userobject) {
doc.userobject = userobject.profile;
return doc
}
});
return DocsWithUserObject;
}
I am going to try this.
The title might sound strange, but I have a website that will query some data in a Mongo collection. However, there is no user system (no logins, etc). Everyone is an anonymouse user.
The issue is that I need to query some data on the Mongo collection based on the input text boxes the user gives. Hence I cannot use this.userId to insert a row of specifications, and the server end reads this specifications, and sends the data to the client.
Hence:
// Code ran at the server
if (Meteor.isServer)
{
Meteor.publish("comments", function ()
{
return comments.find();
});
}
// Code ran at the client
if (Meteor.isClient)
{
Template.body.helpers
(
{
comments: function ()
{
return comments.find()
// Add code to try to parse out the data that we don't want here
}
}
);
}
It seems possible that at the user end I filter some data based on some user input. However, it seems that if I use return comments.find() the server will be sending a lot of data to the client, then the client would take the job of cleaning the data.
By a lot of data, there shouldn't be much (10,000 rows), but let's assume that there are a million rows, what should I do?
I'm very new to MeteorJS, just completed the tutorial, any advice is appreciated!
My advice is to read the docs, in particular the section on Publish and Subscribe.
By changing the signature of your publish function above to one that takes an argument, you can filter the collection on the server, and limiting the data transferred to what is required.
Meteor.publish("comments", function (postId)
{
return comments.find({post_id: postId});
});
Then on the client you will need a subscribe call that passes a value for the argument.
Meteor.subscribe("comments", postId)
Ensure you have removed the autopublish package, or it will ignore this filtering.
To modify a field in an existing entry in mongoose, what is the difference between using
model = new Model([...])
model.field = 'new value';
model.save();
and this
Model.update({[...]}, {$set: {field: 'new value'});
The reason I'm asking this question is because of someone's suggestion to an issue I posted yesterday: NodeJS and Mongo - Unexpected behaviors when multiple users send requests simultaneously. The person suggested to use update instead of save, and I'm not yet completely sure why it would make a difference.
Thanks!
Two concepts first. Your application is the Client, Mongodb is the Server.
The main difference is that with .save() you already have an object in your client side code or had to retrieve the data from the server before you are writing it back, and you are writing back the whole thing.
On the other hand .update() does not require the data to be loaded to the client from the server. All of the interaction happens server side without retrieving to the client.So .update() can be very efficient in this way when you are adding content to existing documents.
In addition, there is the multi parameter to .update() that allows the actions to be performed on more than one document that matches the query condition.
There are some things in convenience methods that you lose when using .update() as a call, but the benefits for certain operations is the "trade-off" you have to bear. For more information on this, and the options available, see the documentation.
In short .save() is a client side interface, .update() is server side.
Some differences:
As noted elsewhere, update is more efficient than find followed by save because it avoids loading the whole document.
A Mongoose update translates into a MongoDB update but a Mongoose save is converted into either a MongoDB insert (for a new document) or an update.
It's important to note that on save, Mongoose internally diffs the document and only sends the fields that have actually changed. This is good for atomicity.
By default validation is not run on update but it can be enabled.
The middleware API (pre and post hooks) is different.
There is a useful feature on Mongoose called Middleware. There are 'pre' and 'post' middleware. The middlewares get executed when you do a 'save', but not during 'update'. For example, if you want to hash a password in the User schema everytime the password is modified, you can use the pre to do it as follows. Another useful example is to set the lastModified for each document. The documentation can be found at http://mongoosejs.com/docs/middleware.html
UserSchema.pre('save', function(next) {
var user = this;
// only hash the password if it has been modified (or is new)
if (!user.isModified('password')) {
console.log('password not modified');
return next();
}
console.log('password modified');
// generate a salt
bcrypt.genSalt(10, function(err, salt) {
if (err) {
return next(err);
}
// hash the password along with our new salt
bcrypt.hash(user.password, salt, function(err, hash) {
if (err) {
return next(err);
}
// override the cleartext password with the hashed one
user.password = hash;
next();
});
});
});
One detail that should not be taken lightly: concurrency
As previously mentioned, when doing a doc.save(), you have to load a document into memory first, then modify it, and finally, doc.save() the changes to the MongoDB server.
The issue arises when a document is edited that way concurrently:
Person A loads the document (v1)
Person B loads the document (v1)
Person B saves changes to the document (it is now v2)
Person A saves changes to an outdated (v1) document
Person A will see Mongoose throw a VersionError because the document has changed since last loaded from the collection
Concurrency is not an issue when doing atomic operations like Model.updateOne(), because the operation is done entirely in the MongoDB server, which performs a certain degree of concurrency control.
Therefore, beware!
I have just started a large project with node.js and I have stumbled upon some callback nightmare issues. I have done some node.js development before, but only small stuff based on tutorials.
I have a user model and a owner model, who is the owner for that user...basically the part I am building in node.js will be a REST service so I need to send a json containing a user and it's owner name. My problem is trying to get the owner name like I would do it in ruby or php and setting it as a property..but it doesn't work.
How do people handle this kind of logic in node.js where you need to change objects in callback functions? Also I do not want it to affect performance.
I am using mysql as the database because this was a requirement. So the code is this:
//find all
req.models.users.find({
'id' : req.query.id
}, function(err, users) {
if (err) console.log(err); //do something better here
else {
var json = [];
users.forEach(function(user) {
user.getOwner(function(err, owner) {
user.accountOwner = owner.name;
json.push(user;
});
});
res.type('application/json');
res.send(json);
}
});
I want to send something like this:
[{
'id': user.id,
'username": user.username,
'email': user.email,
'owner': user.owner.name'
},
{...}]
The problem is you are not understanding the control flow of node code. It doesn't go line by line top to bottom in chronological order. So your issue is your res.send(json) happens BEFORE (in time) your user.getOwner callback executes, so you send your empty json, then you stuff things into the json array after it's already been sent. You need to use something like async.each to do your joins, wait for all of the user`s owners to be populated, and then send the response. Or you could actually let the database join the data by writing a SQL join instead of doing N+1 queries against your database.