We Keep Coding

Mongoose: pre('validate') middleware is not working

This is my mongoose Setup. This happens only when i use class syntax. When i do the same thing with the use of functional programming it works fine. This is the first time i am using class syntax to do this. I think that's where the problem lies. I am doing something wrong with my class definition.
This is my mongoose Setup. This happens only when i use class syntax. When i do the same thing with the use of functional programming it works fine. This is the first time i am using class syntax to do this. I think that's where the problem lies. I am doing something wrong with my class definition.
const mongooseService = require('./services/mongoose.service')
const slugify = require('slugify')
const { marked } = require('marked')
const createDomPurifier = require('dompurify')
const { JSDOM } = require('jsdom')
const dompurify = createDomPurifier(new JSDOM().window)
class ArticleDao {
Schema = mongooseService.getMongoose().Schema
articleSchema = new this.Schema({
title: {
type: String,
required: true,
},
description: {
type: String,
},
markdown: {
type: String,
required: true,
},
createdAt: {
type: Date,
default: new Date(),
},
slug: {
type: String,
required: true,
unique: String,
},
sanitizedHtml: {
type: String,
required: true,
},
})
Article = mongooseService.getMongoose().model('Article', this.articleSchema)
constructor() {
console.log(`created new instance of DAO`)
this.setPreValidation()
}
setPreValidation() {
console.log('h')
this.articleSchema.pre('save', (next) => {
if (this.title) {
this.slug = slugify(this.title, { lower: true, strict: true })
}
if (this.markdown) {
this.sanitizedHtml = dompurify.sanitize(marked(this.markdown))
}
next()
})
}
async addArticle(articleFields) {
const article = new this.Article(articleFields)
await article.save()
return article
}
async getArticleById(articleId) {
return this.Article.findOne({ _id: articleId }).exec()
}
async getArticleBySlug(articleSlug) {
return this.Article.findOne({ slug: articleSlug })
}
async getArticles() {
return this.Article.find().exec
}
async updateArticleById(articleId, articleFields) {
const existingArticle = await this.Article.findOneAndUpdate({
_id: articleId,
$set: articleFields,
new: true,
}).exec()
return existingArticle
}
async removeArticleById(articleId) {
await this.Article.findOneAndDelete({ _id: articleId }).exec()
}
}
module.exports = new ArticleDao()
This is the error i get:
Article validation failed: sanitizedHtml: Path `sanitizedHtml` is required., slug: Path `slug` is required.

Patch request from google sheets overriding previous post request

I am sending 12 cell values from a google sheet to a mongodb database. The reason I'm doing it is because I want to concatenate the 12 cells and do some transforms on the data and output it on a front end somewhere after. I'm also doing this because sheets limit each cell to 50k characters and I have around 500k characters I need to POST to the database each time. My initial assumption here is that I need to create 1 record with all 12 cell values in MondoDB, ( potentially a different avenue would be to just post 12 separate records in a collection ). So the way I'm doing it is doing a POST request, and then potentially 11 PATCH requests on the initially posted record all in one google scripts function inside google sheets. The problem I'm having is that I'm not sure how to form the model & route & request. Everytime I do a patch (shown below) it overrides the previous POST data. I want it to only update the part of the JSON that is being sent via the PATCH request. Currently the PATCH request is changing every other record to null and deleting the previous POSTs data. I know there is a way to patch only a specific record in a collection, but how to patch a specific part of the json in a collection I don't get.
EXPRESS POST ROUTE:
router.post('/', async (req,res) => {
const post = new Post({
title: req.body.title,
en1: req.body.en1,
en2: req.body.en2,
en3: req.body.en3,
en4: req.body.en4,
fr1: req.body.fr1,
fr2: req.body.fr2,
fr3: req.body.fr3,
fr4: req.body.fr4,
de1: req.body.de1,
de2: req.body.de2,
de3: req.body.de3,
de4: req.body.de4
});
try {
const savedPost = await post.save();
res.json(savedPost);
} catch (err) {
res.json({ message: err })
}
})
EXPRESS UPDATE ROUTE:
router.patch('/:postId', async (req,res) => {
try {
const updatedPost = await Post.updateOne(
{ title:req.params.postId },
{ $set: {
title: req.body.title,
en1: req.body.en1,
en2: req.body.en2,
en3: req.body.en3,
en4: req.body.en4,
fr1: req.body.fr1,
fr2: req.body.fr2,
fr3: req.body.fr3,
fr4: req.body.fr4,
de1: req.body.de1,
de2: req.body.de2,
de3: req.body.de3,
de4: req.body.de4
} }
)
res.json(updatedPost)
}catch(err){
res.json({ message: err })
}
})
MONGOOSE MODEL:
const mongoose = require('mongoose')
const PostSchema = mongoose.Schema({
title: {
type: String,
required: false
},
en1: {
type: String,
required: false
},
en2: {
type: String,
required: false
},
en3: {
type: String,
required: false
},
en4: {
type: String,
required: false
},
fr1: {
type: String,
required: false
},
fr2: {
type: String,
required: false
},
fr3: {
type: String,
required: false
},
fr4: {
type: String,
required: false
},
de1: {
type: String,
required: false
},
de2: {
type: String,
required: false
},
de3: {
type: String,
required: false
},
de4: {
type: String,
required: false
},
date: {
type: Date,
default: Date.now
}
})
module.exports = mongoose.model('Posts', PostSchema)
GOOGLE SCRIPT POST REQUEST FROM SHEETS (with only one patch request for now):
function sendInfoToApi() {
const randomId = Math.random()*100000000000000000;
var en1 = SpreadsheetApp.getActive().getSheetByName("Final").getRange('A3').getValues()[0][0];
var en2 = SpreadsheetApp.getActive().getSheetByName("Final").getRange('A4').getValues()[0][0];
// 11111 POST REQUEST //
var data1 = {
"title": randomId,
"en1": en1
}
var payload1 = JSON.stringify(data1)
var url1 = 'https://ag-sheets-api.herokuapp.com/posts';
var fetchParameters1 = {
'method': 'post',
'contentType': 'application/json',
'payload' : payload1,
'muteHttpExceptions' : false
};
try {
var response = UrlFetchApp.fetch(url1, fetchParameters1);
} catch(e){
Logger.log(e)
}
// 22222 PATCH REQUEST //
var data2 = {
"title": randomId,
"en2": en2
}
var payload2 = JSON.stringify(data2)
var url2 = `https://ag-sheets-api.herokuapp.com/posts/${randomId}`;
var fetchParameters2 = {
'method': 'patch',
'contentType': 'application/json',
'payload' : payload2,
'muteHttpExceptions' : false
};
try {
var response = UrlFetchApp.fetch(url2, fetchParameters2);
} catch(e){
Logger.log(e)
}
}
RESULTING RECORD IN MONGODB:
_id:60072da8c52278001791e22e
title:"42612001948065650"
en1:null
date:2021-01-19T19:06:16.052+00:00
__v:0
de1:null
de2:null
de3:null
de4:null
en2:"<div class="qw">
<div class="qe" data-country="gq.svg"> <p cla..."
en3:null
en4:null
fr1:null
fr2:null
fr3:null
fr4:null
Here you can see how the PATCH request is overwriting the data in en1 to null, how can I make it only update en2, and skip the other values?

The issue is that you retrieve all the parameter even if they don't exist in the request. Thus, returning null in all properties.
en1: req.body.en1,
en2: req.body.en2,
en3: req.body.en3,
en4: req.body.en4,
fr1: req.body.fr1,
fr2: req.body.fr2,
fr3: req.body.fr3,
fr4: req.body.fr4,
de1: req.body.de1,
de2: req.body.de2,
de3: req.body.de3,
de4: req.body.de4
You only sent one of them, not all of them.
You should loop on the request keys and those only.
router.patch('/:postId', async (req,res) => {
try {
const updatedObj = {};
for(let i in req.body) {
if(req.body.hasOwnProperty(i) && i.match(/^(title|(en|fr|de)[1-4])$/)) updatedObj[i] = req.body[i];
}
const updatedPost = await Post.updateOne(
{ title:req.params.postId },
{ $set: updatedObj }
)
res.json(updatedPost)
}catch(err){
res.json({ message: err })
}
})
I added some regex validation to the properties send to avoid setting other keys you don't want the web request to update. You can remove that if the request is trusted.

How do I reference mongoose model to another model?

I have a mongoose schema for stories that looks like this:
{
id: {
type: Number,
default: 0
},
title: {
type: String,
maxLength: 60
},
author: {
userid: {
type: Number
},
username: {
type: String
}
}
chapters: [chapter],
numchapters: {
type: Number,
default: 1
},
favs: {
type: Number,
default: 0
},
completed: {
type: Boolean,
default: false
}
}
What I'm trying to do is reference a document in a separate collection (users), and use the values of its userid and username fields in the author field.
how do I do this?
current code:
storyobj.populate('author', {path: 'author', model: 'users', select: 'userid username'}, (err) => {
if (err) {
console.log(err)
}
})
just in case it's relevant, the structure of the users collection looks like this:
{
username: {
type: String,
},
email: {
type: String,
},
password: {
type: String,
},
userid: {
type: Number
},
isAdmin: {
type: Boolean,
default: false
},
banned: {
type: Boolean,
default: false
}
}
EDIT:
I've changed the author field in the Stories model to look like this:
author: {
type: mongoose.Schema.Types.ObjectId,
ref: "User"
}
This is so I tell Mongoose, "Hey, I want this field to reference a user in the User collection".
Here are some more details that I hope will be of help.
Full code:
var storydb = require('../models/stories/story');
var chapterdb = require('../models/stories/chapter');
var userdb = require('../models/user');
const file = JSON.parse(fs.readFileSync('test.json')); // this is a file with the data for the stories I am trying to insert into my database
for (const d in file) {
var storyobj = new storydb({
id: d,
chapters: []
});
for (let e = 0; e < file[d].length; e++) {
var abc = file[d][e];
var updatey = {
chaptertitle: abc.chapter,
chapterid: parseInt(abc.recid),
words: abc.wordcount,
notes: abc.notes,
genre: abc.gid.split(', '),
summary: abc.summary,
hidden: undefined,
loggedinOnly: undefined,
posted: new Date(Date.parse(abc.date)),
bands: abc.bandnames.split(', ')
};
var kbv = getKeyByValue(userlookup, abc.uid);
storyobj.title = abc.title;
storyobj.numchapters = file[d].length;
storyobj.favs = file[d][0].numfavs;
updatey.characters = abc.charid.split(/, |,/g);
storyobj.chapters.push(updatey)
}
storyobj.save();
}
In file, there's a unique ID representing the author of each story. kbv returns the userid associated with that unique ID (note that they're NOT the same).
Now, here's where I'm having trouble:
What I want to do is find a user matching the userid in kbv, and make that the author property in the story model.
The code I'm currently using to try and achieve that:
storydb.findOne({storyobj}, 'author').populate("author", (f) => console.log(f));

const Stories = require("./path/to/model");
Stories
.find({ /* query */ }, { /* projection */ })
.populate("author.username", ["userid", "username"])
.then(/* handle resolve */)
.catch(/* handle rejection */)
For this to work, you have to add a ref key to the userid key in your model, where the ref value is the name of the model it's referencing.
Story.model.js
const StorySchema = new Schema({
author: {
userid: { type: Schema.Types.ObjectId, ref: "users", required: true },
/* other props */
}
/* other props */
});

Dynamic query in mongo and NodeJs asking for fields of documents embedded?

I am trying to make a dynamic query based on multiple selection of the user.
In my application I have the Publication schema that has the Pet schema embedded as follows:
var status = ["public", "private", "deleted"];
var publication_schema = new Schema({
pet:{
type: Schema.Types.ObjectId,
ref: "Pet"
},
status: {
type: String,
enum: status,
default: status[0]
}
});
module.exports = mongoose.model('Publication', publication_schema);
var pet_schema = new Schema({
type: {
type: String,
require: true
},
createdDate: {
type: Date,
default: Date.now
}
});
module.exports = mongoose.model('Pet', pet_schema);
Insyde an async method I build the query, getting all the user input values from the object filter, also I have the query object where I push the different criteria and use it with an $and
let query = {};
let contentQuery = []
if (filter.public && !filter.private) {
contentQuery.push({ status: { $eq: "public" } });
} else if (filter.privada && !filter.public) {
contentQuery.push({ status: { $eq: "private" } });
}
query = { $and: contentQuery }
try {
const publication = await Publication.find(query).populate('pet');
} catch (e) {
console.log(e)
}
the problem is when I want to add more criteria such as follows:
if (filter.specie) { // for example filter.specie equals 'cat'
contentQuery.push({ pet: { type: { $eq: filter.specie } } });
}
I get the error:
'Cast to ObjectId failed for value "{ type: { \'$eq\': \'cat\' } }" at path "pet" for model "Publication"',
name: 'CastError',
stringValue: '"{ type: { \'$eq\': \'cat\' } }"',
kind: 'ObjectId',
value: { type: { '$eq': 'cat' } },
path: 'pet',
reason: undefined,
model: Model { Publication } }
So. How can I do to query the fields of publication and also the pet fields inside publication?

You can have a look on Populate Query Conditions
Instead of .populate('pet') you could do something like
Publication.find({})
.populate({
path: 'pet',
match: { specie: 'cat'},
// You can select the fields you want from pet, or remove the select attribute to select all
select: 'name -_id',
// Here you could add options (e.g. limit)
options: { limit: 5 }
}).exec();
The above query will get you all Publications with pet.specie equals to 'cat'

How to write maintainable JavaScript ala' Bash scripts with advanced reporting features?

I would like to write a script to execute backup procedure for multiple computers in an office. How can I write in a way that I can granually control execution path and also is easy to read and modify? Do I need OOP and SOLID?
The script should make sure that a computer is alive, if not WoL it and leave it in initial state after backup.
Also, the script should do a couple of basic health checks such as smartctl -H and after that execute rsync ... and brtbk ... commands to do the actual backup.
I would like the script to produce a one page report sent to an email address with a clear title indicating wether I should investigate or I can ignore the email.
I already tried to write this in vanilla JS with async/await but failed because of a complex configuration JSON I came up with.
var task = {
type: 'main',
config: {
host: '10.5.1.158',
mac: 'e0:d5:5e:ee:de:3d',
},
task: {
type: 'ensureAlive',
task: [
{
type: 'smartCheck',
dev: '/dev/sda'
},
{
type: 'smartCheck',
dev: '/dev/sdb'
},
{
type: 'failEarly',
task: [
{
type: 'rsync',
config: {
from: `root#{{config.ip}}:/home/VirtualBox\ VMs/a15:/backups/a15/vms/a15/`,
to: '/backups/a15/',
}
},
{
type: 'btrfsSnapshot',
config: {
dir: '/backups/a15/',
},
}
]
}
]
}
};
async function run(ctx, task) {
if (!task) {
return;
}
if (Array.isArray(task)) {
for (var i = 0; i < task.length; i++) {
await run(ctx, task[i]);
}
return;
}
var config = Object.assign({}, ctx.config || {}, task.config || {});
var f = ctx.getFunction(task.type);
try {
var result = await f(config);
task.output = result;
} catch (error) {
task.output = Output({ isOk: false, errorMessage: error.message, errorStack: error.stack })
}
var newCtx = Object.assign({}, ctx, { config });
await run(newCtx, task.task);
}
The recurring run function became too complex to understand and modify/add features.
I was expecting to get something as easy to read as this, no matter wether this is JSON or actual JavaScript. Pseudocode below:
async function a15(report) {
var wasAlive = wakeUp();
try {
await smartCheck();
} catch (error) {
report.addError(error);
}
try {
await smartCheck();
} catch (error) {
report.addError(error);
}
try {
await rsync();
await btrbk();
} catch (error) {
report.addError(error);
}
if (!wasAlive) {
shutDown();
}
}
What am I doing wrong? Is this even possible?
For extra clarification I would like to attach additional config layouts I tried.
Another config try which happened to be too complex to program. Since this config is flat, the main difficulty is related to passing a variable indicating that host is alive (at wakeUp) to the end of the config (at shutDown).
var a15: TaskDescription[] = [
{
type: 'wakeUp',
config: {
host: '10.5.1.252',
mac: 'e0:d5:5e:ee:de:3d'.replace(/:/g, ''),
timeout: '5',
user: 'root',
privateKey: '/Users/epi/.ssh/id_rsa',
},
},
{
type: 'smartCheck',
config: {
dev: '/dev/sda',
},
},
{
type: 'smartCheck',
config: {
dev: '/dev/sdb',
},
},
{
type: 'command',
configTemplateFromConfig: true,
config: {
command: 'rsync -a --inplace --delete -e ssh root#{{host}}:/home/santelab/VirtualBox\ VMs/a15:/mnt/samsung_m3/a15/ /backups/a15/'
},
},
{
type: 'command',
config: {
command: 'btrbk -c /mnt/samsung_m3/a15.conf run'
},
},
{
type: 'shutDown',
runIf: 'wasAlive',
config: {
host: '10.5.1.252',
mac: 'e0:d5:5e:ee:de:3d'.replace(/:/g, ''),
timeout: '5',
user: 'root',
privateKey: '/Users/epi/.ssh/id_rsa',
},
},
];
export interface TaskDescription {
type: string;
config?: TaskConfig;
configTemplateFromConfig?: boolean;
ignoreError?: boolean;
runIf?: string;
}
export type TaskConfig = {
[key: string]: string,
}

I think I made it.
There are 2 key concepts that are required:
never throw an error, always return a result that wraps error and value
use generators - yield magic
Here is a proof of concept for localhost writen in TypeScript. I could use JavaScript as well.
import { WakeUp, Config as WakeUpConfig, Result as WakeUpResult } from './WakeUp';
import { SmartCheck, Config as SmartCheckConfig } from './SmartCheck';
import { Command, Config as CommandConfig, Result as CommandResult } from './Command';
import * as I from './interfaces';
async function* localhost() {
var wakeUpResult = (yield WakeUp({ host: 'localhost', mac: 'e0:d5:5e:ee:de:3d', timeout: 5, tries: 20 })) as WakeUpResult;
if (wakeUpResult.error) { return; }
var echoResult = (yield Command({ command: `echo test` })) as CommandResult;
if (echoResult.error) { return; }
if (!wakeUpResult.value) {
yield Command({ command: 'echo shutdown' });
}
}
(async function () {
var iterator = localhost();
var lastResult: IteratorResult<any> = { value: undefined, done: false };
do {
lastResult = await iterator.next(lastResult.value);
} while (!lastResult.done);
})()
I think the core localhost() is easy to read and allows for easy modifications.