We Keep Coding

Getting error when using objection JS to create query: NJS-012: encountered invalid bind data type in parameter 2

I have been spending hours trying to debug this error. I am using Node, Joi and Oracledb. When I attempted to make a POST request to insert my data from the payload of the request to my table, it gives me the error: NJS-012: encountered invalid bind data type in parameter 2. Nothing I did has been productive to fix the issue. What did I do wrong to cause this issue? The code I use to update my table is in this manager:
manager.js
async function create(payload) {
try {
return await MassRepublishJob.query().insertAndFetch(payload)
} catch (error) {
log.error(`Error while creating mass republish jobs with payload: ${JSON.stringify(payload)}`, error)
}
}
This manager code is called from controller code:
Controller.js
async save(request, reply) {
const instance = await massRepublishJobManager.create(request.payload)
reply(instance)
}
This controller is the handler of my POST route:
method: 'POST',
path: `/${root}`,
config: {
tags: ['api'],
handler: controller.save,
description: 'Create new mass republish job',
notes: 'You can create a mass republish job by either sending a list of content ids or send an object of search querys',
validate: {
payload: {
counts: Joi.object().example(Joi.object({
total: Joi.number().example(1),
completed: Joi.number().example(1),
failed: Joi.number().example(0),
queued: Joi.number().example(0),
})),
type: Joi.string().example('manual'),
content_ids: Joi.array().items(Joi.number().example(11111)),
search_query: Joi.object().example(Joi.object({
query: Joi.string().example('string'),
})),
republishing_reason: Joi.string().example('string'),
duration: Joi.number().example(0),
}
}
}
And finally, I set up the table as described in this sql file, the fields "creation_date", "last_update_date", "created_by", "last_updated_by" are the field of the base model. This model extends that base models with extra fields: "counts", "type", "content_ids", "search_query", "republishing_reason", "duration"
CREATE TABLE wwt_atc_api.mass_republish_jobs (
id NUMBER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
counts CLOB DEFAULT('{}') CHECK(counts IS JSON), -- JSON object of
counts: each has completed, failed, total, queued
type VARCHAR(255) NOT NULL, -- right now only support 'manual' but can be more in the future
content_ids CLOB DEFAULT('[]'),
search_query CLOB DEFAULT('{}') CHECK(search_query IS JSON), -- JSON object of query: each has field, operator, value
republishing_reason VARCHAR(255),
duration NUMBER NOT NULL,
creation_date DATE NOT NULL,
last_update_date DATE NOT NULL,
created_by NUMBER NOT NULL,
last_updated_by NUMBER NOT NULL
);
GRANT SELECT, INSERT, UPDATE, DELETE ON WWT_ATC_API.mass_republish_jobs TO wwt_cf_atc_api;
Knex instance also prints out this debugging message:
method: 'insert',
options: {},
timeout: false,
cancelOnTimeout: false,
bindings: [
[ { id: 11111 } ],
'{"total":1,"completed":1,"failed":0,"queued":0}',
310242,
2022-06-23T18:53:28.463Z,
0,
310242,
2022-06-23T18:53:28.463Z,
'string',
'{"query":"string"}',
'manual',
ReturningHelper { columnName: 'ID' }
],
__.knexQueryUid: 'c5885af0-f325-11ec-a4bd-05b5ac65699c',
sql: 'insert into "WWT_ATC_API"."MASS_REPUBLISH_JOBS" ("CONTENT_IDS", "COUNTS",
"CREATED_BY", "CREATION_DATE", "DURATION", "LAST_UPDATED_BY", "LAST_UPDATE_DATE",
"REPUBLISHING_REASON", "SEARCH_QUERY", "TYPE") values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
returning "ID" into ?',
outBinding: [ [ 'ID' ] ],
returning: [ 'ID' ]
I would really appreciate any helps to get me debug this error.

The issue here is that CONTENT_IDS is not being stringified by Knex because the value is an array. This is a well known issue with Knex that occurs because its QueryBuilder does not have any metadata to determine what format it needs to convert a given binding value into in order for that value to be acceptable to underlying driver/database. Typically objects are stringified but arrays are left as is -hence the inconsistent behavior you are seeing in the bindings. The developers of Knex thus recommend stringifying all JSON values before passing them to the QueryBuilder.
In order to ensure that stringification always properly occurs on JSON values, you should define jsonAttributes on your MassRepublishJob Objection Model.
class MassRepublishJob extends Model {
...
static get jsonAttributes() {
return ["counts", "content_ids","search_query"];
}
}
This will ensure both that your model stringifies these values before binding and that it parses the stringified JSON when mapping rows from the database.
In addition, you can use the $parseJson or $toDatabaseJson lifecycle methods to manually modify how the Objection Model will bind arguments before any queries are run. You can further utilize objection helpers like val,raw and fn in $parseJson for fine grained control on how values for your model properties are bound.
import {fn, ...} from "objection"
...
class MassRepublishJobs extends Model {
...
$parseJson(json: Pojo, opt: ModelOptions) {
const superJson = super.$parseJson(json, opt);
superJson.job_id = randomUUID();
// will format sql with: to_date(?,?) and safely bind the arguments
superJson.creation_date = fn('to_date',new Date().toIsoString(),dateFormat);
return superJson;
}

Can't access property from object in Javascript

EDIT: Re-structured question, cleaer, and cleaner:
I have a data object from Sequelize that is sent by node-express:
{
"page": 0,
"limit": 10,
"total": 4,
"data": [
{
"id": 1,
"title": "movies",
"isActive": true,
"createdAt": "2020-05-30T19:26:04.000Z",
"updatedAt": "2020-05-30T19:26:04.000Z",
"questions": [
{
"questionsCount": 4
}
]
}
]
}
The BIG question is, how do I get the value of questionsCount?
The PROBLEM is, I just can't extract it, these two methods give me undefined result:
category.questions[0].questionsCount
category.questions[0]['questionsCount']
I WAS ABLE to get it using toJSON() (From Sequelize lib I think), like so:
category.questions[0].toJSON().questionsCount
But I'd like to know the answer to the question, or at least a clear explanation of why do I have to use toJSON() just to get the questionsCount?
More context:
I have this GET in my controller:
exports.getCategories = (req, res) => {
const page = myUtil.parser.tryParseInt(req.query.page, 0)
const limit = myUtil.parser.tryParseInt(req.query.limit, 10)
db.Category.findAndCountAll({
where: {},
include: [
{
model: db.Question,
as: "questions",
attributes: [[db.Sequelize.fn('COUNT', 'id'), 'questionsCount']]
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
})
.then(data => {
data.rows.forEach(function(category) {
console.log("------ May 31 ----> " + JSON.stringify(category.questions[0]) + " -->" + category.questions[0].hasOwnProperty('questionsCount'))
console.log(JSON.stringify(category))
console.log(category.questions[0].toJSON().questionsCount)
})
res.json(myUtil.response.paging(data, page, limit))
})
.catch(err => {
console.log("Error get categories: " + err.message)
res.status(500).send({
message: "An error has occured while retrieving data."
})
})
}
I loop through the data.rows to get each category object.
The console.log outputs are:
------ May 31 ----> {"questionsCount":4} -->false
{"id":1,"title":"movies","isActive":true,"createdAt":"2020-05-30T19:26:04.000Z","updatedAt":"2020-05-30T19:26:04.000Z","questions":[{"questionsCount":4}]}
4

https://github.com/sequelize/sequelize/blob/master/docs/manual/core-concepts/model-querying-finders.md
By default, the results of all finder methods are instances of the model class (as opposed to being just plain JavaScript objects). This means that after the database returns the results, Sequelize automatically wraps everything in proper instance objects. In a few cases, when there are too many results, this wrapping can be inefficient. To disable this wrapping and receive a plain response instead, pass { raw: true } as an option to the finder method.
(emphasis by me)
Or directly in the source code, https://github.com/sequelize/sequelize/blob/59b8a7bfa018b94ccfa6e30e1040de91d1e3d3dd/lib/model.js#L2028
#returns {Promise<{count: number, rows: Model[]}>}
So the thing is that you get an array of Model objects which you could navigate with their get() method. It's an unfortunate coincidence that you expected an array, and got an array so you thought it is "that" array. Try the {raw:true} thing, I guess it looks something like this:
db.Category.findAndCountAll({
where: {},
include: [
{
model: db.Question,
as: "questions",
attributes: [[db.Sequelize.fn('COUNT', 'id'), 'questionsCount']]
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
raw: true // <--- hopefully it is this simple
}) [...]
toJSON() is nearby too, https://github.com/sequelize/sequelize/blob/59b8a7bfa018b94ccfa6e30e1040de91d1e3d3dd/lib/model.js#L4341
/**
* Convert the instance to a JSON representation.
* Proxies to calling `get` with no keys.
* This means get all values gotten from the DB, and apply all custom getters.
*
* #see
* {#link Model#get}
*
* #returns {object}
*/
toJSON() {
return _.cloneDeep(
this.get({
plain: true
})
);
}
So it worked exactly because it did what you needed, removed the get() stuff and provided an actual JavaScript object matching your structure (POJSO? - sorry, I could not resist). I rarely use it and thus always forget, but the key background "trick" is that a bit contrary to its name, toJSON() is not expected to create the actual JSON string, but to provide a replacement object which still gets stringified by JSON.stringify(). (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#toJSON_behavior)

try to do so category.data[0].questions.questionCount

As mentioned by others already, you need category.data[0].questions[0].questionCount.
Let me add to that by showing you why. Look at your object, I annotated it with how each part would be accessed:
category = { // category
"page": 0,
"limit": 10,
"total": 2,
"data": [ // category.data
{ // category.data[0]
"id": 1,
"title": "movies",
"createdAt": "2020-05-30T19:26:04.000Z",
"updatedAt": "2020-05-30T19:26:04.000Z",
"questions": [ // category.data[0].questions
{ // category.data[0].questions[0]
"questionCount": 2 // category.data[0].questions[0].questionCount
}
],
"questionsCount": "newValue here!"
}
]
}

try this
category.data[0].questions[0].questionCount
the reason why you have to use toJSON is because it's sometimes it is used to customise the stringification behavior. like doing some calculation before assinging the value to the object that will be returned , so it is most likley been used here to calculate the "numb of questions and then return an object with the property questionscount and the number calculated
so the object you retreived more or less looks like this
var cathegory = {
data: 'data',
questions:[{
// some calulation here to get the questionsCount
result=4,
toJSON () {
return {"questionsCount":this.result}
}
}
]
};
console.log(cathegory.questions[0].toJSON().questionsCount) //4
console.log(JSON.stringify(cathegory)) // {"data":"data","questions":[{"questionsCount":4}]}
console.log("------ May 31 ----> " + JSON.stringify(cathegory.questions[0]) + " -->" + cathegory.questions[0].hasOwnProperty('questionsCount')) //false

Sequelize wrap column with " in where condition

I have a JSONB column in DB.
I'd like to have request to DB where I can check if some value in this JSON it true or false:
SELECT *
FROM table
WHERE ("json_column"->'data'->>'data2')::boolean = true AND id = '00000000-1111-2222-3333-456789abcdef'
LIMIT 1
So, my sequelize request:
const someVariableWithColumnName = 'data2';
Model.findOne({
where: {
[`$("json_column"->'data'->>'${someVariableWithColumnName}')::boolean$`]: true,
id: someIdVariable,
},
order: [/* some order, doesn't matter */],
})
And sequelize generate bad result like:
SELECT *
FROM table
WHERE "(json_column"."->'data'->>'data2')::boolean" = true AND id = '00000000-1111-2222-3333-456789abcdef'
LIMIT 1
Split my column by . and add " to every element.
Any idea how to get rid of adding " to the column in where condition?
Edit:
Here is my query with sequelize.literal():
const someVariableWithColumnName = 'data2';
Model.findOne({
where: {
[sequelize.literal(`$("json_column"->'data'->>'${someVariableWithColumnName}')::boolean$`)]: true,
id: someIdVariable,
},
order: [/* some order, doesn't matter */],
})

You can use Sequelize.literal() to avoid spurious quotes. IMHO, wrapping the json handling in a db function might also be helpful.

I just came across a similar use case.
I believe you can use the static sequelize.where method in combination with sequelize.literal.
Here is the corresponding documentation in sequelize API reference: https://sequelize.org/master/class/lib/sequelize.js~Sequelize.html#static-method-where
And here is an example (although I will admit hard to find) in the regular documentation:
https://sequelize.org/master/manual/model-querying-basics.html#advanced-queries-with-functions--not-just-columns-
In the end for your specific sit try something like this:
const someVariableWithColumnName = 'data2';
Model.findOne({
where: {
[Op.and]: [
// We provide the virtual column sql as the first argument of sequelize.where with sequelize.literal.
// We provide the matching condition as the second argument of sequelize.where, with the usual sequelize syntax.
sequelize.where(sequelize.literal(`$("json_column"->'data'->>'${someVariableWithColumnName}')::boolean$`), { [Op.eq]: true }),
{ id: someIdVariable }
]
})

Cannot infer query fields to set error on insert

I'm trying to achieve a "getOrCreate" behavior using "findAndModify".
I'm working in nodejs using the native driver.
I have:
var matches = db.collection("matches");
matches.findAndModify({
//query
users: {
$all: [ userA_id, userB_id ]
},
lang: lang,
category_id: category_id
},
[[ "_id", "asc"]], //order
{
$setOnInsert: {
users: [userA_id, userB_id],
category_id: category_id,
lang: lang,
status: 0
}
},
{
new:true,
upsert:true
}, function(err, doc){
//Do something with doc
});
What i was trying to do is:
"Find specific match with specified users, lang and category... if not found, insert a new one with specified data"
Mongo is throwing this error:
Error getting/creating match { [MongoError: exception: cannot infer query fields to set, path 'users' is matched twice]
name: 'MongoError',
message: 'exception: cannot infer query fields to set, path \'users\' is matched twice',
errmsg: 'exception: cannot infer query fields to set, path \'users\' is matched twice',
code: 54,
ok: 0 }
Is there a way to make it work?
It's impossible?
Thank you :)

It's not the "prettiest" way to handle this, but current restrictions on the selection operators mean you would need to use a JavaScript expression with $where.
Substituting your vars for values for ease of example:
matches.findAndModify(
{
"$where": function() {
var match = [1,2];
return this.users.filter(function(el) {
return match.indexOf(el) != -1;
}).length >= 2;
},
"lang": "en",
"category_id": 1
},
[],
{
"$setOnInsert": {
"users": [1,2],
"lang": "en",
"category_id": 1
}
},
{
"new": true,
"upsert": true
},
function(err,doc) {
// do something useful here
}
);
As you might suspect, the "culprit" here is the positional $ operator, even though your operation does not make use of it.
And the problem specifically is because of $all which is looking for the possible match at "two" positions in the array. In the event that a "positional" operator was required, the engine cannot work out ( presently ) which position to present. The position should arguably be the "first" match being consistent with other operations, but it is not currently working like that.
Replacing the logic with a JavaScript expression circumvents this as the JavaScript logic cannot return a matched position anyway. That makes the expression valid, and you can then either "create" and array with the two elements in a new document or retrieve the document that contains "both" those elements as well as the other query conditions.
P.S Little bit worried about your "sort" here. You may have added it because it is "mandatory" to the method, however if you do possibly expect this to match "more than one" document and need "sort" to work out which one to get then your logic is slightly flawed.
When doing this to "find or create" then you really need to specifiy "all" of the "unique" key constraints in your query. If you don't then you are likely to run into duplicate key errors down the track.
Sort can in fact be an empty array if you do not actually need to "pick" a result from many.
Just something to keep in mind.

Reuse properties from a remote JSON schema, at the same level of the original schema

I have a remote schema "person.json", saved on another file.
{
"id":"#person",
"type":"object",
"properties": {
"name": {"type":"string"},
"gender": {
"type":"string",
"enum":["m", "f"]
},
"age": {"type":"number"}
},
"additionalProperties": false
}
And I have a "man.json" schema, being this one my original schema.
{
"id":"#man",
"type":"object",
"$ref":"person.json",
"properties": {
"beard":"boolean",
"moustache":"boolean"
},
"required": ["name"],
"additionalProperties": false
}
I want to use the properties: "name, gender, etc" from person.json at the same level as the properties: "beard, moustache" from man.json.
Example for validation
{
name: 'John',
gender: 'm',
age: 29,
beard: false,
moustache: true
}
I want to validate the previously shown example, as you see, with all the properties at the same level (not nested).
Is this possible? If yes, how? Thank you very much.
João

You want to use the allOf keyword, combined with $ref:
{
"id": "/schemas/man.json",
"allOf": [{"$ref": "person.json"}],
...
}
(Note: This is v4. In v3, the same thing was called extends.)
In plain English, it says "Every instance following Man schema must also follow the Person schema".
The bigger issue is in fact your use of additionalPropeties. Since person.json bans additional properties, any instance with a "beard" property is actually not a valid Person. If you're going to be extending Person, I advise you remove the constraint banning additional properties.
(Note: this behaviour is the subject of some conflict in the JSON Schema community, but this is what's in the specs.)

I'm guessing a piece of data cannot satisfy two schemas at once, so you'll need to create a third schema which combines them and validate against that.
var personSchema = JSON.parse(person_json);
var manSchema = JSON.parse(man_json)
for (var personProp in personSchema.properties) {
manSchema.properties[personProp] = personSchema.properties[personProp];
}
var manPersonSchema = JSON.stringify(manSchema);