What I want
First, I use COPY command to Postgresql database and export them into many json files by table.
Each json file contain an array of objects, each object is from each row of table, each object also include its primary key 'id'.
Then, I insert data from those json files into an empty database.
Finally, I want to insert other data succefully, any warning about duplicate primary key won't happen.
DB Schema
Users
id: {
allowNull: false,
autoIncrement: true,
primaryKey: true,
type: Sequelize.INTEGER
},
name: {
type: Sequelize.STRING
}
Comments
id: {
allowNull: false,
autoIncrement: true,
primaryKey: true,
type: Sequelize.INTEGER
},
userId: {
type: Sequelize.INTEGER,
onDelete: 'CASCADE',
references: {
model: 'Users',
key: 'id'
}
},
comment: {
type: Sequelize.STRING
}
Json Files
Users.json
[
{
"id": 1,
"name": "foo"
}
]
Comments.json
[
{
"id": 1,
"userId": 1,
"comment": "hi"
},
{
"id": 2,
"userId": 1,
"comment": "hey"
}
]
What I tried and error showed
I tried these methods to create data from json files, but got the same error message at further insert.
method 1:
const insertData = [{id: 2, name: 'ofo'}]
Users.bulkCreate(insertData)
method2:
const insertData = [{id: 2, name: 'ofo'}]
for (let i = 0; i < insertData.length; i++) {
Users.create(insertData[i])
}
method3:
INSERT INTO "Users"
SELECT *
FROM json_populate_recordset(null::"Users", json [{id: 2, name: 'ofo'}]);
The error message said:
error: duplicate key value violates unique constraint "users_pkey"
detail: 'Key (id)=(1) already exists.'
I also tried other methods to update primary key value, but got these error messages
method4:
sequelize.query(`select make_serial('public', 'Users', 'id');`)
error message:
error: function make_serial(unknown, unknown, unknown) does not exist
hint: 'No function matches the given name and argument types. You might need to add explicit type casts.'
method5:
sequelize.query(`CREATE SEQUENCE user_id_seq OWNED BY "Users".id;
SELECT SETVAL(user_id_seq, (SELECT MAX(id) FROM "Users"));`)
error message:
error: column "users_id_seq" does not exist
version
Postgresql: 14.5
sequelize: 5.22.4
What problem need to help
I want to know that there is any method to insert data from json file that duplicate key value violates unique constraint "users_pkey" this error won't happend.
Otherwise, there is any method I make postgresql to update primary key value.
Hope to get any suggestion, thanks.
I run this code to update id as primary to max value
SELECT SETVAL(E'\"user_id_seq\"', max(id)) FROM "Users";
Related
I'm developing an online store Node.js REST API with Mongoose (MongoDB), which I'm new to. I decided to test the orders service and saw that after I had made 1 successful order (so it worked once), it sent me a duplicate key error for the next one, for a key 'name' with value 'null', of the order.products collection that is an Array, and not a kvp object.
I should note that nowhere in my code is 'products.name' mentioned.
ERROR:
MongoServerError: E11000 duplicate key error collection: store.orders index: products.name_1 dup
at {...}{
key: { products.name: null }
index: 0,
code: 11000,
keyPattern: { 'products.name': 1 },
keyValue: { 'products.name': null },
[Symbol(errorLabels)]: Set(0) {}
}
when the error is handled, this message is received and it makes no sense:
{ "message": "Order with products.name "null" already exists" }
Order schema:
const schema = new Schema({
userId: {
type: Types.ObjectId,
ref: 'User'
},
address: {
type: addressSchema,
required: true
},
products: {
type: [orderProductSchema],
required: true,
validate: nonEmptyArray
},
status: {
type: Number,
validate: inCollection(Object.values(ORDER_STATUS))
},
price: { type: Number, required: true, min: 0 }
}, { timestamps: true });
don't bother with the validators or the address/status/user/price, it has nothing to do with them; what is more, nothing is specified as unique: true
As you can see, the 'products' field is just an array of products, no 'name' is declared
orderProductSchema:
const schema = new Schema({
product: {
_id: { type: Types.ObjectId, required: true },
name: {
type: String,
required: true,
maxLength: 250
},
displayImage: String,
price: { type: Number, required: true, min: 0 }
},
quantity: {
type: Number,
required: true,
validate: isInteger,
min: 1
},
}, { _id: false });
I have a 'name' field here, but it's just the name of a product. The error is thrown even when the names are unique.
Orders service:
// get data and format it to fit the Order model
console.dir(products); // --> all is how it's expected to be in the schema
return await Order.create({
userId,
address,
products,
status: ORDER_STATUS.AWAITING_CONFIRMATION,
price: totalOrderPrice
});
It seems to me that this is some weird MongoDB behaviour/specification that I missed. If someone has any idea what could cause the problem - please help.
I tried removing all parts such as other fields and validations to see if they might've caused the problem but that was not the case. I thought maybe I had formatted the object I send to the database wrong, but I console.log-ed it and it was fine ({products: Array})
Thanks to #user20042973 and #MTN I saw that my 'orders' database had index 'products.name' (no idea how it got there).. I just removed the index and the problem is solved.
I am currently working on a MongoDB Post-Hashtag system, I am using NodeJS and the library "Mongoose". Following the creation of a new "post", I wish to increment the quantity of the "posts" field of the given hashtag's document, else if it does not exist: create a new document for the hashtag. I attempted to do this by utilizing the upsert option and performing a single update (updateOne) for each hashtag. The upserting & new document creation process for a new hashtag appears to work fine, however, pre-existing hashtags do not have their "posts" field incremented.
My code:
await Hashtag.updateOne({hashtag: hashtag}, {
$set: {$inc: {posts: 1}},
$setOnInsert: {
uniqueID: hashtagID,
hashtag: hashtag,
timeOfCreation: now,
posts: 1
}
}, {upsert: true});
I have attempted both with & without the '$set' operator, as well as '$update' and '$inc' to no avail. Omitting it results in the following error:
{
"errorType": "MongoServerError",
"errorMessage": "Updating the path 'posts' would create a conflict at 'posts'",
"code": 40,
"index": 0,
"stack": [
"MongoServerError: Updating the path 'posts' would create a conflict at 'posts'",
...
}
My schema as defined in Mongoose:
const hashtagSchema = new mongoose.Schema({
uniqueID: {
type: String,
required: true
},
hashtag: {
type: String,
required: true
},
timeOfCreation: {
type: Number,
required: true
},
posts: {
type: Number,
required: true
},
});
Remove the $set which is used before $inc.
await Hashtag.updateOne({hashtag: hashtag}, {
$inc: {posts: 1},
$setOnInsert: {
uniqueID: hashtagID,
hashtag: hashtag,
timeOfCreation: now,
posts: 1enter code here
}
}, {upsert: true});
I'm trying to save an object that references other 2 objects in mongoDB, but I'm not getting it. Whenever I try, I get this message.
For this, this using POSTMAN to test the API that I am creating.
{
"message": "Order validation failed: payments.0.credit_card.card: Cast to ObjectID failed for value \"{ number: '4898308633754712',\n holder_name: 'Test test',\n exp_month: 1,\n exp_year: 2022,\n cvv: '1234' }\" at path \"credit_card.card\", customer: Cast to ObjectID failed for value \"{ name: 'Test Test', email: 'test#gmail.com' }\" at path \"customer\""
}
The json object I'm trying to save in mongoDB:
{
"items": [{
"name": "Plano s",
"amount": 12345,
"description": "Descrição do Plano Sanarflix",
"quantity": 1
}],
"customer": {
"name": "Test Test",
"email": "test#gmail.com"
},
"payments": [{
"payment_method": "credit_card",
"credit_card": {
"installments":1,
"capture": true,
"card": {
"number": "4898308633754712",
"holder_name": "Test Test",
"exp_month": 1,
"exp_year": 2022,
"cvv": "1234"
}
}
}]
}
This is the model order I defined:
'use strict';
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
// Model for order
const schema = new Schema({
customer: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Customer',
required: function(){
return this.customer_id;
}
},
items: [{
amount: {
type: Number,
required: true
},
description: {
type: String,
trim: true
},
quantity: {
type: Number,
required: true
}
}],
payments: [{
payment_method: {
type: String,
required: true,
trim: true
},
credit_card: {
installments: {
type: Number,
default: 1
},
capture: {
type: Boolean,
default: true
},
card: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Card'
}
},
}]
});
module.exports = mongoose.model('Order', schema);
What is wrong with it? Because when I import the same json to MongoDB with Studio3 Mongo Manager, I can save it and see the object in the correct way.
an object that references other 2 objects, yes, ObjectId does references, yet you're passing the whole objects.
Three options:
pass the id of customer and card objects (if they exist already)
insert these new data in their respective collections then insert your order using the created IDs (likely the solution)
rewrite your schema to have only the Order collection that would hold everything (not really great).
Also, having card numbers transiting through your server is really dangerous (legally speaking), especially if you're to store them... check out how the payment service you're implementing handles transactions: on any decent service the server never sees banking data, discharging your company of any liability.
So I've been at this for awhile and can't see how my code is different from the documentation.
I've also checked out this question, this question, this question, and this unanswered different question.
For my admin panel I'm trying to query to get all the information associated with a user and display a 'master' user profile to the admin.
My User model looks like this:
module.exports = {
autoPK: true,
attributes : {
id: {
type: 'integer',
primaryKey: true,
unique: true
},
email : {
type : 'email',
unique : true,
required : true,
},
password : {
type : 'string',
minLength : 8,
required : true
},
admin:{
type: 'bool'
},
user_profile:{
collection: 'userprofile',
via: 'user_id',
},
properties: {
collection: 'subjectproperties',
via: 'user_id'
},
employment_info: {
collection: 'employmentinfo',
via: 'user_id'
},
file_uploads: {
collection: 'fileupload',
via: 'user_id'
},
nearest_living_relatives:{
collection: 'nearestlivingrelative',
via: 'user_id'
},
mortgage_info: {
collection: 'mortgageinfo',
via: 'user_id'
},
user_progression_state:{
collection: 'userprogressionstate',
via: 'user_id'
},
users_applied_loan_values:{
collection: 'usersappliedloanvalues',
via: 'user_id'
}
}
}
I don't want to list out all the belongs to user models cause there are a lot of them, but here is one of the simpler one's.
EmploymentInfo.js
module.exports = {
tableName: "employment_info",
attributes : {
employers_name:{
type: 'string',
required: true
},
employers_address:{
type: 'string',
required: true
},
employers_city:{
type: 'string',
required: true
},
employers_state:{
type: 'string',
required: true
},
employers_zip:{
type: 'string',
required: true
},
job_position:{
type: 'string',
required: true
},
years_in_position:{
type: 'string',
required: true
},
years_in_industry:{
type: 'integer',
required: true
},
user_id:{
model:'user'
}
}
};
And as for my controller:
create_admin_user_profile: function(req, res){
var user_id = req.query.userId;
User.find({'id': user_id}).populateAll().exec(function(err, user){
if(err || user.length === 0){
sails.log.verbose(err);
}else{
sails.log.verbose(user);
}
});
},
It doesn't error out but all I see in the terminal is this for the above:
[ { user_profile: [],
properties: [],
employment_info: [],
file_uploads: [],
nearest_living_relatives: [],
mortgage_info: [],
user_progression_state: [],
users_applied_loan_values: [],
id: 5,
email: 'test#test.com',
admin: 1 } ]
Even though there is an entry in all of those tables for that user.
If I change the line:
User.find({'id': user_id}).populateAll().exec(function(err, user){
To:
User.find({'id': user_id}).populate('employment_info').exec(function(err, user){
Same but shorter result:
[ { employment_info: [],
id: 5,
email: 'test#test.com',
admin: 1 } ]
I've tried changing the case, I've tried adding columnName to the user_id attribute, I've tried changing the column name across the entire breadth of the project to not have an under_score in it, though that never seemed to be issue in it picking up the names correctly, but nothing I've done seems to work. I've also tried uninstalling sails, and the sails-mysql adapter and clearing my npm cache.
At this point my just stuck, I really can't see a reason why it's not working.
As for project info:
Sails v: 0.12.11
npm v: 3.10.9
node v: 7.2.0
Additional info asked for in comments:
SQL row taken right from db for user 5
employers_name, employers_address, employers_city, employers_state, employers_zip, job_position, years_in_position, years_in_industry, user_id
'Company', 'Mill Steet', 'SLC', 'Utah', '88888', 'Developer', '2', '2', '5'
And json format returned by find method in EmploymentInfo.js controller
{
"employmentInfo": {
"employers_name": "Company",
"employers_address": "Mill Steet",
"employers_city": "SLC",
"employers_state": "Utah",
"employers_zip": "88888",
"job_position": "Developer",
"years_in_position": "2",
"years_in_industry": 2,
"user": 5
}
}
The reason the last param is user and not user_id is because I rename it in the find method to serve the front-end mvc which also has the ability to work with associations. It's also why the JSON has the format it does.
Code from the find method that replaces user_id:
EmploymentInfo.find({'user_id': user_id}).exec(function(err, profile){
if(err || !profile.length){
return res.json(err);
}else{
res.status(200);
profile[0].user = profile[0].user_id;
delete profile[0].user_id;
res.send({'employmentInfo': profile[0]});
}
});
However I've tried not renaming it; I've also tried getting rid of my find override and just relying on the blueprint find method, neither of those worked either.
I'm running into an issue with beforeValidate(), and can't find any answers online. My model has two relationship attributes that require id numbers in POSTed data. If a user POST strings instead, they get a validation error. I want to enable the user to POST string data, then inside beforeValidate() use findOrCreate() to find or create the attribute's related model, then overwrite the POSTed data's attribute with the relate model's ID.
I have the following model:
attributes: {
reporter : { model: 'reporter', required: true },
location : { model: 'location', required: true },
line : { type: 'boolean', required: true, defaultsTo: true },
count : { type: 'int', required: true, defaultsTo: 0},
composition : { type: 'int', required: true, defaultsTo: 3}
},
beforeValidate: function(values, next) {
if (typeof values.reporter !== 'number') {
Reporter.findOrCreate({name: values.reporter}).exec(function(data){
console.log(data)
values.reporter = data.id;
})
};
next();
},
I'm POSTing this data to the model's create() default blueprint endpoint:
{
"location":"Harry's",
"reporter":"tim",
"count":30,
"composition":3,
"line":false
}
When I log the above values inside beforeValidate(), I get this:
{ location: NaN,
reporter: NaN,
count: '30',
composition: '3',
line: false }
When I replace "location" and "reporter" with ID's, I don't get any errors. Why are the string values getting stripped out in the beforeValidate() function?
maybe u need POST(create) to endpoint location model, then POST(create) to reporter model, and then POST YOUMODEL/id/reporterID, POST YOUMODEL/id/location ID