I have a DynamoDB table with the following attributes:
SubscriptionsTable:
Type: AWS::DynamoDB::Table
DeletionPolicy: Retain
Properties:
TableName: ${self:custom.subscriptionsTableName}
AttributeDefinitions:
- AttributeName: eventName
AttributeType: S
- AttributeName: hookUrl
AttributeType: S
KeySchema:
- AttributeName: eventName
KeyType: HASH
- AttributeName: hookUrl
KeyType: RANGE
I would like to delete an item according to its hookUrl and eventName. That's the code of my repository:
const deleteSubscriptionFromRepository = async ({
hookUrl,
eventName,
}: {
hookUrl: string;
eventName: SubscriptionEventName;
}) => {
const mapper = getDatabaseMapper();
try {
return await mapper.delete(Object.assign(new SubscriptionModel(), { hookUrl, eventName }));
} catch (e) {
throw new DatabaseAccessError(e.message, 'deleteSubscriptionFromRepository');
}
};
Instead of using directly the DynamoDB client, I'm using the DataMapper from '#aws/dynamodb-data-mapper';
But I'm receiving the following error:
DatabaseAccessError: The number of conditions on the keys is invalid
How is that possible to delete an item according to its partition and range key? What I'm doing wrong in the above code?
Related
This question already has answers here:
mongodb/mongoose findMany - find all documents with IDs listed in array
(9 answers)
Closed 3 months ago.
I am trying to search using node.js, ejs and mongoose. All the filter parameters are working perfectly but only categoryIds is not (stored as a collection of ObjectIDs in the mongodb document, referring to the respective document in categories collection), always giving me the empty record set.
For example:
If I need to find the a movie called Cosmos (see the attached screenshot) then I can easily find it with all or any filter except categories. Once I select any category, the record-set will go blank even if the I have selected the one which it belongs to.
model.js
const Model = mongoose.model('Movie', new Schema({
...
categoryIds: [{
type: Schema.Types.ObjectId,
trim: true,
default: null,
ref: 'Category',
}],
copyrightId: {
type: Schema.Types.ObjectId,
trim: true,
default: null,
ref: 'Copyright',
},
...
}, {
timestamps: true
});
Controller.js
Router.get('/', (req, res) => {
const search = req.query;
const conditions = (() => {
let object = {};
['releaseYear', 'languageId', 'copyrightId'].forEach(filter => {
if (search[filter] != '') {
object[filter] = search[filter];
}
});
if (typeof search.categoryIds !== 'undefined') {
object.categoryIds = [];
search.categoryIds.forEach(item => object.categoryIds.push(item));
}
if (search.keywords != '') {
object.title = {
$regex: search.keywords,
$options: 'i'
};
}
return object;
})();
const count = await Model.count(conditions);
const items = await Model.find(conditions, {
__v: false,
imdb: false,
trailer: false,
createdAt: false,
updatedAt: false,
}).sort({
status: -1,
releaseYear: -1,
title: 1
})
.populate('languageId', ['title'])
.populate('copyrightId', ['title'])
.populate('categoryIds', ['title'])
.skip(serialNumber)
.limit(perPage);
...
});
All the fields in the search form
{
categoryIds: [
'6332a8a2a336e8dd78e3fe30',
'6332a899a336e8dd78e3fe2e',
'6332a87ba336e8dd78e3fe2c',
'634574ab339b1a6b09c1e144'
],
languageId: '',
copyrightId: '',
releaseYear: '',
rating: '',
seen: '',
status: '',
keywords: '',
submit: 'search' // button
}
filtered search parameters
{
categoryIds: [
'6332a8a2a336e8dd78e3fe30',
'6332a899a336e8dd78e3fe2e',
'6332a87ba336e8dd78e3fe2c',
'634574ab339b1a6b09c1e144'
]
}
Here is the screenshot of mongodb document.
...
if (typeof search.categoryIds !== 'undefined') {
object.categoryIds = {
$in: []
};
search.categoryIds.forEach(item => object.categoryIds.$in.push(
mongoose.Types.ObjectId(item))
);
}
console.log(object);
return object;
The is the final filter object
{
categoryIds: {
'$in': [
new ObjectId("6332a87ba336e8dd78e3fe2c"),
new ObjectId("634669f4a2725131e80d99f1")
]
}
}
Now, all the filters are working perfectly.
Thank you everyone.
The filter should contain all categoryIds and in the same order to match the document. It's not quite clear from the question if it is the intended functionality. If not, most popular usecases are documented at https://www.mongodb.com/docs/manual/tutorial/query-arrays/
I don't recall how mongoose handles types when you query with array function like $all, so you may need to convert string IDs to ObjectIDs manually, e.g.:
search.categoryIds.forEach(item => object.categoryIds.push(
mongoose.Types.ObjectId(item))
);
I am new to dynamoDB and trying to perform some basic operations to learn the subject.
I have successfully created a table using AWS SDK (so no credentials issue) like this:
const newTable = async () => {
//* it's working!!!
try {
const params = {
AttributeDefinitions: [
{
AttributeName: 'email',
AttributeType: 'S',
},
{
AttributeName: 'password',
AttributeType: 'S',
},
],
KeySchema: [
{
AttributeName: 'email',
KeyType: 'HASH',
},
{
AttributeName: 'password',
KeyType: 'RANGE',
},
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
TableName,
StreamSpecification: {
StreamEnabled: false,
},
};
const command = new CreateTableCommand(params);
const data = await client.send(command);
console.log(data);
} catch (err) {
console.log(err);
}
};
I inserted a new item into the table using the AWS console, and now I'm trying to access it using the SDK as follows:
const getItem = async () => {
try {
const params = {
TableName,
Key: {
email: { S: 'ofer#email.com' },
},
};
const command = new GetItemCommand(params);
const response = await client.send(command);
console.log(response);
} catch (err) {
console.error(err);
}
};
When I try to run the code, the following error is received: "ValidationException: The provided key element does not match the schema"
I couldn't figure out where my mistake was
Since you have a composite key, both HASH and RANGE keys need to be specified when getting an item. Both email and password in your case.
For the primary key, you must provide all of the attributes. For
example, with a simple primary key, you only need to provide a value
for the partition key. For a composite primary key, you must provide
values for both the partition key and the sort key.
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html#DDB-GetItem-request-Key
As a side note, you are unlikely to want to make a password a RANGE key.
I just want to change the value of a key of all the objects inside an array
What I want actually -
The object which I queried from the database is -
{
_id: 61389277fa5c742caf959885,
title: 'What is GRE?',
forumTab: 'GRE',
askedAt: 2021-09-08T10:37:43.979Z,
askedBy: {
_id: 60f0a6a9b4259f7ef9c49cc8,
}
}
I want to add more key-value pairs in the askedBy key by again querying the database for the User with the given _id
Now, the user object which is queried is -
{
role: 'student',
_id: 60f0a6a9b4259f7ef9c49cc8,
firstName: 'Rishav',
lastName: 'Raj'
}
Finally I want to return the below object in response -
{
_id: 61389277fa5c742caf959885,
title: 'What is GRE?',
forumTab: 'GRE',
askedAt: 2021-09-08T10:37:43.979Z,
askedBy: {
_id: 60f0a6a9b4259f7ef9c49cc8,
role: 'student',
firstName: 'Rishav',
lastName: 'Raj'
}
}
I am creating a new array questionsToSend and pushing the object with updated key-value pairs which I am getting after querying the database for each elements in the questions array, I have created functions for respective query that I need to render in sequence, even after rendering the functions in proper sequence why the new array questionsToSend is not populating with the objects before returning the response?
router.get("/questions", async (req, res) => {
if (!req.query.forumTab) return res.status(400).send("something went wrong");
const page = parseInt(req.query.page) - 1;
const perPage = parseInt(req.query.perPage);
let questionsToSend = [];
const func0 = async (callback) => {
const questions = await Question.find({ forumTab: req.query.forumTab })
.sort({ askedAt: -1 })
.limit(perPage)
.skip(perPage * page);
console.log("xxxxxxx");
callback(questions);
};
const func1 = async (questions, callBack) => {
questions.forEach(async (question) => {
const askedUserData = await User.findById(question.askedBy._id);
if (!askedUserData) {
const index = questions.indexOf(question);
questions.splice(index, 1);
return;
}
questionsToSend.push({
..._.pick(question, [
"_id",
"title",
"askedAt",
"tags",
]),
askedUserData,
});
console.log(questionsToSend);
});
console.log("yyyyyyyy");
callBack();
};
func0(
(questions) =>
func1(questions, async () => {
console.log("zzzzzzzz");
res.status(200).send(questionsToSend);
})
);
});
We can use aggregation to achieve this
Question.aggregate([
{
$match: { forumTab: req.query.forumTab }
},
{
$lookup: {
from: 'users',
localField: 'askedBy._id',
foreignField: '_id',
as: "user"
}
},
{ $unwind: "$user"},
{ "$addFields": {
"askedBy": {
"$mergeObjects": ["$askedBy", "$user"]
}
}
},
{ $project: { "user" : 0} },
{ $sort: {"askedAt": -1}},
{ $skip: perPage * page},
{ $limit: perPage},
])
$match is used to apply filter
$lookup is used to do a join on a collection. I have assumed the collection name is users.
$lookup returns the matched result as an array. Converting it to object using $unwind since we get only one back.
$addFields with $mergeObjects is merging the existing askedBy field and newly user field
Removing the user field from the result set with $project.
And then sort, skip and limit.
I'm using Sequelize in my NodeJS app with Postgres as database. Following is my correct SQL query which gives me right set of data while running them on database server :-
SELECT
"expertises"."uuid" AS "id",
"expertises"."display_name" AS "name",
"expertises"."modified_by" AS "modifiedBy",
(
SELECT COUNT(expertise_uuid)
FROM expertise_endorsements
WHERE expertise_uuid = "expertises"."uuid"
) AS "users"
FROM "expertises" AS "expertises"
LEFT JOIN "expertise_endorsements" AS "expertise_endorsements"
ON "expertise_endorsements"."expertise_uuid" = "expertises"."uuid"
LIMIT '16' OFFSET '1';
However when I run the app, the ORM is generating the wrong SQL query without any count (as shown below):-
SELECT
"expertises".*,
"expertise_endorsements"."id" AS "expertise_endorsements.id",
"expertise_endorsements"."expertise_uuid" AS "expertise_endorsements.expertise_uuid"
FROM
(
SELECT
"expertises"."id",
"expertises"."uuid" AS "id",
"expertises"."display_name" AS "name",
"expertises"."modified_by" AS "modifiedBy",
"expertises"."uuid"
FROM
"expertises" AS "expertises"
ORDER BY
"expertises"."name" DESC LIMIT '16' OFFSET '1'
)
AS "expertises"
LEFT OUTER JOIN
"expertise_endorsements" AS "expertise_endorsements"
ON "expertises"."uuid" = "expertise_endorsements"."expertise_uuid"
ORDER BY
"expertises"."name" DESC;
This is my model association :-
const consts = require("../services/static/constants");
const Expertises = require("./models/expertises.model");
const ExpertisesEndorsees = require("./models/expertises_endorsees.model");
const ExpertisesEndorsements = require("./models/expertise_endorsements.model");
exports.setAssociations = (db, Sequelize, type) => {
const expertisesModel = Expertises(db, Sequelize);
const expertisesEndorseesModel = ExpertisesEndorsees(db, Sequelize);
const expertisesEndorsementsModel = ExpertisesEndorsements(db, Sequelize);
expertisesModel.hasMany(expertisesEndorsementsModel, { sourceKey: consts.uuidField, foreignKey: consts.expertiseUuid, as: consts.modelAliases.EXPERTISE_ENDORSEMENTS });
expertisesEndorsementsModel.belongsTo(expertisesModel, { foreignKey: consts.uuidField, as: consts.modelAliases.EXPERTISE_ENDORSEMENTS });
return { expertisesModel, expertisesEndorseesModel, expertisesEndorsementsModel };
};
My logic is below. I'm stuck in understanding and applying the count properly using the Sequelize:-
const models = dbUtils.setAssociations(db, consts.getExpertiseFlow);
const includesData = [
{
model: models.expertisesEndorsementsModel,
attributes: [consts.expertiseUuid],
// attributes: [
// sequelize.fn("COUNT", sequelize.col(`${consts.modelAliases.EXPERTISE_ENDORSEMENTS}.${consts.get_endorsee_uuid}`)), "users"
// ],
as: consts.modelAliases.EXPERTISE_ENDORSEMENTS,
required: false
}
];
let {count: expertiseCount, rows: expertises} = JSON.parse(JSON.stringify(
await manageExpertise.getSortedExpertises(models, consts.get_expertises, pageOffset, pageLimit, orderBy, includesData, false)));
Method:-
exports.getSortedExpertises = (models, attributes, offset, limit, orderBy, includes, raw=true) => {
return models.expertisesModel
.findAndCountAll({
attributes,
include: includes,
offset: offset,
limit: limit,
//order: orderBy,
raw
}).then(data => {
return data;
})
.catch(err => {
throw err;
});
};
Would really appreciate if you can assist me in resolving this issue and let me know the root cause. Thank you
This is one of the fields of StringSet type that is returned from DynamoDb.
permissions:
Set {
wrapperName: 'Set',
values:
[ 'BannerConfigReadOnly',
'CampaignBannerCreate',
'CampaignPromoCreate',
'CampaignReadOnly',
'MasterplanReadOnly',
'SegmentCreate',
'SegmentDownload',
'SegmentUpload' ],
type: 'String' }
}
Now, I am using aws.DynamoDB.Converter.unmarshal function to get it in this format
permissions: ['BannerConfigReadOnly',
'CampaignBannerCreate',
'CampaignPromoCreate',
'CampaignReadOnly',
'MasterplanReadOnly',
'SegmentCreate',
'SegmentDownload',
'SegmentUpload']
But, this is what i get
{}
Any ideas what, I may be doing wrong.
This is my code
const aws = require('aws-sdk');
const documentClient = new aws.DynamoDB.DocumentClient();
documentClient.scan(params, (err, data) => {
if (err) {
reject(err);
} else {
let processedItems = [...data.Items];
var test = aws.DynamoDB.Converter.unmarshall(processedItems[0].permissions);
console.log(`test is ${JSON.stringify(test)}`);
}});
ProcessedItems[0] is this
{ email: 'abc#gmail.com',
tenant: 'Canada',
permissions:
Set {
wrapperName: 'Set',
values:
[ 'BannerConfigReadOnly',
'CampaignBannerCreate',
'CampaignPromoCreate',
'CampaignReadOnly',],
type: 'String' } }
That data is already unmarshalled since you are using the DocumentClient. Consider just using processedItems[0].permissions.values to get the values of the set.