The Issue
I have tried several approaches, but haven't been able to find out how to add numbers to a NS set. This is all running inside a lambda function.
What I'm trying to accomplish
I am creating a dynamodb table where different colors in hex align to a set of ids. I am optimizing the table for fast reads and to avoid duplicates which is why I would like to maintain a set of ids for each hex.
How I'm adding items to the table:
let doc = require('dynamodb-doc');
let dynamo = new doc.DynamoDB();
var object = {
'TableName': 'Hex',
'Item': {
'hex': '#FEFEFE',
'ids': {
'NS': [2,3,4]
}
}
}
dynamo.putItem(object, callback);
Which results in
Then I try to add more ids to the set
Using the Dynamodb Update Item Documentation standards
var params = {
"TableName" : "Hex",
"Key": {
"hex": "#FEFEFE"
},
"UpdateExpression" : "ADD #oldIds :newIds",
"ExpressionAttributeNames" : {
"#oldIds" : "ids"
},
"ExpressionAttributeValues": {
":newIds" : {"NS": ["5", "6"]}
},
};
dynamo.updateItem(params, callback);
This returns the following error, so dynamo thinks :newIds is a map type instead of a set(?)
"errorMessage": "Invalid UpdateExpression: Incorrect operand type for operator or function; operator: ADD, operand type: MAP"
I have also tried these alternative approaches
Try 2:
var setTest = new Set([5, 6]);
var params = {
"TableName" : "Hex",
"Key": {
"hex": "#FEFEFE"
},
"UpdateExpression" : "ADD #oldIds :newIds",
"ExpressionAttributeNames" : {
"#oldIds" : "ids"
},
"ExpressionAttributeValues": {
":newIds" : setTest
},
};
dynamo.updateItem(params, callback);
Error 2 (same error):
"errorMessage": "Invalid UpdateExpression: Incorrect operand type for operator or function; operator: ADD, operand type: MAP"
Try 3:
var params = {
"TableName" : "Hex",
"Key": {
"hex": "#FEFEFE"
},
"UpdateExpression" : "ADD #oldIds :newIds",
"ExpressionAttributeNames" : {
"#oldIds" : "ids"
},
"ExpressionAttributeValues": {
":newIds" : { "NS" : { "L" : [ { "N" : "5" }, { "N" : "6" } ] }}
},
};
dynamo.updateItem(params, callback);
Error 3 (same error):
"errorMessage": "Invalid UpdateExpression: Incorrect operand type for operator or function; operator: ADD, operand type: MAP"
Try 4:
var params = {
"TableName" : "Hex",
"Key": {
"hex": "#FEFEFE"
},
"UpdateExpression" : "ADD #oldIds :newIds",
"ExpressionAttributeNames" : {
"#oldIds" : "ids"
},
"ExpressionAttributeValues": {
":newIds" : [5,6]
},
};
dynamo.updateItem(params, callback);
Error 4 (similar error, but dynamo thinks I'm adding a list this time)
"errorMessage": "Invalid UpdateExpression: Incorrect operand type for operator or function; operator: ADD, operand type: LIST"
Stack Overflow/Github Questions I've Tried
https://stackoverflow.com/a/37585600/4975772 (I'm adding to a set, not a list)
https://stackoverflow.com/a/37143879/4975772 (I'm using javascript, not python, but I basically need this same thing just different syntax)
https://github.com/awslabs/dynamodb-document-js-sdk/issues/40#issuecomment-123003444 (I need to do this exact thing, but I'm not using the dynamodb-document-js-sdk, I'm using AWS Lambda
How to Create a Set and Add Items to a Set
let AWS = require('aws-sdk');
let docClient = new AWS.DynamoDB.DocumentClient();
...
var params = {
TableName : 'Hex',
Key: {'hex': '#FEFEFE'},
UpdateExpression : 'ADD #oldIds :newIds',
ExpressionAttributeNames : {
'#oldIds' : 'ids'
},
ExpressionAttributeValues : {
':newIds' : docClient.createSet([1,2])
}
};
docClient.update(params, callback);
Which results in this dynamodb table:
If the set doesn't exist, then that code will create it for you. You can also run that code with a different set to update the set's elements. Super convenient.
Create a Set and Add Items to a Set (OLD API)
let doc = require('dynamodb-doc');
let dynamo = new doc.DynamoDB();
var params = {
TableName : 'Hex',
Key: {'hex': '#555555'},
UpdateExpression : 'ADD #oldIds :newIds',
ExpressionAttributeNames : {
'#oldIds' : 'ids'
},
ExpressionAttributeValues : {
':newIds' : dynamo.Set([2,3], 'N')
}
};
dynamo.updateItem(params, callback);
(Don't use this code for future development, I only include it to help anyone using the existing DynamoDB Document SDK)
Why the Original Was Failing
Notice how when I asked the question, the resulting set looked like a literal json map object (when viewed in the dynamodb screenshot), which would explain this error message
"errorMessage": "Invalid UpdateExpression: Incorrect operand type for operator or function; operator: ADD, operand type: MAP"
So I was using the wrong syntax. The solution is found in the (now depricated) AWS Labs dynamodb-document-js-sdk docs
The full documentation for the newer Document Client can be viewed here: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html
I've been struggling with this too. I discovered that there are actually 2 api's for DynamoDB.
AWS.DynamoDB
AWS.DynamoDBDocumentClient
Like you, I was not able to make the ADD function work. I tried to import and use the DynamoDB api instead of the DynamoDBDocumentClient api. It solved the problem for me. Below is my code snippet that works with the DynamoDB api but not with the DynamoDBDocumentClient api. I use a String Set instead of a Number Set, but that won’t make a difference I guess.
var AWS = require('aws-sdk');
var dynamo = new AWS.DynamoDB();
// var dynamo = new AWS.DynamoDB.DocumentClient();
...
var params = {};
params.TableName = “MyTable”;
params.ExpressionAttributeValues = { ':newIds': { "SS": ["someId"] } };
// :newIds represents a new dynamodb set with 1 element
params.UpdateExpression = "ADD someAttribute :newIds";
dynamoClient.updateItem(params, function(err, data) { ….}
This answer might be helpful to those who are using npm dynamoDB
module
We had the same issue . AS we were using npm dynamodb module for our queries rather than exporting from AWS.DynamoDB module, given solution of AWS.DynamoDBDocumentClint was implementable untill and unless we could shift from npm dynamoDb module to AWS.DynamoDB queries. So instead of shifting/transforming queries.
We tried to downgrade dynamoDB npm module to version ~1.1.0 and it worked.
Previous version was 1.2.X.
Related
Expected behavior
pgp.helpers.update should be able to update enum and timestamp.
I have a table ("myTable") it has following columns
id (varchar)
comparator (type enum named as comparator_type with values ['eq', 'ne', 'leq', 'geq', 'gt', 'lt'])
updatedAt (timestamp)
The entries to update are
entries = [
{
id: "someId",
comparator: "gt",
updatedAt: new Date().toISOString()
}
]
// Using pgp.helpers.update
const updateQuery = pgp.helpers.update(entries, ["?id", "comparator", "updatedAt"], "myTable") + ' WHERE v.id = t.id';
console.log(updateQuery);
// printing it here for reference
/**
* update "myTable" as t set "comparator"=v."comparator","updatedAt"=v."updatedAt" from (values('someId','gt','0.92','2023-02-17T19:46:38.723Z')) as v("id","comparator","updatedAt") WHERE v.id = t.id
**/
Actual behavior
It is not updating type enum and timestamp. This is following error I'm getting
Steps to reproduce
The following code is being used to run the generated query
await pgDb.any(updateQuery);
The following error I'm getting
{
"name": "error",
"severity": "ERROR",
"code": "42804",
"hint": "You will need to rewrite or cast the expression.",
"position": "112",
"file": "parse_target.c",
"line": "587",
"routine": "transformAssignedExpr",
"query": "<query above>"
}
When I tried to run the same query in Dbever, it starts to give me the following,
ERROR: column "comparator" is of type comparator_type but expression is of type text. Hint: You will need to rewrite or cast the expression.
if I change the "comparator"=v."comparator" to "comparator"='gt', then it given me next error that is
ERROR: column "updatedAt" is of type timestamp without time zone but expression is of type text. Hint: You will need to rewrite or cast the expression.
SO clearly, the query generated by pg-promise is not working.
Environment
Version of pg-promise: 11.2.0
OS type (Linux/Windows/Mac): Linux
Version of Node.js: v16.19.0
Link of gihub issue I created - https://github.com/vitaly-t/pg-promise/issues/866
Using "aws-sdk": "^2.1063.0" and nodejs 12
Inside my lambda I am doing an update to a dynamodb table.
My table has a Primary key: JobUID type string and a Sort key type string.
My parameters look like this:
var params = {
TableName: tableName,
Key: {
"JobUID": payload.JobUID,
"TimeStamp": payload.TimeStamp
},
UpdateExpression:
"set #HasResponse = :v_HasResponse, #ResponseTimeStamp = :v_ResponseTimeStamp, #Recommendation = :v_Recommendation, #ThreadRepComment = :v_ThreadRepComment",
ExpressionAttributeNames: {
"#HasResponse": payload.HasResponse,
"#ResponseTimeStamp": payload.ResponseTimeStamp,
"#Recommendation": payload.Recommendation,
"#ThreadRepComment": payload.ThreadRepComment,
},
ExpressionAttributeValues: {
":v_HasResponse": payload.HasResponse,
":v_ResponseTimeStamp": payload.ResponseTimeStamp,
":v_Recommendation": payload.Recommendation,
":v_ThreadRepComment": payload.ThreadRepComment,
},
// returns only the affected attributes, as they appeared after the update
ReturnValues: "UPDATED_NEW"
};
I have printed out the payload.JobUID and payload.TimeStamp in the log so I know they are what expect.
The latest row in the table has JobUID and TimeStamp exactly as I printed them out.
I want to update the 4 properties in the expression attribute names.
I am getting the error "ValidationException: The provided key element does not match the schema"
I have looked on the web and in SOF at examples of updates and I cannot seem to get this to work.
what is wrong with my key values.
The update call looks like this. Super simple
var returnValue = await dynamo.update(params).promise();
I also tried
Key: {
JobUID: {"S": payload.JobUID},
TimeStamp: {"S":payload.TimeStamp}
},
So this is what I found works:
var params = {
TableName: tableName,
Key: {
JobUID: payload.JobUID,
TimeStamp: payload.TimeStamp
},
UpdateExpression:
"set HasResponse = :v_HasResponse, ResponseTimeStamp = :v_ResponseTimeStamp, Recommendation = :v_Recommendation, ThreadRepComment = :v_ThreadRepComment",
ExpressionAttributeValues: {
":v_HasResponse": payload.HasResponse,
":v_ResponseTimeStamp": payload.ResponseTimeStamp,
":v_Recommendation": payload.Recommendation,
":v_ThreadRepComment": payload.ThreadRepComment || "",
},
// returns only the affected attributes, as they appeared after the update
ReturnValues: "UPDATED_NEW"
};
var returnValue = await dynamo.update(params).promise();
If there is a property is is null or empty, in my case the ThreadRepComment could be null or empty so you need to handle that.
I'm using Dynamoose to simplify my interactions with DynamoDB in a node.js application. I'm trying to write a query using Dynamoose's Model.query function that will search a table using an index, but it seems like Dynamoose is not including all of the info required to process the query and I'm not sure what I'm doing wrong.
Here's what the schema looks like:
const UserSchema = new dynamoose.Schema({
"user_id": {
"hashKey": true,
"type": String
},
"email": {
"type": String,
"index": {
"global": true,
"name": "email-index"
}
},
"first_name": {
"type": String,
"index": {
"global": true,
"name": "first_name-index"
}
},
"last_name": {
"type": String,
"index": {
"global": true,
"name": "last_name-index"
}
}
)
module.exports = dynamoose.model(config.usersTable, UserSchema)
I'd like to be able to search for users by their email address, so I'm writing a query that looks like this:
Users.query("email").contains(query.email)
.using("email-index")
.all()
.exec()
.then( results => {
res.status(200).json(results)
}).catch( err => {
res.status(500).send("Error searching for users: " + err)
})
I have a global secondary index defined for the email field:
When I try to execute this query, I'm getting the following error:
Error searching for users: ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.
Using the Dynamoose debugging output, I can see that the query winds up looking like this:
aws:dynamodb:query:request - {
"FilterExpression": "contains (#a0, :v0)",
"ExpressionAttributeNames": {
"#a0": "email"
},
"ExpressionAttributeValues": {
":v0": {
"S": "mel"
}
},
"TableName": "user_qa",
"IndexName": "email-index"
}
I note that the actual query sent to DynamoDB does not contain KeyConditions or KeyConditionExpression, as the error message indicates. What am I doing wrong that prevents this query from being written correctly such that it executes the query against the global secondary index I've added for this table?
As it turns out, calls like .contains(text) are used as filters, not query parameters. DynamoDB can't figure out if the text in the index contains the text I'm searching for without looking at every single record, which is a scan, not a query. So it doesn't make sense to try to use .contains(text) in this context, even though it's possible to call it in a chain like the one I constructed. What I ultimately needed to do to make this work is turn my call into a table scan with the .contains(text) filter:
Users.scan({ email: { contains: query.email }}).all().exec().then( ... )
I am not familiar with Dynamoose too much but the following code below will do an update on a record using node.JS and DynamoDB. See the key parameter I have below; by the error message you got it seems you are missing this.
To my knowledge, you must specify a key for an UPDATE request. You can checks the AWS DynamoDB docs to confirm.
var params = {
TableName: table,
Key: {
"id": customerID,
},
UpdateExpression: "set customer_name= :s, customer_address= :p, customer_phone= :u, end_date = :u",
ExpressionAttributeValues: {
":s": customer_name,
":p": customer_address,
":u": customer_phone
},
ReturnValues: "UPDATED_NEW"
};
await docClient.update(params).promise();
Sample test data:
"suppliers" : [
{
"supplierName": "xxx supplier"
},
{
"supplierName": "zzz supplier"
}
]
Excerpt of UpdateItem Params:
ExpressionAttributeValues:{
":sA" : {L: [event.suppliers]}
// ":sA" : {L: event.suppliers}
}
I encounter "UnexpectedParameter: Unexpected key '0' found in params". I tried changing the formatting of the AttributeValues but I get other errors like "UnexpectedParameter: Unexpected key 'supplierName' found in params".
I need help properly defining a list AttributeValue. Note that the input sample test data may contain x number of supplierName objects and hence defining the keys (ie. '0', '1', etc) in the AttributeValue is also not an option. I also prefer not to use the documentClient version of UpdateItem.
ExpressionAttributeValues is about to set a value to the parameter of the update expression, for example:
dynamoDb.update({
TableName: ...,
Key: ...,
UpdateExpression: 'set suppliers = :suppliers',
ExpressionAttributeValues: {':suppliers': event.suppliers}
}
You have to provide the key to the record and then you can change the suppliers attribute.
I have parse server which I have this query on
const empList = Parse.Object.extend("EmpList");
const query = new Parse.Query(empList);
query.equalTo("relation", Parse.User.current());
query.find({
success: (results) => {
// results.map((each)=>this.data = each.id)
this.data = results
},
Parse server has 2 classes ( users and EmpList), there is a pointer in EmpList which points correctly to the current user submitted the employee.The returned data also includes the pointer ( which has a name "relation" ) and I can see the username exists as an object in this fashion
Array [
Object {
"createdAt" "2018-07-05T20 17 45.173Z",
"name" "1the latter",
"objectId" "D2kThKcg9z",
"phone" "111",
"relation" Object {
"ACL" Object {
"*" Object {
"read" true,
},
"H2rQJxNyTD" Object {
"read" true,
"write" true,
},
},
"__type" "Object",
"className" "_User",
"createdAt" "2018-07-05T18 43 40.536Z",
"objectId" "H2rQJxNyTD",
"password" undefined,
"sessionToken" "r c45063a034dd81d646bef51ae2055c85",
"updatedAt" "2018-07-05T20 17 35.976Z",
"username" "1",
},
"shift" "tue",
"updatedAt" "2018-07-05T20 22 11.158Z",
},
]
Yet I am unable to extract the username in relation object.
please help me do so by data[0].relation.username or data[0].relation().username
There are two things you can do. The simplest is to eagerly fetch the related object as part of the query using include() ...
const query = new Parse.Query(empList);
query.equalTo("relation", Parse.User.current());
query.include("relation");
(Incidentally, "relation" isn't a great name for that column. A better choice would be something relating to it's meaning, like submittedByUser. Calling it relation is like naming your poodle "Poodle").
The drawback of using include is that it will eagerly fetch all of the related objects on the query, making the query take longer and potentially produce data that you don't need. If you only want the related object on one or a handful of the query results, skip the include() and query the relations individually...
const query = new Parse.Query(empList);
query.equalTo("relation", Parse.User.current());
query.find({
success: results => {
// for one or some of the results...
let submittedByUserRelation = user.relation("relation");
submittedByUserRelation.query().find({
success: user => {
// user.username will be the username
}
});
Check the query() function on the relation that you can use to get all the object in the relation .