There is Dynamo table with fields:
email (primary)
tenant
other stuff
I want to get all the items where email contains 'mike'
In my nodejs server, I have this code
const TableName= 'UserTable';
const db = new aws.DynamoDB();
const email = 'mike.green#abc.com'
params = {
TableName: userTableName,
KeyConditionExpression: '#email = :email',
ExpressionAttributeNames: {
'#email': 'email',
},
ExpressionAttributeValues: {
':email': { S: email },
},
};
db.query(params, (err, data) => {
if (err) {
reject(err);
} else {
const processedItems = [...data.Items].sort((a, b) => a.email < b.email ? -1 : 1);
const processedData = { ...data, Items: processedItems };
resolve(processedData);
}
this works ^^ only if I search entire email mike.green#abc.com
Question 1 -
But, if i want to search mike, and return all items where email contains mike, How can i get that?
Question 2
If I want to get all the rows where email contains mike and tenant is Canada. How can i get that?
I'm not a NodeJS user but hope it will be helpful.
Question 1 - But, if i want to search mike, and return all items where
email contains mike, How can i get that?
Key expressions are reserved to equality constraints. If you want to have more querying flexibility, you need to use a filter expression. Please notice that you won't be able to use filter expression on your partition key. You can find more information on https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html but the most important is:
Key Condition Expression
To specify the search criteria, you use a key condition expression—a
string that determines the items to be read from the table or index.
You must specify the partition key name and value as an equality
condition.
You can optionally provide a second condition for the sort key (if
present). The sort key condition must use one of the following
comparison operators:
a = b — true if the attribute a is equal to the value b
a < b — true if a is less than b
a <= b — true if a is less than or equal to b
a > b — true if a is greater than b
a >= b — true if a is greater than or equal to b
a BETWEEN b AND c — true if a is greater than or equal to b, and less than or equal to c.
The following function is also supported:
begins_with (a, substr)— true if the value of attribute a begins with a particular substring.
......
Question 2 If I want to get all the rows where email contains mike and
tenant is Canada. How can i get that?
You can use a filter expression to do that and use one of available functions https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html#Expressions.OperatorsAndFunctions.Syntax. A filter expression is:
If you need to further refine the Query results, you can optionally
provide a filter expression. A filter expression determines which
items within the Query results should be returned to you. All of the
other results are discarded.
A filter expression is applied after a Query finishes, but before the
results are returned. Therefore, a Query will consume the same amount
of read capacity, regardless of whether a filter expression is
present.
A Query operation can retrieve a maximum of 1 MB of data. This limit
applies before the filter expression is evaluated.
A filter expression cannot contain partition key or sort key
attributes. You need to specify those attributes in the key condition
expression, not the filter expression.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html
To wrap-up:
if e-mail is your partition key, you cannot apply contains on it - you have to query it directly.
eventually you can do a scan over your table and apply filter on it (https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) but I wouldn't do that because of consumed capacity of the table and response time. Scan involves operating over all rows in the table, so if you have kind of hundreds of GB, you will likely not get the information in real-time. And real-time serving is one of purposes of DynamoDB.
Related
I am trying to sort my dynamo db results by the sort and key and also get a range of values based on the sort key and i keep getting invalid key condition expression error. basically im trying to get a range of values based on my sort key and also sort them.
example of saved db item:
{
id: ORDER_287d6df3-bd9f-472d-87ef-ee9b810b5874
issued_at: 2021-05-07T15:09:11.894Z
base_cur: NGN
orderRate(GSI): SELL#500
quantity: 300
quantityRemaining: 300
quantityRemoved: 0
quote_cur: GBP
rate: 500
side: SELL
ticker(GSI): GBP#NGN
}
what i have tried:
const params = {
TableName: process.env.ORDERS_TABLE_NAME,
IndexName: 'ticker_orderRate',
KeyConditionExpression: `ticker = :ticker and orderRate between :maxRate and :baseRate`,
ExpressionAttributeValues: {
':ticker': `${quote_cur}#${base_cur}`,
':maxRate': `${side === 'BUY' ? 'SELL' : 'BUY'}#${rate}`,
':baseRate': `${side === 'BUY' ? 'SELL' : 'BUY'}#0.00`,
}
};
the rate is a number that comes from an SNS topic.
Error message:
ERROR: ValidationException: Invalid KeyConditionExpression: The BETWEEN operator requires upper bound to be greater than or equal to lower bound; lower bound operand: AttributeValue: {S:BUY#620}, upper bound operand: AttributeValue: {S:BUY#0.00}
is there a way i can transform the rate and the base rate to numbers while querying the database? and also how do i sort by ascending order?
Thanks for the help
The issue is in your BETWEEN clause. As the erorr message states, the upper bound (the second value) must be greater than or equal to the lower bound (the first value).
There are a few things to consider here.
Remember, this is a string that you are sorting, not a number. You'll need to make some adjustments to handle that.
You need to make sure the length of the number is always the same. This can be accomplished by zero-padding the value (e.g. 1.23 becomes 001.23). Make sure you zero pad to a value that will always be as long or longer than the biggest number you expect.
If you support negative values then you need to be explicit on positive values too. Include a plus (+) on positive numbers.
Dynamically adjust your upper and lower bounds to be in the right order. If the first value is less than the second leave them, if the first value is greater than the second then swap their order.
Key Condition Expression
I'm learning Dynamodb and for that I installed the local server that comes with a shell at http://localhost:8000/shell
now.. I created the following table:
var serverUpTimeTableName = 'bingodrive_server_uptime';
var eventUpTimeColumn = 'up_time';
var params = {
TableName: serverUpTimeTableName,
KeySchema: [ // The type of of schema. Must start with a HASH type, with an optional second RANGE.
{ // Required HASH type attribute
AttributeName: eventUpTimeColumn,
KeyType: 'HASH',
},
],
AttributeDefinitions: [ // The names and types of all primary and index key attributes only
{
AttributeName: eventUpTimeColumn,
AttributeType: 'N', // (S | N | B) for string, number, binary
},
],
ProvisionedThroughput: { // required provisioned throughput for the table
ReadCapacityUnits: 2,
WriteCapacityUnits: 2,
}
};
dynamodb.createTable(params, callback);
so I created a table only with one hash key called up_time, that's actually the only item in the table.
Now I want to fetch the last 10 inserted up times.
so far I created the following code:
var serverUpTimeTableName = 'bingodrive_server_uptime';
var eventUpTimeColumn = 'up_time';
var params = {
TableName: serverUpTimeTableName,
KeyConditionExpression: eventUpTimeColumn + ' != :value',
ExpressionAttributeValues: {
':value':0
},
Limit: 10,
ScanIndexForward: false
}
docClient.query(params, function(err, data) {
if (err) ppJson(err); // an error occurred
else ppJson(data); // successful response
});
ok.. so few things to notice:
I don't really need a KeyCondition. i just want the last 10 items, so I used Limit 10 for the limit and ScanIndexForward:false for reverse order.
!= or NE are not supported in key expressions for hash keys. and it seems that I must use some kind of index in the query.. confused about that.
so.. any information regarding the issue would be greatly appreciated.
Some modern terminology: Hash is now called Partition, Range is now called Sort.
Thank you Amazon.
You need to understand that Query-ing is an action on hash-keys. In order to initiate a query you must supply a hash-key. Since your table's primary key is only hash key (and not hash+range) you can't query it. You can only Scan it in order to find items. Scan doesn't require any knowledge about items in the table.
Moving on.. when you say "last 10 items" you actually do want a condition because you are filtering on the date attribute, you haven't defined any index so you can't have the engine provide you 10 results. If it were a range key element, you could get the Top-10 ordered elements by querying with a backwards index (ScanIndexForward:false) - again, not your schema.
In your current table - what exactly are you trying to do? You currently only have one attribute which is also the hash key so 10 items would look like (No order, no duplicates):
12312
53453
34234
123
534534
3101
11
You could move those to range key and have a global hash-key "stub" just to initiate the query you're making but that breaks the guidelines of DynamoDB as you have a hot partition and it won't have the best performance. Not sure this bothers you at the moment, but it is worth mentioning.
New to DynamoDB and need to do the above query, but not sure how. Here is what I'm trying currently, and I'm getting the error below.
Btw, I am using this javascript library w/ DynamoDB: https://github.com/awslabs/dynamodb-document-js-sdk
var ids = ['12313','12312313','12312313'];
var params = {
TableName: 'apps',
IndexName: 'state-app_id-index',
KeyConditions: [
DynamoDB.Condition("state", "EQ", "active"),
DynamoDB.Condition("id", "IN", ids)
]
};
DynamoDB.query(params, function(error, response) {});
The error I am getting is as follows:
ValidationException: One or more parameter values were invalid: ComparisonOperator IN is not valid for L AttributeValue type
KeyConditions does not support the IN operator. The documentation for KeyCondition says what operators it does support:
For KeyConditions, only the following comparison operators are
supported:
EQ | LE | LT | GE | GT | BEGINS_WITH | BETWEEN
The EQ operator only works for a single value as well:
EQ : Equal.
AttributeValueList can contain only one AttributeValue of type String, Number, or Binary (not a set type). If an item contains an AttributeValue element of a different type than the one specified in the request, the value does not match. For example, {"S":"6"} does not equal {"N":"6"}. Also, {"N":"6"} does not equal {"NS":["6", "2", "1"]}.
The restrictions are basically the same for KeyConditionExpression, which is the newer, recommended way for filtering on keys. Here is a snippet from the documentation (emphasis mine):
The condition must perform an equality test on a single hash key
value. The condition can also perform one of several comparison tests
on a single range key value. Query can use KeyConditionExpression to
retrieve one item with a given hash and range key value, or several
items that have the same hash key value but different range key values
In your case, you could build out the FilterExpression in a similar way as described in this answer.
The only way to use the IN statement is by using a filter Condition.
However Filter expression can only contain non-primary key attributes.
Thus the query you want to achieve is NOT possible with primary keys.
Something like this
var params = {
TableName: 'apps',
...
FilterExpression: "#id IN (:one,:two)",:
...
Is only possible with non-primary key attributes
The workaround you can apply is using the batch get item
Therefore instead of issuing one query issue multiples in one call and each condition shall contain a value from your ids array.
When it comes to batch get items be aware of the read capacity units (batch get item in node.js).
I'm using YDN-DB (an abstraction on top of IndexedDB) as a local database. I have an object store called 'conversations', and in that store, there's an index called 'participants' where there is a string containing id's for different users in the conversation. For example:
Example Conversation #1:
id: 1234343434353456,
participants: '171e66ca-207f-4ba9-8197-d1dac32499db,82be80e2-2831-4f7d-a8d7-9223a2d4d511'
Example Conversation #2:
id: 4321343434356543,
participants: 'd7fa26b3-4ecc-4f84-9271-e15843fcc83f,171e66ca-207f-4ba9-8197-d1dac32499db'
To try to perform a partial match on an index, I tried using ydn-db-fulltext as a solution. The full text catalog looks like this:
{
name: 'participants',
lang: 'en',
sources: [
{
storeName: 'conversations',
keyPath: 'participants',
weight: 1
}
]
}
I see that the catalog is generated, but there seems to be a problem doing exact matches. For example, if I query using only part of the key in the participants index, I get back a primary key from the catalog:
db.search('participants', 'd7fa26b3').done(function(results) {
if(results.length == 0) console.debug('No results found...');
console.debug(results); // there is 1 object here!
var primaryKey = results[0].primaryKey; // primaryKey exists!
});
However, when using any value past the '-', the search request returns 0 results:
db.search('participants', 'd7fa26b3-4ecc-4f84-9271-e15843fcc83f').done(function(results) {
if(results.length == 0) console.debug('No results found...');
console.debug(results); // there are 0 objects in the array
var primaryKey = results[0].primaryKey; // primaryKey throws undefined since there are 0 results!
});
This makes sense, when reading the documentation, in that '-' and '*' are reserved characters that remove a phrase and match a prefix respectively:
Query format is free text, in which implicit and/or/near logic operator apply for each token. Use double quote for exact match, - to subtract from the result and * for prefix search.
I tried putting double quotes inside the single quotes, using only double quotes, and also escaping all of the '-' characters with a backslash, but none of these seem to work.
So the question is how does one perform a match in an index where the string contains '-' characters?
Have you try db.search('participants', '"d7fa26b3"').
BTW, you are using full text search that is not suppose to do. You have to tokenize your string and index them manually.
If you store the participants field of your object as an array, then you can use the multi-entry flag to the createIndex method called on the participants field, and probably do what you want.
The number of items in the participants property of the object is mutable. When you update an object in the store and it has a different number of items in the partic property, then the index is automatically updated as a result (just like any other index). If you add an item to the prop, then restore (put/override/cursor.update) the object in the store, the index updates.
It helps to review the basics of how a multi-entry index works. You can do this with vanilla js, without a framework, and certainly without full-text searching.
How to Sort Collection Fields by Name & Letter in Meteor JS? means
A Collection contains 3 fields , they are fname,id & email.
Sorting based on fname or fname letters for ex:fname = xyz , sort using x means x is a letter of fname.
Please see the below code and suggest me what to do for above?
Code :
Template.client.clientname = function ()
{
//here Client is collection name & fname is field.
return Client.find();
//return Client.find({name: {$fname: /k$/}});
};
You use sort and order the keys in the order of your preference. There's more details about this in the docs:
http://docs.mongodb.org/manual/reference/method/cursor.sort/
http://docs.meteor.com/#sortspecifiers
So:
Client.find({}, {sort:{fname: 1, id:1, email: 1}});
1 means in ascending order (A-Z) and -1 in descending order (Z-A).