We Keep Coding

Mongodb E11000 duplicate key error on array field

I have a quick question about mongoose schema real quick. Here is the code: https://i.ibb.co/Db8xPMw/5555.png
I tried to create a document without the property "work". It works in the first time, but it didn't start to work on the second time that I do the same thing again.
Do you have any idea?
Basically I create two documents without an "work" property, which causes a duplicate key error. However, I didn't set up unqiue: true though.
Error :
"errmsg" : "E11000 duplicate key error collection: test.user index work_1 dup key: { : null }

From the message it says your collection has an index with name work_1 probably on field work, Since you've created a document without work field then basically you cannot create another document without work field what so ever in the same collection, cause two documents with no work field or even work field with value as null or same cannot exist as it violates unique constraint policies (it says dup key : { : null}) !! Uniques indexes can be created via mongoose schemas or can also be created by manually running queries on database.
Ref : Search for Unique Index and Missing Field in index-unique
So you need to drop the existing index using dropIndex & then if needed recreate it using createIndex. MongoDB would automatically convert a created index to index-multikey (multi-key index - indexes on array fields) if at least one existing document has array value for that indexed field by the time you create index or even if an array value gets inserted for that field in future.
Through code - Drop index : yourSchema.dropIndex({yourFieldName: 1}) && Create index : yourSchema.index({yourFieldName : 1})
NOTE : Just in case if you want to have certain criteria in unique indexes like situation from this question where indexed field can be missing in some documents but it shouldn't be considered as duplicate insertion, then you can take use of partial-indexes (Search for Partial Index with Unique Constraint) which would only index documents where work field exists.
Ex of partial-indexes :-
db.yourCollectionName.createIndex(
{ work: 1 },
{ unique: true, partialFilterExpression: { work: { $exists: true } } }
)
Ref : mongoose-Indexes

AWS Dynamodb Query - get items from table with condition

There is Dynamo table with fields:
email (primary)
tenant
other stuff
I want to get all the items where email contains 'mike'
In my nodejs server, I have this code
const TableName= 'UserTable';
const db = new aws.DynamoDB();
const email = 'mike.green#abc.com'
params = {
TableName: userTableName,
KeyConditionExpression: '#email = :email',
ExpressionAttributeNames: {
'#email': 'email',
},
ExpressionAttributeValues: {
':email': { S: email },
},
};
db.query(params, (err, data) => {
if (err) {
reject(err);
} else {
const processedItems = [...data.Items].sort((a, b) => a.email < b.email ? -1 : 1);
const processedData = { ...data, Items: processedItems };
resolve(processedData);
}
this works ^^ only if I search entire email mike.green#abc.com
Question 1 -
But, if i want to search mike, and return all items where email contains mike, How can i get that?
Question 2
If I want to get all the rows where email contains mike and tenant is Canada. How can i get that?

I'm not a NodeJS user but hope it will be helpful.
Question 1 - But, if i want to search mike, and return all items where
email contains mike, How can i get that?
Key expressions are reserved to equality constraints. If you want to have more querying flexibility, you need to use a filter expression. Please notice that you won't be able to use filter expression on your partition key. You can find more information on https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html but the most important is:
Key Condition Expression
To specify the search criteria, you use a key condition expression—a
string that determines the items to be read from the table or index.
You must specify the partition key name and value as an equality
condition.
You can optionally provide a second condition for the sort key (if
present). The sort key condition must use one of the following
comparison operators:
a = b — true if the attribute a is equal to the value b
a < b — true if a is less than b
a <= b — true if a is less than or equal to b
a > b — true if a is greater than b
a >= b — true if a is greater than or equal to b
a BETWEEN b AND c — true if a is greater than or equal to b, and less than or equal to c.
The following function is also supported:
begins_with (a, substr)— true if the value of attribute a begins with a particular substring.
......
Question 2 If I want to get all the rows where email contains mike and
tenant is Canada. How can i get that?
You can use a filter expression to do that and use one of available functions https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.OperatorsAndFunctions.html#Expressions.OperatorsAndFunctions.Syntax. A filter expression is:
If you need to further refine the Query results, you can optionally
provide a filter expression. A filter expression determines which
items within the Query results should be returned to you. All of the
other results are discarded.
A filter expression is applied after a Query finishes, but before the
results are returned. Therefore, a Query will consume the same amount
of read capacity, regardless of whether a filter expression is
present.
A Query operation can retrieve a maximum of 1 MB of data. This limit
applies before the filter expression is evaluated.
A filter expression cannot contain partition key or sort key
attributes. You need to specify those attributes in the key condition
expression, not the filter expression.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html
To wrap-up:
if e-mail is your partition key, you cannot apply contains on it - you have to query it directly.
eventually you can do a scan over your table and apply filter on it (https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) but I wouldn't do that because of consumed capacity of the table and response time. Scan involves operating over all rows in the table, so if you have kind of hundreds of GB, you will likely not get the information in real-time. And real-time serving is one of purposes of DynamoDB.

DynamoDB query by contains one of the values from array for Node.js

Is it possible to query with an array [apple, orange] on category (also an array. showing below) and get data contains values of either apple or orange?
row1 | "category": [apple, orange, banana, watermelon]
row2 | "category": [banana, watermelon]
row3 | "category": [orange, watermelon]
I expect to get row1, row3 as a result that contains either apple or orange.

You can use CONTAINS function to check for values in the list. However, if you want to check multiple values, you need to use logical OR condition for each value.
CONTAINS is supported for lists: When evaluating "a CONTAINS b", "a"
can be a list; however, "b" cannot be a set, a map, or a list.
Example:-
var params = {
TableName : 'tableName',
FilterExpression: "contains (category, :category1) OR contains (category, :category2)",
ExpressionAttributeValues : {
':category1' : "apple",
':category2' : "orange"
}
};
Note:-
I assumed you are going to use Scan API. If you are using Query API, please include the KeyConditionExpression.

The question asked about Query, but it seems that it is unclear whether on Scan is the technology being used. I will answer for each of those parts.
Scan is very expensive and where possible Query should be used. Scan easily supports using contains to do what you asked.
await db.scan({
TableName : 'tableName',
FilterExpression: "contains (category, :category1) OR contains (category, :category2)",
ExpressionAttributeValues : {
':category1' : "apple",
':category2' : "orange"
}
}).promise();
Given that Scan is inefficient, it would be better to have a strategy where the rows it the table can be returned using the Query. Query does NOT support the Contains syntax and additionally must run only a single HASH value. Two different rows with different HASH values cannot be returned in the same Query. Assuming this criteria is met, then a custom index would need to be constructed. There are a couple of ways to do this, depending on the number of categories that you have and if multiple categories can be assigned to reach row. For 1 cat per row, a single indexed column works. Otherwise, we'll need to construct a indexed column that is able to use the one of the other operators.
That column would need to contain a composite value for all the categories of the row, such that f(column_value, category_value_check) = true for all possible categories. In order to construct such a composite value, Amazon recommends using a Z-Index.

Is it possible to sort indexedDB result set without temp array i.e. by using db queries

Is it possible to sort the result set produced by indexedDB without using push
operation in the temporary array.So that no manual looping is involved.

In some cases this is possible using indexes. For example, if you have records of the form {team_id: 123, name: 'alice'} and wanted to select by team_id then sort by name you could construct an index on [team_id, name], and call openCursor() (or getAll() if supported) with a key range IDBKeyRange.bound([id], [id+1], false, true) to get all records with team_id == id and sorted by name.
If you can't construct an index for this - e.g. if the sort comparison function you want doesn't match IDB key ordering, or if your selection can't be expressed simply as a range - then you'll need to use a temp array.

How do I select a record by matching an index based on a partial string that contains '-' characters?

I'm using YDN-DB (an abstraction on top of IndexedDB) as a local database. I have an object store called 'conversations', and in that store, there's an index called 'participants' where there is a string containing id's for different users in the conversation. For example:
Example Conversation #1:
id: 1234343434353456,
participants: '171e66ca-207f-4ba9-8197-d1dac32499db,82be80e2-2831-4f7d-a8d7-9223a2d4d511'
Example Conversation #2:
id: 4321343434356543,
participants: 'd7fa26b3-4ecc-4f84-9271-e15843fcc83f,171e66ca-207f-4ba9-8197-d1dac32499db'
To try to perform a partial match on an index, I tried using ydn-db-fulltext as a solution. The full text catalog looks like this:
{
name: 'participants',
lang: 'en',
sources: [
{
storeName: 'conversations',
keyPath: 'participants',
weight: 1
}
]
}
I see that the catalog is generated, but there seems to be a problem doing exact matches. For example, if I query using only part of the key in the participants index, I get back a primary key from the catalog:
db.search('participants', 'd7fa26b3').done(function(results) {
if(results.length == 0) console.debug('No results found...');
console.debug(results); // there is 1 object here!
var primaryKey = results[0].primaryKey; // primaryKey exists!
});
However, when using any value past the '-', the search request returns 0 results:
db.search('participants', 'd7fa26b3-4ecc-4f84-9271-e15843fcc83f').done(function(results) {
if(results.length == 0) console.debug('No results found...');
console.debug(results); // there are 0 objects in the array
var primaryKey = results[0].primaryKey; // primaryKey throws undefined since there are 0 results!
});
This makes sense, when reading the documentation, in that '-' and '*' are reserved characters that remove a phrase and match a prefix respectively:
Query format is free text, in which implicit and/or/near logic operator apply for each token. Use double quote for exact match, - to subtract from the result and * for prefix search.
I tried putting double quotes inside the single quotes, using only double quotes, and also escaping all of the '-' characters with a backslash, but none of these seem to work.
So the question is how does one perform a match in an index where the string contains '-' characters?

Have you try db.search('participants', '"d7fa26b3"').
BTW, you are using full text search that is not suppose to do. You have to tokenize your string and index them manually.

If you store the participants field of your object as an array, then you can use the multi-entry flag to the createIndex method called on the participants field, and probably do what you want.
The number of items in the participants property of the object is mutable. When you update an object in the store and it has a different number of items in the partic property, then the index is automatically updated as a result (just like any other index). If you add an item to the prop, then restore (put/override/cursor.update) the object in the store, the index updates.
It helps to review the basics of how a multi-entry index works. You can do this with vanilla js, without a framework, and certainly without full-text searching.