We Keep Coding

DynamoDB query by contains one of the values from array for Node.js

Is it possible to query with an array [apple, orange] on category (also an array. showing below) and get data contains values of either apple or orange?
row1 | "category": [apple, orange, banana, watermelon]
row2 | "category": [banana, watermelon]
row3 | "category": [orange, watermelon]
I expect to get row1, row3 as a result that contains either apple or orange.

You can use CONTAINS function to check for values in the list. However, if you want to check multiple values, you need to use logical OR condition for each value.
CONTAINS is supported for lists: When evaluating "a CONTAINS b", "a"
can be a list; however, "b" cannot be a set, a map, or a list.
Example:-
var params = {
TableName : 'tableName',
FilterExpression: "contains (category, :category1) OR contains (category, :category2)",
ExpressionAttributeValues : {
':category1' : "apple",
':category2' : "orange"
}
};
Note:-
I assumed you are going to use Scan API. If you are using Query API, please include the KeyConditionExpression.

The question asked about Query, but it seems that it is unclear whether on Scan is the technology being used. I will answer for each of those parts.
Scan is very expensive and where possible Query should be used. Scan easily supports using contains to do what you asked.
await db.scan({
TableName : 'tableName',
FilterExpression: "contains (category, :category1) OR contains (category, :category2)",
ExpressionAttributeValues : {
':category1' : "apple",
':category2' : "orange"
}
}).promise();
Given that Scan is inefficient, it would be better to have a strategy where the rows it the table can be returned using the Query. Query does NOT support the Contains syntax and additionally must run only a single HASH value. Two different rows with different HASH values cannot be returned in the same Query. Assuming this criteria is met, then a custom index would need to be constructed. There are a couple of ways to do this, depending on the number of categories that you have and if multiple categories can be assigned to reach row. For 1 cat per row, a single indexed column works. Otherwise, we'll need to construct a indexed column that is able to use the one of the other operators.
That column would need to contain a composite value for all the categories of the row, such that f(column_value, category_value_check) = true for all possible categories. In order to construct such a composite value, Amazon recommends using a Z-Index.

Can't figure out MYSQL query to get the result I want

I have a table called subcategories with columns 'id' and 'name' and a table called goals with columns 'id', 'name' and foreign key 'subcategory_id'.
I want a query that results in an array of subcategory objects, which has a property 'goals' which is an array of goal objects.
Too give an example of how the result would look In JS code:
result = [
{id: 1, name: "name", goals: [{id: 1, name: "goalName"}, {...}, {...}]},
{...},
{...}
]
But (with a different syntax) the result would be the same for other languages..
Thusfar I tried to do this with left-join, like this:
SELECT sc.ID as subcatId, sc.name as subcatName, g.ID as ID, g.name as name
FROM needs_subcategories as sc
LEFT JOIN needs_goals as g
ON sc.ID=g.subcategory_id
But the goals aren't grouped under a single subcategory.. I feel like it should be possible to do with a query, but I can't figure out/google how to do it because I wouldn't know how to phrase the question due to my lack of SQL knowledge..
Hope you guys can help me!
Thanks in advance.

You won't be able to acheive that with a query. MySQL can't do that.
You are currently fetching all goals, each one with their subcategory (subcategories will repeat).
You can convert it to the desired array with some code (example in php, you can translate this to any other language).
$result=array();
$lastSubcatId=null;
$goals=array();
while($row=$query->fetch_object()) { //assuming $query is the resultset
if($lastSubcatId&&$lastSubcatId!=$row->subcatId) {
$row->goals=$goals;
$result[]=$row; //or you could assign each desired property
$goals=array();
}
$goals[]=$row; //or you could assign each desired property
}
//surely, there are items left in $goals
if($lastSubcatId) {
$row->goals=$goals;
$result[]=$row; //or you could assign each desired property
}
But a more efficient way would be, I think, with multiple queries:
$result=array();
$subcats=$db->query("SELECT * FROM needs_subcategories");
while($subcat=$subcats->fetch_object()) {
//you might want to use prepared statements, I'm just simplifying
//it will not only be safer, but reusing the prepared statement will increase the performance considerably
$goals=$db->query("select * from needs_goals where subcategory_id=".$subcat->ID);
$temp=array();
while($goal=$goals->fetch_object()) $temp[]=$goal;
$subcat->goals=$temp;
$result[]=$subcat;
}

In the end I solved this using groupBy as #tadman suggested in his comment.
I created a function (based on the information in this answer) that looks like this:
function processResults(collection, groupKey) {
var result = _.chain(collection)
.groupBy(groupKey)
.toPairs()
.map(function (currentItem) {
// 'text' and 'children' are the keys I want in my resulting object
// children being the property that contains the array of goal objects
return _.zipObject(['text', 'children'], currentItem);
})
.value();
return result;
}
Which results in the array of objects with grouped goals! As I structured the function now (with hard-coded key names) it only works for my specific case, if you want to generalize the function you could add parameters amd replace the hard-coded key names with those.

CouchDB query issues

I will start off by saying while I am not new to CouchDB, I am new to querying the views using JavaScript and the web.
I have looked at multiple other questions on here, including CouchDB - Queries with params, couchDB queries, Couchdb query with AND operator, CouchDB Querying Dates, and Basic CouchDB Queries, just to list a few.
While all have good information in them, I haven't found one that has my particular problem in it.
I have a view set up like so:
function (docu) {
if(docu.status && docu.doc && docu.orgId.toString() && !docu.deleted){
switch(docu.status){
case "BASE":
emit(docu.name, docu);
break;
case "AIR":
emit(docu.eta, docu);
break;
case "CHECK":
emit(docu.checkTime, docu);
break;
}
}
}
with all documents having a status, doc, orgId, deleted, name, eta, and checkTime. (I changed doc to docu because of my custom doc key.
I am trying to query and emit based on a set of keys, status, doc, orgId, where orgId is an integer.
My jQuery to do this looks like so:
$.couch.db("myDB").view("designDoc/viewName", {
keys : ["status","doc",orgId],
success: function(data) {
console.log(data);
},
error: function(status) {
console.log(status);
}
});
I receive
{"total_rows":59,"offset":59,"rows":[
]}
Sometimes the offset is 0, sometimes it is 59. I feel I must be doing something wrong for this not to be working correctly.
So for my questions:
I did not mention this, but I had to set docu.orgId.toString() because I guess it parses the URL as a string, is there a way to use this number as a numeric value?
How do I correctly view multiple documents based on multiple keys, i.e. if(key1 && key2) emit(doc.name, doc)
Am I doing something obviously wrong that I lack the knowledge to notice?
Thank you all.

You're so very close. To answer your questions
When you're using docu.orgId.toString() in that if-statement you're basically saying: this value must be truthy. If you didn't convert to string, any number, other than 0, would be true. Since you are converting to a string, any value other than an empty string will be true. Also, since you do not use orgId as the first argument in an emit call, at least not in the example above, you cannot query by it at all.
I'll get to this.
A little.
The thing to remember is emit creates a key-value table (that's really all a view is) that you can use to query. Let's say we have the following documents
{type:'student', dept:'psych', name:'josh'},
{type:'student', dept:'compsci', name:'anish'},
{type:'professor', dept:'compsci', name:'kender'},
{type:'professor', dept:'psych', name:'josh'},
{type:'mascot', name:'owly'}
Now let's say we know that for this one view, we want to query 1) everything but mascots, 2) we want to query by type, dept, and name, all of the available fields in this example. We would write a map function like this:
function(doc) {
if (doc.type === 'mascot') { return; } // don't do anything
// allow for queries by type
emit(doc.type, null); // the use of null is explained below
// allow queries by dept
emit(doc.dept, null);
// allow for queries by name
emit(doc.name, null);
}
Then, we would query like this:
// look for all joshs
$.couch.db("myDB").view("designDoc/viewName", {
keys : ["josh"],
// ...
});
// look for everyone in the psych department
$.couch.db("myDB").view("designDoc/viewName", {
keys : ["psych"],
// ...
});
// look for everyone that's a professor and everyone named josh
$.couch.db("myDB").view("designDoc/viewName", {
keys : ["professor", "josh"],
// ...
});
Notice the last query isn't and in the sense of a logical conjunction, it's in the sense of a union. If you wanted to restrict what was returned to documents that were only professors and also joshs, there are a few options. The most basic would be to concatenate the key when you emit. Like
emit('type-' + doc.type + '_name-' + doc.name, null);
You would then query like this: key : ["type-professor_name-josh"]
It doesn't feel very proper to rely on strings like this, at least it didn't to me when I first started doing it, but it is a quite common method for querying key-value stores. The characters - and _ have no special meaning in this example, I simply use them as delimiters.
Another option would be what you mentioned in your comment, to emit an array like
emit([ doc.type, doc.name ], null);
Then you would query like
key: ["professor", "josh"]
This is perfectly fine, but generally, the use case for emitting arrays as keys, is for aggregating returned rows. For example, you could emit([year, month, day]) and if you had a simple reduce function that basically passed the records through:
function(keys, values, rereduce) {
if (rereduce) {
return [].concat.apply([], values);
} else {
return values;
}
}
You could query with the url parameter group_level set to 1 or 2 and start querying by year and month or just year on the exact same view using arrays as keys. Compared to SQL or Mongo it's mad complicated and convoluted, but hey, it's there.
The use of null in the view is really for resource saving. When you query a view, the rows contain an _id that you can use in a second ajax call to get all the documents from, for example, _all_docs.
I hope that makes sense. If you need any clarification you can use the comments and I'll try my best.

A filtered and sorted view in CouchDB?

I have some items in CouchDB with a timestamp field and a group field. I use this view function to get a list of all items where group == "foo":
{
"map" : "function(doc) { emit(doc.group, doc); }"
}
http://localhost:5984/my_database/_temp_view?key="foo"
Now I'd like to make the output sorted by the timestamp field. How do I do that? Basically, I want the equivalent of this SQL query:
SELECT * FROM SomeTable WHERE group=? ORDER BY timestamp

Emit the timestamp as a second column:
function(doc) { emit([doc.group,doc.timestamp]); }
Then, query with the following parameters:
view?startkey=["foo"]&endkey=["foo",""]
I am assuming that your timestamps are numbers, not strings. Read this to understand how the numeric timestamps will be sorted in-between ["foo"] and ["foo",""].
Also, don't emit doc as the value, it uses a lot of storage. If you really need the document, use include_docs=true in your query.

Algorithm for data filter

Can you suggest me an algorithm for filtering out data.
I am using javascript and trying to write out a filter function which filters an array of data.I have an array of data and an array of filters, so in order to apply each filter on every data, I have written 2 for loops
foreach(data)
{
foreach(filter)
{
check data with filter
}
}
this is not the proper code, but in short that what my function does, the problem is this takes a huge amount of time, can someone suggest a better method.
I am using the Mootools library and the array of data is JSON array
Details of data and Filter
Data is JSON array of lets say user, so it will be
data = [{"name" : "first", "email" : "first#first", "age" : "20"}.
{"name" : "second", "email" : "second#second", "age" : "21"}
{"name" : "third", "email" : "third#third", "age" : "22"}]
Array of filters is basically self define class for different fields of data
alFilter[0] = filterName;
alFilter[1] = filterEmail;
alFilter[2] = filterAge;
So when I enter the first for loop, I get a single JSON opbject (first row) in the above case.
When I enter the second for loop (filters loop) I have a filter class which extracts the exact field on which the current filter would work and check the filter with the appropriate field of the data.
So in my example
foreach(data)
{
foreach(filter)
{
//loop one - filter name
// loop two - filter email
// loop three - filter age
}
}
when the second loop ends i set a flag denoting if the data has been filtered or not and depending on it the data is displayed.

You're going to have to give us some more detail about the exact structure of your data and filters to really be able to help you out. Are the filters being used to select a subset of data, or to modify the data? What are the filters doing?
That said, there are a few general suggestions:
Do less work. Is there some way you can limit the amount of data you're working on? Some pre-filter that can run quickly and cut it down before you do your main loop?
Break out of the inner loop as soon as possible. If one of the filters rejects a datum, then break out of the inner loop and move on to the next datum. If this is possible, then you should also try to make the most selective filters come first. (This is assuming that your filters are being used to reject items out of the list, rather than modify them)
Check for redundancy in the computation the filters perform. If each of them performs some complicated calculations that share some subroutines, then perhaps memoization or dynamic programming may be used to avoid redundant computation.
Really, it all boils down to the first point, do less work, at all three levels of your code. Can you do less work by limiting the items in the outer loop? Do less work by stopping after a particular filter and doing the most selective filters first? Do less work by not doing any redundant computation inside of each filter?

That's pretty much how you should do it. The trick is to optimize that "check data with filter"-part. You need to traverse all your data and check against all your filters - you'll not going to get any faster than that.
Avoid string comparisons, use data models as native as possible, try to reduce the data set on each pass with filter, etc.
Without further knowledge, it's hard to optimize this for you.

You should sort the application of your filters, so that two things are optimized: expensive checks should come last, and checks that eliminate a lot of data should come first. Then, you should make sure that checking is cut short as soon as an "out" result occurs.

If your filters are looking for specific values, a range, or start of a text then jOrder (http://github.com/danstocker/jorder) will fit your problem.
All you need to do is create a jOrder table like this:
var table = jOrder(data)
.index('name', ['name'], { grouped: true, ordered: true })
.index('email', ['email'])
.index('age', ['age'], { grouped: true, ordered: true, type: jOrder.number });
And then call table.where() to filter the table.
When you're looking for exact matches:
filtered = table.where([{name: 'first'}, {name: 'second'}]);
When you're looking for a certain range of one field:
filtered = table.where([{age: {lower: 20, upper: 21}}], {mode: jOrder.range});
Or, when you're looking for values starting with a given string:
filtered = table.where([{name: 'fir'}], {mode: jOrder.startof});
Filtering will be magnitudes faster this way than with nested loops.

Supposing that a filter removes the data if it doesn't match, I suggest, that you switch the two loops like so:
foreach(filter) {
foreach(data) {
check data with filter
}
}
By doing so, the second filter doesn't have to work all data, but only the data that passed the first filter, and so on. Of course the tips above (like doing expensive checks last) are still true and should additionally be considered.