Parsing PostgreSQL data into Highcharts chart objects correctly - javascript

I'm trying to fetch timeserie data from PostgreSQL & after successful queries and parsing of data, I have some problem in indexing it. This mistake is probably quite small, but I just cant find it.
After I get data from PostgreSQL, it looks like this:
[
{ id: 2,
time: 2019-09-12T03:36:04.433Z,
value: 0.311303124694538
},
{ id: 2,
time: 2019-09-12T03:36:03.434Z,
value: 0.13233108292117
},
{ id: 3,
time: 2019-09-12T03:36:03.434Z,
value: 0.13233108292117 }
]
After this step I'm reducing data by id:
let results = sqlresult.rows.reduce(function(results, row) {
(results[row.id] = results[row.id] || []).push([row.time,row.value]);
return results;
}, {})
let clonedObj = { ...results };
After this step data is formatted like in below:
{ '2':
[ [ 2019-09-12T03:36:04.433Z, 0.311303124694538 ],
[ 2019-09-12T03:36:03.434Z, 0.13233108292117 ],
[ 2019-09-12T03:36:02.432Z, 0.171794173529729 ]
]
}
But once I'm about to drop it into Highchart it won't work. My problem is probably that I didn't fully understand how does that reduce function work and now I'm trying to copy it. If some of you could show me how to avoid this step and to do all in data reduce step, I'd be thankful.
for(let i=0; i< Object.keys(clonedObj).length; i++){
highchart[i] = {
name: Object.keys(clonedObj)[i],
data: clonedObj[i]
}
}
I'm expecting result like this below:
[{"name":1,"data":[["2019-09-12T03:36:00.433Z",20],["2019-09-12T03:35:38.433Z",-20]]},{"name":2,"data":[["2019-09-12T03:36:04.433Z",0.311303124694538]}]]

From your nicely formatted data listings, it looks like you're using Postgres to package rows of data already. This is something I do all the time, but without some pretty narrow limits. I'd like to get better at this, so I figured I'd give your question a bit of time. To start with, I created a table named "reading" with your data:
CREATE TABLE IF NOT EXISTS reading (
id integer,
"time" text,
"value" real
);
I get back a listing like your top one with this query:
select array_to_json(array_agg(row_to_json(reading_row))) as reading_object
from (select id, time, value from reading) as reading_row
Your target output example doesn't parse right for me, I think you're after this:
[
{
"name":1,
"data":[
[
"2019-09-12T03:36:00.433Z",
20
],
[
"2019-09-12T03:35:38.433Z",
-20
]
]
},
{
"name":2,
"data":[
"2019-09-12T03:36:04.433Z",
0.311303124694538
]
}
]
Fair warning: Yeah, I don't really know how to do that, and I'm hoping someone answers with a simple script to generate exactly the format you want on the Postgres side. But I made a start. Check this out:
select id, json_object_agg(time, value order by time)
from reading
group by id
Here's what I get:
2 "{ ""2019-09-12T03:36:03.434Z"" : 0.132331, ""2019-09-12T03:36:04.433Z"" : 0.311303 }"
3 "{ ""2019-09-12T03:36:03.434Z"" : 0.132331 }"
Here's something that's...not right..but getting closer:
select array_to_json(array_agg(row_to_json(reading_row))) as reading_object
from (
select id, json_object_agg(time, value order by time) as data
from reading
group by id
) as reading_row
Which returns:
[
{
"id":2,
"data":{
"2019-09-12T03:36:03.434Z":0.132331,
"2019-09-12T03:36:04.433Z":0.311303
}
},
{
"id":3,
"data":{
"2019-09-12T03:36:03.434Z":0.132331
}
}
]
I took another crack at it here, this might be what you're after, or close. I noticed you're renaming 'id' as 'name', so that's in the final query:
select array_to_json(array_agg(row_to_json(subquery)))
from (
select id as name,
array_to_json(array_agg(json_build_object('time', time, 'value', value))) as data
from reading
group by id
) subquery
The output, pretty-printed, looks like this:
[
{
"name":2,
"data":[
{
"time":"2019-09-12T03:36:04.433Z",
"value":0.311303
},
{
"time":"2019-09-12T03:36:03.434Z",
"value":0.132331
}
]
},
{
"name":3,
"data":[
{
"time":"2019-09-12T03:36:03.434Z",
"value":0.132331
}
]
}
]
This variant has the same structure, but without labels on the elements within the array:
select array_to_json(array_agg(row_to_json(subquery)))
from (
select id as name,
array_to_json(array_agg(array[time, value::text])) as data
from reading
group by id
) subquery
Apart from the numeric value being cast as text, I think this is what you asked for:
select array_to_json(array_agg(row_to_json(subquery)))
from (
select id as name,
array_to_json(array_agg(array[time, value::text])) as data
from reading
group by id
) subquery
[
{
"name":2,
"data":[
[
"2019-09-12T03:36:04.433Z",
"0.311303"
],
[
"2019-09-12T03:36:03.434Z",
"0.132331"
]
]
},
{
"name":3,
"data":[
[
"2019-09-12T03:36:03.434Z",
"0.132331"
]
]
}
]
Note: I don't see where you're getting your output of 20, -20 in your example.
Between array_to_json(), row(), array_agg(), and json_build_object(), it looks like you can get most any format you need.
Here's hoping that someone who actually knows what they're doing chimes in.

Related

JS: Extracting data from an array of objects

I have a complex query with 100s of fields and nested fields. What I want to do is, for each Index, extract the English and French text. As you can see in the array, there is no French text for some indexes. In that case I want to get the English text.
For me extracting the English text works fine because the text is already there, but incase of French, I get undefined errors. What would be the best way to implement this. Is Loadash needed for this or just pure JS methods?
Just to be clear, I have erros with extracting french because in some fields, french text is not available, I want to use the english value in that case.
Also It is recommend if I am able to get the English and French values by it's language field rather than the index. I have no idea how to do that.
Any suggestion, documentation is appreciated. Thank you!
example array:
[
{
id: "1",
name: [
{
language: "en-US",
text: "HOLIDAY"
}
],
order: 6,
Groups: [
{
name: [
{
language: "en-US",
text: "REGULAR"
}
],
code: "REGEARN"
},
{
name: [
{
language: "en-US",
text: "CHARGE"
}
],
code: "CHARGE"
}
]
}
]
and here is the code sandbox that reproduces my error:
CODE SAND BOX
https://codesandbox.io/s/javascript-forked-5073j
EDIT:
EXPECTED OUTPUT:
{
key: key,
englishtext: "Value Here",
frenchtext: "Value Here"
}
below is a working code, but issue is it does not work when there is no french language or that field. I get undefined errors. So is it possible I can get the needed data from the language field?
x.map((y) => ({
key: y.id,
name: y.name[0].text,
groupname: y.Groups ? x.Groups[0].name?.[0].text : 'N/A',
}))
Do you expect result like this? If you don't mind lodash.
const _ = require('lodash');
const getNames = (arr) => {
return arr.map((obj) => {
const id = obj.id;
const englishtext = _.get(obj, 'name[0].text', 'N/A');
const frenchtext = _.get(obj, 'name[1].text', englishtext);
return { id, englishtext, frenchtext };
});
};
console.log(getNames(x));
// [
// { id: '1', englishtext: 'HOLIDAY', frenchtext: 'HOLIDAY' },
// { id: '2', englishtext: 'Stat Holiday', frenchtext: 'Congé Férié' },
// { id: '3', englishtext: 'Over', frenchtext: 'Over' }
// ]

Datatables: dropdown in a cell

i'm trying to generate a table with Datatables.
I receive a json from my controller, here a sample:
this json can change (number of columns, name of the columns) and I can build my table with the good number of column and the good name.
My question is:
How can i do to have a dropdown when the "liste" have an array and a simple input when it's null?
Is it even possible?
EDIT :
I forget to explain something. The Json that I receive is a json to build the table not to fill it. So is it possible to do a columnsDef before the datas are in the cell.
EDIT n°2:
I used the solution that I accepted, but the problem was with my json. I tried to send a json to build and a json to fill the table. So I change my json and I send the list of options in the json to fill the table.
Hope it will help other people.
Thanks
Here are two solutions:
1) With a drop-down.
2) With a formatted array (as an alternative).
1) With a Dropdown
The end result looks like this:
The datatables definition is this:
<script type="text/javascript">
var dataSet = { "records" : [
{ "data" : "123456789",
"liste" : null,
"name" : "Nombre Enfants"
},
{ "data" : "5678901234",
"liste" : [ "Oui", "Non" ],
"name" : "Transport"
}]};
$(document).ready(function() {
$('#example').DataTable( {
data: dataSet.records,
columnDefs: [
{ targets: [ 0 ],
title: "Data",
data: "data" },
{ targets: [ 1 ],
title: "Liste",
data: function ( row ) {
if (row.liste == null) {
return null;
} else {
return buildDropdown(row.liste);
}
} },
{ targets: [ 2 ],
title: "Name",
data: "name" }
]
} );
function buildDropdown(data) {
var dropdown = "<select>";
for (var i = 0; i < data.length; i++) {
var option = "<option value=\"" + data[i] + "\">" + data[i] + "</option>";
dropdown = dropdown + option;
}
dropdown = dropdown + "</select>";
return dropdown;
}
} );
</script>
It builds a drop-down based on the assumption that a non-null value is an array. This may not always be the case in your data - just an assumption on my part.
2) With a formatted array
Just in case this is also of interest, DataTables has a built-in syntax for formatting array data, so it is displayed in a cell like this:
In this case, you no longer need the drop-down builder function. Everything else is the same as option (1) except for this part:
{ targets: [ 1 ],
title: "Liste",
data: "liste[, ]" },
Specifically, the [, ] notation lets you format the array data.
I mention this only because it lets you display all the array data in the cell, rather than neeeding to click a drop-down. But that is just a suggestion.
You may find that other functions such as searching and sorting are better with this option.
Update
The question has clarified that the table needs to be built dynamically from the data provided in the JSON.
You can pass variables to the datatables initializer - for example:
var col1 = { targets: [ 0 ], title: "Data", data: "data" };
var col2 = { targets: [ 1 ], title: "Liste", data: "liste" };
var col2 = { targets: [ 2 ], title: "Name", data: "name" };
var dynamicCols = [ col1, col2, col3 ];
The above col1 variable defines the title for the column, and where the column will get its data (from the dataSet.data fields).
The dynamicCols variable can then be used in a columnDefs as follows:
$(document).ready(function() {
$('#example').DataTable( {
data: dataSet.records,
columnDefs: dynamicCols
} );
However, I am not aware of a way to include a function in a columndef, using this approach (for example to present a cell's data as a drop-down, if needed).
There are additional techniques which can be used to make a datatable even more dynamic - several examples are available online - for example here. Without seeing a more detailed example of the JSON being provided, I am not sure if there are any additional suggestions I can make.

Add property to mongoose query result in nested queries [duplicate]

How do I perform the SQL Join equivalent in MongoDB?
For example say you have two collections (users and comments) and I want to pull all the comments with pid=444 along with the user info for each.
comments
{ uid:12345, pid:444, comment="blah" }
{ uid:12345, pid:888, comment="asdf" }
{ uid:99999, pid:444, comment="qwer" }
users
{ uid:12345, name:"john" }
{ uid:99999, name:"mia" }
Is there a way to pull all the comments with a certain field (eg. ...find({pid:444}) ) and the user information associated with each comment in one go?
At the moment, I am first getting the comments which match my criteria, then figuring out all the uid's in that result set, getting the user objects, and merging them with the comment's results. Seems like I am doing it wrong.
As of Mongo 3.2 the answers to this question are mostly no longer correct. The new $lookup operator added to the aggregation pipeline is essentially identical to a left outer join:
https://docs.mongodb.org/master/reference/operator/aggregation/lookup/#pipe._S_lookup
From the docs:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
Of course Mongo is not a relational database, and the devs are being careful to recommend specific use cases for $lookup, but at least as of 3.2 doing join is now possible with MongoDB.
We can merge/join all data inside only one collection with a easy function in few lines using the mongodb client console, and now we could be able of perform the desired query.
Below a complete example,
.- Authors:
db.authors.insert([
{
_id: 'a1',
name: { first: 'orlando', last: 'becerra' },
age: 27
},
{
_id: 'a2',
name: { first: 'mayra', last: 'sanchez' },
age: 21
}
]);
.- Categories:
db.categories.insert([
{
_id: 'c1',
name: 'sci-fi'
},
{
_id: 'c2',
name: 'romance'
}
]);
.- Books
db.books.insert([
{
_id: 'b1',
name: 'Groovy Book',
category: 'c1',
authors: ['a1']
},
{
_id: 'b2',
name: 'Java Book',
category: 'c2',
authors: ['a1','a2']
},
]);
.- Book lending
db.lendings.insert([
{
_id: 'l1',
book: 'b1',
date: new Date('01/01/11'),
lendingBy: 'jose'
},
{
_id: 'l2',
book: 'b1',
date: new Date('02/02/12'),
lendingBy: 'maria'
}
]);
.- The magic:
db.books.find().forEach(
function (newBook) {
newBook.category = db.categories.findOne( { "_id": newBook.category } );
newBook.lendings = db.lendings.find( { "book": newBook._id } ).toArray();
newBook.authors = db.authors.find( { "_id": { $in: newBook.authors } } ).toArray();
db.booksReloaded.insert(newBook);
}
);
.- Get the new collection data:
db.booksReloaded.find().pretty()
.- Response :)
{
"_id" : "b1",
"name" : "Groovy Book",
"category" : {
"_id" : "c1",
"name" : "sci-fi"
},
"authors" : [
{
"_id" : "a1",
"name" : {
"first" : "orlando",
"last" : "becerra"
},
"age" : 27
}
],
"lendings" : [
{
"_id" : "l1",
"book" : "b1",
"date" : ISODate("2011-01-01T00:00:00Z"),
"lendingBy" : "jose"
},
{
"_id" : "l2",
"book" : "b1",
"date" : ISODate("2012-02-02T00:00:00Z"),
"lendingBy" : "maria"
}
]
}
{
"_id" : "b2",
"name" : "Java Book",
"category" : {
"_id" : "c2",
"name" : "romance"
},
"authors" : [
{
"_id" : "a1",
"name" : {
"first" : "orlando",
"last" : "becerra"
},
"age" : 27
},
{
"_id" : "a2",
"name" : {
"first" : "mayra",
"last" : "sanchez"
},
"age" : 21
}
],
"lendings" : [ ]
}
I hope this lines can help you.
This page on the official mongodb site addresses exactly this question:
https://mongodb-documentation.readthedocs.io/en/latest/ecosystem/tutorial/model-data-for-ruby-on-rails.html
When we display our list of stories, we'll need to show the name of the user who posted the story. If we were using a relational database, we could perform a join on users and stores, and get all our objects in a single query. But MongoDB does not support joins and so, at times, requires bit of denormalization. Here, this means caching the 'username' attribute.
Relational purists may be feeling uneasy already, as if we were violating some universal law. But let’s bear in mind that MongoDB collections are not equivalent to relational tables; each serves a unique design objective. A normalized table provides an atomic, isolated chunk of data. A document, however, more closely represents an object as a whole. In the case of a social news site, it can be argued that a username is intrinsic to the story being posted.
You have to do it the way you described. MongoDB is a non-relational database and doesn't support joins.
With right combination of $lookup, $project and $match, you can join mutiple tables on multiple parameters. This is because they can be chained multiple times.
Suppose we want to do following (reference)
SELECT S.* FROM LeftTable S
LEFT JOIN RightTable R ON S.ID = R.ID AND S.MID = R.MID
WHERE R.TIM > 0 AND S.MOB IS NOT NULL
Step 1: Link all tables
you can $lookup as many tables as you want.
$lookup - one for each table in query
$unwind - correctly denormalises data , else it'd be wrapped in arrays
Python code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "R"}
])
Step 2: Define all conditionals
$project : define all conditional statements here, plus all the variables you'd like to select.
Python Code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "R"},
# define conditionals + variables
{"$project": {
"midEq": {"$eq": ["$MID", "$R.MID"]},
"ID": 1, "MOB": 1, "MID": 1
}}
])
Step 3: Join all the conditionals
$match - join all conditions using OR or AND etc. There can be multiples of these.
$project: undefine all conditionals
Complete Python Code..
db.LeftTable.aggregate([
# connect all tables
{"$lookup": {
"from": "RightTable",
"localField": "ID",
"foreignField": "ID",
"as": "R"
}},
{"$unwind": "$R"},
# define conditionals + variables
{"$project": {
"midEq": {"$eq": ["$MID", "$R.MID"]},
"ID": 1, "MOB": 1, "MID": 1
}},
# join all conditionals
{"$match": {
"$and": [
{"R.TIM": {"$gt": 0}},
{"MOB": {"$exists": True}},
{"midEq": {"$eq": True}}
]}},
# undefine conditionals
{"$project": {
"midEq": 0
}}
])
Pretty much any combination of tables, conditionals and joins can be done in this manner.
You can join two collection in Mongo by using lookup which is offered in 3.2 version. In your case the query would be
db.comments.aggregate({
$lookup:{
from:"users",
localField:"uid",
foreignField:"uid",
as:"users_comments"
}
})
or you can also join with respect to users then there will be a little change as given below.
db.users.aggregate({
$lookup:{
from:"comments",
localField:"uid",
foreignField:"uid",
as:"users_comments"
}
})
It will work just same as left and right join in SQL.
As others have pointed out you are trying to create a relational database from none relational database which you really don't want to do but anyways, if you have a case that you have to do this here is a solution you can use. We first do a foreach find on collection A( or in your case users) and then we get each item as an object then we use object property (in your case uid) to lookup in our second collection (in your case comments) if we can find it then we have a match and we can print or do something with it.
Hope this helps you and good luck :)
db.users.find().forEach(
function (object) {
var commonInBoth=db.comments.findOne({ "uid": object.uid} );
if (commonInBoth != null) {
printjson(commonInBoth) ;
printjson(object) ;
}else {
// did not match so we don't care in this case
}
});
Here's an example of a "join" * Actors and Movies collections:
https://github.com/mongodb/cookbook/blob/master/content/patterns/pivot.txt
It makes use of .mapReduce() method
* join - an alternative to join in document-oriented databases
$lookup (aggregation)
Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. To each input document, the $lookup stage adds a new array field whose elements are the matching documents from the “joined” collection. The $lookup stage passes these reshaped documents to the next stage.
The $lookup stage has the following syntaxes:
Equality Match
To perform an equality match between a field from the input documents with a field from the documents of the “joined” collection, the $lookup stage has the following syntax:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
The operation would correspond to the following pseudo-SQL statement:
SELECT *, <output array field>
FROM collection
WHERE <output array field> IN (SELECT <documents as determined from the pipeline>
FROM <collection to join>
WHERE <pipeline> );
Mongo URL
It depends on what you're trying to do.
You currently have it set up as a normalized database, which is fine, and the way you are doing it is appropriate.
However, there are other ways of doing it.
You could have a posts collection that has imbedded comments for each post with references to the users that you can iteratively query to get. You could store the user's name with the comments, you could store them all in one document.
The thing with NoSQL is it's designed for flexible schemas and very fast reading and writing. In a typical Big Data farm the database is the biggest bottleneck, you have fewer database engines than you do application and front end servers...they're more expensive but more powerful, also hard drive space is very cheap comparatively. Normalization comes from the concept of trying to save space, but it comes with a cost at making your databases perform complicated Joins and verifying the integrity of relationships, performing cascading operations. All of which saves the developers some headaches if they designed the database properly.
With NoSQL, if you accept that redundancy and storage space aren't issues because of their cost (both in processor time required to do updates and hard drive costs to store extra data), denormalizing isn't an issue (for embedded arrays that become hundreds of thousands of items it can be a performance issue, but most of the time that's not a problem). Additionally you'll have several application and front end servers for every database cluster. Have them do the heavy lifting of the joins and let the database servers stick to reading and writing.
TL;DR: What you're doing is fine, and there are other ways of doing it. Check out the mongodb documentation's data model patterns for some great examples. http://docs.mongodb.org/manual/data-modeling/
There is a specification that a lot of drivers support that's called DBRef.
DBRef is a more formal specification for creating references between documents. DBRefs (generally) include a collection name as well as an object id. Most developers only use DBRefs if the collection can change from one document to the next. If your referenced collection will always be the same, the manual references outlined above are more efficient.
Taken from MongoDB Documentation: Data Models > Data Model Reference >
Database References
Before 3.2.6, Mongodb does not support join query as like mysql. below solution which works for you.
db.getCollection('comments').aggregate([
{$match : {pid : 444}},
{$lookup: {from: "users",localField: "uid",foreignField: "uid",as: "userData"}},
])
You can run SQL queries including join on MongoDB with mongo_fdw from Postgres.
MongoDB does not allow joins, but you can use plugins to handle that. Check the mongo-join plugin. It's the best and I have already used it. You can install it using npm directly like this npm install mongo-join. You can check out the full documentation with examples.
(++) really helpful tool when we need to join (N) collections
(--) we can apply conditions just on the top level of the query
Example
var Join = require('mongo-join').Join, mongodb = require('mongodb'), Db = mongodb.Db, Server = mongodb.Server;
db.open(function (err, Database) {
Database.collection('Appoint', function (err, Appoints) {
/* we can put conditions just on the top level */
Appoints.find({_id_Doctor: id_doctor ,full_date :{ $gte: start_date },
full_date :{ $lte: end_date }}, function (err, cursor) {
var join = new Join(Database).on({
field: '_id_Doctor', // <- field in Appoints document
to: '_id', // <- field in User doc. treated as ObjectID automatically.
from: 'User' // <- collection name for User doc
}).on({
field: '_id_Patient', // <- field in Appoints doc
to: '_id', // <- field in User doc. treated as ObjectID automatically.
from: 'User' // <- collection name for User doc
})
join.toArray(cursor, function (err, joinedDocs) {
/* do what ever you want here */
/* you can fetch the table and apply your own conditions */
.....
.....
.....
resp.status(200);
resp.json({
"status": 200,
"message": "success",
"Appoints_Range": joinedDocs,
});
return resp;
});
});
You can do it using the aggregation pipeline, but it's a pain to write it yourself.
You can use mongo-join-query to create the aggregation pipeline automatically from your query.
This is how your query would look like:
const mongoose = require("mongoose");
const joinQuery = require("mongo-join-query");
joinQuery(
mongoose.models.Comment,
{
find: { pid:444 },
populate: ["uid"]
},
(err, res) => (err ? console.log("Error:", err) : console.log("Success:", res.results))
);
Your result would have the user object in the uid field and you can link as many levels deep as you want. You can populate the reference to the user, which makes reference to a Team, which makes reference to something else, etc..
Disclaimer: I wrote mongo-join-query to tackle this exact problem.
playORM can do it for you using S-SQL(Scalable SQL) which just adds partitioning such that you can do joins within partitions.
Nope, it doesn't seem like you're doing it wrong. MongoDB joins are "client side". Pretty much like you said:
At the moment, I am first getting the comments which match my criteria, then figuring out all the uid's in that result set, getting the user objects, and merging them with the comment's results. Seems like I am doing it wrong.
1) Select from the collection you're interested in.
2) From that collection pull out ID's you need
3) Select from other collections
4) Decorate your original results.
It's not a "real" join, but it's actually alot more useful than a SQL join because you don't have to deal with duplicate rows for "many" sided joins, instead your decorating the originally selected set.
There is alot of nonsense and FUD on this page. Turns out 5 years later MongoDB is still a thing.
I think, if You need normalized data tables - You need to try some other database solutions.
But I've foun that sollution for MOngo on Git
By the way, in inserts code - it has movie's name, but noi movie's ID.
Problem
You have a collection of Actors with an array of the Movies they've done.
You want to generate a collection of Movies with an array of Actors in each.
Some sample data
db.actors.insert( { actor: "Richard Gere", movies: ['Pretty Woman', 'Runaway Bride', 'Chicago'] });
db.actors.insert( { actor: "Julia Roberts", movies: ['Pretty Woman', 'Runaway Bride', 'Erin Brockovich'] });
Solution
We need to loop through each movie in the Actor document and emit each Movie individually.
The catch here is in the reduce phase. We cannot emit an array from the reduce phase, so we must build an Actors array inside of the "value" document that is returned.
The code
map = function() {
for(var i in this.movies){
key = { movie: this.movies[i] };
value = { actors: [ this.actor ] };
emit(key, value);
}
}
reduce = function(key, values) {
actor_list = { actors: [] };
for(var i in values) {
actor_list.actors = values[i].actors.concat(actor_list.actors);
}
return actor_list;
}
Notice how actor_list is actually a javascript object that contains an array. Also notice that map emits the same structure.
Run the following to execute the map / reduce, output it to the "pivot" collection and print the result:
printjson(db.actors.mapReduce(map, reduce, "pivot"));
db.pivot.find().forEach(printjson);
Here is the sample output, note that "Pretty Woman" and "Runaway Bride" have both "Richard Gere" and "Julia Roberts".
{ "_id" : { "movie" : "Chicago" }, "value" : { "actors" : [ "Richard Gere" ] } }
{ "_id" : { "movie" : "Erin Brockovich" }, "value" : { "actors" : [ "Julia Roberts" ] } }
{ "_id" : { "movie" : "Pretty Woman" }, "value" : { "actors" : [ "Richard Gere", "Julia Roberts" ] } }
{ "_id" : { "movie" : "Runaway Bride" }, "value" : { "actors" : [ "Richard Gere", "Julia Roberts" ] } }
We can merge two collection by using mongoDB sub query. Here is example,
Commentss--
`db.commentss.insert([
{ uid:12345, pid:444, comment:"blah" },
{ uid:12345, pid:888, comment:"asdf" },
{ uid:99999, pid:444, comment:"qwer" }])`
Userss--
db.userss.insert([
{ uid:12345, name:"john" },
{ uid:99999, name:"mia" }])
MongoDB sub query for JOIN--
`db.commentss.find().forEach(
function (newComments) {
newComments.userss = db.userss.find( { "uid": newComments.uid } ).toArray();
db.newCommentUsers.insert(newComments);
}
);`
Get result from newly generated Collection--
db.newCommentUsers.find().pretty()
Result--
`{
"_id" : ObjectId("5511236e29709afa03f226ef"),
"uid" : 12345,
"pid" : 444,
"comment" : "blah",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f2"),
"uid" : 12345,
"name" : "john"
}
]
}
{
"_id" : ObjectId("5511236e29709afa03f226f0"),
"uid" : 12345,
"pid" : 888,
"comment" : "asdf",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f2"),
"uid" : 12345,
"name" : "john"
}
]
}
{
"_id" : ObjectId("5511236e29709afa03f226f1"),
"uid" : 99999,
"pid" : 444,
"comment" : "qwer",
"userss" : [
{
"_id" : ObjectId("5511238129709afa03f226f3"),
"uid" : 99999,
"name" : "mia"
}
]
}`
Hope so this will help.

How do I properly map attributes of relations in sequelize.js?

I'm creating a recipe-database (commonly known as a cookbook) where I need to have a many-to-many relationship between ingredients and recipes and I'm using sequelize.js in combination with postgresql.
When an ingredient is added to a recipe I need to declare the correct amount of that ingredient that goes into the recipe.
I've declared (reduced example)
var Ingredient = sequelize.define('Ingredient', {
name: Sequelize.STRING
}, {
freezeTable: true
});
var Recipe = sequelize.define('Recipe', {
name: Sequelize.STRING
}, {
freezeTable: true
});
var RecipeIngredient = sequelize.define('RecipeIngredient', {
amount: Sequelize.DOUBLE
});
Ingredient.belongsToMany(Recipe, { through: RecipeIngredient });
Recipe.belongsToMany(Ingredient, {
through: RecipeIngredient,
as: 'ingredients'
});
My problem is with how data is returned when one my REST endpoints do
router.get('/recipes', function(req, res) {
Recipe.findAll({
include: [{
model: Ingredient,
as: 'ingredients'
}]
}).then(function(r) {
return res.status(200).json(r[0].toJSON());
})
});
The resulting JSON that gets sent to the client looks like this (timestamps omitted):
{
"id": 1,
"name": "Carrots",
"ingredients": [
{
"id": 1,
"name": "carrot",
"RecipeIngredient": {
"amount": 12,
"RecipeId": 1,
"IngredientId": 1
}
}
]
}
While all I wanted was
{
"id": 1,
"name": "Carrots",
"ingredients": [
{
"id": 1,
"name": "carrot",
"amount": 12,
}
]
}
That is, I want the amount field from the relation-table to be included in the result instead of the entire RecipeIngredient object.
The database generated by sequelize looks like this:
Ingredients
id name
1 carrot
Recipes
id name
1 Carrots
RecipeIngredients
amount RecipeId IngredientId
12 1 1
I've tried to provide an attributes array as a property to the include like this:
include: [{
model: Ingredient,
as: 'ingredients',
attributes: []
}]
But setting either ['amount'] or ['RecipeIngredient.amount'] as the attributes-value throws errors like
Unhandled rejection SequelizeDatabaseError: column ingredients.RecipeIngredient.amount does not exist
Obviously I can fix this in JS using .map but surely there must be a way to make sequelize do the work for me?
I am way late to this one, but i see it been viewed quite a bit so here is my answer on how to merge
attributes
Some random examples in this one
router.get('/recipes', function(req, res) {
Recipe.findAll({
include: [{
model: Ingredient,
as: 'ingredients',
through: {
attributes: ['amount']
}
}]
})
.then(docs =>{
const response = {
Deal: docs.map(doc =>{
return{
cakeRecipe:doc.recipe1,
CokkieRecipe:doc.recipe2,
Apples:doc.ingredients.recipe1ingredient
spices:[
{
sugar:doc.ingredients.spice1,
salt:doc.ingredients.spice2
}
]
}
})
}
})
res.status(200).json(response)
})
You can use sequelize.literal. Using Ingredient alias of Recipe, you can write as follows. I do not know if this is the right way. :)
[sequelize.literal('`TheAlias->RecipeIngredient`.amount'), 'amount'],
I tested with sqlite3. Received result with alias "ir" is
{ id: 1,
name: 'Carrots',
created_at: 2018-03-18T04:00:54.478Z,
updated_at: 2018-03-18T04:00:54.478Z,
ir: [ { amount: 10, RecipeIngredient: [Object] } ] }
See the full code here.
https://github.com/eseom/sequelize-cookbook
I've gone over the documentation but I couldn't find anything that seems like it would let me merge the attributes of the join-table into the result so it looks like I'm stuck with doing something like this:
router.get('/recipes', function(req, res) {
Recipe.findAll({
include: [{
model: Ingredient,
as: 'ingredients',
through: {
attributes: ['amount']
}
}]
}).then(function(recipes) {
return recipes[0].toJSON();
}).then(function(recipe) {
recipe.ingredients = recipe.ingredients.map(function(i) {
i.amount = i.RecipeIngredient.amount;
delete i.RecipeIngredient;
return i;
});
return recipe;
}).then(function(recipe) {
return res.status(200).json(recipe);
});
});
Passing through to include lets me filter out which attributes I want to include from the join-table but for the life of me I could not find a way to make sequelize merge it for me.
The above code will return the output I wanted but with the added overhead of looping over the list of ingredients which is not quite what I wanted but unless someone comes up with a better solution I can't see another way of doing this.

Mongoose distinct condition not working

I'm using mongoose to query a MongoDB using following scheme:
var File = mongoose.model('File',{
size: Number,
type: String,
filename: String,
path: String,
taxonomies:[{
tags:[{
name:String,
'tag':String
}],
name: String
}]
});
Now I want to do a distinct query as follows:
File.distinct('taxonomies.tags.tag', {'taxonomies.name':'AM'},function(error, tags){
if(err)
res.send(err);
console.log("All distinct tags", tags);
})
to get all unique tags from all files where the name of the taxonomy is 'AM'. The problem is that the query above returns all tags, including those with a different taxonomies.name. Without the condition, it just works as it should.
Do I have a syntactic error in here, or am I misunderstanding how distinct works?
Update (More examples)
Each document has a taxonomy with name SM and one with name AM, something like
taxonomies: [{
tags: [
{
name:"Type",
'tag':kind
}, {
name:"Language",
'tag':lang
},
{
name:"Code",
"tag":code
}],
name:'AM'
}, {
name:'SM',
tags:[{
name:"Sales",
'tag':'has to be sold'
},{
name:"Personal filter",
'tag':'temp'
}]
}]
When I execute the query mentioned above, I get as a result:
All distinct tags [ '4007', 'fr', 'has to be sold', 'temp', 'wol', '16104', 'en' ]
while 'temp' and 'has to be sold are from SM, not AM.
I want as a result only the ones wehre taxonomies.name = 'AM', across all documents, without duplicates
The query selection is selecting documents, not elements from the array:
{'taxonomies.name':'AM'}
That criteria will select the entire document.
As a result, all items in both arrays are considered as valid for the distinct operation. If you only want subsets to be considered, you'll either need to use the aggregation framework to filter, or modify your documents so that the two taxonomies are not stored in the same document.
If you used the aggregation framework instead of the distinct command, there are a lot of options you could use to gather the distinct values:
db.test.aggregate({$unwind: "$taxonomies" },
{ $group :
{ _id: "$taxonomies.name",
tags: { $addToSet: "$taxonomies.tags" } }} )
That should group on both SM and AM (and any others):
[
{
"_id" : "SM",
"tags" : [
[
"d1",
"e1"
],
[
"d",
"e"
]
]
},
{
"_id" : "AM",
"tags" : [
[
"a1",
"b1",
"c1"
],
[
"a",
"b",
"c"
]
]
}
]
You could use $match to filter on a particular name as well.

Categories