We Keep Coding

Simple $lookup "left join" of the ObjectId of two collections not working in Mongo DB

I am using Mongo DB Atlas with node.js and oboe (oboe streams the results to html). I am new to this all, just learning all these technologies, so explaining in simpler terms will be appreaciated.
The goal is to do a $lookup between two collections, in this case, the collection of 'review' to 'place'. I've researched similar answers, but the either didn't work or were using strings, not ObjectIds.
It's quite simple, just connect the ObjectIds of both of the collections, but I am not able to pull the data out from the "left joined" collection of 'places' when using oboe (see oboe code at bottom, FWIW).
Here is a look at a document from both collections, then the code. What am I doing wrong? I have tried converting them to strings and joining with .toString() and .str, plus put 'place_id.ObjectId' and '_id.ObjectId' for localField and foreignField. Another thing too, is how can I see what is in the cursor to know what I am getting? debug(cursor.ToArray()) didn't work. Thanks in advance.
review
({
"_id": { "$oid": "5fd27fd9647f7bb815c4c946" },
"place_id": { "$oid": "5fbc37c4fc13ae680b00002b" }, // connect this...
"user_id": { "$oid": "5fbc10ecfc13ae232d000068" },
"title": "Black Forest -- unforgettable!",
"description": "The forest was great.",
"score": { "$numberInt": "5" }
}
place
{
"_id": { "$oid": "5fbc37c4fc13ae680b00002b" }, // connected to _id above
"name": "Black Forest (Schwarzwald)",
"category": "activity",
"city": "Freiburg",
"country": "Germany",
"description": "The Black Forest (German: Schwarzwald [ˈʃvaʁtsvalt] (About this soundlisten)) is a large forested mountain range.]",
"image": { "filename": "1607629020164_black_forest.jpg", "mime": "image/jpeg" },
"state": ""
})
router.get('/', async (req, res, next) => {
debug('get all reviews api');
try {
const q = req.query.q;
const collation = { locale: 'en_US', strength: 1 };
const matchStage = {};
if (q) {
matchStage.$text = { $search: q };
}
const pipeline = [
{
$match: matchStage,
},
{
$lookup: {
from: 'place',
localField: 'place_id',
foreignField: '_id',
as: 'place',
},
},
];
const connection = await db.connect();
const cursor = connection.collection('review').aggregate(pipeline, { collation: collation });
// write the JSON file
res.type('application/json');
res.write('[\n');
for await (const doc of cursor) {
res.write(JSON.stringify(doc));
res.write(',\n');
}
res.end('null]');
} catch (err) {
sendError(err, res);
}
});
The cursor goes to oboe and becomes an 'item'. I would expect to use a template string such as {item.place.name} to get the data when putting this into html. That's how I would access it, right?
const performSearch = () => {
seen = 0;
$('stream-data-spinner').removeClass('d-none');
$('#search-results').html('');
const formData = $('#search-place-form').serialize();
oboe('/api/review?' + formData)
.node('![*]', (item) => {
if (item) {
chunk.push(item);
if (chunk.length >= 1000) {
showChunk(chunk);
}
}
return oboe.drop;
})
.done((_) => {
// show the last chunk
showChunk(chunk);
// hide the spinner
outputSpinner.classList.add('d-none');
})
.fail((res) => {
// show the error
outputSeen.textContent = `ERROR: network error`;
outputSeen.classList.add('text-danger');
outputSpinner.classList.add('text-danger');
});
};

From your MongoDB aggregation query, your place field is an array. You may want to $unwind it to flatten it into object for your oboe code to access it.

Join With AngularFire 2

I have seen similar questions asked, but those questions had slightly different data structures to what I'm dealing with. I've looked at:
Joining data between paths based on id using AngularFire
AngularFire2: Perform 'Joins' on FirebaseListObservables using RxJS .map()
This is my data structure:
{
"samples": {
"24084": {
"addInfo": "TEST",
"datePrinted": "8/11/2017 9:42:57 AM",
"equipment": "GR028",
"hmisNumber": "100E",
"lotNumber": "GR0030C659-JM",
"productionNumber": "PN0034781",
"userName": "MCorbett"
},
"24342": {
"addInfo": "test",
"datePrinted": "8/15/2017 11:51:55 AM",
"equipment": "GR025",
"hmisNumber": "100",
"lotNumber": "BR0010P835",
"productionNumber": "PN0035616",
"userName": "MCorbett"
}
},
"scans": {
"-Krlb3tv3oFPtYZp2ErX": {
"inTime": 1502997139131,
"sampleId": "24342"
},
"-KrlbdbCT0us6xE9POm3": {
"inTime": 1502997289573,
"outTime": 1502997292524,
"sampleId": "24342"
},
"-Krlc3vsjiQ9czWYGvA9": {
"inTime": 1502997401784,
"outTime": 1502997404864,
"sampleId": "24084"
}
}
}
As you can see, Samples to Scans have a one to many relationship. What I need to do is populated a table with Sample data joined to scan data. It needs to look like this:
"24342": {
"addInfo": "test",
"datePrinted": "8/15/2017 11:51:55 AM",
"equipment": "GR025",
"hmisNumber": "100",
"lotNumber": "BR0010P835",
"productionNumber": "PN0035616",
"userName": "MCorbett",
"inTime": 1502996197213
}
I need to grab all Scans where outTime is undefined, and then join it to it's corresponding Sample data. Here is what I have tried so far:
// Get samples that have a scan where inTime is populated but outTime is not
getOpenSamples() {
console.log('getopensmaples stareted')
let scanWithSampleList = this.scanSvc.getScansList({
orderBy: 'outTime',
startAt: ''
})
.switchMap(scans => {
let sampleObservables = scans.map(scan => this.getSample(scan.sampleId));
console.log("insisde");
return sampleObservables.length === 0 ?
Observable.of(scans) :
Observable.combineLatest(sampleObservables, (samples) => {
scans.forEach((scan, index) => {
scan.productionNumber = samples[index].productionNumer;
scan.lotNumbner = samples[index].lotNumber;
});
return scans;
});
});
}
This gives me this error:
ERROR in C:/Users/corbetmw/workspace/angular/sample-time-tracking/src/app/samples/shared/sample.service.ts (82,54): Propert
y 'productionNumer' does not exist on type '{}'.
What am I doing wrong here? This seems like a simple enough thing, but I'm having a lot of trouble with it. Do I need to change my data structure? Should I make a component that gets the Scans with undefined outTime and stick it in the table with a parent that can pass the sample ID, or vice versa?

I was able to find a solution which returns an observable of type Scan<> with a single Scan inside of it.
I have this function:
getOpenScans() {
const scansList = this.db.list('/scans', ref => ref.orderByChild('outTime').endAt(null));
return scansList.snapshotChanges().map(arr => {
return arr.map(snap => Object.assign(snap.payload.val(), {
$key: snap.key
}))
})
}
Then, I have this guy:
getOpenSamples() {
let openScans = this.scanSvc.getOpenScans();
let fullSamples = openScans.map(scans => {
for (let scan of scans) {
scan.sample = this.getSample(scan.sampleId);
}
return scans;
});
//fullSamples.subscribe(samples => console.log(samples));
return fullSamples;
}
I am now trying to implement this solution with MatTable in Material2.

Populating kendo-tree using from Json

I am in Angular environment using Kendo. All I want to do is following:
Take Json
Produce Kendo tree using it
I have tried it with simple data and it seems to work fine. But this time I have somewhat complex data and it seems like it does not work well with complex Json. I have been trying to have it render Json but it seems like it keeps on thinking and never comes back. I have created a sample Dojo for reference:
http://dojo.telerik.com/EdOqE
I am not sure what am I doing wrong but it just does not seem to work. Can anyone help me with this please?

I presume you have controll over the resultant json, because you'll have to change it a little to fit the TreeView's expected format. Check this out:
{
"items": [{ // Projects
"Id": 0,
"Name": "Your Example Project",
"CreatedOn": "",
"hasChildren": true,
"items": [{ // Analyses
"Id": 0,
"Name": "1.0 - Your Example Run",
"CreatedOn": "",
"hasChildren": true,
"items": [{ // Samples
"Id": 0,
"Name": "Sample 1",
"hasChildren": false,
"Description": "ample frample sample"
}, {
"Id": 0,
"Name": "Sample 2",
"hasChildren": false,
"Description": null
}]
}]
}]
};
The above json is what I did to work in the widget. First of all, the collection properties were renamed to items. All of them, in all levels. With that, kendo will know how property it should deal with. A hasChildren property was added to let it know when it has to show the expand icon. Otherwise it will show the expand option even if the item doesn't haves any children. So user clicks it and get an empty result.
This is the widget initialization options:
{
dataSource: new kendo.data.HierarchicalDataSource({
data: things,
schema: {
data: "items"
}
}),
dataTextField: "Name"
};
With schema.data I tell which property kendo will deal as the collection item. The dataSource expects an array, but if you give him an object, you have to set this property. If it was an array, then kendo would look for item property of each child for default. dataTextField is the name of the property it will use as the label.
Demo
Here is another demo with the data as an array. No need to set schema.data.
Update:
I was afraid you would say that. Yes, there is a way to deal with the data if you can't change it in the server-side. You have to intercept the data at the schema.parse() method and change the resultant data object property to items, so then the widget will understand:
schema: {
data: "items",
parse: function(data) {
if (data.hasOwnProperty("Projects")) {
return { items: data.Projects };
}
else if (data.hasOwnProperty("Analyses")) {
return { items: data.Analyses };
}
else if (data.hasOwnProperty("Samples")) {
return { items: data.Samples };
}
}
}
Demo
Every node when opened will call parse with items collection as data parameter. You have to return a new object with the property name as items instead of Projects, Analysis or Samples.
I forgot you can't touch the data, so can't add hasChildren property as well. Then you have to add a tiny logic into parse to set those properties in each level, otherwise the expand icon would not appear:
schema: {
data: "items",
parse: function(data) {
if (data.hasOwnProperty("Projects")) {
data.Projects.forEach(p => {
p.hasChildren = false;
if (p.hasOwnProperty("Analyses")) {
p.hasChildren = true;
}
});
return { items: data.Projects };
}
else if (data.hasOwnProperty("Analyses")) {
data.Analyses.forEach(a => {
a.hasChildren = false;
if (a.hasOwnProperty("Samples")) {
a.hasChildren = true;
}
});
return { items: data.Analyses };
}
else if (data.hasOwnProperty("Samples")) {
return { items: data.Samples };
}
}
}
Demo
It is ugly, I know. But get used to Kendo, it is the it goes with it.

KnockoutJs - Pushing to an element in an observable array does not work

I have an object (analysisLogData) that I use to generate a table using KnockoutJS. Here's the viewModel containing this object:
function AppViewModel() {
var self = this;
self.analysisLogData = ko.observableArray();
self.analysisLogTitle = ko.observable("Warnings")
self.changeAnalysisLog = function(title) {
self.analysisLogTitle(title)
}
var data =
{
"Warnings": [
{
"number": 3002,
"description": "There may be a problem with the device you are using if you use the default profile"
},
{
"number": 3001,
"description": "There may be a problem with the device you are using if you don't use the default profile"
}
]
,
"Errors": [
{
"number": 1000,
"description": "No networks are loaded"
},
{
"number": 1002,
"description": "No devices are loaded"
}]
}
self.addLog = function (type, content) {
self.analysisLogData()[type].push(content);
}
self.analysisLogData.push(data)
}
ko.applyBindings(new AppViewModel());
You can see the result here in a JSFiddle: http://jsfiddle.net/etiennenoel/V4r2e/5/
I want to be able to add an error or a warning without losing the warnings or errors already present.
I tried to do the following in the self.addLog function:
self.addLog = function (type, content) {
self.analysisLogData()[type].push(content);
}
but it says that it can't push to an undefined object...

Ok, after playing around in fiddle. I believe that you need to do some changes in how you pushed data in the observable array. But without doing a lot of modification check my solution in this link.
jsfiddle example
self.addLog = function (type, content) {
self.analysisLogData()[0][type].push({
"number": 1002,
"description": content
});
}
And data object should be
"Warnings": ko.observableArray([........]),
"Errors": ko.observableArray([..........])
I did two things
Modify Warnings & Errors to be an Observable Array
I pushed the data in this self.analysisLogData()[0][type].push instead of self.analysisLogData()[type].push

self.analysisLogData() is an array which contains arrays of Errors/Warnings.
I'm not sure if that's how you want your data structured.
To get the fiddle to work you can replace the addLog function with this:
self.addLog = function (type, content) {
self.analysisLogData()[0][type].push(content);
}

Ember-Data: How do "mappings" work

I'm currently trying to put something together with ember + emberdata + router + asp.net web api. Most of it seem to work, however I stuck in an error message I get when ember-data tries to findAll through the adapter for my models.
In my backend I have a model like this (C#):
public class Genre {
[Key]
public int Id { get; set; }
[Required]
[StringLength(50, MinimumLength=3)]
public string Name { get; set; }
}
Which in my app I represent it like this using ember-data:
App.Genre = DS.Model.extend({
id: DS.attr("number"),
name: DS.attr("string")
}).reopenClass({
url: 'api/genre'
});
I have also a Store defined in my App using the RESTAdapter like so:
App.store = DS.Store.create({
revision: 4,
adapter: DS.RESTAdapter.create({
bulkCommit: false
})
});
And the store is used in my controller as below:
App.GenreController = Ember.ArrayController.extend({
content: App.store.findAll(App.Genre),
selectedGenre: null
});
The router is defined as
App.router = Em.Router.create({
enableLogging: true,
location: 'hash',
root: Ember.Route.extend({
//...
genre: Em.Route.extend({
route: '/genre',
index: Ember.Route.extend({
connectOutlets: function (router, context) {
router.get('applicationController').connectOutlet('genre');
}
})
}),
//...
})
})
When I run my application, I get the following message for every object that has this same structure:
Uncaught Error: assertion failed: Your server returned a hash with the
key 0 but you have no mappings
For reference, here's the json the service is returning:
[
{
"id": 1,
"name": "Action"
},
{
"id": 2,
"name": "Drama"
},
{
"id": 3,
"name": "Comedy"
},
{
"id": 4,
"name": "Romance"
}
]
I cannot tell exactly what the problem is and since the assertion is mentioning that I need mapping, I'd like to know:
What this mapping is and how to use it.
Since the returned json is an array, should I be using a different type of controller in my app ,or is there anything I should know about when working with this type of json in ember-data? or should I change the JsonFormatter options in the server?
Any help is welcome.
I can definitely add more information if you feel this isn't enough to understand the problem.
EDIT: I've changed a few things in my backend and now my findAll() equivalent action in the server serializes the the output as the following json:
{
"genres": [
{ "id": 1, "name": "Action" },
{ "id": 2, "name": "Drama" },
{ "id": 3, "name": "Comedy" },
{ "id": 4, "name": "Romance" }
]
}
But I still can't get it to populate my models in the client and my error message has changed to this:
Uncaught Error: assertion failed: Your server returned a hash with the
key genres but you have no mappings
Not sure what else I might be doing wrong.
The method that throws this exception is sideload and checks for the mappings like this:
sideload: function (store, type, json, root) {
var sideloadedType, mappings, loaded = {};
loaded[root] = true;
for (var prop in json) {
if (!json.hasOwnProperty(prop)) { continue; }
if (prop === root) { continue; }
sideloadedType = type.typeForAssociation(prop);
if (!sideloadedType) {
mappings = get(this, 'mappings');
Ember.assert("Your server returned a hash with the key " + prop + " but you have no mappings", !!mappings);
//...
This call sideloadedType = type.typeForAssociation(prop); returns undefined and then I get the error message. The method typeForAssociation() checks for the for 'associationsByName' key which returns an empty Ember.Map.
Still no solution for this at the moment.
By the way...
My action is now like this:
// GET api/genres
public object GetGenres() {
return new { genres = context.Genres.AsQueryable() };
}
// GET api/genres
//[Queryable]
//public IQueryable<Genre> GetGenres()
//{
// return context.Genres.AsQueryable();
//}
I had to remove the original implementation which gets serialized by json.NET as I could not find config options to produce a json output as Ember-Data expects ( as in {resource_name : [json, json,...]}). Side effect of this is that I've lost built-in OData support, but I'd like to keep it. Does anyone know how could I configure it to produce different json for a collection?

The mapping can be defined in the DS.RESTAdapter. I think you could try to define something like this:
App.Store = DS.Store.extend({
adapter: DS.RESTAdapter.create({
bulkCommit: true,
mappings: {
genres: App.Genre
},
// you can also define plurals, if there is a unregular plural
// usually, RESTAdapter simply add a 's' for plurals.
// for example at work we have to define something like this
plurals: {
business_process: 'business_processes'
//else it tries to fetch business_processs
}
}),
revision: 4
});
Hope this resolves your problem.
Update:
At this time, this is not well documented, I don't remember if we found it by ourself reading the code, or perhaps Tom Dale pointed on it.
Anyway, here is the point for plurals
For the mappings, I think we were driven by the same error as you, and either we tried, either Tom teached us about this.

The RESTAdapter expects the returned JSON to be of the form:
{
"genres": [{
"id": 1,
"name": "action"
},{
"id": 2,
"name": "Drama"
}]
}
The tests are a good source of documentation, see https://github.com/emberjs/data/blob/master/packages/ember-data/tests/unit/rest_adapter_test.js#L315-329

I'm using Ember Data rev. 11 and it seems that the plurals config in DS.RESTAdapter.create never works. I looked into the codes and found a solution as following:
App.Adapter = DS.RESTAdapter.extend({
bulkCommit: false
})
App.Adapter.configure('plurals', {
series: 'series'
})