We Keep Coding

How can I store mapping orders in BBDD and then eval them

I'm trying to store in MongoDB one document with an object with the properties I want to map latter. My idea it's to create a function that will receive 2 params. First the object where I got to find the mapping, and second the object where I have to take the info from.
For example I want to store this JSON (that would be the first parameter in the function):
{
"name": "client.firstName",
"surname": "client.surname",
"age": "client.age",
"skills": [
{
"skillName": "client.skills[index].name",
"level": "client.skills[index].levelNumber",
"categories": [
{
"categoryName": "client.skills[index].categories[index].name",
"isImportant": "client.skills[index].categories[index].important"
}
]
}
]
}
And the second paramenter would be something like this (it's the object where you find the information.
{
"client": {
"firstName": "Jake",
"surname": "Long",
"age": 20,
"skills": [
{
"name": "Fly",
"level": 102,
"categories": [
{
"name": "air",
"important": true
},
{
"name": "superpower",
"important": false
}
]
},
{
"name": "FastSpeed",
"level": 163,
"categories": [
{
"name": "superpower",
"important": false
}
]
}
]
}
}
The idea it's: with de paths that I have in the first object, find it in the second one.. The problem I found it's when I have arrays, because when I defined the mapping rules I don't know how many positions will have the array I want to map. So in the mapping object (first) I'll only define the path but I'll not put it with the same lenght of the secondone because I don't know how much it will have.

Nested or different mongodb collections for e-commerce products with multi language?

He there, I use to work with Relational DBS, but right now trying to implement e-commerce shop and use mongodb.
I need the product, sub-products and description (multi lang);
I prefere to separate everything by 3 collection (maybe its not a good idea because mongo use 1 collection for one entity, in my example, 3 collection for 1 entity)
"content": [{
pid: 1,
lang: "ru",
title: "Привет"
},
{
pid: 1,
lang: "en",
title: "Hello"
},
{
pid: 2,
lang: "ru",
title: "Пока"
},
{
pid: 2,
lang: "en",
title: "Bye"
}
],
"products": [{
"_id": 1,
"item": "almonds",
"price": 12,
},
{
"_id": 2,
"item": "pecans",
"price": 20,
},
],
"sub": [{
"_id": 11,
"pid": 1,
"features": {
"color": ["red"],
"size": 42
},
"qt": 5
},
{
"_id": 12,
"pid": 1,
"features": {
"color": ["red"],
"size": 43
},
"qt": 2
},
{
"_id": 13,
"pid": 1,
"features": {
"color": ["yellow"],
"size": 44
},
"qt": 3
},
{
"_id": 21,
"pid": 2,
"features": {
"color": ["yellow"],
"size": 41
},
"qt": 6
},
{
"_id": 22,
"pid": 2,
"features": {
"color": ["red"],
"size": 47
},
"qt": 10
}
]
Products should have sub-products in order to use filter, for example when i want to filter items i will seek into sub-product collection find all the yellow t-short for example with size 44, then i just $group the items by main productId and make $lookup with main products and return it.
Also in order to receive main product with description I should to do $lookup with content collection.
Is it a great idea or should i use 1 collection for product and content?
Like:
"products": [{
"_id": 1,
"item": "almonds",
"price": 12,
"content": [{
lang: "ru",
title: "Привет"
},
{
lang: "en",
title: "Hello"
},
},
]
and maybe should I include sub-item also to main product, like:
"products": [{
"_id": 1,
"item": "almonds",
"price": 12,
"content": [{
lang: "ru",
title: "Привет"
},
{
lang: "en",
title: "Hello"
},
},
"sub": [{
"features": {
"color": ["red"],
"size": 42
},
"qt": 5
},
{
"features": {
"color": ["red"],
"size": 43
},
"qt": 2
},
]
]
The main Question is it good idea to compare everything and don't care about size of the collection? And if so how should I do a filter on nested documents ('sub-products')(previously 'sub-products' collection was like a plain collection and I could make aggregation in order to find all items by color for example: {"features.color": { $in: ['red'] }}) how can i manage it with nested document, and it will not be overwhelmed operation?

Your question about the data model design decision of how to split up your entities/documents cannot be answered in a useful way with just the information given, but I can give you some aspects to think about for the decision.
MongoDb is a key-value store and as such is most useful if you can design your data in a way that uses mostly key-based lookups. This can be extended with creating additional indexes on document fields other than _id, which is indexed by default. Everything that is not indexed will result in collection scans and be very inefficient to query. Indexes can substantially increase the amount of storage required, so depending on your scenario cost may be a prohibiting factor to just index every field you want to query by later. That means when designing entities you will also have to consider estimated size of collections and the possibility to reference between collections (by guid for example).
So in order to make the right design decisions for your data model we cannot judge just based on the data schema you want to store, but rather based on the queries you are looking to perform later and the expected collection sizes / future scaling requirements. Those are aspects that you only touch very lightly in your question. For example if you plan to query all kind of complex joins and combinations of property values across all your entities, you may ask yourself if you can afford the extra storage cost of non-normalized data and additional indexes, or if a traditional (or modern) SQL-RDB, or maybe a graph database may be more suitable than a key-value store for your use case. Whereas if your database scale will be small at all times and your main concern is developer productivity these considerations may be worthless.
Your specific question about accessing "sub"-documents within an array of the parent "products" can be answered by that it is supported by using an elemmatch. For example:
db.products.find({sub: {$elemMatch: {'features.size':43, 'features.color':'red'}}})
Please note again that these queries will only be efficient if you index the fields in your query. In case of array sub-documents that means looking into multi-key indexes.
In order to acquire more knowledge for better decisioning around DB models and your questions about queries in MongoDB I recommend reading the official MongoDB data model design guide, the documentation for querying arrays, as well as googling some articles on normalization and the motivation of SQL vs noSQL in terms of scaling/sharding, ACID and eventual consistency.

Query language for JSON or how to use Javascript/Lua in Python?

I'm looking solution to query JSON that extracts specific data.
This solution will be embedded inside our Python software (more exactly Python AWS Lambda) and the user should have the ability to specify such query string in GUI.
For example, we have such JSON and I would extract "nazwisko_weterynarza.value" from "files" array if "typ_dokument" is equal to "szczepienie" and "data_szczepienia" is equal to 3434343.
I've tried this using JSON Path but I've failed.
What about embedded Lua/Javascript in Python, but how to create a 'sandbox' to be sure that such a solution will be safe?
{
"files" :
[
{
"meta": {
"updated": 555555,
"attributes": [
{
"typ_dokumentu":
{
"formula_type": "NONE",
"formula": "",
"value_type": "string",
"value": "szczepienie",
"updated": 5345435435345
}
},
{
"data_szczepienia":
{
"formula_type": "NONE",
"formula": "",
"value_type": "int",
"value": 3434343,
"updated": 5345435225345
},
},
{
"nazwisko_weterynarza":
{
"formula_type": "NONE",
"formula": "",
"value_type": "string",
"value": "Nowak",
"updated": 5345435225345
}
}
]
}
},
{
"meta": {
"updated": 555555,
"attributes": [
{
"typ_dokumentu" :
{
"formula_type": "NONE",
"formula": "",
"value_type": "string",
"value": "Certyfikat urodzin",
"updated": 5345435435345
}
},
{
"data_urodzin" :
{
"formula_type": "NONE",
"formula": "",
"value_type": "int",
"value": 8888888,
"updated": 5345435225345
}
}
]
}
}
]
}

The AWS JSON (reference) path queries are not capable enough. Some more advanced JSONPath processors can do it, eg. using Jayway's JsonPath we could run a query like this:
$.files..meta.[?(#.attributes[*].typ_dokumentu.value contains 'szczepienie' && #.attributes[*].data_szczepienia.value contains '3434343')]..nazwisko_weterynarza.value
Reurns ["Nowak"] (You can try it online).
I don't know what you mean by embedded JavaScript/Lua, but in any event, you could create a web service that does the querying for you leveraging an implementation of your choice. Security wouldn't be an issue as long as your API is properly secured.

coding watson conversation for complex inputs?

I would like to know how can I integrate some coding (python or javascript) in my dialog, in order to avoid the very limited bluemix interface for example for loops, or text mining.
EX : I am creating a pizza chatbot and I need to be able to process this kind of request :
I would like 2 margarita whose one with extra cheese and the other with pepperoni and a regina, with 3 diet cokes and two beers
It is just impossible to do it with bluemix.
If anyone has a solution I would be very grateful.
Thanks

So the first thing to understand is that Conversation has good and bad use cases.
If your user can enter in structured data, then it is better to use a form. Be that in the conversation window (like Watson Virtual Agent), or a form on your site/app.
So your example doesn't look like a good use case.
That said...
With your example it's possible if your main items are entities, and you haver #sys-number system entity enabled. You can then check the entities array to see distance from each other.
Example
Created the following entities.
#PizzaType
#PizzaTopping
#Drinks
When I ask your question, the entities object returns the following:
[
{
"entity": "Drinks",
"location": [
104,
114
],
"value": "diet coke",
"confidence": 1
},
{
"entity": "PizzaType",
"location": [
72,
81
],
"value": "Pepperoni",
"confidence": 1
},
{
"entity": "Drinks",
"location": [
123,
128
],
"value": "beer",
"confidence": 1
},
{
"entity": "PizzaType",
"location": [
88,
94
],
"value": "Regina",
"confidence": 1
},
{
"entity": "PizzaTopping",
"location": [
46,
52
],
"value": "cheese",
"confidence": 1
},
{
"entity": "PizzaType",
"location": [
15,
24
],
"value": "Margarita",
"confidence": 1
},
{
"entity": "sys-number",
"location": [
13,
14
],
"value": "2",
"confidence": 1,
"metadata": {
"numeric_value": 2
}
},
{
"entity": "sys-number",
"location": [
31,
34
],
"value": "1",
"confidence": 1,
"metadata": {
"numeric_value": 1
}
},
{
"entity": "sys-number",
"location": [
101,
102
],
"value": "3",
"confidence": 1,
"metadata": {
"numeric_value": 3
}
},
{
"entity": "sys-number",
"location": [
119,
122
],
"value": "2",
"confidence": 1,
"metadata": {
"numeric_value": 2
}
}
]
A summary of that is:
You can see straight away there are some issues there. If the user doesn't mention a number, you have to account for that. Also something like this question would break it as well (unless you cater for it).
I would like margarita pizzas (two).
But the important point is to build your system with representative questions. Your example question may never be asked by an end user, or you can even shape the conversation to prevent such an input.

Convert Excel file to JSON: design of JSON code check

I want to convert the data from an Excel file to a JSON file. However, I'm not sure about the design of my JSON code (i.e. is it organized in a proper way in order to process it easily?)
I will use this JSON file with D3.js.
This a small part of my Excel file:
I'd like to convert this data into a JSON file in order to use it with D3.js. This is what I have so far:
So my question is: is this a good design (way) for organizing the data in order to use it with D3.js?
This is a sample output:
Thanks in advance!

This is a somewhat subjective question, but from my experience, there is a better way:
Since you're working in d3, you're probably doing something like this:
d3.selectAll('div')
.data(entities)
.enter()
.append('div')
...
So you want entities to be an array. The question is what are your entities? Is there a view where entities are all the countries in the world? Is there a view where entities are all the countries plus all the regions plus the whole world? Or, are all the views going to be simply all the countries in a selected region, not including the region itself?
The unless the JSON structure you're proposing matches the combinations of entities that you plan to display, your code will have to do a bunch of concat'ing and/or filtering of arrays in order to get a single entities array that you can bind to. Maybe that's ok, but it will create some unnecessary amount of coupling between your code and the structure of the data.
From my experience, it turns out that the most flexible way (and also probably the simplest in terms of coding) is to keep the hierarchy flat, like it is in the excel file. So, instead of encoding regions into the hierarchy, just have them in a single, flat array like so:
{
"immigration": [
{
"name": "All Countries"
"values: [
{ "Year": ..., "value": ... },
{ "Year": ..., "value": ... },
...
]
},
{
"name": "Africa"
"values: [
{ "Year": ..., "value": ... },
{ "Year": ..., "value": ... },
...
]
},
{
"name": "Eastern Africa"
"continent": "Africa"
"values": [
{ "Year": ..., "value": ... },
{ "Year": ..., "value": ... },
...
]
},
{
"name": "Burundi"
"continent": "Africa"
"region": "East Africa"
"values": [
{ "Year": ..., "value": ... },
{ "Year": ..., "value": ... },
...
]
},
{
"name": "Djibouti"
"continent": "Africa"
"region": "East Africa"
"values": [
{ "Year": ..., "value": ... },
{ "Year": ..., "value": ... },
...
]
},
...
]
}
Note that even though the array is flat, there is still a hierarchy here -- the region and sub-region properties.
You'll have to do a bit of filtering to extract just the countries/regions you want to show. But that's simpler than traversing the hierarchy you're proposing:
var africanEntities = data.immigration.filter(function(country) {
return country.continent == "Africa";
}); // Includes the region "East Africa"
var justCountries = data.immigration.filter(function(country) {
return country.continent != null && country.region != null;
});
Also, d3 has the awesome d3.nest(), which lets you turn this flat data into hierarchical one with little effort:
var countriesByContinent = d3.nest()
.key(function(d) { return d.continent; })
.map(data.immigration);
var africanEntities = countriesByContinent['Africa'];
Hope that helps....