I am working with d3.js to visualise families of animals (organisms) (up to 4000 at a time) as a tree graph, though the data source could just as well be a directory listing, or list of namespaced objects. my data looks like:
json = {
organisms:[
{name: 'Hemiptera.Miridae.Kanakamiris'},
{name: 'Hemiptera.Miridae.Neophloeobia.incisa'},
{name: 'Lepidoptera.Nymphalidae.Ephinephile.rawnsleyi'},
... etc ...
]
}
my question is: I am trying to find the best way to convert the above data to the hierarchical parent / children data structure as is used by a number of the d3 visualisations such as treemap (for data example see flare.json in the d3/examples/data/ directory).
Here is an example of the desired data structure:
{"name": "ROOT",
"children": [
{"name": "Hemiptera",
"children": [
{"name": "Miridae",
"children": [
{"name": "Kanakamiris", "children":[]},
{"name": "Neophloeobia",
"children": [
{"name": "incisa", "children":[] }
]}
]}
]},
{"name": "Lepidoptera",
"children": [
{"name": "Nymphalidae",
"children": [
{"name": "Ephinephile",
"children": [
{"name": "rawnsleyi", "children":[] }
]}
]}
]}
]}
}
EDIT: enclosed all the original desired data structure inside a ROOT node, so as to conform with the structure of the d3 examples, which have only one master parent node.
I am looking to understand a general design pattern, and as a bonus I would love to see some solutions in either javascript, php, (or even python). javascript is my preference.
In regards to php: the data I am actually using comes from a call to a database by a php script that encodes the results as json.
database results in the php script is an ordered array (see below) if that is any use for php based answers.
Array
(
[0] => Array
(
['Rank_Order'] => 'Hemiptera'
['Rank_Family'] => 'Miridae'
['Rank_Genus'] => 'Kanakamiris'
['Rank_Species'] => ''
) ........
where:
'Rank_Order' isParentOf 'Rank_Family' isParentOf 'Rank_Genus' isParentOf 'Rank_Species'
I asked a similar question focussed on a php solution here, but the only answer is not working on my server, and I dont quite understand what is going on, so I want to ask this question from a design pattern perspective, and to include reference to my actual use which is in javascript and d3.js.
The following is specific to the structure you've provided, it could be made more generic fairly easily. I'm sure the addChild function can be simplified. Hopefully the comments are helpful.
function toHeirarchy(obj) {
// Get the organisms array
var orgName, orgNames = obj.organisms;
// Make root object
var root = {name:'ROOT', children:[]};
// For each organism, get the name parts
for (var i=0, iLen=orgNames.length; i<iLen; i++) {
orgName = orgNames[i].name.split('.');
// Start from root.children
children = root.children;
// For each part of name, get child if already have it
// or add new object and child if not
for (var j=0, jLen=orgName.length; j<jLen; j++) {
children = addChild(children, orgName[j]);
}
}
return root;
// Helper function, iterates over children looking for
// name. If found, returns its child array, otherwise adds a new
// child object and child array and returns it.
function addChild(children, name) {
// Look for name in children
for (var i=0, iLen=children.length; i<iLen; i++) {
// If find name, return its child array
if (children[i].name == name) {
return children[i].children;
}
}
// If didn't find name, add a new object and
// return its child array
children.push({'name': name, 'children':[]});
return children[children.length - 1].children;
}
}
Given your starting input I believe something like the following code will produce your desired output. I don't imagine this is the prettiest way to do it, but it's what came to mind at the time.
It seemed easiest to pre-process the data to first split up the initial array of strings into an array of arrays like this:
[
["Hemiptera","Miridae","Kanakamiris" ],
["Hemiptera","Miridae","Neophloeobia","incisa" ],
//etc
]
...and then process that to get a working object in a form something like this:
working = {
Hemiptera : {
Miridae : {
Kanakamiris : {},
Neophloeobia : {
incisa : {}
}
}
},
Lepidoptera : {
Nymphalidae : {
Ephinephile : {
rawnsleyi : {}
}
}
}
}
...because working with objects rather than arrays makes it easier to test whether child items already exist. Having created the above structure I then process it one last time to get your final desired output. So:
// start by remapping the data to an array of arrays
var organisms = data.organisms.map(function(v) {
return v.name.split(".");
});
// this function recursively processes the above array of arrays
// to create an object whose properties are also objects
function addToHeirarchy(val, level, heirarchy) {
if (val[level]) {
if (!heirarchy.hasOwnProperty(val[level]))
heirarchy[val[level]] = {};
addToHeirarchy(val, level + 1, heirarchy[val[level]]);
}
}
var working = {};
for (var i = 0; i < organisms.length; i++)
addToHeirarchy(organisms[i], 0, working);
// this function recursively processes the object created above
// to create the desired final structure
function remapHeirarchy(item) {
var children = [];
for (var k in item) {
children.push({
"name" : k,
"children" : remapHeirarchy(item[k])
});
}
return children;
}
var heirarchy = {
"name" : "ROOT",
"children" : remapHeirarchy(working)
};
Demo: http://jsfiddle.net/a669F/1/
An alternative answer to my own question....In the past day I have learn't a great deal more about d3.js and in relation to this question d3.nest() with .key() and .entries() is my friend (all d3 functions).
This answer involves changing the initial data, so it may not qualify as a good answer to the specific question i asked. However if someone has a similar question and can change things on the server then this is a pretty simple solution:
return the data from the database in this format:
json = {'Organisms': [
{ 'Rank_Order': 'Hemiptera',
'Rank_Family': 'Miridae',
'Rank_Genus': 'Kanakamiris',
'Rank_Species': '' },
{}, ...
]}
Then using d3.nest()
organismNest = d3.nest()
.key(function(d){return d.Rank_Order;})
.key(function(d){return d.Rank_Family;})
.key(function(d){return d.Rank_Genus;})
.key(function(d){return d.Rank_Species;})
.entries(json.Organism);
this returns:
{
key: "Hemiptera"
values: [
{
key: "Cicadidae"
values: [
{
key: "Pauropsalta "
values: [
{
key: "siccanus"
values: [
Rank_Family: "Cicadidae"
Rank_Genus: "Pauropsalta "
Rank_Order: "Hemiptera"
Rank_Species: "siccanus"
AnotherOriginalDataKey: "original data value"
etc etc, nested and lovely
This returns something very much similar to they array that I described as my desired format above in the question, with a few differences. In particular, There is no all enclosing ROOT element and also whereas they keys I originally wanted were "name" and "children" .nest() returns keys as "key" and "values" respectively.
These alternatives keys are easy enough to use in d3.js by just defining appropriate data accessor functions (basic d3 concept) ... but that is getting beyond the original scope of the question ... hope that helps someone too
Related
I have a JSON data set as follows:
{
"content":[],
"layout":[],
"trail":[
{
"content":[
{
"type":"image",
"media":[
{
"type":"image/jpg",
"width":593,
"height":900,
"url":"https://live.staticflickr.com/65535/48208920877_e6b234d3ea_c_d.jpg",
"flickr":{
"flickr-post":"https://www.flickr.com/photos/riketrs/48208920877",
"flickr-album":"https://www.flickr.com/photos/riketrs/albums/72157709130951466"
}
}
]
},
{
"type":"image",
"media":[
{
"type":"image/jpg",
"width":1600,
"height":900,
"url":"https://live.staticflickr.com/2817/33807326532_91013ef6b1_h_d.jpg",
"flickr":{
"flickr-post":"https://www.flickr.com/photos/146758538#N03/33807326532",
"flickr-album":"https://www.flickr.com/photos/146758538#N03/albums/72157681438471236"
}
}
]
}
],
"colors":{
"c0":"#1e1e1d",
"c1":"#78736f",
"c2":"#b2a89f"
}
}
]
}
I would like to console.log the "url" key for each of the images shown here.
(https://live.staticflickr.com/65535/48208920877_e6b234d3ea_c_d.jpg and https://live.staticflickr.com/2817/33807326532_91013ef6b1_h_d.jpg)
I tried some code but I'm very new to JSON in general, I've looked at some other answers to do with JSON but I'm not quite sure how to achieve what I want.
JSFiddle: https://jsfiddle.net/fj6qveh1/1/
I appreciate all advice, including links to other answers that I potentially missed.
Thank you!
url is a property of an object. There can be many of these in a media array. (This data only shows one object per array.) media itself is an property of objects inside the content array.
Use map, and flatMap.
map to return the URL values from the objects in media, and flatMap to return a flat array of the nested arrays returned by map.
const data={content:[],layout:[],trail:[{content:[{type:"image",media:[{type:"image/jpg",width:593,height:900,url:"https://live.staticflickr.com/65535/48208920877_e6b234d3ea_c_d.jpg",flickr:{"flickr-post":"https://www.flickr.com/photos/riketrs/48208920877","flickr-album":"https://www.flickr.com/photos/riketrs/albums/72157709130951466"}}]},{type:"image",media:[{type:"image/jpg",width:1600,height:900,url:"https://live.staticflickr.com/2817/33807326532_91013ef6b1_h_d.jpg",flickr:{"flickr-post":"https://www.flickr.com/photos/146758538#N03/33807326532","flickr-album":"https://www.flickr.com/photos/146758538#N03/albums/72157681438471236"}},{type:"image/jpg",width:1600,height:900,url:"https://live.dummyimage.com/2817/dummy.jpg",flickr:{"flickr-post":"https://www.flickr.com/photos/146758538#N03/33807326532","flickr-album":"https://www.flickr.com/photos/146758538#N03/albums/72157681438471236"}}]}],colors:{c0:"#1e1e1d",c1:"#78736f",c2:"#b2a89f"}}]};
const content = data.trail[0].content;
const urls = content.flatMap(obj => {
return obj.media.map(inner => inner.url);
});
console.log(urls)
The easiest way is to use map function. Given that you are very new to programming (the solution has little to do with JSON itself, since the first step is to parse JSON string to a JavaScript object), it would be better if you try yourself. But you start with
let urls = trail["content"].map(x => x["media"][0]["url"])
for more about map function look here
There is a table in the table so for each table:
for(let i in trail){
var content = trail[i]["content"];
content.forEach(content => content.media.forEach(media => console.log(media.url)))
}
To access object properties, you can use a dot (.), and to access an array element, you use its index in square brackets ([]). So you just keep repeating these steps as necessary until you get to the content you're looking for.
Here's how that looks on a simplified version of your object, using the forEach method of arrays to apply a custom function to each item in the content array:
const json = getJson();
json.trail[0].content.forEach(item=>console.log(item.media[0].url));
function getJson(){
let obj = {
"trail": [{
"content": [
{ "media": [{ "url":"image #65535/48208920877_e6b234d3ea_c_d.jpg" }]},
{ "media": [{"url":"image #2817/33807326532_91013ef6b1_h_d.jpg"}]}
]
}]
};
return obj;
}
I have an HTML page that contains a stringified JSON object. The object has this structure:
{
"x":{
"key1":[],
"key2":{},
"keyN":{},
"myKey":{
"randomID238492":{
"items":[
{ "value":"zzzz" },
{ "value":"aaaa" },
{ ...}
]
}
}
}
}
I want to replace this object with one in which the "items" array has been sorted. Here is what I will and won't know about the object:
"myKey" and "items" will always be the relevant object keys
"myKey" will contain only one random ID, and the "items" key will always be its first child
I won't know the order of "myKey" in the object.
I won't know the true randomID under which "items" nests.
Is there a clear, efficient way to replace this JSON object with one in which "items" has been sorted? Right now, I do it by using this jQuery function after the page has rendered:
$(function() {
var myData = $( "#myJSON_string" )[0]; // <script> node that contains the string
var myDataJSON = JSON.parse(myData.innerText); // JSON string
var myKeyJSON = myDataJSON["x"]["myKey"]; // object
var myArr = myKeyJSON[Object.keys(myKeyJSON)[0]]["items"]; // array to sort
// Now sort and revise. I'm leaving myCompare() out of the example for brevity
myKeyJSON[Object.keys(myKeyJSON)[0]]["items"] = myArr.sort(myCompare);
myDataJSON["x"]["myKey"] = myKeyJSON;
myDataJSON = JSON.stringify(myDataJSON);
myData.innerText = myDataJSON;
});
This approach works, but it seems rather labored. It might be better, for example, if I could revise the JSON object "in place" without parsing it and then re-stringifying it.
Many SO posts, like this one, speak to the general question of how to sort a JSON array. But I can't see that any speak to the specific question posed here.
I’m trying to learn Lodash right now because I think it could help a lot with my current project. However, I have an issue that I can’t figure out how to solve, and, since I’m a Lodash rookie, I thought I’d post something to see if I could get some guidance.
I have two complex objects (see the structure below) – made using angular.copy(original, copy) (my app is, obviously, an Angular app).
After the copy, both
original and copy =
{
name: 'name',
date: 'date',
rows: [ // array of objects
{
// this is row 1
columns: [ // array of objects
{
// this is column 1
widgets: [ // array of objects
{
// this is 1 widget in this column
createdDate: 'createdDate1',
widgetName: 'widgetName1',
widgetParameters: [ // array of objects
{ parameterName: 'parameterName1', parameterValue: 'parameterValue1' },
{ parameterName: 'parameterName2', parameterValue: 'parameterValue2' },
{ parameterName: 'parameterName3', parameterValue: 'parameterValue3' }
]
}
]
},
{
// this is column 2
widgets: [
{
// this is 1 widget in this column
createdDate: 'createdDate2',
widgetName: 'widgetName2',
widgetParameters: [ // array of objects
{ parameterName: 'parameterName1a', parameterValue: 'parameterValue1a' },
{ parameterName: 'parameterName2a', parameterValue: 'parameterValue2a' }
]
},
{
// this is 2 widget in this column
// ...
}
]
},
{
// this is column 3
widgets: [
{
// this is 1 widget in this column
createdDate: 'createdDate3',
widgetName: 'widgetName3',
widgetParameters: [ // array of objects
{ parameterName: 'parameterName1b', parameterValue: 'parameterValue1a' }
]
},
{
// this is 2 widget in this column
// ...
},
{
// this is 3 widget in this column
// ...
}
]
}
] // end columns array
},
{
// this is row 2
// ... same structure as above with columns, widgets, widgetParameters
}
] // end rows array
}
After some further processing and changing of property values, I now need to re-sync some of the properties (not all of them) in the copy object back into the original object. Specifically, I need to go thru the entire copy object, find all “widgets”, get the “widgetName” value and the complete “widgetParameters” collection from the widget, then go thru the entire original object, find the exact matching “widget”, and update the matching widget's “widgetName” and “widgetParameters” with the values from the copy object.
I’ve got it working with a brute force approach with lots of forEach’ing (see below), but I’m thinking there is probably a more elegant and efficient way in Lodash to do what I’m trying to do.
// brute force it
angular.forEach(copy.rows, function (row) {
angular.forEach(row.columns, function (column) {
angular.forEach(column.widgets, function (widget) {
var widgetName = widget.widgetName;
var createdDate = widget.createdDate;
var widgetParameters = widget.widgetParameters;
angular.forEach(original.rows, function (row1) {
angular.forEach(row1.columns, function (column1) {
angular.forEach(column1.widgets, function (widget1) {
if ((widgetName === widget1.widgetName) && (createdDate === widget1.createdDate))
{
widget1.widgetName = widgetName;
angular.copy(widgetParameters, widget1.widgetParameters);
}
});
});
});
});
});
});
Any thoughts, help, ideas, etc.?
Thanks!
I have had great success with the JavaScript-based deep-diff utility. For example:
// this would be a node.js way to require it, or you could use the DeepDiff object in the browser's global window object
var diff = require('deep-diff').diff;
var differences = diff(original, copy);
If you wanted to apply some changes at the browser level, you could iterate over the changes (differences) and do something like this for each difference received:
DeepDiff.applyChange(original, {}, difference);
I don't know of an elegant and easy way to do this in Lodash, without rebuilding the functionality of the deep-diff library.
What you could do is build the smaller object that contains only the things you want to keep, let's call it diffObject, so then you only need to do:
// Puts the diff back into the original object
_.merge( originalObject, diffObject );
Now to construct such diff object you may still need to go through all your object, or have some clever recursive building function that keeps track of the path it has gone through and all.
As Iam new to javascript, I found handleBar.js can be used to template with dynamic data.
I worked on a sample which worked fine and the json structure was simple and straight forward.
(function()
{
var wtsource = $("#some-template").html();
var wtTemplate = Handlebars.compile(wtsource);
var data = { users: [
{url: "index.html", name: "Home" },
{url: "aboutus.html", name: "About Us"},
{url: "contact.html", name: "Contact"}
]};
Handlebars.registerHelper('iter', function(context, options) {
var fn = options.fn, inverse = options.inverse;
var ret = "";
if(context && context.length > 0) {
for(var i=0, j=context.length; i<j; i++) {
ret = ret + fn($.extend({}, context[i], { i: i, iPlus1: i + 1 }));
}
} else {
ret = inverse(this);
}
return ret;
});
var temp=wtTemplate(data);
$("#content").html(temp);
})();
<script id="some-template" type="text/x-handlebars-template">
{{#iter users}}
<li>
{{name}}
</li>
{{/iter}}
</script>
How to iterate a json with the below structure ? Please do suggest the possible way for iterating and creating the template for the below json structure
var newData = { "NEARBY_LIST": {
"100": {
"RestaurantID": 100,
"ParentRestaurantID": 0,
"RestaurantName": "Chennai Tiffin",
"listTime": [{
"startTime": "10:00",
"closeTime": "23:30"
} ]
},
"101": {
"RestaurantID": 101,
"ParentRestaurantID": 0,
"RestaurantName": "Biriyani Factory",
"listTime": [{
"startTime": "11:00",
"closeTime": "22:00"
}]
}
}
};
Accessing the properties of an object has nothing to do with Handlebars. If you dealing with JSON and you wish to access it in general bracket or dot notation, you must first parse the JSON into a JavaScript object using the JSON.parse() function.
After this is done, you may access the properties as follows.
var property = newData['NEARBY_LIST']['100'].RestaurantName; // "Chennai Tiffin"
Here is a fiddle to illustrate.
http://jsfiddle.net/qzm0cygu/2/
I'm not entirely sure what you mean, but if your question is how you can use/read the data in newData, try this:
newData = JSON.parse(newData); //parses the JSON into a JavaScript object
Then access the object like so:
newData.NEARBY_LIST //the object containing the array
newData.NEARBY_LIST[0] //the first item (key "100")
newData.NEARBY_LIST[1] //the second item (key "101")
newData.NEARBY_LIST[0][0] //the first field of the first item (key "RestaurantID", value "100")
newData.NEARBY_LIST[0][2] //the third field of the first item (key "RestaurantName", value "Chennai Tiffin")
newData.NEARBY_LIST[0][3][0] //the first field of the fourth field of the first item (key "startTime", value "11:00")
I hope this was what you were looking for.
EDIT: as Siddharth points out, the above structure does assume you have arrays. If you are not using arrays you can access the properties by using their names as if they're in an associative array (e.g. newData["NEARBY_LIST"]["100"]. The reason I say "properties" and "as if" is because technically JavaScript doesn't support associative arrays. Because they are technically properties you may also access them like newData.NEARBY_LIST (but I don't recommend that in this case as a property name may not start with a number, so you would have to use a mix of the different notations).
On that note, I would recommend using arrays because it makes so many things easier (length checks, for example), and there are practically no downsides.
EDIT2: also, I strongly recommend using the same camelcasing conventions throughout your code. The way you currently have it (with half your properties/variables starting with capitals (e.g. "RestaurantName", "RestaurantID") and the other half being in lowerCamelCase (e.g. "listTime", "startTime")) is just asking for people (you or colleagues) to make mistakes.
I've looked around for an answer, but I think this is a kind of weird question. How would I convert, as a text file using tabs for spacing, this:
parent
child
child
parent
child
grandchild
grandhcild
to
{
"name" : "parent",
"children" : [
{"name" : "child"},
{"name" : "child"},
]
},
{
"name" : "parent",
"children" : [
{
"name" : "child",
"children" : [
{"name" : "grandchild"},
{"name" : "grandchild"},
{"name" : "grandchild"},
]
},
]
}
JSON probably isn't perfect, but hopefully makes my point clear.
i've had the same problem. Here is the solution:
function node(title,lvl){
var children = [],
parent = null;
return {
title:title,
children:children,
lvl:()=>lvl==undefined?-1:lvl,
parent:()=>parent, //as a function to prevent circular reference when parse to JSON
setParent:p=>{parent=p},
appendChildren: function(c){
children.push(c);
c.setParent(this);
return this
},
}
}
function append_rec(prev,curr) {
if(typeof(curr)=='string'){ //in the recursive call it's a object
curr = curr.split(' ');//or tab (\t)
curr = node(curr.pop(),curr.length);
}
if(curr.lvl()>prev.lvl()){//curr is prev's child
prev.appendChildren(curr);
}else if(curr.lvl()<prev.lvl()){
append_rec(prev.parent(),curr) //recursive call to find the right parent level
}else{//curr is prev's sibling
prev.parent().appendChildren(curr);
}
return curr;
}
root = node('root');
var txt =
`parent
child
child
parent
child
grandchild
grandhcild`;
txt.toString().split('\n').reduce(append_rec,root);
console.log(JSON.stringify(root.children,null,3));
I've just implemented this feature for the tabdown markup language — it does exactly what you sought for.
https://github.com/antyakushev/tabdown
Usage is pretty simple:
var lines = data.toString().split('\n');
var tree = tabdown.parse(lines);
console.log(tree.toString());
You can also use the parse function outside of node.js, it does not depend on any modules.
This is my regex-based, recursion-free approach. It looks a bit "hacky" but makes perfect sense, you can try each step on regexr if you want to. It's written purposely verbose and can probably be compressed a bit. Also, this code assumes your text is tab-indented and only has one "parent", but you should be able to easily replace your indents and add a single "root" parent beforehand.
const string = `
parent
child
grandchild
child
child
grandchild
grandchild
`;
let json = string
.replace(
/(?:(\t+)(\S+)(?=(?:\n(?:(?:(?!\1))|(?:\1\S)))|$))/g,
"$1{\n$1\t\"name\": \"$2\",\n$1\t\"children\": []\n$1},"
) // this one replaces all empty nodes with a simple object with an empty children array
.replace(
/(?<=(^\t*))([^\s{]+)$\n(?=\1\t)/gm,
"{\"name\": \"$2\",\"children\": [\n"
); // this one replaces every immediate parent with an object and a starting children array
const lines = string.split("\n");
const maxDepth = Math.max(
...lines.map(line => line.replace(/[^\t]/g, "").length)
);
// this one basically closes all square brackets and curly braces
// this is a loop because it depends on the max depth of your source text and i also don't like recursion
for (let index = 0; index < maxDepth - 1; index++) {
json = json.replace(
/(^\t+)(.*,)("children": \[)((\n\1\t+[^\t\n]+)+)/gm,
"$1$2\n$1$3$4\n$1]},"
)
}
// this closes the root object brackets and removes trailing commas and newlines
json = `${json}\n]}`.replace(/,(?=\s*\])/g, "").replace(/\n/g, "");
const object = JSON.parse(json);
const betterLookingJson = JSON.stringify(object, null, "\t");
console.log(object);
console.log(betterLookingJson);
Generate JSON from Tab Tree Text File
The links below attack your problem specifically. All you need to do is update the code so the output is formatted to your requirements.
Python file parsing: Build tree from text file
Creating a tree/deeply nested dict from an indented text file in python
Parse Tree - Ruby example for parsing a tab tree. Created for #ruby on FreeNode.
Tab Delimiter to JSON
Converting Tab Delimited Textfile to JSON - A Python solution you could adapt.
Convert CSV To JSON - Online converter with many options.
Mr. Data Converter - Not an exact solution but you could adapt the code (fork on GitHub).
Other Help
How convert tsv to Json
Convert comma separated list into JSON using Javascript