Flatten D3.js Nested Data or Map it to new Dataset - javascript

Total noob to D3.js and working on creating my first grouped bar chart. However I'm having trouble making my data fit with the examples online. I am trying to use this example here, with my data that has been nested JSON with D3.
My problem is i cant use the d3.keys method to retrieve keys because my keys are not the state names. They are just Key.
Not to mention the second half forEach wont work because again the keys are not the State names, they are just the term key. So +d[name] will try d[MAPLE] when really my value is inside a key of d.values[(Get the Value where the key = name)]. Just really confused how to do this once the data has been nested in JSON
How would I go about getting all possible Key Values, and then mapping the keys with the next level of keys and values? Using a similar example as below but with my JSON nested data.
var ageNames = d3.keys(data[0]).filter(function(key) { return key !== "State"; });
data.forEach(function(d) {
d.ages = ageNames.map(function(name) { return {name: name, value: +d[name]}; });
});
My data is as so
{
"key": "1/10/2014",
"values": [
{
"key": "Texas",
"values": 200
},
{
"key": "Colorado",
"values": 300
},
{
"key": "Utah",
"values": 227
}
]
},{
"key": "2/10/2014",
"values": [
{
"key": "Texas",
"values": 225
},
{
"key": "Colorado",
"values": 241
},
{
"key": "Utah",
"values": 500
}
]
}

It's not clear from the question if the aim is to group by state ("Texas", "Colorado"...) or date ("1/10/2014", "2/10/2014"...) along the x-axis.
Assuming date (because that's how the data is currently structured), here's a working plunk: http://plnkr.co/edit/C8lkPMGanFY9BkTc6f1i?p=preview
The code that processes the data into a format that your existing D3 code for grouped bar chart can handle looks like this:
// Call the first mapping function on every 'date' in the data array
processed_data = data.map( function (d) {
var y0 = 0;
var total = 0;
return {
date: d.key,
// Call the second mapping function on every 'state' in the given date array
values: (d.values).map( function (d) {
return_object = {
state: d.key,
count: d.values,
y0: y0,
y1: y0 + d.values
};
// Calculate the updated y0 for each new state in a given date
y0 = y0 + d.values;
// Add the total for a given state to the sum total for that date
total = total + d.values;
return return_object;
}),
total: total
};
});
We use nested array.map transforms to manipulate your two-level nested data into the expected format and calculate y0, y1 and total values. Your processed_data object will then look like this:
The only other tricky bit will be to calculate your list of unique states, in order to define the color.domain. If you don't want to hard-code this (e.g. because the list may vary from dataset to dataset, you could use this approach:
// Define the color domain, by first building a list of states:
var states = [];
// Loop through each date
processed_data.forEach(
function (d) {
// Loop through each state in that date
d.values.forEach(
function(d) {
// add to the array if not already present
if (!(states.indexOf(d.state) > -1)) {
states.push(d.state)
}
}
)
}
);
color.domain(states);

Related

How to transform Javascript Object based on one of the column?

I am really struggling with the Object transformation. I have an array of Object and want to transform it into Highcharts multi line chart input.
For x-axis i need a object of unique dates.
For Y -axis i need a series of objects, each will have equal number of points as unique dates from step 1. For a ID and date if data is available it will add that value otherwise null.
Original:
[{
"date": "1997-09-29",
"Count": 100,
"ID": "AB12-R"
},
{
"date": "1997-12-30",
"Count": 104.7,
"ID": "AB13-R"
},
{
"date": "1997-12-30",
"Count": 1192,
"ID": "BA12-R"
},
{
"date": "1998-03-30",
"Count": 981,
"ID": "BA12-R"
},
{
"date": "1998-06-01",
"Count": 341,
"ID": "BA12-R"
}
]
Output:
[
{
Identiy: 'AB12-R',
data: [100, null, null, null]
},
{
Identiy: 'AB13-R',
data: [null, 104.7, null, null]
}, {
Identiy: 'BA12-R',
data: [null, 1192, 981, 341]
}
]
Explanation:
In Original i have 4 unique dates, Hence the length of data for each identity is 4. Now for each unique date i'll check if there is any entry in Original if yes i'll push it in data if not that null.
My Code:
From some help from, this is what i was trying. But this is not checking the unique dates and just creating the data object length which is equal to number of object in the Original. Here is that implementation https://jsfiddle.net/3u9penko/
const data = [{"date":"1997-09-29","Count":100,"ID":"AB12-R"},{"date":"1997-12-30","Count":104.7,"ID":"AB13-R"},{"date":"1997-12-30","Count":1192,"ID":"BA12-R"},{"date":"1998-03-30","Count":981,"ID":"BA12-R"},{"date":"1998-06-01","Count":341,"ID":"BA12-R"}];
const hcData = [];
data.forEach((d, i) => {
const checkIfExist = hcData.find(data => data.id === d["ID"])
if (checkIfExist) {
checkIfExist.data[i] = d["Count"];
} else {
const initialData = [...Array(data.length)]
initialData.fill(null, 0, data.length)
initialData[i] = d["Count"];
hcData.push({
data: initialData,
id: d["ID"]
})
}
})
console.log(hcData)
Create a Map of unique dates to index using a Set to get the unique dates, and then converting to a Map with the date as key, and the index as value.
Reduce the array to a Map using the ID as the key, and initializing the value with an empty array, that has the length of the dates Map size filled with null. Add the value of the current object to the array of nulls at the index of the date in the dates Map.
Convert the Map to an array of objects using Array.from().
const fn = arr => {
// create a Map of uniqe dates with their index
const dates = new Map(Array.from(
new Set(arr.map(o => o.date)),
(v, i) => [v, i]
))
return Array.from(
arr.reduce((acc, o, i) => {
// if an ID doesn't exist on the Map init it with an empty array of null values (counts)
if(!acc.has(o.ID)) acc.set(o.ID, new Array(dates.size).fill(null))
// add the current ID count to the array of counts
acc.get(o.ID)[dates.get(o.date)] = o.Count
return acc
}, new Map),
([Identity, data]) => ({ Identity, data }) // convert to an array of objects
)
}
const arr = [{"date":"1997-09-29","Count":100,"ID":"AB12-R"},{"date":"1997-12-30","Count":104.7,"ID":"AB13-R"},{"date":"1997-12-30","Count":1192,"ID":"BA12-R"},{"date":"1998-03-30","Count":981,"ID":"BA12-R"},{"date":"1998-06-01","Count":341,"ID":"BA12-R"}]
const result = fn(arr)
console.log(result)

Reducing redundant information in a JSON object with javascript and combining values

I have a CSV of results that looks at a picture and makes a guess at whether or not a picture contains a certain attribute. In this case if the subject in the picture is male or female.
I'm converting this CSV to JSON with javascript/node and I want to take the attributes and their values and put them in an array inside of one object per pciture. Right now each line of the CSV measures and attribute but it means at least two lines per image.
Simple version of the csv:
path, detect_id, score, x-coord, y-coord, w-coord, h-coord, attribute, value
picture_1.jpg,0,1.44855535,74,54,181,181,genderf,0.024716798
picture_1.jpg,0,1.44855535,74,54,181,181,genderm,0.975283206
I can convert this CSV to JSON and then at least group items together by their path/filename.
But that leaves a lot of redundant information out there and I want to put my Attributes and their Value together in a nested object inside of the main one.
Like:
Path: picture_1.jpg
Attributes: [genderf: 0.025,
genderm: 0.985]
other_info: other info
Right now I'm using lodash to create the objects as you see below but if I try to map through the attributes I end up pushing out every element except the last one.
So I can create the object with the following code.
var result =
_([...arr1, ...arr2])
.concat()
.groupBy("path")
.value();
Where arr1 and arr2 is the data from one line of the output csv. All the information is the same except the attribute and its value.
That gets me this object:
{
"picture_1.jpg": [
{
"path": "picture_1.jpg",
"detect_id,": "0",
"score,": "1.44855535",
"coordinates": [
{
"x,": "74",
"y,": "54",
"w": "181",
"h": "181"
}
],
"attribute": "genderf",
"value": "0.024716798"
},
{
"path": "picture_1.jpg",
"detect_id,": "0",
"score,": "1.44855535",
"coordinates": [
{
"x,": "74",
"y,": "54",
"w": "181",
"h": "181"
}
],
"attribute": "genderm",
"value": "0.975283206"
}
]
}
Which at least groups pictures together based on the path heading but a lot of the information is redundant and this is just measuring one attribute.
You could just iterate all csv-lines and build an object/map while keeping track of already found file-names/paths. If you encounter a line whose path already exists in the map, just append the attribute/value pair. Something like this (note that I've changed the coords delimiter for the sake of simplicity and that it needs proper error handling):
const data = ["picture_1.jpg,0,1.44855535,74;54;181;181,genderf,0.024716798", "picture_1.jpg,0,1.44855535,74;54;181;181,genderm,0.975283206"];
function createImageDataMap(dataArr) {
const imageDataResult = {};
for (const imgData of dataArr) {
const currData = parseImgDataLine(imgData);
if (!imageDataResult[currData.path]) {
imageDataResult[currData.path] = {
attributes: [], other_info: {
score: currData.score,
detectId: currData.detectId,
coords: currData.coords
}
}
}
imageDataResult[currData.path].attributes.push({[currData.attribute]: currData.value});
}
return imageDataResult;
}
function parseImgDataLine(line) {
const attributes = line.split(',');
return {
path: attributes[0],
detectId: attributes[1],
score: attributes[2],
coords: attributes[3],
attribute: attributes[4],
value: attributes[5]
}
}
console.log(JSON.stringify(createImageDataMap(data)));
// prints {"picture_1.jpg":{"attributes":[{"genderf":"0.024716798"},{"genderm":"0.975283206"}],"other_info":{"score":"1.44855535","detectId":"0","coords":"74;54;181;181"}}}

How to create a new json out of three jsons?

I have 3 different jsons, I need to extrapolate some data from each and create a new json with it. The three jsons have an id identifier in common, a unique identifier, so We could use that as a match since they are actually three different big jsons.
On json one we have "id":"265", on two and three "article_id":"265", so these can be the reference point when we loop.
I never worked with json this way so I wouldn't know how to approach it. I have put jQuery and JS as tags as they're what I know best.
1
{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
}
2
{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
}
3
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}
So the end result I am looking for is a single json exactly like this, we take id and title as objects from json 1, then we grab original_name from json two and year object from json three and we'll have:
{
"id":"265",
"title":"Battle of Gettysburg",
"original_name":"United States",
"year":"1863"
}
NOTE
The json above are just examples, in reality they are three huge lists, what I could do (manually), is to join them in order to have a single json.
There is some terminology confusion here; based on your comments you could be asking one of two very different questions. Fortunately one of them is very simple to answer so let's do both.
(I am handwaving past the details of loading json strings into the browser and converting them into javascript objects.)
If you have three objects
...then this is just a matter of plucking out the fields you need individually when constructing an output object:
var in1 = {
"id": "265",
"title": "Battle of Gettysburg",
"page_id": "4849",
"language_id": "en",
"original_time": "July 1\u20133, 1863"
};
var in2 = {
"id": "185",
"original_name": "United States",
"country_id": "24",
"article_id": "265"
}
var in3 = {
"id": "73",
"month": "July",
"year": "1863",
"suffix": "",
"article_id": "265"
}
// construct a new object using the selected fields
// from each object in1, in2, or in3:
var out = {
id: in1.id,
title: in1.title,
original_name: in2.original_name,
year: in3.year
}
console.log(out);
If you have three lists of objects:
...in this case it's a lot more complicated (and a lot more interesting). In this case you would need to match fields from the objects in each list which share the same IDs.
The following is definitely not the most efficient or memory-conserving way to do this; I've spread things out to (hopefully) make it easier to follow what it's doing.
I'm making two assumptions:
within each list, all IDs are unique (meaning you won't have two objects with the same ID in one JSON file)
Every ID will appear in all three lists (meaning you don't need to handle missing fields in output)
/* Again handwaving past loading JSON strings and parsing
them into javascript objects, we'll just start with
three arrays: */
var input1 = [{
"id": "265",
"title": "Battle of Gettysburg",
"page_id": "4849",
"language_id": "en",
"original_time": "July 1\u20133, 1863"
},
{
"id": "1",
"title": "Foo",
"page_id": "123",
"language_id": "en",
"original_time": "July 1\u20133, 1863"
}
];
var input2 = [{
"id": "1",
"original_name": "Bar",
"country_id": "24",
"article_id": "265"
},
{
"id": "265",
"original_name": "United States",
"country_id": "24",
"article_id": "265"
}
]
var input3 = [{
"id": "1",
"month": "July",
"year": "Baz",
"suffix": "",
"article_id": "265"
},
{
"id": "265",
"month": "July",
"year": "1863",
"suffix": "",
"article_id": "265"
}
]
/* It would be much easier to find corresponding IDs
across these arrays if they weren't arrays. We'll
start by converting them into objects keyed by the
item ids: */
var convertArray = function(arr) {
var output = {};
arr.forEach(function(o) {
output[o.id] = o;
});
return output;
}
var obj1 = convertArray(input1);
var obj2 = convertArray(input2);
var obj3 = convertArray(input3);
/* Now if we need to find (say) the object with id "foo", we don't
need to search the whole array, but can just use `obj1["foo"]` or
`obj1.foo`.
The last step is to iterate over the list of IDs and repeatedly
do basically the same thing as in the "if you have three objects"
part above. The only difference is that we need to access the
object with the same ID in each of the input lists: */
var constructOutput = function(in1, in2, in3) {
var output = []; // we'll be outputting a list of objects again.
// step through every ID (assuming in1 contains all of them):
Object.keys(in1).forEach(function(id) {
var obj = {
id: id,
title: in1[id].title,
original_name: in2[id].original_name,
year: in3[id].year
}
output.push(obj);
});
return output;
}
var final = constructOutput(obj1, obj2, obj3)
console.log(final)
Essentially what you have to do is mimic a SQL JOIN using JavaScript objects:
Use JSON.parse() on all three JSON collections to turn them into arrays of objects.
Iterate through JSON 1 objects; for each object...
Iterate through JSON 2 objects, testing if article ID matches the ID from JSON 1 that we are iterating over. Save this object.
Iterate through JSON 3 objects, testing if ID matches the ID of the object we found from JSON 2. Save this object.
After you have all three objects, make a new object literal that contains only the fields you want:
{
Id: obj1.id,
Title: obj1.title,
Original_name: obj2.original_name,
Year: obj3.year
}
Should you want to combine n number of JSON objects, e.g. a list of objects you can take a functional approach and utilise reduce + filter.
const data = [{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
},
{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
},
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}];
const final = data.reduce((accu, { id, title }, index, array) => {
// Find any related objects
const matches = array.filter(data => data.article_id === id);
if (matches.length) {
// Flatten them for ease of access. Duplicate keys will override.
const flat = matches.reduce((arr, item) => ({ ...arr, ...item }), [])
// Return new object
return accu.concat({
...flat,
id,
title,
});
}
return accu;
}, []);
console.log(final, '<<')
// Witness
document.getElementById('results').innerHTML = JSON.stringify(final);
<div id="results" style="font-family: Courier; font-size 14px; color: #fff; background: #000; padding: 20px; max-width: 80vw;"></div>
Edited*
Maybe this is what you need?
let arrPages = [{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
}];
let arrArticles = [{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
},
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}];
let getResult = (arrInput, arrCompare) => {
let joinedItems = [];
arrInput.forEach(item => {
let newItem = { id: item.id, title: item.title };
arrCompare.forEach(subItem => {
if(subItem.article_id !== undefined && subItem.article_id === item.id){
if(subItem.original_name !== undefined)
newItem.original_name = subItem.original_name;
if(subItem.year !== undefined)
newItem.year = subItem.year;
}
});
joinedItems.push(newItem);
});
return joinedItems;
};
let result = getResult(arrPages, arrArticles);
console.log(result);
In the first part of the code i create a var that has the json data.
To solve the problema i create 2 functions, the order of the creation dosen't metter, the first function getJSONData() take the json data as parameter and return a object filtered by the keys defined in the array keys. The secound function just check if the current key is present in the array of keys, this function could be replaced by the jQuery.inArray() method.
// JSON data
var json = [{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
},
{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
},
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}]
// keys that i want
var keys = ["title", "original_name", "year"];
// var that will have the filtered data
var newJSON = getJSONData(json);
console.log(JSON.stringify(newJSON))
// this is the main function of the code
// here we iterate in the json creating a new object that has all the tags definid in the keys array
function getJSONData(arrayJSON){
var JSONFiltered = {};
for(var i in arrayJSON){
for(var key in arrayJSON[i]){
if(hasElement(key)){
JSONFiltered[key] = arrayJSON[i][key];
}
}
}
return JSONFiltered;
}
// this function is used to check a key is present in the array of keys
function hasElement(key){
for(var elem in keys){
if(keys[elem] == key) return true;
}
return false;
}

Loading json data to an array in d3

so I am trying to assign json data to an array variable in d3.
Here is my json:
[
{
"Impressions": "273909",
"Clicks": "648",
"CPM": 4.6388278388278,
"Cost": 1266.4,
"CPC": 1.9543209876543,
"Campaign": "Campaign 1"
},
{
"Impressions": "974408",
"Clicks": "14571",
"CPM": 4.0175975359343,
"Cost": 3913.14,
"CPC": 0.26855672225654,
"Campaign": "Campaign 2"
},
{
"Impressions": "76751",
"Clicks": "5022",
"CPM": 8.4675,
"Cost": 643.53,
"CPC": 0.1281421744325,
"Campaign": "Campaign 3"
},
and here is my code to load the json dataset:
d3.json("DS003a_Adwords_AdPerformance_modified.json", function(error, data) {
var topData = data.sort(function(a, b){
return d3.descending(+a.cost, +b.cost);
}).slice(0,10);
topData.forEach(function (d) {
d.CampaignName = d.Campaign;
d.cost = d.Cost;
});
var cost = d3.nest()
.key(function(d) {return d.Cost;})
.entries(data); //fail
var p = d3.select("body").selectAll("p")
.data(topData)
.enter()
.append("p")
.text(function(d,i){
return (i+1) + ". " + d.CampaignName + " cost = " + cost[i];
});
I basically want to save the value of "Cost" to an array variable var cost.
But when I tried my code the result is as followed:
What should i do?
Thank you, your help is appreciated :)
You cannot use nest to directly have an array of values. The two possible output formats of nest are:
a large object
{
key1: value1,
key2: value2,
...
}
or an array of small objects
[
{ key: key1, values: value1 },
{ key: key2, values: value2 },
...
]
Neither is the one you desire. (Remember the first goal of nest: identify a set of keys, and group all pieces of data with the same key in a single batch, possibly with some transformation).
If for some reason you don't want to use your original array as suggested in the comments, then d3.map is what you're needing:
var cost = d3.map(data, function(d) {
return d.cost;
});
This is creating a copy of your cost data (if your data array changes, then you will need to run d3.map again to update your array). So you should use this array only locally if your data may not be constant. This is why in general one prefers using the original data directly, as it also saves this copy step and has less risks of tricky bugs later on.

Populate Highchart Column From JSON

I have to create a column chart in my project using Highchart. I am using $.ajax to populate this data. My current JSON data is like this :
[{
"city": "Tokyo",
"totalA": "10",
"totalB": "15"
},
{
"city": "Seoul",
"totalA": "20",
"totalB": "27"
},
{
"city": "New York",
"totalA": "29",
"totalB": "50"
}]
How to resulting JSON string look like this:
[{
"name": "city",
"data": ["Tokyo", "Seoul", "New York"]
}, {
"name": "totalA",
"data": [10, 20, 29]
}, {
"name": "totalB",
"data": [15, 27, 50]
}]
Thank you.
Assuming all the elements look the same (they all have the same fields): Live Example
// src is the source array
// Get the fields from the first element
var fields = Object.keys(src[0]);
// Map each key to...
var result = fields.map(function(field) {
// Grab data from the original source array
var data = src.reduce(function(result, el) {
// And create an array of data
return result.concat(el[field]);
}, []);
// Format it like you want
return {name: field, data: data};
});
console.log(result);
If they aren't, the code is slightly more complicated: Live Example
// Work out the fields by iterating all of the elements
// and adding the keys that weren't found yet to an array
var fields = src.reduce(function (fields, current) {
return fields.concat(Object.keys(current).filter(function (key) {
return fields.indexOf(key) === -1;
}));
}, []);
var result = fields.map(function (field) {
// Need another step here, filter out the elements
// which don't have the field we care about
var data = src.filter(function (el) {
return !!el[field];
})
// Then continue like in the example above.
.reduce(function (result, el) {
return result.concat(el[field]);
}, []);
return {
name: field,
data: data
};
});
console.log(result);

Categories