We Keep Coding

Reducing redundant information in a JSON object with javascript and combining values

I have a CSV of results that looks at a picture and makes a guess at whether or not a picture contains a certain attribute. In this case if the subject in the picture is male or female.
I'm converting this CSV to JSON with javascript/node and I want to take the attributes and their values and put them in an array inside of one object per pciture. Right now each line of the CSV measures and attribute but it means at least two lines per image.
Simple version of the csv:
path, detect_id, score, x-coord, y-coord, w-coord, h-coord, attribute, value
picture_1.jpg,0,1.44855535,74,54,181,181,genderf,0.024716798
picture_1.jpg,0,1.44855535,74,54,181,181,genderm,0.975283206
I can convert this CSV to JSON and then at least group items together by their path/filename.
But that leaves a lot of redundant information out there and I want to put my Attributes and their Value together in a nested object inside of the main one.
Like:
Path: picture_1.jpg
Attributes: [genderf: 0.025,
genderm: 0.985]
other_info: other info
Right now I'm using lodash to create the objects as you see below but if I try to map through the attributes I end up pushing out every element except the last one.
So I can create the object with the following code.
var result =
_([...arr1, ...arr2])
.concat()
.groupBy("path")
.value();
Where arr1 and arr2 is the data from one line of the output csv. All the information is the same except the attribute and its value.
That gets me this object:
{
"picture_1.jpg": [
{
"path": "picture_1.jpg",
"detect_id,": "0",
"score,": "1.44855535",
"coordinates": [
{
"x,": "74",
"y,": "54",
"w": "181",
"h": "181"
}
],
"attribute": "genderf",
"value": "0.024716798"
},
{
"path": "picture_1.jpg",
"detect_id,": "0",
"score,": "1.44855535",
"coordinates": [
{
"x,": "74",
"y,": "54",
"w": "181",
"h": "181"
}
],
"attribute": "genderm",
"value": "0.975283206"
}
]
}
Which at least groups pictures together based on the path heading but a lot of the information is redundant and this is just measuring one attribute.

You could just iterate all csv-lines and build an object/map while keeping track of already found file-names/paths. If you encounter a line whose path already exists in the map, just append the attribute/value pair. Something like this (note that I've changed the coords delimiter for the sake of simplicity and that it needs proper error handling):
const data = ["picture_1.jpg,0,1.44855535,74;54;181;181,genderf,0.024716798", "picture_1.jpg,0,1.44855535,74;54;181;181,genderm,0.975283206"];
function createImageDataMap(dataArr) {
const imageDataResult = {};
for (const imgData of dataArr) {
const currData = parseImgDataLine(imgData);
if (!imageDataResult[currData.path]) {
imageDataResult[currData.path] = {
attributes: [], other_info: {
score: currData.score,
detectId: currData.detectId,
coords: currData.coords
}
}
}
imageDataResult[currData.path].attributes.push({[currData.attribute]: currData.value});
}
return imageDataResult;
}
function parseImgDataLine(line) {
const attributes = line.split(',');
return {
path: attributes[0],
detectId: attributes[1],
score: attributes[2],
coords: attributes[3],
attribute: attributes[4],
value: attributes[5]
}
}
console.log(JSON.stringify(createImageDataMap(data)));
// prints {"picture_1.jpg":{"attributes":[{"genderf":"0.024716798"},{"genderm":"0.975283206"}],"other_info":{"score":"1.44855535","detectId":"0","coords":"74;54;181;181"}}}

Group and count json data in queue() in d3.js

I am trying to change the format of a json file sourcing a D3 map. The original source is World Choropleth by palewire.
EDIT: working code thanks to below answer in the Plunker:
http://embed.plnkr.co/JYTATyrksAg3OJ0scHAp/
The original json is in a nested format with a count for each value:
{
"id": "IQ-G02-D009",
"rate": "1"
},
{
"id": "IQ-G05-D030",
"rate": "4"
},
{
"id": "IQ-G07-D047",
"rate": "5"
}
]
The new json will use a flat format, something like:
[
{
"id": "IQ-G02-D009"
},
{
"id": "IQ-G05-D030"
},
{
"id": "IQ-G05-D030"
},
{
"id": "IQ-G05-D047"
},
{
"id": "IQ-G07-D047"
}
]
It seems using a rollup and nest function would be the way forward to get the new json in a similar form of the former one, but am getting stuck at implementing those in a queue().
The rollup should be implemented at the .defer level it seems:
queue()
.defer(d3.json, "map.json")
.defer(d3.json, "data.json")
.await(ready);
Can I use the nest and rollup function directly in the queue?

You had some typo's working against you. The general answer to your question, as cal_br_mar said: apply nest and roll up when your data is loaded. So:
queue()
.defer(d3.json, "map.json")
.defer(d3.json, "data.json")
.await(ready)
function ready(error, country, activities) {
activities = d3.nest()
.key(function(d) {
return d.id;
})
.rollup(function(v) {
return v.length;
})
.entries(activities);
}
You had a typo in your last line: it should be activities, not expenses (wrong copy and paste, I guess).
And in the code that follows it should be:
activities.forEach(function(d) {
rateById[d.key] = +d.values;
});
The nest() and rollup puts your data in key and values properties, as you can see in your console output of the nested data:
[{"key":"IQ-G02-D009","values":1},{"key":"IQ-G05-D030","values":2},{"key":"IQ-G05-D047","values":1},{"key":"IQ-G07-D047","values":1}]
Change those fields and you should be fine.

How to create a new json out of three jsons?

I have 3 different jsons, I need to extrapolate some data from each and create a new json with it. The three jsons have an id identifier in common, a unique identifier, so We could use that as a match since they are actually three different big jsons.
On json one we have "id":"265", on two and three "article_id":"265", so these can be the reference point when we loop.
I never worked with json this way so I wouldn't know how to approach it. I have put jQuery and JS as tags as they're what I know best.
1
{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
}
2
{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
}
3
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}
So the end result I am looking for is a single json exactly like this, we take id and title as objects from json 1, then we grab original_name from json two and year object from json three and we'll have:
{
"id":"265",
"title":"Battle of Gettysburg",
"original_name":"United States",
"year":"1863"
}
NOTE
The json above are just examples, in reality they are three huge lists, what I could do (manually), is to join them in order to have a single json.

There is some terminology confusion here; based on your comments you could be asking one of two very different questions. Fortunately one of them is very simple to answer so let's do both.
(I am handwaving past the details of loading json strings into the browser and converting them into javascript objects.)
If you have three objects
...then this is just a matter of plucking out the fields you need individually when constructing an output object:
var in1 = {
"id": "265",
"title": "Battle of Gettysburg",
"page_id": "4849",
"language_id": "en",
"original_time": "July 1\u20133, 1863"
};
var in2 = {
"id": "185",
"original_name": "United States",
"country_id": "24",
"article_id": "265"
}
var in3 = {
"id": "73",
"month": "July",
"year": "1863",
"suffix": "",
"article_id": "265"
}
// construct a new object using the selected fields
// from each object in1, in2, or in3:
var out = {
id: in1.id,
title: in1.title,
original_name: in2.original_name,
year: in3.year
}
console.log(out);
If you have three lists of objects:
...in this case it's a lot more complicated (and a lot more interesting). In this case you would need to match fields from the objects in each list which share the same IDs.
The following is definitely not the most efficient or memory-conserving way to do this; I've spread things out to (hopefully) make it easier to follow what it's doing.
I'm making two assumptions:
within each list, all IDs are unique (meaning you won't have two objects with the same ID in one JSON file)
Every ID will appear in all three lists (meaning you don't need to handle missing fields in output)
/* Again handwaving past loading JSON strings and parsing
them into javascript objects, we'll just start with
three arrays: */
var input1 = [{
"id": "265",
"title": "Battle of Gettysburg",
"page_id": "4849",
"language_id": "en",
"original_time": "July 1\u20133, 1863"
},
{
"id": "1",
"title": "Foo",
"page_id": "123",
"language_id": "en",
"original_time": "July 1\u20133, 1863"
}
];
var input2 = [{
"id": "1",
"original_name": "Bar",
"country_id": "24",
"article_id": "265"
},
{
"id": "265",
"original_name": "United States",
"country_id": "24",
"article_id": "265"
}
]
var input3 = [{
"id": "1",
"month": "July",
"year": "Baz",
"suffix": "",
"article_id": "265"
},
{
"id": "265",
"month": "July",
"year": "1863",
"suffix": "",
"article_id": "265"
}
]
/* It would be much easier to find corresponding IDs
across these arrays if they weren't arrays. We'll
start by converting them into objects keyed by the
item ids: */
var convertArray = function(arr) {
var output = {};
arr.forEach(function(o) {
output[o.id] = o;
});
return output;
}
var obj1 = convertArray(input1);
var obj2 = convertArray(input2);
var obj3 = convertArray(input3);
/* Now if we need to find (say) the object with id "foo", we don't
need to search the whole array, but can just use `obj1["foo"]` or
`obj1.foo`.
The last step is to iterate over the list of IDs and repeatedly
do basically the same thing as in the "if you have three objects"
part above. The only difference is that we need to access the
object with the same ID in each of the input lists: */
var constructOutput = function(in1, in2, in3) {
var output = []; // we'll be outputting a list of objects again.
// step through every ID (assuming in1 contains all of them):
Object.keys(in1).forEach(function(id) {
var obj = {
id: id,
title: in1[id].title,
original_name: in2[id].original_name,
year: in3[id].year
}
output.push(obj);
});
return output;
}
var final = constructOutput(obj1, obj2, obj3)
console.log(final)

Essentially what you have to do is mimic a SQL JOIN using JavaScript objects:
Use JSON.parse() on all three JSON collections to turn them into arrays of objects.
Iterate through JSON 1 objects; for each object...
Iterate through JSON 2 objects, testing if article ID matches the ID from JSON 1 that we are iterating over. Save this object.
Iterate through JSON 3 objects, testing if ID matches the ID of the object we found from JSON 2. Save this object.
After you have all three objects, make a new object literal that contains only the fields you want:
{
Id: obj1.id,
Title: obj1.title,
Original_name: obj2.original_name,
Year: obj3.year
}

Should you want to combine n number of JSON objects, e.g. a list of objects you can take a functional approach and utilise reduce + filter.
const data = [{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
},
{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
},
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}];
const final = data.reduce((accu, { id, title }, index, array) => {
// Find any related objects
const matches = array.filter(data => data.article_id === id);
if (matches.length) {
// Flatten them for ease of access. Duplicate keys will override.
const flat = matches.reduce((arr, item) => ({ ...arr, ...item }), [])
// Return new object
return accu.concat({
...flat,
id,
title,
});
}
return accu;
}, []);
console.log(final, '<<')
// Witness
document.getElementById('results').innerHTML = JSON.stringify(final);
<div id="results" style="font-family: Courier; font-size 14px; color: #fff; background: #000; padding: 20px; max-width: 80vw;"></div>

Edited*
Maybe this is what you need?
let arrPages = [{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
}];
let arrArticles = [{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
},
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}];
let getResult = (arrInput, arrCompare) => {
let joinedItems = [];
arrInput.forEach(item => {
let newItem = { id: item.id, title: item.title };
arrCompare.forEach(subItem => {
if(subItem.article_id !== undefined && subItem.article_id === item.id){
if(subItem.original_name !== undefined)
newItem.original_name = subItem.original_name;
if(subItem.year !== undefined)
newItem.year = subItem.year;
}
});
joinedItems.push(newItem);
});
return joinedItems;
};
let result = getResult(arrPages, arrArticles);
console.log(result);

In the first part of the code i create a var that has the json data.
To solve the problema i create 2 functions, the order of the creation dosen't metter, the first function getJSONData() take the json data as parameter and return a object filtered by the keys defined in the array keys. The secound function just check if the current key is present in the array of keys, this function could be replaced by the jQuery.inArray() method.
// JSON data
var json = [{
"id":"265",
"title":"Battle of Gettysburg",
"page_id":"4849",
"language_id":"en",
"original_time":"July 1\u20133, 1863"
},
{
"id":"185",
"original_name":"United States",
"country_id":"24",
"article_id":"265"
},
{
"id":"73",
"month":"July",
"year":"1863",
"suffix":"",
"article_id":"265"
}]
// keys that i want
var keys = ["title", "original_name", "year"];
// var that will have the filtered data
var newJSON = getJSONData(json);
console.log(JSON.stringify(newJSON))
// this is the main function of the code
// here we iterate in the json creating a new object that has all the tags definid in the keys array
function getJSONData(arrayJSON){
var JSONFiltered = {};
for(var i in arrayJSON){
for(var key in arrayJSON[i]){
if(hasElement(key)){
JSONFiltered[key] = arrayJSON[i][key];
}
}
}
return JSONFiltered;
}
// this function is used to check a key is present in the array of keys
function hasElement(key){
for(var elem in keys){
if(keys[elem] == key) return true;
}
return false;
}

Using dictionary as D3 data source

I want to use this data (below) as a data source for a d3.js program:
{
"component": {
"name1": {
"graphname": {
"title": "foo",
"data": [
{"data": "DATE IN ISOFORMAT", "value": 5},
{"data": "DATE IN ISOFORMAT", "value": 10}
]
}
},
"name2": {
"graphname": {
"title": "foo",
"data": [
{"data": "DATE IN ISOFORMAT", "value": 5},
{"data": "DATE IN ISOFORMAT", "value": 10}
]
}
}
}
"component2": {...
}
D3 accepts only array data? How i can manipulate it to work?
I want to graph for all component any "graphname" aggregated by "name"
Any tips?

D3 does not just accept arrays as data sources, but for looping purposes, arrays are much more convenient than JavaScript objects ("dicts"). There is a very easy way to convert objects to arrays, though. If your object above were called d, then its corresponding array can be created with:
var dlist = d3.entries(d);
Now dlist will be something like:
[ { key: 'component',
value: { name1: ..., name2: ... } },
{ key: 'component2',
value: { name1: ..., name2: ... } } ]
The original dict has been remapped into an array of "records," each with key and value pairs. This "array of records" pattern is very common in D3 work, and JavaScript in general. If you need to loop over the sub-structures (e.g. the name1, name2, ... values, d3.entries can be applied at multiple levels of the original structure, as those dict-to-list transforms are required.
Since this answer is called out in the comments as wrong, here's a working example of using an object ("dict") as a data source for a D3 program: first in a simple loop, then secondarily using the idiomatic d3 .data(...).enter() pipeline.

Flatten D3.js Nested Data or Map it to new Dataset

Total noob to D3.js and working on creating my first grouped bar chart. However I'm having trouble making my data fit with the examples online. I am trying to use this example here, with my data that has been nested JSON with D3.
My problem is i cant use the d3.keys method to retrieve keys because my keys are not the state names. They are just Key.
Not to mention the second half forEach wont work because again the keys are not the State names, they are just the term key. So +d[name] will try d[MAPLE] when really my value is inside a key of d.values[(Get the Value where the key = name)]. Just really confused how to do this once the data has been nested in JSON
How would I go about getting all possible Key Values, and then mapping the keys with the next level of keys and values? Using a similar example as below but with my JSON nested data.
var ageNames = d3.keys(data[0]).filter(function(key) { return key !== "State"; });
data.forEach(function(d) {
d.ages = ageNames.map(function(name) { return {name: name, value: +d[name]}; });
});
My data is as so
{
"key": "1/10/2014",
"values": [
{
"key": "Texas",
"values": 200
},
{
"key": "Colorado",
"values": 300
},
{
"key": "Utah",
"values": 227
}
]
},{
"key": "2/10/2014",
"values": [
{
"key": "Texas",
"values": 225
},
{
"key": "Colorado",
"values": 241
},
{
"key": "Utah",
"values": 500
}
]
}

It's not clear from the question if the aim is to group by state ("Texas", "Colorado"...) or date ("1/10/2014", "2/10/2014"...) along the x-axis.
Assuming date (because that's how the data is currently structured), here's a working plunk: http://plnkr.co/edit/C8lkPMGanFY9BkTc6f1i?p=preview
The code that processes the data into a format that your existing D3 code for grouped bar chart can handle looks like this:
// Call the first mapping function on every 'date' in the data array
processed_data = data.map( function (d) {
var y0 = 0;
var total = 0;
return {
date: d.key,
// Call the second mapping function on every 'state' in the given date array
values: (d.values).map( function (d) {
return_object = {
state: d.key,
count: d.values,
y0: y0,
y1: y0 + d.values
};
// Calculate the updated y0 for each new state in a given date
y0 = y0 + d.values;
// Add the total for a given state to the sum total for that date
total = total + d.values;
return return_object;
}),
total: total
};
});
We use nested array.map transforms to manipulate your two-level nested data into the expected format and calculate y0, y1 and total values. Your processed_data object will then look like this:
The only other tricky bit will be to calculate your list of unique states, in order to define the color.domain. If you don't want to hard-code this (e.g. because the list may vary from dataset to dataset, you could use this approach:
// Define the color domain, by first building a list of states:
var states = [];
// Loop through each date
processed_data.forEach(
function (d) {
// Loop through each state in that date
d.values.forEach(
function(d) {
// add to the array if not already present
if (!(states.indexOf(d.state) > -1)) {
states.push(d.state)
}
}
)
}
);
color.domain(states);