We Keep Coding

Reducing redundant information in a JSON object with javascript and combining values

I have a CSV of results that looks at a picture and makes a guess at whether or not a picture contains a certain attribute. In this case if the subject in the picture is male or female.
I'm converting this CSV to JSON with javascript/node and I want to take the attributes and their values and put them in an array inside of one object per pciture. Right now each line of the CSV measures and attribute but it means at least two lines per image.
Simple version of the csv:
path, detect_id, score, x-coord, y-coord, w-coord, h-coord, attribute, value
picture_1.jpg,0,1.44855535,74,54,181,181,genderf,0.024716798
picture_1.jpg,0,1.44855535,74,54,181,181,genderm,0.975283206
I can convert this CSV to JSON and then at least group items together by their path/filename.
But that leaves a lot of redundant information out there and I want to put my Attributes and their Value together in a nested object inside of the main one.
Like:
Path: picture_1.jpg
Attributes: [genderf: 0.025,
genderm: 0.985]
other_info: other info
Right now I'm using lodash to create the objects as you see below but if I try to map through the attributes I end up pushing out every element except the last one.
So I can create the object with the following code.
var result =
_([...arr1, ...arr2])
.concat()
.groupBy("path")
.value();
Where arr1 and arr2 is the data from one line of the output csv. All the information is the same except the attribute and its value.
That gets me this object:
{
"picture_1.jpg": [
{
"path": "picture_1.jpg",
"detect_id,": "0",
"score,": "1.44855535",
"coordinates": [
{
"x,": "74",
"y,": "54",
"w": "181",
"h": "181"
}
],
"attribute": "genderf",
"value": "0.024716798"
},
{
"path": "picture_1.jpg",
"detect_id,": "0",
"score,": "1.44855535",
"coordinates": [
{
"x,": "74",
"y,": "54",
"w": "181",
"h": "181"
}
],
"attribute": "genderm",
"value": "0.975283206"
}
]
}
Which at least groups pictures together based on the path heading but a lot of the information is redundant and this is just measuring one attribute.

You could just iterate all csv-lines and build an object/map while keeping track of already found file-names/paths. If you encounter a line whose path already exists in the map, just append the attribute/value pair. Something like this (note that I've changed the coords delimiter for the sake of simplicity and that it needs proper error handling):
const data = ["picture_1.jpg,0,1.44855535,74;54;181;181,genderf,0.024716798", "picture_1.jpg,0,1.44855535,74;54;181;181,genderm,0.975283206"];
function createImageDataMap(dataArr) {
const imageDataResult = {};
for (const imgData of dataArr) {
const currData = parseImgDataLine(imgData);
if (!imageDataResult[currData.path]) {
imageDataResult[currData.path] = {
attributes: [], other_info: {
score: currData.score,
detectId: currData.detectId,
coords: currData.coords
}
}
}
imageDataResult[currData.path].attributes.push({[currData.attribute]: currData.value});
}
return imageDataResult;
}
function parseImgDataLine(line) {
const attributes = line.split(',');
return {
path: attributes[0],
detectId: attributes[1],
score: attributes[2],
coords: attributes[3],
attribute: attributes[4],
value: attributes[5]
}
}
console.log(JSON.stringify(createImageDataMap(data)));
// prints {"picture_1.jpg":{"attributes":[{"genderf":"0.024716798"},{"genderm":"0.975283206"}],"other_info":{"score":"1.44855535","detectId":"0","coords":"74;54;181;181"}}}

How to fix creating a new object in JSON file?

I'm trying to create a function that when called will update a specific object in json file. However, it updates the object as well as creating a new one.
I've tried many different methods in trying to get this to work, but all have failed. The closest I've got to it working is the code shown below, but it still doesn't do what is required.
This is my function:
var fs = require('fs');
var _ = require("underscore");
module.exports = {
personalUpdate: function (id, forename, surname, dob, gender, callback) {
let rawdata = fs.readFileSync('data.json');
let data = JSON.parse(rawdata);
let filtered = _.where(data['students'], { id: id });
let all = filtered[0];
all.forename = forename;
all.surname = surname;
all.dob = dob;
all.gender = gender;
data["students"].push(all);
fs.writeFileSync('data.json', JSON.stringify(data, null, 2), (err) => {
if (err) throw err;
});
callback("success");
}
}
And this is the JSON file that I want to update:
{
"teachers": [
{
"name": "",
"email": "",
"password": "",
"formGroup": "",
"id": ""
}
],
"students": [
{
"surname": "test",
"forename": "test",
"dob": "",
"homeAddress": "",
"homePhone": "",
"gender": "",
"tutorGroup": "",
"schoolEmail": "",
"grades": [
{
"french": 8,
"maths": 7
}
],
"id": ""
},
{
"surname": "test2",
"forename": "test2",
"dob": "",
"homeAddress": "test2",
"homePhone": "",
"gender": "",
"tutorGroup": "",
"schoolEmail": "",
"grades": [
{
"french": 9,
"maths": 8
}
],
"id": ""
}
]
}
I had to remove and change the objects and info inside them, as it contained confidential information.
When running this function, it finds the object that is specified in the parameter. It then updates that object, but it then creates another object at the bottom of the original JSON object, which it is not supposed to.
Also, is there a better way to update the specific objects in the JSON file?

tl;dr
The result set is duplicating because you are pushing it into the array
The change is being applied due to the variables holding the same object reference, so they are being mirrored across objects that share the same pointer.
Explanation
It creates a new one due to the data["students"].push(all); instruction.
When you manipulate objects in javascript you need to be aware of how the reference between them work, so you can avoid bugs and use them in your benefit.
For example, take this set of instructions:
let a = {"x": 1};
let b = a;
b.x = 3;
console.log(a) // it will output {"x": 3}
Notice that we:
Create an object with the prop x equal 1 and assign it to the variable a
Initialize a variable b with the value of a
Change the property x on the variable/object b
Then we can observe that the change was also reflected in the variable a, due to the object reference.
So, basically this is exactly what is happening with your instructions when you do all.forename = forename; it changes the variable all, but also the original object which it derives from.
Here is a nice reference that explains this concept more in-depth
#EDIT
I strongly advise you not using the sync version of functions like readFileSync since this blocks the event loop. Here is the official guidelines about it

How to use "this" keyword inside array in javascript?

Here I have an array which consists of n number of objects which are similar to this.
var views = [
{
id: "chart-0",
datasource: "",
type: "batch",
context: "/analytics/device/stats/",
schema: [
{
"metadata": {
"names": [""],
"types": [""]
}
}
],
columns: [
{
"name": "month",
"label": "month",
"type": "linear",
},
{
"name": "minValue",
"label": "minValue",
"type": "linear"
},
{
"name": "maxValue",
"label": "maxValue",
"type": "linear",
"axis": "y"
},
{
"name": "time",
"label": "time",
"type": "time",
"axis": "x"
},
],
callbacks: [
{
type: "click",
callback: function() {}
}
],
data: function() {
var COLUMNS = views[0].columns;
}
}
];
I want to access some of the fields inside data function. I can access id, datasource simply this.id or this.datasource. How to access columns inside data function using this keyword instead of views[0].columns

The only way I know that you can use the this key word to assign id, distance, or any other property would be to pass the data into a function.
// create a function and pass in a reference to the views array and specify the objects index you want to operate on.
function chanceProperty( views[0]) {
var currentColumns = this.columns; // gives you a reference to the objects columns array. "This" is a reference to the object that was passed in.
// you can then do your operations on the properties in that array either with a loop or by specifically targeting an object.
currentColumns[1].name = "myNewMinValue";
// or
currentColumns.forEach( function(val, idx) {
currentColumns[idx].name = "allTheSame";
});
}
update
I thin I see what you are trying to do. You want pass some of your views properties to your data function. You don't need "this" for that. You can just pass the properties you want to the data function.
data: function(id, dataSource, columns) {
var COLUMNS = views[0].columns;
}

You can use:
data: function() {
var COLUMNS = this.columns;
return COLUMNS;
}
And you will be able to access columns inside data function:
views[0].data();

Using this needs care. Well you can access views[0].columns by saying this.columns but under two conditions.
You must call the function like views[0].data(); and only then the this in the data function will refer to the views object. In other words if you might like to pass views[0].data as a callback, you better be careful because if you do like doStg(arg1, arg2, views[0].data) then this will no longer refer to the views[0] object but to the global or window scope. So you must pass callbacks like doStg(arg1, arg2, function(){views[0].data})
Somehow instead of conventional method if you would like to define views[0].data as an arrow function, then forget about using this. Since then this would only refer to the limited scope it resides in which is the function's own scope (not even views[0] object)

Create nested JSON array

I'm trying to create a JSON array to send it to my web service. This is how my json should look like:
[{
"tipus": 1,
"proveidor": 3,
"atributs": {
"atribut":{
"id": 1,
"valor": 8
},
"atribut":{
"id": 2,
"valor": 500
}
}
}]
So, I have two general values "tipus" and "proveidor" and multiple "atributs" each "atribut" is composed with "id" and "valor".
When I construct the json I get this instead of what I want:
[
2:{
"tipus": 1,
"proveidor": 3,
1:{
"id": 1,
"valor": 8
},
0:{
"id": 2,
"valor": 500
}
}]
This is how I'm building the json:
// For every founded in $scope.atrb i need to create an 'atribut' element into my json
$scope.a = [];
var key;
for(key in $scope.atrb){
var newField = {
"idatributs_actiu": $scope.atrb[key].idatributs_actiu,
"nomAtribut": $scope.atrb[key].nomAtribut,
"valor": $scope.atrb[key].valor,
"idActiu": $routeParams.idTipusActiu,
"value": "",
"ordre": $scope.atrb[key].ordre,
"idatributs_generics": $scope.atrb[key].idatributs_generics
};
$scope.a.push(newField);
}
$scope.f = $scope.a;
});
var generics = {
"nom": $scope.nom,
"tipus": $routeParams.idTipusActiu,
"proveidor": $scope.proveidor.id
};
$scope.a.push(generics);
It's my first project with angular and I'm not sure if I'm building the json appropriately, basically i use an array to build a json but I don't know how to nested it 'atribut' inside 'atributs'.
The main idea is to read the 'generics' atributes and then loop through 'atributs' and read all 'atribut' element getting the properties.
Regards

Like S4beR and Kevin B told me, I just need to do an JS array. This is in my controller:
var obj = { generics: g, atributs: $scope.a };
g: it's an object with the generic properties
$scope.a: this is an array with 'atribut' objects which contais all
the properties I need save to.

Does angularjs automatically create nested structures?

a bit confused about how to create and reference items in a nested array using angular.js. I was thinking that I could do:
$scope.zones = [ {z:'north'}, {z:'south'} ];
$scope.zones.times = [ {t:'noon'}, {t:'midnight'} ];
$scope.zones.times.places = [ {p:'here'}, {p:'there'} ];
and angularjs would create a structure in which every zone has two times and every time has two places.
Then I could use something like:
<ul ng-repeat="zone in $scope.zones">
<li>{{zone.z}}</li>
<ul ng-repeat="time in zone.times">
<li>{{time.t}}</li>
<ul ng-repeat="place in time.places">
<li>{{place.p}}</li>
</ul>
</ul>
</ul>
to see the tree structure on my page.
So, does using the dot notation above actually create a nested array of objects? Should I be able to reference them "recursively" as in the directive above? I'm having trouble getting this to work beyond the first two levels.

You're not setting up your data correctly. It should be:
$scope.zones = [ {z:'north'}, {z:'south'} ];
$scope.zones[0].times = [ {t:'noon'}, {t:'midnight'} ];
$scope.zones[1].times = [ {t:'noon'}, {t:'midnight'} ];
Etc...
Then your HTML should work as expected.

In JavaScript using obj.propName accesses the property named propName of the object referenced by obj.
Arrays are also objects (in JavaScript), so these lines of code:
$scope.zones = [...];
$scope.zones.times = [...];
create an array (named zones) and give it a property named times (which is also an array).
Note: This is JavaScript-specific and has nothing to do with Angular.
This is not what you want. You want to give the times property to zones' items, not to zones itself. (Similarly with places.)
In order to achieve that, you need to iterate over the elements of each array and give it the extra properties.
var places = [{p: 'here'}, {p: 'there'}];
var times = [{t: 'noon'}, {t: 'midnight'}];
var zones = [{z: 'north'}, {z: 'south'}];
// Give each `time` a `places` property
times.forEach(function (time) {
time.places = places;
});
// Give each `zone` a `times` property
// (Each `time` already has a `places` property.)
zones.forEach(function (zone) {
zone.times = times;
});
// Assign the augmented `zones` to `$scope.zones`
$scope.zones = zones;
See, also, this short demo.
Note:
In the above implementation, values are passed by reference. This means that is you change $scope.zones[0].times[0].places[0], $scope.zones[1].times[1].places[0] will also be affected, since it references the same object.
This implementation is OK if you just want to read values, as it's more efficient.
If you want to be able to also modify the objects, then you need to create copies of the objects and not assign them by reference.
E.g., instead of time.places = places; and zone.times = times;
write time.places = angular.copy(places); and zone.times = angular.copy(times); respectively.
The resulting $scope.zones object will look liek this:
[{
"z": "north",
"times": [{
"t": "noon",
"places": [
{"p": "here" },
{"p": "there"}
]
}, {
"t": "midnight",
"places": [
{"p": "here" },
{"p": "there"}
]
}]
}, {
"z": "south",
"times": [{
"t": "noon",
"places": [
{"p": "here" },
{"p": "there"}
]
}, {
"t": "midnight",
"places": [
{"p":"here" },
{"p":"there"}
]
}]
}]
Note:
In your HTML code you reference $scope.zones. This is an error !
The corect way is: zones
(All Angular expressions are evaluated in the context of the current Scope, thus ng-repeat="zone in $scope.zones" will look for $scope.$scope.zones.)