JSONata (or JS) - group and sum JSON array / objects - javascript

I want to 'groupsum' an array of objects with JSONata, from the following array:
payload = [
{"id":"Irr-1","name":"Irrigation","timedif_s":7.743},
{"id":"Irr-2","name":"Irrigation","timedif_s":2.749},
{"id":"Lig-2","name":"Lights","timedif_s":1.475},
{"id":"Irr-1","name":"Irrigation""timedif_s":1.07},]
The results must look like:
[{"name":"Irrigation", "sumtimedif": 11.562},
{"name":"Lig-1", "sumtimedif": 1.475}]
I have tried:
payload.name{"name":payload.name,"sumtimedif":$sum(timedif_s)}
But I only get back an empty string.
{ empty }
Any advise?

The following JSONata expression will do this:
payload{
name: $sum(timedif_s) /* group by name */
} ~> $each(function($v, $k) {
{"name": $k, "sumtimedif": $v} /* transform each key/value pair */
})
The first part of the expression does the grouping and aggregation, and the second part transforms each group (key/value pair) into the format you want. http://try.jsonata.org/ByxzyQ0x4

I found another good post that helped me: Most efficient method to groupby on a array of objects.
Not sure if this can be done in JSONata, so I reverted to JS, as follows:
var arr = msg.payload ;
arr1 = groupBy(arr, 'name', 'timedif_s' ) ;
msg.payload = arr1 ;
return msg;
//---------------------------------------------
function groupBy(array, col, value) {
var r = [], o = {};
array.forEach(function (a) {
if (!o[a[col]]) {
o[a[col]] = {};
o[a[col]][col] = a[col];
o[a[col]][value] = 0;
r.push(o[a[col]]);
}
o[a[col]][value] += +a[value];
});
return r;
}

Related

Find Unique value from an array based on the array's string value (Javascript)

so I want to find unique values from an array.
so for example I have this array:
const mainArr = ['shape-10983', 'size-2364', 'size-7800', 'size-4602', 'shape-11073', 'size-15027', 'size-15030', 'size-15033', 'height-3399', 'height-5884']
so I want to find the first matching value for each unique item.
for example, in the array, I have two strings with the shape prefix, six items with the size prefix, and two items with the height prefix.
so I want to output to be something like
const requiredVal = ["shape-10983", "size-2364", "height-3399"]
I want only the first value from any set of different values.
the simplest solution will be to iterate on the list and storing what you got in a dictionary
function removeSimilars(input) {
let values = {};
for (let value of input) {//iterate on the array
let key = value.splitOnLast('-')[0];//get the prefix
if (!(key in values))//if we haven't encounter the prefix yet
values[key] = value;//store that the first encounter with the prefix is with 'value'
}
return Object.values(values);//return all the values of the map 'values'
}
a shorter version will be this:
function removeSimilars(input) {
let values = {};
for (let value of input)
values[value.splitOnLast('-')[0]] ??= value;
return Object.values(values);
}
You could split the string and get the type and use it aks key for an object along with the original string as value. At result take only the values from the object.
const
data = ['shape-10983', 'size-2364', 'size-7800', 'size-4602', 'shape-11073', 'size-15027', 'size-15030', 'size-15033', 'height-3399', 'height-5884'],
result = Object.values(data.reduce((r, s) => {
const [type] = s.split('-', 1);
r[type] ??= s;
return r;
}, {}));
console.log(result);
If, as you mentioned in the comments, you have the list of prefixes already available, then all you have to do is iterate over those, to find each first element that starts with that prefix in your full list of possible values:
const prefixes = ['shape', 'size', 'height'];
const list = ['shape-10983', 'size-2364', 'size-7800', 'size-4602', 'shape-11073', 'size-15027', 'size-15030', 'size-15033', 'height-3399', 'height-5884']
function reduceTheOptions(list = [], prefixes = [], uniques = []) {
prefixes.forEach(prefix =>
uniques.push(
list.find(e => e.startsWith(prefix))
)
);
return uniques;
}
console.log(reduceTheOptions(list, prefixes));
Try this:
function getRandomSet(arr, ...prefix)
{
// the final values are load into the array result variable
result = [];
const randomItem = (array) => array[Math.floor(Math.random() * array.length)];
prefix.forEach((pre) => {
result.push(randomItem(arr.filter((par) => String(par).startsWith(pre))));
});
return result;
}
const mainArr = ['shape-10983', 'size-2364', 'size-7800', 'size-4602', 'shape-11073', 'size-15027', 'size-15030', 'size-15033', 'height-3399', 'height-5884'];
console.log("Random values: ", getRandomSet(mainArr, "shape", "size", "height"));
I modified the #ofek 's answer a bit. cuz for some reason the ??= is not working in react project.
function removeSimilars(input) {
let values = {};
for (let value of input)
if (!values[value.split("-")[0]]) {
values[value.split("-")[0]] = value;
}
return Object.values(values);
}
create a new array and loop over the first array and check the existing of element before in each iteration if not push it to the new array

Search by multiple keys and values, Javascript

I'm trying to make a search function. The number of filters will change dynamically, a number of keys can be different, and the number of values, too.
My code looks like:
var data = [{"id":"123","color":"Red","model":"Tesla"},{"id":"124","color":"Black","model":"Honda"},{"id":"125","color":"Red","model":"Audi"},{"id":"126","color":"Blue","model":"Resla"}]
var keys = ["color", 'model'];
var values = ["Re"];
var result = data.filter(function(e) {
return keys.every(function(a) {
return values.includes(e[a])
})
})
console.log(result);
Is it possible to search with - startsWith() and not includes()? I guess everything should be in toLowerCase() as well?
Also can I have two separate results as two arrays if results found in one key then it should individual array? So results will be like:
[{ colors: [{"id":"123","color":"Red","model":"Tesla"},{"id":"125","color":"Red","model":"Audi"}], models: [{"id":"126","color":"Blue","model":"Resla" }] }]
Thank you very much in advance.
You need to check keys.some and not keys.every. This will check if the value is part of at least one of the keys and not all of them.
For values, you could create a dynamic regex with alternation and test value against the regex. So, values = ["Re", "Ho"] will create /Re|Ho/
const data = [{"id":"123","color":"Red","model":"Tesla"},{"id":"124","color":"Black","model":"Honda"},{"id":"125","color":"Red","model":"Audi"},{"id":"126","color":"Blue","model":"Resla"}],
keys = ["color", 'model'],
values = ["Ho"],
regex = new RegExp(values.join('|')),
output = data.filter(e => keys.some(k => regex.test(e[k])) )
console.log(output);
If you want to individual results for each key, you can loop through the keys and check for the regex individually.
const data = [{"id":"123","color":"Red","model":"Tesla"},{"id":"124","color":"Black","model":"Honda"},{"id":"125","color":"Red","model":"Audi"},{"id":"126","color":"Blue","model":"Resla"}],
keys = ["color", 'model'],
values = ["Ho", "Re"],
regex = new RegExp(values.join('|')),
group = {}
for (const o of data) {
for (const k of keys) {
if (regex.test(o[k])) {
group[k] ||= []
group[k].push(o)
}
}
}
console.log(group);
You can iterate through each key and value and look it up in object array using array#reduce
const data = [{"id":"123","color":"Red","model":"Tesla"},{"id":"124","color":"Black","model":"Honda"},{"id":"125","color":"Red","model":"Audi"},{"id":"126","color":"Blue","model":"Resla"}],
keys = ["color", 'model'],
values = ["Re"],
initial = Object.assign(...keys.map(k => ({[`${k}s`]: [] }))),
result = data.reduce((r, o) => {
keys.forEach(k => {
values.forEach(val => {
if(o[k] && o[k].startsWith(val)) {
r[`${k}s`].push(o);
}
});
});
return r;
},initial);
console.log(result);
.as-console-wrapper { max-height: 100% !important; top: 0; }

Convert JSON to Array of Objects using lodash

I have a JSON object in NoSql database in this format. We are getting this data after migrating some records from some other database and these are multi-valued fields.(Basically we are trying to clean the data for further processing).
{
"BPContName":"aName;bName;cName",
"BPContEmail":"aEmail;bEmail;cEmail",
"BPContPWID":"aPWID;bPWID;cPWID"
}
I want to add another key "bpTableDataName" in the same JSON which should have this format and values,
"bpTableDataName": [
{
"name": "aName",
"email": "aEmail",
"pwdid": "aPWID"
},
{
"name": "bName",
"email": "bEmail",
"pwdid": "bPWID"
},
{
"name": "cName",
"email": "cEmail",
"pwdid": "cPWID"
}
],
Is there a way we can achieve this using lodash?
Try following code -
o = {
"BPContName": "aName;bName;cName",
"BPContEmail": "aEmail;bEmail;cEmail",
"BPContPWID": "aPWID;bPWID;cPWID"
}
map = { "BPContName" : "name", "BPContEmail": "email", "BPContPWID": "pwdid" }
const result = _.reduce(o, (arr, v, k) => ( v.split(";").forEach((x,i) => _.set(arr, `${i}.${map[k]}`, x)), arr ), [])
console.log(result)
<script src="https://cdn.jsdelivr.net/npm/lodash#4.17.11/lodash.min.js"></script>
You can use split() to split the values into an array.
Then iterate over the array and create the require json and then push that into results.
Check this out.
var data = {
"BPContName":"aName;bName;cName",
"BPContEmail":"aEmail;bEmail;cEmail",
"BPContPWID":"aPWID;bPWID;cPWID"
}
var names = data.BPContName.split(';');
var emails = data.BPContEmail.split(';');
var pwids = data.BPContPWID.split(';');
var results = [];
for(var i = 0 ; i < names.length; i++) {
var obj = {
name: names[i],
email: emails[i],
pwdid: pwids[i]
}
results.push(obj);
}
console.log(results)
You could reduce the entries returned by Object.entries like this:
let obj = {
"BPContName": "aName;bName;cName",
"BPContEmail": "aEmail;bEmail;cEmail",
"BPContPWID": "aPWID;bPWID;cPWID"
}
let bpTableDataName = Object.entries(obj).reduce((r, [key, value]) => {
let splits = value.split(";");
key = key.replace("BPCont", "").toLowerCase();
splits.forEach((split, i) => (r[i] = r[i] || {})[key] = split)
return r;
}, [])
obj.bpTableDataName = bpTableDataName;
console.log(obj)
Object.entries returns an array of key-value pair. Loop through each of them
split the each value at ;
get the key by removing BPCont part and making it lowerCase
Loop through the splits and update specific keys of objects at each index
Update:
Since you have an extra d in the output's key, you can create a mapping object:
propertyMap = {
"BPContName": "name",
"BPContEmail": "email",
"BPContPWID": "pwdid"
}
And inside the reduce, change the replace code to this:
key = propertyMap[key]
Using Object.assign, Object.entries, Array#map and the spread operator make this trivial
const inputdata = {
"BPContName":"aName;bName;cName",
"BPContEmail":"aEmail;bEmail;cEmail",
"BPContPWID":"aPWID;bPWID;cPWID"
};
const t1=Object.assign({},...Object.entries(inputdata).map(([k,v])=>({[k]:v.split(';')})));
inputdata.bpTableDataName=t1.BPContName.map((name,i)=>({name,email:t1.BPContEmail[i],pwdid:t1.BPContPWID[i]}));
console.log(inputdata);
Of course, it wouldn't be me without a one-liner
const obj = {
"BPContName":"aName;bName;cName",
"BPContEmail":"aEmail;bEmail;cEmail",
"BPContPWID":"aPWID;bPWID;cPWID"
};
// one line to rule them all
obj.bpTableDataName=Object.entries(obj).reduce((r,[k,v])=>(v.split(';').forEach((v,i)=>(r[i]=r[i]||{})[{BPContName:'name',BPContEmail:'email',BPContPWID:'pwdid'}[k]]=v),r),[]);
//
console.log(obj);
Basically what you need is to zip it.
Snippet:
let obj = {"BPContName":"aName;bName;cName","BPContEmail":"aEmail;bEmail;cEmail","BPContPWID":"aPWID;bPWID;cPWID"},
res = _.zipWith(
..._.map(obj, v => v.split(';')),
(name, email, pwid) => ({name, email, pwid})
);
console.log(res)
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.11/lodash.min.js"></script>
Note, the sequence of the parameters we have to put such a way, the original object give us values when using Object.values or giving us keys when using Object.keys usually it is alphabetical order. But, In case in any env the order is not guranted we can sort it with a sequence of keys as a metadata.
Or else you can explicitly pass the arguments like:
(obj.BPContName.split(';'), obj.BPContEmail.split(';'), obj.BPContPWID.split(';'))
You can use lodash's _.flow() to create a function. Use _.map() with _.overArgs() to create a function that splits the values, format the key, and then converts them to an array of pairs using _.unzip(), for example [['name', 'x'], ['name', 'y']]. Transpose the array of arrays with _.unzip() to combine pairs of different properties. Then use _.map() to iterate, and convert each array of pairs to an object using _.fromPairs().
const { flow, partialRight: pr, map, unzip, overArgs, times, size, constant, split, fromPairs } = _
const keysMap = new Map([['BPContName', 'name'], ['BPContEmail', 'email'], ['BPContPWID', 'pwdid']])
const formatKey = key => keysMap.get(key)
const splitVals = pr(split, ';')
const fn = flow(
pr(map, overArgs(
(vals, k) => unzip([vals, times(size(vals), constant(k))]),
[splitVals, formatKey])
),
unzip, // transpose
pr(map, fromPairs) // convert each pairs array to object
)
const data = {
"BPContName":"aName;bName;cName",
"BPContEmail":"aEmail;bEmail;cEmail",
"BPContPWID":"aPWID;bPWID;cPWID"
}
const results = fn(data)
console.log(results)
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.11/lodash.js"></script>

convert CSV lines into Javascript objects

I have a simple csv file
people.csv:
fname, lname, uid, phone, address
John, Doe, 1, 444-555-6666, 34 dead rd
Jane, Doe, 2, 555-444-7777, 24 dead rd
Jimmy, James, 3, 111-222-3333, 60 alive way
What I want to do it get each line of the CSV, convert it to a JavaScript object, store them into an array, and then convert the array into a JSON object.
server.js:
var http = require('http');
var url = require('url');
var fs = require('fs');
var args = process.argv;
var type = args[2] || 'text';
var arr = [];
var bufferString;
function csvHandler(req, res){
fs.readFile('people.csv',function (err,data) {
if (err) {
return console.log(err);
}
//Convert and store csv information into a buffer.
bufferString = data.toString();
//Store information for each individual person in an array index. Split it by every newline in the csv file.
arr = bufferString.split('\n');
console.log(arr);
for (i = 0; i < arr.length; i++) {
JSON.stringify(arr[i]);
}
JSON.parse(arr);
res.send(arr);
});
}
//More code ommitted
My question is if I am actually converting that CSV lines into Javascript objects when I call the .split('\n') method on bufferString or is there another way of doing so?
By doing this:
arr = bufferString.split('\n');
you will have an array containing all rows as string
["fname, lname, uid, phone, address","John, Doe, 1, 444-555-6666, 34 dead rd",...]
You have to break it again by comma using .split(','), then separate the headers and push it into an Javascript Object:
var jsonObj = [];
var headers = arr[0].split(',');
for(var i = 1; i < arr.length; i++) {
var data = arr[i].split(',');
var obj = {};
for(var j = 0; j < data.length; j++) {
obj[headers[j].trim()] = data[j].trim();
}
jsonObj.push(obj);
}
JSON.stringify(jsonObj);
Then you will have an object like this:
[{"fname":"John",
"lname":"Doe",
"uid":"1",
"phone":"444-555-6666",
"address":"34 dead rd"
}, ... }]
See this FIDDLE
Using ES6/ES7 and some functional programming guidelines:
All variables are const (immutability)
Use map/reduce instead of while/for
All functions are Arrow
No dependencies
// Split data into lines and separate headers from actual data
// using Array spread operator
const [headerLine, ...lines] = data.split('\n');
// Split headers line into an array
// `valueSeparator` may come from some kind of argument
// You may want to transform header strings into something more
// usable, like `camelCase` or `lowercase-space-to-dash`
const valueSeparator = '\t';
const headers = headerLine.split(valueSeparator);
// Create objects from parsing lines
// There will be as much objects as lines
const objects = lines
.map( (line, index) =>
line
// Split line with value separators
.split(valueSeparator)
// Reduce values array into an object like: { [header]: value }
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce
.reduce(
// Reducer callback
(object, value, index) => ({
...object,
[ headers[index] ]: value,
}),
// Initial value (empty JS object)
{}
)
);
console.log("Objects:", objects);
For CSV files using , as separator and quotes string values, you can use this version:
// Split data into lines and separate headers from actual data
// using Array spread operator
const [headerLine, ...lines] = data.split('\n');
// Use common line separator, which parses each line as the contents of a JSON array
const parseLine = (line) => JSON.parse(`[${line}]`);
// Split headers line into an array
const headers = parseLine(headerLine);
// Create objects from parsing lines
// There will be as much objects as lines
const objects = lines
.map( (line, index) =>
// Split line with JSON
parseLine(line)
// Reduce values array into an object like: { [header]: value }
.reduce(
(object, value, index) => ({
...object,
[ headers[index] ]: value,
}),
{}
)
);
return objects;
Note: For big files, it would be better to work with streams, generators, iterators, etc.
You could try using MVC Razor,
<script type="text/javascript">
MyNamespace.myConfig = #Html.Raw(Json.Encode(new MyConfigObject()));
</script>
The Json.Encode will serialize the initialized object to JSON format. Then the Html.Raw will prevent it from encoding the quotes to ".
Here the entire example
You can use lodash (or underscore) to help with this.
var objects = _.map(arr, function(item){return item.split(',');});
var headers = objects[0];
objects.splice(0, 1); // remove the header line
populatedObject = [];
objects.forEach(function(item){
var obj = _.zipObject(headers, item);
populatedObject.push(obj);
});
The .zipObject method will match each header to each value in the items array and produce an object.
Here is a solution if you already have an array and want that the csv header (first line) to be the object's property.
const csvArrayToObj = (csvData) => {
return csvData
.map((csvLine, csvIndex) => {
if (csvIndex === 0 || !csvLine.length) return null; // skip header and empty lines
return csvLine.reduce((a, v, i) => ({ ...a, [csvData[0][i]]: v }), {});
})
.filter((filter) => !!filter); //filter empty lines
};
const csvArray = [
['name', 'age'],
['John Doe', 20],
['Jane Doe', 30],
];
csvArrayToObj(csvArray);
// output
[
{
"name": "John Doe",
"age": 20
},
{
"name": "Jane Doe",
"age": 30
}
]

JSON.stringify serializes to [[]]

If I create a JavaScript object like:
var lst = [];
var row = [];
row.Col1 = 'val1';
row.Col2 = 'val2';
lst.push(row);
And then convert it to a string:
JSON.stringify(lst);
The result is an object containing an empty object:
[[]]
I would expect it to serialize like:
[[Col1 : 'val1', Col2: 'val2']]
Why do the inner objects properties not serialize?
Code snippet at JSFiddle.
Because row is an array, not an object. Change it to:
var row = {};
This creates an object literal. Your code will then result in an array of objects (containing a single object):
[{"Col1":"val1","Col2":"val2"}]
Update
To see what really happens, you can look at json2.js on GitHub. This is a (heavily reduced) snippet from the str function (called by JSON.stringify):
if (Object.prototype.toString.apply(value) === '[object Array]') {
//...
length = value.length;
for (i = 0; i < length; i += 1) {
partial[i] = str(i, value) || 'null';
}
//...
}
//...
for (k in value) {
if (Object.prototype.hasOwnProperty.call(value, k)) {
//...
}
//...
}
//...
Notice that arrays are iterated over with a normal for loop, which only enumerates the array elements. Objects are iterated with a for...in loop, with a hasOwnProperty test to make sure the proeprty actually belongs to this object.
You use your inner array like an object, so make it an object instead of an array.
var lst = [];
var row = {};
row.Col1 = 'val1';
row.Col2 = 'val2';
lst.push(row);
or use it as an array
var lst = [];
var row = {};
row.push( 'val1' );
row.push( 'val2' );
lst.push(row);
You want row to be a dictionary, not a vector. Define it like this:
var row = {};
Since an array is a datatype in JSON, actual instances of Array are stringified differently than other object types.
If a JavaScript Array instance got stringified with its non-numeric keys intact, it couldn't be represented by the [ ... ] JSON array syntax.
For instance, [ "Col1": "val1"] would be invalid, because JSON arrays can't have explicit keys.
{"Col1": "val1"} would be valid - but it's not an array.
And you certainly can't mix'n'match and get { "Col1": "val1", 1, 2, 3 ] or something.
By the way, this works fine:
var lst = [];
var row = {};
row.Col1 = 'val1';
row.Col2 = 'val2';
lst.push(row);
alert(JSON.stringify(lst));​

Categories