We Keep Coding

GroupBy Array elements from CSV file and help reduce code

Each csv file that is imported has the same data structure.
I need to sum the ['Net Charge Amount'] by each '[Service Type'].
I am currently doing this by assigning each unique ['Service Type'] to their own array. My current script is probably overkill but it is very easy to follow, however I am looking for a more compact way of doing this otherwise this script could get very long.
const fs = require('fs')
const { parse } = require('csv-parse')
// Arrays for each service type
const GroundShipments = []
const HomeDeliveryShipments = []
const SmartPostShipments = []
const Shipments = []
The [Shipments] array will hold all data and I would assume this is the array
we want to work with
//functions for each service type
function isGround(shipment) {
return shipment['Service Type'] === 'Ground'
}
function isHomeDelivery(data) {
return data['Service Type'] === 'Home Delivery'
}
function isSmartpost(shipment) {
return shipment['Service Type'] === 'SmartPost'
}
function isShipment(shipment) {
return shipment['Service Type'] === 'Ground' || shipment['Service Type'] === 'Home Delivery' ||
shipment['Service Type'] === 'SmartPost'
}
// Import csv file / perform business rules by service type
// output sum total by each service type
fs.createReadStream('repco.csv')
.pipe(parse({
columns: true
}))
.on('data', (data) => {
//push data to proper service type array
// Ground
if (isGround(data)) {
GroundShipments.push(data)
}
// Home Delivery
if (isHomeDelivery(data)) {
HomeDeliveryShipments.push(data)
}
// Smartpost
if (isSmartpost(data)) {
SmartPostShipments.push(data)
}
// All shipment types, including Ground, Home Delivery, and Smartpost
if (isShipment(data)) {
Shipments.push(data)
}
})
.on('error', (err) => {
console.log(err)
})
.on('end', (data) => {
// sum data by service type
// Ground Only
const sumGround = GroundShipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
// Home Delivery Only
const sumHomeDelivery = HomeDeliveryShipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
// SmartPost Only
const sumSmartPost = SmartPostShipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
// All services
const sumAllShipments = Shipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
//output sum by service type to console
console.log(`${GroundShipments.length} Ground shipments: ${sumGround}`)
console.log(`${HomeDeliveryShipments.length} Home Delivery shipments: ${sumHomeDelivery}`)
console.log(`${SmartPostShipments.length} Smartpost shipments: ${sumSmartPost}`)
console.log(`${Shipments.length} All shipments: ${sumAllShipments}`)
})
Here is the console output:
[1]: https://i.stack.imgur.com/FltTU.png
Instead of separating each ['Service Type'] by its own Array and Function, I would like one Array [Shipments] to output each unique ['Service Type'] and sum total of ['Net Charge Amount']

The two keys to simplifying this are:
separating the CSV parsing from the data processing
using a groupBy function
First, you should parse the CSV into a simple JS array. Then you can use regular JS utility functions to operate on the data, such as the groupBy function. It is a utility that can be found in the lodash and ramda libraries. It's probably going to be added to vanilla JS as the .group method but that's a while from now.
I was looking for a sample problem to play with my own JS evaluation framework, so I answered your question there:
You can explore the underlying val yourself: https://www.val.town/stevekrouse.exampleGroupByShppingCSV
There are a couple things about my answer that wouldn't make sense in a normal NodeJS codebase, but that I had to do to make it work in val.town (async/await, using a custom groupBy method instead of importing one). If you'd like help getting it to work in your application, just let me know.

A solution would be to use a Map instance to keep track of the stats of different service types.
For each shipment find the associated stats (based on service type), or create a new stats object { count: 0, sum: 0 }. Then increment the count, and add the amount to the sum.
When all data is iterated (on end), you can loop through the serviceTypeStats which and log the values. You can also use this loop to calculate the total by adding all count and sum of each service type group.
const serviceTypeStats = new Map();
// ...
.on('data', (shipment) => {
const serviceType = shipment['Service Type'];
const amount = parseFloat(shipment['Net Charge Amount']);
if (!serviceTypeStats.has(serviceType)) {
serviceTypeStats.set(serviceType, { count: 0, sum: 0 });
}
const stats = serviceTypeStats.get(serviceType);
stats.count += 1;
stats.sum += amount;
})
// ...
.on('end', () => {
const total = { count: 0, sum: 0 };
for (const [serviceType, stats] of serviceTypeStats) {
total.count += stats.count;
total.sum += stats.sum;
console.log(`${stats.count} ${shipmentType}: ${stats.sum}`);
}
console.log(`${total.count} All shipments: ${total.sum}`);
})
If you want to loop keys in a specific order you can define the order in an array, or sort the keys of the Map instance.
// pre-defined order
const serviceTypeOrder = ["Ground", "Home Delivery", "SmartPost"];
// or
// alphabetic order (case insensitive)
const serviceTypeOrder = Array.from(serviceTypeStats.keys());
serviceTypeOrder.sort((a, b) => a.localeCompare(b, undefined, { sensitivity: "base" }));
// ...
for (const serviceType of sericeTypeOrder) {
const stats = serviceTypeStats.get(serviceType);
// ...
}

Javascript - Add Item in the middle of an array inside map function

I'm trying to add an item in a specific index inside an array inside a map function and it's been behaving unexpectedly. Here's the code for it
const addItemToLevelTwoArray= (uniqueID, arrayID )=> {
const reportObject = {
id:arrayID,
title:'',
}
data.map(section=>{
section.content.map((report, reportIndex)=>{
if(report.id===uniqueID){
section.content.splice(reportIndex, 0, reportObject);
}
return report;
})
return section;
})
}
Here's a working pen - https://codepen.io/raufabr/pen/vYZYgOV?editors=0011
Expected behaviour is that it would insert an object in the specific index, right above the object where the ID matches.
However, it's acting weirdly and sometimes I'm getting 2 items being added instead of one.
Any tip on what I'm doing would be massively appreciated! I know I'm close but I've been stuck on this for a while now and can't figure out what I'm doing wrong!

Preface: You're using map incorrectly. If you're not using the array that map builds and returns, there's no reason to use it; just use a loop or forEach. More in my post here. And one reason to use an old-fashioned for loop is that you're in control of iteration, which matters because...
However, it's acting weirdly and sometimes I'm getting 2 items being added instead of one.
That's because you're inserting into the array being looped by the map, so on the next pass, it picks up the entry you're adding.
If you do a simple loop, you can easily avoid that by incrementing the index when you insert, or by looping backward; here's the looping backward approach:
const addItemToLevelTwoArray = (uniqueID, arrayID) => {
const reportObject = {
id: arrayID,
title: "",
};
for (const section of data) {
for (let reportIndex = section.content.length - 1; reportIndex >= 0; --reportIndex) {
const report = section.content[reportIndex];
if (report.id === uniqueID) {
section.content.splice(reportIndex, 0, reportObject);
}
}
}
};
Because we're looping backward, we won't pick up the entry we just added on the next pass.
Since the outer loop doesn't have that problem, I used the more convenient for-of.
Since you asked about map, if you do use the array map returns, you can do this by returning an array with the two entries, and then calling flat on the array map builds. (This only works if the array doesn't already contain arrays, because they'll get flattened to.) This is common enough that it's combined in one function: flatMap. It's not what I'd do (I'd do a loop), but it's certainly feasible. Sticking with forEach and flatMap rather than using for-of and for:
const addItemToLevelTwoArray = (uniqueID, arrayID) => {
const reportObject = {
id: arrayID,
title: "",
}
data.forEach(section => {
section.content = section.content.flatMap(report => {
if (report.id === uniqueID) {
// Return the new one and the old one
return [reportObject, report];
}
// Return just the old one
return report;
});
});
};
That assumes it's okay to modify the section object. If it isn't, Alberto Sinigaglia's answer shows creating a new replacement object instead, which is handy in some sitautions.

You can just use flatMap:
const data = [
{
content: [
{
id: 1,
title: "a"
},{
id: 3,
title: "c"
},
]
}
]
const addItemToLevelTwoArray= (uniqueID, arrayID )=> {
const reportObject = {
id:arrayID,
title:'',
}
return data.map(section=> {
return {
...section,
content: section.content.flatMap( report =>
report.id === uniqueID
? [reportObject, report]
: report
)
}
}
)
}
console.log(addItemToLevelTwoArray(3, 2))

The following will extend the inner array .contentwithout modifying the original array data:
const data = [ {id: 0,title:'main',content:[{id:1,title:'Hello'},
{id:2,title:"World"}] } ];
const addItemToLevelTwoArray= (uniqueID, arrayID )=> {
const reportObject = {
id:arrayID,
title:'something new!',
}
return data.map(d=>(
{...d, content:d.content.reduce((acc, rep)=>{
if(rep.id===uniqueID) acc.push(reportObject);
acc.push(rep)
return acc;
},[]) // end of .reduce()
})); // end of .map()
}
const res=addItemToLevelTwoArray(1,123);
console.log(res);

How to number shuffled elements in array?

In my application, I randomized the sequence of specific views. However, I have a problem with displaying the correct sequence number of view.
So, the first view should get number 1, second view number 2, etc.. Since they get shuffled in an array, I have no idea how to access the order number of the view beforehand. The correct number should then get passed as a prop to the specific view component in order to display it.
shuffle(arr) {
var i, j, temp;
for (i = arr.length - 1; i > 0; i--) {
j = Math.floor(Math.random() * (i + 1));
temp = arr[i];
arr[i] = arr[j];
arr[j] = temp;
}
return arr;
};
constructor() {
const componentArray = [<Tasks incrementSequenceCounter={this.incrementSequenceCounter} count={this.state.count} />, <APM incrementSequenceCounter={this.incrementSequenceCounter} count={this.state.count} />, <ICAA incrementSequenceCounter={this.incrementSequenceCounter} count={this.state.count} />]
const shuffledArray = this.shuffle(componentArray);
this.state.shuffledArray = shuffledArray;
}
Somehow the component should be aware of its index in the array, but I don't know where to start.

Firstly I don't like the idea of initialising components in the constructor. That basically makes them static, but that may be what you're after. Here's my attempt:
constructor() {
const componentArray = [
{
type: Tasks,
props: {
incrementSequenceCounter: this.incrementSequenceCounter,
count: this.state.count
}
},
{
type: APM,
props: {
incrementSequenceCounter: this.incrementSequenceCounter,
count: this.state.count
}
},
{
type: ICAA,
props: {
incrementSequenceCounter: this.incrementSequenceCounter,
count: this.state.count
}
}
const shuffledArray = this.shuffle(componentArray);
this.state.shuffledArray = shuffledArray.map(
(componentConstructor, index) => React.createElement(componentConstructor.type, { ...componentConstructor.props, index })
);
}
The basic idea is to construct them after you've determined the order. The obvious disadvantage here is that since this is happening in the constructor any state changes in the parent component are not reflected in these children components. If you don't want that then this should be moved in render and/or componentDidMount/Update
Note: Based on other answers I need to clarify that in my understanding the question is how to pass the index the component ends up in after shuffling to the component itself. This differs from how others have answered so if I am understanding it incorrectly let me know

Your array's element should be wrapped in an object before shuffling:
const componentArray = [
{
index: 0,
component: <MyComponent />
}
];
You could create it from your array with a simple map():
const indexArray = componentArray.map((el, i) => ({ index: i, component: el });
Then you can safely shuffle your indexArray

Javascript's method forEach() creates array with undefined keys

I am building a simple todo app, and I'm trying to get the assigned users for each task. But let's say that in my database, for some reason, the tasks id starts at 80, instead of starting at 1, and I have 5 tasks in total.
I wrote the following code to get the relationship between user and task, so I would expect that at the end it should return an array containing 5 keys, each key containing an array with the assigned users id to the specific task.
Problem is that I get an array with 85 keys in total, and the first 80 keys are undefined.
I've tried using .map() instead of .forEach() but I get the same result.
let assignedUsers = new Array();
this.taskLists.forEach(taskList => {
taskList.tasks.forEach(task => {
let taskId = task.id;
assignedUsers[taskId] = [];
task.users.forEach(user => {
if(taskId == user.pivot.task_id) {
assignedUsers[taskId].push(user.pivot.user_id);
}
});
});
});
return assignedUsers;
I assume the issue is at this line, but I don't understand why...
assignedUsers[taskId] = [];
I managed to filter and remove the empty keys from the array using the line below:
assignedUsers = assignedUsers.filter(e => e);
Still, I want to understand why this is happening and if there's any way I could avoid it from happening.
Looking forward to your comments!

If your taskId is not a Number or autoconvertable to a Number, you have to use a Object. assignedUsers = {};

This should work as you want it to. It also uses more of JS features for the sake of readability.
return this.taskLists.reduce((acc, taskList) => {
taskList.tasks.forEach(task => {
const taskId = task.id;
acc[taskId] = task.users.filter(user => taskId == user.pivot.task_id);
});
return acc;
}, []);
But you would probably want to use an object as the array would have "holes" between 0 and all unused indexes.

Your keys are task.id, so if there are undefined keys they must be from an undefined task id. Just skip if task id is falsey. If you expect the task id to possibly be 0, you can make a more specific check for typeof taskId === undefined
this.taskLists.forEach(taskList => {
taskList.tasks.forEach(task => {
let taskId = task.id;
// Skip this task if it doesn't have a defined id
if(!taskId) return;
assignedUsers[taskId] = [];
task.users.forEach(user => {
if(taskId == user.pivot.task_id) {
assignedUsers[taskId].push(user.pivot.user_id);
}
});
});
});

Conditional observables with forkJoin

I have a case where I may or may not need to add observables to a list. I then want to forkJoin the observables I do have so the page can load once all of the data is available.
let observables: Observable<any>[] = [];
observables.push(this.taskService.getStep(this.housingTransactionId, this.task.stageReferenceId, this.task.stepReferenceId));
if (this.task.associatedChatThreadId) {
observables.push(this.messageHubService.getChatThread(this.housingTransactionId, this.task.associatedChatThreadId));
}
if (this.task.associatedDocuments && this.task.associatedDocuments.length > 0) {
this.task.associatedDocuments.forEach(documentId => {
observables.push(this.documentHubService.getDocumentProperties(this.housingTransactionId, documentId));
});
}
Observable.forkJoin(observables)
.subscribe(([step, chatThread, ...documents]) => {
this.step = step;
this.chatThread = chatThread;
this.documents = documents;
this.isPageLoading = false;
}, error => {
this.isPageLoading = false;
console.log(error);
});
The problem I'm getting is that if I don't have a this.task.associatedChatThreadId, then the observable is not added to the list and when the forkJoin is executed, the ...documents are in the position of the chatThread property in the subscribe method (well, the first document!).
Is there a way to ensure the positioning of the responses from a forkJoin? Or should I/can I use a different approach?

Most easily you can add a dumb Observable.of(null) with null value if the condition is not met in order to keep the same order of responses:
if (this.task.associatedChatThreadId) {
observables.push(this.messageHubService....);
} else {
observables.push(Observable.of(null))
}
Then in the subscription you can check if chatThread === null becauese it'll always be present at the same position.
Alternatively, you could wrap each Observable in observables with some extra object that would make it uniquely identifiable in the subscriber but that would be unnecessarily complicated so I'd personally stick to the first option.

Another approach would be not to use folkJoin but subscribe separately. At the same time, make isPageLoading a BehaviorSubject which counts how many async requests you currently have. Each time when you make a request, you can have isPageLoading.next(1), and isPageLoading.next(-1) when you finish a request.

You could make a helper function that accepts an object which has string keys and observable values and returns an observable that will emit an object with the same keys, but having the resulting values instead of the observables as values.
I would not really say that this is a cleaner version than using of(null) like suggested by martin, but it might be an alternative.
function namedForkJoin(map: {[key: string]: Observable<any>}): Observable<{[key: string]: any}> {
// Get object keys
const keys = Object.keys(map);
// If our observable map is empty, we want to return an empty object
if (keys.length === 0) {
return of({});
}
// Create a fork join operation out of the available observables
const forkJoin$ = Observable.forkJoin(...keys.map(key => map[key]))
return forkJoin$
.map(array => {
const result = {};
for (let index = 0; index < keys.length; index++) {
result[keys[index]] = array[index];
}
}));
}
Please keep in mind, I did not have angular or rxjs running here at the moment, so I could not verify the function really works. But the idea is:
1. Get the keys from the input map.
2. Use the keys to get an array of observables and pass that to fork join.
3. Add a mapping function that converts the resulting array back into an object.