We Keep Coding

GroupBy Array elements from CSV file and help reduce code

Each csv file that is imported has the same data structure.
I need to sum the ['Net Charge Amount'] by each '[Service Type'].
I am currently doing this by assigning each unique ['Service Type'] to their own array. My current script is probably overkill but it is very easy to follow, however I am looking for a more compact way of doing this otherwise this script could get very long.
const fs = require('fs')
const { parse } = require('csv-parse')
// Arrays for each service type
const GroundShipments = []
const HomeDeliveryShipments = []
const SmartPostShipments = []
const Shipments = []
The [Shipments] array will hold all data and I would assume this is the array
we want to work with
//functions for each service type
function isGround(shipment) {
return shipment['Service Type'] === 'Ground'
}
function isHomeDelivery(data) {
return data['Service Type'] === 'Home Delivery'
}
function isSmartpost(shipment) {
return shipment['Service Type'] === 'SmartPost'
}
function isShipment(shipment) {
return shipment['Service Type'] === 'Ground' || shipment['Service Type'] === 'Home Delivery' ||
shipment['Service Type'] === 'SmartPost'
}
// Import csv file / perform business rules by service type
// output sum total by each service type
fs.createReadStream('repco.csv')
.pipe(parse({
columns: true
}))
.on('data', (data) => {
//push data to proper service type array
// Ground
if (isGround(data)) {
GroundShipments.push(data)
}
// Home Delivery
if (isHomeDelivery(data)) {
HomeDeliveryShipments.push(data)
}
// Smartpost
if (isSmartpost(data)) {
SmartPostShipments.push(data)
}
// All shipment types, including Ground, Home Delivery, and Smartpost
if (isShipment(data)) {
Shipments.push(data)
}
})
.on('error', (err) => {
console.log(err)
})
.on('end', (data) => {
// sum data by service type
// Ground Only
const sumGround = GroundShipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
// Home Delivery Only
const sumHomeDelivery = HomeDeliveryShipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
// SmartPost Only
const sumSmartPost = SmartPostShipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
// All services
const sumAllShipments = Shipments.reduce((acc, data) =>
acc + parseFloat(data['Net Charge Amount']), 0)
//output sum by service type to console
console.log(`${GroundShipments.length} Ground shipments: ${sumGround}`)
console.log(`${HomeDeliveryShipments.length} Home Delivery shipments: ${sumHomeDelivery}`)
console.log(`${SmartPostShipments.length} Smartpost shipments: ${sumSmartPost}`)
console.log(`${Shipments.length} All shipments: ${sumAllShipments}`)
})
Here is the console output:
[1]: https://i.stack.imgur.com/FltTU.png
Instead of separating each ['Service Type'] by its own Array and Function, I would like one Array [Shipments] to output each unique ['Service Type'] and sum total of ['Net Charge Amount']

The two keys to simplifying this are:
separating the CSV parsing from the data processing
using a groupBy function
First, you should parse the CSV into a simple JS array. Then you can use regular JS utility functions to operate on the data, such as the groupBy function. It is a utility that can be found in the lodash and ramda libraries. It's probably going to be added to vanilla JS as the .group method but that's a while from now.
I was looking for a sample problem to play with my own JS evaluation framework, so I answered your question there:
You can explore the underlying val yourself: https://www.val.town/stevekrouse.exampleGroupByShppingCSV
There are a couple things about my answer that wouldn't make sense in a normal NodeJS codebase, but that I had to do to make it work in val.town (async/await, using a custom groupBy method instead of importing one). If you'd like help getting it to work in your application, just let me know.

A solution would be to use a Map instance to keep track of the stats of different service types.
For each shipment find the associated stats (based on service type), or create a new stats object { count: 0, sum: 0 }. Then increment the count, and add the amount to the sum.
When all data is iterated (on end), you can loop through the serviceTypeStats which and log the values. You can also use this loop to calculate the total by adding all count and sum of each service type group.
const serviceTypeStats = new Map();
// ...
.on('data', (shipment) => {
const serviceType = shipment['Service Type'];
const amount = parseFloat(shipment['Net Charge Amount']);
if (!serviceTypeStats.has(serviceType)) {
serviceTypeStats.set(serviceType, { count: 0, sum: 0 });
}
const stats = serviceTypeStats.get(serviceType);
stats.count += 1;
stats.sum += amount;
})
// ...
.on('end', () => {
const total = { count: 0, sum: 0 };
for (const [serviceType, stats] of serviceTypeStats) {
total.count += stats.count;
total.sum += stats.sum;
console.log(`${stats.count} ${shipmentType}: ${stats.sum}`);
}
console.log(`${total.count} All shipments: ${total.sum}`);
})
If you want to loop keys in a specific order you can define the order in an array, or sort the keys of the Map instance.
// pre-defined order
const serviceTypeOrder = ["Ground", "Home Delivery", "SmartPost"];
// or
// alphabetic order (case insensitive)
const serviceTypeOrder = Array.from(serviceTypeStats.keys());
serviceTypeOrder.sort((a, b) => a.localeCompare(b, undefined, { sensitivity: "base" }));
// ...
for (const serviceType of sericeTypeOrder) {
const stats = serviceTypeStats.get(serviceType);
// ...
}

Display variable and array

Guys I have a question.
I have an app in which I register data from an external device using BLE.
I have a "time" and an array for "acceleration".
const time = parseInt(
Buffer.from(characteristic.value, "base64")
.readUInt16LE(0)
.toString(16),
16
);
const acc_dx = [2, 4, 6].map(index => {
const hex = Buffer.from(characteristic.value, "base64")
.readInt16LE(index)
.toString(16);
return Number.parseInt(hex, 16);
});
const accUpdate_acc_dx = [...this.state.array_acc_dx, this.state.time, this.state.acc_dx]
this.setState({ array_acc_dx: accUpdate_acc_dx })
the result for array_acc_dx is like:
[1520,[42,-419,-926],1520,[41,-420,-927],1520,[41,-421,-927],1520,[41,-421,-926],1580,[40,-420,-927],1640,[40,-420,-926],1640,[41,-420,-926],1640,[41,-419,-926]
I would obtain this:
1520: [42,-419,-926],
1520: [41,-420,-927],
1520: [41,-421,-927],
1580: [40,-420,-927],
How can I do to have this kind of array?

I've cleaned up a few bits in your code, but I assume the following code does what you want it to do.
I don't think it is necessary to use a seperate buffer for each index you want to read.
I've already mentioned the part about converting a number to hex, just to immediately parse it back to a number; that's useless code.
you can parse the values for time and acc_dx in one go. No need to duplicate code.
when updating the state based on a previous state, use this.setState(previousState => newState)
const buf = Buffer.from(characteristic.value, "base64");
const [time, ...acc_dx] = [0,2,4,6].map(index => buf.readInt16LE(index));
this.setState(state => ({
time,
acc_dx,
array_acc_dx: [
...state.array_acc_dx,
[time, acc_dx]
]
}));

RxJS Debounce with priority

I'm having trouble coming up with this stream.
What I'm looking for is something like debounceTime but with priority.
So if I have events with the shape { type: 'a', priority: 2 }. These events needs to be debounced by a few seconds but instead of the last event being emitted, the event with the highest priority is emitted.
input stream:
------(a|1)--(b|3)---(c|2)-----------------------(a|1)-----------------
output stream:
-----------------------------------(b|3)---------------------(a|1)-----
I've try looking at other operators like window and filtering through the result for the last event but it's not ideal because window work on a fixed cadence where I want the timer to start on the first event like debouncing does.

You have to store and update the item with the highest priority and map to this highest value which you then pass to debounceTime.
let highest = null;
source$.pipe(
map(v => highest = highest && highest.priority > v.priority ? highest : v),
debounceTime(2000),
tap(() => highest = null)
);
You can create your own operator that does this with the help of defer. defer makes sure that every subscriber gets its own highest variable, as every subscriber will get its own new Observable created by calling the given factory function.
function debounceTimeHighest<T>(dueTime: number, getHighest: (curr: T, high: T) => T): MonoTypeOperatorFunction<T> {
return (source: Observable<T>) => defer(() => {
let highest: T = null;
return source.pipe(
map(item => highest = highest ? getHighest(item, highest) : item),
debounceTime(dueTime),
tap(() => highest = null)
);
});
}
// Usage
source$.pipe(
debounceTimeHighest(2000, (v1, v2) => v1.priority >= v2.priority ? v1 : v2)
)
The code above is Typescript. If you want plain Javascript just remove all the types.
https://stackblitz.com/edit/rxjs-hitqxk

I'll offer the following solution, based around using scan to offer up the highest given priority emission so far for consideration by debounceTime(). Note that scan needs to reconsider new data after every successful debounce, so I use the operator window() to split up the emissions, starting a new observable window after every emission by debounceTime().
Here is the CodeSandbox
And here is some simplified code from the CodeSandbox showing the important bits:
const resetScan$ = new Subject();
source$.pipe(
window(resetScan$),
mergeMap(win$ => win$.pipe(
scan((acc, cur) => acc.priority >= cur.priority ? acc : cur )
)),
debounceTime(debounceDelay),
tap(() => resetScan$.next())
);

You can combine the debounceTime and buffer and filter operator to achieve what you need. I have developed this small example for it.
https://stackblitz.com/edit/typescript-lwzt4k
/*
Collect clicks that occur, after 250ms emit array of clicks
*/
clicks$.pipe(
buffer(clicks$.pipe(debounceTime(1000))),
// if array is greater than 1, double click occured
map((clickArray) => {
document.querySelector('#emittedObjects').innerHTML = (`<div>${JSON.stringify(clickArray)}</div>`);
const sortedArray = clickArray.sort((a, b) => {
return a.priority < b.priority ? 1 : -1;
});
const output = sortedArray.length > 0 ? sortedArray[0] : null;
document.querySelector('#mappedOutput').innerHTML = JSON.stringify(output);
return output;
})
)
.subscribe((obj) => {
const str = obj ? JSON.stringify(obj) : 'NULL';
document.querySelector('#throttledOutput').innerHTML = `<div>THROTTLED: ${str}</div>`;
});

RxJs zip operator equivalent in xstream?

Hello I'm trying to figure out if there is an equivalent to the RxJs operator zip in xstream, or at least a way to get the same behaviour. In case anyone needs clarification on the difference the marble diagrams below will show.
zip in rxjs
|---1---2---3-----------5->
|-a------b------c---d----->
"zip"
|-1a----2b------3c-----5d->
whereas 'combineLatest' aka 'combine' in xstream does
|---1---2----------4---5->
|----a---b---c---d------->
"combine"
|-1a----2a-2b-2c-2d-4d-5d>
Any help is appreciated as I'm very new to programming with streams. Thank you in advance!

I also needed a zip operator for xstream. So I created my own from existing operators. It takes an arbitrary number of streams for zipping.
function zip(...streams) {
// Wrap the events on each stream with a label
// so that we can seperate them into buckets later.
const streamsLabeled = streams
.map((stream$, idx) => stream$.map(event => ({label: idx + 1, event: event})));
return (event$) => {
// Wrap the events on each stream with a label
// so that we can seperate them into buckets later.
const eventLabeled$ = event$.map(event => ({label: 0, event: event}));
const labeledStreams = [eventLabeled$, ...streamsLabeled];
// Create the buckets used to store stream events
const buckets = labeledStreams.map((stream, idx) => idx)
.reduce((buckets, label) => ({...buckets, [label]: []}), {});
// Initial value for the fold operation
const accumulator = {buckets, tuple: []};
// Merge all the streams together and accumulate them
return xs.merge(...labeledStreams).fold((acc, event) => {
// Buffer the events into seperate buckets
acc.buckets[event.label].push(event);
// Does the first value of all the buckets have something in it?
// If so, then there is a complete tuple.
const tupleComplete = Object.keys(acc.buckets)
.map(key => acc.buckets[key][0])
.reduce((hadValue, value) => value !== undefined
? true && hadValue
: false && hadValue,
true);
// Save completed tuple and remove it from the buckets
if (tupleComplete) {
acc.tuple = [...Object.keys(acc.buckets).map(key => acc.buckets[key][0].event)];
Object.keys(acc.buckets).map(key => acc.buckets[key].shift());
} else {
// Clear tuple since all columns weren't filled
acc.tuple = [];
}
return {...acc};
}, accumulator)
// Only emit when we have a complete tuple
.filter(buffer => buffer.tuple.length !== 0)
// Just return the complete tuple
.map(buffer => buffer.tuple);
};
}
This can be used with compose.
foo$.compose(zip(bar$)).map(([foo, bar]) => doSomething(foo, bar))

How to "synchronize" Observables?

I have this piece of code:
ngOnInit(): void
{
this.categories = this.categoryService.getCategories();
var example = this.categories.flatMap((categor) => categor.map((categories) => {
var links = this.categoryService.countCategoryLinks(categories.id)
.subscribe(valeur => console.log(valeur));
return categories.id
}));
}
The result are two observables.
One consists in a list of categories.
The second one is the number of items for a particular categories.id.
My question is as follow:
How could I get all this information structured in a particular data structure?
I would like to store categories and the number of items per category in the same data structure to be able to show them up in my TS component.
I went step by step trying to fix my issues and I went to have the following code that is almost the solution:
this.categories = this.categoryService.getCategories();
var example = this.categories.mergeMap((categor) => categor.map((myCateg) =>
{
this.categoryService.countCategoryLinks(myCateg.id)
.map(numlinks => Object.assign(myCateg,{numLinks: numlinks}))
.subscribe(valeur => console.log(valeur));
return myCateg.id
}));
It gives the following output:
Where numLinks is still an object... (containing my count value) Any idea on how to transform it to a json property like categoryName or id??
Thanks in advance and Regards,
Here is the solution to the problem:
ngOnInit(): void
{
this.categories = this.categoryService.getCategories();
const example = this.categories
.mergeMap((categor) => categor
.map((myCateg) => {
this.categoryService.countCategoryLinks(myCateg.id)
.map(numlinks => {
myCateg.numlinks = numlinks.count;
return myCateg;
})
//Object.assign(myCateg,{numLinks: numlinks}))
.subscribe(value => console.log(value));
return myCateg
}));
example.subscribe(val => console.log("value2: "+val));
}

Once more, the solution comes from the mergeMap() operator. :-)
this.categoryService.getCategories()
.mergeMap(val => val) // "Flatten" the categories array
.mergeMap(category =>
this.categoryService.countCategoryLinks(category.id)
// Write the number of links on the `category` object
.map(numLinks => Object.assign(category, {numLinks: numLinks}))
)
.toArray()
.subscribe(allCategoriesWithNumLinks => console.log(allCategoriesWithNumLinks));
I'm not going into the specifics (the first mergeMap to flatten the array, Object.assign() to produce the final object) since it seems like we covered all that in a previous thread we had, but feel free to ask questions if anything is unclear.
Philippe's questions:
Why flatten the categories array? I'm assuming getCategories() emits a SINGLE array containing all the categories. Since you want to run an HTTP request for each category, it's more convenient to have an observable emitting each category individually. That's what the first mergeMap() does: it transforms Observable<Category[]> into Observable<Category>.
Why create an object? You said you wanted to store everything in the same data structure. That's what Object.assign does: it writes the number of links found for each category on the category object itself. This way, you end up with ONE object containing the information from TWO observables (category + numLinks).