We Keep Coding

Zapier breaks down arrays of objects into distinct arrays containing the keys of the objects

Hi everyone I am having an issue with code by zapier (javascript)... (I know, I know)
The issue I am having is I am bringing in my data from airtable. The data comes in as three distinct arrays. They are:
specCategory[]
specName[]
specDescription[]
I iterate over the arrays and split them out by the commas in the arrays. I then form each of these values in each of the arrays into their own object. I then push that object into an array.
My end goal is to push a JSON payload into PDFMonkey in the form:
{
"payload": [
{
specName: "specName data",
specCategory: "specCategory data",
specDescription: "specDescription data"
},
{
specName: "specName data",
specCategory: "specCategory data",
specDescription: "specDescription data"
},
{
specName: "specName data",
specCategory: "specCategory data",
specDescription: "specDescription data"
}
]
}
Zapier seems to return the correct payload for me. That is, an array of objects. However, when I go to access the data in a subsequent step, the data is returned back into three distinct arrays again.
Here is what the output from the zapier code looks like.
specArray
1
specCategory
Kitchen Appliances
specName
Gas Hob
specDescription
Westinghouse WHG 643 SA Gas Hob
2
specCategory
Kitchen Appliances
specName
Range Hood
specDescription
Westinghouse WRR 614 SA Range Hood
3
specCategory
Kitchen Appliances
specName
Oven
specDescription
Westinghouse WVE 613 S Oven
4
specCategory
Doors and Windows (Internal)
specName
Architraves
specDescription
42X12min Splayed Profile F/Joint Pine painted gloss
5
specCategory
External Stairs
specName
External Stair Balustrade
specDescription
Painted pre-primed ladies waist handrail with slats and bottom rails (not included if stair is under 1m in height)
Instead when accessing it in subsequent steps I receive three distinct arrays like:
specArraySpecName: specName[1],specName[2],specName[...],
specArraySpecCategory: specCategory[1],specCategory[2],specCategory[...],
specArraySpecDescription: specDescription[1],specDescription[2],specDescription[...]
Here is my code so you can have a look and see if what I am doing is wrong. When I try to output just the array of objects (instead of first wrapping the array in object tags) it outputs the single value of each object but the problem is that makes zapier loop the subsequent steps each time using each object as an input.
Is there a way to flatten or stringify the JSON object I am trying to create?
My code below for reference:
// this is wrapped in an async function
// you can use await throughout the function
let categories = inputData.specCategories.split(/\s*,\s*/);
let names = inputData.specName.split(/\s*,\s*/);
let descriptions = inputData.specDescriptions.split(/\s*,\s*/);
let specArray = [];
// for loop to input each of the discrete items into an array
for (let i = 0; i < categories.length; i++) {
let spec = {
specCategory: categories[i],
specName: names[i],
specDescription: descriptions[i]
specArray.push(spec);
}
output = { specArray };
I apologise in advance for the formatting but stack overflow would not let me post code blocks due to some not properly formatted code (tried ctrl + k, 4 space, triple back ticks etc) and I could not figure it out.
Thanks for your help!

Great question! It's worth mentioning that this is Zapier "working as expected"... for use cases that don't match yours. This behavior supports line items, a common structure in Accounting & E-Commerce. But, that doesn't help you, but you want Zapier to stop messing with your nicely structured values.
The best way to handle this is probably to stringify the whole JSON. Zapier only mangles arrays, so it'll leave a JSON string unharmed.
That would be something like this in the code:
// ...
output = { result: JSON.stringify(specArray) };
With any luck, you'll be able to use that payload in a body field for PDFMonkey and it'll process correctly!
But, that's how I share JSON between Zapier steps to keep it un-mangled.

How do I check if an object is within an array in Javascript?

I am getting stuck on a project where I am trying to pull out relevant data from a super massive Google Maps Timeline json, and running into the problem of the structure not being as orderly as I thought it was. Essentially, I am trying to pull out an address, time, date and mileage out of this json for every trip in my car. to use this data, I pasted it into a normal javascript file and named it so I can use it as an object. I then take this data and create a string that will format that info like a CSV file.
From going over the structure of the json by looking at only a few trips, I was able to determine the following general structure:
const google = {
timelineObjects: [
0: {activitySegment: {A NUMBER OF OBJECTS}},
1: {placeVisit : {A NUMBER OF OBJECTS}},
2: {activitySegment: {A NUMBER OF OBJECTS}},
3: {placeVisit : {A NUMBER OF OBJECTS}}
]
}
activitySegment has all the travelling info, like distance, travel times, etc. placeVisit has info about the destination. In my small sample, I was able to just loop through each using an if statement with i%2=0, and just change what I wanted to pull out from each, which worked well.
When I tried adding a larger sample, I was finding that Google occasionally did not create a activitySegment object and only had a placeVisit, which was throwing "Uncaught TypeError: Cannot read property 'distance' of undefined".
I am sure that the even/odd sorting will not work out any more. Is there a way to use a conditional to show if google[timelineObjects][i] is either a {activitySegment} or {placeVisit}? Or, is there a better way to figuring out what object is next in the array?

You can test to see if the Object at the particular array index has a given property using Object.prototype.hasOwnProperty():
const google = {
timelineObjects: [
{activitySegment: {}},
{placeVisit : {}},
{activitySegment: {}},
{placeVisit : {}}
]
};
console.log(google.timelineObjects[0].hasOwnProperty("activitySegment")); // true
console.log(google.timelineObjects[1].hasOwnProperty("activitySegment")); // false

If your objective to see what type of object you get. You can iterate over each object, see what the key of the object is and process the data depending on the key value. Something like this.
Object.entries(google).forEach(([key, value]) => {
if(key==='activitySegment') {
//process activeSegment here
}else {
//process placeVisit here
}
})

Zapier - Catch Hook - JSON Array - Loop over each item in array

I need to use a Zapier webhook to take some incoming JSON data, which contains an array of items, loop that array and do an action for each element.
Here's a sample of incoming JSON data:
{
"first_name": "Bryan",
"last_name": "Helmig",
"age": 27,
"data": [
{
"title": "Two Down, One to Go",
"type": "Left"
},
{
"title": "Talk the Talk",
"type": "Right"
},
{
"title": "Know the Ropes",
"type": "Top"
}
]
}
The size of the array will be dynamic.
The problem is that when I import this data in the hook, it gives me
data
title: Two Down, One to Go
type: Left
title: Talk the Talk
type: Right
title: Know the Ropes
type: Top
So, basically it says that data is just a big string of all this stuff together.
Can anyone help me figure out if it's possible to have a Zap loop over this and do something, e.g., insert data into a sheet, for ever item in the array? I'm aware of the "Code" actions, I've chosen JavaScript, which I could parse out the string, but that doesn't seem efficient. Plus, in reality, there will be a lot of data in the objects inside the JSON array.
EDIT: SOLVED! ANSWER BELOW

So, the first part is to Catch Raw Hook for a trigger. It's the normal "Webhooks", but you have to click to show the less common variations. With the Catch Raw Hook, your data will not be turned automatically turned into variables via the Zapier app, you'll have the raw JSON data.
Once you have the raw JSON, in my case, you'll have an action, and this will be the "Code" action. I'm using JavaScript. In my template, I'm grabbing the entire JSON string (your whole imported JSON is a string right now, not an object, so we can't use "." (dot) notation to access parts of it).
You'll need to JSON.parse() the string in the code. But first, let me explain that Zapier has a pre-defined variable called inputData that you'll use in your code. Then in the top of the "Edit Template" section of your "Code" Action, you'll see you can name the variable of that JSON string you imported.
Now the fun part! In the code, you'll type:
// of course, you can change the variables to what you want
// but 'inputData' is unique, can't change that
const myData = JSON.parse(inputData.rawJsonData);
So, my raw data is a string, it's not JSON yet, so this line of code makes it a JSON object. And now, as an object we can loop over it or .map or access 'this.that' or whatever you want.
The next important thing to mention about "Code" in Zapier, is that to get your stuff out, you return. So, in the next few lines, I'm going to return a .map function that returns each item in an array. And it's tough to grasp how Zapier treats this, but it actually runs the next "Action" you create (e.g. adding a row to a sheet) for each time you loop in that .map. So, let's take a look below:
return myData.data.map(item => {
return item;
});
If you remember, I had an array called "data" in my raw JSON I listed in the original question. It will loop over that array and since I'm returning, then it will perform an "Add Row to Sheet" (in my case) for each loop, thus, inserting all of my data as multiple rows in my spreadsheet.
So the finished code:
const myData = JSON.parse(inputData.rawJsonData);
return myData.data.map(item => {
return item;
});

Removing attributes from nested array in D3

I am trying to filter out some attributes from an array in D3. The array contains the values of a csv file.
This all worked well for a small csv file doing it like this:
d3.csv("foods.csv", function(data) {
data.forEach(function(v){ delete v.name });
data.forEach(function(v){ delete v.created_at });
});
This is what the first array looks like:
But when I try to do it for a bigger csv file I get an error saying : "devtools was disconnected from the page. once page is reloaded devtools will automatically reconnect".
This is what the 2nd array looks like.
Why is this not working for the 2nd array? Is the array too big or should I try to address the values recursively because I already tried doing it like this:
function deleteCitation(v) {
if(Object.prototype.toString.call(v) === '[object Array]' ) {
v.forEach(deleteCitation);
}
else {
delete v.citation;
}
}
d3.csv("compounds_foods.csv", function(data) {
data.forEach(deleteCitation);
print(data);
});

I never loaded an CSV with 740 thousand rows. However, I believe you have some alternatives:
Use a row conversion function, or an accessor function:
d3.csv("foods.csv", deleteCitation, function(data) {
//the rest of the code
And then declare the conversion function:
function deleteCitation(d){
delete d.name;
delete d.created_at;
return d;
}
I didn't benchmarked it, maybe the conversion function takes the same time that your forEach (they do pretty much the same thing), but I believe that it's worth to check if this is quicker than calling your deleteCitation function for each object inside the data array.
The second alternative is simpler: don't remove those two properties, just leave them there and don't use them!
When you load an CSV to your data array you don't have to use all the properties in each object for your visualisation. You can simply ignore them. It's possible that you waste more processing time manipulating that huge array than simply leaving those extra two objects there.
The third alternative is the logical one: as there is absolutely no way you're gonna use 740k objects in a dataviz, consider filtering/reducing/cropping this CSV before sending it to the client side.

Algorithm for data filter

Can you suggest me an algorithm for filtering out data.
I am using javascript and trying to write out a filter function which filters an array of data.I have an array of data and an array of filters, so in order to apply each filter on every data, I have written 2 for loops
foreach(data)
{
foreach(filter)
{
check data with filter
}
}
this is not the proper code, but in short that what my function does, the problem is this takes a huge amount of time, can someone suggest a better method.
I am using the Mootools library and the array of data is JSON array
Details of data and Filter
Data is JSON array of lets say user, so it will be
data = [{"name" : "first", "email" : "first#first", "age" : "20"}.
{"name" : "second", "email" : "second#second", "age" : "21"}
{"name" : "third", "email" : "third#third", "age" : "22"}]
Array of filters is basically self define class for different fields of data
alFilter[0] = filterName;
alFilter[1] = filterEmail;
alFilter[2] = filterAge;
So when I enter the first for loop, I get a single JSON opbject (first row) in the above case.
When I enter the second for loop (filters loop) I have a filter class which extracts the exact field on which the current filter would work and check the filter with the appropriate field of the data.
So in my example
foreach(data)
{
foreach(filter)
{
//loop one - filter name
// loop two - filter email
// loop three - filter age
}
}
when the second loop ends i set a flag denoting if the data has been filtered or not and depending on it the data is displayed.

You're going to have to give us some more detail about the exact structure of your data and filters to really be able to help you out. Are the filters being used to select a subset of data, or to modify the data? What are the filters doing?
That said, there are a few general suggestions:
Do less work. Is there some way you can limit the amount of data you're working on? Some pre-filter that can run quickly and cut it down before you do your main loop?
Break out of the inner loop as soon as possible. If one of the filters rejects a datum, then break out of the inner loop and move on to the next datum. If this is possible, then you should also try to make the most selective filters come first. (This is assuming that your filters are being used to reject items out of the list, rather than modify them)
Check for redundancy in the computation the filters perform. If each of them performs some complicated calculations that share some subroutines, then perhaps memoization or dynamic programming may be used to avoid redundant computation.
Really, it all boils down to the first point, do less work, at all three levels of your code. Can you do less work by limiting the items in the outer loop? Do less work by stopping after a particular filter and doing the most selective filters first? Do less work by not doing any redundant computation inside of each filter?

That's pretty much how you should do it. The trick is to optimize that "check data with filter"-part. You need to traverse all your data and check against all your filters - you'll not going to get any faster than that.
Avoid string comparisons, use data models as native as possible, try to reduce the data set on each pass with filter, etc.
Without further knowledge, it's hard to optimize this for you.

You should sort the application of your filters, so that two things are optimized: expensive checks should come last, and checks that eliminate a lot of data should come first. Then, you should make sure that checking is cut short as soon as an "out" result occurs.

If your filters are looking for specific values, a range, or start of a text then jOrder (http://github.com/danstocker/jorder) will fit your problem.
All you need to do is create a jOrder table like this:
var table = jOrder(data)
.index('name', ['name'], { grouped: true, ordered: true })
.index('email', ['email'])
.index('age', ['age'], { grouped: true, ordered: true, type: jOrder.number });
And then call table.where() to filter the table.
When you're looking for exact matches:
filtered = table.where([{name: 'first'}, {name: 'second'}]);
When you're looking for a certain range of one field:
filtered = table.where([{age: {lower: 20, upper: 21}}], {mode: jOrder.range});
Or, when you're looking for values starting with a given string:
filtered = table.where([{name: 'fir'}], {mode: jOrder.startof});
Filtering will be magnitudes faster this way than with nested loops.

Supposing that a filter removes the data if it doesn't match, I suggest, that you switch the two loops like so:
foreach(filter) {
foreach(data) {
check data with filter
}
}
By doing so, the second filter doesn't have to work all data, but only the data that passed the first filter, and so on. Of course the tips above (like doing expensive checks last) are still true and should additionally be considered.