We Keep Coding

Asynchronously and recursively crawl links Javascript

I am making a Blog using Notion as a content management system. There is an unofficial API provided by notion-api-js, with a function getPagesByIndexId(pageId) that returns a page's content, its subpages' contents, and, its parent's contents. So, an array of objects is returned, looking like:
[
{ moreStuff: ...,
Attributes: { slug: "home page slug", id: "home page id", moreStuff... },
},
{ moreStuff: ..., Attributes: { slug: "parent to homepage", id: "homepage's parent id", moreStuff: ... }
{ moreStuff: ..., Attributes: { slug: "sub page slug 0", id: "sub page id 0", moreStuff: ... } },
{ moreStuff: ..., Attributes: { slug: "sub page slug 1", id: "sub page id 1", moreStuff: ... } },
];
I want to build a tree that is created by recursively looping through the given id and the ids that getPagesByIndexId(given id) return to extract all slugs and ids. The function stops recursing when getPagesByIndexId(id) returns objects with ids already crawled through.
I use a crawledIdsList array to keep track of ids already crawled through, fetchPage is the same as getPagesByIndex, and I use flatmap to ignore empty []s passed by from the map function. Thanks in advance! To run this locally on node, the dependency required is npm i notion-api-js
The tree structure of the page I provided the ID with (I provided the id to the "Dev" page in homePageId) looks like:
My current code follows. It hits the "end" and returns successfully, but it is returning many pages a lot more than once.
const Notion = require("notion-api-js").default;
const token_v2 = "543f8f8529f361ab34596f5be9bc972b96ab8d8dc9e6e41546c05751b51a18a6c7d40b689d80794babae3a91aeb5dd5e47c34edb724cc356ceceacf3a8061158bfab92e68b7614516a0699295990"
const notion = new Notion({
token: token_v2,
});
const fetchPage = (id) => {
return notion.getPagesByIndexId(id);
};
const homePageId = "3be663ea-90ce-4c45-b04e-41161b992dda"
var crawledIdsList = [];
buildTree(tree={}, homePageId).then(tree => {console.log(tree)})
function buildTree(tree, id) {
return fetchPage(id).then((pages) => {
tree.subpages = [];
tree.slug = pages[0].Attributes.slug;
tree.id = id;
crawledIdsList.push(id);
return Promise.all(
pages.flatMap((page) => {
var currentCrawlId = page.Attributes.id;
if (crawledIdsList.indexOf(currentCrawlId) === -1) {
// executes code block if currentCrawlId is not used in fetchPage(id) yet
crawledIdsList.push(currentCrawlId);
return buildTree({}, currentCrawlId).then((futureData) => {
tree.subpages.push(futureData);
return tree;
});
} else {
if (crawledIdsList.indexOf(id) >= 0) {
return [];
}
return tree; // end case. futureData passed to earlier calls is tree, which looks like {subpages: [], slug: someSlug, id: someId}
}
})
)
});
}

Vue computed property changes the data object

I have basically this structure for my data (this.terms):
{
name: 'First Category',
posts: [
{
name: 'Jim James',
tags: [
'nice', 'friendly'
]
},
{
name: 'Bob Ross',
tags: [
'nice', 'talkative'
]
}
]
},
{
name: 'Second Category',
posts: [
{
name: 'Snake Pliskin',
tags: [
'mean', 'hungry'
]
},
{
name: 'Hugo Weaving',
tags: [
'mean', 'angry'
]
}
]
}
I then output computed results so people can filter this.terms by tags.
computed: {
filteredTerms: function() {
let self = this;
let terms = this.terms; // copy original data to new var
if(this.search.tags) {
return terms.filter((term) => {
let updated_term = {}; // copy term to new empty object: This doesn't actually help or fix the problem, but I left it here to show what I've tried.
updated_term = term;
let updated_posts = term.posts.filter((post) => {
if (post.tags.includes(self.search.tags)) {
return post;
}
});
if (updated_posts.length) {
updated_term.posts = updated_posts; // now this.terms is changed even though I'm filtering a copy of it
return updated_term;
}
});
} else {
return this.terms; // should return the original, unmanipulated data
}
}
},
filteredTerms() returns categories with only the matching posts inside it. So a search for "angry" returns just "Second Category" with just "Hugo Weaving" listed.
The problem is, running the computed function changes Second Category in this.terms instead of just in the copy of it (terms) in that function. It no longer contains Snake Pliskin. I've narrowed it down to updated_term.posts = updated_posts. That line seems to also change this.terms. The only thing that I can do is reset the entire data object and start over. This is less than ideal, because it would be loading stuff all the time. I need this.terms to load initially, and remain untouched so I can revert to it after someone clears their search criterea.
I've tried using lodash versions of filter and includes (though I didn't really expect that to make a difference). I've tried using a more complicated way with for loops and .push() instead of filters.
What am I missing? Thanks for taking the time to look at this.

Try to clone the object not to reference it, you should do something like :
let terms = [];
Object.assign(terms,this.terms);

let terms = this.terms;
This does not copy an array, it just holds a reference to this.terms. The reason is because JS objects and arrays are reference types. This is a helpful video: https://www.youtube.com/watch?v=9ooYYRLdg_g
Anyways, copy the array using this.terms.slice(). If it's an object, you can use {...this.terms}.

I updated my compute function with this:
let terms = [];
for (let i = 0; i < this.terms.length; i++) {
const term = this.copyObj(this.terms[i]);
terms.push(term);
}
and made a method (this.copyObj()) so I can use it elsewhere. It looks like this:
copyObj: function (src) {
return Object.assign({}, src);
}

Is there a find function in JavaScript that returns what your function returns and not the whole item when the value is truthy

I want to return only the matched item, I solved this problem creating my own high order function, I want to solve this in a completely functional way.
Is there any similar javascript function that does what my function is doing? See the examples below, I wrote some Jest based examples to facilitate what I am expecting.
The function will try to find the value until is different than undefined. If this kind of function does not exist what you guys think of trying implementing it on JavaScript, maybe making a tc39 proposal? Anyone had the same problem as me before?
I know how the Array.prototype.find works and why it does not work when chained to get deep elements.
There are some conditions that I would like to meet:
Return what my function returns and not the whole item if it's truthy.
For performance reasons, when the value is found there is no need to keep looping in the array, in the example below I used the condition anything different from undefined to exit the for loop.
Follow the standard of the others high order functions such as find, map, filter and reduce like this: fn(collection[i], index, collection).
const findItem = (collection, fn) => {
for (let i = 0; i < collection.length; i++) {
const item = fn(collection[i], i, collection)
if (item !== undefined) return item
}
return undefined
}
let groups = [
{ items: [{ id: 1 }, { id: 2 }] },
{ items: [{ id: 3 }, { id: 4 }] },
]
var result = findItem(groups, group =>
findItem(group.items, item => item.id === 4 ? item : undefined))
// works!
expect(result).toEqual(groups[1].items[1])
// Array.prototype.find
var result2 = groups.find(group =>
group.items.find(item => item.id === 4 ? item : undefined))
// returns groups[1], don't work! And I know why it does not work.
expect(result2).toEqual(groups[1].items[1])

Probably horrible, but you could make use of a backdoor in the reduce function that would allow you to exit early on a match
let groups = [
{ items: [{ id: 1 }, { id: 2 }] },
{ items: [{ id: 3 }, { id: 4 }] },
];
const item = groups.slice(0).reduce((val, g, i, arr) => {
for (const item of g.items) {
if (item.id === 4) {
val = item;
arr.splice(1); // exit
}
}
return val;
}, null);
item && console.log(item);
Note - use of slice is to ensure the original array is not mutated

How to delete something in a JavaScript object that can be nested arbitrarily deep?

I have a JavaScript object that manages all of my app's data. It looks something like this:
parent = {
data: 'abcdef',
children: {
'child1' : {
data: 'ghijkl',
children: {...}
},
'child2' : {
data: 'mnopqr',
children: {...}
}
}
}
In order to manage which is the currently active node, I keep an array called 'address' that is a list of which names to follow to traverse the overall structure and reach the desired point. For example [] as address gives the whole structure, ['child1'] as address gives just the 'child1' of the main object.
My problem is that given an address, I need to be able to remove that particular node, which can be arbitrarily deep within the overall structure, how could I go about this? For example remove(['child1']) should remove 'child1' from the example structure, with 'child2' remaining intact. All remove operations should return the entire structure with the single node removed.

This'll do it.
remove = function(address, tree) {
if (address.length === 1) {
delete tree.children[address[0]];
} else {
remove(address.slice(1), tree.children[address[0]]);
}
}
Pass in parent as the tree parameter to start with.

Please excuse if this is not what you are looking for, I 'think' I know what you are going for. But recursion might be your best bet. I just whipped this up, but perhaps someone else has a more succinct solution.
Lets assume, in the example below you want to delete "child1a",which is nested. We should be able to do that.
parent = {
data: 'abcdef',
children: {
'child1' : {
data: 'ghijkl',
children: {
'child1a' : {
data: 'data for child1a',
children: {}
}
}
},
'child2' : {
data: 'mnopqr',
children: {}
}
}
};
function deleteNode(node, child) {
if(node[child]) {
delete node[child];
return;
}
var children = Object.keys(node);
for(var i = 0; i < children.length; i++) {
return deleteNode(node[children[i]].children, child);
}
}
deleteNode(parent.children, 'child1a')

find and modify deeply nested object in javascript array

I have an array of objects that can be of any length and any depth. I need to be able to find an object by its id and then modify that object within the array. Is there an efficient way to do this with either lodash or pure js?
I thought I could create an array of indexes that led to the object but constructing the expression to access the object with these indexes seems overly complex / unnecessary
edit1; thanks for all yours replies I will try and be more specific. i am currently finding the location of the object I am trying to modify like so. parents is an array of ids for each parent the target object has. ancestors might be a better name for this array. costCenters is the array of objects that contains the object I want to modify. this function recurses and returns an array of indexes that lead to the object I want to modify
var findAncestorsIdxs = function(parents, costCenters, startingIdx, parentsIdxs) {
var idx = startingIdx ? startingIdx : 0;
var pidx = parentsIdxs ? parentsIdxs : [];
_.each(costCenters, function(cc, ccIdx) {
if(cc.id === parents[idx]) {
console.log(pidx);
idx = idx + 1;
pidx.push(ccIdx);
console.log(pidx);
pidx = findAncestorsIdx(parents, costCenters[ccIdx].children, idx, pidx);
}
});
return pidx;
};
Now with this array of indexes how do I target and modify the exact object I want? I have tried this where ancestors is the array of indexes, costCenters is the array with the object to be modified and parent is the new value to be assigned to the target object
var setParentThroughAncestors = function(ancestors, costCenters, parent) {
var ccs = costCenters;
var depth = ancestors.length;
var ancestor = costCenters[ancestors[0]];
for(i = 1; i < depth; i++) {
ancestor = ancestor.children[ancestors[i]];
}
ancestor = parent;
console.log(ccs);
return ccs;
};
this is obviously just returning the unmodified costCenters array so the only other way I can see to target that object is to construct the expression like myObjects[idx1].children[2].grandchildren[3].ggranchildren[4].something = newValue. is that the only way? if so what is the best way to do that?

You can use JSON.stringify for this. It provides a callback for each visited key/value pair (at any depth), with the ability to skip or replace.
The function below returns a function which searches for objects with the specified ID and invokes the specified transform callback on them:
function scan(id, transform) {
return function(obj) {
return JSON.parse(JSON.stringify(obj, function(key, value) {
if (typeof value === 'object' && value !== null && value.id === id) {
return transform(value);
} else {
return value;
}
}));
}
If as the problem is stated, you have an array of objects, and a parallel array of ids in each object whose containing objects are to be modified, and an array of transformation functions, then it's just a matter of wrapping the above as
for (i = 0; i < objects.length; i++) {
scan(ids[i], transforms[i])(objects[i]);
}
Due to restrictions on JSON.stringify, this approach will fail if there are circular references in the object, and omit functions, regexps, and symbol-keyed properties if you care.
See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_native_JSON#The_replacer_parameter for more info.

As Felix Kling said, you can iterate recursively over all objects.
// Overly-complex array
var myArray = {
keyOne: {},
keyTwo: {
myId: {a: '3'}
}
};
var searchId = 'myId', // Your search key
foundValue, // Populated with the searched object
found = false; // Internal flag for iterate()
// Recursive function searching through array
function iterate(haystack) {
if (typeof haystack !== 'object' || haystack === null) return; // type-safety
if (typeof haystack[searchId] !== 'undefined') {
found = true;
foundValue = haystack[searchId];
return;
} else {
for (var i in haystack) {
// avoid circular reference infinite loop & skip inherited properties
if (haystack===haystack[i] || !haystack.hasOwnProperty(i)) continue;
iterate(haystack[i]);
if (found === true) return;
}
}
}
// USAGE / RESULT
iterate(myArray);
console.log(foundValue); // {a: '3'}
foundValue.b = 4; // Updating foundValue also updates myArray
console.log(myArray.keyTwo.myId); // {a: '3', b: 4}
All JS object assignations are passed as reference in JS. See this for a complete tutorial on objects :)
Edit: Thanks #torazaburo for suggestions for a better code.

If each object has property with the same name that stores other nested objects, you can use: https://github.com/dominik791/obj-traverse
findAndModifyFirst() method should solve your problem. The first parameter is a root object, not array, so you should create it at first:
var rootObj = {
name: 'rootObject',
children: [
{
'name': 'child1',
children: [ ... ]
},
{
'name': 'child2',
children: [ ... ]
}
]
};
Then use findAndModifyFirst() method:
findAndModifyFirst(rootObj, 'children', { id: 1 }, replacementObject)
replacementObject is whatever object that should replace the object that has id equal to 1.
You can try it using demo app:
https://dominik791.github.io/obj-traverse-demo/

Here's an example that extensively uses lodash. It enables you to transform a deeply nested value based on its key or its value.
const _ = require("lodash")
const flattenKeys = (obj, path = []) => (!_.isObject(obj) ? { [path.join('.')]: obj } : _.reduce(obj, (cum, next, key) => _.merge(cum, flattenKeys(next, [...path, key])), {}));
const registrations = [{
key: "123",
responses:
{
category: 'first',
},
}]
function jsonTransform (json, conditionFn, modifyFn) {
// transform { responses: { category: 'first' } } to { 'responses.category': 'first' }
const flattenedKeys = Object.keys(flattenKeys(json));
// Easily iterate over the flat json
for(let i = 0; i < flattenedKeys.length; i++) {
const key = flattenedKeys[i];
const value = _.get(json, key)
// Did the condition match the one we passed?
if(conditionFn(key, value)) {
// Replace the value to the new one
_.set(json, key, modifyFn(key, value))
}
}
return json
}
// Let's transform all 'first' values to 'FIRST'
const modifiedCategory = jsonTransform(registrations, (key, value) => value === "first", (key, value) => value = value.toUpperCase())
console.log('modifiedCategory --', modifiedCategory)
// Outputs: modifiedCategory -- [ { key: '123', responses: { category: 'FIRST' } } ]

I needed to modify deeply nested objects too, and found no acceptable tool for that purpose. Then I've made this and pushed it to npm.
https://www.npmjs.com/package/find-and
This small [TypeScript-friendly] lib can help with modifying nested objects in a lodash manner. E.g.,
var findAnd = require("find-and");
const data = {
name: 'One',
description: 'Description',
children: [
{
id: 1,
name: 'Two',
},
{
id: 2,
name: 'Three',
},
],
};
findAnd.changeProps(data, { id: 2 }, { name: 'Foo' });
outputs
{
name: 'One',
description: 'Description',
children: [
{
id: 1,
name: 'Two',
},
{
id: 2,
name: 'Foo',
},
],
}
https://runkit.com/embed/bn2hpyfex60e
Hope this could help someone else.

I wrote this code recently to do exactly this, as my backend is rails and wants keys like:
first_name
and my front end is react, so keys are like:
firstName
And these keys are almost always deeply nested:
user: {
firstName: "Bob",
lastName: "Smith",
email: "bob#email.com"
}
Becomes:
user: {
first_name: "Bob",
last_name: "Smith",
email: "bob#email.com"
}
Here is the code
function snakeCase(camelCase) {
return camelCase.replace(/([A-Z])/g, "_$1").toLowerCase()
}
export function snakeCasedObj(obj) {
return Object.keys(obj).reduce(
(acc, key) => ({
...acc,
[snakeCase(key)]: typeof obj[key] === "object" ? snakeCasedObj(obj[key]) : obj[key],
}), {},
);
}
Feel free to change the transform to whatever makes sense for you!