We Keep Coding

"Cleanup called..." Message log Tensorflow.js (Node)

I am using around 60000 + 10000 Images for creating the training and validation dataset using a generator
( Images used are the MNIST Images of Handwritten Digits in the PNG Format ).
But when I fit the model using them, "Cleanup called" message logs are constantly getting printed.
function* dataGenerator(type) {
const dataRoot = `MNIST/${type}-Data`;
const labels = fs.readdirSync(dataRoot);
for (let _idx = 0; _idx < labels.length; _idx++) {
const label = labels[_idx];
const files = fs.readdirSync(`${dataRoot}/${label}`);
for (let idx = 0; idx < files.length; idx++) {
const img = fs.readFileSync(`${dataRoot}/${label}/${files[idx]}`);
const imageTensor = tf.node.decodePng(img, 1);
const oneHotArr = new Array(labels.length).fill(0);
oneHotArr[label] = 1;
const oneHotTensor = tf.tensor1d(oneHotArr);
yield { xs: imageTensor, ys: oneHotTensor };
};
};
}
const trainingDataset = tf.data.generator(() => dataGenerator("Training"))
.shuffle(100)
.batch(100);
const validationDataset = tf.data.generator(() => dataGenerator("Validation"))
.shuffle(100)
.batch(100);
// Fitting the model
await model.fitDataset(trainingDataset, {
epochs: 5,
validationData: validationDataset
});
What am I doing wrong ?

nothing. message is info-only and comes from underlying tensorflow c library (where it was introduced at the wrong message level) used by tfjs-node.
planned upgrade of shared library from 2.7.3 to 2.9.1 will take care of that - it should be part of tfjs 3.21 once its released.

Can you do a recursive, memoized Fibonacci function using a js generator?

The most elegant Fibonacci function I have found isn't even recursive:
async function* fib(x) {
let x1 = 0;
let x2 = 1;
let i = 0;
while (i < x) {
[x1, x2] = [x2, x1 + x2];
i += 1;
}
yield x1;
}
Generators are great.
That being said - they also lend themselves to recursive tasks, since you can execute them 'lazily'.
And traditionally, Fibonacci functions are textbook examples for recursion or memoization.
What I came up with so far doesn't work.
So I was wondering:
How would you do a recursive, memoized Fibonacci generator function in JavaScript?

Some preliminary comments:
async has nothing to do with yield. So remove async here.
For a generator it should be unnecessary to pass an argument that indicates the end of the series (x). This should be a limit that the caller imposes and the generator should not have to worry about. From the viewpoint of the generator, it should just keep going without limit for as long as the caller pulls values from it.
A recursive version would anyhow have to yield the first Fibonacci value before the others, so it would then make sense to apply a pattern that looks like tail recursion. This requires no memoization (except for passing on two consecutive Fibonacci values):
function* genFib (a=0, b=1) {
yield a;
yield *genFib(b, a+b);
}
let limit = 50;
for (let n of genFib()) {
if (n > limit) break;
console.log(n);
}

Trincot's answer covers well the recursive generator aspect of you question and I agree with their assessment re: memoization in that case. Since it looks like you're looking to return a specific Fibonacci number you can simply memoize the function you posted in your question (minus async as it's not needed here). Keep in mind that since it only ever yields once, it could just as easily be a standard function since the cost will be paid regardless as soon as you access the next() (and only) element.
function memoFib() {
const c = new Map();
let m = 0;
let x1 = 0;
let x2 = 1;
return function* (x) {
while (m <= x) {
c.set(m, x1);
[x1, x2] = [x2, x1 + x2];
m++;
}
yield c.get(x)
}
}
const fib = memoFib();
console.log(fib(10).next().value); // updates cache
console.log(fib(2).next().value);
console.log(fib(5).next().value);
console.log(fib(14).next().value); // updates cache
console.log(fib(11).next().value);
console.log(fib(1).next().value);
It's a small next step to expand the above with Trincot's recursive example into a memoized function which returns specific ranges of the series as an iterator. The below snippet uses an array as a cache instead of a Map and accepts start and end indexes, if end is omitted it will return a sequence of 1. This makes better use of the generator and since the cache was already being populated sequentially an array is a better fit than a Map anyway.
function memoFib() {
const c = [];
let m = 0;
let x1 = 0;
let x2 = 1;
return function* fib(start, end) {
end = end ?? start + 1;
while (m <= start) {
c[m] = x1;
[x1, x2] = [x2, x1 + x2];
m++;
}
if (start < end) {
yield c[start]
yield* fib(start + 1, end)
}
}
}
const fib = memoFib();
console.log('Sequence:')
for (const n of fib(0, 5)) {
console.log(n)
}
console.log('\nSingle values:')
console.log(fib(2).next().value);
console.log(fib(11).next().value); // updates cache
console.log(fib(10).next().value);
console.log(fib(14).next().value); // updates cache

Three.js BufferGeometry Vertices Not Updating

I am looking to connect plane buffer geometry grid tiles which have real elevation data from IndexedDB. My issue is the data resolution on the STRM elevation is not perfect so the edges between the tiles are not the same. I need to essentially average out all the grid edges between the touching vertices to create a seamless terrain.
When I copy paste the code into the console in the scene it works. However just in the code it doesn't. The sceneRef that is passed is valid and the rest of the codebase using the sceneRef correctly.
The tiles are a 3 x 3 with the current grid tile being the center at 1,1 from range 0,0 - 2,2.
function connectTiles(currGridKey, sceneRef){
console.log("connectTiles");
console.log("currGridKey");
// Current Tile Connection
for (var lat = 0; lat < currGridKey[0]+2; lat++) {
for (var long = 0; long < currGridKey[1]+2; long++) {
const currentTile = sceneRef.getObjectByName(`${lat}-${long}`);
// Current Grid Tile Per Loop
if (currentTile) {
const currentTileVerts = currentTile.geometry.attributes.position.array,
latPlusTile = sceneRef.getObjectByName(`${lat}-${long+1}`),
longPlusTile = sceneRef.getObjectByName(`${lat+1}-${long}`);
// Connect Latitudinally
if (latPlusTile) {
const latPlusTileVerts = latPlusTile.geometry.attributes.position.array;
for (var z = 0; z < currentTileVerts.length; z+=27) {
const newVertHeight = (currentTileVerts[z] + latPlusTileVerts[z]) / 2;
latPlusTileVerts[z] = newVertHeight;
currentTileVerts[z] = newVertHeight;
}
latPlusTile.geometry.attributes.position.needsUpdate = true;
currentTile.geometry.attributes.position.needsUpdate = true;
}
// Connection Longitudinally
if (longPlusTile) {
const longPlusTileVerts = longPlusTile.geometry.attributes.position.array;
for (var x = 0; x < currentTileVerts.length; x+=3) {
const newVertHeight = (currentTileVerts[x] + longPlusTileVerts[x]) / 2;
longPlusTileVerts[x] = newVertHeight;
currentTileVerts[x] = newVertHeight;
}
longPlusTile.geometry.attributes.position.needsUpdate = true;
currentTile.geometry.attributes.position.needsUpdate = true;
}
}
}
}

If all values inside the array are in fact being updated, maybe they're just not getting uploaded to the GPU. Instead of changing the value inside geometry.attributes.position directly, try using the .setAttribute() method. The docs state that using .setAttribute() and .getAttribute() is preferrable than accessing it directly because it has its own internal storage mechanism.
const latPlusTileVerts = latPlusTile.geometry.getAttribute("position").array;
// ... Loops
latPlusTile.geometry.getAttribute("position").needsUpdate = true;
// Or an alternative is to generate a new attribute...
// in case updating the old one fails
const posAttrib = new THREE.BufferAttribute(latPlusTileVerts, 3);
latPlusTile.geometry.setAttribute("position", posAttrib);

How do I optimize this synchronous "fuzzy search" function and turn it into an async function?

Problem
I'm trying to implement some sort of "fuzzy search" in my Node.js based project.
Fuzzy search is a search that returns results even if the string didn't match exactly.
I found this code in another stackoverflow thread. Code is below.
It's quite good, but the problem is - it's synchronous It slows down the whole program when it searches through a large array.
Question
ES6 methods are welcomed. Needs to work only in the latest Chrome, so any JS methods will work.
Are there any new JS methods that would optimize this function?
Am I doing something wrong there that makes it even slower?
Can I turn this function into an async function that returns a promise? Will it stop freezing the app during the search then?
Are there any better "fuzzy search" implementations you know of? (I found a module called fuzzysort, can't say if it's that much better though, it won't return "folder test" if you type "test folder" (wrong order) so it's not that good)
Code
Calling search function
searchArray is an array of paths it searches through, e.g.: ["C:\\test", "C:\\file.txt"...] (0.5 - 5 million paths)
searchQuery is a string without spaces, e.g.: filetxt
search () {
const fuzzySearch = this.fuzzySearch(this.searchQuery.toLowerCase(), this.searchArray)
let result = fuzzySearch.filter(element => element.relevance >= 0.3)
// sort by relevance
var sortedResults = result.sort((a, b) => parseFloat(b.relevance) - parseFloat(a.relevance)).map(item => item.name);
this.searchResults = sortedResults
},
The fuzzy search function
fuzzySearch (searchQuery, searchArray) {
const get_bigrams = function(string) {
const s = string.toLowerCase();
const v = new Array(s.length - 1);
for (let i = 0, end = v.length; i <= end; i++) {
v[i] = s.slice(i, i + 2);
}
return v;
};
const string_similarity = function(str1, str2) {
if ((str1.length > 0) && (str2.length > 0)) {
const pairs1 = get_bigrams(str1);
const pairs2 = get_bigrams(str2);
const union = pairs1.length + pairs2.length;
let hit_count = 0;
for (let x of Array.from(pairs1)) {
for (let y of Array.from(pairs2)) {
if (x === y) {
hit_count++;
}
}
}
if (hit_count > 0) {
return ((2.0 * hit_count) / union);
}
}
return 0.0;
};
let results = [];
for (let name of searchArray) {
// I added .match to use only the base filename (name+ext) not the whole path, and removed all characters
let filteredPath = name.match(/[^\\\/]+$/)[0].replace(/[^A-Za-z0-9.]+/g, '')
const relevance = string_similarity(searchQuery, filteredPath);
const obj = {name, relevance};
results.push(obj);
}
return results
},

Refactor big switch statement

I have the following switch statement:
switch (type) {
case 1: // 1 BYTE 8-bit unsigned integer
pointer = count > 4 ? offset : pointer;
for (let i = 0; i < count; i++) {
value += dataView.getUint8(pointer + i);
}
tag.value = parseInt(value, 10);
return tag;
case 3: // 3 SHORT 16-bit unsigned integer
pointer = count > 2 ? offset : pointer;
for (let i = 0; i < count; i++) {
value += dataView.getUint16(pointer + 2 * i, littleEnd);
}
tag.value = parseInt(value, 10);
return tag;
case 4: // 4 LONG 32-bit unsigned integer
pointer = count > 1 ? offset : pointer;
for (let i = 0; i < count; i++) {
value += dataView.getUint32(pointer + 4 * i, littleEnd);
}
tag.value = parseInt(value, 10);
return tag;
case 5:
...
and so on.
The pattern is every time the same with some small variations. How can I refactor this? I want to refactor the pattern inside the case and I'm also trying to remove the whole switch block. Is that possible?

(This probably belongs on the Code Review Stack Exchange.)
Without a bit of larger context it's difficult to provide a reasonable refactoring, or even determine if such a refactoring would be worth the effort and additional maintenance.
The nutshell is that you have a number of type that need to be handled. Rather than a switch, you could implement a command pattern where each type is either a small class, or a simple object. (Using a class makes it marginally easier to pass in an "execution context" that contains the variables not shown in the snippet.)
For the sake of brevity, here's a (very) rough outline.
You'd have a base type handler. This wraps up dataView looping and tag value setting. Since I don't know the context, I'm pretending there's a context you pass in. I include all variables that weren't shown in your snippet.
(I didn't include value, which it looks like you should, but I didn't know the intent.)
class BaseTypeHandler {
constructor(ctx) {
this.ctx = ctx
}
getPointer = () => throw new Error('Missing getPointer implementation')
getViewData = () => throw new Error('Missing getViewData implementation')
getValueFromDataView = () => {
let value = 0
for (let i = 0; i < this.ctx.count; i++) {
value += this.getViewData(i, pointer)
}
return value
}
getTag = () => {
const pointer = this.getPointer()
, value = this.getValueFromDataView()
this.ctx.tag.value = parseInt(value, 10)
return this.ctx.tag
}
}
Each subclass implements the required unique functionality, here how to get the pointer, and how to get data from the dataView.
class Type1Handler extends BaseTypeHandler {
getPointer = () =>
this.ctx.count > 4 ? this.ctx.offset : this.ctx.pointer
getViewData = (i, pointer) =>
this.ctx.dataView.getUint8(pointer + i)
}
class Type3Handler extends BaseTypeHandler {
getPointer = () =>
this.ctx.count > 2 ? this.ctx.offset : this.ctx.pointer
getViewData = (i, pointer) =>
this.ctx.dataView.getUint16(pointer + 2 * i, littleEnd);
}
Then you wrap those up in an object of type handlers:
const typeHandlers = {
1: Type1Handler,
3: Type3Handler,
4: Type4Handler
}
const handler = new typeHandlers(type)
, tag = handler.getTag()
TL;DR
Unless you have a tremendous number of these, and you cannot use math to figure out the getPointer and getViewData implementations, you might want to stick with the switch.
Simple objects or immediate functions may be a significantly smaller implementation, although not necessarily easier to reason about. They also have the advantage of being able to close over variables you already have locally.