We Keep Coding

How to make node.js program wait for a PNG to parse?

I am writing a project which requires me to load a lot of low resolution PNGs (~10k) to train my neural network. This is my code
NOTE: This project uses 'pngjs' module
for(var i = 0;i < 100;i++){
var input_dir = dirs[i];
console.log('file number: ', i);
fs.createReadStream(input_dir).pipe(new PNG({
filterType: 4,}))
.on("parsed", function(){
console.log(input_dir);
for(var y = 0; y < this.height; y++) {
for (var x = 0; x < this.width; x++) {
var idx = (this.width * y + x) << 2;
// invert color
this.data[idx] = 255 - this.data[idx];
this.data[idx + 1] = 255 - this.data[idx + 1];
this.data[idx + 2] = 255 - this.data[idx + 2];
var sum = this.data[idx]+this.data[idx + 1]+this.data[idx + 2];
sum /= 3;
arr.push(sum);
}
}
console.log(arr);
});
}
After execution, I a ton of file number: in the terminal, after which all the images show up.
Any way to make the program wait for the image to parse?

createReadStream will run asynchronously each time that you call it, you could wrap it in a function that returns a promise, then await each one of those promises with Promise.all.
function readStreamPromise(input_dir){
return new Promise((resolve) => {
fs.createReadStream(input_dir)
.pipe(new PNG({filterType: 4,}))
.on("parsed", () => {
// do parsing logic
resolve()
});
}
}
then do something like
await Promise.all(dirs.map((inputDir) => {
return readStreamPromise(inputDir)
})
of course this would need to be inside of an async function if you use the await

I suppose you would like to await its loading phase before proceeding. Luckily there is a keyword for that; namely await. It must be paired with async. Check it out.
Edit: The idea in general is that you would make an array of so-called Promises (related to async and await); one Promise for each image. You can then await the completion of all these asynchronous Promises, i.e. when all images are loaded. And after that you can do what you want while still having all images load asynchronously in beforehand. (Hint: Promise.all)

Increment loop but it increments at the end

I’m trying to use an increment loop but I want it to increment at the end of the loop. Sadly, whenever I simply put the i++ at the end of the loop it doesn’t behave like I’d expect or want it to. Anyone mind showing me the proper way of doing it?
The referred increment loop:
for (i = 1; i < 15; i++) {
// do somthing here
}
Here is the loop I’m working with:
for (i = 1; i < 15; i++) {
for (x = 1; x < 15; x++) {
var take = document.getElementById("row" + i + "sm" + x);
Tesseract.recognize(take)
.then(function(result) {
console.log(result.text);
// rows[i][x] = result.text;
})
}
}
What I’d like it to do:
for (i = 1; i < 15) {
for (x = 1; x < 15) {
var take = document.getElementById("row" + i + "sm" + x);
Tesseract.recognize(take)
.then(function(result) {
console.log(result.text);
//rows[i][x] = result.text;
x += 1;
})
i += 1;
}
}
I am using the for loop because I need to iterate over something one by one. How do I properly increment i at the end of the loop?
Here is a video explaining my problem with context and explanation why it is not an ASYNC problem. Sorry if it is hard to follow, ill update it with audio soon so I can explain it propperly.
https://drive.google.com/file/d/1n1ZwNJif5Lb5zfLb2GPpBemObwpOqNf7/view

The problem is that the second one doesnt wait until first one is complete.
You can try with recursion inside then. There maybe some mistake with i,x but you get the point.
You execute first with i=1 and x=1, after the operation is done (then) you call the next until all elements are executed.
function execItem(i, x) {
var take = document.getElementById("row" + i + "sm" + x);
Tesseract.recognize(take)
.then(function(result){
rows[i][x] = result.text;
if (i < 15 && x < 15) {
if (i > 15) {
x += 1
i = 1
} else {
i += 1
}
execItem(i, x)
}
})
}
execItem(1, 1)

As a comment suggests this actually seems likely to be a problem with an asynchronous call (Tesseract...then) inside a loop. By the time the function inside then is called, your values of x and i have already moved on, so you don't get the result you expect.
One way around this would be to use a 'closure' - making a function that creates another function based on the value of i and x.
function getDisplayFunc(row, col) {
function displayRecognisedText(result) {
console.log(row, col, result.text);
//rows[row][col] = result.text;
}
return displayRecognisedText;
}
for (i = 1; i < 15; i++) {
for (x = 1; x < 15; x++) {
var take = document.getElementById("row" + i + "sm" + x);
Tesseract.recognize(take).then(getDisplayFunc(i, x));
}
}

I guess #Mike spot the error on: your code is asynchronous. What does it mean?
So, let's suppose you have this loop:
for (i = 0; i < 10; i++) {
console.log(i);
}
It will print this, right?
0
1
2
3
4
5
6
7
8
9
However, you do not print your value inside the loop directly, but as a follow-up operation to a promise. This makes this code asynchronous. It means that it does not have to execute at the exact moment you call it. I do not have Tesseract here so I will make my loop asynchronous using another very old trick, setTimeout():
for (i = 0; i < 10; i++) {
setTimeout(function() {
console.log(i);
}, 0);
}
If I run it, I get this:
10
10
10
10
10
10
10
10
10
10
What happens is, when I pass the operation we want to do (in this case, printing the i value) to an asynchronous function (recognize().then() in your case, setTimeout() in my case) through a callback (function() {console.log(i);} in my example) the asynchronous function "schedules" the operation to execute as soon as possible, but this "soon" is not faster than the loop. So, the loop finishes executing but our callback is not called, not even once! Since you are not declaring i with let, it is a global variable, so there exists only one i. And since the loop finished, the value of the i variable is 10 already.
It used to be a hard thing to solve, but with ES6 it is quite straightforward: declare i with let!
for (let i = 0; i < 10; i++) {
setTimeout(function() {
console.log(i);
}, 0);
}
The let-ed variable has a new binding at each iteration of the loop, so in practice you have 10 variables called i. The closure of your function will have access only to the one with the right value!

Maybe you should try to use while loop.
Like this:
while i < 15:
//do something
i += 1
For two variables: x, i with embeding:
while x < 15:
//do something
while i < 15:
//do something2
i += 1
x += 1
Hope I understand the problem correctly.

JS crashes sometimes with Timer scramble

My Javascript timer is for people with a rubiks cube with generates a scramble (nevermind all this, but just to tell you I'm generating after each solve a new scramble will be generated) and my scrambles do actually have a while (true) statement. So that does crash my script, but it 95/100 times stops just before the script crashes but I don't wanna have any times.
Let me explain a bit more detailed about the problem.
Problem: javascript crashes because my script takes too long to generate a scramble.
Below you have 3 functions I use.
This function generates a scramble with the Fisher-Yates shuffle.
Timer.prototype.generateScramble = function(array) {
for (var i = array.length - 1; i > 0; i--) {
var j = Math.floor(Math.random() * (i + 1));
var temp = array[i];
array[i] = array[j];
array[j] = temp;
}
return array;
};
This function validates the input e.g. I receive an array as the following:
Here I only have to check the first character. That's why I use the seconds [ ] notation. I don't want people get an F with an F2 e.g.
var scr = ["F","R","U","B","L","D","F2","R2","U2","B2","L2","D2","F'","R'","U'","B'","L'","D'"]
Timer.prototype.validateScramble2 = function(array) {
var last = array.length-1;
for (var i = 0; i < array.length-1; i++) {
if (array[i][0] == array[i+1][0]) {
return false;
}
}
for (var i = 0; i < array.length-2; i++) {
if (array[i][0] == array[i+2][0]) {
return false;
}
}
if (array[0][0] == [last][0]) {
return false;
}
return true;
};
The above functions are just waiting to be called. Well in the function below I use them.
Timer.prototype.updateScramble2 = function(scrambleArr, len, type) {
var self = this;
var scramble = '', j, updatedArr = [];
while (updatedArr.length < len) {
j = (Math.floor(Math.random() * scrambleArr.length));
updatedArr.push(scrambleArr[j]);
}
while (!self.validateScramble2(updatedArr)) {
updatedArr = self.generateScramble(updatedArr);
}
for (var i = 0; i < updatedArr.length; i++) {
scramble += updatedArr[i] + ' ';
}
scrambleDiv.innerHTML = scramble;
};
I assume you guys understand it but let me explain it briefly.
The first while-loop adds a random value from the given array(scrambleArr) into a new array called updatedArr.
The next while-loop calls the validateScramble2() function if there isn't in an array F next to an F2.
The for-loop adds them into a new variable added with a whitespace and then later we show the scramble in the div: scrambleDiv.innerHTML = scramble;
What do I need know after all this information?
Well I wanna know why my updateScramble2() functions lets my browser crash every time and what I do wrong and how I should do it.

I'm not entirely sure I understand the question, but from the way your code looks, I think you have an infinite loop going on. It appears as if validateScramble2 always returns false which causes your second loop in updateScramble2 to perpetually run.
I suggest you insert a breakpoint in your code and inspect the values. You could also insert debugger; statements in your code, works the same way. Open dev tools prior to doing these.
A suggestion is instead of using loops, use a timer. This breaks up your loop into asynchronous iterations rather than synchronous. This allows the browser breathing space for other operations. Here's an example of a forEachAsync:
function forEachAsync(array, callback){
var i = 0;
var timer = setInterval(function(){
callback.call(null, array[i]);
if(++i >= array.length) clearInterval(timer);
}, 0);
}
forEachAsync([1,2,4], function(item){
console.log(item);
});
You can take this further and use Promises instead of callbacks.

Node.js: how to loop through handling incoming data?

I have received this comment:
On the server side of Node.js when handling incoming data if you want to use a for loop you have to create i inside an anonymous function or you will pull your hair out wondering how the hell your variable i is greater than what you limit it to be inside your loop.
Here is the bugfix that was recommended:
var i = 0,
len = that.users.length;
(function(i) {
while(i < len) {
console.log(' - - - - - debug - - - - -');
console.log('i = ' + i );
i++;
}
})(i);
Can someone explain to me why using an anonymous function is necessary?

The problem only shows up when you have asynchronous code inside the loop. For example (I changed to a for loop for simplicity):
var i = 0,
len = that.users.length;
for(i = 0;i < len;i++) {
setTimeout(function() {
console.log('i = ' + i );
}, 500);
}
You will find that running this code causes the value of len to be printed len times, instead of a count up to len. This is because the for loop finishes before any of the print statements run, so the loop has exited because i == len.
The fix for this is to lock i in to each value with an Immediately-Invoked Function Expression (IIFE):
var i = 0,
len = that.users.length;
for(i = 0;i < len;i++) {
(function(i) {
setTimeout(function() {
console.log('i = ' + i );
}, 500);
})(i);
}
This isn't exactly the bugfix you presented, but it is the closest I can think of that makes much sense. Given more context, I could be more sure about what problem it is supposed to solve.

Setting JavaScript Timeout limit in Google Chrome

Is it possible to increase the time out limit for JavaScript?
If I have a script that executes for more than 20/30 seconds chrome will pop-up with the unresponsable page dialog.
Making a more efficient script won't help me because the script sometimes need to iterate through a function for a million or billion times

To split the function on steps/chunks and run those inside setInterval(function(){}).
This way page will be responsive, you will be able to notify user about progress of execution and you will get your job done.
UPDATE: Here is simple function that takes
worker function executing each iteration,
chunksz - number of iteration running in single chunk
maxit - total number of iterations.
function task(worker, chunksz, maxit)
{
var idx = 0;
var xint = null;
function exec_chunk()
{
for(var n = 0; n < chunksz; ++n)
{
if(idx >= maxit) { return; }
worker(idx++);
}
setTimeout(exec_chunk,1);
}
exec_chunk();
}
Here is an example : http://jsfiddle.net/Ed9wL/
As you see you get all iterations in order.
UPDATE2:
Say you have a loop:
for(var i=0; i<100000; ++i) { ... do something ... }
then you need to wrap body of the loop into a function and call the task above with it like this:
task(function(i){ ... do something ... },100, 100000);
or like this:
function loopBody(i){ ... do something ... }
task(loopBody,100, 100000);

When you have lots of processing to do client side, you need to split out your work into separate threads. The browser only has a single thread for handling user input (events) and for processing JS. If you're processing too much JS, without yielding, the UI becomes unresponsive and the browser is not happy.
How can you allow your script to yield? The new way is to use web workers http://www.whatwg.org/specs/web-workers/current-work/ . This works by creating a separate thread to run your JS, thread thread does not access to the DOM and can be run concurrently.
However, this newer technology doesn't exist in all browsers. For older browsers, you can split up your work by having the script call itself through timeouts. Whenever a timeout occurs, the script is yielding to the browser to run its events, once the browser is done, your next timeout will be triggered.
Example http://jsfiddle.net/mendesjuan/PucXf/
var list = [];
for (var i = 0; i < 500000; i++) {
list.push(Math.random());
}
function sumOfSquares(list) {
var total = 0;
for (var i = 0; i < list.length; i++) {
total += list[i] * list[i];
// DOM manipulation to make it take longer
var node = document.createElement("div");
node.innerHTML = "Sync temp value = " + total;
document.body.appendChild(node);
}
return total;
}
function sumOfSquaresAsync(arr, callback) {
var chunkSize = 1000; // Do 1000 at a time
var arrLen = arr.length;
var index = 0;
var total = 0;
nextStep();
function nextStep() {
var step = 0;
while (step < chunkSize && index < arrLen) {
total += arr[index] * arr[index];
// DOM manipulation to make it take longer
var node = document.createElement("div");
node.innerHTML = "Async temp value = " + total;
document.body.appendChild(node);
index++;
step++;
}
if (index < arrLen) {
setTimeout(nextStep, 10);
} else {
callback(total);
}
}
}
sumOfSquaresAsync(list, function(total) {console.log("Async Result: " + total)});
//console.log("Sync result" + sumOfSquares(list));
The example on jsfiddle has the synchronous call commented out, you can put it back in to see the browser come to a crawl. Notice that the asynchronous call does take a long time to complete, but it doesn't cause the long running script message and it lets you interact with the page while calculating (select text, button hover effects). You can see it printing partial results on the pane to the bottom right.
UPDATE http://jsfiddle.net/mendesjuan/PucXf/8/
Let's try to use c-smile's task function to implement sum of squares. I think he's missing a parameter, a function to call back when the task is finished. Using task allows us to create multiple chunked functions without duplicating the work of calling setTimeout and iteration.
/**
* #param {function} worker. It is passed two values, the current array index,
* and the item at that index
* #param {array} list Items to be traversed
* #param {callback} The function to call when iteration is finished;
* #param {number} maxit The number of iterations of the loop to run
* before yielding, defaults to 1000
*/
function task(worker, list, callback, maxit)
{
maxit = maxit || 1000;
var idx = 0;
exec_chunk();
function exec_chunk()
{
for(var n = 0; n < maxit; ++n)
{
if(idx >= list.length) {
callback();
return;
}
worker(idx, list[idx]);
idx++;
}
setTimeout(exec_chunk,1);
}
}
function sumOfSquaresAsync(list, callback)
{
var total = 0;
// The function that does the adding and squaring
function squareAndAdd(index, item) {
total += item * item;
// DOM manipulation to make it take longer and to see progress
var node = document.createElement("div");
node.innerHTML = "Async temp value = " + total;
document.body.appendChild(node);
}
// Let the caller know what the result is when iteration is finished
function onFinish() {
callback(total);
}
task(squareAndAdd, list, onFinish);
}
var list = [];
for (var i = 0; i < 100000; i++) {
list.push(Math.random());
}
sumOfSquaresAsync(list, function(total) {
console.log("Sum of Squares is " + total);
})

If your goal is to suppress "Kill-Wait" message as quick temporary fix for your slow JavaScript then the solution is to open Tools/Developer Tools in Google Chrome and keep it open and minimized somewhere on your desktop while browsing .