While waiting for the back end devs to implement a "cancel all" feature, which cancels all tasks tracked by the back end, I am attempting to makeshift it by cancelling each individual task. The cancel REST service accepts an ID in the form of a data object {transferID: someID}.
I use a FOR loop to iterate over an array of IDs that I have stored elsewhere. Anticipating that people MAY end up with dozens or hundreds of tasks, I wanted to implement a small delay that will theoretically not overflow the number of HTTP requests the browser can handle and will also reduce a blast of load on the back end CPU. Here is some code with comments for the purpose of this discussion:
ta.api.cancel = function (taskArray, successCallback, errorCallback) {
// taskArray is ["task1","task2"]
// this is just the latest attempt. I had an attempt where I didn't bother
// with this and the results were the same. I THOUGHT there was a "back image"
// type issue so I tried to instantiate $.ajax into two different variables.
// It is not a back image issue, though, but one to do with setTimeout.
ta.xhrObjs = ta.xhrObjs || {};
for (var i = 0; i < taskArray.length; i++) {
console.log(taskArray); // confirm that both task1 and task2 are there.
var theID = taskArray[i];
var id = {transferID: theID}; // convert to the format understood by REST
console.log(id); // I see "task1" and then "task2" consecutively... odd,
// because I expect to see the "inside the setTimeout" logging line next
setTimeout(function () {
console.log('inside the setTimeout, my id is: ')
console.log(id.transferID);
// "inside the setTimeout, my id is: task2" twice consecutively! Y NO task1?
ta.xhrObjs[theID] = doCancel(id);
}, 20 * i);
}
function doCancel(id) {
// a $.Ajax call for "task2" twice, instead of "task1" then "task2" 20ms
// later. No point debugging the Ajax (though for the record, cache is
// false!) because the problem is already seen in the 'setTimeout' and
// fixed by not setting a timeout.
}
}
Thing is: I know setTimeout makes the containing function execute asynchronously. If I take out the timeout, and just call doCancel in the iterator, it will call it on task1 and then task2. But although it makes the call async, I don't understand why it just does task2 twice. Can't wrap my head around it.
I am looking for a way to get the iterator to make the Ajax calls with a 20ms delay. But I need it to call on both! Anybody see a glaring error that I can fix, or know of a technique?
You must wrap your function setTimeout and pass the id variable into it, like this:
(function(myId, i) {
setTimeout(function () {
console.log('inside the setTimeout, my id is: ', myId);
}, 20 * i);
}(theId, i));
This pattern does not create a unique variable1 for each instance of the loop as one might expect.
function () {
for (var i = 0; i < length; i++) {
var variable1;
}
}
In javascript variables are "hoisted". To quote Mozilla:
"Because variable declarations (and declarations in general) are
processed before any code is executed, declaring a variable anywhere
in the code is equivalent to declaring it at the top."
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/var
So it should be re-written as:
function () {
var variable1;
for (var i = 0; i < length; i++) {
}
}
What this means is that after the loop has finished, any asynchronous callbacks that reference this variable will see the last value of the loop.
Related
I want to run a simple function asynchronously in Electron, so it doesn't block my render thread. So, something (very roughly) like this (within render.js):
var max = 42; // Somehow needs to be passed to the function
function foo() {
for (var i = 0; i < max; i++) {
// Do something...
// ... and tell the render thread about it.
}
}
foo(); // Should run asynchronously
There are two requirements:
I must pass parameters to the function (here: max). These could not only be integers, but also objects. The function must not run before it recieves these parameters.
While running, there must be a communication channel to the render thread. For example, to periodically report progress from within the for loop all the way to the UI, or to abort the function when an event in the render thread fires.
Here is a more specific minimal working (or rather, not working) example. The purpose is to send serial commands to a physical apparatus, on which a probe should be moved to all locations in a specified grid. So I need two loops (one for x, one for y). The loop body will contain functions that block until the motors have moved, than a mesurement from that location must be communicated back to the UI. Also, before the loops start running, the specifications about the grid must be known (hence my requirement about passing parameters.)
var Parameters = {x_length: 50, y_length: 50};
//These functions interact with a serial device and block until it is done with its task
function moveTo(x, y) {/*...*/};
function measure() {/*...*/};
//This function should eventually be executed asynchronously
function scan() {
for (var x = 0; x < Parameters.x_length; x++) {
for (var y = 0; y < Parameters.y_length; y++) {
moveTo(x, y);
var result = measure();
// Here, we need to tell the render thread about results. I used
// postMessage (WebWorker syntax) as a placeholder.
postMessage({
position: {x: x, y: y},
data: result
});
}
}
}
// This is what will be actually called, e. g. when the user clicks a
// button.
function scan_async() {
//Again, I used WebWorker (or rather, TinyWorker) syntax as a placeholder.
var scan_worker = new Worker(scan);
scan_worker.onmessage = function (msg) {
console.log(msg.data);
};
}
After hours of very frustrating googeling, I found a lot of approaches, but non of them seems like the "definitive" way to do it, or doesn't meet the above points, or seems way to complicated for what I want to achieve.
What I found so far:
Actually using WebWorkers (or rather something TinyWorkers, https://github.com/avoidwork/tiny-worker), like in the code above. Here, there seems to be no elegant way to pass the starting parameters before the worker starts running.
Create a new, hidden BrowserWindow, as mentioned here. However, I can't find anything about this method elsewhere.
So, is there a "correct", more straight-forward way to achieve my goal?
Actually using WebWorkers (or rather something TinyWorkers)
Yes. This is how you move code to a different thread.
Here, there seems to be no elegant way to pass the starting parameters before the worker starts running.
Create the worker. Don't have it do anything except listen for postMessage events.
Send a message to it, with postMessage with the starting parameters
An event listener in the worker gets the message and starts running the job
When it is done, postMessage from the work to the main application
An event listener in the main application gets the message with the results
For my website I am using an API where I need to load several variables from. Each of the variables are dependent on the return value of the previous call (I use the returned variable from call 1 to make call 2 etc).
Example:
Say that I need to make 5 different API calls to gather all of my data and each is dependent on the return value of the previous call. Then in my case I am doing like this. I am passing a callback function to the first function that loads the data. Then that function will make the first API call. When that call is finished it will pass the callback function to the next function that makes the second API call, and so on. When the last API call is finished the callback function gets called and then I know that all the data has been loaded. In code it would look something like this (I am using the Trello API in my application so I will use it in the example below):
function loadData(cb){
//Make the first API call
Trello.get('/member/me/boards', function(boards){
myBoards = boards;
for(var i = 0; i < boards.length; i++){
//Make the second API call
Trello.get('/boards/' + board[i].id + '/lists', function(lists){
board[i].lists = lists;
//Then make the third and fourth and so on
.....
//When all calls are made call the callback function
cb();
});
});
}
As you can see the callback function will be passed a long way into the callstack. I was wondering if there is a better way to load the data and to store it (as of now I just store everything in a large array). And what is some best practices for loading large amount of data from an API?
P.S. In my original code each of the API calls are in separate functions, but I simplified it here to reduce the amount of code in the example.
I don't know if this is an option for you but using TypeScript makes solving this kind of JavaScript problem much more simple:
async function loadData() {
const boards = await Trello.get('/member/me/boards');
return boards.map(async (board) => {
const lists = await Trello.get('/boards/' + board.id + '/lists');
const something = await Trello.get('/...');
const somethingElse = await Trello.get('/...');
// ...more calls
return {
...board,
lists: lists,
something: something,
somethingElse: somethingElse
// ... more attributes
};
});
}
loadData().then((data) => console.log(data));
Without fully understanding your problem this may not be a valid solution, but taking a quick glance at the trello api docs shows a batch call you could make to avoid looping at each level. Batching these would allow for many fewer API calls at each level and would be considered a best practice:
function loadData(cb){
//Make the first API call
Trello.get('/member/me/boards', function(boards){
myBoards = boards;
var boardAPIs = [];
var boardResponses = [];
for(var i = 0; i < boards.length; i++){
boardAPIs.push('/boards/' + board[i].id + '/lists');
//max of 10 at a time per documentation
if (boardAPIs.length == 10 || i >= (boards.length - 1)) {
//Make the second level API call
Trello.get('/batch/?urls=' + boardAPIs.join(','), function(boards){
// collect response information on all boards, then continue with third request
boardResponses.push(...);
if (i >= (boards.length - 1)) {
// all board requests have been made, continue execution at third level
// if this were the last level of calls, you could call cb() here
for(var j = 0; i < boardResponses.length; i++){
// loop inside responses to get individual board responses, build up next set of batch requests
}
}
});
boardAPIs= [];
}
});
});
}
One thing to note here: the docs mentioned that you can only batch 10 requests at a time, so I added some code in there to check for that.
This post provides more information on how to consume the batch service:
this means you get only a single response back, and it looks a little
different from a normal response. The response is an array of objects
– but not of the normal response objects you might expect. Instead,
it’s an object with a single property, with a name set to the HTTP
response code of the request.
You may focus on a deep first approach, so that the first data arrives fast at the client:
function loadData(showChunk){
//Make the first API call
Trello.get('/member/me/boards', function(boards){
myBoards = boards;
(function getboard(i){
//Make the second API call
Trello.get('/boards/' + board[i].id + '/lists', function(lists){
board[i].lists = lists;
//Then make the third and fourth and so on
.....
//When all calls are made for the first board call the callback function, and also continue with the next board
showChunk();
if(i+1<boards.length) setTimeout(getboard, 1, i+1);
});
})(0);
});
}
Out of all the questions I have seen on this site about setTimeout the majority of them people are executing the method in a setTimeout declaration. I am not doing this.
Code:
var nums = [1,2,3,4,5,6,7,8,9];
for(var i=0; i < nums.length; i++){
setTimeout(function(num){
var url = getUrl(num);
//query(url); // prints out query operation and data return
console.log(num);
}, 1000, nums[i]);
}
When running the command to start the app, query will execute all items in the array no matter how large I set the millisecond delay. When running this code with printing out to console I get the same effect. Am I doing this wrong? Can anyone else duplicate this?
I created an app that would run setInterval to test api throttling, and it works as expected. I am thinking I may be missing some kind of backend knowledge about node with setTimeout and loops? setTimeout Docs
The docs do say that it won't guarantee the time that it will execute the callback, but I have 120 items that it is going to fire logging on and it fires them all at once.
since those function setTimeout or setInterval works asynchronously,
You have to call the next loop in the setTimeout/setInterval callback function.
var _i = 0
var doQuery = setInterval(function(){
if(_i == nums.length) clearInterval(doQuery);
var url = getUrl(nums[_i]);
query(url);
_i++;
},1000);
In my previous question I thought I'd got it sorted, but I've found an intermittent edge condition where the "then" part is being carried out before all the promises resolve in the Q.all call.
Simply, I have the following set up where a generic calculation is called multiple times with different parameters: (code simplified a bit to remove irrelevant code and actual parameter names)
var promiseArray = [];
promiseArray.push(genericCalc(param1, param2, param3));
promiseArray.push(genericCalc(param1, param2, param3));
promiseArray.push(genericCalc(param1, param2, param3));
var calcPromise = Q.all(promiseArray);
return calcPromise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone(res) {
calcTotals();
setChart(selectedRow());
console.log("done recalculation routine");
}
function calcsFailed() {
logger.logError("Failure in Calculations", "", null, true);
}
genericCalc (with some irrelevant code removed)
var genericCalc = function (param1, param2, param3) {
//specific calculation is determined using parameters passed in and varies but is same structure for each as below
calcPromise = specificCalc(params...);
return calcPromise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone(res) {
//some minor subtotalling in here using "res" and flag setting
return Q.resolve();
}
function calcsFailed() {
//handle error....
}
};
There are 3 or 4 different specific calculations and they all have roughly the same sort of layout:
function specificCalc1(params...) {
var promise = calcContext.ajaxCalc(params);
return promise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone(res) {
rv(res);
console.log("done specific calc 1");
return Q.resolve();
}
function calcsFailed() {
logger.logError("Failure in specific calc 1", "", null, true);
return Q.resolve();
}
};
Now I know that this is not a great idea to push the result of the ajax calc into a return value, but at present I don't want to change this as there is just too much code change involved and I feel that, even if not the best methodology at present, this is a learning curve for me and I'll address that part once I have my head around this strange edge condition.
So... what throws me is that every now and again when I change some values on screen which trigger a recalculation, one of the "done" console log messages from one of the specific calcs appears AFTER the "done recalculation routine" one from the first section!
I thought it was being caused by a poorly-constructed promise leading to the function being executed immediately, but the REALLY weird thing is that when I put a debug stop in the server code for that calculation, all works correctly and the client browser is paused until the server code is continued and then the client side debugs show that the next points are being hit.
9 times out of 10 it all works perfectly. On the 10th one, I see "done recalculation routine" in the console immediately followed by "done specific calc 2" and this causes the totals to be wrong.
I cannot see how this can be happening. I've artificially put time delay loops in all the specific calc functions to make them take several seconds and never once do I see the out of sequence debug messages, but when there is no artificial slowdown in place, I see it 1 in 10 times approximately.
Please would someone try and put me out of my misery here... I just want to know I can rely on the "Q.all" call working in the first block of code, such that when it hits "calcsDone" it has ALWAYS done the generic and specific calcs!
Edit 1:
To explain the "rv" or returnvalue a bit. In "genericCalc", I declare an observable:
var res = ko.observable(); //holds returned results
And as part of the call to "specificCalc" I pass that return value in, for example:
calcPromise = specificCalc1(isPackaging, res, thisArticle);
In the specificCalc I just put the ajax result into the observable, so it is available in "calcsDone" in "genericCalc" and, once all calcs are completed from the "Q.all()" function, the calculation routines can tot up the individual results from each specific Calc.
Edit 2
The console log, when it goes wrong is:
done specificCalc1
done specificCalc2, isPackaging=true
Done specificCalc3, redraw = false
done calctotals
chart replotted
done recalculation routine
done specificCalc2, isPackaging=false
...you can see the "done specificCalc2" after "done recalculation routine" there.
Edit 3
I reduced the promiseArray to just one item to check, using the parameters for what seems to be the troublesome call (specificCalc2):
var promiseArray = [];
promiseArray.push(genericCalc(param1, param2, param3));
And then I put a stop point in the server code.
The result is that the stop in the server code happens and the console already says "done" so it's a problem with the promise construction after all and was somehow being masked by one of the other calls being done. Once I release the stop in server code, I get my console messages from the ajax call function AND the generic Calc function saying "done".
So, it appears to me as if the problem is in the main Q.all call from what I can tell, as that doesn't wait for the generic calc function being completed before it carries on with its "calcsDone" function.
I just tried returning the genericCalc promise:
return genericCalc("eol_", true, false, false)
//return calcPromise
.then(calcsDone)
.fail(calcsFailed);
...and it instantly fails with "cannot call method "then" of undefined, thus confirming my problem is in the generic calc function, so off to look at that now.
Edit 4
Narrowed the problem down to a further function call within genericCalc. As part of the calculation, this calls a function to remove the impact value as stands before the calculation is done. When the calculation returns it then adds the result back in to the overall amount.
If I "return Q.resolve()" from genericCalc on the line before I do:
impactRemove(currentPrefix, currentArticle); //remove impacts for this prefix
then the promise works, if I do it on the line after, it fails. So for some reason calling that function seems to resolve the promise. Investigating further now.
Edit 5
So the problem is caused when I call a standard function midway through the genericCalc routine. The logic behind this is:
Change on browser form retriggers calculation
Function is called that sets up array of promises (several of which call the same function with different parameters)
Inside that generic function (genericCalc) I call a standard non-promise function that removes the current calculation totals from the project total
Calculation is complete
Standard non-promise function called to add results of calculation back to project total
GenericCalc returns promise to main function
Overall totals updated with latest project totals, graphics are updated
What actually seems to happen is that when I call the standard javascript functions with genericCalc, they execute immediately, therefore resolving the promise and although the ajax call is still done, the Q.all call does not wait as it believes the promise is resolved as "genericCalc" is returning "undefined" and not a promise.
At this point, Bergi is screaming at me about my hideous anti-pattern noob coding and I tend to agree with him. Trouble is, I'd like to get it working this way so I have something to test against when I finally adapt it to work correctly.
So... if I have two functions called from within "genericCalc" like so:
impactRemove(currentPrefix, currentArticle); //remove impacts for this prefix
and
impactAdd(currentPrefix, currentArticle); //addimpacts for this prefix
Which are basically like this:
function impactAdd(prefix, prjArt) {
if (!prjArt) {return} //just get out as there's nothing to calculate
factorArray.forEach(function (f) {
var orig_projValue = pGlobalVar.editProject()[prefix + f]();
var new_projArtValue = prjArt[prefix + f](); //must be set first!
pGlobalVar.editProject()[prefix + f](orig_projValue + new_projArtValue); //add new value back in to project value
});
}
...then how do I call these "midpoint" functions within the promise of genericCalc so that I Can still return a promise when a) impactRemove has been completed, b) the remote ajax call has been completed and c) the impactAdd function has been completed?
Do I need to set up code within genericCalc to do something like:
impactRemove(params...).then(<ajax function that returns new calculation results>).then(impactAdd(params))
...or will (params) after my functions automatically invoke those functions, thus resolving the promise too early? This is all so confusing compared to what I'm used to!
Edit6
All genericCalc does is this:
var genericCalc = function (param1, param2, param3) {
//specific calculation is determined using parameters passed in and varies but is same structure for each as below
calcPromise = specificCalc(params...);
impactRemove(currentPrefix, currentArticle); //remove impacts for this prefix
return calcPromise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone(res) {
//some minor subtotalling in here using "res" and flag setting
impactAdd(currentPrefix, currentArticle); //addimpacts for this prefix
return Q.resolve();
}
function calcsFailed() {
//handle error....
}
};
"specificCalc" returns a promise - that one works as I've checked the contents of the promise at a breakpoint. If I remove the calls to "impactRemove" and "impactAdd" above, then "genericCalc" also works. It is those calls that cause the problem.
This is why I think I need something like:
impactRemove(params)
.then(return calcPromise(params)
.then(impactAdd(params);
...but neither impactAdd nor impactRemove do anything asynchronously and I'm also not sure how I can set this up as I need to use params and yet you said those params will mean the functions are immediately invoked...
Edit 7
So, as mentioned in the lengthy comments section, this is being caused by a "forEach" loop in genericCalc:
var genericCalc = function (calcPrefix, other params...) {
gcCount++;
console.log("generic calc starting for: " + calcPrefix + ", count=" + gcCount);
pGlobalVar.projectIsCalculating(true); //controls spinner gif
var res_Array = ko.observable(); //holds returned results
var _prjArticleArray = []; //create local array to use
if (currentArticle == true) {
_prjArticleArray.push(pGlobalVar.editProjectArticle());
} else {
projectResults().projectArticles().forEach(function (pa) {
_prjArticleArray.push(pa);
});
};
_prjArticleArray.forEach(function (thisArticle) {
var calcPromise;
switch (calcPrefix) {
case "eol_":
calcPromise = Q.all([calcEOL(isPackaging, res_Array, thisArticle)]);
break;
case "use_":
calcPromise = Q.all([calcUse(isPackaging, res_Array, thisArticle)]);
break;
case "air_":
calcPromise = Q.all([calcFlight(isPackaging, res_Array, thisArticle)]);
break;
default:
break;
}
impactRemove(calcPrefix, thisArticle); //remove impacts for this prefix
return calcPromise
.then(calcsDone)
.fail(calcsFailed);
function calcsDone(args) {
//do some calcs and totalling based on returned results
impactAdd(calcPrefix, thisArticle); //add this article impact into project total
console.log("generic calc done for: " + calcPrefix + ", count=" + gcCount);
calcTotals(); //accmulates individual results into the "total_xxx" used on the results table and in the chart
setChart(selectedRow());
pGlobalVar.projectIsCalculating(false);
}
function calcsFailed() {
logger.logError("Failure in " + calcPrefix + "calculation", "", null, true);
impactAdd(calcPrefix); //will add back in results as they were at start of calc
pGlobalVar.projectIsCalculating(false);
}
});
};
The only reason I've posted it in all its ugly glory is to point out that all works perfectly IF I remove the "forEach" loop and just run this for one article. Why on earth would the forEach loop make it die a horrible death ?
I think you just want to exchange the order of the specificCalc() and impactRemove() calls. Even if the first is asynchronous, it will start doing its task right now - only the results will arrive in the next turn. If you want to chain anything after a synchronous task, just put it on the next line ("semicolon monad").
Also, if impactRemove does assign to your global (!) variable calcPromise, it might not be a promise any more and throw when calling a .then() method on it. What you want seems to be
function genericCalc(param1, param2, param3) {
impactRemove(currentPrefix, currentArticle); //remove impacts for this prefix
return specificCalc(params...).finally(function() {
impactAdd(currentPrefix, currentArticle); // add impacts for this prefix
}).then(function calcsDone(res) {
// some minor subtotalling in here using "res" and flag setting
return subtotal;
}, function calcsFailed() {
// handle error....
});
}
Why on earth would the forEach loop make it die a horrible death?
Because you're not returning a promise. A forEach loop has its own callback, from which you can return, but from the genericCalc function nothing is returned. Q.all will not fret about that, but just take the undefined as a value. However, the async action is to be started and you get your callbacks, but Q.all won't wait for it, because it does not know of it.
The solution is a quite simple change and has already been explained here.
I noticed a strange behavior: if I have a series of tasks and wish to defer their execution, then I can use a setTimeout with 0 delay for each of them.
(see http://javascript.info/tutorial/events-and-timing-depth#the-settimeout-func-0-trick)
Everything works perfectly: the tasks are queued and executed as soon as possible.
But ... if the invocation of the various setTimeout is very close, then I found that sometimes (rarely happens!) is not executed in the correct order.
Why?
Nobody ever promised they would be fired in the "correct" order (the tasks with the same timeout will be executed in the order they are set to time out). setTimeout only guarantees that:
each timeout is executed exactly once (unless the page dies in the meantime)
each timeout is executed no sooner than when it is supposed to.
There is no word about execution order. In fact, even if the implementor tried to preserve order (even as a side-effect), most likely there is not enough time resolution to provide a unique sort order to all tasks, and a binary heap (which may well be used here) does not preserve insertion order of equal keys).
If you want to preserve the order of your deferred tasks, you should only enqueue one when the previous one is done.
This should work:
var defer = (function(){
//wrapped in IIFE to provide a scope for deferreds and wrap
var running = false;
var deferreds = [];
function wrap(func){
return function(){
func();
var next = deferreds.shift();
if(next){
setTimeout(wrap(next),0);
}else{
running = false;
}
}
}
return function(func){
if(running){
deferreds.push(func);
}else{
setTimeout(wrap(func),0);
running = true;
}
}
})()
Demo: http://jsfiddle.net/x2QuB/1/
You can consider using jquery deferreds ( or some other implementation of deferreds), which can handle this pattern very elegantly.
The important point to note is that the deferred done callbacks are executed in the order in which they are added.
var createCountFn = function(val){
return function(){
alert(val)
};
}
// tasks
var f1 = createCountFn(1),
f2 = createCountFn('2nd'),
f3 = createCountFn(3);
var dfd = $.Deferred();
dfd.done(f1).done(f2).done(f3);
dfd.resolve();
demo
The HTML5 draft specification states that the setTimeout method can be run asynchronously (implying that the order that the callbacks will be executed may not be preserved), which could be what your browser is doing.
The setTimeout() method must run the following steps:
...
6. Return handle, and then continue running this algorithm asynchronously.
7. If the method context is a Window object, wait until the Document associated with the method context has been fully active for a further timeout milliseconds (not necessarily consecutively).
In any case one could workaround this issue by doing something similar to this:
function inOrderTimeout(/* func1[, func2, func3, ...funcN], timeout */)
{ var timer; // for timeout later
var args = arguments; // allow parent function arguments to be accessed by nested functions
var numToRun = args.length - 1; // number of functions passed
if (numToRun < 1) return; // damm, nothing to run
var currentFunc = 0; // index counter
var timeout = args[numToRun]; // timeout should be straight after the last function argument
(function caller(func, timeout) // name so that recursion is possible
{ if (currentFunc > numToRun - 1)
{ // last one, let's finish off
clearTimeout(timer);
return;
}
timer = setTimeout(function ()
{ func(); // calls the current function
++currentFunc; // sets the next function to be called
caller(args[currentFunc], timeout);
}, Math.floor(timeout));
}(args[currentFunc], timeout)); // pass in the timeout and the first function to run
}