In a language with threads and locks it is easy to implement a lazy load by checking the value of a variable, if it's null then lock the next section of code, check the value again and then load the resource and assign. This prevents it from being loaded multiple times and causes threads after the first to wait for the first thread to complete the action that's needed.
Psuedo code:
if(myvar == null) {
lock(obj) {
if(myvar == null) {
myvar = getData();
}
}
}
return myvar;
JavaScript runs in a single thread, however, it still has this type of issue because of asynchronous execution while one call is waiting on a blocking resource. In this Node.js example:
var allRecords;
module.exports = getAllRecords(callback) {
if(allRecords) {
return callback(null,allRecords);
}
db.getRecords({}, function(err, records) {
if (err) {
return callback(err);
}
// Use existing object if it has been
// set by another async request to this
// function
allRecords = allRecords || partners;
return callback(null, allRecords);
});
}
I'm lazy loading all the records from a small DB table the first time this function is called and then returning the in-memory records on subsequent calls.
Problem: If multiple async requests are made to this function at the same time then the table is going to be loaded unnecessarily from the DB multiple times.
In order to solve this I could simulate a locking mechanism by creating a var lock; variable and setting it to true while the table is loading. I would then put the other async calls into a setTimeout() loop and check back on this variable every (say) 1 second until the data was available and then allow them to return.
The problems with that solution are:
It's fragile, what if the first async call throws and doesn't unset the lock.
How many times do we loop back into the timer before giving up?
How long should the timer be set for? In some environments 1 second might be way too long and inefficient.
Is there a best practise for solving this in JavaScript?
On the first call to the service, initialize an array. Start the fetch operation. Create a Promise, store it in the array.
On subsequent calls, if the data is there, return an already-fulfilled Promise. If not, add another Promise to the array and return that.
When the data arrives, resolve all the waiting Promise objects in the list. (You can throw away the list once the data's there.)
I really like the promise solution in the other answer -- very clever, very interesting. Promises aren't the dominent methodology, so you may need to educate the team. I'm going to go in another direction though.
What you're after is a memoize function -- an in-memory key/value cache of expensive results. JavaScript the Good Parts has a memoize sample towards the end. Lodash has a memoize function. These assume synchronous processing so don't account for your scenario -- which is to say they'd hit the database lots of times until one of the "threads" replied.
The async library also has a memoize function that does exactly what you want. In it's innards, it keeps a queue array of callbacks, and once it gets the answer, it both caches it and calls all the callbacks.
If you're into inventing, by all means, use promises. If you'd just like a plug-n-play answer, use async#memoize.
Related
I am reaching out to a SongKick's REST API in my ReactJS Application, using Axios.
I do not know how many requests I am going to make, because in each request there is a limit of data that I can get, so in each request I need to ask for a different page.
In order to do so, I am using a for loop, like this:
while (toContinue)
{
axios.get(baseUrl.replace('{page}',page))
.then(result => {
...
if(result.data.resultsPage.results.artist === undefined)
toContinue = false;
});
page++;
}
Now, the problem that I'm facing is that I need to know when all my requests are finished. I know about Promise.All, but in order to use it, I need to know the exact URLs I'm going to get the data from, and in my case I do not know it, until I get empty results from the request, meaning that the last request is the last one.
Of course that axios.get call is asynchronous, so I am aware that my code is not optimal and should not be written like this, but I searched for this kind of problem and could not find a proper solution.
Is there any way to know when all the async code inside a function is finished?
Is there another way to get all the pages from a REST API without looping, because using my code, it will cause extra requests (because it is async) being called although I have reached empty results?
Actually your code will crash the engine as the while loop will run forever as it never stops, the asynchronous requests are started but they will never finish as the thread is blocked by the loop.
To resolve that use async / await to wait for the request before continuing the loop, so actually the loop is not infinite:
async function getResults() {
while(true) {
const result = await axios.get(baseUrl.replace('{page}', page));
// ...
if(!result.data.resultsPage.results.artist)
break;
}
}
I initially assumed that passing a bare Promise.mapSeries(...) call as an argument to .then() would be the same as wrapping it in a function, like .then(function() { return Promise.mapSeries(...); }). Having written out this question, I'm no longer entirely sure why it works at all.
In the simplified code below, I open a couple of databases asynchronously (openDBAsync()) and then read a file containing a JS object. I use _.map() to iterate over all of the key/value pairs in the object and asynchronously update their values in a database, while keeping track of which ones meet certain criteria (whether the value is odd, in this toy example). Promise.all() waits for all of the async database calls to settle, and then Promise.mapSeries() is used to process each of the subset of keys, which makes another async database call for each one. Finally, I close all the databases.
function processData(path)
{
var oddKeys = [];
return openDBAsync()
.then(function() { return readFileAsync(path); })
.then(function(dataObject) {
return Promise.all(_.map(dataObject, function(value, key) {
if (value % 2) {
oddKeys.push(key);
}
return updateDBAsync(key, ++value);
}))
.then(Promise.mapSeries(
oddKeys,
function(key) {
return updateOddDBAsync(key);
}
))
})
.then(closeDBAsync);
}
The problem is that the database throws errors complaining that I'm trying to update the database after it's been closed. That means that some of the promises generated in the .mapSeries() call are being called after the final closeDBAsync(). I expected all of them to settle before the final .then() call.
If I wrap the call to Promise.mapSeries() in a function:
.then(function() {
return Promise.mapSeries(
oddKeys,
function(key) {
return updateOddDBAsync(key);
}
);
})
Then I don't get any errors. It also works if I put a .delay(2000) before the close database call, which indicates that Promise.mapSeries() isn't settling all of the promises before it finishes.
This seems like either a bug in Bluebird, or, much more likely, I'm not understanding something fundamental about how Promise.mapSeries() works. Any guidance would be much appreciated.
much more likely, I'm not understanding something fundamental about how Promise.mapSeries() works
Nope, this seems to be more a misunderstanding about how .then(…) works.
The then method of promises does always take a callback function (if you're passing anything else [but null], Bluebird should spit out a warning!). By calling .then(Promise.mapSeries(…)), you were passing a promise instead, which is simply ignored. Being ignored, it also is not awaited by anything, which leads to that error of the database being closed too early.
But the direct call to Promise.mapSeries() doesn't get applied to the array immediately. If it was, the array would be empty and this code wouldn't work at all.
Yes it does. Your array is filled from by the _.map callback, which is executed synchronously, before then and mapSeries are invoked.
So the solution is indeed to wrap the call in a function expression, which will only be executed when the Promise.all(…) fulfills, and whose result will then not be ignored but rather awaited. There might be more, different solutions, depending on what degree of parallel execution you want to allow.
Btw, given you are doing a database transaction, your current code is quite fragile. Look into the promise disposer pattern.
I recently did a lot of coding in AngularJS. After some time it started to feel comfortable with it and also got really productive. But unfortunately there is this one thing I don't understand:
Within my project I need to get data through $http.get and a RESTful API server. This is where I started to stumble first. After implementing promise ($q.defer etc and .then) at functions which are processing data that's necessary to continue, I thought I conquered the problem.
But in this code:
$scope.getObservationsByLocations = function() {
var promise = $q.defer();
var locationCount = 0;
angular.forEach($scope.analysisData, function(loc) { // for each location
$http.get($scope.api + 'Device?_format=json', { // get all devices
params: {
location: loc.location.id
}
}).then(function (resultDevices) {
var data = angular.fromJson(resultDevices);
promise.resolve(data);
// for each device in this location
angular.forEach(angular.fromJson(resultDevices).data.entry.map(function (dev) {
http.get($scope.api + 'Observation?_format=json', { // get all observations
params: {
device: dev.resource.id
}
}).then(function (resultObservations) {
var observations = angular.fromJson(resultObservations);
// for each obervation of that device in this location
angular.forEach(observations.data.entry.map(function(obs) {
$scope.analysisData[locationCount].observations.push({observation: obs.resource});
}));
})
}))
});
locationCount++
});
return promise.promise
};
I can't understand in which order the commands are executed. Since I use the Webstorm IDE and it's debugging feature, it would be more accurate to say I don't know why the commands are executed in an order I don't understand.
Thinking simple, everything included in the forEach have to be executed before the return is reached, because $http.get's are connected through .then's. But following the debugging information, the function iterates over locationCount++ and even returns the promise before it goes deeper (meaning after the first .then() ).
What's that all about? Did I misunderstood this part of the AngularJS concept?
Or is this just really bad practice and I should reach out for a different solution?
If the context is important/interesting: Objects are based on i.e. https://www.hl7.org/fhir/2015May/location.html#5.15.3
With JavaScript you can create only single thread applications, although e.g. here they say it is not guaranteed to be like that.
But we are talking about the real world and real browsers, so your code sample is running as a single thread (by the way the same thread is also used for rendering your CSS and HTML, at least in Firefox).
When it comes to an asynchronous call
$http.get($scope.api + 'Device?_format=json', {
it says "hey, I can do that later". And it waits with that because it must go on with the current thread.
Then, once the current task is done with return it finally can start getting the remote data.
Proof? Check this fiddle:
console.log(1);
for (var i=0;i<1000000;i++) setTimeout(function(){
console.log(2);
},0);
console.log(3);
You see the spike with the for loop? This is the moment when it registers the setTimeout asynchronous calls. Still 3 is printed before 2 because the task is not done until the 3 is printed.
The $http.get is asynchronous, so depending on (among other things) how large the fetched data is, the time it takes to 'complete' the get is variable. Hence why there is no saying in what order they will be completed
I have developed a small lib for the Dynamics CRM REST/ODATA webservice (CrmRestKit). The lib dependes on jQuery and utilizes the promise-pattern, repectivly the promise-like-pattern of jQuery.
Now I like to port this lib to bluebird and remove the jQuery dependency. But I am facing a problem because bluebird does not support the synchronous resolution of promise-objects.
Some context information:
The API of the CrmRestKit excepts an optional parameter that defines if the web-service call should be performed in sync or async mode:
CrmRestKit.Create( 'Account', { Name: "foobar" }, false ).then( function ( data ) {
....
} );
When you pass "true" or omit the last parameter, will the method created the record in sync. mode.
Sometimes it is necessary to perform a operation in sync-mode, for instance you can write JavaScript code for Dynamics CRM that is involed for the save-event of an form and in this event-handler you need to perform sync-operation for validation (e.g. validate that a certain number of child-records exist, in case the right number of records exist, cancel the save-operation and show an error message).
My problem now is the following: bluebird does not support the resolution in sync-mode. For instance when I do the following, the "then" handler is invoked in async fashion:
function print( text ){
console.log( 'print -> %s', text );
return text;
}
///
/// 'Promise.cast' cast the given value to a trusted promise.
///
function getSomeTextSimpleCast( opt_text ){
var text = opt_text || 'Some fancy text-value';
return Promise.cast( text );
}
getSomeTextSimpleCast('first').then(print);
print('second');
The output is the following:
print -> second
print -> first
I would expect that the "second" appears after the "first" because the promise is already resolved with an value. So I would assume that an then-event-handler is immediately invoked when applied on an already resolved promise-object.
When I do the same (use then on an already resolved promise) with jQuery I will have my expected result:
function jQueryResolved( opt_text ){
var text = opt_text || 'jQuery-Test Value',
dfd = new $.Deferred();
dfd.resolve(text);
// return an already resolved promise
return dfd.promise();
}
jQueryResolved('third').then(print);
print('fourth');
This will generate the following output:
print -> third
print -> fourth
Is there a way to make bluebird work in the same fashion?
Update:
The provided code was just to illustrate the problem. The idea of the lib is: Regardless of the execution-mode (sync, async) the caller will always deal with an promise-object.
Regarding "... asking the user... doesn't seems to make any sense": When you provide two methods "CreateAsync" and "CreateSync" it is also up to the user to decide how the operation is executed.
Anyway with the current implementation the default behavior (last parameter is optional) is a async execution. So 99% of the code requires a promise-object, the optional parameter is only use for the 1% cases where you simply need a sync execution. Furthermore I developed to lib for myself and I use in 99,9999% of the case the async mode but I thought it is nice to have the option to go the sync-road as you like.
But I thinks I got the point an sync method should simply return the value. For the next release (3.0) I will implement "CreateSync" and "CreateAsync".
Thanks for your input.
Update-2
My intension for the optional parameter was to ensure a consistend behavior AND prevent logic error. Assume your as a consumer of my methode "GetCurrentUserRoles" that uses lib. So the method will alway return an promise, that means you have to use the "then" method to execute code that depends on the result. So when some writes code like this, I agree it is totally wrong:
var currentUserRoels = null;
GetCurrentUserRoles().then(function(roles){
currentUserRoels = roles;
});
if( currentUserRoels.indexOf('foobar') === -1 ){
// ...
}
I agree that this code will break when the method "GetCurrentUserRoles" changes from sync to async.
But I understand that this I not a good design, because the consumer should now that he deals with an async method.
Short version: I get why you want to do that, but the answer is no.
I think the underlying question being asked is whether a completed promise should immediately run a callback, if the promise has already completed. I can think of a lot of reasons that this might happen - for example, an asynchronous save procedure that only saves data if changes were made. It may be able to detect changes from the client side in a synchronous fashion without having to go through an external resource, but if changes are detected then and only then would an asynchronous operation be required.
In other environments that have asynchronous calls, the pattern seems to be that the developer is responsible for understanding that their work might complete immediately (for example, .NET framework's implementation of the async pattern accomodates this). This is not a design problem of the framework, it's the way it's implemented.
JavaScript's developers (and many of the commenters above) seem to have a different point of view on this, insisting that if something might be asynchronous, it must always be asynchronous. Whether this is "right" or not is immaterial - according to the specification I found at https://promisesaplus.com/, item 2.2.4 states that basically no callbacks can be called until you are out of what I'll refer to as "script code" or "user code"; that is, the specification says clearly that even if the promise is completed you can't invoke the callback immediately. I've checked a few other places and they either say nothing on the topic or agree with the original source. I don't know if https://promisesaplus.com/ could be considered a definitive source of information in this regard, but no other sources that I saw disagreed with it and it seems to be the most complete.
This limitation is somewhat arbitrary and I frankly prefer the .NET perspective on this one. I'll leave it up to others to decide if they consider it "bad code" to do something that might or might not be synchronous in a way that looks asynchronous.
Your actual question is whether or not Bluebird can be configured to do the non-JavaScript behavior. Performance-wise there may be a minor benefit to doing so, and in JavaScript anything's possible if you try hard enough, but as the Promise object becomes more ubiquitous across platforms you will see a shift to using it as a native component instead of custom written polyfills or libraries. As such, whatever the answer is today, reworking a promise in Bluebird is likely to cause you problems in the future, and your code should probably not be written to depend on or provide immediate resolution of a promise.
You might think this is a problem, because there's no way to have
getSomeText('first').then(print);
print('second');
and to have getSomeText "first" printed before "second" when the resolution is synchronous.
But I think you have a logic problem.
If your getSomeText function may be synchronous or asynchronous, depending on the context, then it shouldn't impact the order of execution. You use promises to ensure it's always the same. Having a variable order of execution would likely become a bug in your application.
Use
getSomeText('first') // may be synchronous using cast or asynchronous with ajax
.then(print)
.then(function(){ print('second') });
In both cases (synchronous with cast or asynchronous resolution), you'll have the correct execution order.
Note that having a function being sometimes synchronous and sometimes not isn't a weird or unlikely case (think about cache handling, or pooling). You just have to suppose it's asynchronous, and all will be always fine.
But asking the user of the API to precise with a boolean argument if he wants the operation to be asynchronous doesn't seem to make any sense if you don't leave the realm of JavaScript (i.e. if you don't use some native code).
The point of promises is to make asynchronous code easier, i.e. closer to what you feel when using synchronous code.
You're using synchronous code. Don't make it more complicated.
function print( text ){
console.log( 'print -> %s', text );
return text;
}
function getSomeTextSimpleCast( opt_text ){
var text = opt_text || 'Some fancy text-value';
return text;
}
print(getSomeTextSimpleCast('first'));
print('second');
And that should be the end of it.
If you want to keep the same asynchronous interface even though your code is synchronous, then you have to do it all the way.
getSomeTextSimpleCast('first')
.then(print)
.then(function() { print('second'); });
then gets your code out of the normal execution flow, because it's supposed to be asynchronous. Bluebird does it the right way there. A simple explanation of what it does:
function then(fn) {
setTimeout(fn, 0);
}
Note that bluebird doesn't really do that, it's just to give you a simple example.
Try it!
then(function() {
console.log('first');
});
console.log('second');
This will output the following:
second
first
There are some good answers here already, but to sum up the crux of the matter very succinctly:
Having a promise (or other async API) that is sometimes asynchronous and sometimes synchronous is a bad thing.
You may think it's fine because the initial call to your API takes a boolean to switch off between sync/async. But what if that's buried in some wrapper code and the person using that code doesn't know about these shenanigans? They've just wound up with some unpreditable behavior through no fault of their own.
The bottom line: Don't try to do this. If you want synchronous behavior, don't return a promise.
With that, I'll leave you with this quotation from You Don't Know JS:
Another trust issue is being called "too early." In application-specific terms, this may actually involve being called before some critical task is complete. But more generally, the problem is evident in utilities that can either invoke the callback you provide now (synchronously), or later (asynchronously).
This nondeterminism around the sync-or-async behavior is almost always going to lead to very difficult to track down bugs. In some circles, the fictional insanity-inducing monster named Zalgo is used to describe the sync/async nightmares. "Don't release Zalgo!" is a common cry, and it leads to very sound advice: always invoke callbacks asynchronously, even if that's "right away" on the next turn of the event loop, so that all callbacks are predictably async.
Note: For more information on Zalgo, see Oren Golan's "Don't Release Zalgo!" (https://github.com/oren/oren.github.io/blob/master/posts/zalgo.md) and Isaac Z. Schlueter's "Designing APIs for Asynchrony" (http://blog.izs.me/post/59142742143/designing-apis-for-asynchrony).
Consider:
function result(data) {
console.log( a );
}
var a = 0;
ajax( "..pre-cached-url..", result );
a++;`
Will this code print 0 (sync callback invocation) or 1 (async callback invocation)? Depends... on the conditions.
You can see just how quickly the unpredictability of Zalgo can threaten any JS program. So the silly-sounding "never release Zalgo" is actually incredibly common and solid advice. Always be asyncing.
What about this case, also CrmFetchKit related which in latest version uses Bluebird. I have upgraded from version 1.9 that was based on jQuery. Still the old app code that uses CrmFetchKit has methods the prototypes of which I can't or won't change.
Existing App Code
CrmFetchKit.FetchWithPaginationSortingFiltering(query.join('')).then(
function (results, totalRecordCount) {
queryResult = results;
opportunities.TotalRecords = totalRecordCount;
done();
},
function err(e) {
done.fail(e);
}
);
Old CrmFetchKit implementation (a custom version of fetch())
function fetchWithPaginationSortingFiltering(fetchxml) {
var performanceIndicator_StartTime = new Date();
var dfd = $.Deferred();
fetchMore(fetchxml, true)
.then(function (result) {
LogTimeIfNeeded(performanceIndicator_StartTime, fetchxml);
dfd.resolve(result.entities, result.totalRecordCount);
})
.fail(dfd.reject);
return dfd.promise();
}
New CrmFetchKit implementation
function fetch(fetchxml) {
return fetchMore(fetchxml).then(function (result) {
return result.entities;
});
}
My problem is that the old version had the dfd.resolve(...) where I was able to pass any number of params that I need.
The new implementation just returns, the parent seems to call the callback, I can't call it directly.
I went and made a custom version of the fetch() in the new implementation
function fetchWithPaginationSortingFiltering(fetchxml) {
var thePromise = fetchMore(fetchxml).then(function (result) {
thePromise._fulfillmentHandler0(result.entities, result.totalRecordCount);
return thePromise.cancel();
//thePromise.throw();
});
return thePromise;
}
But the problem is that the callback gets called two times, once when I do it explicitly and second by the framework but it passes it one parameter only. To trick it and "tell" not to call anything because I do it explicitly I try to call .cancel() but it is ignored. I understood why but still how do you do the "dfd.resolve(result.entities, result.totalRecordCount);" in the new version with out having to changes prototypes in the app that uses this library ?
You can in fact do this, yes.
Modify the bluebird.js file (for npm: node_modules/bluebird/js/release/bluebird.js), with the following change:
[...]
target._attachExtraTrace(value);
handler = didReject;
}
- async.invoke(settler, target, {
+ settler.call(target, {
handler: domain === null ? handler
: (typeof handler === "function" &&
[...]
For more info, see here: https://github.com/stacktracejs/stacktrace.js/issues/188
Is there a way to wait on a promise so that you can get the actual result from it and return that instead of returning the promise itself? I'm thinking of something similar to how the C# await keyword works with Tasks.
Here is an example of why I'd like to have a method like canAccess() that returns true or false instead of a promise so that it can be used in an if statement. The method canAccess() would make an AJAX call using $http or $resource and then somehow wait for the promise to get resolved.
The would look something like this:
$scope.canAccess = function(page) {
var resource = $resource('/api/access/:page');
var result = resource.get({page: page});
// how to await this and not return the promise but the real value
return result.canAccess;
}
Is there anyway to do this?
In general that's a bad idea. Let me tell you why. JavaScript in a browser is basically a single threaded beast. Come to think of it, it's single threaded in Node.js too. So anything you do to not "return" at the point you start waiting for the remote request to succeed or fail will likely involve some sort of looping to delay execution of the code after the request. Something like this:
var semaphore = false;
var superImportantInfo = null;
// Make a remote request.
$http.get('some wonderful URL for a service').then(function (results) {
superImportantInfo = results;
semaphore = true;
});
while (!semaphore) {
// We're just waiting.
}
// Code we're trying to avoid running until we know the results of the URL call.
console.log('The thing I want for lunch is... " + superImportantInfo);
But if you try that in a browser and the call takes a long time, the browser will think your JavaScript code is stuck in a loop and pop up a message in the user's face giving the user the chance to stop your code. JavaScript therefore structures it like so:
// Make a remote request.
$http.get('some wonderful URL for a service').then(function (results) {
// Code we're trying to avoid running until we know the results of the URL call.
console.log('The thing I want for lunch is... " + results);
});
// Continue on with other code which does not need the super important info or
// simply end our JavaScript altogether. The code inside the callback will be
// executed later.
The idea being that the code in the callback will be triggered by an event whenever the service call returns. Because event driven is how JavaScript likes it. Timers in JavaScript are events, user actions are events, HTTP/HTTPS calls to send and receive data generate events too. And you're expected to structure your code to respond to those events when they come.
Can you not structure your code such that it thinks canAccess is false until such time as the remote service call returns and it maybe finds out that it really is true after all? I do that all the time in AngularJS code where I don't know what the ultimate set of permissions I should show to the user is because I haven't received them yet or I haven't received all of the data to display in the page at first. I have defaults which show until the real data comes back and then the page adjusts to its new form based on the new data. The two way binding of AngularJS makes that really quite easy.
Use a .get() callback function to ensure you get a resolved resource.
Helpful links:
Official docs
How to add call back for $resource methods in AngularJS
You can't - there aren't any features in angular, Q (promises) or javascript (at this point in time) that let do that.
You will when ES7 happens (with await).
You can if you use another framework or a transpiler (as suggested in the article linked - Traceur transpiler or Spawn).
You can if you roll your own implementation!
My approach was create a function with OLD javascript objects as follows:
var globalRequestSync = function (pUrl, pVerbo, pCallBack) {
httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = function () {
if (httpRequest.readyState == 4 && httpRequest.status == 200) {
pCallBack(httpRequest.responseText);
}
}
httpRequest.open(pVerbo, pUrl, false);
httpRequest.send(null);
};
I recently had this problem and made a utility called 'syncPromises'. This basically works by sending what I called an "instruction list", which would be array of functions to be called in order. You'll need to call the first then() to kick things of, dynamically attach a new .then() when the response comes back with the next item in the instruction list so you'll need to keep track of the index.
// instructionList is array.
function syncPromises (instructionList) {
var i = 0,
defer = $q.defer();
function next(i) {
// Each function in the instructionList needs to return a promise
instructionList[i].then(function () {
var test = instructionList[i++];
if(test) {
next(i);
}
});
}
next(i);
return defer.promise;
}
This I found gave us the most flexibility.
You can automatically push operations etc to build an instruction list and you're also able to append as many .then() responses handlers in the callee function. You can also chain multiple syncPromises functions that will all happen in order.