I started using Twisted in a project that require asynchronous programming and the docs are pretty good.
So my question is, is a Deferred in Twisted the same as a Promise in Javascript? If not, what are the differences?
The answer to your question is both Yes and No depending on why you're asking.
Yes:
Both a Twisted Deferred and a Javascript Promise implement a mechanism for queuing synchronous blocks of code to be run in a given order while being decoupled from other synchronous blocks of code.
No:
So Javascript's Promise is actually more similar to Python's Future, and the airy-fairy way to explain this is to talk about the Promise and the Resolver being combined to make a Deferred, and to state that this affects what you can do with the callbacks.
This is all very well and good in that it's accurate, however it doesn't really make anything any clearer, and without typing thousands of words where I'm almost guaranteed to make a mistake, I'm probably better quoting someone who knows a little something about Python.
Guido van Rossum on Deferreds:
Here's my attempt to explain Deferred's big ideas (and there are a lot
of them) to advanced Python users with no previous Twisted experience.
I also assume you have thought about asynchronous calls before. Just
to annoy Glyph, I am using a 5-star system to indicate the importance
of ideas, where 1 star is "good idea but pretty obvious" and 5 stars
is "brilliant".
I am showing a lot of code snippets, because some ideas are just best
expressed that way -- but I intentionally leave out lots of details,
and sometimes I show code that has bugs, if fixing them would reduce
understanding the idea behind the code. (I will point out such bugs.)
I am using Python 3.
Notes specifically for Glyph: (a) Consider this a draft for a blog
post. I'd be more than happy to take corrections and suggestions for
improvements. (b) This does not mean I am going to change Tulip to a
more Deferred-like model; but that's for a different thread.
Idea 1: Return a special object instead of taking a callback argument
When designing APIs that produce results asynchronously, you find that
you need a system for callbacks. Usually the first design that comes
to mind is to pass in a callback function that will be called when the
async operation is complete. I've even seen designs where if you don't
pass in a callback the operation is synchronous -- that's bad enough
I'd give it zero stars. But even the one-star version pollutes all
APIs with extra arguments that have to be passed around tediously.
Twisted's first big idea then is that it's better to return a special
object to which the caller can add a callback after receiving it. I
give this three stars because from it sprout so many of the other good
ideas. It is of course similar to the idea underlying the Futures and
Promises found in many languages and libraries, e.g. Python's
concurrent.futures (PEP 3148, closely following Java Futures, both of
which are meant for a threaded world) and now Tulip (PEP 3156, using a
similar design adapted for thread-less async operation).
Idea 2: Pass results from callback to callback
I think it's best to show some code first:
class Deferred:
def __init__(self):
self.callbacks = []
def addCallback(self, callback):
self.callbacks.append(callback) # Bug here
def callback(self, result):
for cb in self.callbacks:
result = cb(result)
The most interesting bits are the last two lines: the result of each
callback is passed to the next. This is different from how things work
in concurrent.futures and Tulip, where the result (once set) is fixed
as an attribute of the Future. Here the result can be modified by each
callback.
This enables a new pattern when one function returning a Deferred
calls another one and transforms its result, and this is what earns
this idea three stars. For example, suppose we have an async function
that reads a set of bookmarks, and we want to write an async function
that calls this and then sorts the bookmarks. Instead of inventing a
mechanism whereby one async function can wait for another (which we
will do later anyway :-), the second async function can simply add a
new callback to the Deferred returned by the first one:
def read_bookmarks_sorted():
d = read_bookmarks()
d.addCallback(sorted)
return d
The Deferred returned by this function represents a sorted list of
bookmarks. If its caller wants to print those bookmarks, it must add
another callback:
d = read_bookmarks_sorted()
d.addCallback(print)
In a world where async results are represented by Futures, this same
example would require two separate Futures: one returned by
read_bookmarks() representing the unsorted list, and a separate Future
returned by read_bookmarks_sorted() representing the sorted list.
There is one non-obvious bug in this version of the class: if
addCallback() is called after the Deferred has already fired (i.e. its
callback() method was called) then the callback added by addCallback()
will never be called. It's easy enough to fix this, but tedious, and
you can look it up in the Twisted source code. I'll carry this bug
through successive examples -- just pretend that you live in a world
where the result is never ready too soon. There are other problems
with this design too, but I'd rather call the solutions improvements
than bugfixes.
Aside: Twisted's poor choices of terminology
I don't know why, but, starting with the project's own name, Twisted
often rubs me the wrong way with its choice of names for things. For
example, I really like the guideline that class names should be nouns.
But 'Deferred' is an adjective, and not just any adjective, it's a
verb's past participle (and an overly long one at that :-). And why is
it in a module named twisted.internet?
Then there is 'callback', which is used for two related but distinct
purposes: it is the preferred term used for a function that will be
called when a result is ready, but it is also the name of the method
you call to "fire" the Deferred, i.e. set the (initial) result.
Don't get me started on the neologism/portmanteau that is 'errback',
which leads us to...
Idea 3: Integrated error handling
This idea gets only two stars (which I'm sure will disappoint many
Twisted fans) because it confused me a lot. I've also noted that the
Twisted docs have some trouble explaining how it works -- In this case
particularly I found that reading the code was more helpful than the
docs.
The basic idea is simple enough: what if the promise of firing the
Deferred with a result can't be fulfilled? When we write
d = pod_bay_doors.open()
d.addCallback(lambda _: pod.launch())
how is HAL 9000 supposed to say "I'm sorry, Dave. I'm afraid I can't
do that" ?
And even if we don't care for that answer, what should we do if one of
the callbacks raises an exception?
Twisted's solution is to bifurcate each callback into a callback and
an 'errback'. But that's not all -- in order to deal with exceptions
raised by callbacks, it also introduces a new class, 'Failure'. I'd
actually like to introduce the latter first, without introducing
errbacks:
class Failure:
def __init__(self):
self.exception = sys.exc_info()[1]
(By the way, great class name. And I mean this, I'm not being
sarcastic.)
Now we can rewrite the callback() method as follows:
def callback(self, result):
for cb in self.callbacks:
try:
result = cb(result)
except:
result = Failure()
This in itself I'd give two stars; the callback can use
isinstance(result, Failure) to tell regular results apart from
failures.
By the way, in Python 3 it might be possible to do away with the
separate Failure class encapsulating exceptions, and just use the
built-in BaseException class. From reading the comments in the code,
Twisted's Failure class mostly exists so that it can hold all the
information returned by sys.exc_info(), i.e. exception class/type,
exception instance, and traceback but in Python 3, exception objects
already hold a reference to the traceback.There is some debug stuff
that Twisted's Failure class does which standard exceptions don't, but
still, I think most reasons for introducing a separate class have been
addressed.
But let's not forget about the errbacks. We change the list of
callbacks to a list of pairs of callback functions, and we rewrite the
callback() method again, as follows:
def callback(self, result):
for (cb, eb) in self.callbacks:
if isinstance(result, Failure):
cb = eb # Use errback
try:
result = cb(result)
except:
result = Failure()
For convenience we also add an errback() method:
def errback(self, fail=None):
if fail is None:
fail = Failure()
self.callback(fail)
(The real errback() function has a few more special cases, it can be
called with either an exception or a Failure as argument, and the
Failure class takes an optional exception argument to prevent it from
using sys.exc_info(). But none of that is essential and it makes the
code snippets more complicated.)
In order to ensure that self.callbacks is a list of pairs we must also
update addCallback() (it still doesn't work right when called after
the Deferred has fired):
def addCallback(self, callback, errback=None):
if errback is None:
errback = lambda r: r
self.callbacks.append((callback, errback))
If this is called with just a callback function, the errback will be a
dummy that passes the result (i.e. a Failure instance) through
unchanged. This preserves the error condition for a subsequent error
handler. To make it easy to add an error handler without also handling
a regular resullt, we add addErrback(), as follows:
def addErrback(self, errback):
self.addCallback(lambda r: r, errback)
Here, the callback half of the pair will pass the (non-Failure) result
through unchanged to the next callback.
If you want the full motivation, read Twisted's Introduction to
Deferreds; I'll just end by noting that an errback and substitute a
regular result for a Failure just by returning a non-Failure value
(including None).
Before I move on to the next idea, let me point out that there are
more niceties in the real Deferred class. For example, you can specify
additional arguments to be passed to the callback and errback. But in
a pinch you can do this with lambdas, so I'm leaving it out, because
the extra code for doing the administration doesn't elucidate the
basic ideas.
Idea 4: Chaining Deferreds
This is a five-star idea! Sometimes it really is necessary for a
callback to wait for an additional async event before it can produce
the desired result. For example, suppose we have two basic async
operations, read_bookmarks() and sync_bookmarks(), and we want a
combined operation. If this was synchronous code, we could write:
def sync_and_read_bookmarks():
sync_bookmarks()
return read_bookmarks()
But how do we write this if all operations return Deferreds? With the
idea of chaining, we can do it as follows:
def sync_and_read_bookmarks():
d = sync_bookmarks()
d.addCallback(lambda unused_result: read_bookmarks())
return d
The lambda is needed because all callbacks are called with a result
value, but read_bookmarks() takes no arguments.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I am thinking could it be a good approach to always return promises in functions in JavaScript?
Let's imagine the case where we have a function that validates a username. A master function simply utilises 2 other functions that perform different checks.
Please note, all functions names are just examples.
// Returns a boolean
function validateUsername (username) {
return validateUsernameFormat(username) &&
isUsernameReserved(username);
}
// Returns a boolean
function validateUsernameFormat (username) {
return typeOf(username) === 'string' &&
username.match(/^\[a-z0-9]{8,20}$/);
}
// Returns a boolean
function isUsernameNotReserved (username) {
return ['igor', 'kristina'].indexOf(username) === -1;
}
Now let's imagine we augment our validation in the future by calling API to check if a given username already exists in our database.
// Now returns a promise
function isUsernameNotReserved (username) {
return API.checkIfUserNameAlreadyExists(username);
}
This would mean we will also now have to change the master validateUsername function since it now also needs to return promise. This would also probably mean we will have to modify all functions that use validateUsername function.
But what If we had all function in promises from scratch?
Option A - All functions return promises
// Returns a promise
function validateUsername (username) {
return validateUsernameFormat(username)
.then(() => {
return isUsernameReserved(username);
});
}
// Returns a promise
function validateUsernameFormat (username) {
return (
typeOf(username) === 'string' && username.match(/^\[a-z0-9]{8,20}$/) ?
Promise.resolve() : Promise.reject()
);
}
// Returns a promise
function isUsernameNotReserved (username) {
return (
['igor', 'kristina'].indexOf(username) === -1 ?
Promise.resolve() : Promise.reject()
);
}
Now if we want to augment isUsernameNotReserved with asynchronous API call, we don't need to change anything else.
Option B - Only functions calling another functions return promises
Also, another option would be write functions in promises that call another functions. In that case, only validateUsername should be written as a promise from scratch.
Is this a good approach? What could be the drawbacks apart from performance?
Upd: ran a simple performance test and though running consequent promises is slower, it practically should not make any difference since running 100000 consequent functions takes ~200ms, while running 1000 takes ~3ms in Chrome. Fiddle here https://jsfiddle.net/igorpavlov/o7nb71np/2/
Should I always return promises in all functions in JavaScript?
No.
If you have a function that performs an asynchronous operation or may perform an asynchronous operation, then it is reasonable and generally good design to return a promise from that function.
But, if your function is entirely synchronous and no reasonable amount of forethought thinks this will sometime soon contain an asynchronous operation, then there are a bunch of reason why you should not return a promise from it:
Asynchronous code writing and testing is more complicated than synchronous code writing and testing. So, you really don't want to make code harder to write and test than it needs to be. If your code can be synchronous (and just return a normal value), then you should do so.
Every .then() handler gets called on the next tick (guaranteed asynchronously) so if you take a whole series of synchronous operations and force each function to wait until the next tick of the event loop, you're slowing down code execution. In addition, you're adding to the work of the garbage collector for every single function call (since there's now a promise object associated with every single function call).
Losing the ability to return a normal value from a synchronous function is a huge step backwards in the language tools that you can conveniently use to write normal code. You really don't want to give that up on every single function.
Now if we want to augment isUsernameNotReserved with asynchronous API call, we don't need to change anything else.
A good API design would anticipate whether an asynchronous API is relevant or likely useful in the near future and make the API async only in that case. Because asynchronous APIs are more work to write, use and test, you don't want to unnecessarily just make everything async "just in case". But, it would be wise to anticipate if you are likely to want to use an async operation in your API in the future or not. You just don't want to go overboard here. Remember, APIs can be added to or extended in the future to support new async things that come along so you don't have to over complicate things in an effort to pre-anticipate everything that ever might happen in the future. You want to draw a balance.
"Balance" seems like the right word here. You want to balance the likely future needs of your API with developer simplicity. Making everything return a promise does not draw a proper balance, as it chooses infinite flexibility in the future while over complicating things that don't need to be as complicated.
In looking at a couple of your particular examples:
validateUsernameFormat() does not seem likely to ever need to be asynchronous so I see no reason for it to return a promise.
isUsernameNotReserved() does seem like it may need to be asynchronous at some point since you may need to look some data up in a database in order to determine if the user name is available.
Though, I might point out that checking if a user name is available before trying to just create it can create a race condition. This type of check may still be used as UI feedback, but when actually creating the name, you generally need to create it in an atomic way such that two requests looking for the same name can't collide. Usually this is done by creating the name in the database with settings that will cause it to succeed if the name does not already exist, but fail if it does. Then, the burden is on the database (where it belongs) to handle concurrency issues for two requests both trying to create the same user name.
var identifier = '0';
var serviceCodes = parseServiceCodes(identifier);
console.log(serviceCodes);
function parseServiceCodes(id){
var serviceCodes = id + 'HYJSIXNS';
return serviceCodes
}
0HYJSIXNS is returned in the console. But I thought that since JavaScript is asynchronous, the variable would be assigned before the function is returned, making it null.
Javascript isn't asynchronous by default, and code is executed in order unless you specify that it shouldn't be. You're never making a call to an external resource, so there's nothing to be async here.
JS, like any interpreted language, is run line-by-line. In your example, at the parseServiceCodes callsite, execution goes from line 2 (the callsite) to line 6 (the function body).
Async behavior is introduced for time consuming operations, like HTTP requests. In this case, the expensive function either takes a callback parameter (code to run once the expensive operation is over) or returns a Promise which "resolves" with the computed data when ready.
To illustrate the first case, let's say we have a function getFromServer which needs a URI and a callback:
function getFromServer(uri, callback){ // ...
It doesn't matter exactly what happens under the hood, just that execution is given back to the main program after invocation. This means the program isn't blocked for the duration of this expensive operation. When getFromServer is done, it will come back and call callback with some data.
To learn more about the second case, Promises, I encourage you to read the documentation I linked to earlier in this response.
You seem to misunderstand what it means to say that "JavaScript is asynchronous".
JavaScript is asynchronous because it supports and benefits from usage of asynchronicity, which is considered a common design pattern throughout the language by using features such as asynchronous callbacks, promises and even async/await, but this does not mean that all code is asynchronous.
You can easily write synchronous code, as you just demonstrated in your question, and there is no issue specifying a return value that is not a Promise.
This is because the code invoking it expects the return value to contain a string, which it then assigns to serviceCodes and uses that in the same tick.
If any of these terms above confuse you, please refer to their respective documentation and usage I've provided the links for.
I have a series of questions I want to ask the user, with later questions depending on earlier ones. These are expressed most concisely by pretending we can have synchronous i/o, like this:
//'queryuser' is a callback which provides synchronous user interaction
function seriesOfQuestions(queryuser)
var drink = queryuser("What would you like to drink?",["Tea","Coffee","Orange Juice"]);
if (drink=="Tea" || drink=="Coffee")
{
var milk = queryuser("Would you like milk?",["Yes","No"]);
var sugar = queryuser("Would you like sugar?",["Yes","No"]);
if (milk=="Yes")
var milk_type = queryuser("Normal or soya?",["Moo juice","Hippy beans"]);
}
queryuser("Would you like a biscuit?",["Yes","No"]);
//...etc... deep decision tree, though note code paths can re-join for biscuits
}
Javascript, of course, doesn't work like this; functions such as $ionicPopup.show() do not block execution and instead themselves call a callback (via a promise mechanism) when the user selects an option. So if we use ionicPopup.show for queryuser we'll get all the popups at once, and the decision tree will be executed on null return values.
Is there a way we can wrap ionicPopup.show() or similar in a function that behaves like we want queryuser to behave?
(Preferably less obtuse than Duff's device for writing C co-routines...)
Or failing that, how can a decision tree like the above be expressed without ending deep in callback hell?
You are looking for ES8 async/await if your queries are returning promises.
But since you were asking about coroutines, you can also use generator functions and drive them asynchronously by a query framework. Especially CSP allows a nice abstraction of an asynchronous user interface.
In Draft ECMAScript 6 Specification, what's the rationale behind using StopIteration exception to signal the end of iteration instead of using a dedicated method (hasNext in Java/Scala, and MoveNext in C#) to check for the end of iteration.
Potential performance problem aside, in my opinion, exception should not be used for something that's not really an exception.
I don't claim to be authoritative with this answer. Instead, it is just my recollection of various discussions I read about iterators and/or my own thoughts... Unfortunately I do not have a list of sources (too long ago) nor can I easily produce one (because even googling up stuff takes lots of time sometimes).
StopIteration
The name StopIteration and a lot of the semantics originated from python.
PEP 234 has some remarks regarding alternative approaches:
It has been questioned whether an exception to signal the end of
the iteration isn't too expensive. Several alternatives for the
StopIteration exception have been proposed: a special value End
to signal the end, a function end() to test whether the iterator
is finished, even reusing the IndexError exception.
A special value has the problem that if a sequence ever
contains that special value, a loop over that sequence will
end prematurely without any warning. If the experience with
null-terminated C strings hasn't taught us the problems this
can cause, imagine the trouble a Python introspection tool
would have iterating over a list of all built-in names,
assuming that the special End value was a built-in name!
Calling an end() function would require two calls per
iteration. Two calls is much more expensive than one call
plus a test for an exception. Especially the time-critical
for loop can test very cheaply for an exception.
Reusing IndexError can cause confusion because it can be a
genuine error, which would be masked by ending the loop
prematurely.
hasNext()
Java's hasNext() is just a "bonus". Java's next(), however, has it's own flavor of StopIteration called NoSuchElementException.
hasNext() is hard, if not impossible, to implement, which is best demonstrated with e.g. the following generator:
var i = 0;
function gen() {
yield doSomethingDestructive();
if (!i) {
yield doSomethingElse();
}
}
How would you implement hasNext() for that? You cannot simply "peek" into the generator, because that would actually execute doSomethingDestructive(), which you didn't intend to do or else you would have called next() in the first place.
So you'd have to write some code analyzer that has to always reliably proof that a given code when run with a certain state will always result in either a yield or no yield (halting problem). So even if you could write something like that, the state may still change between .hasNext() and a subsequent .next().
What would hasNext() return in the following example:
var g = gen();
g.next();
if (g.hasNext()) {
++i;
alert(g.next());
}
At that point of time true would be the right answer. However, the subsequent .next() would throw, and the user would probably have a hard time figuring out why...
You could tell people: "Don't do the following things in your iterator implementation" (a contract). Which people will routinely break and complain.
So in my humble opinion, hasNext() was ill-conceived and makes it too easy to write wrong code.
.MoveNext()
C# is pretty much the same as .next(), just that the return value indicates if there was actually a next item instead of or the lack of an exception.
There is one important difference, however: You need to store the current item in the iterator itself, e.g. .Current in C#, which may prolong the life-time of the current item unnecessarily:
var bufferCtor = (function gen() {
for(;;) {
yield new ArrayBuffer(1<<20); // buffer of size 1 megabyte
}
})();
setInterval(function() {
// MoveNext scheme
bufferCtor.MoveNext(); // Not actually Javascript
var buf = bufferCtor.Current; // Not actually Javascript
// vs. StopIteration scheme
var buf = bufferCtor.next();
}, 60000); // Each minute
OK, that example is a little contrived, but I'm sure there are actual use cases where a generator/iterator would return something talking up a lot of space or otherwise occupying resources.
In the StopIteration scheme, buf can be garbage-collected as soon as it goes out of scope.
In .MoveNext(), it cannot be garbage-collected until .next() is called again, because .Current will still hold a reference to it.
Conclusion
In my opinion, of the presented alternatives, the StopIteration one is the least error prone and least ambiguous approach. And if I recall correctly, the python and es-6 people thought about the same.
A number of times, I have to write a callback function.
Example 1:
myTransaction.executeSql(stmt,parameters,mySuccess,myError);
Example 2: $.getScript(url [, mySuccess])
Q: Is there a way to tell JavaScript, "Please execute this line of code, and when you're done, execute the next line of code"?
I don't want to lock up the browser session, I just want it to not run the next line until it's done.
Put the call to the second function in the mySuccess handler function for the first function call. When the first one completes and runs the mySuccess function, that function will then call the second one. That is the design pattern for sequencing asynchronous operations.
You cannot write normal sequential function calls with asynchronous operations. You must use the completion functions to do your sequencing.
Here's an example using an anonymous success handler for the first function:
myTransaction.executeSql(stmt,parameters,function() {
$.getScript(url, mySuccess);
}, myError);
No there is not a way to exactly what you want. But you can use a library that helps make this less hard, like Step: https://github.com/creationix/step
Step(
function() { myTransaction.executeSql(stmt, parameters, this, myError) },
function() { $.getScript(url, this); },
function() { alert('done'); }
);
Or you will need to manually nest the callbacks.
The Javascript library Step provides a framework that encapsulates the solution suggested by #jfriend00.
You can put step 2's call in step 1's success/complete callback, as already mentioned.
Or you can look into the promise/deferred paradigm. It's a little complicated, so I won't try and explain it concisely here (and get it wrong). But given that you already seem to be using jQuery you can probably use its implementation of the deferred object. Here are a couple of articles you can read (though there are plenty more that you'll find with Google):
http://blogs.msdn.com/b/ie/archive/2011/09/11/asynchronous-programming-in-javascript-with-promises.aspx
http://joseoncode.com/2011/09/26/a-walkthrough-jquery-deferred-and-promise/
http://www.infoq.com/articles/surviving-asynchronous-programming-in-javascript
Frame.js is another library, similar to Step, that solves this problem eloquently and provides a host of advantages over setTimeouts or nested callbacks.