Inconsistent interplay between IndexedDB transactions and Promises - javascript

I saw sync-promise posted on Reddit and got into a discussion with the author. We noticed some weird inconsistencies in the relationship between IndexedDB transactions and promises.
IndexedDB transactions auto-commit when all the onsuccess events finish. One complication is that you can't do anything asynchronous inside an onsuccess callback except do another operation on the same transaction. For example, you can't start an AJAX request in an onsuccess and then reuse the same transaction after the AJAX request returns some data.
What do promises have to do with it? As I understand it, promise resolution is supposed to always be asynchronous. This would imply that you can't use promises without auto-committing an IndexedDB transaction.
Here is an example of what I'm talking about:
var openRequest = indexedDB.open("library");
openRequest.onupgradeneeded = function() {
// The database did not previously exist, so create object stores and indexes.
var db = openRequest.result;
var store = db.createObjectStore("books", {keyPath: "isbn"});
var titleIndex = store.createIndex("by_title", "title", {unique: true});
var authorIndex = store.createIndex("by_author", "author");
// Populate with initial data.
store.put({title: "Quarry Memories", author: "Fred", isbn: 123456});
store.put({title: "Water Buffaloes", author: "Fred", isbn: 234567});
store.put({title: "Bedrock Nights", author: "Barney", isbn: 345678});
};
function getByTitle(tx, title) {
return new Promise(function(resolve, reject) {
var store = tx.objectStore("books");
var index = store.index("by_title");
var request = index.get("Bedrock Nights");
request.onsuccess = function() {
var matching = request.result;
if (matching !== undefined) {
// A match was found.
resolve(matching);
} else {
// No match was found.
console.log('no match found');
}
};
});
}
openRequest.onsuccess = function() {
var db = openRequest.result;
var tx = db.transaction("books", "readonly");
getByTitle(tx, "Bedrock Nights").then(function(book) {
console.log('First book', book.isbn, book.title, book.author);
return getByTitle(tx, "Quarry Memories");
}).then(function(book) {
console.log('Second book', book.isbn, book.title, book.author);
// With native promises this gives the error:
// InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable
// With bluebird everything is fine
});
};
(Full disclosure: demo was created by paldepind, not me!)
I've tried it in Chrome and Firefox. It fails in Firefox due to the transaction auto-committing, but it actually works in Chrome! Which behavior is correct? And if Firefox's behavior is correct, is it literally impossible to use "correct" promise implementations with IndexedDB transactions?
Another complication: If I load bluebird before running the above demo, it works in both Chrome and Firefox. Does this imply that bluebird is resolving promises synchronously? I thought it wasn't supposed to do that!
JSFiddle

This is probably due to the difference between microtasks and tasks ("macrotasks"). Firefox has never had a standards-complaint promise implementation that uses microtasks, whereas Chrome, Bluebird, and others correctly use microtasks. You can see this in how a microtask (which executes "sooner" than a macrotask, but still async) falls inside the transaction boundary, whereas a macrotask (e.g. from Firefox's promises) does not.
So, this is a Firefox bug.

Ok, so I've once again taken a deep dive into the IndexedDB, the DOM and the HTML specification. I really need to get this right for SyncedDB since it relies heavily on promises inside transactions.
The crux of the problem is whether or not the delayed execution of the onFulfilled and the onRejected callbacks to then that Promises/A+ compliant must exhibit will trigger an IndexedDB transaction commit.
The IndexedDB rules for a transactions lifetime are actually pretty straight forward when you extract them from the specification and line them up:
Request can only made against a transaction when its active flag is set to true (as specified here).
When a transaction is created it is initially active until control is returned to the the browsers event loop (this is specified in the transaction creation steps).
Every time a success or error event is fired, the transactions active flag is set to true before as the last step before the event is dispatched. After the event dispatch the transaction is flagged as inactive again (this is specified in the steps for firing a success/error event.
When a transaction can no longer become active it will automatically be committed (as specified here).
This roughly translates to:
When you create a transaction you can place as many request against it as you wish.
From then on new request can only be made inside event handlers for another requests success or error event listener.
When all requests has been executed and no new requests placed the transaction will commit.
The question then becomes: If a promise is fulfilled inside a request's success or error event listener will its onFulfilled callbacks be invoked before the IndexedDB sets the transaction as inactive again? I.e. will onFullfilled callbacks be called as part of step 3 in firing a success event?
The step dispatches an event and IndexedDB uses DOM events so the actual operation performed is beyond the IndexedDB specification. The steps for dispatching an event is, instead, specified here in the DOM specification. Going over the steps it becomes clear that at no point is a microtask (which would call the promise callbacks) checkpoint performed. So the initial conclusion is that the transaction will be closed before any onFulfilled callbacks will be invoked.
However, if we attach the event listeners by specifying an onsuccess attribute on the request object things gets more hairy. In that case we are not simply adding an event listener as per the DOM specification. We are instead setting an event handler IDL attribute as defined in the HTML specification.
When we do that the callback is not added directly to the list of event listeners. It is instead "wrapped" inside the the event handlers processing algorithm. This algorithm performs the following important operations:
In step 3 it runs the jump to code entry-point algorithm.
This then performs the steps to clean up after running a callback which
Finally, this performs a microtask checkpoint. Which means that your promise callbacks will be invoked before the transaction is marked as inactive! Hurrah!
This is good news! But it is weird how the answer depends on whether or not you listen for the success event by using addEventListener or set a onsuccess event handler. If you do the former the transaction should be inactive when your promise's onFulfilled callbacks is invoked and if you do the later it should still be active.
I was, however not able to reproduce the difference in existing browsers. With native promises Firefox fails at the example code no matter what and Chrome succeeds even when using addEventListener. It is possible that I've overlooked or misunderstood something in the specifications.
As a final note Bluebird promises will close transactions in Internet Explorer 11. This is due to the scheduling that Bluebird uses in IE. My synchronized promise implementation works inside transactions in IE.

You are correct: Promises are resolved asynchronously, and IndexedDB has some synchronous requirements. While other answers point out that native promises may work correctly with IndexedDB in certain versions of certain browsers, it is likely as a practical matter that you will have to deal with the issue of it not working in some of the browsers you're targeting.
Using a synchronous promise implementation instead, however, is a horrible idea. Promises are asynchronous for very good reasons, and you are introducing needless chaos and potential for bugs if you make them synchronous instead.
There is a fairly straightforward workaround, however: use a Promise library that provides a way to explicitly flush its callback queue, and an IndexedDB wrapper that flushes the promise callback queue after invoking event callbacks.
From the Promises/A+ point of view, there isn't any difference between the handlers being called at the end of the event, or at the beginning of the next tick cycle -- they are still being called after all the code that set up the callbacks has finished, which is the important part of Promise asynchrony.
This allows you to use promises that are asynchronous, in the sense of meeting all the Promises/A+ guarantees, but which still ensure that the IndexedDB transaction isn't closed. So you still get all the benefits of callbacks not happening "all at once".
The catch of course is that you need libraries that support this, and not every Promise implementation exposes a way to specify a scheduler or to flush its callback queue. Likewise, I'm not aware of any open source IndexedDB wrappers that have support for this.
If you are writing your own IndexedDB wrapper with Promsies, though, it would be good for it to use an appropriate Promise implementation, and flush its callback queue accordingly. One easy option would be to embed one of the many "micropromise" implementations that are only 100 lines or so of Javascript, and modify it as needed. Alternately, using one of the larger mainstream Promise libraries with custom scheduling support would be doable.
Do not use a synchronous promise library, the synchronous Bluebird build, or a synchronous scheduler. If you do that, you might as well abandon promises altogether and use straight callbacks.
Follow-up note: one commenter suggests that a synchronous promise is as safe as flushing a callback queue. But they are wrong. Horribly, horribly wrong. You can reason about a single event handler well enough to say "there isn't any other code running here; it's okay to invoke the callbacks now". To make a similar analysis with synchronous promises requires a complete understanding of how everything calls everything else... which is precisely opposite from the reason you want promises in the first place.
In the specific sync-promise implementation, the sync-promise author claims that their promise library is now "safe", and doesn't "release Zalgo". They are once again wrong: it isn't safe, and does release Zalgo. The author apparently did not actually understand the article about "releasing Zalgo", and has successfully reimplemented jQuery promises, which are widely considered horribly broken for a number of reasons, including their Zalgo-ness.
Synchronous promises are simply not safe, no matter your implementation.

Related

May this long polling code cause stackoverflow? [duplicate]

For example I found some api library that is based on promises, and I need to issue api requests using this library in some interval, infinite times (like usual back-end loop). This api requests - actually chain of promises.
So, if I write function like:
function r(){
return api
.call(api.anotherCall)
.then(api.anotherCall)
.then(api.anotherCall)
...
.then(r)
}
Will it cause stack overflow?
Solutions that I come up with is to use setTimeout for a call of r recursively.
function r(){
return api
.call(api.anotherCall)
.then(api.anotherCall)
.then(api.anotherCall)
.then(()=>{setTimeout(r, 0)})
}
So setTimeout will call r actually only when call stack is empty.
Is it good solution, or there is some standard way of calling promises recursively?
Will this cause stackoverflow?
No, it will not. Per the promise specification, .then() waits for the stack to completely unwind and is then called after the stack is clear (essentially on the next tick of the event loop). So, .then() is already called asynchronously after the current event is done processing and the stack is unwound. You do not have to use setTimeout() to avoid stack build-up.
Your first code example will not have any stack build-up or stack overflow, no matter how many times you repeat it.
In the Promises/A+ specification, section 2.2.4 says this:
2.2.4 onFulfilled or onRejected must not be called until the execution context stack contains only platform code. [3.1].
And, "platform code" is defined here in 3.1:
“platform code” means engine, environment, and promise implementation code. In practice, this requirement ensures that onFulfilled and onRejected execute asynchronously, after the event loop turn in which then is called, and with a fresh stack. This can be implemented with either a “macro-task” mechanism such as setTimeout or setImmediate, or with a “micro-task” mechanism such as MutationObserver or process.nextTick. Since the promise implementation is considered platform code, it may itself contain a task-scheduling queue or “trampoline” in which the handlers are called.
The ES6 promise specification uses different words, but generates the same effect. In ES6, promise .then() is performed by enqueing a job and then letting that job get processed and the job only gets processed when no other code is running and the stack is empty.
This is how running such as job is described in the ES6 spec:
A Job is an abstract operation that initiates an ECMAScript computation when no other ECMAScript computation is currently in progress. A Job abstract operation may be defined to accept an arbitrary set of job parameters.
Execution of a Job can be initiated only when there is no running execution context and the execution context stack is empty. A PendingJob is a request for the future execution of a Job. A PendingJob is an internal Record whose fields are specified in Table 25. Once execution of a Job is initiated, the Job always executes to completion. No other Job may be initiated until the currently running Job completes. However, the currently running Job or external events may cause the enqueuing of additional PendingJobs that may be initiated sometime after completion of the currently running Job.

Recursive promises can cause stack overflow?

For example I found some api library that is based on promises, and I need to issue api requests using this library in some interval, infinite times (like usual back-end loop). This api requests - actually chain of promises.
So, if I write function like:
function r(){
return api
.call(api.anotherCall)
.then(api.anotherCall)
.then(api.anotherCall)
...
.then(r)
}
Will it cause stack overflow?
Solutions that I come up with is to use setTimeout for a call of r recursively.
function r(){
return api
.call(api.anotherCall)
.then(api.anotherCall)
.then(api.anotherCall)
.then(()=>{setTimeout(r, 0)})
}
So setTimeout will call r actually only when call stack is empty.
Is it good solution, or there is some standard way of calling promises recursively?
Will this cause stackoverflow?
No, it will not. Per the promise specification, .then() waits for the stack to completely unwind and is then called after the stack is clear (essentially on the next tick of the event loop). So, .then() is already called asynchronously after the current event is done processing and the stack is unwound. You do not have to use setTimeout() to avoid stack build-up.
Your first code example will not have any stack build-up or stack overflow, no matter how many times you repeat it.
In the Promises/A+ specification, section 2.2.4 says this:
2.2.4 onFulfilled or onRejected must not be called until the execution context stack contains only platform code. [3.1].
And, "platform code" is defined here in 3.1:
“platform code” means engine, environment, and promise implementation code. In practice, this requirement ensures that onFulfilled and onRejected execute asynchronously, after the event loop turn in which then is called, and with a fresh stack. This can be implemented with either a “macro-task” mechanism such as setTimeout or setImmediate, or with a “micro-task” mechanism such as MutationObserver or process.nextTick. Since the promise implementation is considered platform code, it may itself contain a task-scheduling queue or “trampoline” in which the handlers are called.
The ES6 promise specification uses different words, but generates the same effect. In ES6, promise .then() is performed by enqueing a job and then letting that job get processed and the job only gets processed when no other code is running and the stack is empty.
This is how running such as job is described in the ES6 spec:
A Job is an abstract operation that initiates an ECMAScript computation when no other ECMAScript computation is currently in progress. A Job abstract operation may be defined to accept an arbitrary set of job parameters.
Execution of a Job can be initiated only when there is no running execution context and the execution context stack is empty. A PendingJob is a request for the future execution of a Job. A PendingJob is an internal Record whose fields are specified in Table 25. Once execution of a Job is initiated, the Job always executes to completion. No other Job may be initiated until the currently running Job completes. However, the currently running Job or external events may cause the enqueuing of additional PendingJobs that may be initiated sometime after completion of the currently running Job.

IndexedDb transaction auto-commit behavior in edge cases

Tx is committed when :
request success callback returns
- that means that multiple requests can be executed within transaction boundaries only when next request is executed from success callback of the previous one
when your task returns to event loop
It means that if no requests are submitted to it, it is not committed until it returns to event loop. These facts pose 2 problematic states :
placing a new IDB request by enqueuing a new task to event loop queue from within the success callback of previous request instead of submitting new request synchronously
in that case the first success callback immediately returns but another IDB request has been scheduled
are all the asynchronous requests executed within the single initial transaction? This is quite essential in case you want to implement result pulling with back-pressure where consumer gives you a feedback in form of a Future that it is ready to consume another response
creating a ReadWrite tx, not placing any requests against it and creating another one before returning to event loop
does creating a new one implicitly commits the previous tx ? If not, serious write lock starvations might occur, because :
If multiple "readwrite" transactions are attempting to access the same
object store (i.e. if they have overlapping scope), the transaction
that was created first must be the transaction which gets access to
the object store first. Due to the requirements in the previous
paragraph, this also means that it is the only transaction which has
access to the object store until the transaction is finished.
The example of enqueuing a new task to event loop queue from within the success callback by recursive request submission with back-pressure :
function recursiveFn(key) {
val req = store.get(key)
req.onsuccess = function() {
observer.onNext(req.result).onsuccess { recursiveFn(nextKey) }
}
}
Observer#onNext // returns Future[Ack] Ack is either Continue or Cancel
Now can onsuccess or onNext do a setTimeout(0) or not to make the whole thing be part of one transaction?
Bonus question :
I think that ReadOnly transactions are exposed to the consumer/user just because it would be hard to detect the end of a batch read if you recursively submit new requests from the success callback of the previous one right? Otherwise I don't see any other reason for them to be exposed to a user, right ?
I'm not sure I understand your question completely but I'll offer an answer on whether you can safely use IDB transaction events to move a state machine.
Yes and no. Yes in theory, no in practice.
I think you understand the transaction lifetime. But to rehash:
The lifetime of a transactions lasts as long as it's referenced: it's "active" so long as it's being referenced, after which it is said to be "finished" and the transaction is committed.
In theory, oncomplete should fire whenever a transaction successfully commits. There's a useful tip in the spec on this that suggests how you could loop:
To determine if a transaction has completed successfully, listen to the transaction’s complete event rather than the IDBObjectStore.add request’s success event, because the transaction may still fail after the success event fires.
To safely use this mechanism be sure to watch for non-success events including onblocked and onabort as well.
Practically speaking, I've found transactions to be flakey when long-lived or done consecutively in batches (as you've noted in another IDB comment). I'm generally not engineering my apps to require tricky behavior because, no matter how the spec says it should behavior, I'm seeing wonky transactions in both Firefox and Chromium (but mostly Blink, interestingly) especially when multiple tabs are open.
I spent many weeks rewriting dash to reuse transactions for supposed performance gains. In the end it could not pass even my basic write tests and I was forced to abandon simultaneous/queued/consecutive transactions and rewrite once again. This time I picked a one-transaction-at-a-time model which is slower but, for me, more reliable (and suggest to avoid my lib and use something like ydn for bulk inserts).
I'm not sure on your application requirements, but in my humble opinion tying in your I/O into your event loop seems like a disastrous idea. If I needed an event loop as what I understand to be the term I would definitely use requestAnimationFrame() and throttle that callback if I needed fewer ticks than one per ~33 milliseconds.

Javascript Promises library to make "long-running-code-non-blocking-UI" in browser?

Update
this an update to the question below and should help finding an answer
Taking up the answer from torazaburo who also quoted part of the prominent Javascript Promise/A+ definition I want to update the question here.
The Promise/A+ specification suggest in point 2.2.4 this:
onFulfilled or onRejected must not be called until the execution
context stack contains only platform code. 3.1.
and further explains
Here “platform code” means engine, environment, and promise
implementation code. In practice, this requirement ensures that
onFulfilled and onRejected execute asynchronously, after the event
loop turn in which then is called, and with a fresh stack. This can be
implemented with either a “macro-task” mechanism such as setTimeout or
setImmediate, or with a “micro-task” mechanism such as
MutationObserver or process.nextTick. Since the promise implementation
is considered platform code, it may itself contain a task-scheduling
queue or “trampoline” in which the handlers are called.
The very issue I look forward to find with this question is having as the crucial point that Promise implementation Javascript code is itself considered platform code and allows to not yield to the eventloop inbetween resolving subsequent promise via calling the onFulfilled onRejected functions associated. This is good in Node.js(server) as it avoids unnessary relinguishing back to the event-loop (leaving the execution stack), but also causes the challange in a Browser that since the execution stack is not exited in between resolving a potentially large number of Promises (which themselves can generate new Promises). Not leaving the execution stack and yielding to the event loop is causing in a Browser the undesired (blocking script warning/problem).
The "trampoline" task-scheduling of the Promise implementation which causes this needs however not necessary refrain from handing back the execution to the Javascript event loop from time to time. Such a feature would allow for using Promises for heavier tasks. Such an implementation for Promises for "long-running-code" is searched/asked for in this question.
Clarification: The "excessive lenght" is not the individual length of the onFulfilled function, but the joining together several those functions/callbacks as result of the Promise resolving process (when done in such a "trampoline" way). I am already aware that if one individual onFulfilled funciton is too long, this cannot be helped in any way by using any sort of Promise implementation.
The deal here is that the subsequent resolvement of x promises (within one excecution stack and hence without handing back to the Javascript event loop) can provoke an excessive length duration of Javascript code execution. This, when in a Browser is bad (because of blocking).
The question
In Javascript, Promises allow to deal with asynchronous programming tasks. Great!
There are already some implementations and libraries arround Q, WinJS or when.js to name just a few.
Having looked at then I see that they tackle some of the "special things" in Javascript asynchronous programming challanges.
Normally I perceive them to do this for promise resolution
Go to the internal list of promises
Check if the promise is fullfilled + run all the associated (via then(onFullfilled,onReject)) functions.
(in some cases we are done here)
(in other cases there will be still "pending" promises)
This case (4) is because to have them (the remaining promises) fullfilled would need the current Javascript Code (which is this very code for promise resolution) to stop running and allow JS event loop to happend (i.e asynchronous things like XHR-requests, or User-UI-interaction). To make this (4) work, the promise resolution normaly schedules a recall (i.e. via setTimeout/setImmediate) and continues after the event loop ran and hence maybe some of the "pending" promises have been settled (=rejected/fullfilled).
My worry is that the step 1 and 2 could be runnning for quite a some time, only releasing execution to the event loop in case it seems indicated to settle some of the "pending" promises. While "okay" in some cases (i.e. on the server/Node.js) it is quite problematic in a browers, because even though it was no problem to release execution to the event loop and have the UI not-blocking, this is not done in the implementations of promises I have seen.
My question therefore is:
Do you know a promise implementation (Javascript Promises library) that cares for the aspect:
to make "long-running-code-non-blocking-UI" in browser?
which would mean that the promise resolution would voluntarily release execution back to the event loop so that CSS animations, user input, mouse interaction, does get enough attention and that there will be no "Warning: Unresponsive script" message.
Any compliant promises implementation will not run the then functions synchronously, but rather only at the next tick. Therefore, your worry that "step 1 and 2 could be runnning for quite a some time" is unfounded.
From the Promises/A+ spec:
onFulfilled or onRejected must not be called until the execution context stack contains only platform code.
Here "platform code" means engine, environment, and promise implementation code. In practice, this requirement ensures that onFulfilled and onRejected execute asynchronously, after the event loop turn in which then is called, and with a fresh stack.
In other words, your formulation under 2) is incorrect. The promises implementation does not "run the associated functions", it schedules them.
This cannot help you--indeed there is no way to help you--if a handler itself is "long-running" code.
I think the solution could be to parse that long-running JavaScript code for example with https://github.com/NeilFraser/JS-Interpreter.
It will make the code be even slower, but you could specify the priority:
const myInterpreter = new Interpreter(myCode);
function nextStep() {
if (myInterpreter.step()) {
window.setTimeout(nextStep, 100/speed);
}
}
nextStep();

Similar pattern to promises/deferred's that support multiple results and cancellation

Similar to the promises pattern I'm looking for an event pattern that avoids needing to pollute objects with addEventListener/etc methods, I want to be able to return an object, that can be cancelled as well as 'resolved' multiple times.
For example, I may write a method that returns an 'interval' object, something like this:
var ticker = createTicker(1000);
var subscription = ticker.then(function() { console.log('tick') });
... later on ...
subscription.cancel();
The key differences here being, similar to a promise the events are standardized, so that I can subscribe without needing to know the event name, however unlike a promise, the "completion" can happen multiple times, and may even be cancelled (this would be equiv of removeEventListener).
I'm interested to see if this is legal with promises, such that the progress handler could be used for multiple callbacks, and the complete handler never used, but more importantly, that there is a concept of unsubscribing from a promise.
If this isn't the case, and promises are specialized to this scenario, is there a standardized pattern for doing what I described?
The ability to cancel can be added to a promise implementation, without breaking the main paradigm of having single-fire success/failure callbacks.
In fact, jQuery already has cancellation for the promise instances that it returns from jQuery.ajax calls:
For backward compatibility with XMLHttpRequest, a jqXHR object will
expose the following properties and methods:
readyState
status
statusText
responseXML and/or responseText when the underlying request responded with xml and/or text, respectively
setRequestHeader(name, value) which departs from the standard by replacing the old value with the new one rather than concatenating the new value to the old one
getAllResponseHeaders()
getResponseHeader()
abort()
You could write a setTimeout wrapper that exposes a promise interface along with an additional cancel method.
However, once you get into the multi-fire territory, I think that is not what promises are intended for. You would have to define a lot of rules and exceptions around how multiple firing will play out alongside regular promise functionality. It doesn't make a lot of sense to me to use promises that way.
Update (based on discussion in comments):
Here's a sample implementation of a promise "proxy" that allows aborting further relaying of the done/fail callbacks:
http://jsfiddle.net/atesgoral/qvtqu/

Categories