I mean, this may seem like a silly question, but I don't think it really is. In programming languages where we can use threads, the concept is very simple. But Javascript is single-threaded and handles events through callbacks. What really makes something "branch-out" the execution process (I know this is the wrong technical term but I hope you understand my point). Is it the timers? Like setTimeout and setInterval?
I mean, if I were to write a Database with JS (I know JS wouldn't fit it but please bear with me for the sake of the example) and I need to check for some hardware operations to be completed before calling a callback function to handle the completion of such operation I cannot for the sake of me think in a possible way to actually make this 'pseudo-code' async. I mean:
function op() {
while (someHardwareOpIsNotCompleted()) doWhatever();
callback("done");
}
How can I make this function run asynchronously without using some form of timer that refers and 'counts' to a different thread? In my mind the only way around it is to pass to setTimeout the function reference as such setTimeout(op). From here on I can implement my callbacks and listeners, or the syntax-sugar Promise or even the syntax sugar-sugar "async".
Hopefully I was able to get my point across and I don't get replies like: "Use promises!" or something of the sort. If the real structure has nothing to do with timers but only simple callbacks, can someone give me an example on how to achieve a non-blocking execution of some piece of code without timers?
Thanks
A function is asynchronous for one of two reasons.
It needs to happen later (e.g. when a timeout has finished or when a click happens)
It takes a long time and shouldn't tie up the JS engine while it runs (e.g. accessing a database / file / network resource)
The asynchronous parts of the latter functions are not written in JavaScript. They are provided by the host environment and are usually written in the same language a that (e.g. C). The host environment exposes an API to provide access to them to the JavaScript.
For example, the source code for Chrome's implementation of XMLHttpRequest is written in C++.
If you need to poll something, then testing it on a timer is the usual way.
It feels like this must be a stupid question but I don't understand something fundamental about making a request in indexedDB.
Why are the requests made before the event handlers are defined? For example, the request = objectStore.add(data) is made before the request.onsuccess and request.onerror functions are declared. Is this correct? Is it possible that the request could complete before the event handlers are registered?
I'm comparing it to the creation of an image element, followed by declaring the event handlers for onload and onerror, all before setting the source attribute to the location of a file and attempting to load it. But a request "element" can't be created before a request is made; so, there's nothing to attach the events to until the request has already been made.
Please let me know what I'm missing here. I've been writing and retrieving data from indexedDB without issue, and think I've been coding it correctly; but I want to make sure this is right and will always work.
Thank you.
Duplicate Response
I read that question and answer sometime ago, when I first started reading about indexedDB, and completely forgot about it. Had I found it again before I wrote this question, I likely would not have submitted it and would've just accepted that the code ought to work out whether I understand it or not. Handling error events and transaction aborts is what got me to thinking about the statement order again.
However, after reading the answer again, I don't understand enough to be any further along than just accepting it and hoping it will always work. I'm not trying to be snide. In one sense, it is just confusing for my limited capacity to think of event loops and epochs and that everything sort of happens all at once.
At the end of the epoch (or the start of the next, whatever you think is easier to understand), the underlying JS engine goes back and looks at what is registered to execute, and then executes everything nearly all at once.
There has to be an order of execution or nothing makes sense, asynchronous or not. I understand that the interpreter does not wait for an anysnchronous process to complete before starting to execute the next line of code. But aren't the synchronous statements processed completely in turn in the order they appear in the code and the asynchronous ones started off in the order they appear in the code, such that if an asynchronous process errored quickly, the event could be missed if the event handlers weren't declared in advance? The event handlers are not hoisted like function declarations, are they? This is the part that I still find confusing.
In this article by Jake Archibald on promises, in the introduction he presents an example concerning the loading of images and writes:
Unfortunately, in the example above, it's possible that the events happened before we started listening for them, so we need to work around that using the "complete" property of images.
and
This doesn't catch images that error'd before we got a chance to listen for them; unfortunately the DOM doesn't give us a way to do that. Also, this is loading one image, things get even more complex if we want to know when a set of images have loaded.
That gives the impression that order is important, such that, in the case of images, when possible, the source should be assigned after declaring all the event handlers, in order to not miss hearing events. The important part for me was the fact that an event could take place before the event handler was declared/registered.
I tried to follow the same pattern of making the request after declaring the event handlers in indexedDB, but it doesn't appear possible because there's nothing to attach the events to until the request is made.
Even when all the statements are asynchronous, such as in this example in the MDN Web Docs on Using IndexedDB, some things are still rather confusing. The objectStore.transaction.oncomplete is an interesting statement. We're waiting for the objectStore to be created before attempting to write data to it. (I think that's considered bad practice, writing data in an onupgradeneeded event; so, we don't use that statement.) But what is confusing is why we don't worry about the objectStore being created before creating an index in it. Why isn't the createIndex statement started at the same time the createObjectStore statement started, if everything is processed all at once? If the createObjectStore statement doesn't complete before the createIndex statement begins, shouldn't an event handler be required or it would fail because objectStore wouldn't yet exist?
I know it works because I've been using the same code pattern, but I really don't understand it.
These two items--the potential to miss events and why an event handler isn't needed in this indexedDB example--are what I'd like to better understand. I don't know if this makes my question different or not, but the answer to the duplicate question doesn't answer these for me. Perhaps, I'd have to understand the JS engine better to understand the answer to these questions.
const dbName = "the_name";
var request = indexedDB.open(dbName, 2);
request.onerror = function(event) {
// Handle errors.
};
request.onupgradeneeded = function(event) {
var db = event.target.result;
// Create an objectStore to hold information about our customers. We're
// going to use "ssn" as our key path because it's guaranteed to be
// unique - or at least that's what I was told during the kickoff meeting.
var objectStore = db.createObjectStore("customers", { keyPath: "ssn" });
// Create an index to search customers by name. We may have duplicates
// so we can't use a unique index.
objectStore.createIndex("name", "name", { unique: false });
// Create an index to search customers by email. We want to ensure that
// no two customers have the same email, so use a unique index.
objectStore.createIndex("email", "email", { unique: true });
// Use transaction oncomplete to make sure the objectStore creation is
// finished before adding data into it.
objectStore.transaction.oncomplete = function(event) {
// Store values in the newly created objectStore.
var customerObjectStore = db.transaction("customers", "readwrite").objectStore("customers");
customerData.forEach(function(customer) {
customerObjectStore.add(customer);
});
};
};
Clarification/Response to Answer/Comments
Thank you for taking the time to respond to my question and for the added explanation.
First, by "before" I simply mean the order in which the statements appear in the script.
I think I follow your analogy and it is a good one. I'm just still not clear on why the employees never submit their work to the secretary until the next day, when it is guaranteed that a secretary will be there to receive it.
It sounds similar to the fact that the javascript interpreter, when it performs its equivalent of compiling the script, hoists the function declarations, such that a function can be invoked in the code before the function declaration has been made.
It would appear that your saying, restated in my simple terms, that the JS engine, at some point before final execution, assigns the event handlers (the secretaries) to be registered in an earlier epoch than the one in which the requests (the employees) that utlimately trigger the events will complete. Therefore, it doesn't matter where in the code the request statements appear relative to the event handlers, that is, as long as they are defined within the same epoch.
The JS engine doesn't know when the request will complete, but only when the event handlers have been registered to start listening and when the request has commenced. As long as the JS engine has a process to order these steps properly independent of the order the statements appear in the code, such that an event cannot be missed, then it's no different to me than the hoisting of function declarations and I don't really have to think much about it any longer to accomplish my tasks.
However, I would still like to better understand what an epoch is, at least in terms of knowing that the statements are made within the same epoch. I don't see any mention of an epoch in the article on "Concurrecny Model and Event Loop" in MDN Web Docs. Would you mind pointing me to any good resources you know of?
Thank you.
Final Notes
I came across these two items through a link, here, on stack overflow. This same question was asked eight years ago and was answered about the same way but with different terminology; that is, instead of epochs, it was that javascript code will "run to completion" or has run-to-completion semantics. This question refers you to this document which can be searched for "run to completion" to read two exchanges about why there is not a race condition in this set up of making a request before registering the event handlers. The older JavaScript book I have by David Flanagan, in discussing the execution of JS "programs," states that, because JS has single-threaded execution, one never has to worry about race conditions; but I don't know if he was referring to this situation exactly.
Thus, this question has been asked and answered multiple times in the past and I guess I'm just another newbie asking an old question as if I was the first to have thought of it, and without enough knowledge of how JS processes.
The article "Concurrency Model and Event Loop", linked above, has a brief "Run-to-completion" section; but I didn't understand its implications until after reading the last document linked above.
I now take it to mean that all the code in a function will run to completion before any other code can begin, which seems to have two interpretations.
One is that the asynchronous request on the database is queued when the statement is reached in the function code but won't actually commence until all the other statements in the function are run, including the event handlers that are declared afterward.
Or, according to that last linked document above, the asynchronous request may run and even complete before the event handlers are registered, but the notification of its completion will remain in the queue and won't execute until after the rest of the statements in the function are run and the event handlers have been registered.
Interpretation 2 appears to be the accurate one but, whichever is the actual case, it all makes adequate sense to me now and explains why the secretary will always be there before the employee submits the work, and why, even if the employee finishes the work in a nanosecond, the employee won't submit the work until the next day when a secretary is guaranteed to be present to receive it. The employee may place the work-complete notification in the queue, but the queue won't sound the notification for a secretary to hear until the next day.
Thanks, Josh, for the additional explanation about what is meant by epochs and how that terminology works out in operation. I accepted your answer and appreciate you taking the time to write it all out.
Now that I seem to understand why the event-handler declarations can be made later in the code than the making of the request, I still don't understand why we can create an object store and then immediately create an index on that object store without having to wait until we know the object store has been successfully created, unless it is synchronous or something else special takes place in a versionchange transaction / onupgradeneeded event. There are no events mentioned in the MDN Web Docs description of createObjectStore and no examples that have any listeners on it; so; I'll just assume it's never necessary.
Thanks again.
Why are the requests made before the event handlers are defined?
It does not matter.
For example, the request = objectStore.add(data) is made before the request.onsuccess and request.onerror functions are declared. Is this correct?
Yes it is correct because again it does not matter.
I would be careful about your use of the word before. Maybe it means something different to me than it does to you. I can't tell. But maybe this is what is tripping you up.
Is it possible that the request could complete before the event handlers are registered?
If you register the event handlers in the same epoch as when you make the request, then no. The request only completes in a later epoch.
Ok, here is my attempt at explaining by example (sorry if this is bad!). Personification is generally a good educational technique, and is less intimidating then using raw technical terms, so let's go with that.
Let's say you are a boss, and have employees. Let's say you ask an employee to do some work for you. Then, you ask that employee to report back to your secretary when they completed the work. Immediately after asking the employee to go do that other work, you carry on doing your own work, without waiting for that employee to finish their work and report back. You are both basically doing work at the same time.
Now, in this situation, what happens if you don't have a secretary at the time you hand the employee a request to do something? Well, no problem. You go and hire another secretary before that employee finishes their work and before that employee even knows who to report back to, which is fine because all the employee knows is they report to your secretary. The employee does not know whether your secretary exists or not at the time of being assigned work, and does not need to know that. The missing secretary did not prevent that employee from getting started, or understanding the work to be done. And by the time that employee completes their work, you have a secretary ready and waiting. Or, you don't, because you don't happen to care to even acknowledge whether the work was actually completed, you just made a command and trusted the employee to do their job, whatever. You really only care about having them report back to your secretary if you need to do some other work that has to wait until after the first project is done, and that is a different concern.
Let's pretend you already had a secretary at the time you assigned the employee the work. What is the difference between this situation of already having a secretary, and the situation where you go and hire one shortly after assigning the work, but before it is done? There is NO difference.
Now, let's try and really address your concern. What you're suggesting is that it seems impossible to reliably go out and hire that secretary before you know whether the employee finished their assignment. This I think is the critical misunderstanding. It is entirely possible to do this. Why is that? I suppose it is not the easiest thing to grasp.
I am going to stretch this metaphor a bit and impose a strange rule. No matter how simple the project you hand off to the employee, even if it is just to run and get you coffee in the morning, they will never ever get back to you the same day. They will always finish their work some later day, at the earliest tomorrow. They might even finish their work within one fleeting nanosecond of you telling them, but they will NEVER get back to you or your secretary right away, they will always be delayed until tomorrow at the earliest.
This means you have all day to go and hire that secretary that did not exist at the time you gave the employee the order. So long as you do it before tomorrow, you're good. That secretary will exist and be working for you by the time the employee responds tomorrow, and will be able to receive the message from the employee.
Edit response to your added comments:
Yep, hoisting is similar in many respects. Things can happen in a different order then written in code. Hoisting is of course synchronous so it is not a perfect similarity, but the out-of-order aspect is still similar.
Epoch is just my own word i use for a single iteration of the event loop. Like in the case of a for loop using i for i from 0 to 2, there are 3 epochs, iteration 0, iteration 1, and iteration 2. i just call them epochs because it is like categories of time.
In a promise case, it might even be a microtask. In a js worker case, it might be thread-like (and workers are the new hotness over the old child-iframe technique). Basically these are all just ways to 'realize' doing more than one thing at a time. Node calls it a tick, and has things like nextTick() that defers code execution until the next tick of its loop. Within a single epoch, things happen in the order they are written (and notably hoisting is all in epoch 0). but some code may be async, and therefore happen across epochs, and therefore may run in a different order than it was written. Code written earlier may happen in a later epoch.
When you make a request, it says, start doing this thing, and get back to me at the earliest in the next epoch. You have up until the end of the current epoch to register handlers for the request.
Some code, like in the case of an image preloader as mentioned in your example, has to take into account that it attaches the listeners too late (images are being preloaded in an alternate timeline and some may already be loaded and in some browsers this means load will not fire), so it wants to check imageElement.complete to catch that case. In other cases of event listener implementations, some dispatcher implementations will fire events to newly added listeners for events that already happened where the new listener was not listening at the time of the event. But that is not a universal characteristic of event listener implementations, just a characteristic of certain implementations.
And in the case of the transaction.oncomplete thing from within onupgradeneeded, that is just not a great example. It is doing stuff it does not need to do.
This is the technical answer to your question:
https://html.spec.whatwg.org/multipage/webappapis.html#event-loops
The JS concurrency model is cooperative with "run-to-completion" semantics (no parallel processing of events in the same queue). This means that any async response will be posted as a message to the window event loop, and all the sequential code you see after the request is guaranteed to execute before the async response processing starts.
That said, from usability point of view, the IndexDB API is not affording the intent in the most expressive manner, and coming from other languages with preemptive threading you are excused for being confused :-)
I'm wondering if there's a way to cause JavaScript to wait for some variable-length code execution to finish before continuing using events and loops. Before answering with using timeouts, callbacks or referencing this as a duplicate, hear me out.
I want to expose a large API to a web worker. I want this API to feel 'native' in the sense that you can access each member using a getter which gets the information from the other thread. My initial idea was to compile the API and rebuild the entire object on the worker. While this works (and was a really fun project), it's slow at startup and cannot show changes made to the API without it being sent to the worker again after modification. Observers would solve part of this, and web workers transferrable objects would solve all, but they aren't adopted widely yet.
Since worker round-trip calls happen in a matter of milliseconds, I think stalling the thread for a few milliseconds may be an alright solution. Of course I would think about terminating in cases where calls take too long, but I'm trying to create a proof of concept first.
Let's say I want to expose the api object to the worker. I would define a getter for self.api which would fetch the first layer of properties. Each property would then be another getter and the process would continue until the final object is found.
worker.js
self.addEventListener('message', function(event) {
self.dataRecieved = true;
self.data = event.data; // would actually build new getters here
});
Object.defineProperty(self, 'api', {
get: function() {
self.dataRecieved = false;
self.postMessage('request api first-layer properties');
while(!self.dataRecieved);
return self.data; // whatever properties were received from host
}
});
For experimentation, we'll do a simple round-trip with no data processing:
index.html (only JS part)
var worker = new Worker("worker.js");
worker.onmessage = function() {
worker.postMessage();
};
If onmessage would interrupt the loop, the script should theoretically work. Then the worker could access objects like window.document.body.style on the fly.
My question really boils down to: is there a way to guarantee that an event will interrupt an executing code block?
From my understanding of events in JavaScript, I thought they did interrupt the current thread. Does it not because it's executing a blank statement over and over? What if I generated code to be executed and kept doing that until the data returned?
is there a way to guarantee that an event will interrupt an executing code block
As #slebetman suggests in comments, no, not in Javascript running in a browser's web-worker (with one possible exception that I can think of, see suggestion 3. below).
My suggestions, in decreasing order of preference:
Give up the desire to feel "native" (or maybe "local" might be a better term). Something like the infinite while loop that you suggest also seems to be very much fighting agains the cooperative multitasking environment offered by Javascript, including when thinking about a single web worker.
Communication between workers in Javascript is asynchronous. Perhaps it can fail, take longer than just a few milliseconds. I'm not sure what your use case is, but my feeling is that when the project grows, you might want to use those milliseconds for something else.
You could change your defined property to return a promise, and then the caller would do a .then on the response to retrieve the value, just like any other asynchronous API.
Angular Protractor/Webdriver has an API that uses a control flow to simulate a synchronous environment using promises, by always passing promises about. Taking the code from https://stackoverflow.com/a/22697369/1319998
browser.get(url);
var title = browser.getTitle();
expect(title).toEqual('My Title');
By my understanding, each line above adds a promise to the control flow to execute asynchronously. title isn't actually the title, but a promise that resolves to the title for example. While it looks like synchronous code, the getting and testing all happens asynchronously later.
You could implement something similar in the web worker. However, I do wonder whether it will be worth the effort. There would be a lot of code to do this, and I can't help feeling that the main consequence would be that it would end up harder to write code using this, and not easier, as there would be a lot of hidden behaviour.
The only thing that I know of that can be made synchronous in Javascript, is XMLHttpRequest when setting the async parameter to false https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest#Parameters. I wonder if you could come up with some sort of way to request to the server that maintains a connection with the main thread and pass data along that way. I have to say, my instinct is that this is quite an awful idea, and would be much slower than just requesting data from the main thread.
For what I know, there is not something native in JS to do this but it is relatively easy to do something similar. I made one some time ago for myself: https://github.com/xpy/whener/blob/master/whener.js .
You use it like when( condition, callback ) where condition is a function that should return true when your condition is met, and callback is the function that you want to execute at that time.
I have a web application that is growing more complex. It makes heavy use of JavaScript based HTML generation and AJAX calls, and herein lies my problem:
Since I can't know how long an ajax call might take getting back to client side, I don't know when the callback gets actually executed. The user might have at that point navigated away from the element that originally caused the AJAX event, in which case this callback can cause some havoc. Is there a way to "expire" old callbacks ?
Are there any libraries that would offer that functionality? (I am using jQuery now but am not 100% familiar with it).
Thanks,
You might want to look into Ajax Queue Manager. There are params you can set to abort old requests before sending a new one. I think that might be what your looking for.
Well, the simple answer is to check for the proper state of your app within your callback functions, before they do whatever it is they are doing that causes problems. For example, you could make sure that certain elements are still being hovered over.
I'm repeatedly coming into troubles with Internet Explorer's "This script is taking too long to run, would you like to continue?" messages. I am wondering if anyone is aware of a clever way to trick the JS engine into keeping quiet? Based on some searching I did, I found that the engine monitors states it thinks potentially could be infinitely looping, so I thought maybe I could add some logic to change up the execution every once in a while to fool it into leaving it alone, but no luck. I also tried breaking up a longer loop into several shorter ones, but that hasn't helped. Specifically the code that is currently causing issues is the expansion of nodes in a tree structure. The code is looping over the current nodes and expanding each. It's a trivial thing to write in Javascript, but I can't allow these timeout errors, so I think my only option might be to request pre-expanded view data via AJAX. I'm currently working in a DEV environment with a small(ish) data set and I know this will not fly in other environments. Has anyone managed to suppress these warnings?
Using setTimeout
A good way is simulating threaded execution using setTimeout() function calls. This requires splitting your whole processing into smaller parts and queueing them one after another. Timeouts can be set quite close to each other but they will run one by one when each of them finishes execution.
How about spacing it using a series of events. So a loop occurs sends an event, listener to event triggers and does a loop. etc..?
Why not break your function into a series of steps and queue them up using jQuery?
http://api.jquery.com/queue/
Have you tried making it output something every once in a while? It might be that it just checks for output and if there hasn't been any in x seconds, it assumes you're in an infinite loop.
If outputting works, you could try something like adding and then immediately deleting something really small (like an empty <span>).
A very common solution for this problem is to use setTimeout function.
The way you do it is that you separate the process into smaller pieces a then execute those pieces one after another using the setTimeout function.
I think this http://www.julienlecomte.net/blog/2007/10/28/ should help you.
There is also another option introduced by HTML5 WebWorkers.
This new standard should allow you to execute long running tasks in a separate thread and then report any results in a callback.
You can read about it here robertnyman.com/2010/03/25/using-html5-web-workers-to-have-background-computational-power/
Unfortunatelly, it is not supported by IE according to html5demos.com/
I think the timeout is more based on the number of statements than timing or heuristics. You could go a long way to increasing the amount your code can handle before triggering the warning by optimizing your code for simple things -- especially if you are using helper APIs on another library like jQuery. For example, change this:
$.each(arr, function(value) {
// do stuff
});
to this:
for (var i = 0, l = arr.length; i < l; i++) {
var value = arr[i];
// do stuff
}
Another easy one -- cache access to fields. If you have two instances of "foo.bar", store the result in a variable and use it, wherever that makes sense.
Obviously I have no idea what your code looks like, but I bet you could do a lot to improve it as these little things really add up when you're talking about this timeout problem.
I managed to do this by using prototypes Function#defer method, which is essentially the same as using the setTimeout method. Thanks everyone!