Signal the end of a series of (n) AJAX calls - javascript

I have a web application which crawls sites in a CMS and looks for data of a certain type.
It works a lot like a recursive file/directory loop:
//pseudo code
var rootWeb = context.site.rootWeb();
var objectThatHoldsAllResults;
recursiveSiteSearch(rootWeb);
function recursiveSiteSearch(webSite) {
//Get all content of a certain type and add to objectThatHoldsAllResults
//Get all SubSites and throw them into a loop that runs recursiveSiteSearch
}
This application lives in the cloud, and has no idea how many subsites live in each CMS that accesses it.
Each time the loop gets all content of a certain type, it makes an AJAX call to the website.
I need to know when the recursion is done, but have no idea how to do so.

Straightforwardly, recursion will have ended when execution falls through to the statement following recursiveSiteSearch(rootWeb);.
However, asynchronicity (ajax) within recursiveSiteSearch may/will(?) mean that some latent activity still exists at that point.
Therefore, you appear to need a mechanism for detecting when all promises (ie. all ajax requests initiated within the recursion) are complete.
jQuery, provides such a mechanism.
pseudo code :
function recursiveSiteSearch(webSite) {
//Get all content of a certain type and add to objectThatHoldsAllResults
//Get all SubSites and throw them into a loop that runs recursiveSiteSearch
//Within the loop, push jqXHR objects onto the externally declared `promises` array.
}
var rootWeb = context.site.rootWeb();
var objectThatHoldsAllResults;
var promises = [];
recursiveSiteSearch(rootWeb);
jQuery.when.apply(jQuery, promises).done(function() {
//statements here will execute when
//recursion has finished and all ajax
//requests have completed.
});
The reason this should work is that jqXHR objects (returned by jQuery.ajax() and its shorthand forms) implement jQuery's Promise interface.

Related

Returning Chrome storage API value without function

For the past two days I have been working with chrome asynchronous storage. It works "fine" if you have a function. (Like Below):
chrome.storage.sync.get({"disableautoplay": true}, function(e){
console.log(e.disableautoplay);
});
My problem is that I can't use a function with what I'm doing. I want to just return it, like LocalStorage can. Something like:
var a = chrome.storage.sync.get({"disableautoplay": true});
or
var a = chrome.storage.sync.get({"disableautoplay": true}, function(e){
return e.disableautoplay;
});
I've tried a million combinations, even setting a public variable and setting that:
var a;
window.onload = function(){
chrome.storage.sync.get({"disableautoplay": true}, function(e){
a = e.disableautoplay;
});
}
Nothing works. It all returns undefined unless the code referencing it is inside the function of the get, and that's useless to me. I just want to be able to return a value as a variable.
Is this even possible?
EDIT: This question is not a duplicate, please allow me to explain why:
1: There are no other posts asking this specifically (I spent two days looking first, just in case).
2: My question is still not answered. Yes, Chrome Storage is asynchronous, and yes, it does not return a value. That's the problem. I'll elaborate below...
I need to be able to get a stored value outside of the chrome.storage.sync.get function. I -cannot- use localStorage, as it is url specific, and the same values cannot be accessed from both the browser_action page of the chrome extension, and the background.js. I cannot store a value with one script and access it with another. They're treated separately.
So my only solution is to use Chrome Storage. There must be some way to get the value of a stored item and reference it outside the get function. I need to check it in an if statement.
Just like how localStorage can do
if(localStorage.getItem("disableautoplay") == true);
There has to be some way to do something along the lines of
if(chrome.storage.sync.get("disableautoplay") == true);
I realize it's not going to be THAT simple, but that's the best way I can explain it.
Every post I see says to do it this way:
chrome.storage.sync.get({"disableautoplay": true, function(i){
console.log(i.disableautoplay);
//But the info is worthless to me inside this function.
});
//I need it outside this function.
Here's a tailored answer to your question. It will still be 90% long explanation why you can't get around async, but bear with me — it will help you in general. I promise there is something pertinent to chrome.storage in the end.
Before we even begin, I will reiterate canonical links for this:
After calling chrome.tabs.query, the results are not available
(Chrome specific, excellent answer by RobW, probably easiest to understand)
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference (General canonical reference on what you're asking for)
How do I return the response from an asynchronous call?
(an older but no less respected canonical question on asynchronous JS)
You Don't Know JS: Async & Performance (ebook on JS asynchronicity)
So, let's discuss JS asynchonicity.
Section 1: What is it?
First concept to cover is runtime environment. JavaScript is, in a way, embedded in another program that controls its execution flow - in this case, Chrome. All events that happen (timers, clicks, etc.) come from the runtime environment. JavaScript code registers handlers for events, which are remembered by the runtime and are called as appropriate.
Second, it's important to understand that JavaScript is single-threaded. There is a single event loop maintained by the runtime environment; if there is some other code executing when an event happens, that event is put into a queue to be processed when the current code terminates.
Take a look at this code:
var clicks = 0;
someCode();
element.addEventListener("click", function(e) {
console.log("Oh hey, I'm clicked!");
clicks += 1;
});
someMoreCode();
So, what is happening here? As this code executes, when the execution reaches .addEventListener, the following happens: the runtime environment is notified that when the event happens (element is clicked), it should call the handler function.
It's important to understand (though in this particular case it's fairly obvious) that the function is not run at this point. It will only run later, when that event happens. The execution continues as soon as the runtime acknowledges 'I will run (or "call back", hence the name "callback") this when that happens.' If someMoreCode() tries to access clicks, it will be 0, not 1.
This is what called asynchronicity, as this is something that will happen outside the current execution flow.
Section 2: Why is it needed, or why synchronous APIs are dying out?
Now, an important consideration. Suppose that someMoreCode() is actually a very long-running piece of code. What will happen if a click event happened while it's still running?
JavaScript has no concept of interrupts. Runtime will see that there is code executing, and will put the event handler call into the queue. The handler will not execute before someMoreCode() finishes completely.
While a click event handler is extreme in the sense that the click is not guaranteed to occur, this explains why you cannot wait for the result of an asynchronous operation. Here's an example that won't work:
element.addEventListener("click", function(e) {
console.log("Oh hey, I'm clicked!");
clicks += 1;
});
while(1) {
if(clicks > 0) {
console.log("Oh, hey, we clicked indeed!");
break;
}
}
You can click to your heart's content, but the code that would increment clicks is patiently waiting for the (non-terminating) loop to terminate. Oops.
Note that this piece of code doesn't only freeze this piece of code: every single event is no longer handled while we wait, because there is only one event queue / thread. There is only one way in JavaScript to let other handlers do their job: terminate current code, and let the runtime know what to call when something we want occurs.
This is why asynchronous treatment is applied to another class of calls that:
require the runtime, and not JS, to do something (disk/network access for example)
are guaranteed to terminate (whether in success or failure)
Let's go with a classic example: AJAX calls. Suppose we want to load a file from a URL.
Let's say that on our current connection, the runtime can request, download, and process the file in the form that can be used in JS in 100ms.
On another connection, that's kinda worse, it would take 500ms.
And sometimes the connection is really bad, so runtime will wait for 1000ms and give up with a timeout.
If we were to wait until this completes, we would have a variable, unpredictable, and relatively long delay. Because of how JS waiting works, all other handlers (e.g. UI) would not do their job for this delay, leading to a frozen page.
Sounds familiar? Yes, that's exactly how synchronous XMLHttpRequest works. Instead of a while(1) loop in JS code, it essentially happens in the runtime code - since JavaScript cannot let other code execute while it's waiting.
Yes, this allows for a familiar form of code:
var file = get("http://example.com/cat_video.mp4");
But at a terrible, terrible cost of everything freezing. A cost so terrible that, in fact, the modern browsers consider this deprecated. Here's a discussion on the topic on MDN.
Now let's look at localStorage. It matches the description of "terminating call to the runtime", and yet it is synchronous. Why?
To put it simply: historical reasons (it's a very old specification).
While it's certainly more predictable than a network request, localStorage still needs the following chain:
JS code <-> Runtime <-> Storage DB <-> Cache <-> File storage on disk
It's a complex chain of events, and the whole JS engine needs to be paused for it. This leads to what is considered unacceptable performance.
Now, Chrome APIs are, from ground up, designed for performance. You can still see some synchronous calls in older APIs like chrome.extension, and there are calls that are handled in JS (and therefore make sense as synchronous) but chrome.storage is (relatively) new.
As such, it embraces the paradigm "I acknowledge your call and will be back with results, now do something useful meanwhile" if there's a delay involved with doing something with runtime. There are no synchronous versions of those calls, unlike XMLHttpRequest.
Quoting the docs:
It's [chrome.storage] asynchronous with bulk read and write operations, and therefore faster than the blocking and serial localStorage API.
Section 3: How to embrace asynchronicity?
The classic way to deal with asynchronicity are callback chains.
Suppose you have the following synchronous code:
var result = doSomething();
doSomethingElse(result);
Suppose that, now, doSomething is asynchronous. Then this becomes:
doSomething(function(result) {
doSomethingElse(result);
});
But what if it's even more complex? Say it was:
function doABunchOfThings() {
var intermediate = doSomething();
return doSomethingElse(intermediate);
}
if (doABunchOfThings() == 42) {
andNowForSomethingCompletelyDifferent()
}
Well.. In this case you need to move all this in the callback. return must become a call instead.
function doABunchOfThings(callback) {
doSomething(function(intermediate) {
callback(doSomethingElse(intermediate));
});
}
doABunchOfThings(function(result) {
if (result == 42) {
andNowForSomethingCompletelyDifferent();
}
});
Here you have a chain of callbacks: doABunchOfThings calls doSomething immediately, which terminates, but sometime later calls doSomethingElse, the result of which is fed to if through another callback.
Obviously, the layering of this can get messy. Well, nobody said that JavaScript is a good language.. Welcome to Callback Hell.
There are tools to make it more manageable, for example Promises and async/await. I will not discuss them here (running out of space), but they do not change the fundamental "this code will only run later" part.
Section TL;DR: I absolutely must have the storage synchronous, halp!
Sometimes there are legitimate reasons to have a synchronous storage. For instance, webRequest API blocking calls can't wait. Or Callback Hell is going to cost you dearly.
What you can do is have a synchronous cache of the asynchronous chrome.storage. It comes with some costs, but it's not impossible.
Consider:
var storageCache = {};
chrome.storage.sync.get(null, function(data) {
storageCache = data;
// Now you have a synchronous snapshot!
});
// Not HERE, though, not until "inner" code runs
If you can put ALL your initialization code in one function init(), then you have this:
var storageCache = {};
chrome.storage.sync.get(null, function(data) {
storageCache = data;
init(); // All your code is contained here, or executes later that this
});
By the time code in init() executes, and afterwards when any event that was assigned handlers in init() happens, storageCache will be populated. You have reduced the asynchronicity to ONE callback.
Of course, this is only a snapshot of what storage looks at the time of executing get(). If you want to maintain coherency with storage, you need to set up updates to storageCache via chrome.storage.onChanged events. Because of the single-event-loop nature of JS, this means the cache will only be updated while your code doesn't run, but in many cases that's acceptable.
Similarly, if you want to propagate changes to storageCache to the real storage, just setting storageCache['key'] is not enough. You would need to write a set(key, value) shim that BOTH writes to storageCache and schedules an (asynchronous) chrome.storage.sync.set.
Implementing those is left as an exercise.
Make the main function "async" and make a "Promise" in it :)
async function mainFuction() {
var p = new Promise(function(resolve, reject){
chrome.storage.sync.get({"disableautoplay": true}, function(options){
resolve(options.disableautoplay);
})
});
const configOut = await p;
console.log(configOut);
}
Yes, you can achieve that using promise:
let getFromStorage = keys => new Promise((resolve, reject) =>
chrome.storage.sync.get(...keys, result => resolve(result)));
chrome.storage.sync.get has no returned values, which explains why you would get undefined when calling something like
var a = chrome.storage.sync.get({"disableautoplay": true});
chrome.storage.sync.get is also an asynchronous method, which explains why in the following code a would be undefined unless you access it inside the callback function.
var a;
window.onload = function(){
chrome.storage.sync.get({"disableautoplay": true}, function(e){
// #2
a = e.disableautoplay; // true or false
});
// #1
a; // undefined
}
If you could manage to work this out you will have made a source of strange bugs. Messages are executed asynchronously which means that when you send a message the rest of your code can execute before the asychronous function returns. There is not guarantee for that since chrome is multi-threaded and the get function may delay, i.e. hdd is busy.
Using your code as an example:
var a;
window.onload = function(){
chrome.storage.sync.get({"disableautoplay": true}, function(e){
a = e.disableautoplay;
});
}
if(a)
console.log("true!");
else
console.log("false! Maybe undefined as well. Strange if you know that a is true, right?");
So it will be better if you use something like this:
chrome.storage.sync.get({"disableautoplay": true}, function(e){
a = e.disableautoplay;
if(a)
console.log("true!");
else
console.log("false! But maybe undefined as well");
});
If you really want to return this value then use the javascript storage API. This stores only string values so you have to cast the value before storing and after getting it.
//Setting the value
localStorage.setItem('disableautoplay', JSON.stringify(true));
//Getting the value
var a = JSON.stringify(localStorage.getItem('disableautoplay'));
var a = await chrome.storage.sync.get({"disableautoplay": true});
This should be in an async function. e.g. if you need to run it at top level, wrap it:
(async () => {
var a = await chrome.storage.sync.get({"disableautoplay": true});
})();

Javascript window.location calls getting lost?

I am having some trouble with a bit of code. I have a function that does some stuff to some data, calls a remote system (activating a script on that system and passing in the data), and then makes another call to the same system to activate a different script (which acts on the data saved above). The problem is that the 1st call to the remote system appears to get lost in the execution.
This is being run in Safari, uses jquery; the function is tied to a button click, which is defined in the javascript code with an onclick function (i.e. it is not defined in the html button definition).
Here's a rough breakdown of the function (cleaned out for viewing purposes - I hope I left enough to make it clear):
function compareJSON() {
// loop through the objects, testing and changing data
// ...
dataSession=({ //build object for output });
$.each( dataSession.chapters , function( indexC, value ) {
//compare objects to some others, testing and changing data
});
// ...
//Call remote script on other system
urlString="url://blah.dee.com/Blar?script=SaveJSON&$JSONobject=";
window.location= urlString + JSON.stringify(dataSession);
//Call remote script on other system
window.location="url://blah.dee.com/Blar?script=EditJSON";
}
The last three lines of code are the two calls. It uses the window.location to actually trigger the remote system, passing the data through the URL. But I need BOTH scripts to get called and run. It appears that only the LAST script in the sequence ever gets run. If I switch them around it remains whatever is in last place.
Is there something about the window.location that doesn't actually process until the end of the function?
This script actually used to be a series of separate function calls, but I figured I was running into asynchronous execution that was causing the various script calls to not register. But once I put the code into this single function, it was still happening.
Any clues would be helpful.
Thanks,
J
Modifing the value of window.location is reserved exclusively for instances in which you'd like to cause a browser redirect.
It looks like you want to trigger a page request instead. You say you already have jQuery loaded, if so, you can trigger such a request using jQuery.get or a similar function.
For example:
// Loads the myscript.php page in the background
$.get('myscript.php');
// You can also pass data (in the form of an object as the second argument)
$.get('myscript.php', { name: "John", time: "2pm" });

Process chain of functions without UI block

I need to perform several functions in my JavaScript/jQuery, but I want to avoid blocking the UI.
AJAX is not a viable solution, because of the nature of the application, those functions will easily reach the thousands. Doing this asynchroniously will kill the browser.
So, I need some way of chaining the functions the browser needs to process, and only send the next function after the first has finished.
The algorithm is something like this
For steps from 2 to 15
HTTP:GET amount of items for current step (ranges somewhere from a couple of hundred to multiple thousands)
For every item, HTTP:GET the results
As you see, I have two GET-request-"chains" I somehow need to manage... Especially the innermost loop crashes the browser near to instantly, if it's done asynchroniously - but I'd still like the user to be able to operate the page, so a pure (blocking) synchronous way will not work.
You can easily do this asynchronously without firing all requests at once. All you need to do is manage a queue. The following is pseudo-code for clarity. It's easily translatable to real AJAX requests:
// Basic structure of the request queue. It's a list of objects
// that defines ajax requests:
var request_queue = [{
url : "some/path",
callback : function_to_process_the_data
}];
// This function implements the main loop.
// It looks recursive but is not because each function
// call happens in an event handler:
function process_request_queue () {
// If we have anything in the queue, do an ajax call.
// Otherwise do nothing and let the loop end.
if (request_queue.length) {
// Get one request from the queue. We can either
// shift or pop depending on weather you prefer
// depth first or breadth first processing:
var req = request_queue.pop();
ajax(req.url,function(result){
req.callback(result);
// At the end of the ajax request process
// the queue again:
process_request_queue();
}
}
}
// Now get the ball rolling:
process_request_queue();
So basically we turn the ajax call itself into a pseudo loop. It's basically the classic continuation passing style of programming done recursively.
In your case, an example of a request would be:
request_queue.push({
url : "path/to/OUTER/request",
callback : function (result) {
// You mentioned that the result of the OUTER request
// should trigger another round of INNER requests.
// To do this simply add the INNER requests to the queue:
request_queue.push({
url : result.inner_url,
callback : function_to_handle_inner_request
});
}
});
This is quite flexible because you not only have the option of processing requests either breadth first or depth first (shift vs pop). But you can also use splice to add stuff to the middle of the queue or use unshift vs push to put requests at the head of the queue for high priority requests.
You can also increase the number of simultaneous requests by popping more than one request per loop. Just be sure to only call process_request_queue only once per loop to avoid exponential growth of simultaneous requests:
// Handling two simultaneous request channels:
function process_request_queue () {
if (request_queue.length) {
var req = request_queue.pop();
ajax(req.url,function(result){
req.callback(result);
// Best to loop on the first request.
// The second request below may never fire
// if the queue runs out of requests.
process_request_queue();
}
}
if (request_queue.length) {
var req = request_queue.pop();
ajax(req.url,function(result){
req.callback(result);
// DON'T CALL process_request_queue here
// We only want to call it once per "loop"
// otherwise the "loop" would grow into a "tree"
}
}
}
You could make that ASYNC and use a small library I wrote some time ago that will let you queue function calls.

Deferring Execution Till Needed

I know that you use jQuery deferreds in the following scenario:
I have a asynchronous request. I'd like to execute one or more sections of code whenever this request comes back.
Here's my issue. I don't want the async request to actually be sent out until it hits the first .when statement.
For instance:
I have an init function that loads up say 50 different deferred objects for various data requests. I obviously don't want all of these to fire at once, just when needed. The data will be loaded (the deferred object resolved) on future .when statements.
Sort of complicated but thanks for reading this! :D
Here's what we have
init() = function {
var data1 = someDeferredRequest();
//blah, lots of these
}
.doSomethingElse() = function {
//I only want the call to data1 to have been started the first instance I try to go get it, like this instance below. I do NOT want it fired in the init
$.when(data1).then();
//other stuff, still want data1 to be completed by here
$.when(data1).then();
}
Have a method on the objects named fire() or run(), whatever suits you that grabs/pulls does whatever you want with the data.
The object will then sit idle until you call object.run().

Sequential web service call not working

A little (!) bit of background before I can get to the question :
I am implementing a web based search solution. Technology used: javascript (jquery), .net, html etc. etc.
All my web service calls are done through javascript (cross domain ws call). I have few sequential web service calls which all have different success callback function.
I am not able to digest - when i call those ws individually in seperate places they are returning me proper results but sequentially sometime they are giving and sometime not.
sample code: this is not giving expected results all the time.
function submitSearchRequest(_queryString, Stores) {
if (Stores[1].length>0) {
//generate 'searchRequestForArtifact' request object
getSearchResponse("successcallForArtifact", _searchRequestForArtifact);
}
if (Stores[2].length > 0) {
//generate 'searchRequestForPerson' request object
getSearchResponse("successcallForPerson", _searchRequestForPerson);
}
}
function successcallForArtifact(response)
{
//show the results
}
function successcallForPerson(response)
{
//show the results
}
}
If you need sequentially you will need to kick off each search only after one has returned. Currently you are making async calls, meaning it gets kicked off then continues with the code. Currently if the second call is simply faster the order will be off. You will either need to make a sync call or simply have the order enforced by calling the second search from the success function for the artifact.
If you are using JQuery which it seems you are you can set the async parameter to false which will force the order you want but it will slow the overall performance of your page. See this question.

Categories