I have an API from which I need to query N pages of data. Because I do not want to overload the API, I want to do it sequentially and without blocking the main thread.
The code would be something like this:
var res = []; // all data from api
var totalPages = 10;
var pageSize = 100;
for (let page = 0; page < totalPages; page++) {
// load using jQuery ajax request
$.get('api.php', { page: page, page_size: pageSize }, function(result) {
res.push(...result); // add data to resulting array
});
}
But this approach has a few issues:
Since it is async, it will just run all requests in parallel, overloading API as a result. I need them to still run async, but each should run only when the previous is done.
Since all the calls are async, by the end of the cycle we still wouldn't have the requested data - it will be loading in the background. We need somehow to wait for all callbacks to be done before returning res to some other code that needs it.
There is no way to make every callback pass its result to the next callback and it is the only way to stop loading when some callback receives "stop loading"/"no more data" request from the server
Is there a way to fix these issues without using some side libraries or promises? Just plain old vanilla javascript.
Sorry if something looks unclear, I am not very experienced in js
You can use a simple recursion here, like this:
var res = []; // all data from api
var totalPages = 10;
var pageSize = 100;
const loader = page => {
// load using jQuery ajax request
$.get('api.php', { page: page, page_size: pageSize }, function(result) {
res.push(...result); // add data to resulting array
if (page < totalPages)
loader(++page)
else
console.log('DONE!');
});
}
loader(0);
Related
So I have a page that is making several ajax calls at the same time. Each ajax call pertains to a block/div with data that was retrived, for instance, lets say there are 5 divs. When the page is rendered, I want one of the main divs to load entirely and display before the other divs/blocks.
Ex. I have a page that has general info at top, and the other divs are more details on a part of the main div. So they should all be called at the same time, but I want the general div to load all of its content first before allowing the more specific divs display on the screen.
I want all the calls to be asynchronous, but somehow delay the appearance of the others. Is this possible?
I understand that async: false will only delay when the ajax calls are called, but I need them to all be called at the same time.
Any ideas? Please and thank you.
You can use setTimeout(foo, 3000); if you need to wait for a specific time interval then call your function foo() to do what you want after that time interval
You can place/call other Ajax calls on success of general div Ajax call.
I think I made it via angular, on my case I need to query all status before executing to other process. Make a array which combines all of your ajax requests then use $q.all equivalent to native javascript's Promise.all()
getAllStatusResult() {
let promises = [];
let statusList = ['all','done','ongoing','cancelled'];
let responseList = [];
let date = new Date(), y = date.getFullYear(), m = date.getMonth();
let deferred = $q.defer();
statusList.map(status => {
let payload = {
start_date: "2017-1-1",
end_date: formatDate(date),
filter: { status }
}
promises.push(ReportsService.getRevenues(payload).then((response) => {
responseList.push({ name: status, total:response.data.total});
}));
});
$q.all(promises).then(() => {
deferred.resolve(responseList);
}).then(response => {
return response;
});
return deferred.promise;
}
For my website I am using an API where I need to load several variables from. Each of the variables are dependent on the return value of the previous call (I use the returned variable from call 1 to make call 2 etc).
Example:
Say that I need to make 5 different API calls to gather all of my data and each is dependent on the return value of the previous call. Then in my case I am doing like this. I am passing a callback function to the first function that loads the data. Then that function will make the first API call. When that call is finished it will pass the callback function to the next function that makes the second API call, and so on. When the last API call is finished the callback function gets called and then I know that all the data has been loaded. In code it would look something like this (I am using the Trello API in my application so I will use it in the example below):
function loadData(cb){
//Make the first API call
Trello.get('/member/me/boards', function(boards){
myBoards = boards;
for(var i = 0; i < boards.length; i++){
//Make the second API call
Trello.get('/boards/' + board[i].id + '/lists', function(lists){
board[i].lists = lists;
//Then make the third and fourth and so on
.....
//When all calls are made call the callback function
cb();
});
});
}
As you can see the callback function will be passed a long way into the callstack. I was wondering if there is a better way to load the data and to store it (as of now I just store everything in a large array). And what is some best practices for loading large amount of data from an API?
P.S. In my original code each of the API calls are in separate functions, but I simplified it here to reduce the amount of code in the example.
I don't know if this is an option for you but using TypeScript makes solving this kind of JavaScript problem much more simple:
async function loadData() {
const boards = await Trello.get('/member/me/boards');
return boards.map(async (board) => {
const lists = await Trello.get('/boards/' + board.id + '/lists');
const something = await Trello.get('/...');
const somethingElse = await Trello.get('/...');
// ...more calls
return {
...board,
lists: lists,
something: something,
somethingElse: somethingElse
// ... more attributes
};
});
}
loadData().then((data) => console.log(data));
Without fully understanding your problem this may not be a valid solution, but taking a quick glance at the trello api docs shows a batch call you could make to avoid looping at each level. Batching these would allow for many fewer API calls at each level and would be considered a best practice:
function loadData(cb){
//Make the first API call
Trello.get('/member/me/boards', function(boards){
myBoards = boards;
var boardAPIs = [];
var boardResponses = [];
for(var i = 0; i < boards.length; i++){
boardAPIs.push('/boards/' + board[i].id + '/lists');
//max of 10 at a time per documentation
if (boardAPIs.length == 10 || i >= (boards.length - 1)) {
//Make the second level API call
Trello.get('/batch/?urls=' + boardAPIs.join(','), function(boards){
// collect response information on all boards, then continue with third request
boardResponses.push(...);
if (i >= (boards.length - 1)) {
// all board requests have been made, continue execution at third level
// if this were the last level of calls, you could call cb() here
for(var j = 0; i < boardResponses.length; i++){
// loop inside responses to get individual board responses, build up next set of batch requests
}
}
});
boardAPIs= [];
}
});
});
}
One thing to note here: the docs mentioned that you can only batch 10 requests at a time, so I added some code in there to check for that.
This post provides more information on how to consume the batch service:
this means you get only a single response back, and it looks a little
different from a normal response. The response is an array of objects
– but not of the normal response objects you might expect. Instead,
it’s an object with a single property, with a name set to the HTTP
response code of the request.
You may focus on a deep first approach, so that the first data arrives fast at the client:
function loadData(showChunk){
//Make the first API call
Trello.get('/member/me/boards', function(boards){
myBoards = boards;
(function getboard(i){
//Make the second API call
Trello.get('/boards/' + board[i].id + '/lists', function(lists){
board[i].lists = lists;
//Then make the third and fourth and so on
.....
//When all calls are made for the first board call the callback function, and also continue with the next board
showChunk();
if(i+1<boards.length) setTimeout(getboard, 1, i+1);
});
})(0);
});
}
How can I get some javascript to run before a document ready function?
I have the following snippet finished first...
var applicantlist = [];
$.getJSON("apps.json", function(appsdata) {
for (var i = 0; i<appsdata.applications.length; i++){
var tempapp = [appsdata.applications[i].name, appsdata.applications[i].server];
applicantlist.push(tempapp);
}
});
I've tested this, and the data gets pushed into the array just fine. The problem is that I need this array to make some ajax calls that are found in my page ready function as follows...
$(document).ready(function(){
window.jsonpCallbacks = {};
alert(applicantlist.length);
for (var i = 0; i < applicantlist.length; i++){
(function(index){
window.jsonpCallbacks["myCallback" + index] = function(data){
myCallback(data,index);
};
})(i);
//Jquery/Ajax call to the WoW API for the data.
$.ajax({
"url":"http://us.battle.net/api/wow/character/" + applicantlist[i][1] + "/" + applicantlist[i][0] + "?jsonp=jsonpCallbacks.myCallback" + i,
"type":"GET",
"data": { fields: "items, talents, progression, professions, audit, guild, stats"},
"dataType":"jsonp",
"contentType":"application/json",
"jsonpCallback":"jsonpCallbacks.myCallback"+i,
"success":function(data1){
}
})
}
All of this fires off before the first snipet, no matter where I seem to put it. So, the array is empty (the alert message just shows "0").
As you can see by the URL of my ajax call, I need that array populated beforehand. I've tried putting the first snippet in a seperate .js file and calling it before all other javascript files on the actual HTML page...
What am I missing?
Move the code that sends the first request to the document.ready. You don't usually want anything happening before the document is ready. Then move the code that sends the next request(s) to the callback of the first request after you populate the array and do whatever else you need to happen first
$(document).ready(function () {
$.getJSON("apps.json", function(appsdata) {
...
// add items to your array
sendNextRequest();
}
});
function sendNextRequest() {
//Jquery/Ajax call to the WoW API for the data.
...
}
This gurantees that the calls to the WoW api don't get fired until the first $.getJSON call completes and you populate your array.
FYI this is a common challenge in javascript. You need one operation to run only after another finishes. When you use ajax, you have callbacks like in my example above that help you achieve this. Outside of ajax requests, you can use jQuery Promises to defer tasks until after something else finishes.
My code is as follows:
$(".qtyfield").each(function (index) {
if (this.value != "" && this.value > 0) {
var fieldname = this.id;
countme = countme + 1;
var tmpProductID = fieldname.split("_")
var ProductID = tmpProductID[1];
var ShowPrice = $(this).closest('td').prev('td').text();
var Quantity = this.value;
ShowPrice = ShowPrice.replace("$", "");
isItemInCart(ProductID).done(function () {
if (isItemInCartVar) {
updateQuantityByProductID(ProductID, Quantity);
}
else {
addToCartWithQty(ProductID, ShowPrice, Quantity);
}
});
this.value = '';
}
});
I need to ensure that this code block completes (all ajax calls included) prior to running any further statements.
A couple of points..
isItemInCart is a function with an Ajax Call
updateQuantityByProductID is a function with an ajax call
addToCartWithQty is a function with an ajax call
I don't know how many items are in the each collection (it could change)
My solution would be to consolidate all this logic in a single call, something like updateCart().
On the client side you would create a list of products that need to be updated, e.g.
[
{productId: 123, quantity: 2, price: 123},
{productId: 456, quantity: 1, price: 200}
]
This data gets sent to the server side where the session data will get updated with the new quantities. Basically, the code on the server side will perform the same logic as you have in all your individual calls, but it will be much faster because there's only a single request.
A single request also reduces the likelihood of session lock contention and improves the consistency of your cart state.
On the client side, there's the advantage of only a single callback function instead of having to create a pipe of individual requests that need to be synchronized (read: big pain in the ass).
I'd refactor the code so that the each loop is server side. This way you can prepare the array on the client, send it with ajax (with a unique call) and then unpack it on the server where you'll have the each loop.
What if your each loop is of billions of elements? Multiply that by the length of an html header.
Instead, you could just prepare the array like this.
var serverSide = [];
$(".qtyfield").each(function (index) {
serverSide.push("the data you need");
});
you could then send it up as a JSON string
var myArg = JSON.stringify(serverSide)
or a comma separated list (for simpler structures)
var myArg = serverSide.join(',');
and have a unique ajax call which can return any value/error message
$.post("link", myArg).success(function(data){ alert("data"); })
I don't know which server side support you have but if you're using PHP you may decode the JSON string with json_decode.
If you're using C#, you may want to have a look at JSON.net.
Either way, you'll have much more flexibility like this. On the contrary, while doing an each loop on the client IS possible it is not the best solution. Ajax calls have headers.
Look for jQuery deferreds. Every $.ajax call returns a 'promise' you can wait for to be resolved:
var promise1 = $.ajax(...)
var promise2 = $.ajax(...)
Then wait for them to be resolved:
$.when(promise1, promise2).then(function(){
//ready to rumble
});
As you mentioned you don't know how many items are in each collection, simply push each promise into an array, which can be used in $.when as follows:
var promises = [];
promises.push($.ajax(...));
$.when.apply(this, promises).then(function(){...});
I'm accessing a json file which has 50 entries per page over x amount of pages.
I have the total number of entries, say 500 - which amounts to 10 pages.
I get the data from json file for page 1, pass the data to an array and then repeat the function but this time for page 2.
I have created the function and it loops perfectly incrementing and fetching each page, but it doesn't wait for the json data to be parsed and passed to the array before looping again.
Basically I want to wait until the data has been processed and then continue on.
My code so far is roughly this:
function getJsonData(metroID){
currentPageNo = 0;
totalPages = 'x';
count = 0;
function jsonLoop(){
meroAreaSearchString = 'http://jsonurl'+currentPageNo;
$.getJSON(meroAreaSearchString,{},function( data ){
if(totalPages == 'x'){
var totalEntries = data.resultsPage.totalEntries;
var perPage = data.resultsPage.perPage;
totalPages = (totalEntries/perPage);
log(totalEntries+', '+perPage+', '+totalPages);
log(Math.round(totalPages));
}
$.each(data.resultsPage.results.event, function(i,item){
var name = item.displayName;
var type = item.type;
var valueToPush = new Array();
valueToPush[0] = name;
valueToPush[1] = type;
valueToPush[3] = count;
locations.push(valueToPush);
count++;
});
});
if(currentPageNo == totalPages){
log(locations);
alert('finished processing all results');
}else{
currentPageNo++;
jsonLoop();
}
currentPageNo++;
jsonLoop();
}
}
Have you tried making the request syncronous?
Just put this piece of code at the top of your function getJsonData
$.ajaxSetup({async:false});
You can specify the async option to be false to get a synchronous Ajax request. This will stop your function until the callback set some data.
The $.getJSON() function fires off an AJAX request, and calls it's callback function when the AJAX call resolves successfully, if that makes any sense.
Basically, that just means that given a call $.getJSON(url,data,callback);, jQuery will fire an AJAX request to url passing data along with it, and call callback when that call resolves. Clear cut straightforward.
The thing you're missing here is that an AJAX call is just that -- as its name implies, its asynchronous. This means that throughout the whole lifetime of the AJAX call, it lets the other logic in your application run instead of waiting for it to finish.
So something like this:
$.getJSON(url, data, callback);
alert('foo');
... will most probably result in an alert() call happening before your AJAX call completes. I hope that made sense.
To make sure that something happens after your AJAX call completes, you put that logic inside the callback. That's really what the callback is for.
$.getJSON(url, data, function (d) {
something_you_want_done_after_ajax_call();
});
In the context of your problem, you just have to put all that conditional recalling of jsonLoop() into your callback. It's not very obvious right now because of your indenting, but it's currently outside your callback.