Sending large javascript array to Web API - javascript

I am struggling with this issue for 2 days...
I have a JavaScript array (20,000K rows and 41 columns). It was originally received in javaScript through an ajax call as shown below,
var dataArray = [];
var dataRequest = {};
dataRequest.SearchCondition = 'some value';
$.ajax({
type: "POST",
url: "api/GetData/ProcessRequest",
dataType: 'json',
cache: false,
async: true,
crossDomain: false,
data: dataRequest ,
success: function (response) {
dataArray = response;
},
error: function (xhr, textStatus, errorThrown) {
dataArray = null;
}
});
In the application, the user will verify the data and send it back to Web API method.
I am trying to send the same data back (dataArray) to web api method but, it fails. Please see the code below,
Option 1: (failed - the request did not hit web api method)
var dataArrayJsonStr = JSON.stringify(dataArray);
$.ajax({
type: "POST",
url: "api/SendData/ProcessRequest",
dataType: 'json',
data: {'dataValue':dataArrayJsonStr },
success: function (response) {
alert('success');
},
error: function (xhr, textStatus, errorThrown) {
alert(errorThrown)
}
});
In IE 8, I am getting 'out of memory' exception popup. (most of our application users still have IE 8)
In Chrome, it crashes.
Option 2 tried: (don't know how to read the value)
I tried to send the same value to web api through XmllHttpRequest
var dataArrayJsonStr = JSON.stringify(dataArr);
var xmlRequest;
if (window.XMLHttpRequest) {
xmlRequest = new XMLHttpRequest();
}
xmlRequest.open("POST", "api/SendData/ProcessRequest", false);
xmlRequest.setRequestHeader('Content-Type', 'application/text');
xmlRequest.send("dataValue=" + dataArrayJsonStr);
Using Chrome, I am able to post the data successfully to Web API, I am seeing the content-length as '128180309'. But, I don't see the values. How do i get the values in Web API?
Please suggest me how to send large data back to web api from javascript.
Thanks,
Vim

I think you create overhead, maybe I wrong, you can edit me.
Did you really need send back all datas back or you just need send modified data?
Because in real life hard to imagine that user will review 20.000 of rows.
Good example is ExtJS stores, you can see example here
Key thing of stores that they send back to the server only modified or deleted data, it save browser, network and server resources.

Try to add more memory for API or more time excecution, also you can try return data in more small parts. Defining the number of parts to send.

Did you try to send the data by chunks?
I mean, you need to split it in small pieces and perform multiple number of requests.
For example, it can be like:
--HELLO SERVER. STARTING TRANSMITION FOR DATA SET #177151--
PIECE 1/13
PIECE 2/13
...
PIECE 13/13
--BUE SERVER--
So, it will take some time, but you can send any amounts of data without memory problems. If you're struggling with it for 2 days, I think you got some time to code it :)
UPD1: Client code example.
Here's an example of client code. This is a simple chunking algorithm.
Have to say I didn't test it, because it would take a lot of time to represent your situation.
So, you should read it and get the point.
You have a simple function, that takes you whole data set and callbacks for each response (to update your progress bar, e.g.), for successful finish and for error.
Hope, it will help you to make some problems.
Also, I can help you to build architecture on the server-side, but I need to know what technologies do you use.
function sendData(data, onEach, onFinish, onError) {
var CHUNK_SIZE = 1000;
var isFailed = false;
var chunkNum = 0;
var chunk, chunkStart, chunkEnd;
while(data.length + CHUNK_SIZE > chunkNum * CHUNK_SIZE) {
if(isFailed) {
return;
}
chunkStart = chunkNum * CHUNK_SIZE;
chunkEnd = chunkStart + CHUNK_SIZE + 1;
chunk = {
num: chunkNum,
data: data.slice(chunkStart, chunkEnd)
};
ajaxCall(chunk);
chunkNum++;
}
function ajaxCall(data) {
$.ajax({
type: "POST",
url: "api/GetData/ProcessRequest",
dataType: 'json',
async: true,
data: dataRequest ,
success: function (response) {
onEach(data, response);
},
error: function (xhr, textStatus, errorThrown) {
isFailed = true;
onError(arguments);
}
});
}
}

Related

AJAX request every 1 second without blocking the webpage, how to approach it

I am making a web app that will control a device that I have. The device has its own webserver (of which I have no control of) and it has its own language.
The way that I am controlling it is by doing ajax GET requests that are executing scripts on the device.
For example "http://127.0.0.1/nameOfScript?varName=varValue
I can also get data from the device through JSON, for example http://127.0.0.1/GetValuesJSON and it sends back the json object that I have created, so far so good.
My problem is that I want to continuously read data from the device and be able to send data at the same time, so I need to run the second URL every 0.5/1second, and be able to run the first URL whenever I want, the device and the webpage will be on a local network, security is not of importance.
Is it possible to do this within the HTML/JavaScript, if so, how? What do I have to research? Or do I need to use an additional webserver for the webpage that will handle this?
#abhinavxeon
^ Gave a solution to the problem, someone downvoted him and the comment was deleted, but the solution works and does exactly what I want it to do!
Here is the solution:
var interval;
function doSomething() {
$.ajax({
type: 'GET',
url: 'someurl',
data: $(this).serialize(),
dataType: 'json',
success: function (data) {
$("#content").text(data)
interval = setTimeout(doSomething, 1000);
}
});
}
doSomething();
function yourfuncation() {
$.ajax({
type: 'GET',
url: 'GETURL',
success: function (data) {
$('#hidden').val(data);// first set the value
}
});
}
use setinterval
var interval = setInterval(yourfunction, 1000);
or you can go with this way
var interval;
function callAjax() {
$.ajax({
type: 'POST',
url: 'increment.php',
data: $(this).serialize(),
dataType: 'json',
success: function (data) {
$('#hidden').val(data);// first set the value
interval = setTimeout(callAjax, 1000);
}
});
}
callAjax();
So you need to poll the server. The way you normally would do it is with websockets. If you can not do websockets, then you have to rely on polling. To do that, you need to use setTimeout or setInterval.
Basic idea to poll a server would be
function grabData () {
fetch('/your/endpoint/')
.then((response) => {
if (!response.ok) {
throw new Error('Something bad happened: ' + response.statusText );
}
return response.json(); // or text() if it is not JSON
})
.then((myJSON) => {
console.log(myJSON)
window.setTimeout(grabData, 2000) // call again after a delay
.catch((error) => {
console.error('There was a problem', error);
});
}
And you can make another function that does the update and call that when you need to send data up.

Ajax url is undefined

I'm making a function to get a xml file and edit it. I've never done that before so I searched a good way to get an xml file. I decided to use ajax, but the file is never returned because the url is undefined.
EDIT :
I edited the code and made the treatment in the success function. Now there is no problem with this file.
Here is the update of the ajax part :
$.ajax({
type: 'GET',
url: 'allrtp.xml',
dataType: 'xml',
success: function(xml) {
//file = $.parseXML(xml);
// Editing the file to have the good dates
$(xml).find('StartDateTime').text(start);
$(xml).find('EndDateTime').text(end);
var strFile;
if (window.ActiveXObject) {
strFile = xml.xml;
} else {
strFile = (new XMLSerializer()).serializeToString(xml);
}
var encoded64 = Base64.encode(strFile); // Encoded in base64
var encodeURL = encodeURIComponent(encoded64); // Encoded URL
var AR = urlAR + encodeURL; // The URL to open
window.open(AR, '_blank');
}
})
Now all is working well about the xml file, I have a little problem with the window.open, which open my url but with %31 at the beggining, but it's another problem.
Thank you for your help !
file is undefined because you are declaring it inside a ajax success function
function openRecords(start, end) {
// Extraction of the xml file
var file;
$.ajax({
type: 'GET',
url: 'allrtp.xml',
dataType: 'xml',
success: function(xml) {
file = $.parseXML(xml);
},
error: function(ex) {
console.log(ex);
}
})
// Test
var start = '2016-02-15T12:57:00+01:00';
var end = '2019-02-16T13:57:00+01:00';
setTimeout(function(){
// Editing the file to have the good dates
file.find('StartDateTime').text(start);
file.find('EndDateTime').text(end);},1500);
}
Add an error callback:
error: function (ex) {}
Many things can be happening, you will get more info with the error callback. Probably you are querying an incorrect url. Do not trust that undefined upon url, see what returns your jquery ajax function. Maybe you should be querying something like '\files\xxx.xml'.
can you give me a picture of Network in your broswer? I want to know the URL is send or not:
1. F12 open your console
2. select the Network tab
3. refresh the broswer
4. check the request is send or not

JQuery Ajax limiting deep recursion "too much recursion"

The idea is that I send lots of synchronous requests to an API to create a JSON that I will need later to do some clusterizations. This API gives me information about articles, review etc. from a site (scopus.com). First I send a request based on a query from which I get a JSON which contains information about some articles. These articles are cited by other articles. I have to get information about these ones too so I need recursion. The problem is that I get an error because of "too much recursion". It seems that the error appears when the recursion is over and the program has to go back to the "root"/the first call. So the program will look like a very deep tree.
Pure Javascript does have this limitation too? What can I do?
Also I have to do SYNCHRONOUS requests otherwise the JSON I will get will be a mess.
EDIT:
I tested the script on queries that need a small recursion such as a tree with 4-5 levels.
var result = '{"entry":[ ';
function query(){
var results = getNumberResults();
if(results>0)
{
var pages = Math.ceil(results/25);
var i;
for(i=0; i<pages; i++){
$.when($.ajax({
url: url,
type: "GET",
async: false,
headers: {'Accept':'application/json'},
success: function(data){
$.each( data['search-results']['entry'], function( i, item ) {
get info from json and save it in my variable
if(data['search-results']['entry'][i]['citedby-count'] > 0)
getCitedBy(data['search-results']['entry'][i]['eid']);
else{
result += '"children-id":[]},';
}
});
}
}));
}
}
result = result.slice(0,-1);
result += "]}";
}
function getCitedBy(eid){
var results = getCitedByNumberResults(eid);
if(results>0)
{
var pages = Math.ceil(results/25);
var i;
for(i=0; i<pages; i++){
$.when($.ajax({
url: url,
type: "GET",
async: false,
headers: {'Accept':'application/json'},
success: function(data){
$.each( data['search-results']['entry'], function( i, item ) {
get info from json and save it in my variable
if(data['search-results']['entry'][i]['citedby-count'] > 0)
getCitedBy(data['search-results']['entry'][i]['eid']);
else{
result += '"children-id":[]},';
}
});
}
}));
}
}
}
function getNumberResults(){
var innerResult;
$.ajax({
url: url,
type: "GET",
async: false,
headers: {'Accept':'application/json'},
success: function(output){
innerResult = output['search-results']['opensearch:totalResults'];
},
error: function (xhr, ajaxOptions, thrownError) {
innerResult = 0;
}
});
return innerResult;
}
function getCitedByNumberResults(eid){
var innerResult;
$.ajax({
url: url,
type: "GET",
async: false,
headers: {'Accept':'application/json'},
success: function(output){
innerResult = output['search-results']['opensearch:totalResults'];
},
error: function (xhr, ajaxOptions, thrownError) {
innerResult = 0;
}
});
return innerResult;
}
The problem was as trincot mentioned and I also thought so was that 2 or more articles are referencing each other making an infinite cycle. I fixed it by searching my string variable for the unique identifier. So if in my variable already exists and article with that identifier I will skip recursion for the current article.
So I tested my script again for relative short queries (few hundreds of articles returned) because there are queries with huge outputs (millions of articles). When I will search for big queries I could come upon string size limitations(browser specific) or even “too much recursion”. If so I will let you know.
Advice: if “too much recursion” error occurs in your ajax request search first for an infinite cycle because this is the most probable cause.

ASP.NET MVC JsonResult return 500

I have this controller method:
public JsonResult List(int number) {
var list = new Dictionary <int, string> ();
list.Add(1, "one");
list.Add(2, "two");
list.Add(3, "three");
var q = (from h in list where h.Key == number select new {
key = h.Key,
value = h.Value
});
return Json(list);
}
On the client side, have this jQuery script:
$("#radio1").click(function() {
$.ajax({
url: "/Home/List",
dataType: "json",
data: {
number: '1'
},
success: function(data) {
alert(data)
},
error: function(xhr) {
alert(xhr.status)
}
});
});
I always get an error code 500. What's the problem?
Thank you
If you saw the actual response, it would probably say
This request has been blocked because
sensitive information could be
disclosed to third party web sites
when this is used in a GET request. To
allow GET requests, set
JsonRequestBehavior to AllowGet.
You'll need to use the overloaded Json constructor to include a JsonRequestBehavior of JsonRequestBehavior.AllowGet such as:
return Json(list, JsonRequestBehavior.AllowGet);
Here's how it looks in your example code (note this also changes your ints to strings or else you'd get another error).
public JsonResult List(int number) {
var list = new Dictionary<string, string>();
list.Add("1", "one");
list.Add("2", "two");
list.Add("3", "three");
var q = (from h in list
where h.Key == number.ToString()
select new {
key = h.Key,
value = h.Value
});
return Json(list, JsonRequestBehavior.AllowGet);
}
While JustinStolle's answer solves your problem, I would pay attention to the error provided from the framework. Unless you have a good reason to want to send your data with the GET method, you should aim to send it with the POST method.
The thing is, when you use the GET method, your parameters gets added to your request url instead of added to the headers/body of your request. This might seem like a tiny difference, but the error hints why it's important. Proxy servers and other potential servers between the sender and the receiver are prone to logging the request url and often ignore the headers and/or body of the request. This information is also often regarded as non important/secret so any data exposed in the url is much less secure by default.
The best practice is then to send your data with the POST method so your data is added to the body instead of the url. Luckily this is easily changed, especially since you're using jquery. You can either use the $.post wrapper or add type: "POST" to your parameters:
$.ajax({
url: "/Home/List",
type: "POST",
dataType: "json",
data: { number: '1' },
success: function (data) { alert(data) },
error: function (xhr) { alert(xhr.status) }
});

Save temporary Ajax parameters in jQuery

I am developing a heavily scripted Web application and am now doing some Error handling. But to do that, I need a way to access the AJAX parameters that were given to jQuery for that specific AJAX Request. I haven't found anything on it at jquery.com so I am asking you folks if you have any idea how to accomplish that.
Here is an example of how I want to do that codewise:
function add_recording(filename) {
updateCounter('addRecording','up');
jQuery.ajax({
url: '/cgi-bin/apps/ajax/Storyboard',
type: 'POST',
dataType: 'json',
data: {
sid: sid,
story: story,
screen_id: screen_id,
mode: 'add_record',
file_name: filename
},
success: function(json) {
updateCounter('addRecording','down');
id = json[0].id;
create_record(id, 1, 1, json);
},
error: function() {
updateCounter('addRecording','error',hereBeData);
}
})
}
hereBeData would be the needed data (like the url, type, dataType and the actual data).
updateCounter is a function which updates the Status Area with new info. It's also the area where the User is notified of an Error and where a Dismiss and Retry Button would be generated, based on the Info that was gathered in hereBeData.
Regardless of calling complete() success() or error() - this will equal the object passed to $.ajax() although the values for URL and data will not always be exactly the same - it will convert paramerters and edit the object around a bit. You can add a custom key to the object to remember your stuff though:
$.ajax({
url: '/',
data: {test:'test'},
// we make a little 'extra copy' here in case we need it later in an event
remember: {url:'/', data:{test:'test'}},
error: function() {
alert(this.remember.data.test + ': error');
},
success: function() {
alert(this.remember.data.test + ': success');
},
complete: function() {
alert(this.remember.data.url + ': complete');
}
});
Of course - since you are setting this data originally from some source - you could rely on the variable scoping to keep it around for you:
$("someelement").click(function() {
var theURL = $(this).attr('href');
var theData = { text: $(this).text(); }
$.ajax({
url: theUrl,
data: theData,
error: function() {
alert('There was an error loading '+theURL);
}
});
// but look out for situations like this:
theURL = 'something else';
});
Check out what parameters you can get in the callback for error.
function (XMLHttpRequest, textStatus, errorThrown) {
// typically only one of textStatus or errorThrown
// will have info
this; // the options for this ajax request
}
You can use the ajax complete event which passes you the ajaxOptions that were used for the request. The complete fires for both a successful and failed request.
complete : function (event, XMLHttpRequest, ajaxOptions) {
//store ajaxOptions here
//1 way is to use the .data on the body for example
$('body').data('myLastAjaxRequest', ajaxOptions);
}
You can then retireve the options using
var ajaxOptions = $('body').data('myLastAjaxRequest');

Categories