I'm running into an issue where large data sets sent over POST using AJAX are not making it to the $_POST variable in PHP.
The user uploads an Excel/CSV file through a webform and is parsed via AJAX. This particular example has 775 records with 13 fields/elements each. Adding additional fields being submitted and there are less than 11,000 elements in the dataset. From the research I've done on the subject, 32-bit browsers (i.e. Firefox, Chrome, etc.) should be able to handle 4.29 billion elements, so I don't see the data size as an issue, especially as the response from the file upload contains all the elements.
The issue only rears its head when the entire form is submitted to be validated and entered into the database. The issue is that the console on both Firebug and Chrome Developer Tools shows that the whole data set is submitted:
Doing a var_dump on the $_POST gives this:
The php.ini has 'post_max_size' set to 200M. Even 'upload_max_filesize' is set to 100M. This issue occurs in both Firefox 32.0.3 and Chrome 37.0.2062.103 m that I have tested personally and other older versions (including IE 10) that UAT has tested.
The AJAX call is:
new wrapper.ajax('/t1.php', {
type: 'POST',
data: data,
form: $('form[name=oppForm]'),
success: function (response)
{
if (response.result)
{
window.location = response.result;
}
},
complete: function ()
{
$("#submit").loading('done');
}
});
The PHP is:
<?php
var_dump($_POST);
Any thoughts?
EDIT
After talking with some other developers, I also checked the output of php://input and found that it DID contain the entire POST data that the browsers were sending, but that the data was not getting translated into $_POST properly. However, it does work properly if I remove 10 keys from the post data, and submit 765 instead of 775.
The issue ended up being that 'max_input_vars' in the php.ini file was not set high enough. The value was set to 10,000 and the user was submitting data of near 11k, thus some of it was getting truncated. Changing this value to be greater is what solved the issue.
Related
I am running a server on Django, one function takes a seed through a URL param GET request, generates some data based on that seed, and sends it back.
The URL format:
mysite.com/api/generate/<seed>
expected result:
submitting a GET on mysite.com/api/generate/99 gets picked up in Django as a seed value of 99. data returned is chosen with random.choice() seeding with random.seed(99) from a database which contains a single column of names. data returned is the following:
Walker Lewis
Dalia Aguilar
Meghan Ford
Theresa Hughes
Kenna Coffey
Kendra Ho
problem
Here's where I'm getting confused (code below for each):
1000 requests in postman, all 1000 return perfectly equal
approx 100 requests from the google chrome console, all are equal
from the generate.js that the server sends with index.html, making the same call, results degenerate (examples below)
Postman call
very simple, GET mysite.com/api/generate/99
jquery from chrome console
$.ajax({
url: "/api/generate/99",
success: function( result ) {
console.log(result.data)
}})
jquery from generate.js
$.ajax({
url: "/api/generate/99",
success: function( result ) {
var data = result.data;
// data is now passed about the script, but debugging at the line above shows that data has already started to vary on a request by request basis
Both Postman and Chrome Console will return the expected results:
Walker Lewis
Dalia Aguilar
Meghan Ford
Theresa Hughes
Kenna Coffey
Kendra Ho
generate.js:
The first two names are always correct
The third is correct the majority of the time
fouth, 20% at best (estimate)
Anything past the forth might as well not be seeded, it just seems to be chosen at random from the database
other information
I have confirmed that each request from each source is being sent and received from the server, and not from a cache
Confirmed that all sources are hitting the same server, in the same state, and the same database
If anyone has any advice on this it would be very appreciated.
So it turned out this was due to me sending multiple AJAX requests "at once". When Django was behind Gunicorn/nginx each request gets a worker and is processed correctly. when requesting directly to docker the front end is returned with the strange data.
Background:
I am making a simple game in PHP, JavaScript and HTML for the web. A player control movements of a box on the screen, and see others fly around with their boxes.
I have the following files, that I upload to my domain via a hosting company:
index.html: a file with some buttons (eg. to start the game) and frames (for putting boxes in).
server.php: PHP script that receives messages from client, performs reads/writes to a database, echoes (using echo) boxes from database to the client. Does not echo the box of the player the message came from.
database.txt: a JSON text file containing data of players and the next free ID number. When empty it looks like this: {"players":[], "id": 1}. players contain objects with values such as ID, position and rotation.
script.js: JavaScript file with script to send/receive messages, display data from messages etc. Linked to index.html. Moves your box.
A screenshot, two players in movement:
Problem: The game crashes, always. Sooner or later. This is what happens:
Client recevies player data from server.php, everything is fine. This could be for 10 seconds or up to some minutes.
The data starts to falter, the message sometimes is null instead of actual data.
The data recevied is always null. The database file is now {"players":null,"id":5}. (The "id" could be any number, does not have to be 5).
Picture of data flow, printing of players from database. Two players. Before this screenshot lots of rows with valid data. Then as seen two null messages. Then after a while null forever.
I am not completely sure where the problem is, but I am guessing it has to do with my read/write in server.php. I feels like a lot of player movement makes the program more likely to crash. Also how often the program sends data affetcs.
Code Piece 1: This is code from server.php, that writes to the database. I have some sort of semaphore (the flock( ... ) ) to prevent clients from reading/writing at the same time (causing errors). I have an other function, read, which is very similar to this. Possible problems here:
The semaphore is incorrect.
The mode for fopen() is incorrect. See PHP docs. The mode w is for write. The tag b is for "If you do not specify the 'b' flag when working with binary files, you may experience strange problems with your data ...".
Something weird happening because I use read() in my writing function?
Code:
// Write $val to $obj in database JSON
function write($obj,$val){
$content = read();
$json = json_decode($content);
$json->{$obj} = $val; // eg. $json->{'id'} = 5;
$myfile = fopen("database.txt", "wb") or die("Unable to open file!");
if(flock($myfile, LOCK_EX|LOCK_NB)) {
fwrite($myfile,json_encode($json));
flock($myfile, LOCK_UN);
}
fclose($myfile);
}
Code Piece 2: This is my code to send data. It is called via a setInterval(). In script.js:
// Send message to server.php, call callback with answer
function communicate(messageFunc,callback){
var message = messageFunc();
if (window.XMLHttpRequest) {
var xmlhttp=new XMLHttpRequest();
}
xmlhttp.onreadystatechange= function() {
if (this.readyState==4 && this.status==200) {
callback(this.responseText);
}
}
xmlhttp.open("GET","server.php?msg="+message,true);
xmlhttp.send();
}
This is my code to receive data, in server.php: $receive = $_GET["msg"].
My current work of solving
This is what I have done so far, but nothing has changed:
Added mode b to fopen().
Added flock() to read/write functions in server.php.
Much reworking on script.js, I would say it looks/works very clean.
Check memory_get_peak_usage(), and check with the hosting company for memory limits. Should be no problem at all.
Looked at PHP garbage collecting and gc_enable() (I don't know why that would change anything).
Lots of testing, looking at the data flow.
Crying.
Conclusion: Is this type of application what PHP is for? What do you think is wrong? If you want more code/info I provide. Thank you very much.
Here is the root of your problem:
$myfile = fopen("database.txt", "wb") or die("Unable to open file!");
Note the behavior of the w open mode (emphasis mine):
Open for writing only; place the file pointer at the beginning of the file and truncate the file to zero length. If the file does not exist, attempt to create it.
This happens before you lock the file. What's happening is that between this fopen() call and the following flock() call, the file's content is zero length, and a reader is coming along during that time and reading the empty file.
Why doesn't this cause an error in PHP when you parse the empty string as JSON? Because json_decode() is defective, and returns null when the input is not valid JSON rather than throwing an exception. Nevermind that the string "null" is valid JSON -- json_decode() gives you no way to differentiate between the cases of valid input representing the null value and invalid input. If json_decode() actually threw an exception or triggered a PHP error (don't ask me why two error-signalling mechanisms are necessary in PHP), you would have a fantastic point to start debugging to figure out why the file is empty, and you might have solved this problem by now!
... sigh ...
PHP's "design" gives me headaches. But I digress.
To fix this whole problem, change the open mode to "cb" and ftruncate($myfile, 0) after you successfully acquire the lock.
Note the behavior of the c mode, which actually specifically mentions the approach you are using (emphasis mine):
Open the file for writing only. If the file does not exist, it is created. If it exists, it is neither truncated (as opposed to 'w'), nor the call to this function fails (as is the case with 'x'). The file pointer is positioned on the beginning of the file. This may be useful if it's desired to get an advisory lock (see flock()) before attempting to modify the file, as using 'w' could truncate the file before the lock was obtained (if truncation is desired, ftruncate() can be used after the lock is requested).
I have a Django web app which has various instances of ck-editor instances on a web page.By using on blur event I am saving whole of data of the instance in the database -
{% for editor in editors %}
CKEDITOR.appendTo("{{editor.ck_id}}" ,
{
on: {
blur: function(event){
var data = event.editor.getData();
console.log("data of {{editor.ck_id}} is " + data);
var request = $.ajax({
url: "/editor/save/",
type: "GET",
data: {
content : data,
content_id : "{{editor.ck_id}}"
},
dataType: "html"
});
}
}
}
// "{{editor.data}}"
);
CKEDITOR.instances['{{editor.ck_id}}'].insertHtml("{{editor.data}}");
Here ck_id and data are the two database fields of ckeditors.Now suppose I write this on one instance of ckeditor -
Tony Stark
When I lose focus on that instance then on blur event is fired and I use 'getData()' to get the html data of that instance.
<p>Tony Stark</p>
is saved into the database.On the python interpreter now when I fetch editor's data it shows -
<p>Tony Stark</p>
which is obvious.
Now When I re start the server and set the data of every ck-editor instance again then this exception is raised -
Uncaught SyntaxError: Unexpected token ILLEGAL
I know why this is happening - due to this -
CKEDITOR.instances['ck_1'].insertHtml("<p>Tony Stark</p>
");
The data which I sent by fetching from database was -
<p>Tony Stark</p>
and it somehow got converted to above mentioned text with illegal tokens.I have tried to use setData() but with no result. Do I need to encode/decode this HTML or something?
Now my question is how to again reset data to a ck-editor instance which I fetched from it and stored in database. I have posted this question on several forums but many people have got no clue of what I am asking?
Is there anyone here who has tried to do same thing and have succeeded in it?
Thanks
PS: Adrian Ghiuta solution seems to be working but there is just problem. When the editor is loaded first time then in google chrome's debugger inserted line is seen as -
"<p>Tony Stark</p>"
which is rendered due to line "{{editor.data|safe}}".But when I change my editor's content to just
Tony
then in database
<p>Tony</p>
is being saved but when I restart the server it does not render and throws this error
Unexpected ILLEGAL TOKEN
due to this -
"<p>Tony</p>
"
Here ending double quotes are in the next line.But during initial data loading it was on the same line.Might this be causing be any problem?
Because my chrome console shows error at that position.
I will add some images to show the condition.
Initial loading -
Initial console -
Editor after editing -
Error being shown after restarting server and reloading editors -
Console -
You can see how error is thrown at line where double quotes are at the next line.Do I to escape html or something?
Sorry for my naivety but I do not have much command over HTML.
I am using this plugin to enable me to display what I am currently listening to live on my website: https://github.com/derekmartinez18/Simple-Ajax-Spotify-Now-Playing
A piece of JavaScript will run on page load, which connects to the PHP which via API checks what I am listening to, grabs the title etc, if successful sends back to the JavaScript and displays via HTML.
I've echoed out what the PHP is getting and can see it's correctly grabbing all the recent songs, names etc. Therefore because nothing is appearing on the page maybe it's the JavaScript. I've pasted it below with pastebin link to php too.
JavaScript
<script type="text/javascript">
function get_spotify() {
$.ajax({
type: 'POST',
url: '/Scripts/last.fm.php',
data: { request: 'true' },
success: function(reply) {
$('.now-playing').html("<p>" + reply + "</p>");
}
});
}
window.onload = get_spotify;
</script>
Pastebin of PHP - http://pastebin.com/eZUH6BNU
A snippet of the API output which is being past over to the PHP (it's huge so didn't paste it all.):
{"recenttracks":{"track":[{"artist":{"#text":"LMFAO","mbid":"ed5d9086-e8cd-473a-b96c-d81ad6c98f0d"},}
Live Link - http://bit.ly/1ewTe8l
Based on the example you have given below your question, the problem is that there is no element with a class of now-playing.
Your page seems to have been cut short, perhaps due to an error, but the ajax request fires and in the console I can see the response (the url is actually echoed before that as well):
I am currently listening to 01. Circles Around The Sun by Dispatch on Spotify.
Adding a <div class="now-playing"></div> should do the trick.
Edit: Note that your key is visible in the response as you are echoing the url you are making a request to. It might be a good idea to change the key when you have solved your problem.
I got this javascript code in a JSP:
$('#myForm').submit(function(event) {
event.preventDefault();
var act = $('#myForm').attr('action') + '&act=' + $('#submitBtn').val();
var dataStr = $("#myForm").serialize();
$.post(act, dataStr, function(data, status) {
$('body').html(data);
});
return false;
});
The facts:
I'm using jquery v1.3.x (and I can't replace with newer version because of company policy).
The jsp is in a pop-up window.
There is a submit button with id=submitBtn
When the submit button is clicked, it will execute above code and send the data into struts action.
The action url:
someAction.do?someParam=blabla&someNo=213354345&mode=POP&act=Update Your Data
(note: I had replaced some sensitive data, but the argument pattern is same)
The problem is at this line:
$('body').html(data);
It produces Error: NS_ERROR_FAILURE: Failure in error console. I heard it's related with cross domain request, but my application is at localhost, how could it happened?
In IE 8, it successfully replaced the current page content with the data value (which is html string). But in firefox (v22), it gave blank page, I checked with right click -> View Source, it's really empty. Then I checked with firebug, the response data indeed contains the result html page string. If I debug with alert(data); before the $('body').html() call, it also show the html string.
So the $.post() function successfully returns the response data but it seems the $('body').html() function failed to render the data.
So what is actually happened? I'm really have no clue.
What is the solution for this?
UPDATE:
I checked the error console, I got this error after submitted the page:
Error: NS_ERROR_FAILURE: Failure
Source File: http://localhost:8080/myapp/script/jquery.js
Line: 19
UPDATE 2:
I debug the java code, I found out that the input html tag in the JSP that has indexed array name like this:
myItem[0].name
myItem[1].name
...
were not successfully submitted to the ActionForm class. I investigated the source of problem might come from a 3rd party javascript library.
try something like this
$.post(act, dataStr, function(data) {
document.body.innerHTML = data;
});