I know there are existing some Questions about Chunking a mysql array in php, but my problem is, that I want to keep the output in JSON.
Scenario:
I want to get data from mysql, do some stuff with it ( like time formatting ) and output it in JSON.
The JSON data is parsed in the browser and visualized over a javascript chart.
Problem:
All of the above is working, but because of the huge amount of data, I'm getting an out of memory error, when I select bigger date ranges to output.
The Idea of directly sending out each x-lines of data is not working because of the JSON format it needs to be. Several JSON chunks won't work, it needs to be one for the chart.
So in the end I need to chunk the data but keep it as one big JSON.
(And setting up the memory limit is not really a solution.)
Ideas:
One Idea would be, to let the browser chunk the date range and ask the data as chunks & then put them together.
Of course this would work, but if there is a way to do this server side, it would be better.
Code:
private function getDB($date1, $date2){
$query = 'SELECT * FROM `db1`.`'.$table.'` WHERE `date` BETWEEN "'.$date1.'" AND "'.$date2.'" order by `date`;';
// date = datetime !
$result = $this->db->query($query);
$array = array();
while ( $row = $result->fetch_assoc () ) {
$array[] = array( strtotime($row[ 'date' ])*1000 , (float)$row[ 'var' ] );
// the formatting needs to be done, so the chart accepts it..
}
$result->close();
return json_encode($array);
}
Since this is not an option,
ini_set("memory_limit","32M")
perhaps you can add LIMIT to the function paramaters and query:
private function getDB($date1, $date2, $start, $pageSize){
$query = 'SELECT * FROM `db1`.`'.$table.'` WHERE `date` BETWEEN "'.$date1.'" AND "'.$date2.'" order by `date` LIMIT $start, $pageSize;';
// date = datetime !
$result = $this->db->query($query);
$array = array();
while ( $row = $result->fetch_assoc () ) {
$array[] = array( strtotime($row[ 'date' ])*1000 , (float)$row[ 'var' ] );
// the formatting needs to be done, so the chart accepts it..
}
$result->close();
return json_encode($array);
}
Then setup a for loop in javascript, call this with Ajax, incrementing the $start variable each time.
Store each responseText.substr(1).substr(-1) in an array.
When the responseText is "", all of the records have been returned.
.join the array with a comma, then add a new opening and closing "{ }", and you should have a JSON equivalent to all records.
Minimal parsing, and you'll be using built-in functions for most of it.
var startRec=0;
var pageSize=50;
var xmlhttp=new XMLHttpRequest();
var aryJSON=[];
var JSON;
xmlhttp.onreadystatechange=function(){
if (xmlhttp.readyState==4 && xmlhttp.status==200){
if(xmlhttp.responseText==""){ //Might need to check for "{}" here instead of ""
//All records are received
JSON="{" + aryJSON.join(",") + "}";
aryJSON=[];
startRec=0
}else{
aryJSON.push(xmlhttp.responseText.substr(1).substr(-1));
startRec+=pageSize;
getNextPage();
}
}
}
function getNextPage(){
xmlhttp.open("GET","your.php?start=" + startRec + "&pageSize=" + pageSize,true);
xmlhttp.send();
}
I would recommend that you have the server send the browser exactly what it needs to create the table. Parsing can be a heavy task, so why have the client do that lifting?
I would have your backend send the browser some kind of data structure that represents the table (i.e. list of lists), with all the formatting already done. Rendering the table should be faster and less memory-intensive.
One way of answer would be, to do the chunking on the server, by giving out the JSON, removing the leading [ & ].
#apache_setenv('no-gzip', 1);
#ini_set('zlib.output_compression', 0);
#ini_set('implicit_flush', 1);
$array = array();
echo '[';
$started = false;
while ( $row = $result->fetch_assoc () ) {
$array[] = [ strtotime($row[ 'datetime' ])*1000 , (float)$row[ 'var' ] ];
if(sizeof($array) == 1000){
if($started){
echo ',';
}else{
$started = true;
}
echo substr(substr(json_encode($array),1), 0, -1);
// converting [[datetime1, value1],[datetime2, value2]]
// to [datetime1, value1],[datetime2, value2]
ob_flush();
$array = array();
}
}
if($started)echo ',';
$this->flushJSON($array);
echo ']';
flush();
$result->close();
This is working and reducing the ram usage to 40%.
Still it seems that Apache is buffering something, so the ram usage increases over the time, the script is running. (Yeah, the flush is working, I debugged that, that's not the problem.)
But because of the remaining increase, the fastest way to achieve a clean chunking is to do this like alfadog67 pointed it out.
Also, to mention it, I had to disable the output compression, otherwise apache wouldn't flush it directly..
Related
currently, im having problem to parse xml node in array using condition where parse with <mo> as separator
this is my array(0)
Array([0] => <mi>x</mi><mo>+</mo><mn>2</mn><mo>=</mo><mn>3</mn>);
i want to parse like this
Array[0] => <mi>x</mi>
Array[1] =><mo>+</mo><mn>2</mn>
Array[2]=><mo>=</mo><mn>3</mn>
this is my coding
<?
$result(0)="<mi>x</mi><mo>+</mo><mn>2</mn><mo>=</mo><mn>3</mn>";
$result1= new simplexml_load_string($result);
$arr_result=[];
foreach($result1 as $key => $value){
$exp_key = explode('<', $key);
if($key[0] == 'mo'){
$arr_result[] = $value;
}
print_r($arr_result);
}
if(isset($arr_result)){
print_r($arr_result);
}
?>
thanks in advance !
The approach with XML seems excessive since what you really want is to pull out substrings of a string based on a delimiter.
Here is a working example. It works by finding the position of <mo> and cutting off that section, then searching for the next <mo> in the remain string.
<?php
$result(0)="<mi>x</mi><mo>+</mo><mn>2</mn><mo>=</mo><mn>3</mn>";
$res = $result(0);
$arr_result=[];
while($pos = strpos($res, "<mo>", 1)) {
$arr_result[] = substr($res, 0, $pos); // grab first match
$res = substr($res, $pos); // grab the remaining string
}
$arr_result[] = $res; // add last chunk of string
print_r($arr_result);
?>
Your code above has several issues.
First:
$result1= new simplexml_load_string($result); // simplexml_load_string() is a function not a class
Second:
$key and $value do not contain the '<' and '>' so, this part:
$exp_key = explode('<', $key); will never do anything and isn't needed.
Third:
If your code did work it would only return array('+', '=') because you are appending the data inside the mo element to the result array.
I'm getting problems while filtering data from a MongoDB where documents have a "iso_date" field like
ISODate("2010-08-01T00:00:00Z")
in the following .php file,
<?php
$date_a = $_GET['date_a'];
$date_b = $_GET['date_b'];
$m = new MongoClient('mongodb://127.0.0.1:xxxxx');
$db = $m->db;
$collection = $db->collection;
$res = $collection->find(array(
'iso_date' => array('$gte' => $date_a, '$lte' => $date_b)
));
$m->close();
echo json_encode($res);
?>
where date_a and date_b are created in javascript. I've tried to add string "T00:00:00Z" to YYYY-MM-DD dates, but the result is always empty (and I'm sure it shouldn't be).
How can I solve this? I'd like not to convert everything to strings, and to keep a datetime format.
Try this :
$collection = $db->collection;
$start = $_GET['date_a'];
$end = $_GET['date_a'];
// find dates between 1/15/2010 and 1/30/2010
$collection->find(array("iso_date" => array('$gt' => $start, '$lte' => $end)));
Your $date_a and $date_b need to be instances of MongoDate (since you're using legacy driver chridam's answer may not work).
I'm looking for a functioning way of reading the last, say, 5 words from a .txt file using file_get_contents while long-polling. I've tried to work on a solution using the commented code below, but it breaks the long-polling – which works fine when displaying the full contents of a .txt-file.
I looked at the offset parameter for file_get_contents as well, but I don't know how to do a good (or functioning) adaption of that in my code. I guess that'd involve first counting all the words in said .txt file (which in itself is dynamic) then taking that number of words minus 5 somehow? There's probably an easier solution than that, and if not how do I go about to work it out with an alternative solution?
get_data.php
$id = $_COOKIE["currentID"];
$filepath = 'stories/'.$id.'.txt';
if (file_exists($filepath)) {
$lastmodif = isset($_GET['timestamp']) ? $_GET['timestamp'] : 0;
$currentmodif = filemtime($filepath);
while($currentmodif <= $lastmodif){
sleep(1);
clearstatcache();
$currentmodif = filemtime($filepath);
}
// I've tried:
// Attempt to getting the last five words of .txt file, does NOT work
$str = file_get_contents($filepath);
$words = explode(' ', $str);
$last = join(" ", array_slice($words, -5, 5));
$response = array();
/* $response['data'] = file_get_contents($filepath); */ //use if not attempting to get last 5 words
$response['data'] = $last; //part of "what I've tried"
$response['timestamp'] = $currentmodif;
echo json_encode($response);
}
else{
// if file doesn't exist
echo('nada data, son.');
}
?>
application.js (for context)
$(function(){
$.longpolling({
pollURL: './get_data.php',
successFunction: pollSuccess,
errorFunction: pollError
});
});
function pollSuccess(data, textStatus, jqXHR){
var json = eval('(' + data + ')');
$('#response').html(json['data']);
document.getElementById('notificationsound').play();
console.log('file updated');
}
function pollError(jqXHR, textStatus, errorThrown){
console.log('Long Polling Error: ' + textStatus);
}
Why not do:
$lastfive = explode(" ", trim(preg_replace("/^([\s\S]+)(( ([^ ]+)){5})$/m", "$2", $code)));
Also, you should never use eval when parsing code retrieved from a server. Try using JSON.parse(data) instead.
I have a JavaScript client which needs to send data to the server which will then save it as a file.
The client doesn't allow websockets so I've tried using JSON to post it then save it using PHP on the server but as the data on the JavaScript client is a Uint32Array PHP doesn't understand what it is.
This is what I have on the client side
var arrayData = new Uint32Array(256);
...
var formdata = new FormData();
formdata.append("data" , JSON.stringify(arrayData));
var xhr = new XMLHttpRequest();
xhr.open( 'post', 'receive.php', true );
xhr.send(formdata);
Then on the server
<?php
if(!empty($_POST['data'])){
$data = json_decode($_POST['data']);
$fname = mktime() . ".txt";//generates random name
file_put_contents("upload/" .$fname, data);
}
?>
All I get in the text file is 'data' and a PHP error in the logs
'Use of undefined constant data - assumed 'data'
I've no idea how to get PHP to write this as binary data so any help would be greatly appreciated!
Replace this line:
file_put_contents("upload/" .$fname, data);
With:
file_put_contents("upload/" .$fname, $data);
You just forgot a $ to make data a variable, and PHP was searching for a constant named data. Not finding it, it assumed (ahhh, PHP!) that you wanted to write "data" (a string).
Added: how to save binary data from your sample
// Parse the JSON. $input would be $_POST['data'] for you
$json = json_decode($input);
// This variable contains the string that we'll write to the file
$write = '';
// Loop through the array
for($i = 0; $i < $json->length; $i++) {
// Append the binary number to $write
// Note for pack(): 32-bit unsigned integers can be represented with:
// L -> machine byte order
// N -> big endian
// V -> little endian
$write .= pack('V', $json->{$i});
}
// Then write $write to file
file_put_contents("upload/" .$fname, $write);
I'm using http://canvasjs.com/ to create a bar graph. So far, when the user clicks 'submit' the entered numbers are displayed on a graph http://jsfiddle.net/jx9sJ/5/ .
Now, I'm trying to change it. So the entered numbers are sent to ajax, calculations done, & then encoded using json_encode($total); I'm struggling to create the graph from the values, which are in json_encode. How can this be done?
Code so far
$fortot2 = 5;
$fortot3 = 2;
if (is_numeric($numwelds) && is_numeric($numconwelds))
{
$total['tot1'] = $numwelds + $numconwelds + $fortot3 ;
$total['tot2'] = $numwelds + $numconwelds + $fortot2 ;
$total['tot3'] = $numwelds + $numconwelds;
$response = json_encode($total);
header("Content-Type: application/json");
echo $response;
exit;
}
Using print_r json_encode($total) is structured as
Array ( [tot1] => 3 [tot2] => 5 [tot3] => 1 )
First of all,
Array ( [tot1] => 3 [tot2] => 5 [tot3] => 1 )
is not json_encoded and rather looks like var_dump( $total ).
Then you have to populate the data object that is used on line 15 of your JSFiddle with the values from JSON, which requires some basic knowledge of AJAX. I suggest that you first familiarize yourself with that.