Safely catch a 'Allowed memory size exhausted' error in PHP

Question 1

I have a gateway script that returns JSON back to the client. In the script I use set_error_handler to catch errors and still have a formatted return.

It is subject to 'Allowed memory size exhausted' errors, but rather than increase the memory limit with something like ini_set('memory_limit', '19T'), I just want to return that the user should try something else because it used to much memory.

Are there any good ways to catch fatal errors?

Question 2

As this answer suggests, you can use register_shutdown_function() to register a callback that'll check error_get_last().

You'll still have to manage the output generated from the offending code, whether by the @ (shut up) operator, or ini_set('display_errors', false)

ini_set('display_errors', false);

error_reporting(-1);

set_error_handler(function($code, $string, $file, $line){
        throw new ErrorException($string, null, $code, $file, $line);
    });

register_shutdown_function(function(){
        $error = error_get_last();
        if(null !== $error)
        {
            echo 'Caught at shutdown';
        }
    });

try
{
    while(true)
    {
        $data .= str_repeat('#', PHP_INT_MAX);
    }
}
catch(Exception $exception)
{
    echo 'Caught in try/catch';
}

When run, this outputs Caught at shutdown. Unfortunately, the ErrorException exception object isn't thrown because the fatal error triggers script termination, subsequently caught only in the shutdown function.

You can check the $error array in the shutdown function for details on the cause, and respond accordingly. One suggestion could be reissuing the request back against your web application (at a different address, or with different parameters of course) and return the captured response.

I recommend keeping error_reporting() high (a value of -1) though, and using (as others have suggested) error handling for everything else with set_error_handler() and ErrorException.

Question 3

Your loop fills the variable $field for no reason (it writes to a different cell on every loop iteration), thereby using up more memory with every line.

You can replace:

$field[$loop] = explode ($delimiter, $line);
$export_date = $field[$loop][0];
$genre_id = $field[$loop][1];
$application_id = $field[$loop][2];

With:

list($export_date, $genre_id, $application_id) = explode($delimiter, $line);

For improved performance, you could take advantage of the ability to insert several lines using REPLACE INTO by grouping N lines into a single query.

Question 4

This answer is an example of how to implement a buffer(a limited array in memory) in your code and when it is filled, flush it's contents to disk, at the end you will find a huge array on disk in JSON format. I used this way in a situation similar to yours and got great result regarding "memory usage", but as I told you in comments you need to rethink why you need that HUGE array in the first place, and if there is a way to avoid it, go with it.

using this function will save you the memory used by your $final_result array and replace it with $final_result string buffer but we are controlling it's use of memory. However your $query_result array will still taking the memory it needs.

Note that you need to alter the function as you need because I used your variables which are undefined in my code.

/**
 * proccess the HUGE array and save it to disk in json format [element,element]
 * 
 * @param string $fileName absulote file name path you want to save the proccessed array in
 * @return int processed elements count
 */
function buildMyArrayInFile($fileName)
{
    $flushCheckPoint = 100;// set the buffer size as needed, depending on the size of PHP allowed memory and average element size
    $processedElements = 0;
    $final_result = "[";

    file_put_contents($fileName, "");//prepare the file and erase anything in it

    foreach($query_result as $row)
    {
        $line_datas =explode(";",$row["linedata_0"]);
        $linedata = [];
        $final = [];
        $d = [];

        for($s =0; $s < count($line_datas); $s++){
            $line_data = explode(",",$line_datas[$s]);
            $d["timestamp"] = utf8_encode($line_data[0]);
            $d["x"]= utf8_encode($line_data[1]);
            $d["y"] = utf8_encode($line_data[2]);

            array_push($linedata,$d);
        }

        $final["id"]= $row["id"];
        $final["isActive"]= $row["isActive"];
        $final["personId"]= utf8_encode($row["personId"]);
        $final["name"] = NULL;
        $final["gender"] = utf8_encode($row["gender"]);
        $final["age"] = utf8_encode($row["age"]);
        $final["linedata"]=$linedata;


        $final_result .= json_encode($final) . ",";
        $processedElements ++;
        if($processedElements % $flushCheckPoint === 0){
            //the array has reached the limit, flush the array to disk
            file_put_contents($fileName, $final_result, FILE_APPEND);
            $final_result = "";
        }

    }

    $final_result = rtrim($final_result, ",");//trim the last comma
    $final_result .= "]";
    //flush the remaning data in $final_result
    file_put_contents($fileName, $final_result, FILE_APPEND);

    return $processedElements;

}

this is another simple version of the function for testing

// test
var_dump(buildMyArrayInFile2("/home/myuser/myArray.json"));
// outputs int(7)



function buildMyArrayInFile2($fileName)
{
    $flushCheckPoint = 2;// set the buffer size as needed, depending on the size of PHP allowed memory and average element size
    $processedElements = 0;
    $final_result = "[";

    file_put_contents($fileName, "");//prepare the file and erase anything in it

    $test_result = [1,2,3,4,"wee","hellonworld",5];
    foreach($test_result as $row)
    {
        $final_result .= json_encode($row) . ",";
        $processedElements ++;
        if($processedElements % $flushCheckPoint === 0){
            //the array has reached the limit, flush the array to disk
            file_put_contents($fileName, $final_result, FILE_APPEND);
            $final_result = "";
        }

    }

    $final_result = rtrim($final_result, ",");//trim the last comma
    $final_result .= "]";
    //flush the remaning data in $final_result
    file_put_contents($fileName, $final_result, FILE_APPEND);

    return $processedElements;
}

Question 5

This error is a fatal error - that means you cannot recover from it. If PHP has hit it's memory limit, it won't be able to allocate any more memory to create your exception and any other memory it needs to carry on its execution.

There is another type of error - "catchable fatal error" which as the name suggests, can be caught in a try/catch, but unfortunately the memory size allocation is not one of them.

Question 6

PHP has a garbage collector. It will free memory for variable containers which reference count is set to 0, meaning that no userland reference exists anymore.

I guess there are still references to variables which you might think to have cleaned. Need to see your code to show you what is the problem.

Safely catch a 'Allowed memory size exhausted' error in PHP

Answers

php,memory