[TYPO3-performance] Re: DataHandler high memory consumption

[Lukas Krieger]

Quote: Jigal van Hemert wrote on Sun, 02 November 2014 19:54
----------------------------------------------------
> Hi,
> 
> On 02/11/2014 17:53, Lukas Krieger wrote:
> > i am using the DataHandler in order to import pages into Typo3 but i
> > have some problems with the php memory consumption.
> >
> > Creating X pages result in X MB Memory usage
> > 54 Pages => 21MB
> > 118 Pages => 41MB
> > 150 Pages => 50MB
> > 281 Pages => 88MB
> >
> > You can say, the DataHandler instance need 0,33MB for each record.
> [...]
> > Is there any solution to use less memory or even to free the memory
> > which the DataHandler is using?
> > Did someone of you experienced the same problem?
> 
> In one project I did a rather large import of pages and content using 
> DataHandler. Because it would be used for many thousands of pages and 
> their content elements I made two scheduler tasks to handle this.
> This way I didn't notice any dramatic memory usage, simply because only 
> a limited number of records would be imported.
> 
> It could be that the DataHandler object adds data for each record to 
> some internal structures. It would be interesting to see where this occurs.
> A workaround could be to destroy the variable with the DataHandler at 
> some point and instantiate a new object.
> 
> -- 
> Jigal van Hemert
> TYPO3 CMS Active Contributor
> 
> TYPO3 .... inspiring people to share!
> Get involved: typo3.org
----------------------------------------------------


Hi,

thanks for your reply so far.
Just some background informations:
I fetch informations from another system via an API and get some JSON results. 
These pages need to be created in typo3 in order to add additional informations like images,text and so on by the online editor.
The initial import of the pages must be done on the same execution. The scheduler will be used to update the informations.

Here are some of my results from debugging.
Task: Update 281 pages and delete/undelete/hide/unhide them



$chunks = array_chunk($cmdPages['pages'],10,true);
$tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler');
$tce->enableLogging=false;
//$tce->stripslashes_values = 0;
//$tce->reverseOrder = 1;
//$tce->storeLogMessages = false;
//$tce->checkSimilar = false;
//$tce->checkStoredRecords = false;
//$tce->checkStoredRecords_loose = false;
//$tce->bypassWorkspaceRestrictions = true;
//$tce->bypassAccessCheckForRecords = true;
foreach($chunks as $chunk)
{

	echo 'mem pre typo3 process: '.$this->getMemUsage();

	$temp['pages'] = $chunk;
			
	$tce->start(array(), $temp);
	$tce->process_cmdmap();

	$temp = null;
	unset($temp);
	
	echo 'mem post typo3 process: '.$this->getMemUsage();
}

I get the following output:

ini mem: 44364784 //initial memory usage at the start of the script

mem pre typo3: 45192576
mem pre typo3 process: 48235416
mem post typo3 process: 52489584
..
mem pre typo3 process: 75972360
mem post typo3 process: 78894784
...
mem pre typo3 process: 93589144
mem post typo3 process: 96514048
...
mem pre typo3 process: 117088552
mem post typo3 process: 120013920
...
mem pre typo3 process: 132067504
mem post typo3 process: 132067504

As you can see, the memory usage is increasing after each process_cmdmap()

I had a closer look at all three functions
1. makeInstance
2. tce->start
3. process_cmdmap 

1. makeInstance http://typo3.org/api/typo3cms/_general_utility_8php_source.html#l04335

Line 4341-4344 will be executed 
Each time $tce = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\DataHandling\\DataHandler'); is called,
the function "protected static function instantiateClass" creates a new object of the class "TYPO3\CMS\Core\DataHandling\DataHandler"

2. tce->start http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l00690
This is just a setter function for the two arrays and some permission checks

3. process_cmdmap http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l02860

So this function must be responsible for the memory consumption.
Here is the output of the memory usage in this function (editing 150 records)

mem pre typo3 process: 47667120
mem pre traverse map 47870080
mem pre command 47892552 //before first command
mem post command 50247504 //after first command
mem pre command 50247816 //before second command
mem post command 50248016 //after second command
...
mem post command 50279600 //last command | after 150 commands
mem post traverse map 50258872
mem pre copyTCE map 50258872
mem post copyTCE map 50275784
mem post remap 50277032
mem post hooks 50277032 //line 2991
mem post process_cmdmap 94548192 //line 2995

As you can see, there is a increase after the first executed command (2.25MB) and a very big increase at the end of the function
if ($this->isOuterMostInstance()) {
                         $this->processClearCacheQueue();
                         $this->resetNestedElementCalls();
                 }

So i had a look at the memory usage after both functions:

Before: 50278080
After processClearCacheQueue : 94549448
After resetNestedElementCalls: 94549448
After: 94549448

It looks like the processClearCache function is responsible for the high memory consumption.

I deleted the function just to be sure that this is the responsible function
And got this output:

ini mem: 44013992 // the initial memory usage when my script is executed
mem pre typo3: 44665112 //before the DataHandler is doing anything
mem pre command 48234496 //before the DataHandler is executing the first command
mem post process_cmdmap 50593792 //after the DataHandler is finished

Conclusion: it is the function processClearCacheQueue http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l06992


So after all that command execution (updating 150 pages)
My script only uses 6MB memory and is much faster then before

Updating 280 pages was done by using just 4.8MB memory.

I will debug the processClearCacheQueue function later and post my results in this thread.

Hope it will help someone :-)