[TYPO3-performance] Re: DataHandler high memory consumption

Lukas Krieger lukas.krieger at me.com
Mon Nov 3 12:45:06 CET 2014


Hi,

using clearCache_disable=1 did not change the memory usage. 
The function is still using ~ 85MB after editing 280 pages (undelete command)
Although the if clause between lines 7004-7064 is not executed.

If i set clearCache_disable=0 back again and run the script undeleting 280 pages. The if clause catches 
and i have the same memory usage.

=> The memory must be used before or after the if clause

My "static::$recordsToClearCacheFor" consists of 280 elements:
Array ( [pages] => Array ( [0] => 2540 [1] => 2539 [2] => 2536 [3] => 2535.....
So just 280 uids of the updated pages.

The foreach on line 6995 is just executed once (iterating the table)
The foreach on line 6996 is executed 280 times (iterating the page uids)

So let us have a look at the memory usage at the start and end of each iterating 

start foreach numb 1 mem usage 37823968
end foreach numb 1 mem usage 38267432
..
start foreach numb 50 mem usage 52395072
end foreach numb 50 mem usage 52689232
..
start foreach numb 100 mem usage 67107552
end foreach numb 100 mem usage 67401888
..
start foreach numb 150 mem usage 81824232
end foreach numb 150 mem usage 82118504
..
start foreach numb 279 mem usage 119803552
end foreach numb 279 mem usage 120098080

As you can see, the usage is increasing by each loop

Just to be sure, i deleted the whole if clause from line 7004-7064 and i got the same output


So there are just three code fragments left which could be responsible for the memory usage:

// Get Page TSconfig relavant:
http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l07001
Line 7001-7003

//clear cache for pages entered in TSconfig
http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l07065
Line 7065-7070

//post processing function for clear-cache
http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l07071
Line 7071-7077

I commented out the whole post processing block and got the same memory usage
After out commenting the "clear cache for pages entered in TSconfig" block the memory usage is still the same

=>
After out commenting the two lines 
 7001 // Get Page TSconfig relavant:
 7002                                 list($tscPID) = BackendUtility::getTSCpid($table, $uid, '');
 7003                                 $TSConfig = $this->getTCEMAIN_TSconfig($tscPID);

The memory usage of my script drops from ~ 85MB to just 5MB!
We have found the place where the high memory usage occurs!


For every page uid in "static::$recordsToClearCacheFor"  the script gets the TSConfig of each page

Line 7002
BackendUtility::getTSCpid($table, $uid, '');
http://typo3.org/api/typo3cms/class_t_y_p_o3_1_1_c_m_s_1_1_backend_1_1_utility_1_1_backend_utility.html#a4a5108691e86b91f24617c203b5b9451
Returns the REAL pid of the record, if possible. If both $uid and $pid is strings, then pid=-1 is returned as an error indication.


Line 7003
$TSConfig = $this->getTCEMAIN_TSconfig($tscPID);
http://typo3.org/api/typo3cms/class_t_y_p_o3_1_1_c_m_s_1_1_core_1_1_data_handling_1_1_data_handler.html#a5ca44413411e7b1ea925da3254296b55
Return TSconfig for a page id

=>

 6546 public function getTCEMAIN_TSconfig($tscPID) {
 6547                 if (!isset($this->cachedTSconfig[$tscPID])) {
 6548                         $this->cachedTSconfig[$tscPID] = $this->BE_USER->getTSConfig('TCEMAIN', BackendUtility::getPagesTSconfig($tscPID));
 6549                 }
 6550                 return $this->cachedTSconfig[$tscPID]['properties'];
 6551         }


At first i thought it would be the caching of the TSConfig for each page but my cachedTSconfig array is empty.
It is just filled with 280 keys and for each key a new array:
Array ( [2540] => Array ( [value] => [properties] => ) [2539] => Array ( [value] => [properties] => ) [2536] =>...

Of course it increases the memory usage but it is only a few KB for 280 pages!

So i commented out the line 6548 (i do not want to fetch my empty TSConfig) and the memory usage dropped to 5MB again!
Thats the problem!

It also explains why there is no benefit of chunking the pages into pieces and let the DataHandler process only i.e. 10 pages at the same time.
It calls external functions and the DataHandler itself does not use that much memory.
(Therefore creating und destroying the DataHandler is no solution for the problem - as i tested before)

I have to leave now but i will dig deeper into the system and have a look at the functions on Line 6548 later:
$this->BE_USER->getTSConfig('TCEMAIN', BackendUtility::getPagesTSconfig($tscPID));




Quote: Tymoteusz Motylewski (tmotylewski) wrote on Mon, 03 November 2014 08:37
----------------------------------------------------
> Hi Lukas,
> Nice findings.
> Can you make a forge issue for that. It would be nice if you can
> investigate a little bit further, what is the root cause for the high
> memory consumption.
> Is it just the size of "recordsToClearCacheFor" array, or the processing of
> this array, or do we have a memory leak somewhere?
> 
> Would the memory consumption be better if you set UserTS clearCache_disable
> to 1 ?
> This should help and doesn't require modifiyng the core.
> 
> Cheers
> Tymoteusz
> 
> 
> > Conclusion: it is the function processClearCacheQueue
> > http://typo3.org/api/typo3cms/_data_handler_8php_source.html#l06992
> >
> >
> > So after all that command execution (updating 150 pages)
> > My script only uses 6MB memory and is much faster then before
> >
> > Updating 280 pages was done by using just 4.8MB memory.
> >
> > I will debug the processClearCacheQueue function later and post my results
> > in this thread.
> >
> > Hope it will help someone :-)
> >
> > _______________________________________________
> > TYPO3-performance mailing list
> > TYPO3-performance (at) lists.typo3.org
> > http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-performance
> >
----------------------------------------------------



More information about the TYPO3-performance mailing list