[TYPO3-performance] DataHandler high memory consumption

[Stephan Großberndt]

Hi,

I think I found (the or another?) cause for your problem because I had a 
similar issue. It is in
TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue

Due to this dubious construct, which is called for each and every record 
added:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
   foreach (array_unique($uids) as $uid) {
     ...

     foreach ($pageIdsThatNeedCacheFlush as $pageId) {
       if ($pageId >= 0) {
         $tagsToClear[] = 'pageId_' . $pageId;
       }
     }
     // Queue delete cache for current table and record
     $tagsToClear[] = $table;
     $tagsToClear[] = $table . '_' . $uid;
     $tagsToClear = array_unique($tagsToClear);
   }
}

I am adding 6500 records, so *FOR EACH AND EVERY* record 'pageId_' . 
$pageId, $table and $table . '_' . $uid are added to $tagsToClear and 
afterwards 'pageId_' . $pageId and $table are removed again for 6499 
times. There is no reason to do it like this. array_unique() is meant 
here to reduce memory consumption in $tagsToClear but for big arrays 
with array_unique() this is terrible - xhprof showed 2,020,470,496 bytes 
(~2 gigabyte !!!) of used memory for array_unique() in 
processClearCacheQueue()

It is possible to set clearCache_disable=1 to disable clearing of the 
cache but there is also a simple code solution to solve this since 
entries in this array are meant to be unique anyway:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
   foreach (array_unique($uids) as $uid) {
     ...

     foreach ($pageIdsThatNeedCacheFlush as $pageId) {
       if ($pageId >= 0) {
         $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
       }
     }
     // Queue delete cache for current table and record
     $tagsToClear[$table] = $table;
     $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
   }
}

array_unique memory consumption drops from

IMemUse%: 13064.1%
Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)

to

IMemUse%: 4.1%
Incl. MemUse (bytes): 629,464 bytes

There may be optimizations possible like

foreach (static::$recordsToClearCacheFor as $table => $uids) {
   foreach (array_unique($uids) as $uid) {
     ...

     foreach ($pageIdsThatNeedCacheFlush as $pageId) {
       $pageKey = 'pageId_' . $pageId;
       if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
         $tagsToClear[$pageKey] = $pageKey;
       }
     }
     // Queue delete cache for current table and record
     if (!isset($tagsToClear[$table])) {
       $tagsToClear[$table] = $table;
     }
     $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
   }
}

but even without those its SOOOO much better...

I've created an issue: https://forge.typo3.org/issues/63615

Regards,
Stephan