[TYPO3-performance] Re: Re: Re: Re: Re: DataHandler high memory consumption

[Lukas Krieger]

Hi Stephan,

i had disabled the cache (clearCache_disable=1) in the first place so that i don't have to worry about page caching problems.
Your problem is another one than mine but i think your observation is solid too.

Since i had no time to create an issue on forge and commit my patch on gerrit, it would be nice if someone could help me.
(i don't know what i should create first, issue or commit? Are commits on gerrit directly related to an issue? and so on)
skype: lukas.krieger91

Stephan, i think you will have the same problem i had without the patch.
It would be nice to see the complete memory consumption for your 6500 add script compared to my patched version.
Please add me on Skype so that we can do this within some minutes.



Quote: Stephan Großberndt wrote on Fri, 05 December 2014 19:29
----------------------------------------------------
> Hi,
> 
> I think I found (the or another?) cause for your problem because I had a 
> similar issue. It is in
> TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue
> 
> Due to this dubious construct, which is called for each and every record 
> added:
> 
> foreach (static::$recordsToClearCacheFor as $table => $uids) {
>    foreach (array_unique($uids) as $uid) {
>      ...
> 
>      foreach ($pageIdsThatNeedCacheFlush as $pageId) {
>        if ($pageId >= 0) {
>          $tagsToClear[] = 'pageId_' . $pageId;
>        }
>      }
>      // Queue delete cache for current table and record
>      $tagsToClear[] = $table;
>      $tagsToClear[] = $table . '_' . $uid;
>      $tagsToClear = array_unique($tagsToClear);
>    }
> }
> 
> I am adding 6500 records, so *FOR EACH AND EVERY* record 'pageId_' . 
> $pageId, $table and $table . '_' . $uid are added to $tagsToClear and 
> afterwards 'pageId_' . $pageId and $table are removed again for 6499 
> times. There is no reason to do it like this. array_unique() is meant 
> here to reduce memory consumption in $tagsToClear but for big arrays 
> with array_unique() this is terrible - xhprof showed 2,020,470,496 bytes 
> (~2 gigabyte !!!) of used memory for array_unique() in 
> processClearCacheQueue()
> 
> It is possible to set clearCache_disable=1 to disable clearing of the 
> cache but there is also a simple code solution to solve this since 
> entries in this array are meant to be unique anyway:
> 
> foreach (static::$recordsToClearCacheFor as $table => $uids) {
>    foreach (array_unique($uids) as $uid) {
>      ...
> 
>      foreach ($pageIdsThatNeedCacheFlush as $pageId) {
>        if ($pageId >= 0) {
>          $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
>        }
>      }
>      // Queue delete cache for current table and record
>      $tagsToClear[$table] = $table;
>      $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
>    }
> }
> 
> array_unique memory consumption drops from
> 
> IMemUse%: 13064.1%
> Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)
> 
> to
> 
> IMemUse%: 4.1%
> Incl. MemUse (bytes): 629,464 bytes
> 
> There may be optimizations possible like
> 
> foreach (static::$recordsToClearCacheFor as $table => $uids) {
>    foreach (array_unique($uids) as $uid) {
>      ...
> 
>      foreach ($pageIdsThatNeedCacheFlush as $pageId) {
>        $pageKey = 'pageId_' . $pageId;
>        if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
>          $tagsToClear[$pageKey] = $pageKey;
>        }
>      }
>      // Queue delete cache for current table and record
>      if (!isset($tagsToClear[$table])) {
>        $tagsToClear[$table] = $table;
>      }
>      $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
>    }
> }
> 
> but even without those its SOOOO much better...
> 
> I've created an issue: https://forge.typo3.org/issues/63615
> 
> Regards,
> Stephan
----------------------------------------------------