[TYPO3-performance] Re: Re: Re: DataHandler high memory consumption

[Lukas Krieger]

Hi Philipp,

thanks :-) it was a lot of pain for me to find that specific function and array :-D

The different parsed TSconfig is already stored in a cache using the CacheManager for every page.
$res = $parseObj->parseTSconfig($pageTS, 'PAGES', $id, $rootLine);
Results in either getting the parsed TSconfig out of the cache or storing it (parsed with conditions)

So the function getPagesTSconfig (http://typo3.org/api/typo3cms/backend_2_classes_2_utility_2_backend_utility_8php_source.html#l01153)
is only doing one tasks:

Line 1191-1195
If there is any user page TS it will combine the parsed TSconfig and the userTS.

Every actions between lines 1171-1190 (except returnPartArray) is already stored by the CacheManager after the first call. The first call is also necessary for this function to use the array cache.

Let us think about the different informations which are used in that function:

1. a highly different TSconfig for each page
2. the global $GLOBALS['BE_USER']->userTS['page.']

The output is one highly different TSconfig because of the different parsed TSconfig (line 1186)

We have three different important parts
1. parsed TSconfig
2. userTS['page']
3. the backend user

We can ignore the third part because the second part is already backend user specific!


Lets get back to the logic of the function.
In order to be able to cache the combined TSconfigs for a page, the page TSconfig must be parsed by the TsConfigParser 
which results in caching that parsed TSconfig. For the same page, it is every time same! To put it another way, it is stored only once
in the CacheManager if the page TSconfig (which was build together between lines 1171-1190) is the same.
The same page TSconfig results in the same hash which is used to store the parsed TSconfig.

So the only thing which has to be done is adding the user TSconfig overlay.

The main problem currently is, that lines 1171-1190 have to be executed in order to get the already cached parsed page TSconfig.
=> lack of performance

But keep in mind that we MUST already have got the hash for that parsed TSconfig because we parsed it with the TsConfigParser. Otherwise it is not possible to have the pagesTSconfig_cache in that function.

If we could extend the TsConfigParser and let him return not only the parsed TSconfig + the cache flag but more important the hash for that
cached record, we could use that hash, combined with the user TSconfig, to cache the complete output of that function.

What do you think?

ps.: i really don't like that forum :-( 


Quote: Philipp Gampe (pgampe) wrote on Tue, 04 November 2014 12:26
----------------------------------------------------
> Hi Lukas,
> 
> Nice analysis btw.
> 
> As I helped writing this code, I can tell you why we cannot do better here.
> :(
> 
> The problem here is that you can use conditions for backend TSconfig. That 
> means the parsed TSconfig might be different for each page, because:
> 1) The page has the PageTS field filled
> 2) The parent page has a condition for this page
> 3) The userTS has a condition for this page
> 4) The global TS has a condition for this page
> 5) Some condition in all of the above based on any global var
> 
> You can also see that the TSconfig might be different for different users 
> and groups.
> At least we have static settings only and nothing like stdWrap ;)
> 
> Therefore you cannot really cache it or what would you use as a key?
> 
> Lukas Krieger wrote:
> 
> > I can think of two patches immediately
> > 
> > 1. Storing a serialized array on line 1198 and output the unserialized
> > array on line 1161 By doing so, the memory usage drops to 18MB which is
> > still big, but not as big as before. Just consider, we are still saving
> > 280 strings of ~36KB each in the array => ~10MB memory usage and each
> > array is the same. Thats still bad!
> 
> This helps in your case, but on a normal call, the function is called for 
> the same page very often (50-100+ times).
> And this is what it is optimized for.
> Optimizing it for mass imports means making slower for regular backend 
> operations.
> 
> > 2. Using the CacheManager which is also used for caching the already
> > parsed TSConfig in the TSConfigParser The parsed TSConfig on line 1186 is
> > already cached in the CacheManager and it would be easy to add a cache for
> > the combined parsed TSConfig+User TSconfig based on the uid of each page.
> 
> You cannot cache the UserTS, because it is different for each user and might 
> have some condition based on the current page.
> And the userTS must be parsed in between the global and the current pageTS.
> 
> Thus if you can find a nice and fast algorithm for a cache key, I would be 
> more than happy to introduce a cache here.
> 
> Best regards
> -- 
> Philipp Gampe  PGP-Key 0AD96065  TYPO3 UG Bonn/Köln
> Documentation  Active contributor TYPO3 CMS
> TYPO3 .... inspiring people to share!
----------------------------------------------------