[TYPO3-performance] Re: Re: Re: DataHandler high memory consumption

[Lukas Krieger]

I have modified the function BackendUtility::getPagesTSconfig using the CacheManager storing the results
and just use the static $pagesTSconfig_cache for storing the hashes for each page.

I also had a look at the execution of that function and this function is executed a few times when using the backend.
Did not found any calls across different page ids, like Philipp mentioned.

My modification is working and the memory usage dropped (to 14MB)
There are still 280 hashes saved in the pagesTSconfig_cache array.

I have some output from my script (updating 280 records) and also from the backend 

my script: 
[04-Nov-2014 16:54:22 UTC] current page id "2540" : stored combined TSconfig
[04-Nov-2014 16:54:22 UTC] current page id "2539" : combined TSconfig is already stored because it is the same
[04-Nov-2014 16:54:22 UTC] current page id "2536" : combined TSconfig is already stored because it is the same
[04-Nov-2014 16:54:22 UTC] current page id "2535" : combined TSconfig is already stored because it is the same
...
[04-Nov-2014 16:54:24 UTC] current page id "2542" : combined TSconfig is already stored because it is the same

normal typo3 backend:

[04-Nov-2014 16:57:27 UTC] current page id "43" : stored combined TSconfig
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache
[04-Nov-2014 16:57:28 UTC] current page id "43" : gets the TSconfig out of the cache


The pagesTSconfig_cache array looks like this for normal backend calls

id => hash
1 => dsf22dd(hash)

and like this for cross page calls

id => hash
34 => dsf22dd(hash)
35 => dsf22dd(hash)
36 => dsf22dd(hash)
37 => dsf22dd(hash)
38 => f3gdds2(hash) // different
39 => dsf22dd(hash)

On one last test i added TSconfig to only one page id 2536.
Here is the output:
[04-Nov-2014 17:26:37 UTC] current page id "2540" : stored combined TSconfig
[04-Nov-2014 17:26:37 UTC] current page id "2539" : combined TSconfig is already stored because it is the same
[04-Nov-2014 17:26:37 UTC] current page id "2536" : stored combined TSconfig
[04-Nov-2014 17:26:37 UTC] current page id "2535" : combined TSconfig is already stored because it is the same
[04-Nov-2014 17:26:37 UTC] current page id "2534" : combined TSconfig is already stored because it is the same
[04-Nov-2014 17:26:37 UTC] current page id "2533" : combined TSconfig is already stored because it is the same

As you can see, i start with page 2540, which is stored in the cache.
2539 is already in the cache because it is the same as 2540
2536 has new page TSconfig so it must be stored in the cache as a new one
2535,2534,2533 are already stored in the cache because they are the same as 2540 and 2539 and they get the same hash

There should be also no lack of performance.

Philipp, do you have any specific tests i should run?

And again, forge issue?

best,
Lukas

Quote: Lukas Krieger (lkrieger) wrote on Tue, 04 November 2014 14:14
----------------------------------------------------
> Hi Philipp,
> 
> thanks :-) it was a lot of pain for me to find that specific function and array :-D
> 
> The different parsed TSconfig is already stored in a cache using the CacheManager for every page.
> $res = $parseObj->parseTSconfig($pageTS, 'PAGES', $id, $rootLine);
> Results in either getting the parsed TSconfig out of the cache or storing it (parsed with conditions)
> 
> So the function getPagesTSconfig (http://typo3.org/api/typo3cms/backend_2_classes_2_utility_2_backend_utility_8php_source.html#l01153)
> is only doing one tasks:
> 
> Line 1191-1195
> If there is any user page TS it will combine the parsed TSconfig and the userTS.
> 
> Every actions between lines 1171-1190 (except returnPartArray) is already stored by the CacheManager after the first call. The first call is also necessary for this function to use the array cache.
> 
> Let us think about the different informations which are used in that function:
> 
> 1. a highly different TSconfig for each page
> 2. the global $GLOBALS['BE_USER']->userTS['page.']
> 
> The output is one highly different TSconfig because of the different parsed TSconfig (line 1186)
> 
> We have three different important parts
> 1. parsed TSconfig
> 2. userTS['page']
> 3. the backend user
> 
> We can ignore the third part because the second part is already backend user specific!
> 
> 
> Lets get back to the logic of the function.
> In order to be able to cache the combined TSconfigs for a page, the page TSconfig must be parsed by the TsConfigParser 
> which results in caching that parsed TSconfig. For the same page, it is every time same! To put it another way, it is stored only once
> in the CacheManager if the page TSconfig (which was build together between lines 1171-1190) is the same.
> The same page TSconfig results in the same hash which is used to store the parsed TSconfig.
> 
> So the only thing which has to be done is adding the user TSconfig overlay.
> 
> The main problem currently is, that lines 1171-1190 have to be executed in order to get the already cached parsed page TSconfig.
> => lack of performance
> 
> But keep in mind that we MUST already have got the hash for that parsed TSconfig because we parsed it with the TsConfigParser. Otherwise it is not possible to have the pagesTSconfig_cache in that function.
> 
> If we could extend the TsConfigParser and let him return not only the parsed TSconfig + the cache flag but more important the hash for that
> cached record, we could use that hash, combined with the user TSconfig, to cache the complete output of that function.
> 
> What do you think?
> 
> ps.: i really don't like that forum :-( 
> 
> 
> Quote: Philipp Gampe (pgampe) wrote on Tue, 04 November 2014 12:26
> ----------------------------------------------------
> > Hi Lukas,
> > 
> > Nice analysis btw.
> > 
> > As I helped writing this code, I can tell you why we cannot do better here.
> > :(
> > 
> > The problem here is that you can use conditions for backend TSconfig. That 
> > means the parsed TSconfig might be different for each page, because:
> > 1) The page has the PageTS field filled
> > 2) The parent page has a condition for this page
> > 3) The userTS has a condition for this page
> > 4) The global TS has a condition for this page
> > 5) Some condition in all of the above based on any global var
> > 
> > You can also see that the TSconfig might be different for different users 
> > and groups.
> > At least we have static settings only and nothing like stdWrap ;)
> > 
> > Therefore you cannot really cache it or what would you use as a key?
> > 
> > Lukas Krieger wrote:
> > 
> > > I can think of two patches immediately
> > > 
> > > 1. Storing a serialized array on line 1198 and output the unserialized
> > > array on line 1161 By doing so, the memory usage drops to 18MB which is
> > > still big, but not as big as before. Just consider, we are still saving
> > > 280 strings of ~36KB each in the array => ~10MB memory usage and each
> > > array is the same. Thats still bad!
> > 
> > This helps in your case, but on a normal call, the function is called for 
> > the same page very often (50-100+ times).
> > And this is what it is optimized for.
> > Optimizing it for mass imports means making slower for regular backend 
> > operations.
> > 
> > > 2. Using the CacheManager which is also used for caching the already
> > > parsed TSConfig in the TSConfigParser The parsed TSConfig on line 1186 is
> > > already cached in the CacheManager and it would be easy to add a cache for
> > > the combined parsed TSConfig+User TSconfig based on the uid of each page.
> > 
> > You cannot cache the UserTS, because it is different for each user and might 
> > have some condition based on the current page.
> > And the userTS must be parsed in between the global and the current pageTS.
> > 
> > Thus if you can find a nice and fast algorithm for a cache key, I would be 
> > more than happy to introduce a cache here.
> > 
> > Best regards
> > -- 
> > Philipp Gampe  PGP-Key 0AD96065  TYPO3 UG Bonn/Köln
> > Documentation  Active contributor TYPO3 CMS
> > TYPO3 .... inspiring people to share!
> ----------------------------------------------------
> 
----------------------------------------------------