[TYPO3-mvc] Memory Usage with 2000 Domain Objects

[Christian Zenker]

On Tue, 22 Mar 2011 12:22:18 +0100, Tim Schoch | GSTALTIG  
<tim.schoch at gstaltig.ch> wrote:

Hi Tim.

You are working with fluid, right?
In my experience fluid is very memory hungry. Extbase usually is not the  
problem - I fetched 1000 objects from the repo myself in a scheduler task  
and even 64MB would have been enough and i could have fetched far more  
objects without reaching the limit.

Try to reduce the number of nodes in your template. This could save you  
some space. You could use a very simple viewHelper debug library I  
wrote[1] to debug fluid to find where most of the memory is required.
I had problems once with a very simple gallery I wrote. I had something  
around of 30 images and the memory of 128MB was entirely used. I found out  
that each call of a partial increased the memory consumption by about 1MB.  
For some reason the space was not freed afterwards so that all summed up.  
When I did not use the partial memory consumption was very very low. I  
could not figure out why that happened but maybe this could also be a  
solution for you. (just to add: this happened in TYPO3 4.4 - I'm not sure  
if this behaviour is still present)

Best regards,
Christian.

[1] https://github.com/czenker/fluid_debug

> Hello List
>
> I'm trying to aggregate an index with something around 2000 items and  
> all I get is a «Fatal error:  Allowed memory size of 171966464 bytes  
> exhausted». =[
>
> For each item I have up to 7 optional 1:n Relations (lazy, Value  
> Objects) and 2 n:n Relations(lazy, Value Objects). If the item has a  
> filter, I add the filterUid/categoryUid combination to the index. The  
> fully populated index has around 300 combinations, stored in an array  
> like this: $index[categoryUid][filterUid] = filterName. The size of the  
> index itself can't be the problem. So I've tried to track down where the  
> memory is cluttered and found some ways to cut down the memory  
> footprint, but it's still not good enough:
>
>
> 1. Subquery ( copied the code from the paginate widget )
> $all = $this->repo->findAll();
> for( $page < $totalPages ) {
>   $subQuery = $all->getQuery();
>   $subResults = $subQuery->setLimit( 100 )->setOffset( 100 * $page  
> )->execute();
>   foreach( $subResults as $item ) { ... }
> }
>  (code is shortened)
>
>
> 2. unregistering Items
> $this->persistenceManager->getSession()->unregisterReconstitutedObject(  
> $item );
> $this->identityMap->unregisterObject( $item );
> As I don't need the items for further operations I added this call  
> inside the foreach( $item ) loop. This removes the items from the  
> session and identity map cache but it hasn't that much of an impact  
> (around 1.5 kb per item or 3.5mb for all items).
>
>
> 3. unsetting what's not needed
> unset( $item );
> unset( $subResults );
> unset( $subQuery );
> I tried to unset everything that I don't need, but it's not that much of  
> an impact, as it gets overridden by the next foreach-loop anyway.  
> ("saves" 20kb per loop)
>
> ---------------------------
>
> I added some memory_get_usage() calls to measure the impact and found  
> out, that every Subquery leaves me with a memoryleak of aprox 6 mb. When  
> debugging, i can't find where these 6mb are comming from?
>
> Item No     /   Begin Loop    /  After clean up  /
> 000 - 099   /    34717432    /        41533200  / + 6815768
> 100 - 199   /    41534136    /        47588240  / + 6054104  < the index  
> is 99% completed after the second run
> 200 - 299   /    47588936    /        53908520  / + 6319584
> 300 - 399   /    53909016    /        59850248  / + 5941232
>
> ---------------------------
>
> After each sub query run, the sessionManager shows exactly 9 Objects, so  
> the unregistering is successfull. I've looked into every single  
> dispatcher object using PhpEds debug tool and found no other occurences  
> of any of my objects... but still... after 20 sub query runs, the 6 mb  
> add up to 120mb...
>
> Are there further ways to decrease memory usage of extbase, when dealing  
> with complex objects?
>
> Thanks Tim