[TYPO3-mvc] Memory Usage with 2000 Domain Objects

[Tim Schoch | GSTALTIG]

Hello List

I'm trying to aggregate an index with something around 2000 items and all I get is a «Fatal error:  Allowed memory size of 171966464 bytes exhausted». =[

For each item I have up to 7 optional 1:n Relations (lazy, Value Objects) and 2 n:n Relations(lazy, Value Objects). If the item has a filter, I add the filterUid/categoryUid combination to the index. The fully populated index has around 300 combinations, stored in an array like this: $index[categoryUid][filterUid] = filterName. The size of the index itself can't be the problem. So I've tried to track down where the memory is cluttered and found some ways to cut down the memory footprint, but it's still not good enough:


1. Subquery ( copied the code from the paginate widget )
$all = $this->repo->findAll();
for( $page < $totalPages ) {
  $subQuery = $all->getQuery();
  $subResults = $subQuery->setLimit( 100 )->setOffset( 100 * $page )->execute();
  foreach( $subResults as $item ) { ... }
}
 (code is shortened)


2. unregistering Items
$this->persistenceManager->getSession()->unregisterReconstitutedObject( $item );
$this->identityMap->unregisterObject( $item );
As I don't need the items for further operations I added this call inside the foreach( $item ) loop. This removes the items from the session and identity map cache but it hasn't that much of an impact (around 1.5 kb per item or 3.5mb for all items).


3. unsetting what's not needed
unset( $item );
unset( $subResults );
unset( $subQuery );
I tried to unset everything that I don't need, but it's not that much of an impact, as it gets overridden by the next foreach-loop anyway. ("saves" 20kb per loop)

---------------------------

I added some memory_get_usage() calls to measure the impact and found out, that every Subquery leaves me with a memoryleak of aprox 6 mb. When debugging, i can't find where these 6mb are comming from?

Item No     /   Begin Loop    /  After clean up  /
000 - 099   /    34717432    /        41533200  / + 6815768
100 - 199   /    41534136    /        47588240  / + 6054104  < the index is 99% completed after the second run
200 - 299   /    47588936    /        53908520  / + 6319584
300 - 399   /    53909016    /        59850248  / + 5941232

---------------------------

After each sub query run, the sessionManager shows exactly 9 Objects, so the unregistering is successfull. I've looked into every single dispatcher object using PhpEds debug tool and found no other occurences of any of my objects... but still... after 20 sub query runs, the 6 mb add up to 120mb...

Are there further ways to decrease memory usage of extbase, when dealing with complex objects?

Thanks Tim