[TYPO3-mvc] [solved] Memory Usage with 2000 Domain Object

[Christian Zenker]

On Wed, 23 Mar 2011 09:11:22 +0100, Tim Schoch | GSTALTIG  
<tim.schoch at gstaltig.ch> wrote:

Hi Tim.

I really appreciate your research, as I had to do something similar in my  
two extensions "cz_simple_cal" and "extensible_sitemap". But your point  
number two was the thing I could not figure out how to do.
I guess that unregistering objects could even help saving time: I already  
worked with subqueries, but persistAll gets significantly slower the more  
objects you have fetched from the DB. I assume that this happens because  
each object has to be checked if it was changed. As I can't be sure the  
script won't reach its time or memory limit I do a persist all after each  
subquery. So most objects will be checked if they need persistence  
multiple times. But by design, I can be sure that I won't need any object,  
that I've already seen, again. So it would be helpful to unregister  
objects that were already processed.
I'll give this a try as soon as possible.

As it seems, fetching a whole bunch of objects from the DB and processing  
them is not that uncommon. Maybe we could implement a class in Extbase  
that could simplify that task. This class would basically be a  
Persistence_QueryResult that implements the Iterator Interface (not the  
ArrayAccess Interface) and holds the logic for doing subqueries and  
unregisters objects.

Best regards,
Christian.


> Hi
>
> Oukay, first thing I found the 6 mb: One of the tested filter was  
> spelled wrong so call_user_func( array( $item, 'get' . $filter ) )
> threw a Notice every time it got called. I fixed that, but still got the  
> memory fed up with cached objects.
> So my conclusion when iterating over a large amount of objects is:
>
> 1. Large Query vs multiple Subqueries
> There is near to no difference in terms of runtime but the memory impact  
> with one large query is huge, as all the objects get
> loaded on the first iteration and stored in the session and identityMap.  
> With multiple subqueries only the subset of objects are
> stored and will be cleared to 0 after every subquery is done. For me,  
> 100 items per cycle works fine, but you could reduce this
> number even further to decrease the memorpeak. ( see next adn conclusion  
> )
>
> 2. Clear Session and IdentityMap
> If you don't need the fetched Object Models you can free the memory by  
> calling
>
>           $this->persistenceManager->getSession()->unregisterReconstitutedObject(  
> $item );
>           $this->identityMap->unregisterObject( $item );
>           unset( $item );
>
> for every item you've processed.
>
> 3. lazy loading is not the problem
> I found, that the lazy loading proxy isn't the problem as we thought  
> first. In terms of performance you should leave it on.
>
>> Conclusion
> To collect the data of 2500 Items each with 10 relations ( both 1:n and  
> m:n ) it takes around 30 seconds. The memory peak for the
> whole process is 25mb and after it's finished the used memory has  
> increased by 12mb, but thats mainly because I save the
> generated index to other Objects that are not persistet at this point  
> yet.
>
> For me this solution is ok, I hope this helps others to.
> Thank you to Lienhart and Christian
>
> Cheers, Tim