[TYPO3-mvc] How to speed up the creation of 100.000+ objects in an extbase extension?

[Thomas Mammitzsch]

hi Claus

yes, of course, it can't consume more memory than allowed in 
memory_limit - i missed that. The basic idea is to not keep thousands of 
objects unpersisted in memory. I had the same problems like Roland, the 
job was running for a very long time to be finished. Persisting after 
creating 1000 objects may be also fine if not even better. I personally 
only tested it with 100 and that worked fine for me. If the import is 
not via cli but apache, i think there is no way around chunking.

regards, Thomas

Am 05.12.2012 19:07, schrieb Claus Due:
> Hi Thomas, I'm not sure that's the reason. PHP is unable to use that 
> much RAM (unless of course you intentionally limit the virtual ram 
> available to a value below max_memory_limit, which is unlikely). 
> There's a more likely reason, which is that Extbase tracks dirty 
> objects and this tracking grows out of crontrol with many thousands of 
> records. I would let Extbase import a thousand records at a time. 
> Overusing persistAll() is unwise (and once per 100 operations is, 
> imho, overusing persistAll). A value of 1000 seems more reasonable. 
> Even when used, persistAll still relies on the automatic garbage 
> collection in PHP which has a VERY high triggering limit. And finally: 
> Extbase is not Flow; there's a whole additional DB layer and there's 
> the unavoiable use of the extremely antique mysql PHP extension 
> (rather than the new, improved mysqli). I'd say: use chunked 
> importing. That way you won't stress Extbase and it should perform 
> much better during huge imports. But for ease, please do try Thomas' 
> suggestion to see if that on itself gives enough improvement :) 
> Cheers, Claus _______________________________________________ 
> TYPO3-project-typo3v4mvc mailing list 
> TYPO3-project-typo3v4mvc at lists.typo3.org 
> http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-project-typo3v4mvc