[TYPO3-mvc] How to speed up the creation of 100.000+ objects in an extbase extension?

[Claus Due]

On Dec 5, 2012, at 6:45 PM, Thomas Mammitzsch <thomas at visualworx.de> wrote:

> did that those days in flow and have about 160000 entries imported in around 8 minutes (via cli of course). Without that it takes ages because of high ram load and swapping i think.

Hi Thomas,

I'm not sure that's the reason. PHP is unable to use that much RAM (unless
of course you intentionally limit the virtual ram available to a value below
max_memory_limit, which is unlikely).

There's a more likely reason, which is that Extbase tracks dirty objects and
this tracking grows out of crontrol with many thousands of records.

I would let Extbase import a thousand records at a time. Overusing persistAll()
is unwise (and once per 100 operations is, imho, overusing persistAll). A value
of 1000 seems more reasonable. Even when used, persistAll still relies on the
automatic garbage collection in PHP which has a VERY high triggering limit.

And finally: Extbase is not Flow; there's a whole additional DB layer and there's
the unavoiable use of the extremely antique mysql PHP extension (rather than
the new, improved mysqli).

I'd say: use chunked importing. That way you won't stress Extbase and it should
perform much better during huge imports. But for ease, please do try Thomas'
suggestion to see if that on itself gives enough improvement :)

Cheers,
Claus