[TYPO3-mvc] How to speed up the creation of 100.000+ objects in an extbase extension?

[Claus Due]

Hi Roland,

Stefano said:

> DIG EXTBASE for this operation
> go straight with DB->insert in the fe_user table!

And I agree. But if for some reason you depend on this happening in Extbase:

Try with chunked imports. From your example it does not look like you need
to create relations when importing so I assume you could import ~1000 records
at a time?

If you are able to do it this way it is likely to go faster. With the example you gave
below I would estimate that each chunk of ~1000 records would take a couple
of seconds to execute, totalling a lot less than 60 minutes.

And of course avoid selecting all rows while you are importing; make the import
run without the need to check existing records and you will have saved a LOT of
resources if importing through Extbase.

Last one; maybe unneccesary to even mention, but don't display the data you
import inbetween chunks - Extbase is not very efficient at doing this with that
much data; assume your list of 1000 records takes ~1 second to render you just
lost a good, full 1000 seconds - which is almost 17 minutes spent on just
rendering imported records ;)

Cheers,
Claus