[TYPO3-core] RFC #5045, #5077, #8649: Import/Export requires lot of memory

[Martin Kutschker]

Bernhard Kraft schrieb:
> Martin Kutschker schrieb:
> 
>> What could be done is to write all files gzipped as temp files. No you
>> know the gzipped size and may create a manifesto/index for all files.
>> After this you'll append the manifesto and all temp files to one file.
>>
>> The final file coulbe appended to the .t3d or live as an addional file
>> together with the xml file.
>>
>> But I don't know if this would make much of a difference.
> 
> You stated below that large files get compressed more efficient than
> many small files ... so I would not try to implement this option

This fact only occured later to me.

>> What could be done is to optimize the whole thing for speed and
>> compression:
>>
>> * group text files together
>>
>> It's said (where?) that it's more efficient to zip a large file than to
>> zip many small.
>>
>> * don't encode/compress/deflate images and other compressed file formats
>>
>> This saves us CPU time.
> 
> Well. Depends on "why" you want to save CPU time in this case - just
> because to avoid a script-timeout - or for saving ressources ?

Because it makes me wait longer. And I would want that feature for any
archiver: don't compress files that match a certain pattern. Why bother
compressing already compressed files?

> An Export/Import is nothing "usual" which is performed every few seconds
> - but rather something occasional only occuring when an admin requires
> too ...

Maybe this will gain you those few extra seconds that saves you from a
PHP timeout error. But it's hard to tell how much you would gain.

> When it is about saving CPU ressources it would be better to optimize
> the FE rendering ... This is one of my goals for this summer ...

Sure.

> If its about avoiding script-timeouts (max_execution_time) I would
> rather take the approach to modify the import/export script in such a
> way, it works similar to the direct-mailer engine.

Do you think my scheme would be so hard to implement? I guessed it would
be rather easy to modify your currents scheme so that it arranges the
files in a certain order.

> The script should run for a specific amount of time - it can check when
> the script-timeout will occur. And before this happens it stores
> everything to disk, sends HTML with a "refresh" JS-code, and loads
> itself again ... this way you can create scripts which can handle really
> a lot of data, and avoid script-timeout by reloading it. It would even
> be possible to give some kind of progress-bar on reload.

Now, *that* sounds complicated. Maybe this can be done better with a
view AJAX calls. I don't trust self-refreshing URLs.

> But this would be step-2 of my changes. As I wrote in a mail before I
> would like to commit this first achievement - it is not very promising
> to only get feature-requests for a new feature. And then see that noone
> reviews the code so it can get commited :(
> 
> We could make a deal. I review one of your larger changes, and you have
> a look at my code ;)

Great. But I'd welcome if you would also review my smaller
contributions. I think I have some stuff pending on this list as well.
As do others...

Masi