[TYPO3-core] RFC #5045, #5077, #8649: Import/Export requires lot of memory

[Bernhard Kraft]

Martin Kutschker schrieb:

> What could be done is to write all files gzipped as temp files. No you
> know the gzipped size and may create a manifesto/index for all files.
> After this you'll append the manifesto and all temp files to one file.
> 
> The final file coulbe appended to the .t3d or live as an addional file
> together with the xml file.
> 
> But I don't know if this would make much of a difference.

You stated below that large files get compressed more efficient than 
many small files ... so I would not try to implement this option (hey - 
this all requires quite some time !)


> What could be done is to optimize the whole thing for speed and compression:
> 
> * group text files together
> 
> It's said (where?) that it's more efficient to zip a large file than to
> zip many small.
> 
> * don't encode/compress/deflate images and other compressed file formats
> 
> This saves us CPU time.

Well. Depends on "why" you want to save CPU time in this case - just 
because to avoid a script-timeout - or for saving ressources ?
An Export/Import is nothing "usual" which is performed every few seconds 
- but rather something occasional only occuring when an admin requires 
too ...

When it is about saving CPU ressources it would be better to optimize 
the FE rendering ... This is one of my goals for this summer ...

If its about avoiding script-timeouts (max_execution_time) I would 
rather take the approach to modify the import/export script in such a 
way, it works similar to the direct-mailer engine.
The script should run for a specific amount of time - it can check when 
the script-timeout will occur. And before this happens it stores 
everything to disk, sends HTML with a "refresh" JS-code, and loads 
itself again ... this way you can create scripts which can handle really 
a lot of data, and avoid script-timeout by reloading it. It would even 
be possible to give some kind of progress-bar on reload.

But this would be step-2 of my changes. As I wrote in a mail before I 
would like to commit this first achievement - it is not very promising 
to only get feature-requests for a new feature. And then see that noone 
reviews the code so it can get commited :(

We could make a deal. I review one of your larger changes, and you have 
a look at my code ;)



greets,
Bernhard
-- 
Freiheit ist immer Freiheit des Andersdenkenden.
Rosa Luxemburg, 1871-1919
--------------------------------------------------
www.think-open.at