[TYPO3-core] RFC #5045, #5077, #8649: Import/Export requires lot of memory

[Martin Kutschker]

Bernhard Kraft schrieb:
> Steffen Kamper schrieb:
>>
>> I did a simular in mysqldumper. The code is old, so it's not best
>> style, but it works writing to gz.
>> I simply used
>> $fp = gzopen ($df,'ab');
>> gzwrite ($fp,$dump['data']);
>>
>> so you append the data without any seek.
>> Do you write the data not sequentially?
> 
> Probably writing of data would be no problem - the problem occurs when
> reading the data - it can be that when reading the file-container a seek
> is required - seek for gzipped files takes very long as this gets
> "emulated" ...

What could be done is to write all files gzipped as temp files. No you
know the gzipped size and may create a manifesto/index for all files.
After this you'll append the manifesto and all temp files to one file.

The final file coulbe appended to the .t3d or live as an addional file
together with the xml file.

But I don't know if this would make much of a difference.

What could be done is to optimize the whole thing for speed and compression:

* group text files together

It's said (where?) that it's more efficient to zip a large file than to
zip many small.

* don't encode/compress/deflate images and other compressed file formats

This saves us CPU time.

Combining these ideas means that first you build three groups: text,
binary, compressed binaries. Then the appended text and binary data will
be compressed. To this the already compressed binaries are added. Of
course this also means that you need a header which tells you where the
uncompressed data begins.

Masi