[TYPO3-core] RFC #5045, #5077, #8649: Import/Export requires lot of memory

[Bernhard Kraft]

Martin Kutschker schrieb:

> I didn't get it right from readig the patch. You have create an external
> temporary zip file which you append to the .t3d and flag it so that the
> importer can detach and uncompress it. Is it working like this?

*smile* I hoped someone asks this ...

As usual I like to be very explicit and go into every detail when 
explaining something. So I hope it is not too long for you too read 
(come on - manuals have hundreds of pages - this is just a  half A4 :).

We use hundreds of lines of code for solving our "requirements" - so I 
think it should be ok to use tens of lines of writings/mails do 
"describe" our solutions.

****************************

You surely remember good old games like "Monkey Island", "Indiana Jones" 
or other Lucas Arts Adventures, or games like "Doom". Before those games 
a computer game in DOS worlds was often built up of hundreds of small 
files like "sprt0001.spr"  for small sprite or sound effect.
But games like noted above started to take an approach which has one or 
more "container" files (when games were distributed playable from 5.25" 
floppies or similar they simply used more than one container).

In the container of the first disk was a kind of "index" which told the 
game-engine at which byte-offset it could find which game-element 
(sprite, sound effect, level map, items or whatever) and which length in 
bytes this data element had.

So I used a similar approach. In the code of the originial import export 
module you find a block reading:
---------------------------------------------
$fileRec['content'] = t3lib_div::getUrl($fI['ID_absFile']);
$fileRec['content_md5'] = md5($fileRec['content']);
---------------------------------------------
Here the complete file gets read into array key 'content' and a md5 sum 
is created.

Now this is the biggest problem in the import/export tool - if you 
export i.e. 50 files each 2MB in overall size you will fill up 100 MB of 
memory - you get the point. Exporting "small" page trees is possible - 
large trees with lot of pictures/pdfs, etc. wouldn't work.

I replaced above code by:
--------------------------------------------
$fileRec['content'] = 
$this->export_appendFileToContainer($fI['ID_absFile']);
$fileRec['content_md5'] = $fileRec['content']['md5'];
--------------------------------------------
So I set the 'content' key of the $fileRec array to the return value of 
"export_appendFileToContainer". This method simply appends the file with 
supplied name to a temporarily created container (plain at this time - 
not zipped) and returns the byte offset and length where the file has 
been put to (in the container) ...

$fileRec gets simply handled like before my patch - inserted into the 
export-data-array which will be serialized and stored as .T3D file. But 
after serialization the contents of the temporarily created container 
gets appended to the .T3D file in a manner it can get "extracted" from 
the T3D file again (when importing)

When importing the same game goes in the other direction - a file going 
to be saved from the import-t3d is not simply saved from the 'content' 
key of the array - but rather "fetched" out of the container.



> I guess this could work for the xml export as well if you ship the file
> zip as an extra file. Not quite as convenient as a single file but it'd
> work. Though I don't get it why one would need an xml export.

The thing about the "zip" or better "gzip" methods is the following: 
When creating the .t3D and compression is active (this is the default) 
the serialized-data gets written to disk (or stdout) using the "gzwrite" 
method.

When writing the files to the container I can not use "gzwrite" as later 
on - when importing it might be required to extract a file from the 
"middle" of the container - and gz-files must/should always get 
read/written serially - no skipping should occur:
---------------------------
http://de.php.net/manual/en/function.gzseek.php
If the file is opened for reading, this function is emulated but can be 
extremely slow. If the file is opened for writing, only forward seeks 
are supported; gzseek() then compresses a sequence of zeroes up to the 
new starting position.
---------------------------

So I simply compress the whole container when finished - and append the 
whole compressed container. When importing I first decompress the whole 
file container - and then simply proceed as usual.


> I don't want to go into nitpicking (like "use of new instead of
> t3lib_div::createInstance) right now as I like the idea :-)

Sorry. At the moment of writing I didn't have t3lib_div::makeInstance in 
mind :) I will fix this.



> PS: Ok, onme thing: could you please name the feature internally
> sonething like "externalFiles" or "fileContainer"? "saveMemory" doesn't
> really tell what happens. ;-)

Yes. Give me 1 hour and you get a fixed version.



greets,
Bernhard
-- 
Freiheit ist immer Freiheit des Andersdenkenden.
Rosa Luxemburg, 1871-1919
--------------------------------------------------
www.think-open.at