[TYPO3-dev] Over 2Million images rendering using GIFBUILDER (but not all at once...)

[Ries van Twisk]

Here are the patches for whoever is interested:

All files are store stored under the GB and pics directory, and sub- 
directories are created based on there
filename's data.

There are 4 additional settings added to config_default.php to set the  
levels and filename's length for gifbuilder and imagemagick.

The functions fileName and createTempSubDir are backwards compatible
and behave the same.

However the function fileName now as an option to not add a tempDir to  
a generated filename. This was
done to prevent funky str_replaces for path creations in the gifBuild  
function

The function createTempSubDir now uses mkdir_deep from t3lib_div to  
create
'deep' (recursive) sub directories

I believe that the patch might not be CGL compatible, I hope to hear  
your comments on that.
If you have any other suggestions/comments/questions. Please let me  
know.

Ries



# This patch file was generated by NetBeans IDE
# Following Index: paths are relative to: /Users/rvt/Sites/topdeals43/ 
typo3/sysext/cms/tslib
# This patch can be applied using context Tools: Patch action on  
respective folder.
# It uses platform neutral UTF-8 encoding and \n newlines.
# Above lines and this line are ignored by the patching process.
Index: class.tslib_gifbuilder.php
--- class.tslib_gifbuilder.php Base (BASE)
+++ class.tslib_gifbuilder.php Locally Modified (Based On LOCAL)
@@ -317,18 +317,37 @@
  	 */
  	function gifBuild()	{
  		if ($this->setup)	{
-			$gifFileName = $this->fileName('GB/');	// Relative to PATH_site
-			if (!file_exists($gifFileName))	{		// File exists
+			    // Filename's hash length with a minimum of 10 maximum of 32  
is set by shortMD5
+			$gb_filename_hash_length = max(10,  
intval($GLOBALS['TYPO3_CONF_VARS']['GFX']['gb_filename_hash_length']));

+			    // Get the filename
+			$filename=$this->fileName('', FALSE);
+
+			    // Create levels for this filename based on the letters with a  
maximum depth of 5
+			$gb_subdirectory_creation_levels = min(5, max(0,  
intval($GLOBALS['TYPO3_CONF_VARS']['GFX'] 
['gb_subdirectory_creation_levels'])));
+
+			    // Protect against maximum of directorys being bigger then the  
filename
+			 
$gb_subdirectory_creation_levels=min($gb_subdirectory_creation_levels,  
$gb_filename_hash_length);
+
+			    // generate path for file
+			$directory='GB/';
+			for($i=0; $i<$gb_subdirectory_creation_levels;$i++) {
+			    $directory.=substr($filename, $i, 1).'/';
+			}
+
+			    // Calculate the new path to the file
+			$pathImageFile=$this->tempPath.$directory.$filename;
+			if (!file_exists($pathImageFile)) {
+
  					// Create temporary directory if not done:
-				$this->createTempSubDir('GB/');
+				$this->createTempSubDir($directory);

  					// Create file:
  				$this->make();
-				$this->output($gifFileName);
+				$this->output($pathImageFile);
  				$this->destroy();
  			}
-			return $gifFileName;
+			return $pathImageFile;
  		}
  	}

@@ -624,13 +643,15 @@
  	 * @return	string		The relative filepath (relative to PATH_site)
  	 * @access private
  	 */
-	function fileName($pre)	{
+	function fileName($pre, $withTmpDirectorypath=TRUE)	{
+		    // Filename's hash length with a minimum of 10 maximum of 32 is  
set by shortMD5
+		$gb_filename_hash_length = max(10,  
intval($GLOBALS['TYPO3_CONF_VARS']['GFX']['gb_filename_hash_length']));

  			// WARNING: In PHP5 I discovered that rendering with freetype of  
Japanese letters was totally corrupt. Not only the wrong glyphs are  
printed but also some memory stack overflow resulted in strange  
additional chars - and finally the reason for this investigation: The  
Bounding box data was changing all the time resulting in new images  
being generated all the time. With PHP4 it works fine.
-		return $this->tempPath.
+		return ($withTmpDirectorypath?$this->tempPath:'').
  				$pre.
  				($GLOBALS['TSFE']->config['config'] 
['meaningfulTempFilePrefix'] ? $GLOBALS['TSFE']- 
 >fileNameASCIIPrefix(implode('_',array_merge($this- 
 >combinedTextStrings,$this- 
 >combinedFileNames)),intval($GLOBALS['TSFE']->config['config'] 
['meaningfulTempFilePrefix']),'_') : '').
-				t3lib_div::shortMD5(serialize($this->setup)).
\ No newline at end of file
+				t3lib_div::shortMD5(serialize($this->setup),  
$gb_filename_hash_length).
\ No newline at end of file
  				'.'.$this->extension();
  	}




# This patch file was generated by NetBeans IDE
# Following Index: paths are relative to: /Users/rvt/Sites/topdeals43/ 
t3lib
# This patch can be applied using context Tools: Patch action on  
respective folder.
# It uses platform neutral UTF-8 encoding and \n newlines.
# Above lines and this line are ignored by the patching process.
Index: class.t3lib_stdgraphic.php
--- class.t3lib_stdgraphic.php Base (BASE)
+++ class.t3lib_stdgraphic.php Locally Modified (Based On LOCAL)
@@ -2325,19 +2325,34 @@
  				$command = $this->scalecmd.' '.$info[0].'x'.$info[1].'! '. 
$params.' ';
  				$cropscale = ($data['crs'] ? 'crs-V'.$data['cropV'].'H'. 
$data['cropH'] : '');

+					// Filename's hash length with a minimum of 10 maximum of 32 is  
set by shortMD5
+				$im_filename_hash_length = max(10,  
intval($GLOBALS['TYPO3_CONF_VARS']['GFX']['im_filename_hash_length']));
+
  				if ($this->alternativeOutputKey)	{
-					$theOutputName = t3lib_div::shortMD5($command. 
$cropscale.basename($imagefile).$this->alternativeOutputKey.$frame);
+					$theOutputName = t3lib_div::shortMD5($command. 
$cropscale.basename($imagefile).$this->alternativeOutputKey.$frame,  
$im_filename_hash_length);
  				} else {
-					$theOutputName = t3lib_div::shortMD5($command.$cropscale. 
$imagefile.filemtime($imagefile).$frame);
+					$theOutputName = t3lib_div::shortMD5($command.$cropscale. 
$imagefile.filemtime($imagefile).$frame, $im_filename_hash_length);
  				}
  				if ($this->imageMagickConvert_forceFileNameBody)	{
  					$theOutputName = $this->imageMagickConvert_forceFileNameBody;
  					$this->imageMagickConvert_forceFileNameBody='';
  				}

+					// Create levels for this filename based on the letters with a  
maximum depth of 5
+				$im_subdirectory_creation_levels = min(5, max(0,  
intval($GLOBALS['TYPO3_CONF_VARS']['GFX'] 
['im_subdirectory_creation_levels'])));
+
+					// Protect against maximum of directorys being bigger then the  
filename
+				 
$im_subdirectory_creation_levels=min($im_subdirectory_creation_levels,  
$im_filename_hash_length);
+
+					// generate path for file
+				$directory='pics/';
+				for($i=0; $i<$im_subdirectory_creation_levels;$i++) {
+					$directory.=substr($theOutputName, $i, 1).'/';
+				}
+
  					// Making the temporary filename:
-				$this->createTempSubDir('pics/');
-				$output = $this->absPrefix.$this->tempPath.'pics/'.$this- 
 >filenamePrefix.$theOutputName.'.'.$newExt;
+				$this->createTempSubDir($directory);
+				$output = $this->absPrefix.$this->tempPath.$directory.$this- 
 >filenamePrefix.$theOutputName.'.'.$newExt;

  					// Register temporary filename:
  				$GLOBALS['TEMP_IMAGES_ON_PAGE'][] = $output;
@@ -2776,7 +2791,7 @@

  			// Making the temporary filename:
  		if (!@is_dir($tmpPath.$dirName))	 {
-			return t3lib_div::mkdir($tmpPath.$dirName);
\ No newline at end of file
+			return t3lib_div::mkdir_deep($tmpPath,$dirName);
\ No newline at end of file
  		}
  	}




# This patch file was generated by NetBeans IDE
# Following Index: paths are relative to: /Users/rvt/Sites/topdeals43/ 
t3lib
# This patch can be applied using context Tools: Patch action on  
respective folder.
# It uses platform neutral UTF-8 encoding and \n newlines.
# Above lines and this line are ignored by the patching process.
Index: config_default.php
--- config_default.php Base (BASE)
+++ config_default.php Locally Modified (Based On LOCAL)
@@ -51,6 +51,10 @@
  		'im_noFramePrepended' => FALSE,			// Boolean. If set, the [x]  
frame indicator is NOT prepended to filenames in stdgraphic. Some IM5+  
version didn't work at all with the typical [0]-prefix, which allow  
multipage pdf's and animated gif's to be scaled only for the first  
frame/page and that seriously cuts down rendering time. Set this flag  
only if your ImageMagick version cannot find the files. Notice that  
changing this flag causes temporary filenames to change, thus the  
server will begin scaling images again which were previously cached.
  		'im_stripProfileCommand' => '+profile \'*\'', // String. Specify  
the command to strip the profile information, which can reduce  
thumbnail size up to 60KB. Command can differ in IM/GM, IM also know  
the -strip command. See http://www.imagemagick.org/Usage/thumbnails/#profiles 
  for details
  		'im_useStripProfileByDefault' => TRUE,	// Boolean. If set, the  
im_stripProfileCommand is used with all IM Image operations by  
default. See tsRef for setting this parameter explocit for IMAGE  
generation.
+		'im_subdirectory_creation_levels' => 0,     // Integer (0-5), Set  
the number of directorys will be created for storing the images in the  
'GB/' directory. This is usefull when you need to process many images  
(>50.000) and you don't want to store them into one directory.
+		'im_filename_hash_length' => 10,            // Integer (10-32), Set  
the length of the filename created for storing the image. This is  
usefull if you process many images (>50.000)and there is a resonable  
reason for filename clashes.
+		'gb_subdirectory_creation_levels' => 0,     // Integer (0-5),  Set  
the number of directorys will be created for storing the images in the  
'pics/' directory. This is usefull when you need to process many  
images (>50.000) and you don't want to store them into one directory.
+		'gb_filename_hash_length' => 10,            // Integer (10-32), Set  
the length of the filename created for storing the image. This is  
usefull if you process many images (>50.000)and there is a resonable  
reason for filename clashes.
  		'jpg_quality' => 70,					// Integer. Default JPEG generation quality
  		'enable_typo3temp_db_tracking' => FALSE,	// Boolean. If set, then  
all files in typo3temp will be logged in a database table. In addition  
to being a log of the files with original filenames, it also serves to  
secure that the same image is not rendered simultaneously by two  
different processes.
  		'TTFLocaleConv' => '',					// String. Enter locale conversion  
string used to recode input to TrueType functions. Eg.  
'cp1250..UTF-8'. Deprecated from ver. 3.6.0 of TYPO3. Set up [BE] 
[forceCharset] as strings are automatically converted from database  
charset to UTF-8.





			regards, Ries van Twisk

-------------------------------------------------------------------------------------------------
tags: Freelance TYPO3 Glassfish JasperReports JasperETL Flex Blaze-DS  
WebORB PostgreSQL DB-Architect
email: ries at vantwisk.nl        web:   http://www.rvantwisk.nl/     
skype: callto://r.vantwisk
Phone: +1-810-476-4196    Cell: +593 9901 7694                   SIP:  
+1-747-690-5133










			regards, Ries van Twisk

-------------------------------------------------------------------------------------------------
tags: Freelance TYPO3 Glassfish JasperReports JasperETL Flex Blaze-DS  
WebORB PostgreSQL DB-Architect
email: ries at vantwisk.nl        web:   http://www.rvantwisk.nl/     
skype: callto://r.vantwisk
Phone: +1-810-476-4196    Cell: +593 9901 7694                   SIP:  
+1-747-690-5133