[TYPO3-core] RFC #12232: Bug: md5_file() to check if a file has been changed is very expensive [performance]

[Vladimir Podkovanov]

On 21.10.2010 15:54, André Stösel wrote:
>>> +1, I will argue against every new option, if I don't see the big
>>> need for it - but support everybody who makes efforts to remove some
>>> of the useless config options!
>>>
>>> So -1 for a new option.
>>>
>>> In the last time, we usually used filemtime + filesize as check,
>>> whether a file has changed.
>>
>> The new patch (without any options) is attached.
>
> Today i read the last patch again and changed it a bit.
> The filesize-condition before the filemtime-condition doesn't make any
> sense without the flag to force the md5_file-check.
>
> v3 attached ;)

Hi André!
Thank you for the patch, though I think it is not ready to go.

I think Steffen meant that filemtime + filesize are enough and 
md5_file() could be dropped if no one knows usecase for it. If somebody 
uses symlinks he could find a workaround like updating them with shell 
touch command after file changed.
Anyways the t3lib_stdgraphic class will not work with every file 
uploaded for example via FTP because it doesn't understand some symbols 
in filenames, so best way to provide files is upload via TYPO3, all 
other ways of providing files should be careful and take into account 
TYPO3 core limitations. So IMHO we shouldn't too much care for unusual 
cases here as core itself doesn't provide mush freedom with files.

On your patch ver 3:
I don't like that patch is working only on new images, and still uses 
md5_file() for old ones. It means it is not fully fixes bug for existing 
sites. For example I have on one site 100 gigabytes of user uploaded 
photos, so if I want to fully fix the bug I need to empty table and 
rebuild all images and few days needed for it.

I think it could be done like this:

a) make patch which doesn't use md5_file() and doesn't take into account 
old files, so if filemtime or filesize don't match then image rebuilds

b) by the way we don't need new DB field then, we can use old md5hash 
field, so no DB upgrade needed:
'md5hash' => md5(filesize($identifyResult[3]) . 
filemtime($identifyResult[3])),

c) AND make an extension that will parse cache_imagesizes table and 
recalculates all hashes, it will be fast and no rebuilds needed then.

-- 
-rgds-
Vladimir