[TYPO3-english] two questions about table sys_file

[Christian Reiter]

> as the storage is readonly (it is a readonly external mount in unix)
> nobody is able to modify the files on the TYPO3 server. only graphics
> department can modify files, and so they created files with wrong names
> (mixed case + special signs) they needed to rename a lot of files.
> now these files exists with correct names, but TYPO3 does not know them
> with their new name.

This is hardcore ;)

The sha1 approach described in my other post should work. Filesize and 
sha1 hash should have stayed the same if they only renamed files and did 
not overwrite file content.
(if they did both: you need to fix by hand/semi-intelligent guess script 
unless you have captured the file history in some other way.)

So to solve the renames, you could write a small script that

(1) first scandir's your filesystem recursively and registers the 
filename (basename), path (dirname), filesize and and generates SHA1 
hash of file content.

(2) then goes through your sys_file records for that storage. For any 
entry where file_exists() fails for the given storage + identifier, 
check in the result set from (1) if you have a file where the calculated 
SHA1 from filesystem matches the sha1 entry in the sys_file record. 
Really that should be the same file as sha1 collisions are very unlikely.

* Now if that also has the same filesize and dirname but only a 
different basename you are 111% sure it's really the same thing with 
another name and you can definitely change the sys_file record to point 
to it. (Perhaps log the old name)

* If in (2) you find any sys_file entries with non-existing file, and no 
corresponding sha1 from (1), the file is really gone and you can set 
missing=1

>> When you run the scheduler "indexer" task for FAL , those sys_file
>> records should get their "missing" flag.
>
> that is one problem: they does not get the flag.
> Due to a failure in the indexer which I can't get hands on.

I guess there you would need to dig deeper - run some test cases, dig 
into the code.
But since the indexer can't fix the "externally renamed file" problem 
(as in relinking to the new file name) and you will probably need the 
above script anyway, it can do that as well.

I've used the sha1-check successfully in several installations, mostly 
to consolidate duplicated files.

A crazy approach that existed back in DAM/4.x days in one project that 
versioned all its files, was adding the DAM ID to a SVN property of the 
file. This meant that even if you changed the name AND content of the 
file, you could still tell which DAM record it belonged to. Relations 
became practically unbreakable.
Also, an external system that worked with the same repository could 
generate proper references to TYPO3 DAM just by having the file. So if 
you got "document.XYZ.odt", overwrote it with new content, and renamed 
it, an external application could still generate a webservice request 
with the right ID to TYPO3 to exchange metadata about the file. Crazy 
but fun ;)

Best regards,

Christian