[Typo3-dev] Utf8 problems

[Martin T. Kutschker]

Kasper Skårhøj wrote:
> 
> BTW; the far greater problem is that content stored in the database
> might be corrupted. For instnace if a 100 char string (in utf-8, say it
> will be 163 bytes) stored in a varchar(100) will... be cropped! I don't
> know if MySQL 4 has a solution for this or what. In any case TYPO3 will
> issue a warning if cropping occurs right after saving the contnet...

If have implemented and sent you a PHP-implentation (with may use 
mbstring when configured) to do a string truncation on byte lenght. The 
function ensures the integrity of the UTF-8 code and may drop the last 
bytes so no broken UTF-8 character remains.

Though this code has to be put somewhere in the data strorage routines.

Obviously this has to be done also for every other multi-byte encoding. 
I think that I've got the ones right with a similar concept as EUC. East 
Asian users are please get in contact with me, so we can do some testing.

Note on DB: AFAIR POstgreSQL returns an error if you try to store a 
string into a text column that is longer than allowed. No silent truncation!

Masi