[TYPO3-dev] problem:: report & analysis: 4.1.2 is coming with codepage inconsistancy

[Martin Bless]

Hi folks,

with this posting I'm trying to contribute. I tracked down something
nasty. 

->->->->->->->->->->->->->->->->->->->->->->->->->->->->->->
Problem:
In a freshly installed Dummy-TYPO3-4.1.2 all UTF-8 installation you
will find that your MySql-dump will contain UTF-8 errors right away.

Analysis:
I did a check on all SQL-files coming with 4.1.2. With two exceptions
they're all Ascii and therefore do fit Latin-1 and UTF-8 as well. But
the two files [file_001] and [file_002] are different. See below.

----- Begin of analysis dump: -----
isAscii,isLatin1,isUtf8,filename
1,1,1,./misc/example_MM_relationTables.sql
1,1,1,./t3lib/stddb/tables.sql
1,1,1,./typo3/sysext/adodb/adodb/session/adodb-sessions.mysql.sql
1,1,1,./typo3/sysext/adodb/adodb/session/adodb-sessions.oracle.clob.sql
1,1,1,./typo3/sysext/adodb/adodb/session/adodb-sessions.oracle.sql
1,1,1,./typo3/sysext/cms/ext_tables.sql
0,1,0, [file_001]./typo3/sysext/cms/ext_tables_static+adt.sql
1,1,1,./typo3/sysext/dbal/ext_tables.sql
1,1,1,./typo3/sysext/impexp/ext_tables.sql
1,1,1,./typo3/sysext/indexed_search/ext_tables.sql
1,1,1,./typo3/sysext/rtehtmlarea/ext_tables.sql
1,1,1,./typo3/sysext/sys_action/ext_tables.sql
1,1,1,./typo3/sysext/sys_note/ext_tables.sql
1,1,1,./typo3/sysext/tsconfig_help/ext_tables.sql
0,1,1,[file_002]./typo3/sysext/tsconfig_help/ext_tables_static+adt.sql
----- End of analysis dump. -----

[file_002] is UTF-8 encoded and since I'm using an all UTF-8
installation works fine.

[file_001] is the problem. It is - in general - Latin-1 encoded except
for six times where the byte sequence ï, ¿, ½ occurs.
These 3 bytes are UTF-8 for Unicode � which is the 'unkown
value' symbol.

Fix:

Step 1: I opened [file_001] in Latin-1 mode and replaced six times the
three bytes '\xef\xbf\xbd' by three quotations marks ('???'). This
will keep file size the same and later on allow to spot these six
instances. 

Step 2: I converted [file_001] to UTF-8.

Step 3: Using the installer I imported the static data again.

Result: The MySql-dump doesn't contain errors any more.

<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-<-

I don't know where to put this information so I'm posting it here. Who
cares? 

It's my feeling that a freshly installed TYPO3 shouldn't have codepage
inconsistancies. It gave me a lot of headache since I was trying to
locate codepage problems anyway and didn't expect TYPO3 to bring along
its own.

Please let me know if and how I can help fixing this in the
distribution.

Have a nice day,

Martin

-- 
http://team-mensch-und-computer.de