[TYPO3-core] RFC #11361: Backend TYPO3 AJAX-Interface: json encoded result empty if it contains non-utf-8 characters

[Benjamin Mack]

Hey all,

this is SVN patch request.

Branches: 4.2 and trunk

BT reference: http://bugs.typo3.org/view.php?id=11361

Type: a classic bug

Description:
Currently, in the TYPO3 AJAX interface for the backend there is an 
option to encode the response that is sent back to the browser via JSON.

The php-internal function "json_encode" is used for sending back the 
returned array as a JSON string. However, this function (as documented) 
works only on UTF-8 encoded data. Test results have proven that if you 
send json_encode an array with a string that contains non-unicode 
encoded symbols (like Umlauts in iso-8859-1) that an empty result is 
returned from this function.

This basically means that before any data is sent back to the browser 
via the JSON format that it needs to be converted to Unicode, otherwise 
the response will be empty.

Solution:
Let's check the localCharset (e.g. with forceCharset) if it is unicode 
before converting the response array to a JSON string. Send the JSON 
encoded string as UTF-8 then.

Thus: *every* server call that is returned via JSON needs to be 
UTF8-encoded. Should probably update the documentation as well then. I 
will take care of this as well.

Steps to reproduce:
1. Make sure your TYPO3 Backend runs in a non-unicode environment
2. Look for a registered AJAX call where you return a json response
3. Make sure the data that is sent back to the browser from the server 
includes some umlauts or something non-standard.
4. See with Firebug what the response is with JSON. Without the patch, 
the response will be empty (there will be no error), with the patch, the 
response will be as it should be.
Additional Information 	Please note:
This is different issue than we had before with the $_REQUEST parameters 
being encoded in a wrong format (as javascript always sends unicode to 
the server, and we need to convert them to the force charset). However 
this issue is related to *sending back* data FROM the server TO the browser.

Btw: MAYBE this is also related to the mystery of the "relogin does not 
work anymore" errors...

FAQ:
Why hasn't this been an issue before?
Well, most of the core-related AJAX calls so far deliver either 
non-problematic symbols or data in XML or plain text format, not JSON. 
Of course, this problem only occurs when you have a non-unicode 
environment running.

How did you stumble over this?
In the Advanced Frontend Editing we send quite a lot of data (the 
TCEforms :)) back to the browser with JSON. Although we'll probably 
change the way this is sent (using an iframe), it did not work on my 
system with the current configuration.


Thanks for testing and reviewing.

All the best,
Benni.