[TYPO3-core] RFC #11361: Backend TYPO3 AJAX-Interface: json encoded result empty if it contains non-utf-8 characters

[Niels Pardon]

Hi Benni!

I guess that this issue is related: http://bugs.typo3.org/view.php?id=10517

Could you take a look at it?

Thanks!

Greets,

Niels


Benjamin Mack schrieb:
> Hey all,
> 
> this is SVN patch request.
> 
> Branches: 4.2 and trunk
> 
> BT reference: http://bugs.typo3.org/view.php?id=11361
> 
> Type: a classic bug
> 
> Description:
> Currently, in the TYPO3 AJAX interface for the backend there is an
> option to encode the response that is sent back to the browser via JSON.
> 
> The php-internal function "json_encode" is used for sending back the
> returned array as a JSON string. However, this function (as documented)
> works only on UTF-8 encoded data. Test results have proven that if you
> send json_encode an array with a string that contains non-unicode
> encoded symbols (like Umlauts in iso-8859-1) that an empty result is
> returned from this function.
> 
> This basically means that before any data is sent back to the browser
> via the JSON format that it needs to be converted to Unicode, otherwise
> the response will be empty.
> 
> Solution:
> Let's check the localCharset (e.g. with forceCharset) if it is unicode
> before converting the response array to a JSON string. Send the JSON
> encoded string as UTF-8 then.
> 
> Thus: *every* server call that is returned via JSON needs to be
> UTF8-encoded. Should probably update the documentation as well then. I
> will take care of this as well.
> 
> Steps to reproduce:
> 1. Make sure your TYPO3 Backend runs in a non-unicode environment
> 2. Look for a registered AJAX call where you return a json response
> 3. Make sure the data that is sent back to the browser from the server
> includes some umlauts or something non-standard.
> 4. See with Firebug what the response is with JSON. Without the patch,
> the response will be empty (there will be no error), with the patch, the
> response will be as it should be.
> Additional Information     Please note:
> This is different issue than we had before with the $_REQUEST parameters
> being encoded in a wrong format (as javascript always sends unicode to
> the server, and we need to convert them to the force charset). However
> this issue is related to *sending back* data FROM the server TO the
> browser.
> 
> Btw: MAYBE this is also related to the mystery of the "relogin does not
> work anymore" errors...
> 
> FAQ:
> Why hasn't this been an issue before?
> Well, most of the core-related AJAX calls so far deliver either
> non-problematic symbols or data in XML or plain text format, not JSON.
> Of course, this problem only occurs when you have a non-unicode
> environment running.
> 
> How did you stumble over this?
> In the Advanced Frontend Editing we send quite a lot of data (the
> TCEforms :)) back to the browser with JSON. Although we'll probably
> change the way this is sent (using an iframe), it did not work on my
> system with the current configuration.
> 
> 
> Thanks for testing and reviewing.
> 
> All the best,
> Benni.
>