[TYPO3-core] RFC: Bug #13471: List module eats up tons of memory if displaying records with many references

[Christian Kuhn]

This is a SVN patch request.

Type: bugfix, performance improvement

Branches: trunk, 4_3

BT: http://bugs.typo3.org/view.php?id=13471

Problem:
The list module is very wasteful with memory if handling rows with many 
references. It loads a SELECT * from sys_refindex for every displayed 
row of a table to a local class array, just to show a count() and a 
cropped title tag of the gathered data.

Solution:
- Introduce a new method with a db query to get and store counts of 
referenced records in a local array.
- Introduce a new method to fetch referenced records from db as long as 
crop length for title tag has not been reached, stop then and free query 
result.
- Use new methods within the list module and deprecate old code.

Notes:
- This improvement dropped memory consumption of the list module for a 
page listing a fe_group row, referenced by 45.0000 fe_users, from ~270MB 
down to ~10MB. Parsetime dropped from ~1800ms to ~130ms.
- For 4.3 I propose a not so radical change in the current method that 
changes the current reference query from "select *" to "select only 
needed fields". This typically halfs the memory usage.
- This leak was found with php quick profiler [1]. In contrast to xdebug 
it's possible to easily profile memory usage.
- Only the _trunk and _4-3 patches will be committed, the rest of the 
attachments are for a test setup.

How to test:
1. Install attached pqp profiler extension
2. Apply _example patch
3. Import the t3d file that comes with a sysfolder page containing ~4k 
fe_users, all referencing one fe_group
4. Call the list module on the new page, pqp should show a memory usage 
of ~20MB, this will raise for every new fe_user in the fe_group
5. Apply _trunk or _4-3 patch and watch significant reduction in 
parsetime and memory usage.

Regards
Christian


[1] http://particletree.com/features/php-quick-profiler/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 13471_02_4-3.diff
Type: text/x-patch
Size: 496 bytes
Desc: not available
URL: <http://lists.typo3.org/pipermail/typo3-team-core/attachments/20100417/b00073ac/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 13471_02_example.diff
Type: text/x-patch
Size: 652 bytes
Desc: not available
URL: <http://lists.typo3.org/pipermail/typo3-team-core/attachments/20100417/b00073ac/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 13471_02_trunk.diff
Type: text/x-patch
Size: 6067 bytes
Desc: not available
URL: <http://lists.typo3.org/pipermail/typo3-team-core/attachments/20100417/b00073ac/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: T3D__17-04-18-36-10-z.t3d
Type: application/octet-stream
Size: 132067 bytes
Desc: not available
URL: <http://lists.typo3.org/pipermail/typo3-team-core/attachments/20100417/b00073ac/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: T3X_phpqp-0_0_0-z-201004171822.t3x
Type: application/octet-stream
Size: 12801 bytes
Desc: not available
URL: <http://lists.typo3.org/pipermail/typo3-team-core/attachments/20100417/b00073ac/attachment-0003.obj>