[TYPO3-core] Taming the Performance in 6.2

[Thomas Maroschik]

Hi,

Am 05.11.13 16:02, schrieb Ernesto Baschny:
 > Package Management
 >
 > Since beta1 together with the Package Management we introduced a new
 > Caching Backend for loading PHP classes and it's class aliases. The new
 > concepts minimizes the amount of memory required by not keeping one huge
 > array in memory, but it seems that has still some performance drawbacks
 > due to heavy use of "require_once". Since this affects all hits (even in
 > frontend with all cached), tackling this area will most probably bring
 > us the biggest benefit.

With the introduced package management several concepts have been 
introduced/reworked along that had to be reflected in the class loader. 
If you ask yourself where do all that require files result from read on:

"Arbitrary length vendor and package namespace"
So now you can use YourCompany\YourSolution\YourWhatever as base 
namespace. But there can also be another package present that uses 
YourCompany\YourSolution as base namespace. This is why you cannot 
derive the package namespace deterministically from the class name. 
Composer and Flow solves that by sorting the base namespaces (called 
prefixes in composer) by length and checking for a match. See [1] for 
code. This means for the worst case, in a system with 60 packages the 
class name to load is been checked up to 59 unmatching prefixes. A 
typical frontend requests loads about 200 classes. This means at worst 
200 * 60 = 12000 strpos calls to resolve the package path of the class 
name. This is still done now but a match is saved in the caching 
framework resulting in a "proxy require" file.

"Keeping nothing more in memory"
With the introduction of namespaces the class alias mechanism has been 
established. Due to the renaming of all class names into more meaningful 
ones a map of about 1350 old to new class names had to be kept in 
memory, because if the old name entered the class loader first the class 
loader has to know that this class name is an old one. The inverse map 
had to be kept too, because if the new name entered the class loader 
first the aliases for the old ones have to be established. This results 
in loading two array with 1350 entries upon bootstrap. Further the 
ext_autoload.php files of extensions have been merged and loaded too, 
which results in another potentially large array. The upfront loading 
took constant long time upon every request. The process changed in the 
new implementation so that a cache entry for each ext_autoload entry and 
class alias is set up resulting in a "proxy require". After the first 
request there is no need to load the 3 arrays anymore and logic to 
retrieve the class file from the original and aliased class name is 
greatly simplified.

"Resolving concurrent write issues"
The old class loader kept a map of class names to class file paths in 
memory. Upon request shutdown when any changes to that map happened. it 
has been simply serialized to a cache entry. If two request run in 
parallel and load different classes new to their class name to class 
file map the longer running request overwrites the cache entry which can 
result in constant writes on the class name to class file  map cache 
file. The "proxy require" file approach mitigates that by creating an 
atomic cache entry for every class name to class file mapping.

All this happens at the expense of IO which impact greatly depends on 
your opcode cache (you really should use one) and the speed of your 
filesystem (if you don't use opcode cache). Basically I'm using the 
filesystem as key value store where the filename is the key and the 
value the class file path and the aliases.

 From the feedback in the community I gathered some more insight into 
their configurations and their filesystems. The performance can be 
increased by a more performant key value store. My current approach I'm 
working on includes getting rid of the proxy requires, the special class 
loader cache backend, replacing that one by a more efficient file based 
key value store based on a append only btree implementation and beeing 
able to use also other existing cache backends as alternative.

Kind regards,

Thomas (Tom) Maroschik

[1] 
https://github.com/composer/composer/blob/master/src/Composer/Autoload/ClassLoader.php#L222

-- 
Thomas Maroschik
TYPO3 CMS Active Contributor

TYPO3 .... inspiring people to share!
Get involved: typo3.org