[TYPO3-dev] The extbase dilemma

[Daniel Brün]

Hi folks,

writing this post is feeling awkward to me, but I have to write it
nevertheless.

I have been around the TYPO3-universe for quite some time now and my company
has implemented many TYPO3-projects and many, many extensions (most of them
not in TER).
Last year we started developing salt_news, a powerful news-system for TYPO3
based on extJS and extbase that I talked about at T3CON10.

Today I consider the decision to develop the system using extbase a major
mistake on our side.

Let me tell you why:

extbase is a backport of a framework (FLOW3) that is not production-ready
itself now, and certainly was far from it by the time the first version of
extbase surfaced. Strange enough, many extbase-based extensions are already
used in production context, while there are no real FLOW3-projects live.

During the last years, I have on many occasions discussed FLOW3 and its
architecture with the FLOW3-team and I still believe this is going to be one
of the most sophisticated PHP-frameworks around - once it is finished. Maybe
this influenced our decision to use the backport for a bigger project than
the average edit-this-in-backend-and-show-that-in-frontend extension.
When we started developing salt_news we had to decide which framework and
technology to use and only after reading rave reviews and success stories
about extbase we decided to build our system upon it - dropping symfony and
scala/lift from our list. Having intensively used all kinds of other
languages and frameworks we were eager to see the future of TYPO3.

The first thing we noticed: extbase does not scale. At all. And no, it's not
our coding style, it's just a framework that burns CPU, and Fluid puts oil
onto the fire. OK, we thought, just setup a cluster of apaches and mysqls,
make use of nginx and varnish for balancing and caching and everything will
be fine. Turns out that the few requests reaching the apaches still manage
to bog the servers down. After implementing more caching and bypassing the
persistance layer we could rise the performance to an acceptable level -
thinking about the hardware involved for this project makes me sick though.

The quest for performance followed us through the entire project and I won't
go into details regarding performance here.

But there are other, way more important things going wrong in extbase.
Instead of making use of an existing ORM (like propel or doctrine) the
extbase-team decided to develop something ORM-like by themselves - as a
byproduct. Needless to say, doctrine and propel have been under development
for a couple of years to reach their current speed and stability.
What does this mean for extbase? First of all: no transactions, no locks,
and therefore - race conditions everywhere. Don't believe this? Setup a
simple domain model for authors and books. An author may have written many
books, and books may have multiple authors, requiring a many-to-many
relationship between books and authors. Now add a book to the list of books
by an author and persist the changes. Guess what happens? Right: First of
all, the list of associated books is deleted in the database. Then a new
list (including the new book) is written to the database. Any queries
inbetween deliver a wrong result, or no result at all. The great thing about
this: you can not tell whether the result is correct or not. Your cache will
save the result, so will your varnish. Of course, this will only happen now
and then. Great for debugging!
This kind of race condition is found in many different places all around the
framework and it is a major task for our developers to code around them.
Another example: Create a new object. When persisting this object a new row
in the database is created with default values. It is only after this
operation that the real values are written using UPDATE. Bad luck for any
query that is executed in between. *shakehead*
Did I mention no transactions? Basically every save-operation that touches
more than a single row in the database is racy.
The more traffic there is on your site, the more likely is the appearance of
race-conditions. Think of any serious business-application, bookings,
calculated sums, ...

In addition to the lack of transactions, extbase does not support database
constraints, either. This means, that all relations between objects and
their data have to be taken care of in PHP, instead of the database.
Obviously, this can not work out given the size of a typical race window.
 Whenever writing to the database is unsuccesful, your application is in an
undefined state and in most cases, extbase simply crashes. Due to the lack
of transactions the database is then left in the intermediate state is was
in when extbase crashed... no rollback possible, your database is now
inconsistent.

In order to find out which data has to be persisted, extbase goes through
_all_ objects used in the current request, even if they were just used for
reading. It does so, because there is no dirty-flag. This check is extremely
expensive and gets even more expensive if there are many extbase-plugins on
a page, because then _all_ objects are checked again for _every_ plugin! 10
plugins on a page with 100 objects each = 10 times (10 plugins * 100
objects) = 10.000 objects checked every time you visit that page in your
browser. No, I am not kidding, _every_ object registered for any plugin on a
page is checked for every plugin.

The good feeling you get from domain driven design is heavily hampered by
the weak spot in extbase: the persistance layer. In some circumstances,
touching an object the wrong way initiates a persist-cascade, that not only
takes ages to complete but also has some other nifty side-effects. MM-tables
can not have primary keys? Aha. No composite keys possible, just one single
UID? Hmm. No additional attributes for M:N-relations supported? Grmmml.
Of course, some of these limitations are inherited from TYPO3. The
difference being that in a classic TYPO3-extension you have all means to
access whatever database you want in any way you like. extbase isolates you
from the underlying database, which is ok for a full-blown ORM that allows
for almost any operation that you could think of in SQL. extbase does not
have such an ORM.
Generally speaking, a framwork should be able to handle whatever domain
model you throw at it. In extbase, you often have to alter your domain model
in order to be able to implement it at all. Bad!

I mentioned earlier that other ORMs grew over the years and sport a big deal
of performance optimizations and other great ideas to make handling even
huge amounts of data comfortable. These ORMs leverage the databases
capabilities instead of imitating their behaviour. Unfortunately, in extbase
this is not the case, making queries on large datasets incredibly slow.

extbase is not able to model complex queries that involve more than simple
inner joins. Ever tried to build a tag cloud in extbase? No group-by
available, no complex joins in the current release. Speaking of new
releases: the latest and greatest extbase release, the one bundled with
TYPO3 4.5.2, is so incompatible to last years release that it requires a
major rewrite of our codebase! Hooray.

The main purpose of extbase was to prepare developers for the next major
release of TYPO3: Version 5. Well, it really is not much more than some
slight "preparation". Looking at the code I can not believe that anything
can be easily transferred from extbase to v5. A complete  rewrite will be
the cleaner and faster solution in most cases - want to bet?

Thanks to extbase, we are in trouble. We have a huge project that can not be
easily upgraded to the latest version of TYPO3, it is slow, it has more
workarounds than real code. In no way does it live up to our standards. All
I have written above is still valid for the current release of extbase,
while first perfomance tests even show a 30% decrease in perfomance. This
mixture of design flaws and simple bugs make me feel very, very
uncomfortable. And worst: our developers simply do not want to use extbase
ever again.
Our solution to this problem is simple: We will drop extbase and replace the
core of our system with something stable, scalable and reliable.

You might ask why I write all this. It's because I like TYPO3, I have seen
it grow up, I have met all you great guys who code it and heck - I really
dig FLOW3! This is going to be awesome!
And that is exactly the problem: We managed to make TYPO3 enterprise-ready
(whatever that means), we have a great framework and an even greater CMS at
the horizon... and we mess it all up by making some halfheartedly put
together semi-framework that is neither capable of running enterprise-scale
projects nor is it in any thinkable way future proof.

There are just two ways out of this:
a) Fix extbase as soon as possible. Given the kind of weaknesses we found,
this is not as easy as it sounds... and I am sure we only scratched the
surface.
b) Get real and make clear that using extbase is optional and experimental.
I do not want to see others run into the same problems we ran into.

Regards,
Dan