[TYPO3-dev] The extbase dilemma

[Oliver Hader]

Hi Daniel et all,

thanks for bringing this topic up and discussing the concerns on Extbase
in a constructive way here in the list. We (some of the v4 and v5 guys)
just had a R&D Meeting on Skype and also discussed about this very thread.

Since it seems to be easier to find solutions for particular problems in
a real-time discussion, we aim to have a Skype Meeting with those who
experienced problems and those who take over responsibilities in the
accordant development teams. This meeting shall happend during the next
two weeks and will hopefully help to find a strategy on how to handle
the situation now and in the future. Invitations will be sent
individually very soon.

Best regards,
Olly


Am 18.05.11 11:19, schrieb Daniel Brün:
> Hi folks,
> 
> writing this post is feeling awkward to me, but I have to write it
> nevertheless.
> 
> I have been around the TYPO3-universe for quite some time now and my company
> has implemented many TYPO3-projects and many, many extensions (most of them
> not in TER).
> Last year we started developing salt_news, a powerful news-system for TYPO3
> based on extJS and extbase that I talked about at T3CON10.
> 
> Today I consider the decision to develop the system using extbase a major
> mistake on our side.
> 
> Let me tell you why:
> 
> extbase is a backport of a framework (FLOW3) that is not production-ready
> itself now, and certainly was far from it by the time the first version of
> extbase surfaced. Strange enough, many extbase-based extensions are already
> used in production context, while there are no real FLOW3-projects live.
> 
> During the last years, I have on many occasions discussed FLOW3 and its
> architecture with the FLOW3-team and I still believe this is going to be one
> of the most sophisticated PHP-frameworks around - once it is finished. Maybe
> this influenced our decision to use the backport for a bigger project than
> the average edit-this-in-backend-and-show-that-in-frontend extension.
> When we started developing salt_news we had to decide which framework and
> technology to use and only after reading rave reviews and success stories
> about extbase we decided to build our system upon it - dropping symfony and
> scala/lift from our list. Having intensively used all kinds of other
> languages and frameworks we were eager to see the future of TYPO3.
> 
> The first thing we noticed: extbase does not scale. At all. And no, it's not
> our coding style, it's just a framework that burns CPU, and Fluid puts oil
> onto the fire. OK, we thought, just setup a cluster of apaches and mysqls,
> make use of nginx and varnish for balancing and caching and everything will
> be fine. Turns out that the few requests reaching the apaches still manage
> to bog the servers down. After implementing more caching and bypassing the
> persistance layer we could rise the performance to an acceptable level -
> thinking about the hardware involved for this project makes me sick though.
> 
> The quest for performance followed us through the entire project and I won't
> go into details regarding performance here.
> 
> But there are other, way more important things going wrong in extbase.
> Instead of making use of an existing ORM (like propel or doctrine) the
> extbase-team decided to develop something ORM-like by themselves - as a
> byproduct. Needless to say, doctrine and propel have been under development
> for a couple of years to reach their current speed and stability.
> What does this mean for extbase? First of all: no transactions, no locks,
> and therefore - race conditions everywhere. Don't believe this? Setup a
> simple domain model for authors and books. An author may have written many
> books, and books may have multiple authors, requiring a many-to-many
> relationship between books and authors. Now add a book to the list of books
> by an author and persist the changes. Guess what happens? Right: First of
> all, the list of associated books is deleted in the database. Then a new
> list (including the new book) is written to the database. Any queries
> inbetween deliver a wrong result, or no result at all. The great thing about
> this: you can not tell whether the result is correct or not. Your cache will
> save the result, so will your varnish. Of course, this will only happen now
> and then. Great for debugging!
> This kind of race condition is found in many different places all around the
> framework and it is a major task for our developers to code around them.
> Another example: Create a new object. When persisting this object a new row
> in the database is created with default values. It is only after this
> operation that the real values are written using UPDATE. Bad luck for any
> query that is executed in between. *shakehead*
> Did I mention no transactions? Basically every save-operation that touches
> more than a single row in the database is racy.
> The more traffic there is on your site, the more likely is the appearance of
> race-conditions. Think of any serious business-application, bookings,
> calculated sums, ...
> 
> In addition to the lack of transactions, extbase does not support database
> constraints, either. This means, that all relations between objects and
> their data have to be taken care of in PHP, instead of the database.
> Obviously, this can not work out given the size of a typical race window.
>  Whenever writing to the database is unsuccesful, your application is in an
> undefined state and in most cases, extbase simply crashes. Due to the lack
> of transactions the database is then left in the intermediate state is was
> in when extbase crashed... no rollback possible, your database is now
> inconsistent.
> 
> In order to find out which data has to be persisted, extbase goes through
> _all_ objects used in the current request, even if they were just used for
> reading. It does so, because there is no dirty-flag. This check is extremely
> expensive and gets even more expensive if there are many extbase-plugins on
> a page, because then _all_ objects are checked again for _every_ plugin! 10
> plugins on a page with 100 objects each = 10 times (10 plugins * 100
> objects) = 10.000 objects checked every time you visit that page in your
> browser. No, I am not kidding, _every_ object registered for any plugin on a
> page is checked for every plugin.
> 
> The good feeling you get from domain driven design is heavily hampered by
> the weak spot in extbase: the persistance layer. In some circumstances,
> touching an object the wrong way initiates a persist-cascade, that not only
> takes ages to complete but also has some other nifty side-effects. MM-tables
> can not have primary keys? Aha. No composite keys possible, just one single
> UID? Hmm. No additional attributes for M:N-relations supported? Grmmml.
> Of course, some of these limitations are inherited from TYPO3. The
> difference being that in a classic TYPO3-extension you have all means to
> access whatever database you want in any way you like. extbase isolates you
> from the underlying database, which is ok for a full-blown ORM that allows
> for almost any operation that you could think of in SQL. extbase does not
> have such an ORM.
> Generally speaking, a framwork should be able to handle whatever domain
> model you throw at it. In extbase, you often have to alter your domain model
> in order to be able to implement it at all. Bad!
> 
> I mentioned earlier that other ORMs grew over the years and sport a big deal
> of performance optimizations and other great ideas to make handling even
> huge amounts of data comfortable. These ORMs leverage the databases
> capabilities instead of imitating their behaviour. Unfortunately, in extbase
> this is not the case, making queries on large datasets incredibly slow.
> 
> extbase is not able to model complex queries that involve more than simple
> inner joins. Ever tried to build a tag cloud in extbase? No group-by
> available, no complex joins in the current release. Speaking of new
> releases: the latest and greatest extbase release, the one bundled with
> TYPO3 4.5.2, is so incompatible to last years release that it requires a
> major rewrite of our codebase! Hooray.
> 
> The main purpose of extbase was to prepare developers for the next major
> release of TYPO3: Version 5. Well, it really is not much more than some
> slight "preparation". Looking at the code I can not believe that anything
> can be easily transferred from extbase to v5. A complete  rewrite will be
> the cleaner and faster solution in most cases - want to bet?
> 
> Thanks to extbase, we are in trouble. We have a huge project that can not be
> easily upgraded to the latest version of TYPO3, it is slow, it has more
> workarounds than real code. In no way does it live up to our standards. All
> I have written above is still valid for the current release of extbase,
> while first perfomance tests even show a 30% decrease in perfomance. This
> mixture of design flaws and simple bugs make me feel very, very
> uncomfortable. And worst: our developers simply do not want to use extbase
> ever again.
> Our solution to this problem is simple: We will drop extbase and replace the
> core of our system with something stable, scalable and reliable.
> 
> You might ask why I write all this. It's because I like TYPO3, I have seen
> it grow up, I have met all you great guys who code it and heck - I really
> dig FLOW3! This is going to be awesome!
> And that is exactly the problem: We managed to make TYPO3 enterprise-ready
> (whatever that means), we have a great framework and an even greater CMS at
> the horizon... and we mess it all up by making some halfheartedly put
> together semi-framework that is neither capable of running enterprise-scale
> projects nor is it in any thinkable way future proof.
> 
> There are just two ways out of this:
> a) Fix extbase as soon as possible. Given the kind of weaknesses we found,
> this is not as easy as it sounds... and I am sure we only scratched the
> surface.
> b) Get real and make clear that using extbase is optional and experimental.
> I do not want to see others run into the same problems we ran into.
> 
> Regards,
> Dan


-- 
Oliver Hader
TYPO3 v4 Core Team Leader

TYPO3 .... inspiring people to share!
Get involved: http://typo3.org