[TYPO3-dev] The extbase dilemma

[Michael Knoll]

Hi Daniel et al,

first of all, thanks for sharing your thoughts and taking the time of 
writing down all this stuff. As I was probably one of the guys to "push" 
the usage of Extbase, I wanted to give some feedback, too.


1) Concerning Documentation:

Since the beginning of the project, I was wondering why there is a) 
almost no documentation available and b) the documentation that was 
available is only given in a format that least people can work with. 
After converting the Docbook stuff to readable PDF and uploading on my 
blog, the number of visits on my blog kind of exploded - so there was a 
real need for people to get this stuff. Up to this point in time, there 
is still no valid and useful documentation, the mailinglist explodes and 
thanks to some very very helpful people there, questions that raise up 
the 100th time still get answered. Most of the time, some "critical" 
question araises, I find myself stepping through the code for hours, 
trying to find out what happens and where the problem lies.

That is not at all what I think of, stat-of-the-art extension 
development should look like and covering the whole T3 project, it's the 
thing, why most people are not willing to use T3.

AFAIK there is no other platform for documentation on this project than 
forge-wiki and forge-wiki is not the platform for documenting such a 
project. Compared to other projects like Symfony or CakePHP a 
project-site is simply missing.


2) Concerning FLUID

As I did not have the time to take a look at FLOW3 until now so I don't 
know what's the state of FLUID there and whether FLUID is the same in 
Extbase and FLOW3. But what I do know from several Extbase projects is, 
that FLUID has some severe problems - even in the parsing layer - and 
they simply don't seem to get fixed. Under the cover, a bunch of HUGE 
regexes are in charge of doing the parsing job. Whereas other template 
enginges like Smarty use a table-driven parser generated by a grammar 
that seems to be much more maintainable to me. The concept of 
viewhelpers is great, the implementation lacks a real parser. I don't 
know, whether there is some kind of PHP-code generated that could / or 
should be cached to make it faster. If not, that should be done. If it 
should come to a rewrite of the parser, I'm willing to help out!


3) Concerning persistence layer

About 3 years ago, I started using Propel for my T3 projects. Later I 
used Doctrine for one of my projects and it worked perfectly besides a 
memory error when trying to import 30.000 records "at once" which I'm 
sure could be fixed. The reason of starting to write a persistence layer 
from the scratch due to be downward compatible with what seems to me 
like the biggest design flaws in T3 is a really sad reason for what we 
have today in Extbase. Even if some of the general tasks are covered by 
the Extbase persistence layer (pids, hidden, deleted,...) it's real PITA 
(new word and I love it!) if you "don't want to have this" e.g. when 
writing backend modules for handling hidden or deleted records. In our 
last project, we started using our own hidden flag, as we were not able 
to handle this problem with standard Extbase functionality. Even if it 
would be possible to do - there simply is no documentation on how to do 
it anywhere.

So I really don't know, why we shouldn't use a well-documented ORM like 
Doctrine for this stuff and concentrate on things that really have to be 
coded from the scratch to fit T3 needs. If I'm right, FLOW3 has just 
integrated Doctrine as ORM - so why don't we "backport" this?


4) Concerning the "Blog Example"

Among some of my friends, we are always laughing about the 
"Blog-or-wiki-out-of-the-box-frameworks", as there has been a time when 
the quality and simplicity of a framework had to be shown by writing a 
blog or a wiki with as few lines of code as possible. So I always 
wondered when there is the one and only Blog-framework that comes up with:

$blog = BlogFramework::getInstance();
$blog->render();

or something like that... What I want to say is, that using something 
"simple" like such an example doesn't provide reliable information about 
the problems the raised during Daniel's project or during some of our 
projects. Wanting to be a "Enterprise CMS", we should not rely on a 
framework that is merely capable of rendering a simple web blog. When 
trying to bring Extbase on a next level, serious tests with serious work 
load, race condition scenarios and more complex domain logic should be 
applied to prove those things.

A second problem is the fact, that blog-example is used as some kind of 
reference-extension for documentation. But shouldn't there be a more 
complex extension so show up real-world problems?


5) Concerning community

"Inspiring people to share" is the credo of T3. And I like it - really. 
Almost every line of code I wrote so far went to TER, Forge or github 
and is public. I tried to raise a discussion about the code I wrote on 
mailinglists and write about it on my blog - to help other people cope 
the same problems I had before.

Sadly though - this is not the case for most of the other programmers 
out there. Tons of code is written, thousands of questions are asked, 
hundreds of answers are read - but nothing comes back. That is not a 
community sharing their thoughts and ideas and helping each other on 
bringing this thing to a next level. I really miss the spirit of working 
together and improving the infrastructure, everyone is using. I know, 
that people are not paid for working on Extbase - so it's completely 
wrong, that a hand full of people are working on it and improving it, 
while thousands of people use it and do not contribute at all. That's 
not how Open-Source software and community-based development can work!


6) Concerning problems and suggestions

During the last year, we patched or let's say pimped several issues with 
Extbase and wrote about that in the mailing list as RFCs. Sadly none of 
the were integrated or it wasn't even discussed about it. Most of the 
problems concerned writing extensions in Extbase that can later be 
extended by 3rd party extensions, where convention-over-configuration 
eventually won't do the job and we needed some TS-configuration. Or 
Ajax-dependent problems as there still isn't an out-of-the-box way to 
handle Ajax requests. Most of those posts were ignored and however asked 
again some days later. When Gerrit was introduced, I was already tired 
about proposing things, as they never got incorporated. I will give it 
another try!



By writing this email, I don't want to offend anybody, at least not the 
people like Jochen, Sebastian and the other Extbase guys who are 
spending their leisure time to work on this project. But I think, after 
the experience made by different people, it's now time to reconsider 
decisions and working on a plan for the future. I'd be happy to 
contribute to a session on T3DD or one of the T3CONs. It would be a 
great thing to get this framework on a next level!


Hope this helps to get an idea of where the problems lie. As I said, I'm 
willing to give you a helping hand!


Greetings

Mimi






Am 18.05.11 11:19, schrieb Daniel Brün:
> Hi folks,
>
> writing this post is feeling awkward to me, but I have to write it
> nevertheless.
>
> I have been around the TYPO3-universe for quite some time now and my company
> has implemented many TYPO3-projects and many, many extensions (most of them
> not in TER).
> Last year we started developing salt_news, a powerful news-system for TYPO3
> based on extJS and extbase that I talked about at T3CON10.
>
> Today I consider the decision to develop the system using extbase a major
> mistake on our side.
>
> Let me tell you why:
>
> extbase is a backport of a framework (FLOW3) that is not production-ready
> itself now, and certainly was far from it by the time the first version of
> extbase surfaced. Strange enough, many extbase-based extensions are already
> used in production context, while there are no real FLOW3-projects live.
>
> During the last years, I have on many occasions discussed FLOW3 and its
> architecture with the FLOW3-team and I still believe this is going to be one
> of the most sophisticated PHP-frameworks around - once it is finished. Maybe
> this influenced our decision to use the backport for a bigger project than
> the average edit-this-in-backend-and-show-that-in-frontend extension.
> When we started developing salt_news we had to decide which framework and
> technology to use and only after reading rave reviews and success stories
> about extbase we decided to build our system upon it - dropping symfony and
> scala/lift from our list. Having intensively used all kinds of other
> languages and frameworks we were eager to see the future of TYPO3.
>
> The first thing we noticed: extbase does not scale. At all. And no, it's not
> our coding style, it's just a framework that burns CPU, and Fluid puts oil
> onto the fire. OK, we thought, just setup a cluster of apaches and mysqls,
> make use of nginx and varnish for balancing and caching and everything will
> be fine. Turns out that the few requests reaching the apaches still manage
> to bog the servers down. After implementing more caching and bypassing the
> persistance layer we could rise the performance to an acceptable level -
> thinking about the hardware involved for this project makes me sick though.
>
> The quest for performance followed us through the entire project and I won't
> go into details regarding performance here.
>
> But there are other, way more important things going wrong in extbase.
> Instead of making use of an existing ORM (like propel or doctrine) the
> extbase-team decided to develop something ORM-like by themselves - as a
> byproduct. Needless to say, doctrine and propel have been under development
> for a couple of years to reach their current speed and stability.
> What does this mean for extbase? First of all: no transactions, no locks,
> and therefore - race conditions everywhere. Don't believe this? Setup a
> simple domain model for authors and books. An author may have written many
> books, and books may have multiple authors, requiring a many-to-many
> relationship between books and authors. Now add a book to the list of books
> by an author and persist the changes. Guess what happens? Right: First of
> all, the list of associated books is deleted in the database. Then a new
> list (including the new book) is written to the database. Any queries
> inbetween deliver a wrong result, or no result at all. The great thing about
> this: you can not tell whether the result is correct or not. Your cache will
> save the result, so will your varnish. Of course, this will only happen now
> and then. Great for debugging!
> This kind of race condition is found in many different places all around the
> framework and it is a major task for our developers to code around them.
> Another example: Create a new object. When persisting this object a new row
> in the database is created with default values. It is only after this
> operation that the real values are written using UPDATE. Bad luck for any
> query that is executed in between. *shakehead*
> Did I mention no transactions? Basically every save-operation that touches
> more than a single row in the database is racy.
> The more traffic there is on your site, the more likely is the appearance of
> race-conditions. Think of any serious business-application, bookings,
> calculated sums, ...
>
> In addition to the lack of transactions, extbase does not support database
> constraints, either. This means, that all relations between objects and
> their data have to be taken care of in PHP, instead of the database.
> Obviously, this can not work out given the size of a typical race window.
>   Whenever writing to the database is unsuccesful, your application is in an
> undefined state and in most cases, extbase simply crashes. Due to the lack
> of transactions the database is then left in the intermediate state is was
> in when extbase crashed... no rollback possible, your database is now
> inconsistent.
>
> In order to find out which data has to be persisted, extbase goes through
> _all_ objects used in the current request, even if they were just used for
> reading. It does so, because there is no dirty-flag. This check is extremely
> expensive and gets even more expensive if there are many extbase-plugins on
> a page, because then _all_ objects are checked again for _every_ plugin! 10
> plugins on a page with 100 objects each = 10 times (10 plugins * 100
> objects) = 10.000 objects checked every time you visit that page in your
> browser. No, I am not kidding, _every_ object registered for any plugin on a
> page is checked for every plugin.
>
> The good feeling you get from domain driven design is heavily hampered by
> the weak spot in extbase: the persistance layer. In some circumstances,
> touching an object the wrong way initiates a persist-cascade, that not only
> takes ages to complete but also has some other nifty side-effects. MM-tables
> can not have primary keys? Aha. No composite keys possible, just one single
> UID? Hmm. No additional attributes for M:N-relations supported? Grmmml.
> Of course, some of these limitations are inherited from TYPO3. The
> difference being that in a classic TYPO3-extension you have all means to
> access whatever database you want in any way you like. extbase isolates you
> from the underlying database, which is ok for a full-blown ORM that allows
> for almost any operation that you could think of in SQL. extbase does not
> have such an ORM.
> Generally speaking, a framwork should be able to handle whatever domain
> model you throw at it. In extbase, you often have to alter your domain model
> in order to be able to implement it at all. Bad!
>
> I mentioned earlier that other ORMs grew over the years and sport a big deal
> of performance optimizations and other great ideas to make handling even
> huge amounts of data comfortable. These ORMs leverage the databases
> capabilities instead of imitating their behaviour. Unfortunately, in extbase
> this is not the case, making queries on large datasets incredibly slow.
>
> extbase is not able to model complex queries that involve more than simple
> inner joins. Ever tried to build a tag cloud in extbase? No group-by
> available, no complex joins in the current release. Speaking of new
> releases: the latest and greatest extbase release, the one bundled with
> TYPO3 4.5.2, is so incompatible to last years release that it requires a
> major rewrite of our codebase! Hooray.
>
> The main purpose of extbase was to prepare developers for the next major
> release of TYPO3: Version 5. Well, it really is not much more than some
> slight "preparation". Looking at the code I can not believe that anything
> can be easily transferred from extbase to v5. A complete  rewrite will be
> the cleaner and faster solution in most cases - want to bet?
>
> Thanks to extbase, we are in trouble. We have a huge project that can not be
> easily upgraded to the latest version of TYPO3, it is slow, it has more
> workarounds than real code. In no way does it live up to our standards. All
> I have written above is still valid for the current release of extbase,
> while first perfomance tests even show a 30% decrease in perfomance. This
> mixture of design flaws and simple bugs make me feel very, very
> uncomfortable. And worst: our developers simply do not want to use extbase
> ever again.
> Our solution to this problem is simple: We will drop extbase and replace the
> core of our system with something stable, scalable and reliable.
>
> You might ask why I write all this. It's because I like TYPO3, I have seen
> it grow up, I have met all you great guys who code it and heck - I really
> dig FLOW3! This is going to be awesome!
> And that is exactly the problem: We managed to make TYPO3 enterprise-ready
> (whatever that means), we have a great framework and an even greater CMS at
> the horizon... and we mess it all up by making some halfheartedly put
> together semi-framework that is neither capable of running enterprise-scale
> projects nor is it in any thinkable way future proof.
>
> There are just two ways out of this:
> a) Fix extbase as soon as possible. Given the kind of weaknesses we found,
> this is not as easy as it sounds... and I am sure we only scratched the
> surface.
> b) Get real and make clear that using extbase is optional and experimental.
> I do not want to see others run into the same problems we ran into.
>
> Regards,
> Dan