[TYPO3-dev] The extbase dilemma

[Björn Steinbrink]

On 2011.05.18 14:26:14 +0200, Sebastian Kurfürst wrote:
> > The first thing we noticed: extbase does not scale. At all. And no, it's not
> > our coding style, it's just a framework that burns CPU, and Fluid puts oil
> > onto the fire.
> I'd be curious to see concrete use cases on where Fluid is slow. Best
> would be a simple controller which contains demo data; and then all
> required templates/layout/partials for the rendering.
> We have some ideas how Fluid can be compiled to native PHP code, and
> such real-world tests would be very helpful for us to benchmark Fluid.

I'll see whether I can come up with a stripped down example that uses
static data, to remove things outside of fluid.

> One more question with regards to Performance: Do you use the TYPO3
> caching mechanisms, or is this not possible in your application domain?

We've added even one more layer in between to cache partial content.
For example, we have templates that show teasers for multiple articles.
For those, we cache the HTML snippets for the individual teasers to be
able to avoid reloading that article and re-rendering that teaser, when
e.g. the position of that teaser on the page changes, but the article
itself is unchanged. Still, using e.g. frontend editing capabilities to
reposition articles, things get quite slow.

> > [Persistence]
> You're right that we built our own persistence layer, but the most
> prominent reason is that we wanted integration into TYPO3 practices;
> meaning we needed support for stuff like comma-separated lists in
> relations, and other weird things.

Sounds like a questionable choice to me. I'd dare say that it would have
made sense to me to primarily support sane data storage models and tell
people that they'll have to manually do the weird stuff, if they want to
keep doing that. Or provide some wrappers as some kind of helper for
backwards compatibility. But "encouraging" things like the
comma-separated list stuff? I dunno.

> Today, I am not able to say if a pre-made persistence layer would have
> bought us less problems -- but it could certainly be. I am with you that
> the Persistence Layer is maybe the most complex part of Extbase; and
> because of that, nobody dares to touch it (including me, I have to say)...
> 
> > What does this mean for extbase? First of all: no transactions, no locks,
> > and therefore - race conditions everywhere.
> You're right; and that's the same for TYPO3 v4 -- no use of transactions
> there as well (correct me if I'm wrong); so while I agree that
> transactions would be nice, we have lived in TYPO3 v4 for ages without them.

As Dmitry already pointed out, extbase takes control of the database
connection, while previously the developer had that control and was
able to implement transactions as needed.

> > In order to find out which data has to be persisted, extbase goes through
> > _all_ objects used in the current request, even if they were just used for
> > reading. It does so, because there is no dirty-flag.
> How does a dirty-flag help? IMHO it is not possible to get notified when
> a property of an object changes (in PHP); so one does not know if an
> object has changed. As soon as any method on an object has been called,
> we cannot know anymore if the internal state has changed.
> It'd be cool to get some notifications as soon as object properties
> changed, but apart from a custom PHP extension[1] I once wrote I think
> it is not possible with plain PHP -- although I'd love to be proven wrong ;)

Force users to set that flag. Generate base classes that force you to
use their get/set methods which set the flag. Or if you insist on having
everything implicit by means of annotations, go for something insane
like the example at the end of this mail (I guess I'll get some flak for
that kind of code tomorrow at work ;-)).

But making the common case expensive to save a bit of work for the
"rare" case seems just wrong.

> > This check is extremely
> > expensive and gets even more expensive if there are many extbase-plugins on
> > a page, because then _all_ objects are checked again for _every_ plugin!
> Good point, that's a bug which needs to be fixed. The Persistence Layer
> needs to be reset after each plugin run, or alternatively detect if it
> is run on the last plugin on the page. Could you open a bug report for
> this, or even provide a patch? I think that's pretty easy to solve (at
> least the first solution).

IIRC it's not that easy, or at least wasn't in T3 4.4, due to some
implementation details. I don't have any details at hand right now
though.  I'll see if I quickly dig in there tomorrow (err, today
*sigh*).  But what you face irrespective of how things are implemented,
is that you don't control the object lifetime. If some "smart" developer
gets the idea of caching the objects on his own to avoid reloading them,
simply resetting the persistence layer will get you some hard to debug
problems. Thus, at the very least, you'd then want to detect
"invalid"/outdated objects in some way, but right now, I have no idea as
to how you could do that.

> > access whatever database you want in any way you like. extbase isolates you
> > from the underlying database, which is ok for a full-blown ORM that allows
> > for almost any operation that you could think of in SQL.
> You can always directly write SQL inside Repositories if Extbase is slow
> or does not support a particular use-case. Extbase does not force you to
> use the persistence layer, and in some cases (like building a tag
> cloud), a handmade SQL statement will be the better choice I think.

The problem is that you then give up _everything_. You'll have to handle
the hidden flag, the deletion flag, pids, versioning, etc. all yourself.
That's ok if I have to do that anyway, but not if half of the time some
kind of ORM does it for me, and the other half I have to do it myself,
always having to be on the lookout to exactly match the behaviour of the
ORM to get consistent behaviour throughout my code.

Compare that to a system like e.g. Propel, where you can consistenly use
"building blocks" for all your query building tasks, so you write them
down once, and then simply re-use them. Sure, that system isn't without
flaws either but it's far from extbase's all-or-nothing.

> > TYPO3 4.5.2, is so incompatible to last years release that it requires a
> > major rewrite of our codebase! Hooray.
> I'd be curious on what changes are there; as we tried to document all
> breaking changes on [2]

Just because it's documented doesn't mean that it's not a PITA to deal
with it ;-)

> What other difficulties did you experience when using the publicly
> exposed API?

The QueryResult change is a major annoyance, since we use current() and
array_*() in quite a few places. The dispatcher change was a break for
me at some place (I don't recall where exactly), although it's said to
be only deprecated. The ConfigurationManager stuff isn't public API, but
was IIRC required at some point to handle some cross-extension
interaction (IIRC some singleton [naturally] got very confused). Some
classes got new methods, which we happened to also implement in derived
classes (e.g. getPid() on the domain objects IIRC). And "fixing" the
caching (USER vs. USER_INT for the plugins) will likely take some time
(ok, admittedly, that was just completely broken in extbase 1.2).

I don't recall that many details, as we tried to port it around the time
4.5 was released, and I simply gave up after like 2-3 hours of not
getting anywhere at all.

Björn


Insane dirtyness state tracking approach, extremely simplified:

<?php
class Base {
  private $persistables = array();

  private $isDirty = false;

  public function __construct() {
    // Fill $persistables from the annotations in the derived class
    $this->persistables = Whatever::getPersistables(get_class($this));
  }

  public function __get($name) {
    if (in_array($name, $this->persistables))
      $name = "_{$name}";

    return $this->$name;
  }

  public function __set($name, $val) {
    if (in_array($name, $this->persistables)) {
      $this->isDirty = true;
      $name = "_{$name}";
    }

    $this->$name = $val;
  }

  public function _isDirty() {
    return $this->isDirty;
  }
}

class Ext extends Base {
  /**
   * @var int
   */
  protected $_foo;

  /**
   * @var int
   */
  protected $_bar;

  protected $notPersistable;
}

class Whatever {
  static public function getPersistables($cls) {
    // Hardcoded for simplicity
    return array('foo', 'bar');
  }
}

$x = new Ext;
echo '$x is ', ($x->_isDirty() ? '' : 'not '), 'dirty', PHP_EOL;
$x->notPersistable = 10;
echo '$x is ', ($x->_isDirty() ? '' : 'not '), 'dirty', PHP_EOL;
$x->foo = 5;
echo '$x is ', ($x->_isDirty() ? '' : 'not '), 'dirty', PHP_EOL;
echo $x->foo, PHP_EOL;

-- 
*Björn Steinbrink*
*fon*: +49 (0) 521 - 329647-21
*fax*: +49 (0) 521 - 329647-40
*e-mail*: steinbrink at saltation.de
http://www.saltation.de
---------------

saltation GmbH & Co. KG | Niederwall 43 | 33602 Bielefeld

Sitz Bielefeld | Amtsgericht Bielefeld HRA 15344

Persönlich haftende Gesellschafterin:

saltation Beteiligungs-GmbH | Niederwall 43 | 33602 Bielefeld

Sitz Bielefeld | Amtsgericht Bielefeld HRB 39339
Geschäftsführer: Daniel Brün