[TYPO3-dev] The extbase dilemma

[Mathias Schreiber [wmdb]]

Am 20.05.11 11:20, schrieb Peter Niederlag:
> Hello,
>
> Am 19.05.2011 19:55, schrieb Dmitry Dulepov:
>> Hi!
> [...]
>> The first good thing to do now would be to get some numbers. Two people
>> claimed they make Extbase solutions with huge data. It would be good to
>> know:
>
> don't know if you wre refering to me but anyway I think I already
> claimed that.... ;)
>
>> - how huge is it?
>
> We still lack a true key/description on what we refer to by "huge". ;)

I know, I tried to get real data from everybody who stated "huge" here 
but you are te first one to actually answer it :)
BIG thanks for this.

> Table has > 1000.000 rows with > 15 columns(in TCA and Model), of course
> read/insert/update always affect one ore a handful of rows, so that
> probably keeps it from being considered "huge". ;)
> export/import is done with plain sql.

Since the lazy-load thing got in I don't think the pure number of rows 
is the bottleneck.
 From my experience problems come up when there are nested relations.
1-n is handled pretty well, n-n gets a bit slower.
This is where the method you mentioned below gets its problems.
If we stick to the blog example:
100 blogs with 100 posts each and 100 comments per post.
You want to display the latest comment per blog.
IIRC this would result in at least 100 queries, right?
Or I fiddle around with custom queries which then in return does not let 
me use persistence or does it?

Generally speaking it seems to get worse the more nested relations there 
are.

I for myself almost always have one of the two extremes.
Either have a shallow model with like 2-3 related tables or have 40+ 
table monsters. :/

> I don't really see a use case where the persistance layout of extbase
> must handle more than say 100 rows in total. I'd assume that's probably
> where problems could start?

Movies <-> Actors
or
TV Shows <-> air dates
are valid examples.

> Even doctrine has a special chapter on "mass updates" because an ORM
> always causes extra overload and IMO for updating masses of data one has
> to make special cautions. Not sure about "mass reading" and "mass
> insert" as for efficiency I always head for a small footprint (sql) there.
>
>> - are there relations between data types and what kind of relations?
>
> IIRC mostly manually managed relations[1] as this had been done > 1 year
> ago where I wasn't brave enough to use them all the way and also have
> some special relations using other keys than uid due to imported data.

This is very good feedback.
I tried to think of a large dataset which was not imported from 
somewhere else and only came up with an online shop that runs on TYPO3 
for more than 6 years now :)
So maybe one could bring up the assumption that the larger datasets are 
mostly imported from somewhere else?


cheers
Mathias


-- 
Ernesto, Nov. 9th 2010:
"In the graphics generation routines of TYPO3 *anything* could cause a
side effect."