[TYPO3-ect] Extension Rating System and Reviews Status

[Patrick Rodacker]

Hello Michael.

Michael Scharkow wrote on 22.07.2006 11:59:

> First off, I think your approach is too complex. As Elmar pointed out,
> there are severe motivational and cognitive difficulties to face if you
> want to make QA of extensions reliable.

There are indeed some difficulties to overcome, but I had a long term
concept in mind when writing down the concept. Some of the tasks are
performed already, others are hopefully starting soon and others have
still to be defined. I don't see a problem with that, because the
results from active task could easily be integrated in the definition
process of the other inactive tasks.


> Ratings are simple and fast but very rough and may be completely useless
> in individual cases. That's why we need a large number of them. Reviews
> on the other hand need a lot of effort and motivation, and they may
> still be unreliable in some cases unless you have certified reviewers.
> While with ratings the law of large numbers will make sure that they are
> reliable (remember that each rating/review is per extension-*version*),
> you need either ratings of reviews or certification of some sort for
> detailed reviews. And even this may help against unreliability, but not
> against reviews becoming old/obsolete really fast.

I can agree to most of the points and just want to add a note about
reviews getting old or obsolete. I don't think that the main
functionality of an extension differs to much from one version to
another. Even in the case of major changes reviews of an older version
will still provide very useful information about the extension. Updates
after major changes could be added as review updates with minimal effort.


> After having witnessed the failure of *all* reviewing efforts I strongly
> believe that detailed reviews (both security and other) are bound to
> fail because of the huge and growing number of extension-versions. If
> you cannot review all extensions, you need not review any because you'll
> most likely choose those that are already popular/well-rated, so
> detailed reviews don't add much information, plus they become obsolete
> very quickly.

If the task is to go for all or any I still say, "ok let's go for all".
Many people think it is impossible to do at all, and of course I can
understand them. Of course we have to limit the number of almost 1800
extensions. A start could be to review only extensions which have the
status stable or beta. And you don't have to start a new review on every
version of an extension. Review updates are enough imho at that point.
We could also plan different review cycles and start the first cycle
concentrating on frontend plugins e.g. I think there are a lot of
possibilities to consider and we should not give up before we started at
all, though I can understand your position due to the negative
experience you made in the past with other review efforts.



> On another note, I'd very much prefer not mixing any official typo3.org
> stuff with the T3N magazine. Not because I don't like the magazine, but
> because it excludes all non-German speakers from those reviews. On the
> other hand, I like Jochen Weilands article comparing galleries very
> much, and I'd appreciate it even more if it was posted in English on
> typo3.org, but I don't think there's a need to institutionalize these
> comparisons too much.

This is not about institutionalizing extension comparison, but about
activities from within the community to give users and administrator
some guide through the extension jungle. I think that ext comparisons
focused on different areas are a very good start. I appreciate the work
from Jochen Weiland and the approaches of the TUG Hamburg to set up some
general methods for comparing extensions. You are right about the T3N
magazine and excluding all non-German speakers. Of course we need those
reviews and comparisons in the default language to integrate them into
the concept or make use of them in an official way. I don't think this
is much of a problem. I have different solutions in mind.

I want to integrate this work, because I see there are motivated people
within the community doing an excellent job. Letting the ideas and
efforts be in vain would be a shame.


> Conclusion: Wait for the simple ratings (they have a free text field)
> and see how much they help structuring TER2 already. Don't invest time
> and work in detailed reviews because they'll most likely fail - there
> are always too many extension versions with too few reviewers whose
> motivation will decrease as they realize that last week's 3-page-review
> is obsoleted by a great new version.

Well, I only agree on the first point. Hopefully a lot of users will
make use of the rating system. I am very curious about the results.
Another task "easy" to perfom are the feature specifications, which
could have a huge impact and will definitely aid to the transparency of
the extension features. No additional man-power needed, the developer
will choose the supported features and the user can easily look them up
in the TER. I will provide some more detailed explanation of my thoughts
on this task within the next week, but I hope you get my point. And I
think you have to let go on the point of too many extension versions,
because imho not every version needs a new review. I know that the
rating is bound to the version, but I don't think this is necessary in
case of reviews.

Regards
Patrick