[TYPO3-dev] [TYPO3-v4] Minutes from the 28th meeting of the release 4.5 team

[Ernesto Baschny [cron IT]]

Hi,

We had a meeting today, one of the last meetings before our final
release of 4.5.0.

Present were: Olly Hader (core team leader), Ben van't Ende (community
manager), Steffen Kamper (4.5 technical leader) and Ernesto Baschny (4.5
release manager).

Later we had a meeting with Steffen Kamper, Ernesto Baschny and Helmut
Hummel (Security Team), which is embedded in these minutes.

-----------------------------------------------------------------------
Pending issues
-----------------------------------------------------------------------

We discussed the pending issues briefly which are crucial for the final
release:

t3editor won't save
-----------------------------

Patch from Tobi in core list,
"17078":http://bugs.typo3.org/view.php?id=17078, needs further reviews.
[Post-Note: Resolved by now]

Upgrade Wizard "Workspaces"
-----------------------------

Workspaces Upgrade Wizard cannot be "skipped" (using the new "Next>",
"Next>" approach, which means everybody will get Workspaces installed
upon upgrade. Patch from Francois solves this by making this step
optional. Issue "17260":http://bugs.typo3.org/view.php?id=17260 pending.
Needs more reviews.

CSH in Extbase
-----------------------------

Issue reported by Georg
("17265":http://bugs.typo3.org/view.php?id=17265): CSH not working for
Flexform in Extbase extensions. Pending already in the core list, needs
more review.

New EM with DBAL
-----------------------------

New Extension manager's "Remote Repository" tab uses subselects which
are not DBAL compatible. Steffen will try to find a solution (which is
doing subselects only if DBAL is not activated, else do the calculation
in PHP afterwards).

Other issues
-----------------------------

Olly create a link pointing to all issues reported since RC1:
http://tinyurl.com/4ps9sms

He will do some triage to get a more comprehensive list. Also Ernesto
will work on organizing the open issues.

-----------------------------------------------------------------------
Performance issues
-----------------------------------------------------------------------

Steffen talked with Rupi Germann which did some performance checks on
our latest code and compared it with previous releases. We have added
some features improving the overall performance of the backend (removing
the FRAMESET) but on the other hand many features came in which require
more processing performance.

The main issue tested is the amount of MySQL queries, which increased
dramatically in 4.5. Mainly three issues were analyzed:

CSRF
-----------------------------

New "Form protection" adds several queries when tokens are generated
(and later deleted). E.g. the list view with the extended view open
creates several hundreds of queries. For each icon, one additional SQL
query is added.

See later on for the minutes of our post-meeting with Helmut, where we
discussed the issue.

Pagetree (initial load)
-----------------------------

As soon as the pagetree is loaded for the first time (backend.php reload
or hitting the "Refresh" button) the whole tree of pages is fetched
through individual queries. Up to 500 pages.

Steffen later talked to Stefan Galinski and this seems to be a
pre-loading of pages (up to 500) to speed up the user interface on
future clicks on the tree. Steffen will come up with a solution to have
this pre-loading configurable because its effectiveness depends on
various factors (it was defaulting to 500 pages).

Ernesto mentioned "nested sets" which would help speeding up loading
certain tree parts. Steffen knows this, but there was yet no time to
integrate such things. At least we now have a very stable API and a good
abstraction layer to be able to very easily add such features without
the fear of breaking some API => Optimization can start!

T3 Compressor
-----------------------------

Analysing with kCacheGrind seems to point to t3lib_compressor as a main
performance brake (10%). Apparently doing serialize / unserialize
operations. Ernesto mentioned that the t3_compressor doesn't do any
serialization (it "just" gets md5 hashes from files and compares...) so
this would require further analysis to get solved.

ExtDirect / Ajax calls
-----------------------------

Since we now have lots of more Ajax requests in our backend, we have to
keep in mind that every one of them goes through the whole init.php
process to get the correct environment to work with (BE User, etc). Even
a simply CSH request does that.

This would be in the scope of the "init.php cleanup
project":http://forge.typo3.org/projects/typo3v45-projects/wiki/Initphp_enhancements,
to get that cleaned up and optimized (in a similar way that we already
have for the Frontend).

Performance-Sprint after the release
-----------------------------------------

We will concentrate on major issues we still have left until the final
release which has a settled date already. We will then try to organize a
code sprint to tackle the most annoying performance issues after the
release one by one and with proper time for testing.

In the past we have avoided performance improving patches in released
versions (due to the risk of breaking something), but Ernesto considers
the first weeks after the release a good time to further improve
performance of the backend, as we will see it performing "in the field".
Of course every patch needs to be carefully reviewed and tested.

-----------------------------------------------------------------------
Cross Site Request Forgery (CSRF)
-----------------------------------------------------------------------

Later today we had a Skype meeting about CSRF in particular (Steffen
Kamper, Ernesto Baschny and Helmut Hummel from the Security Team).

We talked about the open issues which are basically:

* Performance (list module, too many queries)
* Race conditions (ExtDirect, multiple IFRAMEs)
* Other non-protected places

And also potential solutions to them.

Performance
-----------------------------

Currently "to be save" the code does a "persis token" after every new
generated token, because at first sight there is no central point in the
code where the persistence of such things could be done.

After some brainstorming the idea came to generate all tokens during one
execution in memory and persist in $this->doc->endPage() method, which
is basically the exit point for all modules using template.php. On
ExtDirect / Ajax calls, still persist on every call.

Race condition
-----------------------------

Race condition happen when two scripts (e.g. nav + content) inits BE
users almost on the same time but with different timings. Since tokens
are stored in be_users's session (which is a huge serialized array), the
problem pops up if one fetches the session data, the second one also,
then the first one writes it back later (with new tokens) and the second
one also (overwriting the first tokens).

Potential solutions are:
# locking shortly before token persistence: get session data, lock,
change it, save, unlock
# store tokens in PHP session: PHP does the locking (session_start(),
..) and also no further SQL is required for tokens.

For the simplicity of it and also the potential performance improvement,
we chose the PHP session path. For this to work reliably, Helmut will
re-use the tx_install_session class, which already allows more secure
PHP sessions in the Install Tool.

More protection
-----------------------------

There is already a pending patch to add CSRF protection to the
Web>Template module ("17189":http://bugs.typo3.org/view.php?id=17189,
needs testing and reviews).

Helmut wants to add CRSF also to file-upload (tce_file.php), because he
thinks its a crucial place. The patch is almost ready for core.

Working on it
-----------------------------

Helmut is currently working on these tasks and we will see the results
later on tonight, and hopefully be able to add those missing pieces to
the puzzle before RC2.

-----------------------------------------------------------------------
Release Candidate 2
-----------------------------------------------------------------------

We decided that we will be releasing a Release Candidate (RC2) tomorrow
morning between 10 AM - 11 AM (GMT+1). This gives the opportunity to
everybody to contribute in the "night hours" to solve the remaining issues.

Ernesto has informed all participant teams, so that they merge their
projects in time.

It has been mentioned that the fixed date of the final release should
have been postponed before. Ernesto thinks we all have to get used to
the rhythm of keeping the promised dates. We're proud to be able to
release at the date which we set up on the first day where we started
this journey, which has never happened on a release before. And even
with the lack of time for more intensive testing and some remaining
issues, we have overall a very stable and feature-rich version which we
will all be proud of.

This also means that our active development team needs to be aware that
we require more testing environments throughout the whole development
process and also a more early reviews and helping out teams by noticing
eventual performance bottlenecks and security issues at an earlier stage.

-----------------------------------------------------------------------
Documenting new features
-----------------------------------------------------------------------

Ernesto mentioned that he wants to see more Buzz articles on new
features so that we could link to them in the release notes. Suggestion
is to use a common prefix on these articles: "TYPO3 4.5 LTS: <topic>".
If the information requires lots of code snippets or screenshots, this
could be added to the Wiki and linked in the Buzz article. So the buzz
articles should be the "perma-URL" to these new feature descriptions,
which then link to the relevant technical documentations.

He already talked to Xavier about an article about "Prepared
statements", and the "article is already
available":http://buzz.typo3.org/teams/core/article/typo3-45-lts-prepared-queries-are-the-way-to-go/.

Ernesto also contacted Francois on his opinion on this approach.

-----------------------------------------------------------------------
Further steps
-----------------------------------------------------------------------

* We will release RC2 tomorrow morning.
* We will release 4.5.0 on Wednesday throughout the day.
* More tiny meetings will happen in between as we think is necessary.
* We will make a pause after the release to renew the energies (Steffen
Kamper mentioned the need of "90h of nonstop sleep" => he deserves it!)


Cheers,
Ernesto