[Typo3-typo3org] Hosting the TYPO3 sites

[Jan-Hendrik Heuing [NF]]

Hello!


As Robert already wrote, we had a chat together on Saturday morning. Here is 
a summary for all the others who did not participate ;)




Some details about history:




--> MySQL-Cluster:
Roughly a little more than a year ago, netfielders did a lot of experiments 
with clustering MySQL. Actualy we already fixed a deal with EMIC Networks 
about licenses for a MySQL-Cluster.
We had successfully set up a MySQL cluster which in the end did not work 
because of some strange incapabilities of that product in relation with some 
TYPO3 operations.

Although there could have been a chance to fix this on our side by modifying 
TYPO3, our requests have not been taken that serious that time.

--> Analysing .org sept. 04
After the forecast when doing those cluster-experiments, we finally figured 
out that the only way would be to change typo3.org. As there was still no 
changes in pipeline from Kaspers' side, we had a look at typo3.org in 
detail. We  figured out a couple of problems including that TER saves it's 
extensions in a database table etc.


At this point we reported this to DH, Rob and Kasper. Unfortunately we got a 
few comments feedback but no active help which helped us to solve those 
problems. Maybe it was because of a lack of time, but anyway - we really 
tried to solve the situation but have been slowed down by missing help from 
the core devs.


Actualy we offered one of our employees for some limited time (a couple of 
weeks if I remember right) fulltime to fix a few things, even extracting 
ext-files from the table, doing that in the filesystem would help'ed a lot! 
Well, after asking a few times, there was no feedback on this!!




--> Setting up Proxy
Last December, after nothing really happened, we started to enable a proxy 
server. We had it running perfectly, but it didn't cache any pages, just 
images. Why ? Because headers are not send, which are needed for caching. So 
we figured out that it does not only cache in the proxy, but also not at the 
client-level. That means, that EVERY request a user sends to typo3.org is 
seriously send, instead of caching that page.




After some discussion with us and some other people, Kasper finally became 
aware of the cache-control headers which are now implemented in TYPO3 3.8.0.




Result of this was that since December we want to update. After asking every 
once in a while, implementation has been finished. Asking Kasper for a 
timetable (while already discussing 3.8.0), I got the only feedback "no 
timetable".

This time, Rob already was in charge of typo3.org, but because of missing 
communication, we didn't know about that and kept updating other people 
until Rob complained....




--> SSH access (see below, one of Rob's concerns) + some infrastructure 
details:
Great, that there was no communication between DH, Kasper and Rob, as I did 
tell DH and Kasper a few times about the status what we wanted to do:

Right now typo3.org is hosted on webservers, db-server, file-server, 
loadbalancer. There is no access to the webserver at the moment, because 
than there would also be access to the fileserver straight away. There was 
no plan to open up the webserver via ssh for anyone beside us, only this 
strategy gave as the security we wanted. To access the webservers, you'd go 
through the loadbalancer, which you then would have full access to as well.


So, the idea was to also move those webservers to the virtuozzo platform, 
which we were already running on other servers. Virtuozzo separates between 
the real hardware node and virtual servers. So you can use i.e. redhat or 
debian within the virtual server, while you can not break the system on the 
node level (and other virtual servers). Virtuozzo is something similiar to 
the Linux vServer project. It separates the hardware node from infividual 
implementations within virtual servers.


This way Rob could have full root-access to a virtual server which makes use 
of the whole server performance, as there would be no other virtual server 
running. (btw: www.sw-soft.com for details). In case of people making things 
not work (apache, whatever), we could always just shut down that virtual 
server, start a new one (which could be pre configured) and here we go. 
Files and DB would still be external. You do not need to access those via 
SSH, you can always access those via remote (nfs and mysql).




--> new Platform beside typo3.org:
Beside what I just wrote, here some details about the rest of it: typo3.com 
and subsites like gov are already running on virtual servers like that on 
separate hardware. This is after splitting up typo3.org and .com soon ago. 
So Rob has got access to a control center to add new virtual servers, 
applications etc. with a few clicks, beside having access to the virtual 
server in any way he wants (including root-ssh). Lists.netfielders.de are 
also running on of those, but using debian as virtual server. BTW: Thanks 
Mr. Stucki for the huge amount of time you spend on the listserver though 
the last years. The list- and news-server is also running on a virtual 
server which is debian based. Through the last year, Michael had full root 
ssh access all the time!


Statistics: We added those sites to a urchin-log-analyzer. We will add those 
individual sites like edu.typo3.com as well. Beside that we'd install UTM an 
all those pages, so that we get a detailed analysis of screensizes, better 
understanding of clickpath etc.




--> Cooperation in general, and hosting of .org:
After all those things we did, after more than a year without cooperation in 
anyway. It's like setting up the stats and sending access and links to it 4 
or 5 times, what is the reason for this?  Of course people were busy, people 
set priorities to other things, but still.
If there is no cooperation like it has been for a quite long time, then I am 
not interested to get on with hosting typo3.org. However it seems as if Rob 
would take the chance to change this to something good. In this case I'd do 
anything to keep this going on our servers. The debugging session already 
was very productive and efficient. Thanks Rob.
BTW: We never advertised with it or anything, it has always been dedication 
to TYPO3.

It's not a matter of the hosting platform, but rather depends on 
architecture, which you all know now, is a huge problem. Just some figures: 
typo3.org produces roughly 3-4 times the traffic than .com. While typo3.org 
would need like 4 webservers, 3-4 database servers (as cluster) to run 
alright also in peak times, typo3.com is hosted on a single server which is 
not loaded at all.



--> future:
next steps as internaly communicate (=not on the lists), this is what we 
(Rob and I) will do:
- finding cache BUG in 3.8.0
- update to 3.8.0 after finding the bug
- enable proxy, at the next possible weekend after updating
- running for about 2-3 weeks to make sure proxy is stable, time to tweak 
some details....
- some time after running the proxy, after we make sure there are no 
problems, we will reinstall the webservers to make use of the virtuozzo 
platform, so that Rob has got real access for the next updates or debugging, 
which will hopefully not be needed very soon.
- Beside proxy-teaking, installation of webservers, we will set up new 
monitoring on a separate server (US) on a separate virtual server, parallel 
to one in the same infrastructure. We should think about the status page in 
this case. Right now we use Mesh-monitoring, which does not check for 
correct results, just for http.

Details of this will have to be fixed, one thing depends on the other in 
some cases, but the next steps are already fixed.



This is a _rough_ overview, not going into details, and not writing about 
all those details in between...






Thanks for listening,
Jan-Hendrik






PS: If people feel that I do not like them, this is mostly not the case, it's 
more that I do not like things which has happened through the last month and 
years.