[TYPO3-caretaker] RFC: caretaker_instance mass installation

[Christoph Moeller]

Hi guys,

first of all: a massive "thank you" to you all for commiting so much 
work in this wonderful set of extensions! We are just starting to really 
make wider use of the caretaker and are already saving quite some effort 
by doing so.

To summarize what I've just talked about with Tobias on the phone, let 
me introduce what we are currently planning to do with caretaker:

We are running tens of our own server systems and we're providing 
support for several customer-run TYPO3 systems, each hosting 1 to 50 
TYPO3 instances. These systems are already monitored by nagios, but 
rather on a "per system, low-level" base.

We are running the caretaker master on our internally hosted nagios 
system, which is not accessible from the internet. This system can of 
course do outgoing connects to the TYPO3 systems that we want to monitor 
with caretaker.

So now, after having completed some really promising manual tests 
(including manually installing EXT:caretaker_instance in some selected 
sites and setting up/exchanging the SSL keys, creating the instance 
records in the caretaker server, etc.), we'd like to do some 
mass-rollout. We'd like to have a means of "integrating all TYPO3 
instances on this system" - without any manual interaction, since we're 
talking about hundreds of TYPO3 instances.

What we've accomplished, so far, is doing the part of mass installation 
in the instances:
* a shell script to find all TYPO3 instances in the current working 
directory [runs remotely]

* a shell script to loop through all TYPO3 instances, which fetches 
EXT:caretaker_instance from SVN, installs it, generates all needed 
settings (SSL keypair, IP whitelists), clears caches [runs remotely]


What's still missing is the instance record on the master. After setting 
up these instance records in the master system and applying some tests, 
the communication instantly works and we're receiving data.

Now I'd like to discuss how to get this final manual step automated.

Idea #1:
--------
* write a webservice extension for the caretaker master and expose it to 
the internet/the instances
* do some fancy security checking
* accept "integrate me" calls from the instances
* create the instance records and apply tests, as centrally defined

Pros:
* clean and fully automatic
* could be used by other system integrations

Cons:
* another vector of attack to the internal network, pretty bad security 
concerns
* over-complex at the moment
* much more effort


Idea #2:
--------
* have the shell script (the one that remotely installs 
EXT:caretaker_instance) output a list of instances to integrate in a 
defined format (e.g. CSV/XML/SQL)

* extend the caretaker master extension with a new mode "Mass-Import 
instances"

* upload the generated instance list file, run the import and select 
what tests should be applied, what instance groups the newly created 
systems should be grouped in, etc.

* have EXT:caretaker create all needed records/relations

Pros:
* easy to accomplish, small effort needed
* does all we need a.t.m.
* more secure, cannot be triggered from "the outside"

Cons:
* semi-automatic instead of fully automatic (that could of course be 
implemented later, using a CLI-Mode script)


We strongly prefer Idea #2 at the moment due to less complexity and 
better security.

What do you think? Have you implemented s.th. like that, already?

Let's share some thoughts and code.

Best wishes from Cologne,
Chris