[TYPO3-core] TYPO3 CMS Deployment with code and database

[Loek Hilgersom]

Hi Simon,

On 18-03-15 23:22, Simon Schaufelberger (Schaufi) wrote:
> Hi Loek,
> your scripts look quite complicated to understand and I have to dig a bit deeper
> to understand what exactly is going on.

Eh, yes. They 'evolved' so to say, in fact, they have been even more complex ;-)
And there are some features in there that are no longer used, or that seemed 
useful but were never really used in the end. As it goes with this sort of 
project: only when you are ready you know how it should have been done.

The changes I'm currently thinking of are these:
1. simplify (get rid of unused code and features)
2. restructure (make a library of functions, there is quite some duplicate code now)
3. get rid of Ant (it has got a few nice features like templating, but some 
basic tasks are near impossible, I prefer to use just one thing and as long as 
its structured and documented, that 'thing' could just be Bash.)

> So far I understood that you update the DB based on TCA, is that right?

Correct. It uses the extension t3deploy [1] to do that. It adds new tables, 
fields and changes, but never automatically removes any.

> How do
> you handle content changes? I didn't find a place where you handle that.

That's done in apply-deltas.sh. It checks the version no. in .data-version on 
the target system, applies all deltas higher than the current version (and 
increases the version number).

Deltas consist of a directory with a 3-digit version no. for a name (eg. 1.2.1). 
The directory contains an SQL file updates.sql with (mostly update) queries, but 
can also contain a directory tree with new files to be copied into the existing 
docroot, and - though seldom used - it can hold a script file updates.sh which 
will be executed when the deltas are applied.

The apply-deltas.sh first runs the TCA-update, and then all the deltas that the 
target system does not have yet.

It works quite ok, except when you have multiple development tracks working 
parallel you have to pay attention: when one team adds delta 1.5.0 but the 
change is not merged yet when the other team deploys 1.6.0, then the changes of 
1.5.0 will not be deployed automatically afterwards. O.t.o.h., for a testing 
environment you anyway want to use the built-in feature to automatically restore 
a backup before you apply your new changes, so that you always test on a clean, 
known state. And for a production deployment I suppose you'll pay some attention 
anyway ;-)

But it would be more perfect to use something which keeps track of exactly which 
delta's have been applied on a certain system, like doctrine does.


> I thought about a hook that will be executed on every db change (insert, update,
> delete) in the TYPO3 backend which will eventually write the sql changes to a
> file which then can be deployed. That would be the cleanest way imho.

I have also been thinking about that, using tcemain to apply db updates. 
Theoretically the way to go, but I wouldn't be surprised if it would create some 
unexpected obstacles. And I guess will require a different syntax for the 
updates. Now I just write SQL: the bits that I added manually on my dev machine 
I export using PhpMyAdmin and just copy parts of the output into the updates.sql.

The Core API extension [2] is something to keep an eye on, although the DataApi 
has been under the heading 'planned/comming soon' for a while.


>> I'd be happy to be involved in further
>> discussions/developments on this sort of thing.
>
> yes, let's keep the ball rolling together! ;)
>

Great!

Loek


[1] https://github.com/AOEmedia/t3deploy
[2] https://github.com/TYPO3-coreapi/ext-coreapi


> Am 18.03.15 um 14:47 schrieb Loek Hilgersom:
>> Hi Simon,
>>
>> You can find my deployment scripts here, maybe you get a few ideas from
>> that:
>> https://github.com/netcoop/typo3-installer
>>
>> I have presented this at T3dd12 and the scripts have been updated and
>> improved since, although I'd do it slightly different if I would do it
>> again.
>>
>> Average deployment takes me between 15 and 30 seconds, of which there is
>> hardly any real downtime. I don't remove tables to import them again,
>> just using deltas (mostly update queries).
>>
>> Hope it helps you somehow, I'd be happy to be involved in further
>> discussions/developments on this sort of thing.
>>
>> Loek
>>
>>
>> On 18-03-15 11:28, Simon Schaufelberger (Schaufi) wrote:
>>> Hello,
>>> I have done a lot of research about deploying a TYPO3 cms website for
>>> many
>>> months already but when it comes to deploying TYPO3 cms, I wish there
>>> would be
>>> more "Inspiring people to share" ;) I hope I can inspire people to
>>> share a bit
>>> now! :D
>>>
>>> I want to have a continuous deployment with files and database but
>>> without
>>> deploying "everything" for every release since that's quite a lot of
>>> data that
>>> needs to be copied.
>>>
>>> For the files to sync I use git but the database is really difficult
>>> since the
>>> database is changed (only) on the dev machine and then exported (with
>>> mysqldump)
>>> and imported on live again. The big problem here is the few seconds of
>>> downtime
>>> during deployment when the tables are deleted and imported again. I
>>> would like
>>> to have a system without any "broken state".
>>>
>>> What I'm looking for is a proper way to:
>>> - copy database (export, rename, import)
>>> - import new data into new database with mysql import
>>> - switch database somehow (maybe through symlink change in
>>> LocalConfiguration.php ?). Another idea: Is there a way to use and change
>>> environment variables somehow so that I can tell TYPO3 which database
>>> to use?
>>>
>>> PS: I have written a deployment extension that runs with the scheduler
>>> and
>>> performs shell scripts so in the end I want a shell script ;)
>