[Flow] TYPO3 Surf questions

[Mark Kuiphuis]

Hi all,

Well, I have made good progress over the last couple of days in regards 
to TYPO3.Surf (I think), but I don't think I am there yet.

I have been able to start using migrations, with the suggestions from 
this thread. No complaints anymore with an existing database that tables 
can't be modified anymore etc. etc. :-)

I did a successful deployment to a local folder where some packages are 
downloaded from Packagist and some from our own GIT server.

Then I made a (small) change to one of our own packages and pushed that 
to the GIT server. Upon running the deployment again (./flow surf:deploy 
<deploymentFile>) I would have expected that our own modified package 
was going to be downloaded again including this change, but it looked 
like that wasn't the case.

With your (Christian) suggestion to store the composer.lock into the git 
repo, I was able to deploy the site for the first time, but upon the 2nd 
deployment, no changes were made. It looks like composer thinks that it 
is already using the stable version (without my change).

When I changed the stability of my two packages in the root 
composer.json from dev-master to @dev, they disappeared from the 
composer.lock file. Running the deployment again causes the 2 packages 
not to be downloaded at all...

The deployment also in my opinion still takes quite a while, so I am 
unsure whether the cached versions of TYPO3.Flow, TYPO3.Fluid, etc. etc. 
are being used or downloaded again.

Shouldn't the "cache" folder contain a cached version of TYPO3.Flow, 
TYPO3.Fluid, and all other packages (or are the composer cached versions 
stored elsewhere?)

My root composer.json currently looks like this. (formatting can be 
corrupt due to Thunderbird having 80 chars per line):

{
     "name": "typo3/flow-base-distribution",
     "description": "TYPO3 Flow Base Distribution",
     "license": "LGPL-3.0+",
     "config": {
         "vendor-dir": "Packages/Libraries",
         "bin-dir": "bin"
     },
     "require": {
         "typo3/flow": "2.2.*",
         "doctrine/migrations": "@dev",
         "vendor/package": "@dev",
         "vendor/package-client": "@dev",
         "typo3/swiftmailer": "dev-master",
         "famelo/pdf": "dev-master"
     },
     "repositories": [
         {
             "type": "vcs",
             "url": "ssh://user@server.tld:22/home/git/Vendor.Package.git"
         },
         {
             "type": "vcs",
             "url": 
"ssh://user@server.tld:22/home/git/Vendor.Package.Client.git"
         }
     ],
     "suggest": {
         "ext-pdo_sqlite": "For running functional tests out-of-the-box 
this is required"
     },
     "scripts": {
         "post-update-cmd": 
"TYPO3\\Flow\\Composer\\InstallerScripts::postUpdateAndInstall",
         "post-install-cmd": 
"TYPO3\\Flow\\Composer\\InstallerScripts::postUpdateAndInstall",
         "post-package-update": 
"TYPO3\\Flow\\Composer\\InstallerScripts::postPackageUpdateAndInstall",
         "post-package-install": 
"TYPO3\\Flow\\Composer\\InstallerScripts::postPackageUpdateAndInstall"
     }
}

This "package" also contains the "Configuration" folder including the 
Settings.yaml files as well as the Routes.yaml.

And last (but not least) the composer.lock which I believe is causing me 
grief to either not download the updated version of my package (with 
'dev-master' stability flag or not downloading it at all (with '@dev' 
stability flag))

Sorry for the long post.....

Cheers, Mark Kuiphuis

On 30/10/2014 5:02 pm, Christian Loock wrote:
> Hey,
>
> Am 29.10.2014 um 22:11 schrieb Mark Kuiphuis:
>> Thanks guys....
>>
>> Might need to some work on this then :-) But I wouldn't be where I am
>> currently at without both your help, so really appreciated :-)
>>
> I know surf can be somewhat confusing, since the documentation isn't all
> that good. I took me quite a while to get into it by experimenting
> around quite a bit. But once you get it running, the deployment works
> really good. It also not too hard to use surf to deploy other
> applications. We is it to deploy our Oxid Shop aswell, which shows how
> flexible surf is. I really love it and am glad I could help :)
>
>> So if I understand you correctly, on every deploy you are downloading
>> TYPO3.Flow, TYPO3.Fluid, Libraries like Doctrine, Yaml, etc. etc.?
>>
> As mentioned before, composer caches libraries, so once they are
> downloaded, composer will prefer the cached version, unless the
> requested version changed.
>
>
>> Cheers, Mark
>>
>> P.S. Might need to work on doctrine:migrate a bit more because up
>> until now we have only used doctrine:update :( Thanks for this tip....
>>
>> On 29/10/2014 6:20 pm, Christian Loock wrote:
>>> Just to add to this:
>>>
>>> Be aware that using doctrine:update, even if it might seem very
>>> convient, might screw up your possibilites to generate migrations with
>>> doctrine:migrationgenerate. We used to use doctrine:update on our loca
>>> working copies aswell, untill we tried to deploy our application the
>>> first time to our production environment, and realized that our
>>> migrations were broken, since they were missing stuff. Since then, we
>>> made it a convention to always use doctrine:migrate and never update the
>>> database directly with doctrine:update. This adds a little step, but
>>> makes your migrations safe and tight.
>>>
>>> Also note, that composer has a cache, that will cache repositories it
>>> checks out, for later use. So as long as you have you composer.lock
>>> versioned, it should only download the repos at the first time you
>>> deploy, and then take them straight from the cache, which is very quick.
>>>
>>> Cheers
>>>
>>>
>>> Am 29.10.2014 um 09:09 schrieb Beat Guggisberg:
>>>> Hey Mark
>>>>
>>>> If you copied the Database with all Data in it, then it should have
>>>> also copied the Migrations Table. So without a codechange there should
>>>> not have been any migration nessessary.
>>>>
>>>> On a Development system you should have no problem just to just clear
>>>> your database and make a migration to come to the correct Databasae
>>>> structure. To test your new things that change the database you can
>>>> update manually with doctrine:update. Never do that on a production
>>>> server, because it destroys future migrations.
>>>>
>>>> To create the migrations you need on the production server, clear the
>>>> DB on the testserver, do a doctrine:migrate and then a
>>>> doctrine:migrationgenerate. Now you have a file with all the DB
>>>> changes in it. You can split up that file for your different packages.
>>>>
>>>> For your problem with the downloads, i have created local copies of
>>>> all nessessary Git repositorys. You can do that with "git clone
>>>> --mirror", to update them use "git remote update".
>>>>
>>>> Now you have to add them as repository in your json.
>>>>
>>>> Kind Regards
>>>> Beat
>>>>
>>>>
>>>> Am Mittwoch, 29. Oktober 2014 01:54 CET, Mark Kuiphuis
>>>> <mark at capesso.com.au> schrieb:
>>>>   Thanks Christian and Beat :-)
>>>>
>>>> Since this is all still very new to me, I wanted to test things
>>>> thoroughly on a local server before putting it to use on a Live
>>>> Production Server.
>>>>
>>>> I therefore made a copy of my development database instead of creating
>>>> an empty database (which I did refer to by the way in the Settings.yaml
>>>> in both Production and Development Contexts).
>>>>
>>>> It turned out that, because there was already a database (with
>>>> tables of
>>>> my packages and the TYPO3.Party tables) it could not rename the
>>>> party_abstractparty to typo3_party_domain_model_abstractparty, because
>>>> that table already existed.....
>>>>
>>>> I now need to find a way to move a lot of folders from the releases
>>>> folder to the "shared" folder and symlink them, because I don't want to
>>>> download TYPO3.Flow, TYPO3.Fluid, and all libraries when I made a
>>>> change
>>>> to one of my own packages and want to deploy these changes.
>>>>
>>>> And while my own packages have been downloaded succesfully from our own
>>>> git server, it did not create the necessary tables upon first
>>>> run....but
>>>> that could maybe be due to the lack of a migration script....
>>>> (not something that I want to do upon each deploy anyway as it should
>>>> reuse existing tables and modify them instead of creating (for example
>>>> party_abstractparty which then should be renamed again to
>>>> typo3_party_domain_model_abstractparty)...
>>>>
>>>> Cheers, Mark
>>>>
>>>> On 27/10/2014 4:47 pm, Christian Loock wrote:
>>>>> Hey,
>>>>>
>>>>> you are probably just missing the Config for the Production Context
>>>>> in you
>>>>> base repo. Surf will automatically call everything in Production
>>>>> context,
>>>>> so you need your Configuration/Production/Settings.yaml to reflect to
>>>>> your
>>>>> actual production database.
>>>>>
>>>>> Also you are right, Surf will automatically create the whole folder
>>>>> structur for you. Make sure your vhost on your production node
>>>>> points to
>>>>> %projectdir%/releases/current/Web where %projectdir%= the directory
>>>>> you
>>>>> deploy to.