[Typo3-typo3org] Comments on the mirroring concept

[Robert Lemke]

Hi Jürgen,

> --- Jürgen Egeling wrote: ------------------------------------------------
> 
> As a server admin I would like to have the chance to let people
> only connect to *one given* mirror. E.G. in a big hosting, I probably want
> to have some "supported" EXT where the hoster
> has tested and will answer questions to the EXT, ...

You won't neccessarily know to which mirror the EM connects and it really
doesn't matter as long as it is fast enough: It will just display and fetch
the extensions which are available on any other mirror or the main
repository.

> Same in a big company, where people might be able to install
> their own EXT, but only from a pool.
> IMHO this could be easily achived if there is a configuration in the
> install tool like "all EXT, or own EXT".

Yes, that means you would disable the EM connection to the main repository
and only use the custom repository.

But in my opinion that does not make sense: Any site developer who is
allowed to install extensions could just trick TYPO3 out by using some
TypoScript or PHP to override your setting to disable external connection.

> Also it might make sense to limit the server access region wide,
> as if you suggest "the ext manager tries to connect to the fist..."
> there is the chance that this first server is behind a 64k line, ...
> (beeing the first in the list, does not mean beeing the fastest
> to respond to one specific user.)
> So it might be an idea to let people choose which server they want.
> I quite like the concept of sourceforge, where I can choose the
> download location of my own (and where lucky to pick fast ones, ...)

The idea is that the super servers provide your EM with a list of servers
that is already sorted by

  - proximity to your own server
  - high bandwith first
  - short pings first

The bandwith and ping times will be checked by the super servers
periodically. Additionally it is possible for the TYPO3.org administrators
to prioritize mirrors so they will be prefered.

> Better we have a few good ones, where people can choose, than
> letting *all* setting up a server and supporting it with the EXT,
> with the EXT that has no chance to guarantee any performance.
> As a bad competitor I would buy 20 64kBit leased lines and
> put them in as supporting mirrors ;-)))
> Or even uglieer: Have VMWare simulate 20 servers on my old
> IBM 386 ;-)) (OK, enough)

Yes of course it is better to have a few good ones. And we will have them,
and additionally we will have some others. We might think about a minimum
of bandwith a mirror must provide and we can easily check that with some
automated scripts.

> another thing: The MD5 is ok, but if you provide this as "typo3.org"
> for the people, you are close to guaranteeing something. I would not
> try to be put in this corner, ...
> Tell them these are mirros, and thats it.

No, I think that this is vital. We guarantee that it is the same version
which has been uploaded to the TER, nothing more, nothing less. I think
that is easy to achieve with a minimum of risk.

If MD5 is not strong enough we'll have to look for a newer hashing
algorithm.

> Setting up a supporting mirror:
> I would let people get in contact with us, and try to find good
> supporting sponsors, not "anyone". Look at the sourceforge list.
> I bet there is a reason why there are so few servers.

Yes, I think you are right. If so, we'd have to build up some criteria like
bandwith, uptime and ping times. We should be able to check that of course.

> I would not push the EXT to the mirrors, I would let the mirrors
> *get* the files. This might get into non consistent EXT repositories,
> but this is what I would fix by letting the EXT client manager connect
> to a repository and get all information from there. 

I don't want to push the EXT to the mirrors, they retrieve it by anonymous
rsync from the main repository.

> Than after getting 
> the EXT there *could* be an MD5 check, but there is no need. If you have
> an FTP session to a mirror, you get the snapshot. But as they are
> usually mirrored in the night, is is fast enough I think.

The idea is not to mirror every night but every 15 minutes! Of course we'd
have to test how much load a 15 or 30 minutely rsync puts on the
repository.

> My summary: It goes very much in the direction I wanted to set this
> up, but I really want to discuss some details.

Well, let's discuss it here anymways and of course all those being
interested can meet at the Cebit!

Cheers,

-- 
robert