[TYPO3-typo3org] robots.txt at typo3.org - searchable extensions' source code

[Wolfgang Zenker]

Hello,

* Krystian Szymukowicz <"t33kRE.MO.VE."@RE.MO.VE.prolabium.com> [070904 18:41]:
> are there any specific reason why robots.txt looks like this?

> User-agent: *
> Disallow: /extensions/repository/view/

yes, there are specific reasons for that:
- the source view highlights any source files from the extensions,
  which puts quite a bit of load on the servers when several robots
  index extension views in parallel
- there is currently no way to restrict the view to the current version
  of any extension, so robots try to index the detail view of all available
  versions of all extensions. Unfortunately that are millions of different
  page versions, which leads to extreme cache growth on the server.

> In depth of typo3.org/extensions/repository/view/* there is a huge 
> amount of working code examples that may be easily searched and be very 
> useful for beginner programmers.
> Unfortunately with such robots.txt soon it will be not possible.

> Are there any chance to make it searchable again?

There is a team working on the re-design of the typo3.org website,
which includes a redesign of the extension repository web interface.
After this redesign is concluded I expect the extension repository
to be searchable again.

> If this is performance issue maybe there should be "extension source 
> archive". Only pure source code without any other information about 
> extensions. Static files - no performance issue. Updated once a 3 month.

> It could be placed for example under: typo3.org/extensions/source-archive/

> I can make such archive and keep it up to date.

Thank you for your offer; but I think the new typo3.org website is not
that far away that the effort would be justified.

Regards,
Wolfgang Zenker
-- 
punkt.de GmbH         Internet - Service - Consulting
Vorholzstr. 25        Tel. +49 721 9109 -0 Fax: -100
D-76137 Karlsruhe     http://punkt.de
Gf: Jürgen Egeling    AG Mannheim HRB 108285