[TYPO3-dev] Google blocked in Forge

[Christian Weiske]

Hello J,


> not sure if this is the right place to inquire about this, but I was
> always wondering why no results for pages in forge.typo3.org showed up
> in Google. Today I set my useragent to googlebot and got the message
> "Forbidden. You don't have permission to access [...] on this server."
> No wonder!

This has been discussed on typo3.teams.typo3org:

From: Steffen Gebert <steffen.gebert at typo3.org>
Subject: Re: [TYPO3-typo3org] Please allow indexing of wiki content on
forge by search engines Date: Mon, 23 May 2011 18:09:02 +0200
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10 Newsgroups:
typo3.teams.typo3org Organization: TYPO3 Core Team

> Please allow indexing of wiki content on forge.typo3.org by search
> engines.
>
> There's so many helpful documentation in the project wikis which can't
> be found via search engines.  
After you mentioned this, I reported two issues yesterday:

* forge not indexed by Google
   http://forge.typo3.org/issues/26980
   Probably because the UA filter you mentioned

* robots.txt disallow indexing
   http://forge.typo3.org/issues/26981
   This disallows indexing of the issue trackers. So nobody will find 
TYPO3 bugs *yeah*

> There's some mechanism which filters by UA strings and redirects
> searchbots to 403 page.  
I suppose it's from the times, when bots indexed forge and made it 
really slow. Wget access is also disallowed. This really has to be
changed.



-- 
Viele Grüße
Dipl.-Inf. Christian Weiske

Senior Developer
Netresearch GmbH & Co. KG