[TYPO3-core] RFC: Bug #7787: Yahoo, MSN, Ask.com and Alexa are not recognized in TypoScript robot condition

[Ernesto Baschny [cron IT]]

Martin Kutschker wrote: on 17.03.2008 20:58:

>> Martin Kutschker schrieb:
>>> Ok, but the list is a bit short. Some links to make it longer:
>>>
>>> http://www.user-agents.org/
>>> http://www.botsvsbrowsers.com/category/1/index.html
>>> http://www.robotstxt.org/db.html
>>> http://www.jafsoft.com/searchengines/webbots.html
>>
>> Thanks for the list. We have now two possibilites:
>> 1) Use the very short list of this RFC as requested in the bugtracker 
>> for TYPO3 4.2 and find a good solution to have much more user-agents 
>> with TYPO3 4.3
>> 2) Postpone this RFC to TYPO3 4.3 and do the "long-list-thing" then
>>
>> What do you think?
> 
> Shortlist now (also for 4.1), better stuff in 4.3

I would use get_browser() PHP function [1] and check for "crawler". The 
crawler-matching list is already being maintained for ages now in the 
browscap.ini file [2].

So I don't see why we should "reinvent" the wheel, maybe just provide a 
"deprecated" fallback in case the server hasn't got a valid browscap 
setting (in this case, we should recommend the user to configure that).

Cheers,
Ernesto

[1] http://www.php.net/get_browser
[2] http://browsers.garykeith.com/downloads.asp