[Typo3-dev] Web crawler for indexed search plugin !!

[Gaurav Prasad]

Boris Senker wrote:

>
>
> Gaurav Prasad wrote:
>
>> Hi,
>> I was wondering if there is any php based crawler which crawl all 
>> links in typo3 site to index it
>> may be we can integrate this with indexed search engine later on
>>
>> I was also looking at http://www.phpmini.com/crawler.php
>
>
> Crawls meta tags.
>
>>
>> have any attempted this before or if there is better solution pl tell me
>
>
> Maybe you'll find this one better - http://phpcrawl.cuab.de/index.html
>
this(php crawl) looks good.... we can easily extend to crawl pages in 
typo3 so that they indexed automatically but questions how do we crawl 
protected content ?

regards,
gaurav