[Typo3-dev] Web crawler for indexed search plugin !!

[Gaurav Prasad]

Hi,
     I was wondering if there is any php based crawler which crawl all 
links in typo3 site to index it
may be we can integrate this with indexed search engine later on

I was also looking at http://www.phpmini.com/crawler.php

have any attempted this before or if there is better solution pl tell me

regards.
gaurav