[Typo3-dev] extension "crawler"

[Peter Russ]

Kraft Bernhard wrote:

> Stefan Beylen wrote:
> 
>> i dont know much about crawlers nor cronjobs, but if i call every page 
>> in lynx it should be indexed by the indexed search, right?
> 
> 
> 
> wget -m http://www.yoursite.com/
> 
> will make you life easier than fetching every single page. It creates a 
> directory called
> www.yoursite.com in the directory where it is called ... so you most 
> probably want to
> put it into a little shell script which changes directory to /tmp first 
> (don't know in which directory
> jobs executed from cron run)
> 
> 
> greets,
> Bernhard
man wget is your friend:
wget -r -nd --delete-after http://whatever.com/

check for it ;-)

Regs. Peter.