[Typo3-dev] extension "crawler"

Ben Wardle ben.typo3 at netefficiency.co.uk
Wed Jun 8 16:54:35 CEST 2005


with wget --spider it won't even save the content in the first place :-)

Using --load-cookies=FILE it shld prob also be possible to spider 
restricted access sections of your site...

Stefan Beylen wrote:
> Peter Russ wrote:
> 
>> wget -r -nd --delete-after http://whatever.com/
> 
> 
> even better...cheers Peter!




More information about the TYPO3-dev mailing list