Ben Wardle wrote: > with wget --spider it won't even save the content in the first place :-) > > Using --load-cookies=FILE it shld prob also be possible to spider > restricted access sections of your site... thanks! I am just wondering why the extension "crawler" takes so long to develop then...what else can it do once finished?