[TYPO3] crawler extension & external non-typo3 sites

[Elijah]

Hi,

It seems that the typo3 crawler could only go through similar cached 
typo3 sites (pls correct me if I'm wrong) but what about other dynamic 
non-typo3 sites?

I've just tested it on one external non-typo3 link and made it crawl 
through 3 levels and it only fetched the top level link and never went 
through the links. But while testing it on a cached typo3 site it 
traversed through the page tree nicely.


Elijah