[TYPO3-dev] linkvalidator's memory footprint

[Philipp Gampe]

Hi Xavier,

Xavier Perseguers wrote:

> Just an idea, could we have some kind of "dynamic" list of such servers
> (not hardcoded) that would allow to only fallback to GET if really needed?

Well, I think the problem was that we found no way to create this list 
statically, because we run the check code for every link.
Also, each link would need to be checked for each entry in that list... that 
could eat up some performance. 

After adding Request2 to core, we might want to try a more dynamic approach:
1. check all links with HEAD
2. if it fails recheck with GET, but only define a range (0-4Kb or so)
3. if it still fails, mark the link as failed

So we only download that much, that we need to see all header lines. We do 
not care for the rest of the content.

Best regards
-- 
Philipp Gampe