[TYPO3] External Crawler

[André Esch]

Ok, the hole Story:

We have a quit large website: Round about 10.000 pages and around
10 million PIs. I found that there are a lot queries to the "index_"-tables
while FrontEnd Indexing is enabled and the server load was very high
(to high) in these times . So I disabled it. After that, the server load 
gone
done. Now i'd like to do the indexing during the night, when the server
is bored.

While my research I found this:
"Currently the extension is under observation because instances of heavy
server load/unstability has been reported. It is not yet clear if THIS 
extension
has anything to do with. So it's only under suspicion at this point until 
further
data has been collected. But for now it is adviced to be careful with the
application of the extension for mission critical, high-load environments.

It's still uncertain how performance is under heavy load conditions and when
MANY pages are indexed. Currently benchmarks has been done only up to
2000 pages indexed/approx. 400.000 relation records. It is probably that
some parts has to be optimized for such scenarios."
in here
www.eurolab.co.at/fileadmin/pdf/typo3/indexedSearch.pdf
which confirmed my thoughts. This is why I need a external crawler/indexer.

I hope to made my Problem clear now. Sorry, my english isn't to good.

with best thanks
André

"Elmar Hinz" <elmar.DOT.hinz at team.MINUS.red.DOT.net> schrieb im Newsbeitrag 
news:mailman.1.1137515759.11688.typo3-english at lists.netfielders.de...
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> André Esch schrieb:
>> Maybee I'm too stupid, but how can wget index the site, when
>> Frontend Indexing is diabled?
>>
>
> Maybe I missunderstand what you mean by a BE indexer. With an external 
> crawler
> you can only access the FE, cause the BE is internal.
>
> Why does indexing produces a too heavy load? Google indexes your page 
> every few
> days. Does it break down?
>
> If you disable caching/indexing the processor load will only rise.
>
> Regards
>
> Elmar
>
>
> - --
> Climate change 2006 is killing people: floods in California, drought and 
> fires
> in Australia, Texas, Sahel, Oklahoma, South Africa. The Bush 
> administration is
> responsible for corruption of the Kyoto Protocol. The US majority is 
> responsible
> to the world for reelection of a convictable [...censored by Echelon...].
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFDzRzuO976RNoy/18RAkKeAKCjV6cLg+s9vudL33QubYjO5XJ4IACg6oql
> qynyRKI2GSEexqPXJrJIqr4=
> =xxCn
> -----END PGP SIGNATURE-----