[TYPO3] search for large sites

[Valery Romanchev]

Hi Dave
> Just wondering how many people here find problems with indexed search on 
> large sites. It's a subject that comes up again and again on the lists but 
> never seems to get a huge amount of response, which is probably to be 
> expected as many people don't reach the limits of what indexed search can 
> do.

I have this slow search problem with web site with 300 pages and 3000 
tt_news items (www.wciom.ru,  limited english version - www.wciom.com)

index_phash   3,303
index_rel   1,291,063  	
index_words  113,693


> Our site has ~27,000 pages of content which correlates to a index_rel table 
> with over 2 million rows of data. The search slows to a crawl (somewhere 
> between 10 and 30 seconds to get a search result) and pushes server load up 
> considerably when multiple searches are being performed.

I have the same situation -  15-40 sec to get result in default search.
My solution:
- use search by string as default (much faster)
- use search in tt_news as option for users
- proper crawling of tt_news items (with some patches to crawler)
The result is  acceptable:
http://wciom.ru/no_cache/poisk/

> What has everyone done who has a large site, have you tried optimising 
> indexed search, or have you just given up and looked for an external 
> solution? 

This was discussed some times:
mnogosearch.com is possible as site search engine for up to 1000000 
pages. But this require some efforts to install/configure

I think MySQL FULLTEXT search in MySQL 4.x and 5.x can handle 50000 
documents
I use it for search in description of 170000 images in 
http://www.vostock-photo.com - results usially below 10 sec.

So may be no external solution needed - just proper MySQL FULLTEXT usage.

---
Valery Romanchev
web-developer
Russian TYPO3 Laboratory
http://www.typo3lab.ru