[TYPO3-Solr] Tomcat & Solr 4.8 / ulimit

[Lorenz Ulrich]

Hello Johann

Thanks for your help. useCompoundFile as well as the mergeFactor 
improved the situation a little bit. However the problem isn't solved 
and now I'm pretty sure what the reason is: The number of open files is 
growing together with the number of files in the 
typo3cores/data/[language]/tlog directory.

After 'grep'ing the files I found out that the transaction logs are all 
from an index queue of a custom extension. We're doing a nightly import 
of around 600 addresses from an XML file. Before that, all existing 
addresses are truncated, afterwards the new ones are imported by a 
CommandController through Extbase persistence.

After persisting, I'm re-initializing the index queue so that the new 
addresses are put to the Solr index. Apart from some custom built fields 
the setup looks pretty normal:

     // enables indexing of addresses
     tx_useruniluaddress_domain_model_address = 1
     tx_useruniluaddress_domain_model_address {
         fields {

             title = TEXT
             title.dataWrap = {field:last_name} {field:first_name}

	    prefix_stringS = title
             email_stringS = e_mail
             website_stringS = website

	    telephone_stringS = telephone

             url = TEXT
             url {
                 typolink.parameter = 
{$plugin.tx_useruniluaddress.singlePid}
                 typolink.additionalParams = 
&tx_useruniluaddress_addresses[address]={field:uid}&L={field:__solr_index_language}&tx_useruniluaddress_addresses[action]=show&tx_useruniluaddress_addresses[controller]=Address
                 typolink.additionalParams.insertData = 1
                 typolink.returnLast = url
                 typolink.useCacheHash = 1
             }
         }
     }

So far everything works fine, but in this process, Solr obviously 
creates, opens and never closes a translation log file for each  (
(!) address. So after around 7 days I have 4000 open files and the 
ulimit is reached.

What puzzles me is the fact that every record import is an own 
transaction. Wouldn't it be better - performance-wise - to only commit 
after a relevant number of items?

Does anyone have an idea why the transaction log files are not closed 
and removed?

Thanks for your help.

Best regards,

Lorenz


Am 03.09.2014 10:30, schrieb Johann Höchtl:
> Hallo Lorenz,
>
> try to set useCompoundFile true in your solrconfig.xml  (see https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig).
> If that’s not enough, set the mergeFactor to a value between 2 and 10.
>
> Best regards,
> Hans

>