[TYPO3-Solr] Tomcat & Solr 4.8 / ulimit

[Lorenz Ulrich]

Anyone? :-)

Am 13.09.2014 01:12, schrieb Lorenz Ulrich:
> Hello Johann
>
> Thanks for your help. useCompoundFile as well as the mergeFactor
> improved the situation a little bit. However the problem isn't solved
> and now I'm pretty sure what the reason is: The number of open files is
> growing together with the number of files in the
> typo3cores/data/[language]/tlog directory.
>
> After 'grep'ing the files I found out that the transaction logs are all
> from an index queue of a custom extension. We're doing a nightly import
> of around 600 addresses from an XML file. Before that, all existing
> addresses are truncated, afterwards the new ones are imported by a
> CommandController through Extbase persistence.
>
> After persisting, I'm re-initializing the index queue so that the new
> addresses are put to the Solr index. Apart from some custom built fields
> the setup looks pretty normal:
>
>      // enables indexing of addresses
>      tx_useruniluaddress_domain_model_address = 1
>      tx_useruniluaddress_domain_model_address {
>          fields {
>
>              title = TEXT
>              title.dataWrap = {field:last_name} {field:first_name}
>
>          prefix_stringS = title
>              email_stringS = e_mail
>              website_stringS = website
>
>          telephone_stringS = telephone
>
>              url = TEXT
>              url {
>                  typolink.parameter =
> {$plugin.tx_useruniluaddress.singlePid}
>                  typolink.additionalParams =
> &tx_useruniluaddress_addresses[address]={field:uid}&L={field:__solr_index_language}&tx_useruniluaddress_addresses[action]=show&tx_useruniluaddress_addresses[controller]=Address
>
>                  typolink.additionalParams.insertData = 1
>                  typolink.returnLast = url
>                  typolink.useCacheHash = 1
>              }
>          }
>      }
>
> So far everything works fine, but in this process, Solr obviously
> creates, opens and never closes a translation log file for each  (
> (!) address. So after around 7 days I have 4000 open files and the
> ulimit is reached.
>
> What puzzles me is the fact that every record import is an own
> transaction. Wouldn't it be better - performance-wise - to only commit
> after a relevant number of items?
>
> Does anyone have an idea why the transaction log files are not closed
> and removed?
>
> Thanks for your help.
>
> Best regards,
>
> Lorenz
>
>
> Am 03.09.2014 10:30, schrieb Johann Höchtl:
>> Hallo Lorenz,
>>
>> try to set useCompoundFile true in your solrconfig.xml  (see
>> https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig).
>>
>> If that’s not enough, set the mergeFactor to a value between 2 and 10.
>>
>> Best regards,
>> Hans
>
>>
>