[TYPO3-Solr] Indexing hangs on localized records

[Stephan Schuler]

You're mixing different things.

CLI and CGI usually go with completely different php.ini files. What your browser shows in phpinfo() is only valid for CGI and does not count at all for CLI.

The number you mentioned (536870912) should be exactly 512MB. So it's not configured somehow lower then 512MB for CLI but your CLI simply needs even more than 512MB.

The frontend indexing works very different then record indexing.

For record indexing, the CLI loads each record (DAM or news or whatever), builds its indexing strategy by pure typoscript. Then it walks through several filter methods and pushes the result array to Solr. The memory here comes only from class loading (that should not be too much), the database results for the records (that should not be too much, too), accumulated generated data defined in your indexing configuration (even not so much memory, usually) and sometimes external value by some foreign bash commands, such as pdftotext or something.

The frontend indexing is completely different. The indexer loads the page records and fetches the frontend by CURL. So when indexing a single frontend page, there are two PHP processes: One CLI process that does the queue runner and another one that was triggered by your webserver and servers your webpage. The second one is a little different from default page rendering: A post processing mechanism cuts the output data and does a huge string manipulation operation over all of the output content.

You could try to find out which one dies with memory errors: The record indexer, the CLI part of the page indexer or the frontend part of the page indexer.

I really don't believe that allowing for more than 512MB memory for a single index document is the way you want the problem to be solved. Which means: I'm completely unsure if raising the memory solves your problem. But if it does, it's although the wrong way.

Regards,


Stephan Schuler

Web-Entwickler

Telefon: +49 (911) 539909 - 0
E-Mail: Stephan.Schuler at netlogix.de
Website: media.netlogix.de


--
netlogix GmbH & Co. KG
IT-Services | IT-Training | Media
Andernacher Straße 53 | 90411 Nürnberg
Telefon: +49 (911) 539909 - 0 | Fax: +49 (911) 539909 - 99
E-Mail: info at netlogix.de | Internet: http://www.netlogix.de

netlogix GmbH & Co. KG ist eingetragen am Amtsgericht Nürnberg (HRA 13338)
Persönlich haftende Gesellschafterin: netlogix Verwaltungs GmbH (HRB 20634)
Umsatzsteuer-Identifikationsnummer: DE 233472254
Geschäftsführer: Stefan Buchta, Matthias Schmidt



-----Ursprüngliche Nachricht-----
Von: typo3-project-solr-bounces at lists.typo3.org [mailto:typo3-project-solr-bounces at lists.typo3.org] Im Auftrag von Tomita Militaru
Gesendet: Donnerstag, 14. März 2013 10:19
An: Apache Solr search for TYPO3 project
Betreff: Re: [TYPO3-Solr] Indexing hangs on localized records


Hi Stephan,
>
>
> 512MB for indexing is quite a lot.
I know :)
>
> What's the number of simultaneously indexed records? I go with 50
> records every 60 seconds.
> But if 50 is too much per process, you could go even lower.
> Every 60 second is not too much, since the queue runner only runs once
> at a time. So if an earlier process is not finished yet, the next one
> will simply not be started.
I tried it with 1 record and I get same error.
>
> Do you know what kind of data you have in the malfunction record? Is
> this a DAM record pointing to PDF or DOCX or something else the
> indexer might open and read? Maybe you simply have a huge file
> attached to this which gets passed completely to the TYPO3 process
> memory during indexing?
I index both dam & tt_news records. DAM records have mostly PDF documents, but as I said before, I don't encounter problems on another clone environment. I tried indexing a localized tt_news record and I have the same problem. The problem is with localized records that the frontend is initialized and I guess there it needs more memory which shouldn't be a problem  with 512 MB, unless there is a CLI memory limitation.
I am waiting for an answer from the hosting company about this.
>
> But to be honest, the last indexer I configured was allowed to use as
> much memory as it needed: "memory_limit = -1" for cli.
I can't make changes like that, I need to ask the hosting company, but where would the cli memory limit be since phpinfo shows 512M?

Thanks,
Tomita

_______________________________________________
TYPO3-project-solr mailing list
TYPO3-project-solr at lists.typo3.org
http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-project-solr