[TYPO3-mvc] memory_limit: 12000 records too much for extbase?

Wed Feb 12 09:45:08 CET 2014

Hi Gerd,

As Your have limited Resources on your machine.

You might fall back to Raw SQL and skip the Extbase stuff to keep your memory usage low. 

Also MySQL is able to Write CSV files directly. If you don’t do complicated Processing of the Data this 
Might be soloution for your export. See link below:

http://www.tech-recipes.com/rx/1475/save-mysql-query-results-into-a-text-or-csv-file/

regards chris

-----Ursprüngliche Nachricht-----
Von: typo3-project-typo3v4mvc-bounces at lists.typo3.org [mailto:typo3-project-typo3v4mvc-bounces at lists.typo3.org] Im Auftrag von Gerhard Mehsel
Gesendet: Mittwoch, 12. Februar 2014 00:42
An: typo3-project-typo3v4mvc at lists.typo3.org
Betreff: Re: [TYPO3-mvc] memory_limit: 12000 records too much for extbase?

Hello Stephan,

thank you very much for your explanation!
I did what you suggested but I did not use all 77 columns because they are not mapped yet (it's a non TYPO3 table; it's mapped through TS persistence settings). I used 3 columns + uid column as I did before when started this thread.

1) 47.114M (TYPO3)
2) 107.18k (after fetching one record - TYPO3)
3) 3.752B (memory consumption of one record)
3.1) 47.223M (total memory after 1000 records)
4) 47.204k (= $r3 * all 12883 records)

The SQL dump of that table consumes 430k (3 columns + uid column).
If I look at 3) and 4) I cannot see a reason why my memory should get exhausted, do you?.

And you are right, 12000records/request is unusual and unusable, but I 
have to export that result to an Excel file :(

Gerd

Am 11.02.14 22:43, schrieb Stephan Schuler:
> Hey there.
>
> I don't think that it's unlikely. But keep in mind that the memory isn't only caused by 12'000 domain objects but by lots of environment stuff, too.
>
> You could dig a bit deeper with memory_get_usage() to see how much of that memory is actually used by your payload and how much is caused by the bootstrap and framework stuff.
>
> 1: Fetching memory_get_usage() right before you do anything gives you an offset. TYPO3 uses this amount of memory just to be there. Currenty this is huge on 6.1 due to several reasons. Let's see what is going on with this in the next releases, but intil now you have to live with that.
>
> 2: Call "findByUid(1)" exactly once and fetch memory_get_usage() right after that. This triggers some autoloading mechanisms (class loaders, for example). This shold be a tiny step compared to the offset before, but it should be noticable.
>
> 3: Call "findAllWithLimit(1000)" (or whatever your method is called that queries only 1000 records) and fetch memory_get_usage() once again.
>
> The ($1) is huge, as explained. Calc ($2-$1), to see if there is lots of framework stuff to be loaded dynamically. Then calc (($3-$2)/1000) to get the raw amount of data a single object needs. That should be less then ($2-$1) because there shouldn't be any autoloader and other environment being involved.
>
> Additionally, there's a huge difference if you call a single request for the very first time after clearing caches or if it is a follow up request. Even for no_cached requests, things like e.g. TypoScript is cached on follow up requests but needs a lot of more memory on the first request to be calculated and cached.
>
> If you pass your data to a fluid template, then you need additional memory because the two reasons. First: Fluid is cachable and cached. The very first request to a certain fluid template calls the template rendering, which uses memory in order to create an executable closure of that template. So the first call to a fluid template uses more memory because than the actual execution of that template needs. Second: The output of every TYPO3 request is calculated completely in memory. If all output data is completely available, then the $GLOBALS[TSFE]->content is echoed.
>
> There are lots of reasons why you need more memory. And you need to know which data could be simplified and reduced in any way in order to optimize.
>
> But honestly, 12000 records for a single request is really unusual. Most situations introduce pagination for both, memory/perforance as well as usability.
>
> You could just limit those 12000 records of 77 colums to less columns and create relations. Those could allow you to use lazy loading of foreign data. Of course when creationg a table with nearly a million cells, you need all of those data. But you most likely can live without lots of those data and only fetch what you need.
>
> Just to compare: When you "mysqldump" this particular table of 12000 records, could you tell me the size the output file grows to? I'm kind of interested in knowing if the PHP stuff ($3-$2) causes like five times the memory of the raw text output of the database or like 50 times.
>
>
> Regards,
> Stephan.
>
>
> Stephan Schuler
> Web-Entwickler
>
> Telefon: +49 (911) 539909 - 0
> E-Mail: Stephan.Schuler at netlogix.de
> Website: media.netlogix.de
>
>
>
> --------------------
> Geben Sie Cyber-Kriminellen keine Chance
> Lückenloser Schutz mit Security-Lösungen von Sophos: Erfahren Sie, wie Sie Ihr Unternehmen wirksam schützen und gewinnen Sie einen Boomball Pro Mini-Lautsprecher:
> https://www.cyber-gesindel-a.de/netlogix
> --------------------
>
>
>
> --
> netlogix GmbH & Co. KG
> IT-Services | IT-Training | Media
> Neuwieder Straße 10 | 90411 Nürnberg
> Telefon: +49 (911) 539909 - 0 | Fax: +49 (911) 539909 - 99
> E-Mail: info at netlogix.de | Internet: http://www.netlogix.de
>
> netlogix GmbH & Co. KG ist eingetragen am Amtsgericht Nürnberg (HRA 13338)
> Persönlich haftende Gesellschafterin: netlogix Verwaltungs GmbH (HRB 20634)
> Umsatzsteuer-Identifikationsnummer: DE 233472254
> Geschäftsführer: Stefan Buchta, Matthias Schmidt
>
>
>
> ________________________________________
> Von: typo3-project-typo3v4mvc-bounces at lists.typo3.org [typo3-project-typo3v4mvc-bounces at lists.typo3.org]&quot; im Auftrag von &quot;Ralf-Rene Schröder [ralf.rene at online.de]
> Gesendet: Dienstag, 11. Februar 2014 21:07
> An: typo3-project-typo3v4mvc at lists.typo3.org
> Betreff: Re: [TYPO3-mvc] memory_limit: 12000 records too much for extbase?
>
> Am 11.02.2014 20:23, schrieb Gerhard Mehsel:
>> Hello,
>>
>> I have an extbase model that points to a flat mysql table with ~12.000
>> records, 77 columns, no relations to other models. It's 77 columns
>> because I flattened the table at import time with mysql joins to avoid a
>> lot of model-relations to gain performance.
>>
>> The memory_limit is set to 90M because of my hoster.
> this is a little bit low
> http://typo3.org/about/typo3-the-cms/system-requirements/
>
> i drive at least 128M (lower value only for mini-installations)
> for an installation with your 12000 Records
> i think you should have the normal recomended value of 256M
>
>
> --
> image[FORMAT] - Ralf-René Schröder
> http://www.image-format.eu ... Wir geben Ihrem Image das richtige Format
>
> aktueller TIPP: www.startnext.de/typo3-theme-packages
> _______________________________________________
> TYPO3-project-typo3v4mvc mailing list
> TYPO3-project-typo3v4mvc at lists.typo3.org
> http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-project-typo3v4mvc
>

_______________________________________________
TYPO3-project-typo3v4mvc mailing list
TYPO3-project-typo3v4mvc at lists.typo3.org
http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-project-typo3v4mvc