[TYPO3-mvc] memory_limit: 12000 records too much for extbase?

[Stephan Schuler]

Hey there.

I don't think that it's unlikely. But keep in mind that the memory isn't only caused by 12'000 domain objects but by lots of environment stuff, too.

You could dig a bit deeper with memory_get_usage() to see how much of that memory is actually used by your payload and how much is caused by the bootstrap and framework stuff.

1: Fetching memory_get_usage() right before you do anything gives you an offset. TYPO3 uses this amount of memory just to be there. Currenty this is huge on 6.1 due to several reasons. Let's see what is going on with this in the next releases, but intil now you have to live with that.

2: Call "findByUid(1)" exactly once and fetch memory_get_usage() right after that. This triggers some autoloading mechanisms (class loaders, for example). This shold be a tiny step compared to the offset before, but it should be noticable.

3: Call "findAllWithLimit(1000)" (or whatever your method is called that queries only 1000 records) and fetch memory_get_usage() once again.

The ($1) is huge, as explained. Calc ($2-$1), to see if there is lots of framework stuff to be loaded dynamically. Then calc (($3-$2)/1000) to get the raw amount of data a single object needs. That should be less then ($2-$1) because there shouldn't be any autoloader and other environment being involved.

Additionally, there's a huge difference if you call a single request for the very first time after clearing caches or if it is a follow up request. Even for no_cached requests, things like e.g. TypoScript is cached on follow up requests but needs a lot of more memory on the first request to be calculated and cached.

If you pass your data to a fluid template, then you need additional memory because the two reasons. First: Fluid is cachable and cached. The very first request to a certain fluid template calls the template rendering, which uses memory in order to create an executable closure of that template. So the first call to a fluid template uses more memory because than the actual execution of that template needs. Second: The output of every TYPO3 request is calculated completely in memory. If all output data is completely available, then the $GLOBALS[TSFE]->content is echoed.

There are lots of reasons why you need more memory. And you need to know which data could be simplified and reduced in any way in order to optimize.

But honestly, 12000 records for a single request is really unusual. Most situations introduce pagination for both, memory/perforance as well as usability.

You could just limit those 12000 records of 77 colums to less columns and create relations. Those could allow you to use lazy loading of foreign data. Of course when creationg a table with nearly a million cells, you need all of those data. But you most likely can live without lots of those data and only fetch what you need.

Just to compare: When you "mysqldump" this particular table of 12000 records, could you tell me the size the output file grows to? I'm kind of interested in knowing if the PHP stuff ($3-$2) causes like five times the memory of the raw text output of the database or like 50 times.


Regards,
Stephan.


Stephan Schuler
Web-Entwickler

Telefon: +49 (911) 539909 - 0
E-Mail: Stephan.Schuler at netlogix.de
Website: media.netlogix.de



--------------------
Geben Sie Cyber-Kriminellen keine Chance
Lückenloser Schutz mit Security-Lösungen von Sophos: Erfahren Sie, wie Sie Ihr Unternehmen wirksam schützen und gewinnen Sie einen Boomball Pro Mini-Lautsprecher:
https://www.cyber-gesindel-a.de/netlogix
--------------------



--
netlogix GmbH & Co. KG
IT-Services | IT-Training | Media
Neuwieder Straße 10 | 90411 Nürnberg
Telefon: +49 (911) 539909 - 0 | Fax: +49 (911) 539909 - 99
E-Mail: info at netlogix.de | Internet: http://www.netlogix.de

netlogix GmbH & Co. KG ist eingetragen am Amtsgericht Nürnberg (HRA 13338)
Persönlich haftende Gesellschafterin: netlogix Verwaltungs GmbH (HRB 20634)
Umsatzsteuer-Identifikationsnummer: DE 233472254
Geschäftsführer: Stefan Buchta, Matthias Schmidt



________________________________________
Von: typo3-project-typo3v4mvc-bounces at lists.typo3.org [typo3-project-typo3v4mvc-bounces at lists.typo3.org]" im Auftrag von "Ralf-Rene Schröder [ralf.rene at online.de]
Gesendet: Dienstag, 11. Februar 2014 21:07
An: typo3-project-typo3v4mvc at lists.typo3.org
Betreff: Re: [TYPO3-mvc] memory_limit: 12000 records too much for extbase?

Am 11.02.2014 20:23, schrieb Gerhard Mehsel:
> Hello,
>
> I have an extbase model that points to a flat mysql table with ~12.000
> records, 77 columns, no relations to other models. It's 77 columns
> because I flattened the table at import time with mysql joins to avoid a
> lot of model-relations to gain performance.
>
> The memory_limit is set to 90M because of my hoster.
this is a little bit low
http://typo3.org/about/typo3-the-cms/system-requirements/

i drive at least 128M (lower value only for mini-installations)
for an installation with your 12000 Records
i think you should have the normal recomended value of 256M


--
image[FORMAT] - Ralf-René Schröder
http://www.image-format.eu ... Wir geben Ihrem Image das richtige Format

aktueller TIPP: www.startnext.de/typo3-theme-packages
_______________________________________________
TYPO3-project-typo3v4mvc mailing list
TYPO3-project-typo3v4mvc at lists.typo3.org
http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-project-typo3v4mvc