[TYPO3-mvc] findOneBy memory usage

[Bastian Waidelich]

Saverio Vigni wrote:

Hi Saverio,

> I'm making a simple web spider for checking the link structure of a
> quite big vacation rental website [...]
>
> $newPage = $this->pageRepository->findOneByUrl($link->getUrl());
> if(!$newPage){
>      $newPage =
> $this->objectManager->create('Tx_Webspider_Domain_Model_Page');
>      $newPage->setUrl($link->getUrl());
>      $this->pageRepository->add($newPage);
> }

> this is inside the loop of the links found in the page I'm processing,
> the cal to the findOneBy makes the memory usage grow from 48MB to
> hundreds of megabytes.


1) Add an index to the "url" column of your page table (as the url 
should be unambiguous this could be a unique index)

2) try replacing $this->pageRepository->findOneByUrl by 
$this->pageRepository->countByUrl - maybe that reduces memory consumption

If both doesn't help you could query the database directly form your 
repository or some service class:

public function pageWithUrlExists($url) {
	$tableName = 'tx_yourextension_page';
	$where = 'url = ' . $GLOBALS['TYPO3_DB']->fullQuoteStr($url, $tableName);
	return $GLOBALS['TYPO3_DB']->exec_SELECTcountRows('*', $tableName, 
$where) > 0;
}

(untested)

If you're aware of the implications (less abstraction, no model mapping, 
..) and if you encapsulate it to separate classes there's no harm in 
accessing the db directly.


BTW: Note, that new pages are found only after persistAll() was called 
(automatically after the controller has been dispatched). So you won't 
find a page that has been added in the same session otherwise.
Calling persistAll() for each iteration would be an overkill, but you 
could store the urls in memory (in an array for example) and check this 
before querying the database.

HTH

-- 
Bastian Waidelich
TYPO3 Core Team Member

TYPO3 .... inspiring people to share!
Get involved: typo3.org