[TYPO3-english] R: Again: How to configure crawler 3.0.0?

Tue Nov 3 09:56:29 CET 2009

Hello Tolleiv,
Thanks for your answer, but
I'm pretty sorry that the whole topic is new for me, so... is there a "crawler for dummies" guide somewhere? I should start from zero :(

You wrote

> Running it through the backend was never a good option (since it will 
> take away too much resources from the webserver).

I know, but first of all, I want to understand how to make it work manually and THEN how to configure it for handling automatic jobs.

> The 3.x version of the 
> crawler requires that TYPO3 has support for popen to run the crawler in 
> a new process when you press that button in the backend. popen might not 
> be available in safe_mode

How can I check such things? 
In my php.ini 
safe_mode = Off

regards

**************************************
Riccardo De Contardi

B Human Srl - www.bhuman.it 
Via Canzio, 15 - 20131 Milano
Tel +39.02 20 23 271 - Fax +39.02 20 240 561
-----Messaggio originale-----
Da: typo3-english-bounces at lists.typo3.org [mailto:typo3-english-bounces at lists.typo3.org] Per conto di Tolleiv.Nietsch
Inviato: martedì 3 novembre 2009 0.49
A: typo3-english at lists.typo3.org
Oggetto: Re: [TYPO3-english] Again: How to configure crawler 3.0.0?

Hi -
the crawler is meant to run through the cli_dispatcher of typo3 ... did 
you ever try this out?

Running it through the backend was never a good option (since it will 
take away too much resources from the webserver). The 3.x version of the 
crawler requires that TYPO3 has support for popen to run the crawler in 
a new process when you press that button in the backend. popen might not 
be available in safe_mode.

Cheers,
Tolleiv

De Contardi Riccardo schrieb:
> Sorry to bother you again about this topic, but I'd need an answer
> Thank you very much!
>  
> 
> Could you help me with the last version of Crawler (3.0.0)? 
> 
> It's changed a lot from the previous one (2.1.0) and I can't get it
> working!
> 
>                 
> 
> On the old version, to crawl my pages manually (for indexing, for
> example), I followed these steps:
> 
>                 
> 
> Module: INFO    
> 
> + Select from dropdown: Site crawler
> 
> + Select from dropdown: Start Crawling 
> 
> + Select from dropdown the Depth (e.g. "Infinite")
> 
> + select URL (e.g. Home page (ID=1)
> 
> + select the Processing Instructions (e.g. "Re-indexing")
> 
> + click on "Update"
> 
> + click on "Crawl URLS"
> 
> + click on "continue"
> 
> + Select from dropdown: CLI status
> 
> + Click on "Run now"
> 
>  
> 
> And that was OK!
> 
>  
> 
> Now:
> 
>  
> 
> Module: INFO
> 
> + Select from dropdown: Site crawler
> 
> + Select from dropdown: Start Crawling 
> 
> + Select from dropdown the Depth (e.g. "Infinite")
> 
> + Mark the "configurations"
> 
> + click on "Update"
> 
> + click on "Crawl Urls"
> 
> + click on "continue"
> 
> + Select from dropdown: Crawling Processes
> 
> + Click on button "enable crawling"
> 
> + Click on button "Add processes" (The plus with green background)
> 
>  
> 
> It appears a message with orange border "New process has been started,
> refresh to monitor the state"
> 
>  
> 
> I click on  "refresh" only to discover that nothing was done at all!
> 
> Other messages:
> 
> General Information:
> 
> Pending Entries (assigned / overall):     0 / 125
> 
> Current server time:       22-10-09 12:59:31
> 
> Processes count (running / max)           0 / 1
> 
> CLI-Path:          /usr/bin/php
> /var/lib/typo3-sites/dummy-4.2.9_test_din2L/typo3/cli_dispatch.phpsh
> crawler
> 
>  
> 
> I'm pretty sure there's something I miss somewhere...could you help me?
> 
> Thanks!                                            
> 
>  
> 
> **************************************
> 
> Riccardo De Contardi
> 
>  
> 
> B Human Srl - www.bhuman.it
> <file:///C:\Documents%20and%20Settings\riccardo.decontardi\Dati%20applic
> azioni\Microsoft\Firme%20elettroniche\www.bhuman.it>  
> 
> Via Canzio, 15 - 20131 Milano
> 
> Tel +39.02 20 23 271 - Fax +39.02 20 240 561
> 
>  
> 
> 
> 
> Chi riceve il presente messaggio e' tenuto a verificare se lo stesso non
> gli sia pervenuto per errore. In tal caso e' pregato di avvisare
> immediatamente il mittente e, tenuto conto delle responsabilita connesse
> all'indebito utilizzo e/o divulgazione del messaggio e/o delle
> informazioni in esso contenute, voglia cancellare l'originale e
> distruggere le varie copie o stampe.
> 
> The receiver of this message is required to check if he/she has received
> it erroneously. If so, the receiver is requested to immediately inform
> the sender and - in consideration of the responsibilities arising from
> undue use and/or disclosure of the message and/or the information
> contained therein - destroy the original message and any copy or
> printout thereof.
> 
> 
> _______________________________________________
> TYPO3-english mailing list
> TYPO3-english at lists.typo3.org
> http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-english
> 
> 
> Chi riceve il presente messaggio e' tenuto a verificare se lo stesso non gli sia pervenuto per errore. In tal caso e' pregato di avvisare immediatamente il mittente e, tenuto conto delle responsabilita connesse all'indebito utilizzo e/o divulgazione del messaggio e/o delle informazioni in esso contenute, voglia cancellare l'originale e distruggere le varie copie o stampe.
> 
> The receiver of this message is required to check if he/she has received it erroneously. If so, the receiver is requested to immediately inform the sender and - in consideration of the responsibilities arising from undue use and/or disclosure of the message and/or the information contained therein - destroy the original message and any copy or printout thereof.
> 
> 

-- 
Tolleiv Nietsch
www.tolleiv.de - www.aoemedia.de
_______________________________________________
TYPO3-english mailing list
TYPO3-english at lists.typo3.org
http://lists.typo3.org/cgi-bin/mailman/listinfo/typo3-english

Chi riceve il presente messaggio e' tenuto a verificare se lo stesso non gli sia pervenuto per errore. In tal caso e' pregato di avvisare immediatamente il mittente e, tenuto conto delle responsabilità connesse all'indebito utilizzo e/o divulgazione del messaggio e/o delle informazioni in esso contenute, voglia cancellare l'originale e distruggere le varie copie o stampe.

The receiver of this message is required to check if he/she has received it erroneously. If so, the receiver is requested to immediately inform the sender and - in consideration of the responsibilities arising from undue use and/or disclosure of the message and/or the information contained therein - destroy the original message and any copy or printout thereof.