[Typo3] Weirdness with google indexing

Boris Senker typo3 at dvotocka.hr
Thu Jun 16 20:54:15 CEST 2005

"rodrigo cea" <public2 at altamira.cl> wrote in message 
news:mailman.1.1118932992.13994.typo3-english at lists.netfielders.de...
> List,
> I recently created a website, using typo3.7. My awstats shows the 
> googlebot having visited the site 3 times, and only requesting 1 page each 
> time, "+1" which means it also requested the robots.txt.
> If I put our address into google, it only shows the front page of the 
> site. Putting in exact text from inside pages doesn't give any results.
> I have the "meta extended" extension installed, but only set the constants 
> for keywords and description, though I initially had the instruction 
> "all", thinking this would make the robots index my whole site.
> The site is generating almost-compliant xhtml (if I use the html cleanup 
> extension, it's fully compliant but my 1 flash animation doesn't show), 
> using css, all the navigation is done via text+<a> tags, using realURL, 
> etc.
> Questions: Are there any issues regarding google and typo I should be 
> aware of? Does typo generate a "robots.txt" or do I need to write my own?

Hi Rodrigo,

you have to create your own robots.txt in the root of your site. Instruction 
'all' would make them index your whole site (although it might save you some 
CPU load and some bandwidth if you had 'index, follow' in your metatags and 
if you disallowed indexing of unnecessary folders in your robots.txt in the 

User-agent: *
Disallow: /uploads/

etc. See www.robotstxt.org for recommendations.

Nevertheless, here is what happens. When you submit a site to Google (or it 
gets to it following some link to your site), the initial robot visits, 
checking for instructions, checking if site is 'there', if it's elligible 
for main indexing etc. and adds it to the Big Pappa Crawler ie. main Google 
indexer's queue. Then you wait your turn, depending where one of Google's 
big crawler robots are at the moment in their cyclic runs. While your site 
waits in the queue, a little Google bot comes from time to time seeing if it 
is still there until the big one arrives.

This waiting can last from 2 days to 2 months, depending on where the big 
indexer is when you submit a site in it's cyclic run. I don't have any 
information on the current Googlebot cycle timings, writing you as it used 
to be.

So patience, Big Google Pappa will come crawling back ;)

Boris Senker

: dvotocka design

Graphic Design for Print and Web, Prepress, Website Production, TYPO3 

J. Laurencica 8, 10000 Zagreb, Croatia

More information about the TYPO3-english mailing list