[Typo3] Weirdness with google indexing
typo3 at dvotocka.hr
Thu Jun 16 20:54:15 CEST 2005
"rodrigo cea" <public2 at altamira.cl> wrote in message
news:mailman.1.1118932992.13994.typo3-english at lists.netfielders.de...
> I recently created a website, using typo3.7. My awstats shows the
> googlebot having visited the site 3 times, and only requesting 1 page each
> time, "+1" which means it also requested the robots.txt.
> If I put our address into google, it only shows the front page of the
> site. Putting in exact text from inside pages doesn't give any results.
> I have the "meta extended" extension installed, but only set the constants
> for keywords and description, though I initially had the instruction
> "all", thinking this would make the robots index my whole site.
> The site is generating almost-compliant xhtml (if I use the html cleanup
> extension, it's fully compliant but my 1 flash animation doesn't show),
> using css, all the navigation is done via text+<a> tags, using realURL,
> Questions: Are there any issues regarding google and typo I should be
> aware of? Does typo generate a "robots.txt" or do I need to write my own?
you have to create your own robots.txt in the root of your site. Instruction
'all' would make them index your whole site (although it might save you some
CPU load and some bandwidth if you had 'index, follow' in your metatags and
if you disallowed indexing of unnecessary folders in your robots.txt in the
etc. See www.robotstxt.org for recommendations.
Nevertheless, here is what happens. When you submit a site to Google (or it
gets to it following some link to your site), the initial robot visits,
checking for instructions, checking if site is 'there', if it's elligible
for main indexing etc. and adds it to the Big Pappa Crawler ie. main Google
indexer's queue. Then you wait your turn, depending where one of Google's
big crawler robots are at the moment in their cyclic runs. While your site
waits in the queue, a little Google bot comes from time to time seeing if it
is still there until the big one arrives.
This waiting can last from 2 days to 2 months, depending on where the big
indexer is when you submit a site in it's cyclic run. I don't have any
information on the current Googlebot cycle timings, writing you as it used
So patience, Big Google Pappa will come crawling back ;)
: dvotocka design
Graphic Design for Print and Web, Prepress, Website Production, TYPO3
J. Laurencica 8, 10000 Zagreb, Croatia
More information about the TYPO3-english