[Typo3] Robots.txt and indexed search

[Jody Cleveland]

Hello,

I'll try to clarify what it is I'm trying to do. I've got a site that
has a few hundred pages, and a few hundred pdf files. I've got a few
pages that link to those pdf files, and I've got typo set to index
external files. I've got a handful of those pdf files that I don't want
indexed. So, to satisfy google, and other search engines, I use a
robots.txt file so they aren't indexed.

My question is, how do I get typo to honor that robots.txt file, and
skip indexing those handful of pdf files?

- jody 

> -----Original Message-----
> From: typo3-english-bounces at lists.netfielders.de 
> [mailto:typo3-english-bounces at lists.netfielders.de] On Behalf 
> Of Dmitry Dulepov
> Sent: Friday, June 03, 2005 3:29 AM
> To: typo3-english at lists.netfielders.de
> Subject: Re: [Typo3] Robots.txt and indexed search
> 
> Hi!
> 
> I did not try but you can: add
> 
>  <!--TYPO3SEARCH_begin--><!--TYPO3SEARCH_end--> on those pages.
> 
> This will cause indeved search to "index" empty text between this tag.
> 
> Dmitry.
> 
> Jody Cleveland wrote:
> > Hello,
> > 
> > I'm running typo 3.7 on a redhat 3.0AS server.
> > 
> > I've got the Indexed Search Engine version 1.2.8 running in 
> there too.
> > It's setup to also index external files, which is does 
> well. There are a
> > few files I don't want indexed, which I've got listed in a 
> robots.txt
> > file. How do I get typo's indexed search to honor robots.txt for
> > searching external files?
> > 
> > --
> > Jody Cleveland
> > Computer Support Specialist
> > cleveland at winnefox.org
> _______________________________________________
> Typo3-english mailing list
> Typo3-english at lists.netfielders.de
> http://lists.netfielders.de/cgi-bin/mailman/listinfo/typo3-english
>