[TYPO3-german] robots.txt Weiterleitung
usenet at schani.com
usenet at schani.com
Wed Oct 28 11:38:50 CET 2015
Hallo,
ich beobachte bei einigen meiner TYPO3 6.2.15 Installationen in den
Apache Log Files Einträge wie /robots.txt 500
[Wed Oct 28 09:41:10 2015] [error] [client 66.249.78.136] Request
exceeded the limit of 10 internal redirects due to probable
configuration error. Use 'LimitInternalRecursion' to increase the limit
if necessary. Use 'LogLevel debug' to get a backtrace.
Ich habe dann natürlich das File geprüft, neu erstellt und Rechte
geprüft. Dann hab ich noch den Apachen geprüft, der aber keine
Einschränkungen auf den vhost hat. Wenn da was wäre dann würde die Seite
ja auch Probleme machen. Alles soweit OK. Es lässt sich auch über den
Browser öffen: www.beispieldomain.de/robots.txt
Über die Google Search Console kann ich es auch prüfen und Google ist
damit zufrieden.
Trotzdem stehen in den Logfiles ständig die 500er Fehler drin.
Jetzt habe ich das robots.txt File mal gelöscht, da sowieso alles
freigegeben ist und nichts gesperrt werden sollte.
Trotzdem sind die 500er Fehler noch da, die von 66.249.x.x, also von
Google kommen.
Natürlich nutze ich RealURL und auch das .htaccess File das aus dem
6.2.15 Core stammt, in meiner Root Ebene der Domain. Soweit funktioniert
es auch.
Ich mache mir nur Sorgen das die 500er Meldung zu Abwertungen bei den
Suchmaschinen führt.
Hab Ihr da Erfahrungen?
Es betrifft auf dem Server nur die TYPO3 Installationen. Bei WP und
eigenen PHP Seiten tritt das nicht auf.
Besten Dank für Tipps und Hilfe
Christian
Hier noch das .htaccess File
#####
#
# Example .htaccess file for TYPO3 CMS - for use with Apache Webserver
#
# This file includes settings for the following configuration options:
#
# - Compression
# - Caching
# - MIME types
# - Cross Origin requests
# - Rewriting and Access
# - Miscellaneous
# - PHP optimisation
#
# If you want to use it, you have to copy it to the root folder of your
TYPO3 installation (if its
# not there already) and rename it to '.htaccess'. To make .htaccess
files work, you might need to
# adjust the 'AllowOverride' directive in your Apache configuration file.
#
# IMPORTANT: You may need to change this file depending on your TYPO3
installation!
# Consider adding this file's content to your webserver's
configuration directly for speed improvement
#
# Lots of the options are taken from
https://github.com/h5bp/html5-boilerplate/blob/master/dist/.htaccess
#
####
### Begin: Compression ###
# Compressing resource files will save bandwidth and so improve loading
speed especially for users
# with slower internet connections. TYPO3 can compress the .js and .css
files for you.
# *) Uncomment the following lines and
# *) Set $GLOBALS['TYPO3_CONF_VARS']['BE']['compressionLevel'] = 9 for
the Backend
# *) Set $GLOBALS['TYPO3_CONF_VARS']['FE']['compressionLevel'] = 9
together with the TypoScript properties
# config.compressJs and config.compressCss for GZIP compression of
Frontend JS and CSS files.
<FilesMatch "\.js\.gzip$">
AddType "text/javascript" .gzip
</FilesMatch>
<FilesMatch "\.css\.gzip$">
AddType "text/css" .gzip
</FilesMatch>
AddEncoding gzip .gzip
<IfModule mod_headers.c>
<FilesMatch "\.(js|css|xml|gz|gzip)$">
Header append Vary: Accept-Encoding
</FilesMatch>
</IfModule>
<IfModule mod_deflate.c>
# Force compression for mangled `Accept-Encoding` request headers
<IfModule mod_setenvif.c>
<IfModule mod_headers.c>
SetEnvIfNoCase
^(Accept-EncodXng|X-cept-Encoding|X{15}|~{15}|-{15})$
^((gzip|deflate)\s*,?\s*)+|[X~-]{4,13}$ HAVE_Accept-Encoding
RequestHeader append Accept-Encoding "gzip,deflate"
env=HAVE_Accept-Encoding
</IfModule>
</IfModule>
# Compress all output labeled with one of the following media types
<IfModule mod_filter.c>
AddOutputFilterByType DEFLATE application/atom+xml \
application/javascript \
application/json \
application/ld+json \
application/manifest+json \
application/rdf+xml \
application/rss+xml \
application/schema+json \
application/vnd.geo+json \
application/vnd.ms-fontobject \
application/x-font-ttf \
application/x-javascript \
application/x-web-app-manifest+json \
application/xhtml+xml \
application/xml \
font/eot \
font/opentype \
image/bmp \
image/svg+xml \
image/vnd.microsoft.icon \
image/x-icon \
text/cache-manifest \
text/css \
text/html \
text/javascript \
text/plain \
text/vcard \
text/vnd.rim.location.xloc \
text/vtt \
text/x-component \
text/x-cross-domain-policy \
text/xml
</IfModule>
<IfModule mod_mime.c>
AddEncoding gzip svgz
</IfModule>
</IfModule>
### End: Compression ###
### Begin: Browser caching of resource files ###
# This affects Frontend and Backend and increases performance.
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 1
month"
ExpiresByType text/css "access plus 1
year"
ExpiresByType application/json "access plus 0
seconds"
ExpiresByType application/ld+json "access plus 0
seconds"
ExpiresByType application/schema+json "access plus 0
seconds"
ExpiresByType application/vnd.geo+json "access plus 0
seconds"
ExpiresByType application/xml "access plus 0
seconds"
ExpiresByType text/xml "access plus 0
seconds"
ExpiresByType image/vnd.microsoft.icon "access plus 1
week"
ExpiresByType image/x-icon "access plus 1
week"
ExpiresByType text/x-component "access plus 1
month"
ExpiresByType text/html "access plus 0
seconds"
ExpiresByType application/javascript "access plus 1
year"
ExpiresByType application/x-javascript "access plus 1
year"
ExpiresByType text/javascript "access plus 1
year"
ExpiresByType application/manifest+json "access plus 1
week"
ExpiresByType application/x-web-app-manifest+json "access plus 0
seconds"
ExpiresByType text/cache-manifest "access plus 0
seconds"
ExpiresByType audio/ogg "access plus 1
month"
ExpiresByType image/bmp "access plus 1
month"
ExpiresByType image/gif "access plus 1
month"
ExpiresByType image/jpeg "access plus 1
month"
ExpiresByType image/png "access plus 1
month"
ExpiresByType image/svg+xml "access plus 1
month"
ExpiresByType image/webp "access plus 1
month"
ExpiresByType video/mp4 "access plus 1
month"
ExpiresByType video/ogg "access plus 1
month"
ExpiresByType video/webm "access plus 1
month"
ExpiresByType application/atom+xml "access plus 1
hour"
ExpiresByType application/rdf+xml "access plus 1
hour"
ExpiresByType application/rss+xml "access plus 1
hour"
ExpiresByType application/vnd.ms-fontobject "access plus 1
month"
ExpiresByType font/eot "access plus 1
month"
ExpiresByType font/opentype "access plus 1
month"
ExpiresByType application/x-font-ttf "access plus 1
month"
ExpiresByType application/font-woff "access plus 1
month"
ExpiresByType application/x-font-woff "access plus 1
month"
ExpiresByType font/woff "access plus 1
month"
ExpiresByType application/font-woff2 "access plus 1
month"
ExpiresByType text/x-cross-domain-policy "access plus 1
week"
</IfModule>
### End: Browser caching of resource files ###
### Begin: MIME types ###
# Proper MIME types for all files
<IfModule mod_mime.c>
# Data interchange
AddType application/atom+xml atom
AddType application/json json map topojson
AddType application/ld+json jsonld
AddType application/rss+xml rss
AddType application/vnd.geo+json geojson
AddType application/xml rdf xml
# JavaScript
AddType application/javascript js
# Manifest files
AddType application/manifest+json webmanifest
AddType application/x-web-app-manifest+json webapp
AddType text/cache-manifest appcache
# Media files
AddType audio/mp4 f4a f4b m4a
AddType audio/ogg oga ogg opus
AddType image/bmp bmp
AddType image/svg+xml svg svgz
AddType image/webp webp
AddType video/mp4 f4v f4p m4v mp4
AddType video/ogg ogv
AddType video/webm webm
AddType video/x-flv flv
AddType image/x-icon cur ico
# Web fonts
AddType application/font-woff woff
AddType application/font-woff2 woff2
AddType application/vnd.ms-fontobject eot
AddType application/x-font-ttf ttc ttf
AddType font/opentype otf
# Other
AddType application/octet-stream safariextz
AddType application/x-bb-appworld bbaw
AddType application/x-chrome-extension crx
AddType application/x-opera-extension oex
AddType application/x-xpinstall xpi
AddType text/vcard vcard vcf
AddType text/vnd.rim.location.xloc xloc
AddType text/vtt vtt
AddType text/x-component htc
</IfModule>
# UTF-8 encoding
AddDefaultCharset utf-8
<IfModule mod_mime.c>
AddCharset utf-8 .atom .css .js .json .manifest .rdf .rss .vtt
.webapp .webmanifest .xml
</IfModule>
### End: MIME types ###
### Begin: Cross Origin ###
# Send the CORS header for images when browsers request it.
<IfModule mod_setenvif.c>
<IfModule mod_headers.c>
<FilesMatch "\.(bmp|cur|gif|ico|jpe?g|png|svgz?|webp)$">
SetEnvIf Origin ":" IS_CORS
Header set Access-Control-Allow-Origin "*" env=IS_CORS
</FilesMatch>
</IfModule>
</IfModule>
# Allow cross-origin access to web fonts.
<IfModule mod_headers.c>
<FilesMatch "\.(eot|otf|tt[cf]|woff2?)$">
Header set Access-Control-Allow-Origin "*"
</FilesMatch>
</IfModule>
### End: Cross Origin ###
### Begin: Rewriting and Access ###
# You need rewriting, if you use a URL-Rewriting extension (RealURL,
CoolUri).
<IfModule mod_rewrite.c>
# Enable URL rewriting
RewriteEngine On
# Store the current location in an environment variable CWD to use
# mod_rewrite in .htaccess files without knowing the RewriteBase
RewriteCond $0#%{REQUEST_URI} ([^#]*)#(.*)\1$
RewriteRule ^.*$ - [E=CWD:%2]
# Rules to set ApplicationContext based on hostname
#RewriteCond %{HTTP_HOST} ^dev\.example\.com$
#RewriteRule .? - [E=TYPO3_CONTEXT:Development]
#RewriteCond %{HTTP_HOST} ^staging\.example\.com$
#RewriteRule .? - [E=TYPO3_CONTEXT:Production/Staging]
#RewriteCond %{HTTP_HOST} ^www\.example\.com$
#RewriteRule .? - [E=TYPO3_CONTEXT:Production]
# Rule for versioned static files, configured through:
# - $GLOBALS['TYPO3_CONF_VARS']['BE']['versionNumberInFilename']
# - $GLOBALS['TYPO3_CONF_VARS']['FE']['versionNumberInFilename']
# IMPORTANT: This rule has to be the very first RewriteCond in
order to work!
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)\.(\d+)\.(php|js|css|png|jpg|gif|gzip)$ $1.$3 [L]
# Access block for folders
RewriteRule _(?:recycler|temp)_/ - [F]
RewriteRule fileadmin/templates/.*\.(?:txt|ts)$ - [F]
RewriteRule typo3temp/logs/ - [F]
RewriteRule
(?:typo3conf/ext|typo3/sysext|typo3/ext|typo3/vendor)/[^/]+/(?:Configuration|Resources/Private|Tests?)/
- [F]
# Access block for files or folders starting with a dot
RewriteCond %{SCRIPT_FILENAME} -d [OR]
RewriteCond %{SCRIPT_FILENAME} -f
RewriteRule (?:^|/)\. - [F]
# Stop rewrite processing, if we are in the typo3/ directory or any
other known directory
# NOTE: Add your additional local storages here
###RewriteRule
(?:typo3/|fileadmin/|typo3conf/|typo3temp/|uploads/|favicon\.ico) - [L]
# If the file/symlink/directory does not exist => Redirect to
index.php.
# For httpd.conf, you need to prefix each '%{REQUEST_FILENAME}'
with '%{DOCUMENT_ROOT}'.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule ^.*$ %{ENV:CWD}index.php [QSA,L]
</IfModule>
# Access block for files
<FilesMatch
"(?i:^\.|^#.*#|^(?:ChangeLog|ToDo|Readme|License)(?:\.md|\.txt)?|^composer\.(?:json|lock)|^ext_conf_template\.txt|^ext_typoscript_constants\.txt|^ext_typoscript_setup\.txt|flexform[^.]*\.xml|locallang[^.]*\.(?:xml|xlf)|\.(?:bak|co?nf|cfg|ya?ml|ts|dist|fla|in[ci]|log|sh|sql(?:\..*)?|sw[op]|git.*)|.*(?:~|rc))$">
# # Apache < 2.3
<IfModule !mod_authz_core.c>
Order allow,deny
Deny from all
Satisfy All
</IfModule>
# # Apache ≥ 2.3
<IfModule mod_authz_core.c>
Require all denied
</IfModule>
</FilesMatch>
# Block access to vcs directories
<IfModule mod_alias.c>
RedirectMatch 404 /\.(?:git|svn|hg)/
</IfModule>
### End: Rewriting and Access ###
### Begin: Miscellaneous ###
# 404 error prevention for non-existing redirected folders
Options -MultiViews
# Make sure that directory listings are disabled.
<IfModule mod_autoindex.c>
Options -Indexes
</IfModule>
<IfModule mod_headers.c>
# Force IE to render pages in the highest available mode
Header set X-UA-Compatible "IE=edge"
<FilesMatch
"\.(appcache|crx|css|eot|gif|htc|ico|jpe?g|js|m4a|m4v|manifest|mp4|oex|oga|ogg|ogv|otf|pdf|png|safariextz|svgz?|ttf|vcf|webapp|webm|webp|woff2?|xml|xpi)$">
Header unset X-UA-Compatible
</FilesMatch>
# Reducing MIME type security risks
Header set X-Content-Type-Options "nosniff"
</IfModule>
# ETag removal
<IfModule mod_headers.c>
Header unset ETag
</IfModule>
FileETag None
### End: Miscellaneous ###
# Add your own rules here.
RewriteEngine On
##RewriteRule sitemap.xml$ /index.php?eID=dd_googlesitemap [L,R=301]
RewriteRule ^typo3$ - [L]
RewriteRule ^typo3/.*$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule .* index.php
More information about the TYPO3-german
mailing list