As I planned move of my site to Linode, first I needed a place to test. It was easy enough to create test domain and fill it with migrated data but I didn't want Google (or any other bot) to index it. The easiest way to do so was to require authentication. In Apache configuration that can be done using Directory directive:
<Directory "/var/www/html">
AuthType Basic
AuthUserFile "/var/www/.htpasswd"
Require valid-user
</Directory>
However, this also means that my robots.txt
with disallow statements was also forbidden. What I really wanted was to allow only access to robots.txt while forbidding everything else.
A bit of modification later, this is what I came up with:
<Directory "/var/www/html">
AuthType Basic
AuthUserFile "/var/www/.htpasswd"
Require valid-user
<Files "robots.txt">
Allow from all
Satisfy Any
</Files>
</Directory>