# /robots.txt file for http://www.boystown.com.au # Hash preceeds a comment # To exclude certain pages specifically add this meta tag into the head # # All robots will spider the domain #User-agent: * #Disallow: # To disallow robots access to certain directories add # the directories on a separate line starting with # Disallow: # Disallow directory(s)