Re: Hide webserver contents from search engines / Robots.txt
I spent some time learning all about search optimization few years back (keywords/description meta tags, robots.txt, etc). All of my sites are well visible in google with the description text displayed as I want it. The rating currently is not at its best but it has been in the past. I have always used robots.txt to control what gets crawled.