The robots.txt file is used by the search engine robots that crawl the web and build their search indexes. It is a plain text file specifying directories you want the robots to exclude from their index, however a snooper can easily download this file and see what you are trying to hide from the world (unready content, backend/middleware systems, stats, administration tools). The easiest option is security by obscurity, say for example that you have a /admin directory, place an index.html in there stating “Access Denied” or similar, and use something like index_real.html to access. A more secure approach would be to make use Basic HTTP Authentication.
