What is a Robots.txt File?
Robots.txt is a plain text file that Search Engines use to see if there are areas within your website which should not be indexed. 
This file must be placed in your 'www' (home directory) in order for a search engine to see it. 

The sample below tells search engines not to index pages in the specified folders. 

---- robots.txt  (contents follow)

User-agent: * 
Disallow: /cgi-bin/
Disallow: /images
Disallow: /secret
Disallow: /members