In order to avoid unwanted content in the search yicrosoft directory results may order Webmaster indices, spiders not to explore certain files or directories default robots.txt at the root of the field. In addition, can be explicitly excluded a page of a search engine with a database of meta tag for robots. If a search engine visited a website, robots.txt located in the root directory is the first file will be explored. The robots.txt file is then analyzed, and the use of robots, the pages are not explored. The crawlers search engines May, a cached copy of this file, it can sometimes be an exploration of pages webmaster does not want explored. Generally unable to login pages crawled Special includes sites such as baskets and user specific content such as results of internal research. In March 2007, Google has warned webmasters, to prevent indexing of pinoy money talk research results internally, as these sites are considered spam research.