robot.txt
Posted by Planet Malaysia on April 1, 2006
Just to share about “robot.txt” from [tag]SEO[/tag] [tag]Marketing[/tag] & [tag]Webcrawler[/tag] help.
*What is the robots.txt file*?
The [tag]robots.txt[/tag] file is an ASCII text file that has specific instructions for search engine robots about specific ontent that they are not allowed to index. These instructions are the deciding factor of how a search engine indexes your website’s pages. The universal address of the robots.txt file is:
www.example.com/robots.txt . This is the first file that a robot visits. It picks up instructions for indexing the site content and follows them. This file contains two text fields.
Lets study this example:
User-agent: *
Disallow:
The User-agent field is for specifying robot name for which the access policy follows in the Disallow field. Disallow field specifies URLs which the specified robots have no access to. An example:
User-agent: *
Disallow: /
Here “*” means all robots and “/ ” means all URLs. This is read as, “No access for any search engine to any URL”. Since all URLs are preceded by “/ ” so it bans access to all URLs when nothing follows after “/ “. If partial access has to be given, only the banned URL is specified in the Disallow field. Lets consider this example:
Research access for Googlebot.
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /concepts/new/
Here we see that both the fields have been repeated. Multiple commands can be given for different user agents in different lines. The above commands mean that all user agents are banned access to /concepts/new/ except Googlebot which has full access. Characters following # are ignored up to the line termination as they are considered to be comments.
*Working with the robots.txt file*
*1.* The robots.txt file is always named in all lowercase (e.g. Robots.txt or robots.Txt is incorrect)
*2.* Wildcards are not supported in both the fields. Only * can be used in the User-agent fields’ command syntax because it is a special character denoting “all”. Googlebot is the only robot that supports some wildcard file extensions.
*Ref*: http://www.google.com/webmasters/remove.html
*3.* The robots.txt file is an exclusion file meant for search engine robot reference and not obligatory for a website to function. An empty or absent file simply means that all robots are welcome to index any part of the website.
*4.* Only one file can be maintained per domain.
*5.* Website owners who do not have administrative rights cannot sometimes make a robots.txt file. In such situations, the Robots Meta Tag *6.* Separate lines are required for specifying access to different user agents and Disallow field should not carry more than one command in a line in the robots.txt file. There is no limit to the number of lines though i.e. both the User-agent and Disallow fields can be repeated with different commands any number of times. Blank lines will also not work within a single record set of both the commands.
can be configured to serve the same purpose. Here we must keep in mind that lately, questïons have been raised about robot behavior regarding the Robot Meta Tag. Some robots might skip it altogether. Protocol makes it obligatory for all
robots to start with the robots.txt thereby making it the default starting point for all robots.
*7.* Use lower-case for all robots.txt file content. Please also note that filenames on Unix systems are case sensitive. Be careful about case sensitivity when defining directory or files for Unix hosted domains.
*Advantages of the robots.txt file*
*1.* Protocol demands that all search engine robots start with the robots.txt file. This is the default entry point for robots if the file is present. Specific instructions can be placed on this file to help index your site on the web. Major search engines will nevër violate the Standard for Robots Exclusion.
*2.* The robots.txt file can be used to keep out unwanted robots like email retrievers, image strippers etc.
*3.* The robots.txt file can be used to specify the directories on your server that you don’t want robots to access and/or index e.g. temporary, cgi, and private/back-end directories.
*4.* An absent robots.txt file could generate a 404 errör and redirect the robot to your default 404 errör page. Here it was noticed after careful research that sites that do not have a robots.txt file present and had a customized 404-errör page, would serve the same to the robots. The robot is bound to treat it as the robots.txt file, which can confuse its indexing.
*5.* The robots.txt file is used to direct select robots to relevant pages to be indexed. This especially comes in handy where the site has multilingual content or where the robot is searching for only specific content.
*6.* The need for the robots.txt file was also necessary to stop robots from deluging servers with rapid-fire requests or re-indexing the same files repeatedly. If you have duplicate content on your site for any reason, the same can be prevented from getting indexed. This will help you avoid any duplicate content penalties.
*Disadvantages of the robots.txt file*
Careless handling of directory and filenames can lead hackers to snoop around your site by studying the robots.txt file, as you sometimes may also list filenames and directories that have classified content. This is not a serious issue as deploying some effective security checks to the content in question can take care of it. For example, if you have your traffïc log on your site on a URL such as www.example.com/stats/index.htm which you do not want robots to index, then you would have to add a command to your robots.txt file. As an example:
User-agent: *
Disallow: /stats/
However, it is easy for a snooper to guess what you are trying to hide and simply typing the URL www.example.com/stats in his browser would enable access to the same. This calls for one of the following remedies -
*1. Change file names*:
* Change the stats filename from index.htm to something different, such as stats-new.htm so that your stats URL becomes
www.example.com/stats/stats-new.htm
* Place a simple text file containing the text, “Sorry you are not authorized to view this page”, and save it as index.htm in your /stats/directory.
Enjoy!
Possibly Related Posts:
- TMNET Unifi Really Fast?
- Leklai
- Apple iPhone Don’t Like Me
- Web Hosting Choice
- Finally! Apple iPhone 16G!
Comments
Leave a Reply