Internet Marketing April 29th, 2010
Have you ever wondered how to hide information from search engines?
What if you don’t want search engines crawling your website administration folder or pages which you have more secure data on? What if you don’t want certain search engines to crawl any of your content at all?
That is what the robots.txt file is for! All you need to do is put it in your root folder and follow a few standard rules and your good to go! The robots.txt file is what you use to tell search robots, also known as Web Wanderers, Crawlers, or Spiders, which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
If you have multiple copy on various pages – maybe one for printing and another for viewing in the browser, chances are high that Search Engines will penalize for duplicate content issue. Using robots.txt this can be avoided.
So don’t try to use /robots.txt to hide information or to make the website secure. It;s just about showing the robots which are all doors are opened or closed for crawling.
The location of robots.txt is very important. Create a regular text file in notepad called “robots.txt” and upload it to your root directory (www.your-site.com/robots.txt). Note that search engines will look for the robots.txt only in the root folder and not in any other location of the website. In other words, put it in the top-level directory of your web server. If no robots.txt exists on the root folder bots index everything they find along the way. Remember to use all lower case for the filename: “robots.txt“, not “Robots.TXT.
The structure of a robots.txt is simple. it is an endless list of user agents and disallowed files and directories.
“User-agent:” are search engines’ crawlers or bots and “disallow:” lists the files /directories to be excluded from indexing.
Click here to find a list of robots.
Here all search engine crawlers are told (using *) not to index any of the pages in the website. “/” means root of the folder.
This will disallow Google image bot to crawl your website. These kind of tricks will help save the bandwidth of your host.
The above code will disallow the folder css and file internet-marketing.php under services folder.
Applicable to all bots.
Here all bots are disallowed access but MSNBot can access all the folders except images
and admin folder.So the rules of specificity apply, not inheritance.
There are no advanced features in robots.txt. You can do most of the things with the examples provided above.
Have any others to share? Did you find this helpful? Tell us about it below!