The TechWyse 'Rise to the Top' Internet Marketing Blog

How To Use The robots.txt File To Improve Results

Have you ever wondered how to hide information from search engines?
What if you don’t want search engines crawling your website administration folder or pages which you have more secure data on? What if you don’t want certain search engines to crawl any of your content at all?

That is what the robots.txt file is for! All you need to do is put it in your root folder and follow a few standard rules and your good to go! The robots.txt file is what you use to tell search robots, also known as Web Wanderers, Crawlers, or Spiders, which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Advantages of Using robots.txt

If you have multiple copy on various pages – maybe one for printing and another for viewing in the browser, chances are high that Search Engines will penalize for duplicate content issue. Using robots.txt this can be avoided.

Important Considerations Before Using robots.txt:

So don’t try to use /robots.txt to hide information or to make the website secure. It;s just about showing the robots which are all doors are opened or closed for crawling.

Creating a robots.txt File

The location of robots.txt is very important. Create a regular text file in notepad called “robots.txt” and upload it to your root directory ( Note that search engines will look for the robots.txt only in the root folder and not in any other location of the website. In other words, put it in the top-level directory of your web server. If no robots.txt exists on the root folder bots index everything they find along the way. Remember to use all lower case for the filename: “robots.txt“, not “Robots.TXT.

The structure of a robots.txt is simple. it is an endless list of user agents and disallowed files and directories.

User-agent:” are search engines’ crawlers or bots and “disallow:” lists the files /directories to be excluded from indexing.

Click here to find a list of robots.

Some Great robots.txt Operators?

1. To exclude all robots from the entire server

User-agent: *

Disallow: /

Here all search engine crawlers are told (using *) not to index any of the pages in the website. “/” means root of the folder.

2. To exclude a single robot

User-agent: Googlebot-Image

Disallow: /

This will disallow Google image bot to crawl your website. These kind of tricks will help save the bandwidth of your host.

3. To exclude all robots from part of the server

User-agent: *
Disallow: /css/
Disallow: /services/internet-marketing.php

The above code will disallow the folder css and file internet-marketing.php under services folder.

Applicable to all bots.

4. Multiple robots giving different access like:

User-agent: *

Disallow: /

User-agent: MSNBot

Disallow: /images/

Disallow: /admin/

Here all bots are disallowed access but MSNBot can access all the folders except images

and admin folder.So the rules of specificity apply, not inheritance.

5. To allow all robots complete access

User-agent: *


There are no advanced features in robots.txt. You can do most of the things with the examples provided above.

Have any others to share?  Did you find this helpful?  Tell us about it below!

Post By Jay (6 Posts)

Website: →


Exit mobile version