¿ How do they work ?

Basically, it is a visualized program that responds to the configuration entered by the user or webmaster, sending HTTP (HyperText Transport Protocol) commands via the Internet. Web pages are HTML files that contain text, codes and other data that define the content of the website.

The visualizer recuperates the URL (Uniform Resource Locator) of the destination to the present page. It establishes a connection with the remote server, transmits a Get http code to recuperate the HTML file and presents the document on the screen just as we see it in the browser.

¿ What do they do ?

The robots that create indices of the Web use algorithms to generate summaries of documents that are stored in enormous databases. Other robots identify links towards pages that no longer exist, others keep track of statistics of the most popular sites, to name a few examples.

¿ How to control their behavior ?

Thanks to SRE (Standard for Robot Exclusion) protocol, administrators of websites are permitted to indicate instructions toward distinct robots. For example, they can instruct the robots to only revise partial content of a site, to not index pages under construction or to not enter certain determined directories, furthermore speeding up their work and making their tasks more efficient.

Instructions to configure the robots.txt file

In order to configure the instructions we desire for a spider (robot), it is only necessary to create a text file denominated "robots.txt". This file is created with Notepad. This text file must contain commands in a a simple language indicating access policies of a site to these programs.

We will show you here below a simple example of a file that will ask the robot to stay away from /drawings/images and its subdirectories.

# Example robots.txt file
User-agent:*
Disallow: /drawings/images

The first line corresponds to a comment (#). The second line indicates to a robot that it is making a request; in this case, the order is for all robots (*). The third line shows which directory route has had spider access removed.

Give a command to a robot:

# Example robots.txt file

User-agent:*

Disallow: /drawings/images

Disallow: /under_contruction

User-agent:Scooter

Disallow:

For example, the Scooter robot has limited access within the site, but prohibits others from entering the specified routes. [Scooter is the AltaVista robot].

If you would like to prohibit access to any robot whatsoever to your site:

# Example robots.txt file

User-agent: *

Disallow: /