This link has been bookmarked by 111 people . It was first bookmarked on 15 Nov 2007, by Mike Howe.
-
08 Feb 19
-
For example,
Disallow: /helpdisallows both/help.htmland/help/index.html, whereasDisallow: /help/would disallow/help/index.htmlbut allow/help.html.
-
-
05 May 16
yumeiki"Although it is not part of this specification, some example code in Perl is available in norobots.pl. It is a bit more "
-
02 Aug 13
-
11 Apr 13
-
25 May 12
-
02 Apr 12
Basem Tahaa page for introducing the crawlers and to get you know about them.
-
04 Feb 12
-
01 Feb 12
-
19 Dec 11
-
30 Sep 11
-
28 Jul 11
-
27 Jun 11
-
27 May 11
-
29 Mar 11
-
26 Mar 11
-
15 Nov 10
-
15 Sep 10
-
26 Jul 10
-
18 Jul 10
-
Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.
-
-
22 Apr 10
-
24 Feb 10
-
24 Nov 09
Rodrigo de OliveiraThis document represents a consensus on 30 June 1994 on the robots mailing list (robots-request@nexor.co.uk), between the majority of robot authors and other people with an interest in robots. It has also been open for discussion on the Technical World Wi
robots.txt robô webcrawler busca engine padrão referência dica bot google importante from_delicious
-
06 Nov 09
-
24 Sep 09
-
is the name of the robot
-
If the value is '
*', the record describes the default access policy for any robot that has not matched any of the other records.
-
-
10 Jun 09
-
05 May 09
-
17 Apr 09
-
09 Apr 09
-
30 Mar 09
-
14 Mar 09
-
13 Mar 09
-
02 Feb 09
-
28 Oct 08
-
14 Sep 08
-
12 Sep 08
-
14 Jul 08
-
08 Jul 08
-
03 Jul 08
-
01 Jul 08
-
14 Jun 08
-
09 May 08
-
04 May 08
-
22 Apr 08
-
Comments can be included in file using UNIX bourne shell conventions: the '
#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded.
-
-
04 Apr 08
-
27 Mar 08
-
18 Mar 08
-
28 Jan 08
-
16 Jan 08
-
02 Jan 08
-
01 Jan 08
-
27 Dec 07
-
03 Dec 07
-
20 Nov 07
-
15 Nov 07
-
21 Jul 04
accentcompetencehttp://www.robotstxt.org/wc/norobots.html
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.