Official Google Blog: Controlling how search engines access and index your web... - The Diigo Meta page

• 06 Jun 13
  

  simspace

  seo webdev tutorial article google

• 06 Dec 10
  

  diigouser2

  "Fine-grained control
  In addition to the robots.txt file -- which allows you to concisely specify instructions for a large number of files on your web site -- you can use the robots META tag for fine-grain control over individual pages on your site. To implement this, simply add specific META tags to HTML pages to control how each individual page is indexed. Together, robots.txt and META tags give you the flexibility to express complex access policies relatively easily."

  robot webdev index meta

• 23 Sep 08
  

  Simon Hardaker

  google

• 19 Sep 08
  

  Kat Kreugar

  bookmarks search seo robots web blog tutorial

• 18 Jul 08
  

  Mats Hernvall

  google search webdesign robots.txt searchengine

• 21 Mar 08
  

  Ramon Tayag

  Search google searchengine

• 14 Feb 08
  

  elle m

  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and no

  google robots search robots.txt

• 08 Jan 08
  

  aanaedwards

  computers del.icio.us electronics imported internet marketplace search-engine

• 20 Nov 07
  

  google seo toRead HowTo

• 10 Oct 07
  

  mario64

  Controlling how search engines access and index your website

  google robots seo

• 05 Oct 07
  

  losamorales g

  articles del.icio.us development google howto imported robots.txt search searchengine seo web webdesign

• 05 Sep 07
  

  google

• 29 Apr 07
  

  ten grrl

  searchengine web delicious

• 08 Mar 07
  

  Brennan O'Keefe

  Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want.

  howto webdesign reference delicious

• 

  Wayne

  imported:del.icio.us searching seo webdesign

• 

  Ethan Gardner

  With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but it

  robots.txt seo del.icio.us

• 27 Feb 07
  

  Stefan

  html programming webdesign

• 26 Feb 07
  

  Jon Leirdal

  Tips til hvordan sette opp sider og robots.txt slik at bare det du ønsker blir indeksert av google

  google search cx portal robots.txt webdesign searchengine web imported

• 25 Feb 07
  

  Matthew Heinrich

  webdev search

• 

  Richard Kendall

  Google search seo robots.txt robots webdev indexing security

• 24 Feb 07
  

  Matteo Campofiorito

  To implement this, simply add specific META tags to HTML pages to control how each individual page is

  Google seo

• 23 Feb 07
  

  How do Google and search engines use the Robots Exclusion Protocol to index your site? See also
  part 2
  
  
  

  google seo delicious_import

• 06 Feb 07
  

  Moultrie Creek

  robots.txt seo webdev

  • What does robots.txt do?
    

    The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams)

    Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.
    
    In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.
    
    However, you may have a few pages on your site you don't want in Google's index. For example, you might have a directory that contains internal logs, or you may have news articles that require payment to access. You can exclude pages from Google's crawler by creating a text file called robots.txt and placing it in the root directory. The robots.txt file contains a list of the pages that search engines shouldn't access. Creating a robots.txt is straightforward and it allows you a sophisticated level of control over how search engines can access your web site.
• 05 Feb 07
  

  Dave Kiihne

  SEO webdesign searchengine howto optimization deliciousImport

• 

  claudedechauny

  google how-to indexation robots.txt comprendre

• 

  Betty Sixtysix

  robots

  robots.txt

• 

  mr bones

  robots

  robots.txt

• 31 Jan 07
  

  kzutter23

  webdesign Google SEO

• 

  Richard Klein

  google seo robots.txt

• 30 Jan 07
  

  Tudor J

  Controlling how search engines access and index your website

  google SEO webdesign searchengine configuration development tips internet Website optimization tools tool fromDELICIOUS

• 

  t t

  Internet Search Engines Technology Hardware MKT Teaching

  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in
  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in
  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in
  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in
  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in
  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in
  • Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.  The key is a simple file called robots.txt that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With robots.txt you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.  What does robots.txt do?  The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web. (with profound apologies to Douglas Adams) Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.  In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.  However, you may have a few pages on your site you don't want in

  5 more annotations...

• 

  Andy Coulson

  article google linux Reference software tips tools web web2.0 webdesign website work searchengine Search seo robots.txt

• 

  Rebecca Weger

  internet

• 

  Jaxton Dragoon

  articles tool Tools blogging toread blogs Web todo tips technology Software Security Search robots Robot reading privacy optimization Lists HTML File Engine Work webdev development webdesign web2.0 Computers Google seo robots.txt searchEngine delicious

• 

  Ruben Quijada

  Explicación de como controlar la manera en que los buscadores actuales indexan los sitios.

  blogs web2.0 utilidades windows mac linux

• 

  Ruben Q

  Explicación de como controlar la manera en que los buscadores actuales indexan los sitios.

  blogs web2.0 utilidades windows mac linux

• 29 Jan 07
  

  Brian Epstein

  I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and no

  websites search

• 

  klotfi

  searchEngine webmaster

• 

  Allison C

  WebDesign

• 

  Rich G

  SEO

• 

  oronmiz

  google search seo robots.txt blog

• 

  Seth Miller

  google search seo robots.txt blog todo

• 

  Jason Anderson

  google seo blogs

• 

  gailst

  google searchengine webdev

• 

  Abiel Guerra

  Controlling how search engines access and index your website

  Controlling how search engines access and index your website 4x4

  • Controlling how search engines access and index your website
  • Controlling how search engines access and index your website
• 

  cinefilo

  seo google articulo ingles

• 

  Chrys R

  robots.txt google search howto seo

• 

  Eric Bouchut

  Controlling how search engines access and index your website

  SearchEngine search indexation reference web robots

• 28 Jan 07
  

  webdevelopment seo

• 

  google search howto reference tips article

• 

  Andrea Riezzo

  google search seo robots.txt web searchengine webdesign

• 

  Ariadne Gomes

  google seo robots.txt delicious

• 

  maurizio savoni
• 

  Isaac Pigott

  Controlling how search engines access and index your website 1/26/2007 11:36:00 AM Posted by Dan Crow, Product Manager I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site own

  Bookmarks google search website development optimization reference tips searchengine seo robots.txt webdesign web delicious

• 

  FirstN@me L@stN@me

  seo

• 

  Ivan Rivera

  Controlling how search engines access and index your website

  **** google search robots.txt web searchengine robots tips tools

• 

  riveraivan

  Controlling how search engines access and index your website

  **** google search robots.txt web searchengine robots tips tools

• 

  Total Kaos

  google search seo searchengine

• 

  taylor banks

  The good news is that those who publish on the web have a lot of control over which pages should appear in search results.

  google search seo web optimization marketing security privacy #delicious

• 

  Peter

  howto robots.txt

• 

  krysto

  Controlling how search engines access and index your website

  google search seo for:avatarez for:ivokunior delicious

• 27 Jan 07
  

  unbowed

  blogs design development marketing reference search technology tips tools Google robots.txt searchengine how to

• 

  Slow Finger

  web bots Google motori_di_ricerca search robot Googlebot crawlers file

• 

  Érico Andrei

  google search seo robots.txt web robots searchengine webdesign for:andren for:knassif for:shank for:tamosauskas

• 

  haveboard

  meta robots search google

• 

  oneaustin

  blog search

• 

  brianmerkel

  Web

• 

  rchk !

  Official Google Blog: Controlling how search engines access and index your website

  google howto robots.txt search import-1

• 

  blogspot engines google googleblog quickd search webdesigners websites

• 

  epharma

  seo google

• 

  Doug Belshaw

  google search webdesign

• 

  David Sky

  internet

• 

  Gary Burge

  Effective use of robots.txt gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use robots.txt to control access to your content.

  Google search robots.txt tutorials

• 26 Jan 07
  

  Adam McCrea

  google search seo robots

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.