Suzannah Porter's Library tagged Robots → View Popular

Filter:
All
Annotated

12 Oct 08

Article | 10 Ways to Increase Your Site Crawl Rate

Ways to invite google to crawl your site.

www.searchenginejournal.com/...7159 - Preview

Articles SEO Google Robots

- Smarty, Ann. 2008. 10 Ways to Increase Your Site Crawl Rate. Blog. Search Engine Journal. June 24. http://www.searchenginejournal.com/10-ways-to-increase-your-site-crawl-rate/7159/.
  > - on 2008-10-12
Add Sticky Note
Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.
15 more annotations...
- Note that you can’t force Googlebot to visit you more often - what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:
- Update your content often and regularly (and ping Google once you do) - well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
- Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
- Mind your page load time: note that the crawl works on a budget - if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
- Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
- Get more back links from regularly crawled sites.
- Adjust the crawl speed via Google Webmaster tools.
- Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
- Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
- Make sure you have unique title and meta tags for each of your pages.
- Monitor Google crawl rate for your site and see what works and what not:
- access crawl stats via Google Webmaster tools:
- Take advantage of this great Wordpress Plugin that tracks crawl rate for Google, Yahoo and MSN:

11 Oct 08

Article | Insane SEO: Blocking Robots.txt from Being Indexed

Robots.txt, indexing, google, and nofollow.

www.searchenginejournal.com/...6961 - Preview

Articles SEO Robots

- Smarty, Ann. 2008. Insane SEO: Blocking Robots.txt from Being Indexed. Blog. Search Engine Journal. May 28. http://www.searchenginejournal.com/insane-seo-blocking-robotstxt-from-being-indexed/6961/.
  > - on 2008-10-11
Add Sticky Note
Google currently indexes 62,100 robots.txt files. Many of them have a decent PR while others have no backlinks at all (according to Yahoo Site Explorer at least):
6 more annotations...
- The irony is that:
  
  you can’t use robots.txt to block robots.txt (that’s truly insane, as in this case a search engine would be unable to crawl robots.txt file and thus to find out that it is unable to do that);
  you are unable to use meta tags in a robots.txt file;
  you can’t remove the file using Google Webmaster Tools because for that you either need to block it in robots.txt or use meta tags (you are unable to do that) or return 404 header which is also impossible (because it actually exists).
- According to the forum member:
  
  At any rate, this does bring up the crazy question, how can you remove a robots.txt file from Google’s index? If you use robots.txt to block it, that would mean that googlebot should not even request robots.txt - an insane loop. And of course, you don’t use meta tags in a robots.txt file.
  
  Interesting, isn’t it?
- Another board member suggested using an X-Robots-Tag in the HTTP header:
  
  <FilesMatch “robots\.txt”>
  
  Header set X-Robots-Tag “noindex, nofollow”
  
  </FilesMatch>
- The solution looks pretty good and that’s also nice that SEOs started at last seeing value in the X-Robots-Tag which is vaguely used.
  
  Another question is why on Earth you would need to block your robots.txt file from being indexed and ranked (a much easier solution would be removing the file completely). But that is not at all important in this case. The truth remains the same: webmasters should have and be aware of the ways to hide any of their pages from search crawlers or prevent it from appearing in SERPs.
- Barry Schwartz
  on May 28, 2008 at 1:05 pm
  
  We have a good comment on this thread, which we covered on May 15th, from Googler JohnMu:
  
  Check out his comment at http://www.seroundtable.com/archives/017139.html#comment-929618
  
  Ann Smarty
  on May 29, 2008 at 8:31 am
  
  @Barry : I can’t believe I missed it! I do check your blog daily… Thank you for pointing that out to me!
  
  Ok, so two possible solutions:
  
  1/ disallow robots.txt in robots.txt (which won’t prevent the bot to check it);
  
  2/ use an X-Robots-Tag in the HTTP header