SEO Inc. Blog

To Robots.txt or not to Robots.txt (that is the question)

There are many a blog post out there detailing the advantages and features of robots.txt files, but does your site even need one? And does it have any impact on search engine optimization strategy.

Robots.txt is used to block search engine spiders from crawling certain pages or directories. On most sites private information is already blocked because it is delivered using SSL, so there is no point to having a robots.txt file.

But Steve you say, Google recommends using a robots.txt file to “tell the crawlers which directories can or cannot be crawled.” Well sure, but if you have no problem with Google-bot crawling everything, there is no need for the robots.txt file, right? Why tell it to do what it is already going to try to do? — Crawl everything!

There are really only two cases in which I recommend using a robots.txt file. The first case is (and I think the Google Webmaster guidelines says it best) “to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.” The other situation where there is the need for a robots.txt file is that if you cannot verify your sites XML sitemap for some reason (that should really be fixed), you can use robots.txt to specify its location on your server.

Stevecrest – out!

About Garry Grant

Garry Grant is a veteran expert in search engine optimization and the digital marketing industry. With nearly 20 years of experience, Garry has successfully built a multi-service operation at SEO, Inc., developing proprietary technologies through complex strategic solutions. He has extensive experience in key initiatives and operational responsibilities grounded in information technology and performance management.

Garry’s expertise and esteemed reputation, coupled with SEO Inc.’s impressive client success record has earned him such accolades as Entrepreneur Magazine's 2005 Hot List for the Hottest Internet Property, Inc. 500 2007 Honorary award for Fastest Growing Private Company in America, an Inc. 500 top 50 Company in San Diego, and interviews with The New York Times, The Wall Street Journal, WIRED, Entrepreneur and The Huffington Post.

Garry Grant began his online career in 1993 creating strategic Web and e-business solutions for, The Rush Limbaugh Show, Premiere Radio Networks, Clear Channel Communications, EarthLink and Artisan Motion Pictures. Today, Garry and SEO Inc.’s highly skilled digital strategists develop proprietary technology and strategic digital marketing direction for Fortune 500 companies including, SC Johnson, McAfee,, Inc Magazine, IGN, Tacorri, LPL Financial, National Kidney Foundation, G4 TV, Fuel TV and Sony, just to name a few.