Internet Marketing Tips, Suggestions, & Ramblings

To Crawl or Not to Crawl: Google Webmaster Tools Announces Updated robots.txt Testing Tool

Big news just announced on the Google Webmaster Central Blog: Google has updated their robots.txt tool. They are now allowing websites the opportunity to test their robots.txt files after making changes, and before pushing it live.

You know the robots, right? As a refresher: the robots.txt file essentially tells the search engines which pages on a site to crawl and which pages not to crawl. “Why not crawl all my pages, though?” Well, there are certain instances and URLs associated with nearly every website out there, where there is no point, nor any good reason for a search engine to crawl them. This can be login screen URLs to access your backend, comment sections of a website, and a number of other pages that have no business being crawled or indexed.

That’s pretty much the brass tacks of what a .robots.txt file does for a website. However, it can be a tricky process if you’re not well-versed in how to go about creating this file. In Google’s attempt to make each and every website the “best” it possibly can be, they have announced a brand new tool to help you out! According to Google, “[With this new tool, you] can test new URLs to see whether they’re disallowed for crawling. To guide your way through complicated directives, it will highlight the specific one that led to the final decision. You can make changes in the file and test those too, you’ll just need to upload the new version of the file to your server afterwards to make the changes take effect.”

Google Robots Image For more information regarding this new tool and how it can be of benefit to you and your site, check out the Webmaster Central blog post.   Connect with SEO Inc on Social!

About Garry Grant

Garry Grant is a veteran expert in search engine optimization and the digital marketing industry. With nearly 20 years of experience, Garry has successfully built a multi-service operation at SEO, Inc., developing proprietary technologies through complex strategic solutions. He has extensive experience in key initiatives and operational responsibilities grounded in information technology and performance management.

Garry’s expertise and esteemed reputation, coupled with SEO Inc.’s impressive client success record has earned him such accolades as Entrepreneur Magazine's 2005 Hot List for the Hottest Internet Property, Inc. 500 2007 Honorary award for Fastest Growing Private Company in America, an Inc. 500 top 50 Company in San Diego, and interviews with The New York Times, The Wall Street Journal, WIRED, Entrepreneur and The Huffington Post.

Garry Grant began his online career in 1993 creating strategic Web and e-business solutions for, The Rush Limbaugh Show, Premiere Radio Networks, Clear Channel Communications, EarthLink and Artisan Motion Pictures. Today, Garry and SEO Inc.’s highly skilled digital strategists develop proprietary technology and strategic digital marketing direction for Fortune 500 companies including, SC Johnson, McAfee,, Inc Magazine, IGN, Tacorri, LPL Financial, National Kidney Foundation, G4 TV, Fuel TV and Sony, just to name a few.