Internet Marketing Tips, Suggestions, & Ramblings

Google Introduces New Googlebot User-Agent for More Effective Crawling On Smartphone Content

Google announced via Google Webmaster Central Blog that the old user-agent “Googlebot-Mobile” mobile-specific crawler will be retired in the next 3 – 4 weeks. “Googlebot-Mobile” user-agents have been tested since 2011 to improve the mobile search experience for feature phone and smartphone users.

According to Zhijian He, Google’s Smartphone Search Engineer, “We’ve seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they really meant to block just feature phone crawling or indexing.” Over the years, as Google developed new mobile crawlers, the company struggled to index mobile content for feature phones versus smartphones.

So now Google has taken action to give webmasters more control by revealing “Googlebot,” the new user-agent built specifically to crawl content on smartphones. “Googlebot-Mobile” will continue to crawl content for feature phone content.

Google clarified the different mobile user-agents as:

The new Googlebot for smartphones user-agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)

AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e

Safari/8536.25 (compatible; Googlebot/2.1;

+http://www.google.com/bot.html)

Photo Source intomobile.com
Image: intomobile.com

The Googlebot-Mobile for smartphones user-agent we will be retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)

AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e

Safari/8536.25 (compatible; Googlebot-Mobile/2.1;

+http://www.google.com/bot.html)

Regular Googlebot user-agent:
Mozilla/5.0 (compatible; Googlebot/2.1;

+http://www.google.com/bot.html)

The two Googlebot-Mobile user-agents for feature phones:

  • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

Google made sure to specify this important implication of the user-agent update, “The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile.”

Based on a Google internal analysis, the update should affect less than 0.001% of URLs. As the update rolls out webmasters can test sites, He says, “Using the Fetch as Google feature in Webmaster Tools, and they have full access to the full list of existing crawlers in the Help Center.”

About Garry Grant

Garry Grant is a veteran expert in search engine optimization and the digital marketing industry. With nearly 20 years of experience, Garry has successfully built a multi-service operation at SEO, Inc., developing proprietary technologies through complex strategic solutions. He has extensive experience in key initiatives and operational responsibilities grounded in information technology and performance management.

Garry’s expertise and esteemed reputation, coupled with SEO Inc.’s impressive client success record has earned him such accolades as Entrepreneur Magazine's 2005 Hot List for the Hottest Internet Property, Inc. 500 2007 Honorary award for Fastest Growing Private Company in America, an Inc. 500 top 50 Company in San Diego, and interviews with The New York Times, The Wall Street Journal, WIRED, Entrepreneur and The Huffington Post.

Garry Grant began his online career in 1993 creating strategic Web and e-business solutions for Homepage.com, The Rush Limbaugh Show, Premiere Radio Networks, Clear Channel Communications, EarthLink and Artisan Motion Pictures. Today, Garry and SEO Inc.’s highly skilled digital strategists develop proprietary technology and strategic digital marketing direction for Fortune 500 companies including, SC Johnson, McAfee, Entrepreneur.com., Inc Magazine, IGN, Tacorri, LPL Financial, National Kidney Foundation, G4 TV, Fuel TV and Sony, just to name a few.