Google’s New Googlebot User-Agent Effective Crawling on Smartphone Content
Google announced via Google Webmaster Central Blog that the old user-agent “Googlebot-Mobile” mobile-specific crawler will be retired in the next 3 – 4 weeks. “Googlebot-Mobile” user-agents have been tested since 2011 to improve the mobile search experience for feature phone and smartphone users.
According to Zhijian He, Google’s Smartphone Search Engineer, “We’ve seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they really meant to block just feature phone crawling or indexing.” Over the years, as Google developed new mobile crawlers, the company struggled to index mobile content for feature phones versus smartphones.
So now Google has taken action to give webmasters more control by revealing “Googlebot,” the new user-agent built specifically to crawl content on smartphones. “Googlebot-Mobile” will continue to crawl content for feature phone content.
Google clarified the different mobile user-agents as:
The new Googlebot for smartphones user-agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)
AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e
Safari/8536.25 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
The Googlebot-Mobile for smartphones user-agent we will be retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)
AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e
Safari/8536.25 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Regular Googlebot user-agent:
Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
The two Googlebot-Mobile user-agents for feature phones:
- SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
- DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Google made sure to specify this important implication of the user-agent update, “The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile.”
Based on a Google internal analysis, the update should affect less than 0.001% of URLs. As the update rolls out webmasters can test sites, He says, “Using the Fetch as Google feature in Webmaster Tools, and they have full access to the full list of existing crawlers in the Help Center.”
