Google’s New Googlebot User-Agent Effective Crawling on Smartphone Content
Google announced via Google Webmaster Central Blog that the old user-agent “Googlebot-Mobile” mobile-specific crawler will be retired in 3 – 4 weeks. “Googlebot-Mobile” user agents have been tested since 2011 to improve the mobile search experience for feature phone and smartphone users.
According to Zhijian He, Google’s Smartphone Search Engineer, “We’ve seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they meant to block just feature phone crawling or indexing.” Over the years, as Google developed new mobile crawlers, the company struggled to index mobile content for feature phones versus smartphones.
So now Google has taken action to give webmasters more control by revealing “Googlebot,” the new user-agent built specifically to crawl smartphone content. “Googlebot-Mobile” will continue to crawl content for feature phone content.
Google clarified the different mobile user agents as follows:
The new Googlebot for smartphone user-agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)
AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e
Safari/8536.25 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
The Googlebot-Mobile for smartphones user-agent we will be retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)
AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e
Safari/8536.25 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)
Regular Googlebot user-agent:
Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
The two Googlebot-Mobile user agents for feature phones:
- SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
- DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Google specified this critical implication of the user-agent update, “The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile.”
The update should affect less than 0.001% of URLs based on a Google internal analysis. As the update rolls out, webmasters can test sites; He says, “Using the Fetch as Google feature in Webmaster Tools, and they have full access to the full list of existing crawlers in the Help Center.”
