In consideration of the differing capabilities between old-school feature phones and newer-age smartphones, Google is introducing a new user-agent for Googlebot to replace Googlebot-Mobile. Webmasters have tried – and failed – in the past to instruct Google web crawlers to block feature-phone indexing while at the same time allowing smartphone indexing. Programmers ended up blocking all mobile-related indexing instead, causing Google to neglect all of their smartphone-optimized content.
The new Googlebot is designed to allow indexing or crawling on smartphones when the intent is to block only feature-phone indexing or crawling. This will allow webmasters the ability to have more control over content search on smartphones. However, the old Googlebot-Mobile will still be available for feature-phone content.
Clearly, there are different mobile-user agreements in this setup and it is aimed specifically at the difference between smartphones and feature phones, which have very distinct capabilities. Smartphone users have been frustrated because of lack of Google visibility to certain sites and understandably so.
On its official Google Webmaster Central blog, Google delineated its new Googlebot user-agent agreement for smartphones and Googlebot-Mobile. They are:
New Googlebot for smartphones user-agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
The Googlebot-Mobile for smartphones user-agent retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
The regular Googlebot user-agent is: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Feature phones have the following user-agents for Googlebot-Mobile:
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/126.96.36.199.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
It is clear that the new Googlebot designed for smartphones will follow robots.txt, HTTP header directives and robots meta tags for Googlebot rather than Googlebot-Mobile. This will affect only 0.001 percent of URLs. Google also told webmasters to use the Fetch as Google feature in Webmaster Tools.
Whether using a laptop, smartphone, tablet or any other device to access the web, functionality is imperative. Googlebot for smartphones makes the search unified and effective. The use of smartphones to search the Internet is growing dramatically and Google has addressed a major problem associated with mobile Internet search. Finding what you need when you search the web presumes that the web crawler is not blocked in any way. Since more people are using smartphones to do the searches they would normally do from a desktop, this move by Google is much appreciated and applauded.