AppleBot

Barry Schwartz (May 6, 2015): Apple Confirms Their Web Crawler: Applebot

After much speculation around an Apple Web Crawler, Apple has finally posted a help document confirming the existence of AppleBot, their web crawler. 

Apple said, Applebot is the web crawler for Apple. AppleBot is “used by products including Siri and Spotlight Suggestions,” the company said. [...] Apple says it will respect the customary robots.txt rules and robots meta tags. AppleBot currently originates in the 17.0.0.0 net block. If you do not mention AppleBot in your robots.txt directive, Apple will follow what you mention for Googlebot. So if you want to block AppleBot and GoogleBot, you can just block GoogleBot, but I’d recommend you block each individually.

If you notice unusual AppleBot activity, you can reach Apple about it at Apple-NOC “at” apple.com.
It is unclear if Apple plans on building and competing with Google on search but this is one step closer to that.

http://searchengineland.com/apple-confirms-their-web-crawler-applebot-220423See also: Welt.de (08.05.15): Apple attackiert Google mit eigener Suchmaschine. http://www.welt.de/wirtschaft/article140664446/Apple-attackiert-Google-mit-eigener-Suchmaschine.html