Apple Webcrawler scours the internet.

Looks like apple is kick up a notch its search functions to provide spotlight with more to work with.  When you compare Google now vs spotlight/Siri the need for more information becomes obvious.  Apple does have a long way to go to be able to compete against Google the seemingly insurmountable search giant.

Currently it looks like the crawler is targeting just the html and not the css or JavaScript, however there will probable be a shift to grabbing that as well in the future.

How this fairs with Bing must be unnerving as Microsoft’s search engine currently is essential the backbone of what apple relies on for web search back fill.