Apple's relationship with Google as its search partner is taking a new turn with the company's introduction of visual search, or “Visual Intelligence,” as the company announced it at its “Glowtime” event on Monday. Apple already pays Alphabet roughly $20 billion a year to make Google the default search engine in its Safari browser. iPhone 16 users will be able to access Google's search engine, and its visual search capabilities, with a click of the device's new camera control button.
OpenAI's ChatGPT, which will be accessible via Siri, was also featured as a third-party partner in the demo, allowing users to point their phone camera at their notes in class and receive help understanding a concept or problem with the click of a button.
Apple explained that the Camera Control will let users quickly take photos and record videos, framing their shot by sliding their finger across the button, and adjusting options like zoom, exposure and depth of field in a new camera preview experience. But the button will also give iPhone 16 users access to Apple's new “visual intelligence” search features, and this is where the Google partnership comes in handy.
When first announced, the iPhone 16's camera control resembled a “shutter button” in Apple parlance, but as the event progressed, Apple explained that there was a lot more that this new hardware feature could do. Visual intelligence gives you more than just an easy way to learn about what's in your camera's field of view; it also gives you another way to access third-party services without having to launch a standalone app.
Visual Intelligence, which is essentially a visual search feature similar to Google Lens or Pinterest Lens, was described by Apple as a way for the app to instantly learn about everything you see. To give a few examples, Apple showed how clicking the camera control button can bring up information about a restaurant you see while walking down the street, or how the feature can be used to identify the breed of a dog you see on a walk. The feature can also turn a poster of an event on the wall into a calendar entry with all the details.
Apple's senior vice president of software engineering, Craig Federighi, casually mentioned that the feature could also be used to access Google search.
“Camera Control is also a gateway to third-party tools, making it very quick and easy to leverage expertise in a specific field. So if you see the exact same type of bike you're looking to buy, just tap to search it on Google and find out where you can buy a similar bike,” he said.
Image credit: Apple
In the demo, you tap the camera control button while pointing your iPhone at a bike and a pop-up window appears overlaid on top of the camera's field of view, listing similar options available for purchase. After a grid of matching bike images and descriptions, a small on-screen button appears saying “More Google Results,” indicating you can tap again to continue the Google search.
What Apple didn't explain was when or how pressing the camera control button would seek answers from third-party partners, rather than Apple's built-in services like Apple Maps, which were featured in the restaurant demo. The company also didn't elaborate on how users can control or configure this feature. Instead, Federighi said somewhat vaguely, “Of course, users always have control over when third-party tools are used.”
Reached for comment, a Google spokesperson said the company has nothing to announce about the partnership at this stage. Apple did not respond to a request for comment, but it is understood the deal is part of the two companies' existing relationship and does not involve Google's Gemini AI.
This feature is interesting because it presents a new paradigm for interacting with software and services beyond what Apple packs in the box with your iPhone, and it arrives at a time when the concept of the App Store is starting to become antiquated.
AI technology allows users to ask questions, perform productivity tasks, and get creative with images and videos — things that consumers previously did using apps, but can now do through new interfaces where they talk or text their AI assistants.
Rather than rushing to build a competitor to ChatGPT, Apple is presenting itself as a platform for accessing third-party services, including AI technology, search services, and potentially other providers in the future. Moreover, rather than relying on in-app transactions as a means of generating revenue, Apple can achieve these connections through behind-the-scenes agreements with partners, such as its collaboration with OpenAI on certain AI features.
It also cleverly prevents Apple’s reputation from being damaged if a third party like ChatGPT gets it wrong (as is common with AI) or if a Google search doesn’t produce useful results.