Apple's relationship with Google as its search partner has taken a new turn as Apple introduced visual search, or “visual intelligence” as the company dubbed it, at yesterday's iPhone 16 launch event. Apple already pays Alphabet roughly $20 billion per year to make Google the default search engine on its Safari browser. iPhone 16 users will now be able to access Google's search engine and its visual search capabilities with just a click of the device's new camera control button.
OpenAI's ChatGPT, which will be accessible via Siri, was also featured as a third-party partner in the demo, allowing users to point their phone camera at their notes in class and receive help understanding a concept or problem with the click of a button.
Apple explained that the Camera Control lets users quickly take photos or record videos, and then slide their finger across the button to frame a shot or adjust options like zoom, exposure, and depth of field with a new Camera Preview experience. But the button also gives iPhone 16 users access to Apple's new “visual intelligence” search features, and this is where its partnership with Google comes in handy.
When first announced, the iPhone 16's camera control resembled a “shutter button” in Apple jargon, but as the event progressed, Apple explained that this new hardware feature could do a lot more. Visual intelligence not only makes it easier for users to know what's in the camera's field of view, but it also gives them another way to access third-party services without having to launch a standalone app.
Essentially a visual search feature similar to Google Lens or Pinterest Lens, Visual Intelligence is described by Apple as “a way for the app to instantly learn about everything it sees.” Apple gave a few examples, showing how a user could click a camera control button to see information about a restaurant they saw while walking around town, or how the feature could be used to identify the breed of a dog they saw on a walk. The feature can also turn an event poster on the wall into a calendar entry with all the details.
Apple's senior vice president of software engineering, Craig Federighi, casually mentioned that the feature could also be used to access Google search.
“Camera Control is also a gateway to third-party tools, making it very quick and easy to leverage expertise in a specific field. So if you see the exact same type of bike you're looking to buy, just tap to search it on Google and find out where you can buy a similar bike,” he said.
Image credit: Apple
In the demo, you tap the camera control button while pointing your iPhone at a bike and a pop-up window appears overlaid on top of the camera's field of view, listing similar options available for purchase. After a grid of matching bike images and descriptions, a small on-screen button appears saying “More Google Results,” indicating you can tap again to continue the Google search.
What Apple didn't explain was when or how pressing the camera control button would seek answers from third-party partners, rather than Apple's built-in services like Apple Maps, which were featured in the restaurant demo. The company also didn't elaborate on how users can control or configure this feature. Instead, Federighi said somewhat vaguely, “Of course, users always have control over when third-party tools are used.”
Asked for comment, a Google spokesman said the company had nothing to announce about the partnership at this time. Apple did not respond to a request for comment.
This feature is interesting because it presents a new paradigm for interacting with software and services beyond what Apple packs in the box with your iPhone, and it arrives at a time when the concept of the App Store is starting to become antiquated.
AI technology allows users to ask questions, perform productivity tasks, and get creative with images and videos — things that consumers previously did using apps, but can now do through new interfaces where they talk or text their AI assistants.
Rather than rushing to build a competitor to ChatGPT, Apple is presenting itself as a platform for accessing third-party services, including AI technology, search services, and potentially other providers in the future. Moreover, rather than relying on in-app transactions as a means of generating revenue, Apple can achieve these connections through behind-the-scenes agreements with partners, such as its collaboration with OpenAI on certain AI features.
It also cleverly prevents Apple’s reputation from being damaged if a third party like ChatGPT gets it wrong (as is common with AI) or if a Google search doesn’t produce useful results.