Google Photos is bringing AI to life with the release of Ask Photos, an experimental feature that leverages Google's Gemini AI model. Launching later this summer, the new feature will allow users to search their entire Google Photos collection using natural language queries that leverage AI's understanding of a photo's content and other metadata.
Before users could search for specific people, places, or things in photos, thanks to natural language processing, AI upgrades can eliminate manual search processes and make relevant content more intuitive. will be discoverable, Google announced Tuesday at its annual Google I. /O 2024 Developer Conference.
For example, instead of searching for something specific in a photo, such as the Eiffel Tower, you can now ask AI to do something more complex, such as finding the best photo from each national park you visit. The AI uses a variety of signals to determine what makes a photo the “best” of a given set, including lighting, blur, and undistorted background. We then combine that with the geolocation of a set of photos or dates to get only images taken in US national parks.
Image credit: Google
This feature builds on Google Photos' recently launched Photo Stacks, which groups near-duplicate photos and uses AI to highlight the best photos in the group. Like Photo Stacks, its purpose is to help people find the photos they need as their digital collections grow. According to Google, more than 6 billion images are uploaded to Google Photos every day, which gives you an idea of how big it is.
Additionally, the “Ask a Photo” feature allows users to ask questions and get other types of helpful answers. Users can not only ask for the best photos from vacations and other groups, but also ask questions that require a human-like understanding of what's in the photos.
For example, if a parent asks Google Photos what themes have been used for their child's last four birthday parties, Google Photos will provide a quick answer along with photos and videos about the previous mermaid, princess, and unicorn themes. is returned.
Image credit: Google
This type of query is possible because Google Photos understands not only the keywords you enter, but also natural language concepts, such as “themed birthday parties.” You can also leverage the multimodal capabilities of AI to understand if there is text within the photo that may be relevant to your query.
Another example, demonstrated to the press by CEO Sundar Pichai ahead of today's Google I/O developer conference, shows a user asking the AI to show them their child's swimming progress. Ta. The AI has packaged photo and video highlights of the child swimming over time.
Another new feature uses search to find answers from text in photos. That way, you can take a photo of something you want to remember later, like a license plate or passport number, and ask the AI to retrieve that information when you need it.
If the AI gets something wrong and you correct it (for example, flagging a photo that isn't a birthday party photo or a photo you don't want highlighted during a vacation), the AI will remember your response and It will improve over time. This means that the longer you interact with the AI, the more personalized it becomes to you.
When you find a photo you're ready to share, AI helps you draft a caption that summarizes the photo's content. However, for now this is a basic overview and does not offer the option to choose from different styles. (However, given that it uses Gemini under the hood, a cleverly crafted prompt might work to return a certain style if you try it.)
Google says it will put guardrails in place to not respond in certain cases (you probably wouldn't ask an AI for “best nudity”, right?). Also, potentially offensive content was not included when training the model. However, this feature is starting as an experiment and may require additional controls over time as Google adapts to how users use it.
The Ask Photos feature will initially be supported in English in the US before being rolled out to more markets. Additionally, it is currently only a text-based feature, similar to asking an AI chatbot a question. However, over time, it may become more deeply integrated with Gemini running on the device, similar to Android.
The company says that users' personal data in Google Photos will not be used for advertising. Google also says, “Humans do not review her AI conversations or personal data in Ask Photos, except in rare cases to address abuse or harm.” People's personal data in Google Photos is also not used to train other generative AI products, such as Gemini.