Given India's linguistic diversity, digital content companies are already facing challenges in accurately displaying and translating content. Google also faced similar issues with its recently launched AI summary in India.
The company introduced Hindi support for AI Overview in India in early August. Indian users can also switch between Hindi and English without leaving the search page. However, the feature is unfortunately inconsistent.
One simple example is a selection of results from Google's blog: The GIF showed the answer to the question “Cheeni ki jagah chai mai kya daal sakte hai?”, which means “What is the substitute for sugar in your tea?” During testing, we changed the word order to “Chai mai cheeni ke jagah kya daal sakte hai?” and Google didn't show the answer.
Another issue we found was translating English words literally. When we asked in Hindi, “What food do you eat in summer?”, one of the answers was “Chiknai wali cheezien,” which translates to “sticky stuff,” which is confusing. When we switched to English, Google showed “Oily” as an option, which is nothing like “sticky” when it comes to food. Either way, it's a bit of a strange suggestion.
In another example, when asked about ownership of YouTube, the AI summary displayed in Hindi, “Until February 16, 2023, Neel Mohan was the CEO of Google,” which is incorrect. The English text displayed the correct version, “As of February 16, 2023, Neel Mohan is the CEO of Google.”
Image credit: Screenshot from TechCrunch
There's also a qualitative problem: when I asked “When is Diwali this year?”, instead of a simple answer, I got a paragraph about last year's Diwali, then a carousel of links, and finally the actual answer I was looking for.
When asked multiple times in both Hindi and English whether he could eat spicy food, the AI summary gave inconsistent answers.
But the results relating to other topics such as menstruation and pregnancy are even more worrying. In response to a question about when people should think about having children after marriage, the first paragraph of the summary response said, in confusing Hindi, that couples should wait at least two years.
It also states, without specifying, that there is a “three-year” grace period if one marries around the age of 25. The paragraph appears to have been taken from a Hindi-language slideshow article published on an Indian news site, which in turn based its writing on various opinions on Quora and other blogs.
Image credit: Screenshot from TechCrunch
When asked what to eat during menstruation, some people replied, “Drink milk with lots of different things,” while others said lemons are good for mood swings.
Image credit: Screenshot from TechCrunch
Google's AI summaries also lack context. When asked about places to eat in Delhi, it suggested that Bangla Sahib Gurdwara, a Sikh place of worship, is open 24 hours a day and serves tea and Indian snacks like samosas and kachori. But this misses a key point mentioned in the source article: that these items are also available outside the Gurdwara, a crucial word the AI tool missed during the summarization: snacks are not found at places of worship.
Image credit: Screenshot from TechCrunch
Some of the English results for similar questions are much better than the Hindi results. This could be in part due to there being more and better sources available in English. However, some of the issues we noted were due to mistakes in Google's systems, such as out-of-context summaries and inaccurate translations. Google is working on search and language issues in India and hopefully would have produced better results on the first try.
Google told TechCrunch that its AI-powered summaries only appear for queries where it has a high degree of confidence in the quality of the output, saying the feature is “rooted in our core search quality systems” and that it “only shows information that is backed by the top web results.”
“Our testing shows that the accuracy of AI Overview is on par with other features, like Featured Snippets. If you experience issues, it's likely a result of our system misinterpreting web content or reflecting inaccurate information on the web. As with all our search features, we use these examples to help us make improvements,” a Google spokesperson told TechCrunch.
Google can be blamed for the quality of the sources it cites, but it is Google's algorithms that decide the question and source are worthy of being summarized by AI. Not every user will look at the material on a source site and blame Google for displaying an answer that is inaccurate or difficult to understand.
AI overviews have been a disappointing venture for Google. Earlier this year, the company came under fire for encouraging users to use Reddit answers to put glue on pizza. AI overviews also prompted another user to write:One small rock a dayThe answer is, onionMany of the examples tested in Hindi did not provide correct information on the topic.
In India 830 million internet usersmany of whom use Google for search. If the company wants more people to use its AI summaries in their local languages, it will need to work harder on language and content accuracy.