Last week, OpenAI released ChatGPT Search, its long-awaited search product to take on Google. The industry has been preparing for this moment for months, pushing Google to introduce AI-generated answers into its core product earlier this year, creating some embarrassing hallucinations in the process. This incident led many to believe that OpenAI's search engine would indeed become a “Google killer.”
But after using ChatGPT Search as my default search engine for about a day (you can also use OpenAI's extensions), I quickly switched back to Google. OpenAI's search product has been impressive in some ways, offering a glimpse of what AI search interfaces will one day look like. But for now, it's still impractical to use as a daily driver.
ChatGPT Search has sometimes helped us find real-time answers to questions that we would otherwise have had to wade through lots of ads and SEO-optimized articles to search. The end result is a concise answer in a nice format. You'll see links to information sources on the right, along with a heading and a short snippet confirming that the AI-generated text you just read is correct.
OpenAI's answer to your Google search is: (Image credit: Maxwell Zeff/openAI)
However, it often felt impractical for everyday use.
ChatGPT search, in its current form, is unreliable for what users use Google most often: short navigation queries. Queries of less than four words make up the majority of searches on Google. Often these are just a few keywords that will get you to the right web page. These are the types of searches that most people barely even realize they're doing all day long, and Google tends to do this very well.
I'm talking about “Celtics scores,” “cotton socks,” “library hours,” “San Francisco weather,” “cafes near me,” and a bunch of other things that billions of people Google. It's about queries that make it the gateway to the Internet.
Test runs on ChatGPT Search can be very frustrating, and as a result, I've become more conscious of how many keyword searches I'm running per day. I couldn't reliably find information with short queries, and for the first time in years I yearned for Google Search.
Don't get me wrong, Google's quality has declined over the last decade or so. The main reason is that we are flooded with advertising and SEO. Still, ChatGPT Search didn't get me the right answers or web pages, so I kept Google open in a separate window during testing.
Which wins: ChatGPT search or short queries?
I typed in “Nuggets score” to check out the live NBA game between the Denver Nuggets and the Minnesota Timberwolves. ChatGPT reported that the Nuggets were winning when they were actually losing, while also showing the Timberwolves' score 10 points lower than it actually was, according to Google results.
Comparison of ChatGPT search (left) and Google search (right) for NBA live scores (Image credit: Maxwell Zeff/OpenAI)
Another time, I tried Today's Earnings to see which companies were reporting quarterly results on Friday that could affect stock prices. ChatGPT told me that Apple and Amazon reported their results on Friday, but both companies had already done so a day earlier. In other words, he hallucinated and fabricated the information.
In another test, I typed in the name of a tech executive to find their contact information. ChatGPT displayed a summary of the person's Facebook profile and hallucinated a link to a LinkedIn page, which when clicked resulted in an error message.
Another time, I wanted to shop and typed in “baggy denim jeans.” ChatGPT Search explained what baggy denim jeans were in the first place (a definition I didn't need) and suggested I go to Amazon.com to find some nice jeans.
ChatGPT Search for “Baggy Denim Jeans”. (Image credit: maxwell zeff/OpenAI)
I could go on, but do you get the idea? My first day using ChatGPT Search was marked by broken links, hallucinations, and random responses.
It may become a “Google killer” someday, but not today.
This was not a major announcement for OpenAI. Sam Altman, who is known for downplaying his startup's AI capabilities, praised the feature as “really good.” The reason this time is different may have something to do with search being one of the biggest businesses on the internet, and OpenAI's version could pose a real threat to its biggest competitor, Google. .
To be fair, Google Search is a 25-year-old product and ChatGPT Search is brand new. OpenAI said in a blog post that it plans to improve its features based on user feedback in the coming months, making this likely to be a key investment area for the startup.
ChatGPT search is suitable for long questions. (Image credit: Maxwell zeff/Openai)
To its credit, ChatGPT Search is pretty good at answering long-form research questions. Something like, “What is the most diverse American professional sports league?” It's not an easy question to answer with Google, but ChatGPT Search is very good at scraping multiple websites and getting a decent answer in just a few seconds. (Perplexity is also very good at these questions, and its search product has been around for over a year.)
Compared to previous versions of ChatGPT, which already had web access, the search feature feels like a better interface for browsing the web. There are now clearer links to the sources from which ChatGPT obtains information. For news articles, ChatGPT will use media companies with which it has all licensing agreements.
ChatGPT search powered by OpenAI's news partners (Image credit: Maxwell Zeff/Openai)
The problem is that most searches on Google aren't very long questions. To truly replace Google, OpenAI needs to improve on the more actionable, shorter searches that people already do throughout the day.
OpenAI doesn't shy away from the fact that ChatGPT Search struggles with short queries.
“ChatGPT search allows users to ask questions in a more natural way than they have previously done using other search tools,” OpenAI spokesperson Nico Felix said in an emailed statement to TechCrunch. It was observed that there was a tendency for people to start “At the same time, web navigation queries, which tend to be short, are also very common. We plan to improve the experience for these types of queries in the future.”
That said, these short keyword queries have made Google indispensable, and until OpenAI gets them right, Google will remain the mainstay for many people.
There are several reasons why OpenAI struggles with these short queries. The first is that ChatGPT relies on Microsoft Bing. Bing is widely considered to be an inferior engine compared to Google. The second reason is that large language models may not be well suited for these short prompts. LLMs typically require fully documented questions to get effective answers. Perhaps some re-prompting (running short queries as long questions through LLM) may be required before ChatGPT Search can perform such searches properly.
OpenAI just launched its search product, but Perplexity's proprietary AI search tools already serve 100 million search queries every week. Perplexity is also touted as a “Google killer”, but it runs into the same problem with short queries.
At TechCrunch Disrupt earlier this week, Perplexity CEO Aravind Srinivas talked about how people are using his company's product differently compared to Google Search: “The median word count for a Google query is between 2 and 3. For Perplexity, it's around 10 to 11 words. So it's clear that a lot of the use of Perplexity is that people can come and ask questions directly. , Google allows you to type in a few keywords and get instant access to a specific link.”
I think the fact that people aren't using these products for web navigation is causing a bigger problem than OpenAI and Perplexity. This means that ChatGPT Search and Perplexity will not replace Google Search for the task it does best: web navigation.
Instead, these AI products are filling new niches and bringing to the surface information that traditional search would otherwise hide. Don't get me wrong, it has its own value.
Both OpenAI and Perplexity claim to work on improving these short queries. Until then, I don't think any of these products can fully replace Google. If OpenAI wants to replace the gateway to the internet, it needs to create a better gateway.