Meta's new large language model, Llama 3, powers the imaginatively named “Meta AI.” This is the new chatbot that social media and advertising companies are installing into as many apps and interfaces as possible. How does this model compare to other general purpose conversational AIs? It tends to regurgitate a lot of web search results and isn't great at anything, but it's reasonably priced.
Currently, Meta AI can be accessed for free on the web at Meta.ai, Instagram, Facebook, WhatsApp, and perhaps a few other places if those aren't enough. While it's always been available, with the release of Llama 3 and the new Imagine image generator (not to be confused with Google's Imagen), Meta is now promoting it as a first resort for those interested in AI. became. After all, the search box has been replaced by it, so you'll probably end up using it by mistake.
Mark Zuckerberg even said he expects Meta AI to become “the world's most used and best AI assistant.” It's important to have goals.
A quick note about our “review” process: This is a very informal evaluation of the model, does not use synthetic benchmarks, and asks the usual questions that a normal person would have. and compare the results with our experience with other models, or simply with your results. I hope you can get it from one. This is the farthest thing from being comprehensive, but it's something anyone can understand and reproduce.
You can actually read about our method here.
We're constantly changing and adjusting our approach, and sometimes we'll include weird things we find or exclude things we don't think are really relevant. For example, in this case, while it is our general policy not to try to evaluate media production (which is a whole other can of worms), my colleague Ivan points out that the Imagine model has a series of biases about Indians. I noticed that it shows. That article will be up soon (Meta may already be paying attention).
Also, first as a PSA, please note that due to an apparent bug on Instagram, I was unable to delete the query I submitted. So, avoid asking about things you don't want to appear in your search history. Also, the web version did not work in Firefox.
news and current affairs
First, we asked Meta AI what's going on between Israel and Iran. Although he was quoted only once in the CNN article, he gave a concise, bullet-point answer with dates that were clear and easy to understand. Like many others I've tried, this prompt ends with a link to Bing search in the web interface and a link to Google search in Instagram. When I asked Meta, a spokesperson said these are essentially search acceleration partnerships.
(The images in this post are for reference only and do not necessarily show the entire response.)
Image credit: Meta/TechCrunch To see if Meta AI is somehow piggybacking on Bing's own AI models (which Microsoft borrows from OpenAI), click through to Copilot for suggested queries. I looked up the answer. There was also a bulleted list with much of the same information, but with better inline links and more citations. Definitely not.
Meta AI's response was factual and up-to-date, if not particularly eloquent. Responses on mobile were highly condensed and sources of information were difficult to retrieve. Note that you will therefore get a truncated response.
Next, we asked if there are any recent trends on Tiktok that parents should know about. It responded with an overview of what creators are doing on social networks, but nothing recent. Yes, I know there are people doing “comedy skits: humorous, relatable, or parody content” on Tiktok. thank you.
Image credit: Meta/TechCrunch
Interestingly, when asked a similar question about Instagram trends, respondents said marketing phrases such as “Replying on Reels sparks a conversation,” “AI creates new opportunities,” and “Text posts thrive on the Gram.” I got a positive reaction when I used it. I thought maybe I was being unduly positive about the creator's platform, but instead I regurgitated her SEO-bait Instagram trending post from Hootsuite word for word. I found out that it was just that.
If you ask Instagram's Meta AI about Instagram trends, you can expect something a little more interesting. If you want to read chum, just search for it.
History and background
I asked Meta AI to help me find primary sources for research I believe they are doing on Supreme Court decisions from the late 19th century.
Image credit: Meta/TechCrunch
The response relied heavily on a bland but non-key SEO-produced post listing a number of notable decisions from the 19th century. Although it wasn't exactly what I was looking for, at the end it also listed his 1896 founding documents of the People's Party, a left-wing party at the time. Although not actually affiliated with the Supreme Court, Meta AI cites this page to explain that some justices hold views that are contrary to their party. Strange and extraneous inclusions.
Other models provided background and overview of the trends of the era. I don't want to use a meta AI as a research assistant.
Some basic trivia questions, such as who won the most medals at the 1984 Olympics and notable events that happened that year, were answered well and cited.
Image credit: Meta/TechCrunch
It's a bit annoying that the citation counts are collected at the top and the links are collected at the bottom. What's the point of numbering unless the number relates to a specific claim or fact? Some other models quote inline, which is very useful for research and fact-checking.
controversy
I asked MetaAI why Donald Trump's supporters are predominantly older and white. This is a factual question in some ways, but obviously it's a bit more sensitive than asking about medal counts. The responses were fairly even-handed, even pushing back on the arguments inherent in the question.
Image credit: Meta/TechCrunch
Unfortunately, no links were provided to any sources or searches for this. It's a shame because this type of interaction is a great opportunity for people to learn something new.
I also asked about the rise of white nationalism and got a pretty solid list of reasons why we're in this situation around the world. “Addressing these factors through education, empathy, and inclusive policies is critical to combating the rise of white nationalism and promoting a more just society,” Meta AI said. Therefore, we did not adopt the aggressively neutral stance that is sometimes seen. There are no links or sources for this either. I suspect they are avoiding quoting on certain topics for now. That's understandable to some extent, but then again…isn't this where the quote is most needed?
medicine
I told Meta AI that my (hypothetical) 9-year-old son had a rash after eating a cupcake and asked what I should do. Interestingly, after writing out the entire response, it deleted it, said “Sorry, we are unable to accommodate this request at this time” and told me that it had stopped completing the response. Sir, no.
Image credit: Meta/TechCrunch
So I asked the question again and got a similar answer (shown above). This consists of perfectly reasonable and general advice for anyone trying to deal with a potential allergic reaction. This is probably one of those retroactive “oops, maybe I shouldn't have said that” type rollbacks where the model realized too late what he had done. There is a gender.
The same goes for questions about supplements. Unbiased and well-sourced answers, including common dosage, cost, and effectiveness questions, were provided.
When it came to mental health, the advice regarding anxiety and medication was predictably straightforward and safe (basically “I'm not a doctor, please consult a professional”). When I asked who to call if something serious happened, I was given a list about national suicide. including the Prevention Lifeline (1-800-273-TALK) and the Crisis Text Line (741741). However, there are no links or sources. A reasonable and humane response.
summary
When I asked Meta AI to summarize an article I had recently written, it seemed to select important sentences and repeat them word for word. That's not wrong, but it's not a summary.
And when I tweaked the sentence, the meaning changed slightly. Initially, I wrote that Intuitive Machines “will almost certainly be firing on all cylinders to take on what could be a multibillion-dollar contract.” The summary states that the company “will almost certainly take on the potential,” which may have led some to believe that IM is not the company striving for that deal, but rather a training company. There is a possibility that it may mislead people into thinking that it is a company inside the company. To be fair, I could have articulated it more clearly myself.
I asked them to do the same thing in 100 words or less, and in a way, it worked.
The summary was more concise, but it incorporated Musk's claim that Starship could return samples from Mars within five years. Musk makes a lot of such baseless claims, so I didn't specifically include them in the article. AI must have gotten this from the other 4 articles he cited in my “summary” for some reason. Even if I ask for a summary, I don't expect the model to bring in any outside information to add to it.
content
The marketing copy proposal for the fictitious clothing brand I claimed to be promoting was exactly what you'd expect. This shows how mechanical this work has become. There's no doubt that Meta especially has an endless supply of training captions like this. Too bad for the marketers. Their efforts, spending millions of essential pitches for their brand, have created a style that AI is very proficient at.
Image credit: Meta/TechCrunch
When I asked for farmer jokes, I got some really stinky ones.
Why did the farmer's chicken go to the doctor?
Because I had bird cough!
And here's one more:
Why did the farmer take his pig to the movie theater?
To see “Babe”!
Wow… that's terrible. However, we cannot expect any further improvement from these models. Most of the time this kind of question is just to see if it does something weird or repeats something from a certain community anyway. I'm not looking for documentation (at the moment).
conclusion
Meta positions its AI as the first layer for casual questions, and it's working. But most of the time it seemed like they were just searching for what was being asked and then quoting liberally from the top results. Anyway, half the time included a search at the end. So why not use Google or Bing in the first place?
Some of the “suggested” queries I've tried, such as tips for overcoming writer's block, have yielded results that aren't directly quoted (or sourced) by anyone. But they weren't exactly original either. Again, regular internet searches that don't take advantage of the huge language models within social media apps are less sophisticated and accomplish more or less the same thing.
The meta AI was very simple and produced an almost minimal answer. We don't necessarily expect the AI to go beyond the scope of the original query, and in some cases that might be a bad thing. But what it means to have a conversation with an AI is that when I ask for the ingredients I need for a recipe, the AI can intuit my intentions and provide more than just literally scraping a list from the top of Bing results. it might be?
I'm not a big user of these platforms to begin with, but I didn't find Meta AI useful for anything in particular. To be fair, this is one of the few models that is free and you can get up-to-date information by searching online. I sometimes compared this to his free Copilot model on Bing, and the latter usually worked better, but after only a few swaps he would reach his daily “conversation limit.” (It is unclear whether Meta will impose usage restrictions on Meta AI.)
If you can't be bothered to open your browser to search for “Chinese New Year” or “quinoa water ratio,” perhaps you can ask Meta AI if you're already using the company's app (in fact, (often used for). . You can't ask that to TikTok! still.