Close Menu
TechBrunchTechBrunch
  • Home
  • AI
  • Apps
  • Crypto
  • Security
  • Startups
  • TechCrunch
  • Venture

Subscribe to Updates

Subscribe to our newsletter and never miss our latest news

Subscribe my Newsletter for New Posts & tips Let's stay updated!

What's Hot

Klarna CEO and Sutter Hill wins lap after Jony Ive's Openai deal

May 22, 2025

Bluesky begins to check for “notable” users

May 22, 2025

Microsoft says Lumma Password Stealer Malware found on 394,000 Windows PCs

May 22, 2025
Facebook X (Twitter) Instagram
TechBrunchTechBrunch
  • Home
  • AI

    OpenAI seeks to extend human lifespans with the help of longevity startups

    January 17, 2025

    Farewell to the $200 million woolly mammoth and TikTok

    January 17, 2025

    Nord Security founder launches Nexos.ai to help enterprises move AI projects from pilot to production

    January 17, 2025

    Data proves it remains difficult for startups to raise capital, even though VCs invested $75 billion in the fourth quarter

    January 16, 2025

    Apple suspends AI notification summaries for news after generating false alerts

    January 16, 2025
  • Apps

    Bluesky begins to check for “notable” users

    May 22, 2025

    Mozilla shuts down its Read-It-Later app pocket

    May 22, 2025

    Opening a Social Web Browser Surf makes it easy for anyone to create custom feeds

    May 22, 2025

    Anthropic's new Claude4 AI model can be inferred in many steps

    May 22, 2025

    Strava buys athletic training app – First Runna, and now Breakaway

    May 22, 2025
  • Crypto

    Starting from up to $900 from Ticep, 90% off +1 in 2025

    May 22, 2025

    Early savings for 2025 will end on May 25th

    May 21, 2025

    Coinbase says its data breach will affect at least 69,000 customers

    May 21, 2025

    There are 6 days to save $900 to destroy 2025 tickets

    May 20, 2025

    Save $900 to destroy 2025 tickets before prices rise on May 25th

    May 19, 2025
  • Security

    Microsoft says Lumma Password Stealer Malware found on 394,000 Windows PCs

    May 22, 2025

    Signal's new Windows update prevents the system from capturing screenshots of chat

    May 22, 2025

    Wyden: AT&T, T-Mobile and Verizon did not inform senators of surveillance requests

    May 21, 2025

    US students agree to plead guilty to hacking affecting tens of millions of students

    May 21, 2025

    The people in Elon Musk’s DOGE universe

    May 20, 2025
  • Startups

    7 days left: Founders and VCs save over $300 on all stage passes

    March 24, 2025

    AI chip startup Furiosaai reportedly rejecting $800 million acquisition offer from Meta

    March 24, 2025

    20 Hottest Open Source Startups of 2024

    March 22, 2025

    Andrill may build a weapons factory in the UK

    March 21, 2025

    Startup Weekly: Wiz bets paid off at M&A Rich Week

    March 21, 2025
  • TechCrunch

    OpenSea takes a long-term view with a focus on UX despite NFT sales remaining low

    February 8, 2024

    AI will save software companies' growth dreams

    February 8, 2024

    B2B and B2C are not about who buys, but how you sell

    February 5, 2024

    It's time for venture capital to break away from fast fashion

    February 3, 2024

    a16z's Chris Dixon believes it's time to focus on blockchain use cases rather than speculation

    February 2, 2024
  • Venture

    Klarna CEO and Sutter Hill wins lap after Jony Ive's Openai deal

    May 22, 2025

    Wild story of how Moxxie-led Intestinal Toilet Startup Sloan was registered as a gut toilet startup throne

    May 22, 2025

    Submitted submission raises $17 million to automate tax preparation dr voyages

    May 21, 2025

    In a busy VC landscape, Elizabeth Weil's graffiti venture shows that networks are still important

    May 21, 2025

    A comprehensive list of 2025 tech layoffs

    May 21, 2025
TechBrunchTechBrunch

AGI tests are close to resolution, but may have flaws

TechBrunchBy TechBrunchDecember 10, 20244 Mins Read
Facebook Twitter Pinterest Telegram LinkedIn Tumblr WhatsApp Email
Share
Facebook Twitter LinkedIn Pinterest Telegram Email


A well-known test of artificial general intelligence (AGI) is nearing resolution. However, the test's creators argue that this is not a genuine research breakthrough, but rather indicates a flaw in the test's design.

In 2019, Francois Chollet, a leading figure in the AI ​​world, introduced the ARC-AGI benchmark, which stands for “Abstract and Reasoning Corpus for Artificial General Intelligence.” ARC-AGI, designed to assess whether an AI system can effectively learn new skills on data other than the data used to train it, remains the only AI test that measures progress toward general intelligence. Francois argues (although other tests have been proposed as well).

Until this year, the best-performing AIs could only solve just under a third of ARC-AGI's tasks. Chollet criticizes the industry's focus on large-scale language models (LLMs), which he believes lack real “reasoning” capabilities.

“LLM is completely reliant on memorization, which makes it difficult to generalize,” he said in a series of posts about X in February. “They analyze anything that isn’t included in the training data.”

As Chollet points out, the LLM is a statistical machine. They have been trained on many examples and have learned patterns in those examples, such as “to whom” is typically placed before “may be of concern” in an email. make predictions.

Chollet argues that while LLMs may be able to memorize “inference patterns,” they are unlikely to be able to generate “new inferences” based on new situations. “If you have to train on many examples of a pattern, even implicitly, in order to learn a reusable expression, you end up memorizing it,” Chollet argued in another post.

To encourage research beyond the LLM, Chollet and Zapier co-founder Mike Knoop launched a $1 million contest in June to build an open source AI that can beat ARC-AGI. Ta. Out of 17,789 entries, the highest score was 55.5%. While this was short of the 85% “human level” threshold needed to win, it was about 20% higher than the highest scorer in 2023.

However, Knoop says this doesn't mean it's up to 20% closer to AGI.

Today we are announcing the winners of the ARC Prize 2024. We will also publish an extensive technical report on what we learned from the competition (linked in the following tweet).

Advanced technology rose from 33% to 55.5%, the largest single-year increase since 2020.

— François Chollet (@fchollet) December 6, 2024

In a blog post, Knoop said that many of the submissions to ARC-AGI can be solved using “brute force,” and that the “vast majority” of ARC-AGI tasks are “[don’t] It conveys a very useful signal to the general intelligence. ”

ARC-AGI consists of puzzle-like questions in which the AI ​​must generate the correct “answer” grid from a given grid of differently colored squares. These problems are designed to force the AI ​​to adapt to new problems it has never seen before. However, it is not clear whether they are achieving this.

ARC-AGI BenchmarkARC-AGI benchmark tasks. The model should solve the “problem” in the top row. The bottom line shows the solution. Image credit: ARC-AGI

“[ARC-AGI] It hasn't changed since 2019 and it's not perfect,” Knoop acknowledged in the post.

Francois and Knoop have also faced criticism for overselling ARC-AGI as a benchmark for AGI, at a time when the very definition of AGI is hotly contested. One OpenAI staff member recently argued that if AGI is defined as an AI that is “better than most humans at most tasks,” then AGI has “already” been achieved.

Knoop and Chollet say they plan to release a second generation ARC-AGI benchmark to address these issues in time for the 2025 competition. “We will continue to focus our research community's efforts on what we believe to be the most important unsolved problems in AI and accelerate our timeline for AGI,” Chollet wrote in the X post.

The fix probably won't be easy. If the shortcomings of the first ARC-AGI test are any indication, the definition of intelligence for AI will be just as unwieldy and inflammatory as it is for humans.





Source link

Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

Related Posts

OpenAI seeks to extend human lifespans with the help of longevity startups

January 17, 2025

Farewell to the $200 million woolly mammoth and TikTok

January 17, 2025

Nord Security founder launches Nexos.ai to help enterprises move AI projects from pilot to production

January 17, 2025

Data proves it remains difficult for startups to raise capital, even though VCs invested $75 billion in the fourth quarter

January 16, 2025

Apple suspends AI notification summaries for news after generating false alerts

January 16, 2025

Nvidia releases more tools and guardrails to help enterprises adopt AI agents

January 16, 2025

Leave A Reply Cancel Reply

Top Reviews
Editors Picks

7 days left: Founders and VCs save over $300 on all stage passes

March 24, 2025

AI chip startup Furiosaai reportedly rejecting $800 million acquisition offer from Meta

March 24, 2025

20 Hottest Open Source Startups of 2024

March 22, 2025

Andrill may build a weapons factory in the UK

March 21, 2025
About Us
About Us

Welcome to Tech Brunch, your go-to destination for cutting-edge insights, news, and analysis in the fields of Artificial Intelligence (AI), Cryptocurrency, Technology, and Startups. At Tech Brunch, we are passionate about exploring the latest trends, innovations, and developments shaping the future of these dynamic industries.

Our Picks

Klarna CEO and Sutter Hill wins lap after Jony Ive's Openai deal

May 22, 2025

Bluesky begins to check for “notable” users

May 22, 2025

Microsoft says Lumma Password Stealer Malware found on 394,000 Windows PCs

May 22, 2025

Subscribe to Updates

Subscribe to our newsletter and never miss our latest news

Subscribe my Newsletter for New Posts & tips Let's stay updated!

© 2025 TechBrunch. Designed by TechBrunch.
  • Home
  • About Tech Brunch
  • Advertise with Tech Brunch
  • Contact us
  • DMCA Notice
  • Privacy Policy
  • Terms of Use

Type above and press Enter to search. Press Esc to cancel.