Demis Hassabis, CEO of Google's DeepMind AI Institute, said in February that putting large amounts of compute into the types of AI algorithms currently widely used could lead to diminishing returns. I was warned that there would be. Hassabis said that reaching the “next level” of AI, so to speak, will require breakthroughs in basic research that create viable alternatives to today's entrenched approaches.
Former Tesla engineer George Morgan agrees. So he founded the startup Symbolica AI to do just that.
“Traditional deep learning and generative language models require unimaginable scale, time, and energy to produce useful results,” Morgan told TechCrunch. “By building a building [novel] Symbolica can achieve higher accuracy with lower data requirements, lower training time, lower cost, and proven correct structured output. ”
Morgan dropped out of the University of Rochester to join Tesla and work on the team developing Autopilot, Tesla's suite of advanced driver assistance features.
Morgan said that during his time at Tesla, he realized that current AI methods, most of which are centered around scaling up computing, were not sustainable over the long term.
“With the current approach, there's only one dial to turn: scale up and expect new behavior,” Morgan said. “But scaling requires more compute, more memory, more money for training, and more data. But in the end, [this] It does not significantly improve performance. ”
Morgan is not alone in reaching that conclusion.
Two executives from semiconductor manufacturing company TSMC said in a memo this year that if AI trends continue at their current pace, the industry will need 1 trillion transistor chips, or chips containing 10 times the average number of transistors. He said it would be. Today's chip he will realize within 10 years.
It is unclear whether it is technically feasible.
Additionally, a report co-authored by Stanford University and independent AI research institute Epoch AI finds that the cost of training cutting-edge AI models has increased significantly over the past year and its changes. doing. The report's authors estimate that OpenAI and Google spent approximately $78 million and $191 million on training for GPT-4 and Gemini Ultra, respectively.
With costs set to rise further (see OpenAI and Microsoft's reported $100 billion AI data center plans), Morgan has begun investigating what he calls “structured” AI models. Rather than trying to approximate insights from huge datasets as traditional models do, these structured models encode the underlying structure of the data, thereby reducing the overall amount of computing. , will be able to achieve what Morgan characterizes as better performance.
“It is possible to generate domain-tailored structured inference capabilities in much smaller models, combining breakthroughs in deep mathematics toolkits and deep learning,” he said.
Structured models, also known as symbolic AI, are not an entirely new concept. These date back several decades and are rooted in the idea that AI can be built based on symbols that represent knowledge using a set of rules.
Symbolic AI solves tasks by defining a set of symbol manipulation rules specifically for a specific job, such as editing a line of text in word processing software. This is in contrast to neural networks, which attempt to solve tasks through statistical approximations and learning from examples.
Neural networks are the basis of powerful AI systems such as OpenAI's DALL-E 3 and GPT-4. But Morgan insists they are not final. In fact, symbolic AI may be well-suited to efficiently encode knowledge of the world, reason through complex scenarios, and “explain” how to arrive at an answer, Morgan said. claim.
“Our model is more reliable, more transparent and more accountable,” Morgan said. “The commercial applications for structured inference capabilities are vast, and existing products are inadequate, especially for code generation, where large codebases are inferred to produce useful code.”
Designed by a team of 16 people, Symbolica's product is a toolkit for creating symbolic AI models and pre-trained models for specific tasks, such as generating code or proving mathematical theorems. The exact business model is in flux. However, Morgan said Symbolica may offer consulting services and support to companies that want to use its technology to build bespoke models.
Today marks the secret launch of Symbolica, so the company has no customers — at least, no customers it wants to talk about publicly. However, Morgan revealed that Symbolica secured a $33 million investment led by Khosla Ventures earlier this year. Other investors include Abstract Ventures, Buckley Ventures, Day One Ventures, and General Catalyst.
$33 million is no small number. Symbolica's backers are clearly confident in the startup's science and roadmap. Vinod Khosla, founder of Khosla Ventures, said in an email that he believes Symbolica is “addressing some of the most important challenges facing the AI industry today.”
“To enable large-scale commercial AI deployment and regulatory compliance, we need models with structured outputs that can achieve higher accuracy with fewer resources,” said Khosla. “George has assembled one of the best teams in the industry to do just that.”
But some are less convinced that symbolic AI is the right path to take.
candidate at the University of Washington, who focuses on law and data ethics, said that because symbolic AI models rely on highly structured data, the models are “very fragile” and that context and points out that it depends on specificity. This means that symbolic AI requires well-defined knowledge to work, and defining that knowledge can be very labor-intensive.
“If you combine the benefits of deep learning and symbolic approaches, this could still be interesting,” Keys said, referring to DeepMind's recently published AlphaGeometry, which is a neural network and AI-inspired symbolic approach. He said he solved a difficult geometric problem by combining algorithms. “But only time will tell.”
Morgan counters that promising alternatives are worth investing in, as current training methods will soon no longer be able to meet the needs of companies wanting to leverage AI for their purposes, and Symbolica is strategically positioned in this regard. He claimed to be in a favorable position. It has a “multi-year” runway with the latest funding tranche and is promising given its model is relatively small (and therefore cheap) to train and run.
“For example, to automate software development at scale, you need models with formal reasoning capabilities to parse large code databases and generate and iterate on useful code. , which requires lower operating costs,” he said. “The public perception around AI models is still that all you need is scale. To make progress in this field, thinking symbolically is absolutely necessary. Meeting these demands requires structured and explainable output with formal reasoning capabilities. ”
There is little to prevent large AI labs like DeepMind from building their own symbolic AI or hybrid models. And aside from Symbolica's point of differentiation, Symbolica is entering a very crowded and well-capitalized AI space. But Morgan expects growth, predicting that San Francisco-based Symbolica's workforce will double by 2025.