A year ago, Databricks acquired MosaicML for $1.3 billion. Now rebranded as Mosaic AI, the platform has become an integral part of Databricks' AI solutions. Today, at the company's Data + AI Summit, the company is announcing several new features for the service. Ahead of the announcement, I spoke with Databricks co-founder and CEO Ali Ghodsi and CTO Matei Zaharia.
At the conference, Databricks will be announcing five new Mosaic AI tools: Mosaic AI Agent Framework, Mosaic AI Agent Evaluation, Mosaic AI Tools Catalog, Mosaic AI Model Training, and Mosaic AI Gateway.
“This year has been great. There's been great developments in Gen AI. Everyone's excited about it,” Ghodshi told me. “But everyone's still concerned about the same three things: how do we improve the quality and reliability of these models? Secondly, how do we ensure cost-effectiveness? There's a huge variance in cost between models, and there are huge orders of magnitude differences in price. And thirdly, how do we do that while maintaining the privacy of the data?”
Today’s announcement aims to address most of these concerns for Databricks customers.
Zaharia also noted that companies currently deploying large language models (LLMs) in production use systems with multiple components. This often means invoking a model (or multiple models) multiple times and using a variety of external tools to access databases, perform search augmentation generation (RAG), etc. These composite systems speed up LLM-based applications, save costs by using cheaper models for certain queries or by caching results, and perhaps most importantly, make results more reliable and relevant by augmenting the underlying models with their own data.
“We see that as the future of really high-impact, mission-critical AI applications,” he explained. “If you think about it, if you're doing something really mission-critical, you want your engineers to have control over every aspect of it. And that's what a modular system does. So we're doing a lot of fundamental research into the best way to create these applications.” [systems] It's designed for a specific task, so developers can work easily, connect all the bits, trace everything, and see what's going on.”
To actually build these systems, Databricks is launching two services this week: the Mosaic AI Agent Framework and the Mosaic AI Tools Catalog. The AI Agent Framework takes the company's serverless vector search capability, which became generally available last month, and gives developers the tools to build their own RAG-based applications on top of it.
Ghodsi and Zaharia emphasized that the Databricks Vector Search system takes a hybrid approach, combining traditional keyword-based search with embedded search. All of this is deeply integrated with the Databricks data lake, with data in both platforms always automatically synchronized. This includes governance features across the Databricks platform (specifically, the Databricks Unity Catalog governance layer) to ensure, for example, that personal information does not leak into the Vector Search service.
Speaking of the Unity Catalog (which the company is now gradually open sourcing), it's worth noting that Databricks is now extending the system to allow companies to curate the AI tools and functions that these LLMs can call when generating answers. Databricks says the catalog will make these services more discoverable across the enterprise.
Ghodsi also emphasized that developers can now use all these tools to build their own agents by chaining together models and functions using, for example, Langchain and LlamaIndex, and in fact, Zaharia said that many of Databricks' customers are already using these tools.
“There are a lot of companies using this, including agent-like workflows. I think a lot of people would be surprised at how many there are, but this seems to be the direction of the future. And even in our own internal AI applications, such as the assistant application on our platform, we're finding that this is the way to build AI,” he said.
To evaluate these new applications, Databricks is also releasing Mosaic AI Agent Evaluation, an AI-assisted evaluation tool that combines LLM-based adjudication to test how well an AI works in production. It also enables companies to get rapid feedback from users (and label initial data sets). Quality Lab includes a UI component based on Lilac, which Databricks acquired earlier this year, that allows users to visualize and search large text data sets.
“All of our clients say they need to label in-house, so they're going to get their employees to do it. All they need is 100 or 500 answers and they can give them to the LLM examiners,” Ghodsi explained.
Another way to improve results is to use fine-tuned models, and for this, Databricks now offers the Mosaic AI model training service, which allows users to fine-tune models using their organization's private data to improve performance for specific tasks.
The final new tool is the Mosaic AI Gateway, which the company describes as “a unified interface to query, manage, and deploy open source or proprietary models.” The idea here is to allow users to query any LLM in a managed way, using a centralized credential store. After all, no enterprise wants their engineers sending random data to a third-party service.
In times of shrinking budgets, AI gateways also allow IT departments to set rate limits on various vendors to keep costs manageable, and they also provide usage tracking and tracing for debugging these systems.
As Ghodsi told me, all of these new features are a response to how Databricks’ users are currently using LLM. “We’ve seen a big shift in the market in the last quarter and a half. At the beginning of last year, no matter who you talked to, they were saying, we’re for open source. Open source is great. But when you really convince people, they were using Open AI. Regardless of what they were saying, regardless of how much they were touting how great open source is, everyone was using Open AI behind the scenes.” Now, these customers are much more sophisticated and using open models (though of course, very few of them are actually open source), and so they’re having to adopt a whole new set of tools to tackle the problems (and opportunities) that come with that.