This article summarizes a talk discussing the challenges and potential solutions for building effective AI agents. While there's significant interest in agents, current ambitious visions are often unrealized. The talk outlines three main reasons for this and suggests ways to improve AI agent development.
The Timeliness of Agent Development
There's considerable interest in AI agents across product development, industry, academic labs, and research. Agents are seen as crucial for scaling language models, especially as we move closer to Artificial General Intelligence (AGI). Even tools like ChatGPT and Claude are considered rudimentary agents with input/output filters and task-execution capabilities. We see mainstream products offering agent functionalities like open-ended internet tasks and report writing.
Unmet Expectations and the Need for Improvement
Despite the promise, ambitious visions of AI agents are far from being realized. Many products have failed to deliver on their initial claims. This isn't to criticize specific products but to highlight the challenge of building AI agents that truly work for users. The following sections will explore the key reasons for these failures and potential solutions.
Evaluating Agents is Genuinely Hard
Real-World Failures as Examples
Several examples demonstrate the difficulty in evaluating agents effectively.
-
DoNotPay: This startup, claiming to automate legal work, faced FTC fines for making false performance claims. The claims were not backed by real-world results.
-
LexisNexis and Westlaw: Leading lawtech firms, touted hallucination-free legal report generation but showed significant hallucination rates in evaluations by Stanford researchers, even reversing the original intent in legal texts.
-
Sakana AI: Claimed to have built an AI research scientist capable of automating scientific research. However, Princeton researchers, using the Core Bench benchmark, found that these agents couldn't reliably reproduce research papers, even with provided code and data.
-
Sakana AI CUDA Kernel Optimization: Claimed significant performance improvements over standard CUDA kernels, but analysis revealed they were exceeding the theoretical maximum, indicating a flawed evaluation process.
The Importance of Rigorous Evaluation
These examples underscore the need for rigorous evaluation as a core component of AI engineering. Without it, the risk of failures remains high. Lack of rigorous evaluation is a significant factor in overhyped AI performance and subsequent failures.
Limitations of Static Benchmarks
The Difference Between Models and Agents
Static benchmarks, often used for evaluating language models, are misleading for agents. Language model evaluations focus on input and output strings, while agents interact with real-world environments and take actions. This makes evaluation considerably more complex.
Challenges in Evaluating Agent Performance
-
Open-Ended Actions: Agents can take unlimited actions, leading to complex recursions and variable costs.
-
Cost as a Factor: Cost must be a primary evaluation metric, alongside accuracy and performance, to understand agent effectiveness.
-
Purpose-Built Agents: Agents are often designed for specific tasks, making it difficult to use general benchmarks for evaluation.
-
Meaningful Metrics: Meaningful multi-dimensional metrics are needed instead of relying on single benchmarks.
Consequences of Static Evaluations
Optimizing for single benchmarks without considering cost and other factors results in an incomplete picture of agent performance.
Holistic Agent Leaderboards
Efforts like the Holistic Agent Leaderboard (HAL) at Princeton aim to address these issues by evaluating agents with cost alongside accuracy. A Pareto frontier can show that a model may perform a couple of percentage points higher, but cost 10 times more than another, making the choice obvious.
The Jevons Paradox and Cost Considerations
Despite decreasing costs of language models, the Jevons paradox suggests that overall usage and associated costs will continue to increase. The Jevons Paradox states that as the cost of using a technology decreases, its use increases, leading to a greater overall consumption of the resource. Therefore, cost remains a crucial factor in agent evaluations.
Over-Reliance on Benchmarks and Human-in-the-Loop Validation
Benchmarks and Funding
Benchmark performance often influences funding decisions, but it rarely translates to real-world success. Real-world evaluations reveal the limitations of relying solely on benchmark results.
The Need for Human Validation
Frameworks that incorporate human domain experts into the evaluation loop can lead to better results. Humans proactively edit criteria based on which the models are evaluated leading to much better overall results.
Capability vs. Reliability: A Key Distinction
The Importance of Reliability
Capability refers to what a model can do, while reliability means consistently getting the correct answer. For real-world applications, especially those involving consequential decisions, reliability is paramount. The reliability of agent products and the potential for failure in the absence of it is why products like Humane Spin and Rabbit R1 have failed.
The Job of the AI Engineer
Closing the gap between capability and reliability is the job of the AI engineer. It requires focusing on software optimizations and abstractions to work with inherently stochastic language models.
Verifiers and Their Limitations
Verifiers, such as unit tests, can improve reliability but are not foolproof. False positives in verifiers can lead to decreased performance as the system tries more and more answers.
AI Engineering as Reliability Engineering
AI engineering should be viewed as a reliability engineering field, focusing on system design rather than just modeling. This requires a mindset shift towards ensuring the reliability of AI systems.
Lessons from Early Computing
The early days of computing, with unreliable vacuum tubes, provide a historical precedent. Engineers focused on improving reliability to make the technology usable. AI engineers need to focus on fixing reliability issues in AI agents to ensure they are reliable for end users.
{#