Revolutionizing Automotive Retail: Motorway and AWS Forge a New Era of AI-Powered Dealer Efficiency

Motorway, a prominent UK-based online car marketplace, has partnered with AWS Prototyping and AI Customer Engineering (PACE) to introduce a groundbreaking AI-powered dealer stock search agent. This innovative solution significantly enhances the efficiency of dealers by replacing hours of manual vehicle filtering with intuitive natural language queries. The collaboration has resulted in a robust evaluation pipeline that drastically reduces errors and accelerates issue detection, marking a significant leap forward in the operational capabilities of automotive dealerships.

The Challenge: Ensuring Reliability in High-Stakes AI Deployment

The core challenge lay in ensuring the absolute reliability of a sophisticated AI agent operating in a high-stakes commercial environment. Motorway’s daily auctions involve up to 8,000 dealers bidding on as many as 2,500 vehicles. When an AI agent is tasked with assisting dealers in finding specific vehicles, especially with real money on the line, any inaccuracy or unreliability can have immediate and substantial financial consequences. The critical question was: how could Motorway and AWS definitively prove that this powerful, yet complex, AI system would perform consistently and accurately under real-world pressure? The traditional metrics for evaluating Large Language Models (LLMs), which focus on text generation quality, were insufficient for assessing the intricate, multi-tool functionality of an AI agent designed for complex tasks.

The Solution: An End-to-End Evaluation Pipeline

To address this critical need for robust validation, Motorway and AWS jointly developed an end-to-end evaluation pipeline. This sophisticated system has demonstrated remarkable success, reducing incorrect search results from one in every eight queries to an impressive one in fifty. Furthermore, the time required to detect and address issues has been slashed from several hours to mere minutes. This transformative improvement is largely attributed to the seamless integration of the Strands Agents SDK with Amazon Bedrock AgentCore, a fully managed service designed for the scalable deployment and operation of AI agents.

This collaborative effort has yielded a deployable blueprint, available in a companion GitHub repository, which serves as a foundation for building similar evaluation pipelines for other AI agents. While the blueprint leverages AWS services, the underlying principles—including a three-layer evaluation framework and the application of the pass^k metric for consistency—are universally applicable to any production-ready AI agent.

Prerequisites for Implementation

For organizations looking to replicate this success, a few prerequisites are essential. The initial deployment of the sample evaluation suite is estimated to take between 30 to 45 minutes. Customizing the system for a specific domain may require an additional two to three hours. Financially, the estimated costs for running the sample evaluation suite are modest, hovering around $5 to $10 in Amazon Bedrock inference charges. Production monitoring costs will, of course, vary based on the chosen sampling rate.

Security has been a paramount concern throughout this development. The companion repository adheres to best practices, implementing least-privilege AWS Identity and Access Management (IAM) roles, securely storing API keys in AWS Systems Manager Parameter Store (rather than less secure environment variables), and utilizing typed parameters to mitigate injection attacks. Detailed security measures are thoroughly outlined in the repository’s README file.

The Worked Example: Motorway’s Dealer Stock Search Agent

The foundation of this advanced evaluation system is Motorway’s dealer stock search agent, built upon the Strands Agents SDK and Amazon Bedrock AgentCore. This agent boasts an impressive eight distinct tools, capable of executing structured filtering across more than 89 vehicle attributes. It also incorporates vector similarity search, powered by LanceDB and Amazon Titan Text Embeddings V2, enabling it to understand nuanced and semantically similar queries.

Historically, dealers would dedicate significant portions of their day to sifting through vehicle listings using cumbersome CSV files and rigid, predefined filters. The introduction of this conversational AI agent has revolutionized this process. Dealers can now simply articulate their needs in natural language, such as, "Find me diesel SUVs under £25,000 near my dealership," or "I’m looking for something sporty and automatic for a family." This conversational interface dramatically reduces the time and effort required to find suitable inventory.

With approximately 1,500 concurrent users during peak hours, ensuring the agent’s behavior is consistently accurate is not merely desirable but imperative. A single tool selection error or a misinterpretation of a semantic search query can directly erode user trust. The agent’s runtime architecture, as illustrated in Figure 1, begins with dealers submitting natural language queries via a web interface. These queries are then routed to Amazon Bedrock AgentCore Runtime, which orchestrates calls to the eight available tools. The runtime leverages Amazon Bedrock models, specifically Claude for reasoning and Amazon Titan for embeddings, to process these requests. Responses from the tools are then funneled back through the runtime to generate the final, dealer-facing results.

Why Agent Evaluation is Fundamentally Different

Evaluating AI agents presents a unique set of challenges distinct from traditional LLM evaluation. While LLM evaluation primarily focuses on the quality of text generation—assessing coherence, factual accuracy, and relevance—agent evaluation delves into a much broader spectrum of performance. It’s akin to comparing an engine’s performance (LLM evaluation) to how an entire car navigates diverse traffic conditions, weather, and passenger loads (agent evaluation).

Standard LLM metrics fail to ascertain whether the Motorway agent correctly identified and invoked the appropriate search tool for a query like "Grade 1 Suzuki models." They also cannot reveal if the agent passed accurate filter parameters to LanceDB or if a dealer refining a previous search query receives the correct subsequent response.

The critical evaluation dimensions for AI agents are multifaceted:

• Task Completion: Agents often execute multi-step workflows, where partial completion can be a common outcome. The evaluation must account for the success rate of these complex sequences.
• Tool Use Correctness: The selection of the wrong tools or the passing of incorrect parameters can completely derail an agent’s workflow, leading to erroneous results.
• Reasoning Coherence: Flawed reasoning can lead to unpredictable failures, especially as the complexity of queries or environmental conditions changes.
• Reliability and Consistency: The inherent non-determinism of LLMs means that the same input can sometimes yield different outcomes. Evaluation must ensure a high degree of consistency.
• Safety and Compliance: Autonomous agents interacting with real-world systems require rigorous checks to prevent unintended actions and ensure adherence to safety protocols.
• Cost and Efficiency: An agent that requires an excessive number of API calls per task might be economically unviable, even if its functional performance is otherwise adequate.

A precise query like "Volkswagen Golf 7-12 years old" might be handled flawlessly. However, a more colloquial request, such as "I’m looking for an older VW," could fail if the semantic search layer isn’t robustly evaluated against a wide array of natural language variations.

Catching Issues Before Deployment with strands-agents-evals

The blueprint effectively integrates build-time evaluation into the GenAIOps lifecycle, ensuring that issues are identified and rectified before an agent is deployed into a production environment. This is complemented by production evaluation, which captures nuances that synthetic tests might miss. The three-layer evaluation framework, visualized in Figure 2, mandates that agents must pass specific thresholds in tool usage, reasoning, and output quality before they can be deployed.

During the development and Continuous Integration/Continuous Deployment (CI/CD) phases, the pipeline leverages the strands-agents-evals framework. This framework is specifically designed to catch pre-deployment issues and offers capabilities for output validation, trajectory evaluation, multi-turn conversation simulation, and automated experiment generation, all working seamlessly with agents built on the Strands Agents SDK. The framework employs three core primitives:

1. Structure Tests by Layer: Tests are organized into distinct layers. Layer 1 focuses on deterministic, code-based grading for tool selection accuracy. Layers 2 and 3 utilize LLM-as-judge evaluators, where an LLM scores agent outputs, to assess reasoning and output quality.
2. Domain-Specific Customization: Custom Evaluator subclasses are crucial for addressing domain-specific concerns. For Motorway’s agent, these include validating data freshness, ensuring dealer data scoping, and implementing safety guardrails. Each agent will have its own unique set of domain constraints.
3. Three Types of Graders: The evaluation framework employs three distinct grader types, each tailored for specific evaluation needs:
   • Code-based deterministic graders: Ideal for Layer 1, they ensure tool selection, parameter passing, and trajectory ordering are accurate. They are fast, cost-effective, and reproducible.
   • LLM-as-judge (e.g., Claude Sonnet 4.6): Suitable for Layers 2 and 3, these graders offer flexibility in assessing reasoning quality, output helpfulness, and goal success. While non-deterministic, their behavior can be controlled using the pass^k metric.
   • Human review: This is reserved for calibrating LLM judge prompts and assessing edge cases and safety concerns. While expensive, it provides crucial ground truth.

In practice, evaluating the agent’s final output often proves more effective than grading its internal process. The primary concern is that the user receives relevant results, rather than the specific sequence of tool calls the agent made.

The Three-Layer Assessment Framework in Detail

Build-time evaluation is meticulously structured across three distinct layers, each with its own pass/fail thresholds:

• Layer 1: Tool Usage (Threshold > 95 percent): This layer verifies if the agent invoked the correct tools with the appropriate parameters. This is assessed deterministically using the ToolSelectionGrader to check tool invocation and the TrajectoryOrderGrader to confirm the sequence of calls.
• Layer 2: Reasoning (Threshold > 85 percent): This layer scrutinizes the logical coherence of the agent’s decision-making process. The HelpfulnessEvaluator and TrajectoryEvaluator from strands-agents-evals, employing LLM-as-judge scoring, determine if the agent’s reasoning is sound. An agent that reaches a correct answer through flawed logic may fail unpredictably as conditions change.
• Layer 3: Output Quality (Threshold > 90 percent): This layer assesses whether the agent’s response was helpful, accurate, and actionable. The OutputEvaluator and GoalSuccessRateEvaluator from strands-agents-evals use LLM-as-judge evaluation to confirm that the user received a useful and well-formatted response.

All three layers must pass before an agent can be deployed, with any failure blocking the pipeline.

Handling Non-Determinism for Consistent Performance

The inherent variability of LLM outputs necessitates robust methods for handling non-determinism. Single-trial results can be misleading, so the run_all_layers() function in the companion repository incorporates a num_trials parameter to mitigate this. Two metrics derived from code generation research are vital for measuring reliability:

• pass@k: This metric measures the probability that at least one out of k independent trials of the agent will succeed.
• Mean@k: This metric calculates the average number of successful trials out of k attempts.

For customer-facing agents, pass@k is of paramount importance. An agent with a 75 percent per-trial success rate has only a 42 percent chance of passing three consecutive trials (0.75³). Users expect consistent, high-quality interactions every time they engage with the agent. The companion code’s run_all_layers(task_fn, registry, num_trials=5) function facilitates multi-trial evaluation and gates deployment on pass^k, ensuring a higher level of reliability.

Comprehensive Test Case Management

Test cases are meticulously organized by category to ensure thorough evaluation:

• Functional tests: Verify the agent’s ability to perform specific tasks correctly.
• Regression tests: Ensure that fixes or new features haven’t introduced unintended side effects.
• Adversarial tests: Challenge the agent with edge cases and unexpected inputs to uncover potential vulnerabilities.

When production monitoring identifies an issue, that specific interaction is converted into a new test case. Motorway’s test suite rapidly expanded from an initial 50 cases to 150 within three months, with each case grounded in actual user behavior. A recommended starting point is 20 to 50 test cases, allowing production data to naturally grow the suite over time.

Crucially, the test suite must include negative cases—scenarios where the agent should explicitly not perform certain actions. For instance, a profile query should invoke the profile tool, not a general search tool. Similarly, a structured query should utilize structured search capabilities rather than falling back to raw SQL. One-sided evaluations, which only test for positive outcomes, can lead to one-sided optimization and overlooked failure modes.

Multi-Turn Conversation Testing: The Next Frontier

Single-turn evaluations, while valuable, often fail to capture a critical dimension of user interaction: conversational coherence. Dealers frequently refine their searches across multiple turns, building upon previous interactions. For example, a dealer might first search for "SUVs under £30,000," then follow up with "Make them diesel and under 50,000 miles."

The strands-agents-evals framework addresses this through its ActorSimulator, which generates realistic multi-turn interactions, and InteractionsEvaluator, which scores context retention across these turns. Multi-turn tests are essential for identifying issues such as context drift, accumulating filter errors, and failures in pronoun resolution that single-turn evaluations would entirely miss.

Monitoring Production Behavior with AgentCore Evaluations

Once an agent is deployed to Amazon Bedrock AgentCore Runtime, the built-in AgentCore Evaluations provide continuous monitoring capabilities. This system integrates with Strands Agents via OpenTelemetry instrumentation, the industry-standard framework for observability. Figure 3 illustrates the production architecture, where observability traces are sampled at a rate of 1-5 percent, and metrics are aggregated into Amazon CloudWatch.

Two Powerful Monitoring Approaches

AgentCore Evaluations offers two complementary monitoring modes:

1. On-demand evaluation: This mode allows for the analysis of specific agent interactions by selecting relevant spans from Amazon CloudWatch logs. It is particularly useful for debugging identified issues or validating the effectiveness of implemented fixes.
2. Online evaluation: This automated mode continuously samples live traffic, applying configured evaluators in the background. Users can set a sampling rate (1-5 percent is recommended) and select up to 10 evaluators to monitor agent performance in real-time.

Built-in and Custom Evaluators for Comprehensive Oversight

AgentCore provides a suite of pre-configured evaluators for common scenarios. These include:

• Builtin.Helpfulness: Assesses the helpfulness of the agent’s response on a 0-1 score across seven levels.
• Builtin.GoalSuccessRate: Determines whether the user’s overall objective was achieved within a session.
• Builtin.ToolSelection: Evaluates the appropriateness of the tools selected by the agent.
• Builtin.Correctness: Measures the factual accuracy of the agent’s responses.

For domain-specific requirements that built-in evaluators may not cover, custom evaluators can be created using an LLM-as-a-judge configuration. These custom evaluators are crucial for enforcing domain constraints such as data freshness, access scoping, forbidden actions, latency budgets, and cost limits. The companion repository provides examples of custom Evaluator subclasses, including:

• DataFreshnessEvaluator: Ensures the agent does not surface stale inventory by validating auction-cycle timestamps.
• SafetyGuardrailEvaluator: Prevents the agent from initiating automated bidding actions.
• DealerDataScopingEvaluator: Enforces data isolation by ensuring queries are scoped to specific dealers.

Additional examples for LatencyEvaluator and CostEvaluator are also available within the companion repository.

Key Metrics for Proactive Monitoring

The companion repository includes an AWS CDK stack that provisions CloudWatch dashboards and alarms for critical metrics. Tracking these metrics provides essential insights into agent health:

Metric	Target	Alert Threshold
Task completion rate	>95%	<80%
Tool selection accuracy	>95%	<90%
Helpfulness score (0-1)	>0.83	<0.58
Response latency P50 / P99	<2s / <10s	>5s / >15s
Hallucination rate	<2%	>5%
Cost per interaction	Monitor trend	>2x baseline

Making Quality Checks a Deployment Gate

Effective evaluation is not an afterthought but a critical quality gate within the deployment pipeline. Figure 4 illustrates a comprehensive five-stage deployment pipeline that incorporates evaluation at each phase: build-time evaluation, staging validation, shadow mode, A/B testing, and phased production rollout. Failures at any stage block further deployment.

The pipeline stages are as follows:

1. Build-time evaluation: Automated tests run during the CI/CD process.
2. Staging validation: A pre-production environment for comprehensive testing.
3. Shadow mode: The candidate agent processes a copy of production traffic without impacting users.
4. A/B testing: Gradual rollout to a subset of users to compare performance against the existing version.
5. Production rollout: Full deployment to all users.

Each phase has defined thresholds; for instance, tool selection accuracy below 95 percent or task completion below 80 percent will halt deployment. For major releases, multi-trial evaluation with num_trials=5 gates deployment on pass^k, mitigating non-deterministic failures.

Shadow Mode: A Critical Pre-Production Step

Positioned between staging and full production, shadow mode allows the candidate agent to process real queries alongside the existing system without affecting users. By receiving a copy of production traffic and processing it in parallel, the candidate agent’s outcomes can be directly compared. Running shadow mode for at least four hours before proceeding to A/B testing is recommended. A deviation threshold, typically set at 2 percent, can automatically pause deployment if performance deviates significantly.

Shadow mode is invaluable for uncovering issues that other stages might miss, such as:

• Subtle differences in real-world data distributions compared to staging environments.
• Performance degradation under actual production load.
• Interactions with other integrated systems that may not be fully replicated in staging.

Getting Started: A Phased Approach

Adopting this robust evaluation methodology can be approached in a phased manner:

1. Phase 1: Build Your Test Suite: Begin by compiling 20-50 test cases derived from actual user queries. Include both positive scenarios (what the agent should do) and negative scenarios (what it should not do).
2. Phase 2: Configure Build-Time Evaluation: Align grader types with the specific evaluation needs: deterministic graders for tool selection and LLM-as-judge for reasoning and output quality.
3. Phase 3: Enable Production Monitoring: Set up AgentCore Evaluations with an initial sampling rate of 1-5 percent. Monitor evaluator costs and adjust the sampling rate as needed.
4. Phase 4: Close the Feedback Loop: Convert production failures into new test cases. This continuous feedback loop, where production issues become regression tests, consistently enhances evaluation quality across teams.

Results and Impact: A Tangible Transformation

Prior to the implementation of this sophisticated evaluation pipeline, Motorway’s agent exhibited an 87% tool selection accuracy, meaning approximately 1 in 8 dealer queries yielded incorrect results. The team was addressing an average of 12 production incidents per month, with an average detection time of four hours after issues began impacting dealers.

Following the implementation of the pipeline, the results have been transformative:

Metric	Before	After
Tool selection accuracy	87%	98%
Task completion rate	82%	96%
Context retention (multi-turn)	71%	94%
Production incidents (monthly)	12	2
Mean time to detect issues	few hours	few minutes

The business impact is profound: dealers can now complete vehicle searches in minutes, not hours, with a high degree of confidence in the accuracy and currency of the results. This efficiency translates directly into increased sales opportunities and enhanced customer satisfaction.

Troubleshooting Common Issues

For organizations implementing this solution, the companion repository’s README file offers guidance on common troubleshooting scenarios, including trajectory grader failures, LLM-as-judge variance, empty evaluation results, and cost threshold tuning.

Key Takeaways for Building Robust AI Agent Evaluation

Several core principles underpin the success of this evaluation pipeline:

• Beyond Fluency: A fluent response does not guarantee correct agent behavior. Verification of tool selection, parameter accuracy, reasoning coherence, and consistency across repeated runs is paramount.
• Integrated Approach: Combining rigorous build-time testing with strands-agents-evals and continuous production monitoring via AgentCore Evaluations is essential for deploying agents with proven reliability.
• Domain Specificity: Tailoring evaluation to specific domain constraints is critical for ensuring agents meet unique business requirements.
• Continuous Improvement: Establishing a feedback loop where production failures inform test cases drives ongoing refinement and enhances evaluation quality.

Conclusion: A New Standard for Production AI Agents

This detailed exploration has demonstrated how to construct a robust evaluation pipeline for production AI agents on AWS, using Motorway’s dealer stock search agent as a practical example. The fundamental lesson is that the quality of an AI agent’s output must be rigorously verified, extending beyond mere conversational fluency to encompass the correctness of its underlying actions.

By integrating build-time testing with strands-agents-evals and production monitoring with AgentCore Evaluations, organizations can confidently deploy agents that function precisely as designed. These patterns are broadly applicable to a wide range of multi-tool, customer-facing agents, whether they are designed for customer service, financial advisory, or healthcare triage.

To embark on this journey:

• Explore the companion repository for a deployable blueprint.
• Familiarize yourself with the Amazon Bedrock AgentCore documentation for in-depth guidance.
• Discover the Strands Agents SDK on GitHub for building and evaluating agents.

Clean Up Resources

To prevent incurring ongoing charges after completing this walkthrough, it is essential to delete the resources created. This typically involves removing the CloudFormation stack that provisions the resources. Note that S3 buckets containing objects may require manual deletion. It is also advisable to export any evaluation data or logs you wish to retain before initiating the cleanup process.

Resources for Further Exploration

• Companion Repository
• Amazon Bedrock AgentCore Documentation
• Strands Agents SDK