agent_inspect.tools.error_analysis package

Submodules

agent_inspect.tools.error_analysis.base_error_analysis module

Error Analysis Module - Class Hierarchy

This module defines the base class for error analysis implementations.

Class Hierarchy Diagram:

                                ErrorAnalysis (ABC)
                                      |
                                      | (abstract base)
                ______________________|_______________________
               |                                              |
        ToolCallErrorAnalysis (ABC)              SubgoalErrorAnalysis (ABC)
        (tool call errors)                       (subgoal errors)
               |                                              |
     __________|__________                           _________|_________
    |                     |                         |                   |
SemisupervisedTool      DeterministicTool       UnsupervisedSubgoal     SemisupervisedSubgoal
CallErrorAnalysis       CallErrorAnalysis       ErrorAnalysis           ErrorAnalysis
(LLM-based              (Regex-based            (Dynamic clustering     (Predefined clusters
classification)         classification)         with LLM)               with LLM)

Algorithm Types:

  1. Unsupervised: Discovers error patterns dynamically without predefined categories - Uses LLM for error summarization and semantic clustering - Generates error categories from data

  2. Semi-supervised: Classifies errors into predefined error clusters - Uses LLM with structured prompts for classification - More consistent and interpretable results - Supports custom error cluster definitions

  3. Deterministic: Rule-based classification using regex patterns - No LLM required (faster, cheaper) - Works only for tool call errors with specific explanation formats

class agent_inspect.tools.error_analysis.base_error_analysis.ErrorAnalysis(config=None)[source]

Bases: ABC

Abstract base class for all error analysis implementations.

This class provides common functionality for error analysis operations:

  • Configuration management via config dictionary

  • Thread pool executor for concurrent processing

  • Retry logic for LLM requests with JSON decode errors

Parameters:

config (Optional[Dict[str, Any]]) –

Optional configuration dictionary. Supported keys:

  • max_workers: Maximum number of concurrent workers (default: 20)

agent_inspect.tools.error_analysis.deterministic_tool_call_error_analysis module

class agent_inspect.tools.error_analysis.deterministic_tool_call_error_analysis.DeterministicToolCallErrorAnalysis(config=None)[source]

Bases: ToolCallErrorAnalysis

Deterministic (rule-based) error analysis for tool call validations.

This class classifies tool validation failures into predefined error clusters using regex pattern matching on validation explanations. It provides a fast, LLM-free alternative to SemisupervisedToolCallErrorAnalysis.

Cluster Labels (in priority order):

Priority Behavior: When multiple failures occur (e.g., both argument AND output fail), the highest priority label is assigned based on the order above.

Parameters:

config (Optional[Dict[str, Any]]) –

Optional configuration dictionary. Supported keys:

  • max_workers: Maximum number of concurrent workers (default: 20)

Example:

>>> from agent_inspect.tools import DeterministicToolCallErrorAnalysis
>>> from agent_inspect.models.tools import ToolCallErrorAnalysisDataSample
>>>
>>> analyzer = DeterministicToolCallErrorAnalysis(config={"max_workers": 1})
>>> result = analyzer.analyze_batch(tool_validation_data_samples)
>>> for cluster_label, validations in result.analyzed_validations_clustered_by_errors.items():
...     print(f"{cluster_label}: {len(validations)} errors")

agent_inspect.tools.error_analysis.llm_output_schemas module

agent_inspect.tools.error_analysis.llm_templates module

agent_inspect.tools.error_analysis.semisupervised_subgoal_error_analysis module

class agent_inspect.tools.error_analysis.semisupervised_subgoal_error_analysis.SemisupervisedSubgoalErrorAnalysis(llm_client, config=None, error_clusters=None)[source]

Bases: SubgoalErrorAnalysis

Semi-supervised algorithm for subgoal-based error analysis.

This implementation uses predefined error clusters to classify failures, making it more efficient and consistent than unsupervised approaches. The algorithm:

  1. Error Summarization: For each failed subgoal validation, generates a concise error description using LLM analysis of judge trial explanations

  2. Semi-Supervised Clustering: Classifies errors into predefined categories using semantic matching. When error doesn’t fit existing categories, it will be assigned to “Unclassified”. No new clusters will be created in this approach.

  3. Result Organization: Groups analyzed validations by error cluster

Notes: There is a small efficiency optimization compared with the unsupervised approach: When subgoal validations have mixed judge scores (some complete, some incomplete), only analyzes the first incomplete judge trial explanation rather than performing majority voting across all trials.

Default predefined clusters:

Parameters:

  • llm_client (LLMClient) – The LLMClient for performing error analysis and clustering.

  • config (Optional[Dict[str, Any]]) –

    Optional configuration dictionary. Supported keys:

    • max_workers: Maximum number of concurrent workers (default: 20)

  • error_clusters (Optional[List[ErrorCluster]]) – Optional list of predefined error clusters. If not provided, uses the default clusters.

Raises:

InvalidInputValueError – If llm_client is None or error_clusters are invalid (wrong format, empty list, etc.)

Example:
>>> from agent_inspect.clients.azure_openai_client import AzureOpenAIClient
>>> from agent_inspect.models.tools import SubgoalErrorAnalysisDataSample
>>> from agent_inspect.tools import SemisupervisedSubgoalErrorAnalysis
>>>
>>> llm_client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096)
>>>
>>> # Prepare data samples with subgoal validations
>>> data_samples: List[SubgoalErrorAnalysisDataSample] = [...]
>>>
>>> # Run analysis with default clusters
>>> analyzer = SemisupervisedSubgoalErrorAnalysis(llm_client)
>>> result = analyzer.analyze_batch(data_samples)
>>>
>>> # Or use custom clusters
>>> from agent_inspect.models.tools.error_cluster import ErrorCluster
>>> from dataclasses import dataclass
>>>
>>> @dataclass(frozen=True)
... class APIRateLimitCluster(ErrorCluster):
...     cluster_label: str = "API Rate Limiting"
...     description: str = "Agent hit rate limits on external APIs"
>>>
>>> custom_clusters = [APIRateLimitCluster()]
>>> analyzer = SemisupervisedSubgoalErrorAnalysis(llm_client, error_clusters=custom_clusters)
>>> result = analyzer.analyze_batch(data_samples)

agent_inspect.tools.error_analysis.semisupervised_tool_call_error_analysis module

class agent_inspect.tools.error_analysis.semisupervised_tool_call_error_analysis.SemisupervisedToolCallErrorAnalysis(llm_client, config=None, error_clusters=None)[source]

Bases: ToolCallErrorAnalysis

Performs semi-supervised error analysis on tool call validations using LLMs.

This implementation uses a direct classification approach where each validation error is classified into predefined error clusters using an LLM judge. The validation explanation serves as the base error description, and the LLM assigns the most appropriate error cluster label.

Unlike unsupervised methods that discover patterns through clustering, this semi-supervised approach leverages predefined error categories to provide more interpretable and consistent error classification across evaluations. When a validation error does not fit well into any of the predefined clusters, it will be labeled as “Unclassified”. No new clusters will be created in this approach.

Default predefined clusters:

Parameters:

  • llm_client (LLMClient) – The LLMClient to use for error classification. Must implement the LLMClient interface for making LLM requests.

  • config (Optional[Dict[str, Any]]) –

    Optional configuration dictionary. Supported keys:

    • max_workers: Maximum number of concurrent workers for processing validations (default: 20)

  • error_clusters (Optional[List[ErrorCluster]]) – Optional list of error cluster definitions. If not provided, uses the default tool call error clusters.

Raises:

InvalidInputValueError – If llm_client is None or if error_clusters are invalid.

Example:
>>> from agent_inspect.clients.azure_openai_client import AzureOpenAIClient
>>> from agent_inspect.models.tools.analysis_models import ToolCallErrorAnalysisDataSample
>>>
>>> # Initialize with LLM client
>>> llm_client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096)
>>> analyzer = SemisupervisedToolCallErrorAnalysis(
...     llm_client=llm_client,
...     config={"max_workers": 10}
... )
>>>
>>> # Prepare data samples with tool call validations
>>> data_samples: List[ToolCallErrorAnalysisDataSample] = [...]
>>>
>>> # Run error analysis with default clusters
>>> results = analyzer.analyze_batch(data_samples)
>>>
>>> # Or provide custom error clusters
>>> from agent_inspect.models.tools.error_cluster import ErrorCluster
>>> from dataclasses import dataclass
>>>
>>> @dataclass(frozen=True)
>>> class CustomErrorCluster(ErrorCluster):
...     cluster_label: str = "Custom Error Type"
...     description: str = "Description of when this error occurs"
>>>
>>> custom_clusters = [CustomErrorCluster()]
>>> custom_analyzer = SemisupervisedToolCallErrorAnalysis(
...     llm_client=llm_client,
...     error_clusters=custom_clusters
... )
>>> results = custom_analyzer.analyze_batch(data_samples)
>>>
>>> # Inspect results
>>> for cluster_label, validations in results.analyzed_validations_clustered_by_errors.items():
...     print(f"{cluster_label}: {len(validations)} errors")

agent_inspect.tools.error_analysis.statistic_analysis module

agent_inspect.tools.error_analysis.subgoal_error_analysis module

class agent_inspect.tools.error_analysis.subgoal_error_analysis.SubgoalErrorAnalysis(llm_client, config=None)[source]

Bases: ErrorAnalysis

Abstract base class for subgoal-based error analysis implementations.

This class provides common functionality for analyzing errors in subgoal validations, including:

  • LLM client management

  • Helper methods for processing judge trial explanations

  • Splitting validations by completeness status

  • Concurrent processing of data samples using thread pool executors

Subclasses implement specific algorithms (unsupervised, semi-supervised) for identifying and clustering errors.

Parameters:

  • llm_client (LLMClient) – The LLM client used for error analysis operations.

  • config (Optional[Dict[str, Any]]) –

    Optional configuration dictionary. Supported keys:

    • max_workers: Maximum number of concurrent workers (default: 20)

analyze_batch(data_samples)[source]

Analyze a batch of data samples and return results.

This is a synchronous wrapper around analyze_batch_async(). Use this method when calling from a synchronous context. If you’re already in an async context (i.e., inside an async function with an event loop running), use analyze_batch_async() instead.

Parameters:

data_samples (List[SubgoalErrorAnalysisDataSample]) – a List of SubgoalErrorAnalysisDataSample objects to analyze.

Return type:

SubgoalErrorAnalysisResult

Returns:

an SubgoalErrorAnalysisResult object with clustered errors.

async analyze_batch_async(data_samples)[source]

Performs error analysis on a batch of data samples.

Use this method when calling from an async context (i.e., when an event loop is already running). If you are calling from a synchronous context, use the analyze_batch() method instead

Returns the SubgoalErrorAnalysisResult containing the clustered error types with their associated subgoal validations and the rest of subgoal validations that don’t have errors.

Parameters:

data_samples (List[SubgoalErrorAnalysisDataSample]) – a List of SubgoalErrorAnalysisDataSample objects to perform error analysis on. Each data sample contains multiple subgoal validations.

Return type:

SubgoalErrorAnalysisResult

Returns:

an SubgoalErrorAnalysisResult object containing the error analysis results:

  • analyzed_validations_clustered_by_errors: Dictionary mapping clustered error types to lists of incomplete subgoal validations exhibiting those errors

  • completed_subgoal_validations: List of subgoal validations that were successfully completed without errors

agent_inspect.tools.error_analysis.tool_call_error_analysis module

class agent_inspect.tools.error_analysis.tool_call_error_analysis.ToolCallErrorAnalysis(config=None)[source]

Bases: ErrorAnalysis

Abstract base class for tool call-based error analysis implementations.

This class provides common functionality for analyzing errors in tool call validations. Tool call analysis focuses on lower-level tool execution errors such as:

  • Wrong tool selection

  • Incorrect tool inputs

  • Incorrect tool output handling

Subclasses implement specific algorithms (semi-supervised, deterministic) for identifying and clustering these errors.

Parameters:

  • config (Optional[Dict[str, Any]]) –

    Optional configuration dictionary. Supported keys:

    • max_workers: Maximum number of concurrent workers (default: 20)

  • error_clusters – Optional list of predefined error clusters for classification.

analyze_batch(data_samples)[source]

Analyze a batch of data samples and return results.

This is a synchronous wrapper around analyze_batch_async(). Use this method when calling from a synchronous context. If you’re already in an async context (i.e., inside an async function with an event loop running), use analyze_batch_async() instead.

Parameters:

data_samples (List[ToolCallErrorAnalysisDataSample]) – a List of ToolCallErrorAnalysisDataSample objects to analyze.

Return type:

ToolCallErrorAnalysisResult

Returns:

an ToolCallErrorAnalysisResult object with clustered errors.

async analyze_batch_async(data_samples)[source]

Async version of analyze_batch(). Performs error analysis on a batch of data samples with tool call validations.

Use this method when calling from an async context (i.e., when an event loop is already running).

Returns the ToolCallErrorAnalysisResult containing the clustered error types with their associated tool call validations.

Parameters:

data_samples (List[ToolCallErrorAnalysisDataSample]) – a List of ToolCallErrorAnalysisDataSample objects to perform error analysis on. Each data sample contains multiple tool call validations.

Return type:

ToolCallErrorAnalysisResult

Returns:

an ToolCallErrorAnalysisResult object containing the error analysis results:

  • analyzed_validations_clustered_by_errors: Dictionary mapping clustered error types to lists of incomplete tool call validations exhibiting those errors.

agent_inspect.tools.error_analysis.unsupervised_subgoal_error_analysis module

class agent_inspect.tools.error_analysis.unsupervised_subgoal_error_analysis.UnsupervisedSubgoalErrorAnalysis(llm_client, config=None)[source]

Bases: SubgoalErrorAnalysis

Unsupervised error analysis for subgoal validations.

This class implements an unsupervised learning approach to error analysis that: 1) Identifies low-level errors from individual subgoal validation failures 2) Dynamically clusters similar low-level errors into error categories 3) Error clusters are generated on-the-fly based on the low-level errors, without relying on any predefined error categories.

The analysis follows a two-step process:

  • Step 1: For each failed subgoal validation, summarize the issues raised by the judge trials into a concise base error description.

  • Step 2: Cluster all base errors semantically using an LLM to identify common failure patterns.

This approach is useful when you want to discover error patterns without prior assumptions about what kinds of errors might occur.

Parameters:

  • llm_client (LLMClient) – The LLMClient for performing error analysis operations.

  • config (Optional[Dict[str, Any]]) –

    Optional configuration dictionary. Supported keys:

    • max_workers: Maximum number of concurrent workers (default: 20)

Example:
>>> from agent_inspect.models.tools import SubgoalErrorAnalysisDataSample
>>> from agent_inspect.tools.error_analysis import UnsupervisedSubgoalErrorAnalysis
>>> from agent_inspect.clients.azure_openai_client import AzureOpenAIClient
>>>
>>> # Prepare your data samples
>>> data_samples: List[SubgoalErrorAnalysisDataSample] = [...]
>>>
>>> # Initialize the analyzer
>>> llm_client = AzureOpenAIClient(model="gpt-4.1", max_tokens=4096)
>>> analyzer = UnsupervisedSubgoalErrorAnalysis(
...     llm_client=llm_client,
...     config={"max_workers": 10}
... )
>>>
>>> # Run error analysis
>>> results = analyzer.analyze_batch(data_samples)
>>> error_categories = list(results.analyzed_validations_clustered_by_errors.keys())
>>> print(f"Discovered error categories: {error_categories}")

Module contents