agent_inspect.models.tools package

Submodules

agent_inspect.models.tools.analysis_models module

class agent_inspect.models.tools.analysis_models.AnalyzedSubgoalValidation(data_sample_id, base_error, subgoal_validation, *, agent_run_id=None)[source]

Bases: AnalyzedValidation

Represents the error analysis result for a single subgoal validation within a data sample.

Parameters:

  • data_sample_id (int)

  • base_error (str | None)

  • subgoal_validation (SubGoalValidationResult)

  • agent_run_id (int | None)

subgoal_validation: SubGoalValidationResult

The subgoal validation result being analyzed.

class agent_inspect.models.tools.analysis_models.AnalyzedToolValidation(data_sample_id, base_error, tool_call_validation, *, agent_run_id=None)[source]

Bases: AnalyzedValidation

Represents the error analysis result for a single tool call validation within a data sample.

Parameters:

  • data_sample_id (int)

  • base_error (str | None)

  • tool_call_validation (ToolCallValidationResult)

  • agent_run_id (int | None)

tool_call_validation: ToolCallValidationResult

The tool call validation result being analyzed.

class agent_inspect.models.tools.analysis_models.AnalyzedValidation(data_sample_id, base_error, *, agent_run_id=None)[source]

Bases: object

Base class for analyzed validation results.

Attributes:

data_sample_id: The unique identifier of the data sample

base_error: A description of the identified error

agent_run_id: Optional unique identifier for the agent run

Parameters:

  • data_sample_id (int)

  • base_error (str | None)

  • agent_run_id (int | None)

agent_run_id: Optional[int] = None

Unique identifier for the agent run associated with this data sample.

base_error: Optional[str]

A description of the identified error in the validation.

data_sample_id: int

The unique identifier of the data sample that this result originates from.

class agent_inspect.models.tools.analysis_models.ErrorAnalysisDataSample(data_sample_id, agent_run_id=None)[source]

Bases: object

Base class for error analysis data samples.

Attributes:

data_sample_id: Unique identifier for the data sample

agent_run_id: Optional unique identifier for the agent run

Parameters:

  • data_sample_id (int)

  • agent_run_id (int | None)

agent_run_id: Optional[int] = None

Unique identifier for the agent run associated with this data sample.

data_sample_id: int

Unique identifier for the data sample. Final results of error analysis will reference this id.

class agent_inspect.models.tools.analysis_models.ErrorAnalysisResult(analyzed_validations_clustered_by_errors)[source]

Bases: Generic[T]

Base class for error analysis results.

Attributes:
analyzed_validations_clustered_by_errors: Mapping from error cluster labels

to lists of analyzed validations that exhibit those errors

Parameters:

analyzed_validations_clustered_by_errors (Dict[str, List[T]])

analyzed_validations_clustered_by_errors: Dict[str, List[TypeVar(T, bound= AnalyzedValidation)]]

A mapping from error cluster labels to lists of analyzed validations that exhibit those errors.

class agent_inspect.models.tools.analysis_models.StatisticAnalysisResult(data_sample_id, subgoal_validations, judge_expectation=None, judge_std=None, agent_run_id=None)[source]

Bases: object

Represents the statistical analysis result for a single data sample.

Parameters:

  • data_sample_id (int)

  • subgoal_validations (List[SubGoalValidationResult])

  • judge_expectation (float | None)

  • judge_std (float | None)

  • agent_run_id (int | None)

agent_run_id: Optional[int] = None

Unique identifier for the agent run associated with this data sample.

data_sample_id: int

The unique identifier of the data sample that this statistic analysis result corresponds to.

judge_expectation: Optional[float] = None

The computed expectation (mean) of judge scores across all subgoals in one data sample.

judge_std: Optional[float] = None

The computed standard deviation of judge scores across all subgoals in one data sample.

subgoal_validations: List[SubGoalValidationResult]

A list of subgoal validation results to perform error analysis on.

class agent_inspect.models.tools.analysis_models.SubgoalErrorAnalysisDataSample(data_sample_id, agent_run_id=None, subgoal_validations=<factory>)[source]

Bases: ErrorAnalysisDataSample

Represents a single data sample with its associated subgoal validations.

Note: subgoal_validations is technically optional with an empty default to satisfy dataclass inheritance constraints, but in practice should always be provided.

Parameters:
subgoal_validations: List[SubGoalValidationResult]

A list of ordered subgoal validation results to perform error analysis on.

class agent_inspect.models.tools.analysis_models.SubgoalErrorAnalysisResult(analyzed_validations_clustered_by_errors, completed_subgoal_validations=<factory>)[source]

Bases: ErrorAnalysisResult[AnalyzedSubgoalValidation]

Represents the overall error analysis result for a set of data samples (subgoal-based).

Parameters:

  • analyzed_validations_clustered_by_errors (Dict[str, List[T]])

  • completed_subgoal_validations (List[AnalyzedSubgoalValidation])

completed_subgoal_validations: List[AnalyzedSubgoalValidation]

A list of analyzed subgoal validations consisting of the subgoal validations that were marked completed, and thus have no associated errors.

class agent_inspect.models.tools.analysis_models.ToolCallErrorAnalysisDataSample(data_sample_id, agent_run_id=None, tool_call_validations=<factory>)[source]

Bases: ErrorAnalysisDataSample

Represents a single data sample with its associated tool call validations.

Note: tool_call_validations is technically optional with an empty default to satisfy dataclass inheritance constraints, but in practice should always be provided.

Parameters:
tool_call_validations: List[ToolCallValidationResult]

A list of tool call validation results to perform error analysis on.

class agent_inspect.models.tools.analysis_models.ToolCallErrorAnalysisResult(analyzed_validations_clustered_by_errors)[source]

Bases: ErrorAnalysisResult[AnalyzedToolValidation]

Represents the overall error analysis result for a set of tool validation data samples.

This result only contains failed validations clustered by error type. Passed validations are not included as they don’t require error analysis.

Parameters:

analyzed_validations_clustered_by_errors (Dict[str, List[T]])

agent_inspect.models.tools.error_cluster module

Error cluster definitions for error analysis.

This module defines error cluster types used by error analysis implementations. Error clusters categorize common failure modes in agent behavior.

class agent_inspect.models.tools.error_cluster.ErrorCluster(cluster_label, description)[source]

Bases: ABC

Base class for error cluster definitions.

Error clusters represent categories of agent failures. Each cluster has: - cluster_label: A short, human-readable name for the error type - description: A detailed explanation of when this error occurs

This dataclass is frozen (immutable) to prevent accidental modification of the default cluster definitions.

Attributes:

cluster_label: Short name identifying the error type description: Detailed explanation of the error category

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str
description: str
to_dict()[source]

Convert cluster to dictionary format for LLM prompts.

Return type:

dict

Returns:

Dictionary with cluster_label and description keys

class agent_inspect.models.tools.error_cluster.FaithfulnessErrorCluster(cluster_label='Faithfulness Error (Ungrounded)', description="Agent's response contradicts or is not supported by the output of the tools it used.")[source]

Bases: ErrorCluster

Agent’s response contradicts or is not supported by the output of the tools it used.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Faithfulness Error (Ungrounded)'
description: str = "Agent's response contradicts or is not supported by the output of the tools it used."
class agent_inspect.models.tools.error_cluster.IncompleteCommunicationCluster(cluster_label='Incomplete or Vague Communication', description="Agent's response is missing necessary information or is too vague to be useful.")[source]

Bases: ErrorCluster

Agent’s response is missing necessary information or is too vague to be useful.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Incomplete or Vague Communication'
description: str = "Agent's response is missing necessary information or is too vague to be useful."
class agent_inspect.models.tools.error_cluster.IncorrectToolInputCluster(cluster_label='Incorrect Tool Input', description='Agent called a tool with incorrect, missing, or malformed input parameters.')[source]

Bases: ErrorCluster

Agent called a tool with incorrect, missing, or malformed input parameters.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Incorrect Tool Input'
description: str = 'Agent called a tool with incorrect, missing, or malformed input parameters.'
class agent_inspect.models.tools.error_cluster.IncorrectToolOutputHandlingCluster(cluster_label='Incorrect Tool Output Handling', description='Agent failed to correctly interpret, parse, or use the output returned by a tool.')[source]

Bases: ErrorCluster

Agent failed to correctly interpret, parse, or use the output returned by a tool.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Incorrect Tool Output Handling'
description: str = 'Agent failed to correctly interpret, parse, or use the output returned by a tool.'
class agent_inspect.models.tools.error_cluster.InstructionFollowingErrorCluster(cluster_label='Instruction Following Error', description='Agent failed to follow explicit instructions or constraints from the user.')[source]

Bases: ErrorCluster

Agent failed to follow explicit instructions or constraints from the user.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Instruction Following Error'
description: str = 'Agent failed to follow explicit instructions or constraints from the user.'
class agent_inspect.models.tools.error_cluster.LogicalReasoningErrorCluster(cluster_label='Logical Reasoning Error', description="Agent's reasoning process was flawed, got stuck in a loop, or gave up before the task was complete.")[source]

Bases: ErrorCluster

Agent’s reasoning process was flawed, got stuck in a loop, or gave up before the task was complete.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Logical Reasoning Error'
description: str = "Agent's reasoning process was flawed, got stuck in a loop, or gave up before the task was complete."
class agent_inspect.models.tools.error_cluster.MissedToolCallCluster(cluster_label='Missed Tool Call', description='Agent failed to call a necessary tool to complete the subgoal.')[source]

Bases: ErrorCluster

Agent failed to call a necessary tool to complete the subgoal.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Missed Tool Call'
description: str = 'Agent failed to call a necessary tool to complete the subgoal.'
class agent_inspect.models.tools.error_cluster.UnclassifiedErrorCluster(cluster_label='Unclassified', description='Errors that do not fit into any of the other predefined error categories. ')[source]

Bases: ErrorCluster

Errors that do not fit into any of the other predefined categories.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Unclassified'
description: str = 'Errors that do not fit into any of the other predefined error categories. '

Also use this when you cannot identify a clear agent behavior or decision that caused the failure.

class agent_inspect.models.tools.error_cluster.WrongToolSelectionCluster(cluster_label='Wrong Tool Selection', description='Agent selected the wrong tool for the task or called tools in an incorrect sequence.')[source]

Bases: ErrorCluster

Agent selected the wrong tool for the task or called tools in an incorrect sequence.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Wrong Tool Selection'
description: str = 'Agent selected the wrong tool for the task or called tools in an incorrect sequence.'

Module contents

class agent_inspect.models.tools.AnalyzedSubgoalValidation(data_sample_id, base_error, subgoal_validation, *, agent_run_id=None)[source]

Bases: AnalyzedValidation

Represents the error analysis result for a single subgoal validation within a data sample.

Parameters:

  • data_sample_id (int)

  • base_error (str | None)

  • subgoal_validation (SubGoalValidationResult)

  • agent_run_id (int | None)

base_error: Optional[str]

A description of the identified error in the validation.

data_sample_id: int

The unique identifier of the data sample that this result originates from.

subgoal_validation: SubGoalValidationResult

The subgoal validation result being analyzed.

class agent_inspect.models.tools.AnalyzedToolValidation(data_sample_id, base_error, tool_call_validation, *, agent_run_id=None)[source]

Bases: AnalyzedValidation

Represents the error analysis result for a single tool call validation within a data sample.

Parameters:

  • data_sample_id (int)

  • base_error (str | None)

  • tool_call_validation (ToolCallValidationResult)

  • agent_run_id (int | None)

base_error: Optional[str]

A description of the identified error in the validation.

data_sample_id: int

The unique identifier of the data sample that this result originates from.

tool_call_validation: ToolCallValidationResult

The tool call validation result being analyzed.

class agent_inspect.models.tools.AnalyzedValidation(data_sample_id, base_error, *, agent_run_id=None)[source]

Bases: object

Base class for analyzed validation results.

Attributes:

data_sample_id: The unique identifier of the data sample

base_error: A description of the identified error

agent_run_id: Optional unique identifier for the agent run

Parameters:

  • data_sample_id (int)

  • base_error (str | None)

  • agent_run_id (int | None)

agent_run_id: Optional[int] = None

Unique identifier for the agent run associated with this data sample.

base_error: Optional[str]

A description of the identified error in the validation.

data_sample_id: int

The unique identifier of the data sample that this result originates from.

class agent_inspect.models.tools.ErrorAnalysisDataSample(data_sample_id, agent_run_id=None)[source]

Bases: object

Base class for error analysis data samples.

Attributes:

data_sample_id: Unique identifier for the data sample

agent_run_id: Optional unique identifier for the agent run

Parameters:

  • data_sample_id (int)

  • agent_run_id (int | None)

agent_run_id: Optional[int] = None

Unique identifier for the agent run associated with this data sample.

data_sample_id: int

Unique identifier for the data sample. Final results of error analysis will reference this id.

class agent_inspect.models.tools.ErrorAnalysisResult(analyzed_validations_clustered_by_errors)[source]

Bases: Generic[T]

Base class for error analysis results.

Attributes:
analyzed_validations_clustered_by_errors: Mapping from error cluster labels

to lists of analyzed validations that exhibit those errors

Parameters:

analyzed_validations_clustered_by_errors (Dict[str, List[T]])

analyzed_validations_clustered_by_errors: Dict[str, List[TypeVar(T, bound= AnalyzedValidation)]]

A mapping from error cluster labels to lists of analyzed validations that exhibit those errors.

class agent_inspect.models.tools.ErrorCluster(cluster_label, description)[source]

Bases: ABC

Base class for error cluster definitions.

Error clusters represent categories of agent failures. Each cluster has: - cluster_label: A short, human-readable name for the error type - description: A detailed explanation of when this error occurs

This dataclass is frozen (immutable) to prevent accidental modification of the default cluster definitions.

Attributes:

cluster_label: Short name identifying the error type description: Detailed explanation of the error category

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str
description: str
to_dict()[source]

Convert cluster to dictionary format for LLM prompts.

Return type:

dict

Returns:

Dictionary with cluster_label and description keys

class agent_inspect.models.tools.FaithfulnessErrorCluster(cluster_label='Faithfulness Error (Ungrounded)', description="Agent's response contradicts or is not supported by the output of the tools it used.")[source]

Bases: ErrorCluster

Agent’s response contradicts or is not supported by the output of the tools it used.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Faithfulness Error (Ungrounded)'
description: str = "Agent's response contradicts or is not supported by the output of the tools it used."
class agent_inspect.models.tools.IncompleteCommunicationCluster(cluster_label='Incomplete or Vague Communication', description="Agent's response is missing necessary information or is too vague to be useful.")[source]

Bases: ErrorCluster

Agent’s response is missing necessary information or is too vague to be useful.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Incomplete or Vague Communication'
description: str = "Agent's response is missing necessary information or is too vague to be useful."
class agent_inspect.models.tools.IncorrectToolInputCluster(cluster_label='Incorrect Tool Input', description='Agent called a tool with incorrect, missing, or malformed input parameters.')[source]

Bases: ErrorCluster

Agent called a tool with incorrect, missing, or malformed input parameters.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Incorrect Tool Input'
description: str = 'Agent called a tool with incorrect, missing, or malformed input parameters.'
class agent_inspect.models.tools.IncorrectToolOutputHandlingCluster(cluster_label='Incorrect Tool Output Handling', description='Agent failed to correctly interpret, parse, or use the output returned by a tool.')[source]

Bases: ErrorCluster

Agent failed to correctly interpret, parse, or use the output returned by a tool.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Incorrect Tool Output Handling'
description: str = 'Agent failed to correctly interpret, parse, or use the output returned by a tool.'
class agent_inspect.models.tools.InstructionFollowingErrorCluster(cluster_label='Instruction Following Error', description='Agent failed to follow explicit instructions or constraints from the user.')[source]

Bases: ErrorCluster

Agent failed to follow explicit instructions or constraints from the user.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Instruction Following Error'
description: str = 'Agent failed to follow explicit instructions or constraints from the user.'
class agent_inspect.models.tools.LogicalReasoningErrorCluster(cluster_label='Logical Reasoning Error', description="Agent's reasoning process was flawed, got stuck in a loop, or gave up before the task was complete.")[source]

Bases: ErrorCluster

Agent’s reasoning process was flawed, got stuck in a loop, or gave up before the task was complete.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Logical Reasoning Error'
description: str = "Agent's reasoning process was flawed, got stuck in a loop, or gave up before the task was complete."
class agent_inspect.models.tools.MissedToolCallCluster(cluster_label='Missed Tool Call', description='Agent failed to call a necessary tool to complete the subgoal.')[source]

Bases: ErrorCluster

Agent failed to call a necessary tool to complete the subgoal.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Missed Tool Call'
description: str = 'Agent failed to call a necessary tool to complete the subgoal.'
class agent_inspect.models.tools.StatisticAnalysisResult(data_sample_id, subgoal_validations, judge_expectation=None, judge_std=None, agent_run_id=None)[source]

Bases: object

Represents the statistical analysis result for a single data sample.

Parameters:

  • data_sample_id (int)

  • subgoal_validations (List[SubGoalValidationResult])

  • judge_expectation (float | None)

  • judge_std (float | None)

  • agent_run_id (int | None)

agent_run_id: Optional[int] = None

Unique identifier for the agent run associated with this data sample.

data_sample_id: int

The unique identifier of the data sample that this statistic analysis result corresponds to.

judge_expectation: Optional[float] = None

The computed expectation (mean) of judge scores across all subgoals in one data sample.

judge_std: Optional[float] = None

The computed standard deviation of judge scores across all subgoals in one data sample.

subgoal_validations: List[SubGoalValidationResult]

A list of subgoal validation results to perform error analysis on.

class agent_inspect.models.tools.SubgoalErrorAnalysisDataSample(data_sample_id, agent_run_id=None, subgoal_validations=<factory>)[source]

Bases: ErrorAnalysisDataSample

Represents a single data sample with its associated subgoal validations.

Note: subgoal_validations is technically optional with an empty default to satisfy dataclass inheritance constraints, but in practice should always be provided.

Parameters:
data_sample_id: int

Unique identifier for the data sample. Final results of error analysis will reference this id.

subgoal_validations: List[SubGoalValidationResult]

A list of ordered subgoal validation results to perform error analysis on.

class agent_inspect.models.tools.SubgoalErrorAnalysisResult(analyzed_validations_clustered_by_errors, completed_subgoal_validations=<factory>)[source]

Bases: ErrorAnalysisResult[AnalyzedSubgoalValidation]

Represents the overall error analysis result for a set of data samples (subgoal-based).

Parameters:

  • analyzed_validations_clustered_by_errors (Dict[str, List[T]])

  • completed_subgoal_validations (List[AnalyzedSubgoalValidation])

analyzed_validations_clustered_by_errors: Dict[str, List[TypeVar(T, bound= AnalyzedValidation)]]

A mapping from error cluster labels to lists of analyzed validations that exhibit those errors.

completed_subgoal_validations: List[AnalyzedSubgoalValidation]

A list of analyzed subgoal validations consisting of the subgoal validations that were marked completed, and thus have no associated errors.

class agent_inspect.models.tools.ToolCallErrorAnalysisDataSample(data_sample_id, agent_run_id=None, tool_call_validations=<factory>)[source]

Bases: ErrorAnalysisDataSample

Represents a single data sample with its associated tool call validations.

Note: tool_call_validations is technically optional with an empty default to satisfy dataclass inheritance constraints, but in practice should always be provided.

Parameters:
data_sample_id: int

Unique identifier for the data sample. Final results of error analysis will reference this id.

tool_call_validations: List[ToolCallValidationResult]

A list of tool call validation results to perform error analysis on.

class agent_inspect.models.tools.ToolCallErrorAnalysisResult(analyzed_validations_clustered_by_errors)[source]

Bases: ErrorAnalysisResult[AnalyzedToolValidation]

Represents the overall error analysis result for a set of tool validation data samples.

This result only contains failed validations clustered by error type. Passed validations are not included as they don’t require error analysis.

Parameters:

analyzed_validations_clustered_by_errors (Dict[str, List[T]])

class agent_inspect.models.tools.UnclassifiedErrorCluster(cluster_label='Unclassified', description='Errors that do not fit into any of the other predefined error categories. ')[source]

Bases: ErrorCluster

Errors that do not fit into any of the other predefined categories.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Unclassified'
description: str = 'Errors that do not fit into any of the other predefined error categories. '

Also use this when you cannot identify a clear agent behavior or decision that caused the failure.

class agent_inspect.models.tools.WrongToolSelectionCluster(cluster_label='Wrong Tool Selection', description='Agent selected the wrong tool for the task or called tools in an incorrect sequence.')[source]

Bases: ErrorCluster

Agent selected the wrong tool for the task or called tools in an incorrect sequence.

Parameters:

  • cluster_label (str)

  • description (str)

cluster_label: str = 'Wrong Tool Selection'
description: str = 'Agent selected the wrong tool for the task or called tools in an incorrect sequence.'