gen_ai_hub.evaluations.models.evaluation_run module

class gen_ai_hub.evaluations.models.evaluation_run.EvaluationRun(run_id: str, execution_id: str, ai_core_client: AICoreV2Client, configuration_id: str = None, artifact_id: str = None, resource_group: str = None, object_store_credentials: _AWSObjectStoreData = None, metrics_list: List[str] = None)

Bases: object

Represents an individual EvaluationRun object and its associated context.

Parameters:

  • run_id (str) – Unique identifier for the evaluation run

  • execution_id (str) – ID of the AI Core execution

  • ai_core_client (AICoreV2Client) – AI Core client instance

  • configuration_id (str) – ID of the configuration, defaults to None

  • artifact_id (str) – ID of the artifact, defaults to None

  • resource_group (str) – Resource group name, defaults to None

  • object_store_credentials (_AWSObjectStoreData) – Object store credentials, defaults to None

  • metrics_list (List[str]) – List of metrics to evaluate, defaults to None

__init__(run_id: str, execution_id: str, ai_core_client: AICoreV2Client, configuration_id: str = None, artifact_id: str = None, resource_group: str = None, object_store_credentials: _AWSObjectStoreData = None, metrics_list: List[str] = None)
set_cached_results_data(data)

Set the cached results data from the child results class.

Parameters:

data (Any) – Results data to cache

wait_for_completion(timeout: int | None = None)

Wait for the evaluation run to complete by polling status.

Parameters:

timeout (Optional[int]) – Maximum time to wait in seconds, defaults to 3600 (1 hour)

get_current_status()

Get the current status of the evaluation run.

Returns:

Current status of the run

Return type:

Status

Raises:

ValueError – If failed to retrieve the current status

get_debug_info() → ExecutionStatusDetails

Provide debug information when execution status is FAILED or DEAD.

Returns:

Execution status details including failed pod information

Return type:

ExecutionStatusDetails

get_debug_logs()

Get the complete trace of execution logs.

Returns:

List of log entries as dictionaries

Return type:

list

results()

Get the results of the evaluation run.

Returns:

Results object for accessing completion and metric results

Return type:

Results

Raises:

ValueError – If execution is not completed

load_results_tables()

Download results from S3 and load the required table data.

Returns:

Dictionary containing completions and metrics table data

Return type:

dict

Raises:

RuntimeError – If failed to download results

class gen_ai_hub.evaluations.models.evaluation_run.Results(run: EvaluationRun)

Bases: object

Represents the Results handler for an EvaluationRun object.

This class provides methods to access completion results, metric results, and aggregated results for a specific evaluation run.

Parameters:

run (EvaluationRun) – The parent EvaluationRun object

__init__(run: EvaluationRun)
completions()

Get the completion results for the run.

Returns:

DataFrame containing completion results for the run

Return type:

pd.DataFrame

Raises:

ValueError – If error occurs while fetching completions

metrics()

Get the metric-level results for the run.

Returns:

DataFrame containing metric results for the run

Return type:

pd.DataFrame

Raises:

ValueError – If error occurs while fetching metric results

aggregations()

Get the aggregated results for the run from the tracking service.

Returns:

JSON response containing aggregated metric results

Return type:

dict

Raises:

ValueError – If error occurs while fetching aggregation results