gen_ai_hub.evaluations.client module

class gen_ai_hub.evaluations.client.EvaluationClient(base_url: str, auth_url: str = None, client_id: str = None, client_secret: str = None, cert_str: str = None, key_str: str = None, cert_file_path: str = None, key_file_path: str = None, resource_group: str = None, aws_access_key_id: str = None, aws_secret_access_key: str = None, ai_core_client: AICoreV2Client = None, orchestration_url: str = None, input_object_store_secret_name: str = None, provider_name: str = 'aws')

Bases: object

Base Client for the Evaluations service

__init__(base_url: str, auth_url: str = None, client_id: str = None, client_secret: str = None, cert_str: str = None, key_str: str = None, cert_file_path: str = None, key_file_path: str = None, resource_group: str = None, aws_access_key_id: str = None, aws_secret_access_key: str = None, ai_core_client: AICoreV2Client = None, orchestration_url: str = None, input_object_store_secret_name: str = None, provider_name: str = 'aws')

EvaluationsClient root object to be used for Evaluations.

Parameters:

  • base_url (str) – Base URL of the AI Core instance (must include /v2 suffix).

  • auth_url (str, optional) – Authentication URL used to retrieve access tokens.

  • client_id (str, optional) – OAuth client ID.

  • client_secret (str, optional) – OAuth client secret.

  • cert_str (str, optional) – X.509 certificate content as a string.

  • key_str (str, optional) – X.509 private key content as a string.

  • cert_file_path (str, optional) – File path to X.509 certificate.

  • key_file_path (str, optional) – File path to X.509 private key.

  • resource_group (str, optional) – Resource group name within the AI Core instance.

  • aws_access_key_id (str, optional) – AWS access key ID.

  • aws_secret_access_key (str, optional) – AWS secret access key.

  • ai_core_client (AICoreV2Client, optional) – Pre-configured AI Core client instance.

  • orchestration_url (str, optional) – Pre-existing orchestration deployment URL.

  • input_object_store_secret_name (str, optional) – Name of input object store secret.

  • provider_name (str, optional) – Hyperscaler provider name (e.g., “aws”).

Raises:

ValueError – If required hyperscaler provider parameters are missing.

static from_env(profile_name: str = None, **kwargs)

Alternative way to create an EvaluationClient object.

Parameter resolution precedence: 1. Explicit keyword arguments 2. Environment variables 3. Configuration file 4. VCAP_SERVICES environment variable

Parameters:

  • profile_name (str, optional) – Profile name defined in configuration.

  • kwargs – Additional parameters passed to constructor.

Returns:

Configured EvaluationClient instance.

Return type:

EvaluationClient

validate_secret_type(secret_type: str, creator_mapping: dict)
create_or_update_object_store_secret(*, context, secret_body: dict, is_default: bool, result_key: str, attr_name: str, creator_mapping: dict, replace_existing: bool, result: dict)
resolve_orchestration_deployment_url() → str

Resolves the orchestration deployment URL.

For non-default resource groups, creates a new deployment. For default resource group, attempts to discover existing deployment with the default config name using the orchestration service, or creates one if not found.

Returns:

The orchestration deployment URL.

Return type:

str

setup(input_secret_body: dict | None = None, default_secret_body: dict | None = None, replace_existing: bool = False)

One time setup function which does object store secrets creation and orchestration deployment url creation if not provided.

evaluate(evaluation_configs: List[EvaluationConfig]) → List[EvaluationRun]

Main evaluate function to create the Evaluation job

Parameters:

evaluation_configs (List[EvaluationConfig]) – A list of one or more of the EvaluationConfig objects

Returns:

A list of EvaluationRun objects, one for each EvaluationConfig provided.

Return type:

List[EvaluationRun]

list_available_models()

Method to list all the available llm models

get_system_supported_metrics() → List[str]

helper method to get the list of all supported metric ids