gen_ai_hub.evaluations.utils.aicore_utils module

gen_ai_hub.evaluations.utils.aicore_utils.generate_random_id()

generates and returns a random uuid everytime

gen_ai_hub.evaluations.utils.aicore_utils.find_configuration_id_by_name(configurations_list: List[Configuration], target_name: str)

gen_ai_hub.evaluations.utils.aicore_utils.get_all_configurations(ai_core_client: AICoreV2Client, resource_group: str, scenario_id: str) → List[Configuration]

gen_ai_hub.evaluations.utils.aicore_utils.get_running_deployments_by_configuration_id(ai_core_client: AICoreV2Client, configuration_id: str, resource_group: str) → List[Deployment]

gen_ai_hub.evaluations.utils.aicore_utils.create_deployment_by_configuration_id(ai_core_client: AICoreV2Client, configuration_id: str, resource_group: str)

gen_ai_hub.evaluations.utils.aicore_utils.create_llm_orchestration_deployment_url(ai_core_client: AICoreV2Client, resource_group: str)

creates the llm-orchestration configuration based on orchestration global scenario and then creates a deployment using that configuration

gen_ai_hub.evaluations.utils.aicore_utils.wait_for_target_status(status_fetcher: Callable[[], Any], target_status: Status, extract_url: Callable[[Any], str] | None = None, timeout: int = 1200, initial_interval: int = 120, pending_interval: int = 40) → str | None

Reusable polling function to wait until a resource reaches target_status.

Parameters:

• status_fetcher (Callable[[], Any]) – Function to get current status response

• target_status (Status) – Target status to wait for (Status enum)

• extract_url (Optional[Callable[[Any], str]]) – Optional function to extract URL from response, defaults to None

• timeout (int) – Maximum time to wait in seconds, defaults to 1200

• initial_interval (int) – Initial polling interval in seconds, defaults to 120

• pending_interval (int) – Polling interval for pending/running status in seconds, defaults to 40

Returns:

Extracted URL if extract_url is provided and status reached, None otherwise

Return type:

Optional[str]

gen_ai_hub.evaluations.utils.aicore_utils.read_data_from_artifact(object_store_credentials: _AWSObjectStoreData, object_store_secret_metadata_details: Dict[str, str], s3_file_key: str, file_type: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.build_s3_file_key(object_store_secret_metadata_details: Dict[str, str], artifact_url_relative_path: str, artifact_source: ArtifactSource)

gen_ai_hub.evaluations.utils.aicore_utils.resolve_artifact_path(artifact_source: ArtifactSource, ai_core_client: AICoreV2Client, object_store_credentials: _AWSObjectStoreData, resource_group: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.fetch_deployment_config(deployment_id: str, ai_core_client: AICoreV2Client, resource_group: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.fetch_configuration_by_id(configuration_id: str, ai_core_client: AICoreV2Client, resource_group: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.call_orchestration_service_with_v2_config(test_orch_config: dict, ai_core_client: AICoreV2Client, orchestration_deployment_url: str, resource_group: str, error_collector: ValidationCollector, proxy_client=None)

gen_ai_hub.evaluations.utils.aicore_utils.upload_file_to_aws_s3(object_store_credentials: _AWSObjectStoreData, object_store_secret_metadata_details: Dict[str, str], file_data: Any, file_key: str, file_type: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.upload_evaluation_dataset_data(evaluation_config_data: _EvaluationConfigData, object_store_credentials: _AWSObjectStoreData, object_store_secret_name: str, ai_core_client: AICoreV2Client, resource_group: str, error_collector: ValidationCollector)

Method to upload the evaluation config data using the object store secrets data passed

gen_ai_hub.evaluations.utils.aicore_utils.register_aicore_artifact(artifact_folder_path: str, ai_core_client: AICoreV2Client, resource_group: str, object_store_secret_name: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.register_aicore_configuration(aicore_artifact_id: str, ai_core_client: AICoreV2Client, resource_group: str, accumulated_config_data: _EvaluationConfigData, orchestration_url: str, dataset_file_key: str, run_ids_list: List[str], llm_model_config: str, template_config: List, orchestration_registry_config: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.register_aicore_execution(ai_core_client: AICoreV2Client, configuration_id: str, resource_group: str, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.list_available_llm_models(ai_core_client: AICoreV2Client, resource_group: str)

gen_ai_hub.evaluations.utils.aicore_utils.fetch_orchestration_config_from_registry(orchestration_registry_reference: str, ai_core_client: AICoreV2Client, error_collector: ValidationCollector)

gen_ai_hub.evaluations.utils.aicore_utils.resolve_metric_identifiers(metrics: List[MetricConfig], ai_core_client: AICoreV2Client, resource_group: str, error_collector: ValidationCollector) → List[Dict]

Resolves metric identifiers to metric template metadata.

gen_ai_hub.evaluations.utils.aicore_utils.resolve_metric_names(metric_configs_list: List[MetricConfig], error_collector: ValidationCollector)