gen_ai_hub.evaluations.utils.validation_utils module¶

gen_ai_hub.evaluations.utils.validation_utils.validate_filtered_models(configuration_param_bindings, orchestration_config_data: List[dict], error_collector: ValidationCollector)¶

gen_ai_hub.evaluations.utils.validation_utils.fetch_and_validate_orchestration_config(ai_core_client: AICoreV2Client, configuration_id: str, orchestration_config_data: List[dict], resource_group: str, error_collector: ValidationCollector)¶

gen_ai_hub.evaluations.utils.validation_utils.validate_orchestration_url_across_configs(accumulated_config_data: List[_EvaluationConfigData] | _EvaluationConfigData, orchestration_url: str, ai_core_client: AICoreV2Client, resource_group: str, error_collector: ValidationCollector, proxy_client=None)¶: wrapper function to perform validation of fetched config data in case of single vs multiple executions flow

gen_ai_hub.evaluations.utils.validation_utils.extract_deployment_id(orch_url) → str¶

gen_ai_hub.evaluations.utils.validation_utils.validate_orchestration_url(evaluation_config_data: _EvaluationConfigData, orchestration_url: str, ai_core_client: AICoreV2Client, resource_group: str, error_collector: ValidationCollector, proxy_client=None)¶: Validates if the orchestration deployment url provided via config resides in same resourceGroup as workload or not. Also validates if url is valid and orchestration deployment is not in terminal state

gen_ai_hub.evaluations.utils.validation_utils.validate_orchestration_configuration(orchestration_config_data: List[dict], error_collector: ValidationCollector)¶: Validates the Orchestration configuration provided by user

gen_ai_hub.evaluations.utils.validation_utils.validate_input_config(orchestration_config_data: List[dict], metrics: List[str], metric_templates: List[dict], error_collector: ValidationCollector)¶: Validates the input parameters of run data and metrics

gen_ai_hub.evaluations.utils.validation_utils.validate_variable_mapping_with_input_config(orchestration_config_data: List[dict], dataset_data: dict, variable_mapping: dict, metrics: List[str], metric_templates: List[dict], error_collector: ValidationCollector)¶

Validates all the required variable mappings provided in input config with a zero-tolerance failure threshold.

Parameters:

orchestration_config_data (list) – Orchestration run configuration
dataset_data (dict) – Dataset rows to validate
variable_mapping (dict) – The variable mapping provided in the input configuration.
metrics (list[str]) – List of metrics provided in the input configuration.
metric_templates (list[dict]) – Metric templates information resolved from Metric Management Service
error_collector (ValidationCollector) – To accumulate the errors occurred during the process

Raises:

ValidationError – If any required variable mapping is invalid or the default column does not exist in the dataset.

gen_ai_hub.evaluations.utils.validation_utils.validate_config_data_collection(accumulated_config_data: List[_EvaluationConfigData] | _EvaluationConfigData, error_collector: ValidationCollector)¶: wrapper function to perform validation of fetched config data in case of single vs multiple executions flow

gen_ai_hub.evaluations.utils.validation_utils.validate_merged_config_data(evaluation_config_data: _EvaluationConfigData, error_collector: ValidationCollector)¶: handles the validation of config provided from the user