gen_ai_hub.evaluations.utils.gen_utils module

gen_ai_hub.evaluations.utils.gen_utils.update_variable_mapping(variable_mapping: dict, prefix_key: str, variable_mapping_dict: dict) → dict

gen_ai_hub.evaluations.utils.gen_utils.get_accumulated_config_data(evaluation_configs_data: List[_EvaluationConfigData]) → _EvaluationConfigData

gen_ai_hub.evaluations.utils.gen_utils.set_model_details_from_run_configs(orch_config) → Tuple[str, str] | None

Sets the model name and version from the run data if available. If not available, it returns None.

gen_ai_hub.evaluations.utils.gen_utils.create_model_versions_map_from_orch_configs(orchestration_configs_data: List[dict], error_collector: ValidationCollector) → Dict[str, List[str]] | None

gen_ai_hub.evaluations.utils.gen_utils.parse_model_filter_list(param, error_collector: ValidationCollector) → List

Parse and return model filter list from a param.

gen_ai_hub.evaluations.utils.gen_utils.build_model_versions_map(model_list) → Dict[str, List[str]]

Builds a map of model names to their versions.

gen_ai_hub.evaluations.utils.gen_utils.create_model_versions_map_from_configuration_param_bindings(param_bindings, error_collector: ValidationCollector) → Tuple[Dict[str, List[str]], str | None]

gen_ai_hub.evaluations.utils.gen_utils.create_model_versions_map_from_custom_metric_config(custom_metric_config_data) → Dict[str, List[str]]

gen_ai_hub.evaluations.utils.gen_utils.select_model_details_randomly(orchestration_config_data: List[dict], error_collector: ValidationCollector) → Tuple[str, str] | None

Selects at random, model name and version from the list of model names and versions provided by the users run data.

gen_ai_hub.evaluations.utils.gen_utils.update_test_orch_config(model_name, model_version, error_collector: ValidationCollector) → dict | None

gen_ai_hub.evaluations.utils.gen_utils.has_filter_key(orch_config: dict) → bool

Check if the ‘filtering’ key is present in the orchestration config.

Parameters:

orch_config (dict) – Orchestration configuration dictionary.

Returns:

True if filtering key is present, False otherwise.

Return type:

bool

gen_ai_hub.evaluations.utils.gen_utils.get_filter_config(orch_config: dict) → dict

Extract the filtering configuration from the orchestration config.

Parameters:

orch_config (dict) – Orchestration configuration dictionary.

Returns:

Filtering configuration dictionary, or empty dict if not present.

Return type:

dict

gen_ai_hub.evaluations.utils.gen_utils.check_if_content_filter_provider_supported(orch_config: dict, error_collector: ValidationCollector) → bool

Validates if all filters in the filtering module configuration are of supported types.

Parameters:

• orch_config (dict) – Orchestration configuration dictionary.

• error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

True if all filters are supported or no filtering is configured, False otherwise.

Return type:

bool

gen_ai_hub.evaluations.utils.gen_utils.remove_filter_metrics_if_provider_not_supported(orchestration_config_data: List[dict], metrics: List[str], error_collector: ValidationCollector) → None

Removes content filter-related metric IDs from the metrics list if the content filter provider is not supported for any of the runs in orchestration_config_data.

gen_ai_hub.evaluations.utils.gen_utils.create_custom_metric_name(custom_metric_config: dict, error_collector: ValidationCollector) → str

Creates a custom metric name based on the provided custom metric configuration.

Parameters:

• custom_metric_config – Dictionary containing metric configuration.

• error_collector – ValidationCollector instance for collecting validation errors.

Returns:

A string representing the custom metric name.

Raises:

ValidationError – If required fields are missing or invalid.

gen_ai_hub.evaluations.utils.gen_utils.validate_metric_name(metric: str, all_supported_metrics: List, error_collector: ValidationCollector) → None

Validates if metrics name is not empty and the value actually exists in the list of supported metrics

gen_ai_hub.evaluations.utils.gen_utils.count_user_prompts_from_template_list(template_list) → int

gen_ai_hub.evaluations.utils.gen_utils.get_template_list_from_orch_config(orch_config) → List

gen_ai_hub.evaluations.utils.gen_utils.validate_prompts_in_templating_module(orchestration_config_data: List[dict], metric: str, error_collector: ValidationCollector) → None

Checks whether the templating config provided in the Orchestration Config has exactly one user prompt

gen_ai_hub.evaluations.utils.gen_utils.get_custom_metric_ids_from_input(custom_metric_config_data: List[dict], error_collector: ValidationCollector) → list[str]

Retrieve custom metric ids from the file data provided by the user.

Parameters:

• custom_metric_config_data (List[dict]) – List of dictionaries containing custom metric definitions.

• error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

List of custom metric ids.

Return type:

list[str]

gen_ai_hub.evaluations.utils.gen_utils.check_if_metric_is_defined(metrics: List[str], metric_templates: List[dict], error_collector: ValidationCollector) → None

gen_ai_hub.evaluations.utils.gen_utils.is_value_in_json(value, name, data: dict[str, str]) → bool

Check if a value matches a name or exists in a mapping dictionary.

Parameters:

• value (Any) – The value to search for.

• name (str) – The name to compare against.

• data (dict[str, str]) – Dictionary to search in (keys or values).

Returns:

True if value matches name or is found in data, False otherwise.

Return type:

bool

gen_ai_hub.evaluations.utils.gen_utils.validate_metrics(metrics: List[str], metric_templates: List[dict], orchestration_config_data: List[dict], error_collector: ValidationCollector) → None

Validates if metrics list is empty or metric name is invalid

gen_ai_hub.evaluations.utils.gen_utils.list_prompt_variables(format_string: str) → list[str]

Get all fields (parameters) of the form {{ ?param_name }} from the template. Optionally return the raw field names without stripping spaces and ‘?’.

gen_ai_hub.evaluations.utils.gen_utils.extract_dataset_columns(template_variables) → List[str]

Extracts column names from the template variables provided. If the value is a list, extracts column names from the first rows; else, extracts from the template_variables directly.

gen_ai_hub.evaluations.utils.gen_utils.get_prompt_variables_from_orch_config(orch_config: dict) → Set[str]

gen_ai_hub.evaluations.utils.gen_utils.get_grounding_config_from_orch_config(orch_config) → dict

Extracts the grounding configuration from the orchestration configuration. :param orch_config: Orchestration configuration. :type orch_config: dict

Returns:

Grounding configuration if present, otherwise an empty dictionary.

Return type:

dict

gen_ai_hub.evaluations.utils.gen_utils.get_grounding_output_param_key(orch_config: dict) → str

Determines the correct key to extract the grounding output parameter based on the API version. :param orch_config: Orchestration configuration. :type orch_config: dict

Returns:

The key to extract the grounding output parameter.

Return type:

str

gen_ai_hub.evaluations.utils.gen_utils.get_defaults(orchestration_configuration: dict) → dict

Returns the default field from the orchestration configuration.

gen_ai_hub.evaluations.utils.gen_utils.get_mapped_value_if_exists(key, mapping_keys, variable_mapping, dataset_columns) → str

Gets the first valid mapped value from the list of keys if it exists in variable mapping, else returns the first key

gen_ai_hub.evaluations.utils.gen_utils.validate_variable_mapping_of_prompts(orchestration_config_data: list, dataset_data: List[dict], variable_mapping: dict, error_collector: ValidationCollector) → None

Validates the variable mapping for prompts with a zero-tolerance failure threshold.

Parameters:

• orchestration_config_data (list) – Orchestration run configuration

• dataset_data (dict) – Dataset rows to validate

• variable_mapping (dict) – The variable mapping provided in the input configuration.

Raises:

ValidationError – If any prompts variable mapping is invalid or does not exist in the dataset.

gen_ai_hub.evaluations.utils.gen_utils.validate_all_metrics_mapping(variable_mapping: dict, dataset_columns: list, error_collector: ValidationCollector) → None

Validates the variable mapping for ‘all_metrics’ with a zero-tolerance failure threshold.

Parameters:

• variable_mapping (dict) – The variable mapping provided in the input configuration.

• dataset_columns (list) – List of column names in the dataset.

Raises:

ValidationError – If any ‘all_metrics’ mapping is invalid or the direct column does not exist in the dataset.

gen_ai_hub.evaluations.utils.gen_utils.extract_metrics_variables(metric_templates, metric_name: str = None) → Set

Extracts unique set of variables from the ‘variables’ key.

gen_ai_hub.evaluations.utils.gen_utils.validate_individual_metrics(metrics: list[str], variable_mapping: dict, dataset_columns: list, metric_dependent_variables: set, error_collector: ValidationCollector) → None

Validates the variable mapping for any metric mapping with a zero-tolerance failure threshold.

Parameters:

• metrics (list) – List of metrics provided in the input configuration.

• variable_mapping (dict) – The variable mapping provided in the input configuration.

• dataset_columns (list) – List of column names in the dataset.

• metric_dependent_variables (set) – Set of dependent variables for all metrics

Raises:

ValidationError – If any metric mapping is invalid or the direct column does not exist in the dataset.

gen_ai_hub.evaluations.utils.gen_utils.validate_variable_mapping_of_metrics(metrics: list[str], metric_templates: list[dict], dataset_data: List[dict], variable_mapping: dict, error_collector: ValidationCollector) → None

Validates variable mapping of metrics with tolerance to zero failure threshold

Parameters:

• metrics – List of metrics provided in the input config

• metric_templates (list[dict]) – Metric templates information resolved from Metric Management Service

• dataset_data – Dataset rows to validate

• variable_mapping – variable mapping provided in input config

Returns:

Validates and throws validation error even if one variable mapping related to metrics is invalid.

gen_ai_hub.evaluations.utils.gen_utils.flatten_prompt_configuration(prompt_config: dict) → str

Flatten a nested prompt configuration dictionary into a readable string.

gen_ai_hub.evaluations.utils.gen_utils.validate_individual_custom_metrics(variable_mapping: dict, dataset_columns: list, custom_metric_ids: list, custom_metric_variables: set, error_collector: ValidationCollector) → None

Validates the variable mapping for any metric mapping with a zero-tolerance failure threshold.

gen_ai_hub.evaluations.utils.gen_utils.handle_missing_dependent_variables_in_dataset(dataset_data: List[dict], metrics: list[str], metric_templates: list[dict], variable_mapping: dict, error_collector: ValidationCollector) → None

validates whether all the dependent variables for the metrics list are either directly present as columns in dataset or a variable mapping is provided :param dataset_data: Dataset rows to validate (list of row dictionaries) :type dataset_data: List[dict] :param metrics: List of metrics provided in the input configuration. :type metrics: list[str] :param metric_templates: Metric templates information resolved from Metric Management Service :type metric_templates: list[dict] :param variable_mapping: The variable mapping provided in the input configuration. :type variable_mapping: dict

Raises:

ValidationError – If any dependent variable is missing in the dataset and the variable mapping is invalid for that metric.

gen_ai_hub.evaluations.utils.gen_utils.populate_dataset_data_if_data_missing(dataset_data: list, variable_mapped_key, error_collector: ValidationCollector) → None

Validates and Populates the dataset_data with missing data fields if golden truth is present and throws error if dataset is partially filled

gen_ai_hub.evaluations.utils.gen_utils.populate_dataset_data_if_single_schema_provided(dataset_data, variable_mapping, collector) → None

Populates the dataset_data with missing json schema column entries across rows of dataset_data

gen_ai_hub.evaluations.utils.gen_utils.handle_json_schema_match(metrics: list[str], dataset_data: list, variable_mapping: dict, error_collector: ValidationCollector) → None

gen_ai_hub.evaluations.utils.gen_utils.validate_language_code_and_data_population(dataset_data: list, variable_mapping: dict, error_collector: ValidationCollector) → None

Populates the dataset_data with missing language column entries across rows of dataset_data

gen_ai_hub.evaluations.utils.gen_utils.handle_language_match(metrics: list, dataset_data: list, variable_mapping: dict, error_collector: ValidationCollector) → None

gen_ai_hub.evaluations.utils.gen_utils.populate_dataset_data_if_single_reference_provided(dataset_data: List[dict], variable_mapping: dict, collector) → None

Populates the dataset_data with missing reference column entries across rows of

dataset_data for only all metrics case where a golden reference is present

Parameters:

• dataset_data (List[dict]) – Dataset rows to validate (list of row dictionaries)

• variable_mapping (dict) – The variable mapping provided in the input configuration.

gen_ai_hub.evaluations.utils.gen_utils.populate_dataset_data_if_individual_metric_reference_provided(dataset_data: List[dict], variable_mapping: dict, metrics: list, error_collector: ValidationCollector) → None

populates reference value across all rows of dataset_data if individual metric reference is provided and is different than all metrics reference provided. This population happens if the provided reference is a golden instance

Parameters:

• dataset_data (List[dict]) – Dataset rows to validate (list of row dictionaries)

• variable_mapping (dict) – The variable mapping provided in the input configuration.

• metrics (list) – List of metrics provided in the input configuration.

gen_ai_hub.evaluations.utils.gen_utils.handle_reference_missing_rows(dataset_data: List[dict], variable_mapping: dict, metrics: list, error_collector: ValidationCollector) → None

validates whether the reference columns in the rows are missing in the dataset for all metrics and for each individual metrics

Parameters:

• dataset_data (List[dict]) – Dataset rows to validate (list of row dictionaries)

• variable_mapping (dict) – The variable mapping provided in the input configuration.

• metrics (list) – List of metrics provided in the input configuration.

Raises:

ValidationError – If any required variable mapping is invalid or the default column does not exist in the dataset.

gen_ai_hub.evaluations.utils.gen_utils.update_artifact_dict(artifact_reference: ArtifactSource, artifact_dict_count: dict) → None

gen_ai_hub.evaluations.utils.gen_utils.resolve_orchestration_config_v2(template_data: List[PromptTemplate], llm: LLMModelDetails) → dict