gen_ai_hub.evaluations.utils.orch_config_utils module¶

gen_ai_hub.evaluations.utils.orch_config_utils.validate_mandatory_modules(orch_config: dict, error_collector: ValidationCollector, module_key: str, required_keys: list[str], config_keys: list[str]) → None¶

Validates the presence of mandatory modules and their structure.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.
module_key (str) – The key of the module to validate.
required_keys (list[str]) – List of required keys in the module.
config_keys (list[str]) – List of configuration keys to validate as dictionaries.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_orch_config_mandatory_modules(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if the outer structure of the orchestration config is valid and exists.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.get_model_name(orch_config: dict, keys: list[str], error_collector: ValidationCollector) → Any¶

Retrieves the value from orch_config using the list of keys provided. If any key in the path is missing, logs an error and returns None.

Parameters:

orch_config (dict) – The orchestration configuration JSON object.
keys (list) – List of keys representing the path to the desired field.
error_collector (ValidationCollector) – The error collector to log errors.

Returns:

The value at the specified path in orch_config, or None if any key is missing.

gen_ai_hub.evaluations.utils.orch_config_utils.validate_model_name(orch_config: dict, keys: list[str], error_collector: ValidationCollector) → None¶

Validates if llm_module_config is a dict and the model name exists.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
keys (list[str]) – List of keys representing the path to the model name.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_model_name_in_llm_module_config(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if the model configuration in llm_module_config is a dict and the model name exists.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.get_prompt_templating_config(orch_config: dict, error_collector: ValidationCollector) → None¶

Returns the prompt_templating configuration from the orchestration config.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_template_ref_absent_in_config(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if template_ref is given in templating module config and raises an error.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.get_template_key(orch_config: dict, keys: list[str], error_collector: ValidationCollector) → Any¶

Retrieves the value from orch_config using the list of keys provided.

If any key in the path is missing, logs an error and returns None.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
keys (list[str]) – List of keys representing the path to the desired field.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

The value at the specified path in orch_config, or None if any key is missing.

Return type:

Any | None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_if_template_list_is_empty(orch_config: dict, keys: list[str], error_collector: ValidationCollector) → None¶

Validates if template list exists and is not empty.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
keys (list[str]) – List of keys representing the path to the template list.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_if_template_list_is_empty_in_templating_module_config(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if template list exists and is not empty in the templating module config.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.get_template_list_from_orch_config(orch_config: dict) → list¶

Returns the template list from the orchestration config.

Parameters:: orch_config (dict) – The orchestration configuration dictionary.
Returns:: The template list from the orchestration config.
Return type:: list

gen_ai_hub.evaluations.utils.orch_config_utils.validate_if_content_inside_template_is_empty_in_templating_module_config(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if the template list is an array and is a valid dict and content exists in template.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_if_image_url_is_provided_in_content_type_inside_templating_module_config(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if inside the template list of templating_module_config if it has image_url type inside the content.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_if_grounding_output_present_in_prompt_variables(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if the grounding output parameter is present in prompt variables.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_if_all_grounding_input_params_present_in_prompt_variables(orch_config: dict, error_collector: ValidationCollector) → None¶

Validates if all the input params of the grounding module exist in the prompt variables for v2 configuration.

Parameters:

orch_config (dict) – The orchestration configuration dictionary.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.validate_orchestration_params_from_evaluation_config(evaluation_configs: List[EvaluationConfig], error_collector: ValidationCollector) → None¶

Validates orchestration parameters from evaluation configuration.

Ensures that either orchestration_registry_reference is provided alone, or both template and llm are provided together.

Parameters:

evaluation_configs (List[EvaluationConfig]) – List of evaluation configuration objects.
error_collector (ValidationCollector) – ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.orch_config_utils.to_comparable(x) → dict | Any¶

Converts an object to a comparable format (dict or primitive).

Handles Pydantic v1, Pydantic v2, custom classes, and primitives.

Parameters:: x (Any) – The object to convert.
Returns:: Dictionary representation or the primitive value.
Return type:: dict | Any