gen_ai_hub.evaluations.models.evaluation_config module

class gen_ai_hub.evaluations.models.evaluation_config.EvaluationConfig(dataset_config: Dataset, metrics: List[MetricConfig], llm: LLMModelDetails | None = None, template: str | PromptTemplateSpec | TemplateRef | None = None, orchestration_registry_reference: str | None = None, template_variable_mapping: dict | None = None, test_row_count: int | None = -1, repetitions: int | None = 1, tags: dict | None = '{}', debug_mode: bool | None = False)

Bases: object

Defines the evaluation configuration object for the Evaluations flow.

This class encapsulates all configuration parameters needed to run an evaluation job, including the model/template configuration, dataset, metrics, and execution settings.

At least one of the following must be provided:

  • llm and template combination (using orchestration_v2 models)

  • orchestration_registry_reference (UUID of a registered orchestration configuration)

Parameters:

  • dataset_config (Dataset) – Dataset configuration object specifying the evaluation dataset

  • metrics (List[MetricConfig]) – List of metric configurations for evaluation

  • llm (Optional[LLM]) – LLM configuration from orchestration_v2 (LLMModelDetails)

  • template (Optional[Union[str, PromptTemplateSpec, TemplateRef]]) – Prompt template as string, PromptTemplateSpec, or TemplateRef

  • orchestration_registry_reference (Optional[str]) – UUID of registered orchestration configuration

  • template_variable_mapping (Optional[dict]) – Variable mapping for the prompt template

  • test_row_count (Optional[int]) – Number of rows to sample from dataset (-1 for all rows), defaults to -1

  • repetitions (Optional[int]) – Number of times to repeat evaluation over the dataset, defaults to 1

  • tags (Optional[dict]) – User-defined metadata as key-value pairs, defaults to “{}”

  • debug_mode (Optional[bool]) – Enable debug logs in hyperscaler output path, defaults to False

Note

This module uses orchestration_v2 models directly.

Example using TemplateRef with ID:

>>> from gen_ai_hub.evaluations.models import EvaluationConfig, Dataset, MetricConfig
>>> from gen_ai_hub.orchestration_v2.models.llm_model_details import LLMModelDetails as LLM
>>> from gen_ai_hub.orchestration_v2.models.template_ref import TemplateRef, TemplateRefByID
>>> config = EvaluationConfig(
...     dataset_config=Dataset("data/test.jsonl"),
...     metrics=[MetricConfig(name="accuracy")],
...     llm=LLM(name="gpt-4", version="latest"),
...     template=TemplateRef(template_ref=TemplateRefByID(id="template-id-here")),
...     test_row_count=100
... )

Example using TemplateRef with scenario/name/version:

>>> from gen_ai_hub.orchestration_v2.models.template_ref import TemplateRefByScenarioNameVersion
>>> config = EvaluationConfig(
...     dataset_config=Dataset("data/test.jsonl"),
...     metrics=[MetricConfig(name="accuracy")],
...     llm=LLM(name="gpt-4", version="latest", params={"temperature": 0.7}),
...     template=TemplateRef(template_ref=TemplateRefByScenarioNameVersion(
...         scenario="foundation-models", name="prompt1", version="1.0"
...     )),
...     test_row_count=100
... )
__init__(dataset_config: Dataset, metrics: List[MetricConfig], llm: LLMModelDetails | None = None, template: str | PromptTemplateSpec | TemplateRef | None = None, orchestration_registry_reference: str | None = None, template_variable_mapping: dict | None = None, test_row_count: int | None = -1, repetitions: int | None = 1, tags: dict | None = '{}', debug_mode: bool | None = False)

Initialize an EvaluationConfig instance.

Parameters:

  • dataset_config (Dataset) – Dataset configuration object

  • metrics (List[MetricConfig]) – List of metric configurations

  • llm (Optional[LLM]) – LLM object from orchestration_v2 (LLMModelDetails), defaults to None

  • template (Optional[Union[str, PromptTemplateSpec, TemplateRef]]) – Prompt template (string, PromptTemplateSpec, or TemplateRef), defaults to None

  • orchestration_registry_reference (Optional[str]) – UUID of orchestration config, defaults to None

  • template_variable_mapping (Optional[dict]) – Variable mapping for prompt template, defaults to None

  • test_row_count (Optional[int]) – Number of dataset rows to sample (-1 for all), defaults to -1

  • repetitions (Optional[int]) – Number of evaluation repetitions (minimum: 1), defaults to 1

  • tags (Optional[dict]) – Key-value metadata pairs applied to all runs, defaults to “{}”

  • debug_mode (Optional[bool]) – Enable debug logging, defaults to False

Raises:

ValueError – If neither (llm, template) nor orchestration_registry_reference is provided