gen_ai_hub.orchestration_v2.service module

Module for orchestration service handling requests and responses.

Provides synchronous and asynchronous methods to run orchestration pipelines.

class gen_ai_hub.orchestration_v2.service.OrchestrationService(api_url: str | None = None, config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, proxy_client: GenAIHubProxyClient | None = None, deployment_id: str | None = None, config_name: str | None = None, config_id: str | None = None, timeout: int | float | Timeout | None = None)

Bases: object

A service for executing orchestration requests, allowing for the generation of LLM-generated content through a pipeline of configured modules.

This service supports both synchronous and asynchronous request execution. For streaming responses, special care is taken to not close the underlying HTTP stream prematurely.

See https://api.sap.com/api/ORCHESTRATION_API_v2/overview

Args:

api_url: The base URL for the orchestration API.

config: The default orchestration configuration.

config_ref: The reference to default orchestration configuration.

proxy_client: A GenAIHubProxyClient instance.

deployment_id: Optional deployment ID.

config_name: Optional configuration name.

config_id: Optional configuration ID.

timeout: Optional timeout for HTTP requests.

__init__(api_url: str | None = None, config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, proxy_client: GenAIHubProxyClient | None = None, deployment_id: str | None = None, config_name: str | None = None, config_id: str | None = None, timeout: int | float | Timeout | None = None)

Initializes the OrchestrationService.

Parameters:

  • api_url (Optional[str], optional) – the base URL for the orchestration API, defaults to None

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional) – the orchestration configuration reference, defaults to None

  • proxy_client (Optional[GenAIHubProxyClient], optional) – the GenAIHubProxyClient instance, defaults to None

  • deployment_id (Optional[str], optional) – the deployment ID, defaults to None

  • config_name (Optional[str], optional) – the configuration name, defaults to None

  • config_id (Optional[str], optional) – the configuration ID, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout for HTTP requests, defaults to None

Raises:

ValueError – if both config and config_ref are provided.

run(config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, placeholder_values: dict | None = None, history: List[SystemMessage | UserMessage | AssistantMessage | ToolChatMessage | DeveloperChatMessage | ResponseChatMessage] | None = None, timeout: int | float | Timeout | None = None) → CompletionPostResponse

Executes an orchestration request synchronously (non-streaming).

Parameters:

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional) – the orchestration configuration reference, defaults to None if not provided, the default configuration is used.

  • placeholder_values (Optional[dict], optional) – the template values, defaults to None

  • history (Optional[List[ChatMessage]], optional) – the message history, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

Returns:

the CompletionPostResponse object

Return type:

CompletionPostResponse

stream(config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, placeholder_values: dict | None = None, history: List[SystemMessage | UserMessage | AssistantMessage | ToolChatMessage | DeveloperChatMessage | ResponseChatMessage] | None = None, timeout: int | float | Timeout | None = None) → Iterable[StreamCompletionPostResponse]

Executes an orchestration streaming request synchronously.

Parameters:

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional if not provided, the default configuration is used.) – the orchestration configuration reference, defaults to None

  • placeholder_values (Optional[dict], optional) – the template values, defaults to None

  • history (Optional[List[ChatMessage]], optional) – the message history, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

Returns:

An Iterable[StreamCompletionPostResponse] object

Return type:

Iterable[StreamCompletionPostResponse]

async arun(config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, placeholder_values: dict | None = None, history: List[SystemMessage | UserMessage | AssistantMessage | ToolChatMessage | DeveloperChatMessage | ResponseChatMessage] | None = None, timeout: int | float | Timeout | None = None) → CompletionPostResponse

Executes an orchestration request asynchronously (non-streaming).

Parameters:

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional) – the orchestration configuration reference, defaults to None

  • placeholder_values (Optional[dict], optional) – the template values, defaults to None

  • history (Optional[List[ChatMessage]], optional) – the message history, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

Returns:

the CompletionPostResponse object

Return type:

CompletionPostResponse

async astream(config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, placeholder_values: dict | None = None, history: List[SystemMessage | UserMessage | AssistantMessage | ToolChatMessage | DeveloperChatMessage | ResponseChatMessage] | None = None, timeout: int | float | Timeout | None = None) → AsyncSSEClient

Executes an orchestration streaming request asynchronously.

Parameters:

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional) – the orchestration configuration reference, defaults to None

  • placeholder_values (Optional[dict], optional) – the template values, defaults to None

  • history (Optional[List[ChatMessage]], optional) – the message history, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

Returns:

the AsyncSSEClient object

Return type:

AsyncSSEClient

run_with_retries(config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, placeholder_values: dict | None = None, history: List[SystemMessage | UserMessage | AssistantMessage | ToolChatMessage | DeveloperChatMessage | ResponseChatMessage] | None = None, timeout: int | float | Timeout | None = None, max_retries: int = 10, base_delay: float = 1.0) → OrchestrationResponseWithRetries | None

Executes an orchestration request with automatic retry on rate limits (429) and server errors.

Parameters:

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional) – the orchestration configuration reference, defaults to None

  • placeholder_values (Optional[dict], optional) – the template values, defaults to None

  • history (Optional[List[ChatMessage]], optional) – the message history, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

  • max_retries (int, optional) – the maximum number of retry attempts, defaults to 10

  • base_delay (float, optional) – the initial delay between retries in seconds, defaults to 1.0

Returns:

the OrchestrationResponseWithRetries with retry count information

Return type:

OrchestrationResponseWithRetries | None

Raises:

  • ValueError – if no configuration is provided.

  • OrchestrationError – if request fails after all retries (includes retry count).

handle_retry(retry_count: int, base_delay: float, error: OrchestrationError, max_retries: int) → float

Handles retry logic with exponential backoff and jitter. If Retry-After header exists, use it as min_delay to add jitter on top

Parameters:

  • retry_count (int) – the incremented retry attempt number

  • base_delay (float) – the initial delay between retries in seconds

  • error (OrchestrationError) – the exception that occurred

  • max_retries (int) – the maximum number of retry attempts

Raises:

error – throws the original error if no retry should be attempted

Returns:

the number of seconds to wait before next retry

Return type:

float

async arun_with_retries(config: OrchestrationConfig | None = None, config_ref: CompletionRequestConfigurationReferenceByIdConfigRef | CompletionRequestConfigurationReferenceByNameScenarioVersionConfigRef | None = None, placeholder_values: dict | None = None, history: List[SystemMessage | UserMessage | AssistantMessage | ToolChatMessage | DeveloperChatMessage | ResponseChatMessage] | None = None, timeout: int | float | Timeout | None = None, max_retries: int = 10, base_delay: float = 1.0) → OrchestrationResponseWithRetries | None

Executes an orchestration request asynchronously with automatic retry on rate limits (429) and server errors. Uses exponential backoff with jitter to handle rate limiting gracefully.

Parameters:

  • config (Optional[OrchestrationConfig], optional) – the orchestration configuration, defaults to None

  • config_ref (Optional[OrchestrationConfigReference], optional) – the orchestration configuration reference, defaults to None

  • placeholder_values (Optional[dict], optional) – the template values, defaults to None

  • history (Optional[List[ChatMessage]], optional) – the message history, defaults to None

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

  • max_retries (int, optional) – the maximum number of retry attempts, defaults to 10

  • base_delay (float, optional) – the initial delay between retries in seconds, defaults to 1.0

Returns:

the OrchestrationResponseWithRetries with retry count information

Return type:

OrchestrationResponseWithRetries | None

Raises:

  • ValueError – if no configuration is provided.

  • OrchestrationError – if request fails after all retries (includes retry count).

embed(config: EmbeddingsOrchestrationConfig, input: EmbeddingsInput, timeout: int | float | Timeout | None = None) → EmbeddingsPostResponse

Executes an embeddings request synchronously.

Parameters:

  • config (EmbeddingsOrchestrationConfig) – the embeddings orchestration configuration

  • input (EmbeddingsInput) – the input text to embed

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

Returns:

the EmbeddingsPostResponse object

Return type:

EmbeddingsPostResponse

async aembed(config: EmbeddingsOrchestrationConfig, input: EmbeddingsInput, timeout: int | float | Timeout | None = None) → EmbeddingsPostResponse

Executes an embeddings request asynchronously.

Parameters:

  • config (EmbeddingsOrchestrationConfig) – the embeddings orchestration configuration

  • input (EmbeddingsInput) – the input text to embed

  • timeout (Union[int, float, httpx.Timeout, None], optional) – the timeout overwrite per request, defaults to None

Returns:

the EmbeddingsPostResponse object

Return type:

EmbeddingsPostResponse

close_http_connection()

Closes the httpx synchronous client.

async aclose_http_connection()

Closes the httpx asynchronous client.