Batch API

caution

The @sap-ai-sdk/llm-batch package is experimental and may change at any time without prior notice.

The Batch API lets you submit multiple LLM requests as a single asynchronous job, reducing cost and avoiding rate limits compared to making real-time requests.

The Batch API supports Azure OpenAI models only.

Installation

npm install @sap-ai-sdk/llm-batch @sap-ai-sdk/ai-api @sap-ai-sdk/foundation-models

Making Requests

A typical batch workflow consists of four steps: create a job, poll for completion, retrieve results, and optionally manage jobs.

Create a Batch Job

Use createBatchInput from @sap-ai-sdk/foundation-models to generate the input Blob from typed chat completion requests and upload it to an object store registered as a secret in SAP AI Core:

import { createBatchInput } from '@sap-ai-sdk/foundation-models';
import { FileApi } from '@sap-ai-sdk/ai-api';

const blob = createBatchInput([
  {
    model: 'gpt-5',
    messages: [{ role: 'user', content: 'What is machine learning?' }],
    max_tokens: 150
  },
  {
    model: 'gpt-5',
    messages: [
      { role: 'user', content: 'Explain neural networks in simple terms' }
    ],
    max_tokens: 150
  }
]);

await FileApi.fileUpload(
  'MY_OBJECT_STORE/input-batch.jsonl',
  blob,
  { overwrite: true },
  { 'AI-Resource-Group': 'MY_RESOURCE_GROUP' }
).execute();

Then create the batch job referencing the input file and an output directory using the ai://SECRET_NAME/ URI format:

import { BatchesApi } from '@sap-ai-sdk/llm-batch';

const response = await BatchesApi.createBatch(
  {
    type: 'llm-native',
    input: { uri: 'ai://MY_OBJECT_STORE/input-batch.jsonl' },
    output: { uri: 'ai://MY_OBJECT_STORE/output/' },
    spec: { provider: 'azure-openai', model: 'gpt-5' }
  },
  { 'AI-Resource-Group': 'MY_RESOURCE_GROUP' }
).execute();

console.log('Batch job created:', response.id);

Poll for Completion

Batch jobs are processed asynchronously. Poll the status endpoint until a terminal state is reached:

const TERMINAL_STATUSES = ['COMPLETED', 'FAILED', 'CANCELLED'];

function delay(ms: number) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

let status = '';
while (!TERMINAL_STATUSES.includes(status)) {
  const result = await BatchesApi.getBatchStatus(response.id!, {
    'AI-Resource-Group': 'MY_RESOURCE_GROUP'
  }).execute();

  status = result.current_status ?? '';
  console.log('Current status:', status);

  if (!TERMINAL_STATUSES.includes(status)) {
    await delay(60_000); // wait 1 minute between polls
  }
}

tip

Batch jobs can take minutes to hours depending on the number of requests. Adjust the polling interval based on your expected workload — for large batches, consider polling every 30–60 minutes.

The possible statuses are:

Status	Description
PENDING	Job is queued
PREPARING_INPUT	Input file is being read from object store
RUNNING	LLM requests are being processed
COMPLETED	All requests finished successfully
FAILED	Job failed
CANCELLING	Cancellation is in progress
CANCELLED	Job was cancelled

Retrieve Results

Once the job reaches COMPLETED status, download the output file from the object store. The output is written to {output.uri}{batchId}/output.jsonl. For example, if output.uri is ai://MY_OBJECT_STORE/output/, the file will be at ai://MY_OBJECT_STORE/output/{batchId}/output.jsonl.

Each line in the output JSONL corresponds to one input request, matched via custom_id:

{"custom_id": "request-1", "response": {"status_code": 200, "body": {"id": "chatcmpl-abc", "choices": [{"message": {"role": "assistant", "content": "Machine learning is a subset of AI..."}}], "usage": {"prompt_tokens": 12, "completion_tokens": 45, "total_tokens": 57}}}, "error": null}
{"custom_id": "request-2", "response": {"status_code": 200, "body": {"id": "chatcmpl-def", "choices": [{"message": {"role": "assistant", "content": "Neural networks are computing systems..."}}], "usage": {"prompt_tokens": 13, "completion_tokens": 42, "total_tokens": 55}}}, "error": null}

Managing Batch Jobs

List all batch jobs:

const { resources } = await BatchesApi.listBatches({
  'AI-Resource-Group': 'MY_RESOURCE_GROUP'
}).execute();

console.log(`Total jobs: ${resources?.length}`);

Get job details:

const details = await BatchesApi.getBatchById(batchId, {
  'AI-Resource-Group': 'MY_RESOURCE_GROUP'
}).execute();

Cancel a running job:

await BatchesApi.cancelBatch(batchId, {
  'AI-Resource-Group': 'MY_RESOURCE_GROUP'
}).execute();

Delete a job:

await BatchesApi.deleteBatch(batchId, {
  'AI-Resource-Group': 'MY_RESOURCE_GROUP'
}).execute();

note

A batch job can only be deleted after it reaches a terminal status: COMPLETED, FAILED, or CANCELLED.