Documentation

• 1: Concepts
• 1.1: CAP Operator Overview
• 1.1.1: Controller
• 1.1.2: Subscription Server
• 1.2: CAP Application Components
• 2: What's New
• 3: Installation
• 3.1: Prerequisites
• 3.2: Using Helm
• 3.2.1: Helm Values
• 3.3: Using CAP Operator Manager
• 4: Configuration
• 5: Usage
• 5.1: Prerequisites
• 5.2: Deploying a CAP Application
• 5.3: Tenant Subscription
• 5.4: Application Upgrade
• 5.5: Services Only Applications
• 5.6: Version Monitoring
• 5.7: Operator Metrics
• 5.8: Resources
• 5.8.1: CAPApplication
• 5.8.2: CAPApplicationVersion
• 5.8.3: CAPTenant
• 5.8.4: CAPTenantOperation
• 5.8.5: CAPTenantOutput
• 6: Troubleshooting
• 7: Support
• 8: Reference

1 - Concepts

Provisioning and operating an SAP Cloud Application Programming Model application on a Kubernetes cluster requires the deployment of various components in addition to the CAP application server (see a list of typical components). Some of these components can be created at the time of system provisioning, while others need to be created (or updated) at different points during the lifecycle of the application (DAY 2 operational tasks).

Using Helm charts to manage the deployment of a CAP application can support the initial system provisioning, but further lifecycle operations (such as tenant provisioning) that are initiated from external components (SAP BTP) require manual adjustment of the deployed resources. An example of such an instance would be the creation of VirtualServices (part of Istio service mesh) during tenant provisioning to route application (HTTP) requests submitted on the new tenant subdomain to the application server. Another limitation of using helm charts is the lack of control over the order in which resources are created.

You can get more control over the deployment and further automation of lifecycle operations by extending the Kubernetes API with custom resources that describe the components and the configuration of CAP applications, and controllers to reconcile them. Similar to standard controllers of Kubernetes, the custom controllers watch for changes in the custom resource objects and work towards moving the cluster state to the desired state.

CAP Operator comprises of custom resource definitions that describe the CAP application components, the controller to reconcile these resources, and other components that support the lifecycle management.

1.1 - CAP Operator Overview

CAP Operator is comprised of the following components:

1. CAP Controller: a native Kubernetes controller that reconciles custom resources defined as part of the operator
2. Webhooks: validating webhooks to ensure consistency of custom resource objects submitted to the Kubernetes API server
3. Subscription Server: web server for handling HTTP requests submitted by the SAP BTP saas-registry service instances during tenant subscription (and unsubscribe)

The following diagram depicts how the main components interact when deployed to a cluster:

Looking for more details about the CAP Operator components? Go to the next pages.

1.1.1 - Controller

To implement the CAP controller, you use the client-go from Kubernetes, which provides the required tools and utilities to interact with the Kubernetes API server. It manages custom resources that are included in CAP Operator.

The controller uses Informers to watch certain resources and invokes registered event handlers when these resources are modified. To streamline the processing of such notifications, rate limiting queues are implemented, which store the changes and allow the processing of these items in independent reconciliation threads (go routines). Such a design allows sequential processing of the changed items and avoids conflicts.

The following namespaced custom resources have been defined to be reconciled by the CAP controller:

• CAPApplication: defines a high-level application, its domains, and the consumed SAP BTP services
• CAPApplicationVersion: defines a child resource of the CAPApplication, which contains container images that will be used to deploy application components (workloads) of a specific version
• CAPTenant: represents a child resource of the CAPApplication which corresponds to an SAP BTP subaccount subscribed to the application
• CAPTenantOperation: represents a provisioning, deprovisioning, or upgrade operation on a tenant that is scheduled as a child resource of a CAPTenant and executed as a sequence of specified steps.

Parent-child relationships between custom resources are established by defining owner references for the children.

1.1.2 - Subscription Server

The Subscription Server handles HTTP requests from the SAP Software-as-a-Service Provisioning service for tenant subscription operations on SAP Cloud Application Programming Model applications that have been installed in the cluster.

During the creation of a saas-registry service instance (in the provider subaccount), callback URLs are configured, which point to the subscription server routes.

When a consumer tenant subscribes to an application managed by the operator, a subscription callback is received by the subscription server, which then generates the CAPTenant custom resource object.

The subscription server returns an Accepted (202) response code and starts a routine/thread, which keeps polling for the tenant status until the changes to the CAPTenant are then independently reconciled by the controller.

Once the tenant provisioning process has completed (or has failed), the tracking routine will return the appropriate status to the SaaS Registry via an asynchronous callback (by obtaining the necessary authorization token).

(More details about asynchronous tenant subscription.)

Such an asynchronous processing allows us to avoid timeouts during synchronous calls, as well as schedule dedicated jobs (via CAPTenantOperation) for completion of the subscription and perform any further tasks required in the cluster (for example, create a VirtualService corresponding to the tenant subdomain).

1.2 - CAP Application Components

A full-stack application built with the SAP Cloud Application Programming Model has the following components:

SAP BTP Service Instances

Multi-tenant CAP-based applications consume services from SAP BTP such as SAP Authorization and Trust Management Service, SAP Software-as-a-Service Provisioning service, and so on. You need to create these service instances within an SAP BTP provider account as well as service keys (bindings) for these instances, which generate the credentials used by the application for accessing these services.

CAP Application Server

The application provides data models that will be deployed to the connected database. An HTTP server exposes defined services and handles server-side application logic. For more details, check out the documentation of SAP Cloud Application Programming Model. It’s also possible to split the application into multiple servers (services) that work together.

CAP Components to Support Multitenancy

CAP provides the module @sap/cds-mtxs, which can be operated as a sidecar (component running independently from the application server). This component is responsible for handling requests related to tenant management such as onboarding, which then creates the required schema in the connected database. This module also supports triggering tenant management tasks as CLI commands.

Approuter

The Approuter, or an extended version of it, takes care of authenticating requests (using the SAP Authorization and Trust Management Service) and routes the requests to the application servers or related services (for example, SAP HTML5 Application Repository service for SAP BTP).

SAP Fiori Applications

Multiple SAP Fiori front-end applications can connect to the CAP application back end. These UI5 applications are deployed to the SAP HTML5 Application Repository service for SAP BTP and served from there. Similarly, the application can have content specific to other services that need to be deployed, such as the SAP Cloud Portal service.

2 - What's New

New updates

3 - Installation

This page provides an overview of available methods to install CAP Operator on a Kubernetes cluster.

3.1 - Prerequisites

We recommend that you use a “Gardener” managed cluster to deploy CAP applications that are managed with CAP Operator.

The Kubernetes cluster must be set up with the following prerequisites before you install CAP Operator:

Istio (version >= 1.22)

Istio service mesh is used for HTTP traffic management. CAP Operator creates Istio resources to manage incoming HTTP requests to the application as well as to route requests on specific (tenant) subdomains.

It’s required that you determine the public ingress Gateway subdomain and the overall shoot domain for the system and specify them in the chart values

Note: Istio promoted many of its APIs to v1 in 1.22 release. Hence as of CAP Operator release v0.11.0 istio version >= 1.22 is a prerequisite.

sap-btp-service-operator or cf-service-operator

These operators can be used for managing SAP BTP service instances and service bindings from within the Kubernetes cluster.

If some SAP BTP services are not available for Kubernetes platforms, you may use cf-service-operator, which creates the services for a Cloud Foundry space and inserts the required access credentials as Secrets into the Kubernetes cluster.

Please note that service credentials added as Kubernetes Secrets to a namespace by these operators, support additional metadata. If you don’t use this feature of these operators, use secretKey: credentials in the spec of these operators to ensure that the service credentials retain any JSON data as it is. We recommend that you use secretKey, even when credential metadata is available to reduce the overhead of interpreting parsing multiple JSON attributes.

“Gardener” certificate management

This component is available in clusters managed by “Gardener” and will be used to manage TLS certificates and issuers. “Gardener” manages encryption, issuing, and signing of certificates. Alternatively, you can use cert-manager.io cert-manager.

3.2 - Using Helm

To install CAP operator components, we recommend using the Helm chart that is published as an OCI package at oci://ghcr.io/sap/cap-operator-lifecycle/helm/cap-operator.

Installation

Create a namespace and install the Helm chart in that namespace by specifying the domain and the dnsTarget for your subscription server, either

• As command line parameters:

  kubectl create namespace cap-operator-system
  helm upgrade -i -n cap-operator-system cap-operator oci://ghcr.io/sap/cap-operator-lifecycle/helm/cap-operator --set subscriptionServer.domain=cap-operator.<CLUSTER-DOMAIN> --set subscriptionServer.dnsTarget=public-ingress.<CLUSTER-DOMAIN>
  

• Or as a YAML file with the values:

  kubectl create namespace cap-operator-system
  helm upgrade -i -n cap-operator-system cap-operator oci://ghcr.io/sap/cap-operator-lifecycle/helm/cap-operator -f my-cap-operator-values.yaml
  

  In this example, the provided values file, my-cap-operator-values.yaml, can have the following content:

  subscriptionServer:
    dnsTarget: public-ingress.<CLUSTER-DOMAIN>
    domain: cap-operator.<CLUSTER-DOMAIN>   
  

Optional steps

• Enable Service Monitors for metrics emitted by controller and subscription server

  To enable Monitoring via metrics emitted by CAP Operator components, the following value can be specified:

  monitoring:
    enabled: true # <-- This enables creation of service monitors, for metrics emitted by the cap operator components
  

  Detailed operational metrics for the controller can be enabled with the following config:

  controller:
      detailedOperationalMetrics: true
  

• Setup Prometheus Integration for Version Monitoring

  To use the Version Monitoring feature of the CAP Operator, a Prometheus server URL can be provided to the CAP Operator. When installing the CAP Operator using the Helm chart, the following values can be specified in the values:

  controller:
    versionMonitoring:
      prometheusAddress: "http://prometheus-operated.monitoring.svc.cluster.local:9090" # <-- example of a Prometheus server running inside the same cluster
      promClientAcquireRetryDelay: "2h"
      metricsEvaluationInterval: "30m" # <-- duration after which version metrics are evaluated
  

  When the controller is started, the operator will try to connect to the Prometheus server and fetch runtime information to verify the connection. If the connection is not successful, it will be retried after the duration specified as controller.versionMonitoring.promClientAcquireRetryDelay. Check default values for these attributes here.

  Note

  • When connecting the controller to a Prometheus server running inside the cluster, please ensure that NetworkPolicies required for connecting to the service in the namespace where Prometheus is running are also created.
  • If the Prometheus service is configured to use TLS, the relevant CA root certificates which need to be trusted can be mounted as volumes to the controller.

3.2.1 - Helm Values

Values

3.3 - Using CAP Operator Manager

To install the CAP Operator using CAP Operator Manager, please execute the following commands:

kubectl apply -f https://github.com/SAP/cap-operator-lifecycle/releases/latest/download/manager_manifest.yaml

The above command will create namespace cap-operator-system with CAP Operator Manager installed. Once the CAP Operator Manager pod is running, you can install the CAP operator by executing the following command:

kubectl apply -n cap-operator-system -f https://github.com/SAP/cap-operator-lifecycle/releases/latest/download/manager_default_CR.yaml

This would work only if the ingressGatewayLabels in your clusters matches the following values

ingressGatewayLabels:
  - name: istio
    value: ingressgateway
  - name: app
    value: istio-ingressgateway

If not, you will have to manually create the CAPOperator resource. For more details on the same, please refer to link.

4 - Configuration

Here’s a list of environment variables used by CAP Operator.

Controller

• CERT_MANAGER: specifies the certificate manager to be used for TLS certificates. Possible values are:
  • gardener: “Gardener” certificate management
  • cert-manager.io: cert-manager.io cert-manager
• DNS_MANAGER: specifies the external DNS manager to be used. Possible values are:
  • gardener: “Gardener” external DNS manager
  • kubernetes: external DNS management from Kubernetes
• PROMETHEUS_ADDRESS: URL of the Prometheus server (or service) for executing PromQL queries e.g. http://prometheus-operated.monitoring.svc.cluster.local:9090. If no URL is supplied, the controller will not start the version monitoring function.
• PROM_ACQUIRE_CLIENT_RETRY_DELAY: Time delay between retries when a Prometheus client creation and connection check fails.
• METRICS_EVAL_INTERVAL: Time interval between subsequent iterations where outdated versions are identified and queued for evaluation.

5 - Usage

5.1 - Prerequisites

Prepare the SAP BTP global account and provider subaccount

CAP-based applications make use of various SAP BTP services that are created in a provider subaccount. So, before you can deploy the application, create a global account and assign the required services that will be used. To do so, use SAP BTP Control Center. Once done, create a provider subaccount, where the required service instances can be created.

Create service instances and bindings

A multi-tenant CAP-based application consumes the following SAP BTP services. While creating these service instances, some of the parameters supplied require special attention. Service keys (bindings) are then created to generate access credentials, which in turn should be provided as Kubernetes Secrets in the namespace where the application is being deployed.

Other services (not listed here) may also be used depending on the requirement (for example, SAP HTML5 Application Repository service for SAP BTP, Business Logging, and so on).

Note: If some SAP BTP services are not available on Kubernetes, enable Cloud Foundry for the provider subaccount to create certain services. In such cases you may use the cf-service-operator for managing the service instances and service bindings directly from within the Kubernetes cluster. Based on the service bindings, it automatically generates the secrets containing the service access credentials.

SAP Authorization and Trust Management Service

The parameter oauth2-configuration.redirect-uris must include the domain used by the application. For instance, if the application is hosted in a “Gardener” managed cluster, the entry may have the form https://*<application-specific-prefix>.<cluster-id>.<gardener-project-id>.shoot.url.k8s.example.com/**.

Scope required to make asynchronous tenant subscription operations need to be included. Additionally, check the CAP Multitenancy documentation for additional scopes which are required.

parameters:
  authorities:
    - $XSAPPNAME.mtcallback
    - $XSAPPNAME.mtdeployment
  oauth2-configuration:
    redirect-uris:
      - https://*my-cap-app.cluster-x.my-project.shoot.url.k8s.example.com/**
  role-collections:
    ...
  role-templates:
    ...
  scopes:
    - description: UAA
      name: uaa.user
    - description: With this scope set, the callbacks for tenant onboarding, offboarding, and getDependencies can be called
      grant-as-authority-to-apps:
        - $XSAPPNAME(application,sap-provisioning,tenant-onboarding)
      name: $XSAPPNAME.Callback
    - description: Async callback to update the saas-registry (provisioning succeeded/failed)
      name: $XSAPPNAME.subscription.write
    - description: Deploy applications
      name: $XSAPPNAME.mtdeployment
    - description: Subscribe to applications
      grant-as-authority-to-apps:
        - $XSAPPNAME(application,sap-provisioning,tenant-onboarding)
      name: $XSAPPNAME.mtcallback
    ...

When using mulitple SAP Authorization and Trust Management Service instances in the app (for example, one for the application and other apiaccess). The primary instance can be set using the annotation: “sme.sap.com/primary-xsuaa” with the value being the name of the service instance, as shown below:

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplication
metadata:
  annotations:
    "sme.sap.com/primary-xsuaa": "my-cap-app-uaa" # This let's the CAP Operator determine/use the right UAA instance for the application.
  name: test-cap-01
  ...
spec:
  btp:
    services:
      - class: xsuaa
        name: my-cap-app-uaa-api
        secret: my-cap-app-uaa-api-bind-cf
      - class: xsuaa
        name: my-cap-app-uaa
        secret: my-cap-app-uaa-bind-cf
      - class: saas-registry
        name: my-cap-app-saas-registry
        secret: my-cap-app-saas-bind-cf
      ...
  btpAppName: my-cap-app
  ...

SAP Software-as-a-Service Provisioning service

When creating an instance of the SaaS Provisioning service, use asynchronous tenant subscription callbacks in the configuration. See Register Your Multi-Tenant Application/Service in SaaS Provisioning for more details.

parameters:
  appName: <short-application-name>
  appUrls:
    callbackTimeoutMillis: 300000 # <-- used to fail subscription process when no response is received
    getDependencies: https://<provider-subaccount-subdomain>.<cap-app-name>.cluster-x.my-project.shoot.url.k8s.example.com/callback/v1.0/dependencies # <-- handled by the application
    onSubscription: https://<cap-operator-subscription-server-domain>/provision/tenants/{tenantId} # <-- the /provision route is forwarded directly to CAP Operator (Subscription Server) and must be specified as such
    onSubscriptionAsync: true
    onUnSubscriptionAsync: true

SAP HANA Cloud

An SAP HANA Cloud instance (preferably shared and accessible from the provider subaccount) is required. The Instance ID of the database must be noted for usage in relevant workloads. SAP HANA Schemas & HDI Containers service must also be entitled for the provider subaccount.

SAP Service Manager service

The SAP Service Manager service allows CAP to retrieve schema-(tenant-)specific credentials to connect to the SAP HANA Cloud database.

5.2 - Deploying a CAP Application

Just by defining two resources provided by CAP Operator, namely capapplications.sme.sap.com and capapplicationversions.sme.sap.com, it’s possible to deploy a multi-tenant CAP application and start using it. These resources are namespaced and so the CAP Operator will create all related resources (deployments, gateways, jobs etc.) within the same namespace.

The object, CAPApplication, describes the high-level attributes of an application such as the SAP BTP account where it is hosted, the consumed SAP BTP services, domains where the application routes will be made available etc. See API Reference.

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplication
metadata:
  name: cap-app-01
  namespace: cap-app-01
spec:
  btpAppName: cap-app-01 # <-- short name (similar to SAP BTP XSAPPNAME)
  btp:
    services:
      - class: xsuaa # <-- SAP BTP service technical name
        name: app-uaa # <-- name of the service instance
        secret: cap-app-01-uaa-bind-cf # <-- secret containing the credentials to access the service existing in the same namespace
      - class: saas-registry
        name: app-saas-registry
        secret: cap-app-01-saas-bind-cf
      - class: service-manager
        name: app-service-manager
        secret: cap-app-01-svc-man-bind-cf
      - class: destination
        name: app-destination
        secret: cap-app-01-dest-bind-cf
      - class: html5-apps-repo
        name: app-html5-repo-host
        secret: cap-app-01-html5-repo-bind-cf
      - class: html5-apps-repo
        name: app-html5-repo-runtime
        secret: cap-app-01-html5-rt-bind-cf
      - class: portal
        name: app-portal
        secret: cap-app-01-portal-bind-cf
  domains:
    istioIngressGatewayLabels: # <-- labels used to identify the Istio ingress gateway (the values provided here are the default values)
      - name: app
        value: istio-ingressgateway
      - name: istio
        value: ingressgateway
    primary: "cap-app-01.cluster.shoot.url.k8s.example.com" # <-- primary domain where the application is exposed. Each tenant will have access to a subdomain of this domain. Ensure that this is at most 62 chars long.
    secondary:
      - "alt.shoot.example.com"
  globalAccountId: global-account-id
  provider:
    subDomain: cap-app-01-provider
    tenantId: e55d7b5-279-48be-a7b0-aa2bae55d7b5

The object, CAPApplicationVersion, describes the different components of an application version including the container images to be used and the services consumed by each component. See API Reference.

The CAPApplicationVersion must be created in the same namespace as the CAPApplication and refers to it.

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  name: cav-cap-app-01-1
  namespace: cap-app-01
spec:
  capApplicationInstance: cap-app-01 # <-- reference to CAPApplication in the same namespace
  version: "1" # <-- semantic version
  registrySecrets:
    - regcred
  workloads:
    - name: cap-backend
      consumedBTPServices: # <-- these are services used by the application server (already defines as part of CAPApplication resource). Corresponding credential secrets will be mounted onto the component pods as volumes.
        - app-uaa
        - app-service-manager
        - app-saas-registry
      deploymentDefinition:
        type: CAP # <-- indicates the CAP application server
        image: app.some.repo.example.com/srv/server:0.0.1
        env:
          - name: CDS_ENV
            value: production
          - name: CDS_MTX_PROVISIONING_CONTAINER
            value: '{"provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345"}}'
    - name: app-router
      consumedBTPServices:
        - app-uaa
        - app-destination
        - app-saas-registry
        - app-html5-repo-runtime
        - app-portal
      deploymentDefinition:
        type: Router
        image: app.some.repo.example.com/approuter/approuter:0.0.1
        env:
          - name: PORT
            value: 4000
          - name: TENANT_HOST_PATTERN
            value: "^(.*).(cap-app-01.cluster.shoot.canary.k8s-hana.ondemand.co|alt.shoot.example.com)"
    - name: service-content
      consumedBTPServices:
        - app-uaa
        - app-html5-repo-host
        - app-portal
      jobDefinition:
        type: Content
        image: app.some.repo.example.com/approuter/content:0.0.1
        backoffLimit: 1

NOTE: The example above is a minimal CAPApplicationVersion that can be deployed. For a more supported configuration and their explanations, see here.

The controller component of CAP Operator reacts to these objects and creates further resources, which constitute a running application:

• Deployment (and service) for the application server with credentials (from secrets) to access SAP BTP services injected as VCAP_SERVICES environment variable
• Deployment (and service) for the approuter with destination mapping to the application server and subscription server (for the tenant provisioning route)
• Job for the version content deployer
• TLS certificates for the domains provided using either “Gardener” cert-management or cert-manager.io cert-manager
• Istio gateway resource for the application domains

The content deployer is used to deploy content or configuration to SAP BTP services, before using them.

Once these resources are available, the CAPApplicationVersion status changes to Ready. The controller proceeds to automatically create an object of type CAPTenant, which corresponds to the tenant of the provider subaccount. Please see tenant subscription for details on how the CAPTenant resource is reconciled.

The CAPApplicationVersion resource is meant to be immutable - it’s spec should not be modified once it is deployed. This is also prevented by our web-hooks which we recommend to always keep active (default).

5.3 - Tenant Subscription

From the perspective of CAP Operator, a valid tenant for an application is represented by the resource CAPTenant. It refers to the CAPApplication it belongs to and specifies the details of the SAP BTP subaccount representing the tenant.

apiVersion: sme.sap.com/v1alpha1
kind: CAPTenant
metadata:
  name: cap-app-01-provider
  namespace: cap-app-01
spec:
  capApplicationInstance: cap-app-01 # <-- reference to the CAPApplication
  subDomain: app-provider
  tenantId: aa2bae55d7b5-1279-456564-a7b0-aa2bae55d7b5
  version: "1.0.0" # <-- expected version of the application
  versionUpgradeStrategy: always # <-- always / never

Tenant Provisioning

The process of tenant provisioning starts when a consumer subaccount subscribes to the application, either via the SAP BTP cockpit or using the APIs provided by the SaaS provisioning service. This, in turn, initiates the asynchronous callback from the SaaS provisioning service instance into the cluster, and the request is handled by the subscription server. The subscription server validates the request and creates an instance of CAPTenant for the identified CAPApplication.

The controller, observing the new CAPTenant, will initiate the provisioning process by creating the resource CAPTenantOperation, which represents the provisioning operation.

apiVersion: sme.sap.com/v1alpha1
kind: CAPTenantOperation
metadata:
  name: cap-app-01-provider-sgz8b
  namespace: cap-app-01
spec:
  capApplicationVersionInstance: cav-cap-app-01-1 # <-- latest CAPApplicationVersion in Ready state
  subDomain: app-provider
  tenantId: aa2bae55d7b5-1279-456564-a7b0-aa2bae55d7b5
  operation: provisioning # <-- valid values are provisioning, deprovisioning and upgrade
  steps:
    - name: cap-backend # <-- derived from workload of type CAP (when workload of type TenantOperation is not specified)
      type: TenantOperation

The CAPTenantOperation is further reconciled to create Kubernetes jobs (steps), which are derived from the latest CAPApplicationVersion, which is in Ready state. The steps comprise of a TenantOperation job and optional CustomTenantOperation steps. The TenantOperation step uses built in CLI-based tenant operations from @sap/cds-mtxs to execute tenant provisioning.

The CAPTenant reaches a Ready state, only after

• a successful completion of all CAPTenantOperation steps.
• the creation of Istio VirtualService, which allows HTTP requests on the tenant subdomain to reach the application.

Tenant Deprovisioning

Similar to the tenant provisioning process, when a tenant unsubscribes from the application, the request is received by the subscription server. It validates the existence and status of the CAPTenant and submits a request for deletion to the Kubernetes API server.

The controller identifies that the CAPTenant has to be deleted, but withholds deletion until it can create and watch for a successful completion of a CAPTenantOperation of type deprovisioning. The CAPTenantOperation creates the corresponding jobs (steps), which execute the tenant deprovisioning.

5.4 - Application Upgrade

An important lifecycle aspect of operating multi-tenant CAP applications is the tenant upgrade process. With CAP Operator, these tenant upgrades can be fully automated by providing a new instance of the capapplicationversions.sme.sap.com custom resource. As you’ve already seen during the initial deployment, the CAPApplicationVersion resource describes the different components (workloads) of an application version that includes the container image to be used and the services consumed by each component. To upgrade the application, provide a new CAPApplicationVersion with the relevant image for each component and use a newer (higher) semantic version in the version field. See API Reference.

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  name: cav-cap-app-01-2
  namespace: cap-app-01
spec:
  capApplicationInstance: cap-cap-app-01 # <-- reference to CAPApplication in the same namespace
  version: "2.0.1" # <-- semantic version
  registrySecrets:
    - regcred
  workloads:
    - name: cap-backend
      consumedBTPServices:
        - app-uaa
        - app-service-manager
        - app-saas-registry
      deploymentDefinition:
        type: CAP # <-- indicates the CAP application server
        image: app.some.repo.example.com/srv/server:0.0.2
        env:
          - name: CDS_ENV
            value: production
          - name: CDS_MTX_PROVISIONING_CONTAINER
            value: '{"provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345"}}'
    - name: app-router
      consumedBTPServices:
        - app-uaa
        - app-destination
        - app-saas-registry
        - app-html5-repo-runtime
        - app-portal
      deploymentDefinition:
        type: Router
        image: app.some.repo.example.com/approuter/approuter:0.0.2
        env:
          - name: PORT
            value: 4000
          - name: TENANT_HOST_PATTERN
            value: "^(.*).(cap-app-01.cluster.shoot.canary.k8s-hana.ondemand.co|alt.shoot.example.com)"
    - name: service-content
      consumedBTPServices:
        - app-uaa
        - app-html5-repo-host
        - app-portal
      jobDefinition:
        type: Content
        image: app.some.repo.example.com/approuter/content:0.0.2
        backoffLimit: 1
    - name: tenant-operation
      consumedBTPServices:
        - app-uaa
        - app-service-manager
        - app-saas-registry
      jobDefinition:
        type: TenantOperation
        image: app.some.repo.example.com/approuter/content:0.0.2
        env:
          - name: CDS_ENV
            value: production
          - name: CDS_MTX_PROVISIONING_CONTAINER
            value: '{"provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345"}}'
    - name: notify-upgrade
      consumedBTPServices: []
      jobDefinition:
        type: CustomTenantOperation
        image: app.some.repo.example.com/approuter/content:0.0.2
        command: ["npm", "run", "notify:upgrade"]
        backoffLimit: 1
        env:
          - name: TARGET_DL
            value: group_xyz@sap.com
  tenantOperations:
    upgrade:
      - workloadName: tenant-operation
      - workloadName: notify-upgrade
        continueOnFailure: true

Note that in this version (compared to version “1” used for the initial deployment), new workloads of type TenantOperation and CustomTenantOperation have been added.

The controller component of CAP Operator reacts to the new CAPApplicationVersion resource and triggers another deployment for the application server, router and triggers the content deployment job. Once the new CAPApplicationVersion is Ready, the controller proceeds to automatically upgrade all relevant tenants i.e. by updating the version attribute on the CAPTenant resources.

The reconciliation of a CAPTenant changes its state to Upgrading and creates the CAPTenantOperation resource of type upgrade.

apiVersion: sme.sap.com/v1alpha1
kind: CAPTenantOperation
metadata:
  name: cap-app-01-provider-fgdfg
  namespace: cap-app-01
spec:
  capApplicationVersionInstance: cav-cap-app-01-2
  subDomain: cap-provider
  tenantId: aa2bae55d7b5-1279-456564-a7b0-aa2bae55d7b5
  operation: upgrade # possible values are provisioning / upgrade / deprovisioning
  steps:
    - name: "tenant-operation"
      type: TenantOperation
    - name: "notify-upgrade"
      type: CustomTenantOperation
      continueOnFailure: true # <-- can be set for workloads of type CustomTenantOperation to indicate that the success of this job is optional for the completion of the overall operation

The CAPTenantOperation creates jobs for each of the steps involved and executes them sequentially until all the jobs are finished or one of them fails. The CAPTenant is notified about the result and updates its state accordingly.

A successful completion of the CAPTenantOperation will cause the VirtualService managed by the CAPTenant to be modified to route HTTP traffic to the deployments of the newer CAPApplicationVersion. Once all tenants have been upgraded, the outdated CAPApplicationVersion can be deleted.

5.5 - Services Only Applications

This guide explains how to deploy a CAP application containing only service workloads (those that are tenant-independent).

Deploying Services Only Applications

Services only applications don’t require tenant-specific configurations. Therefore, the provider section is omitted from the CAPApplication resource, and the CAPApplicationVersion should only define deployments of type Service, alongside optional Content jobs.

Because these applications are tenant-independent, no tenant related resources are created.

The service workloads may be exposed externally via serviceExposures configuration on the version.

Application Configuration (CAPApplication)

Create a CAPApplication resource without a provider section, as shown below:

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplication
metadata:
  name: test-ca-01
  namespace: default
spec:
  btp:
    services:
      - class: xsuaa
        name: cap-uaa
        secret: cap-cap-01-uaa-bind-cf
      - class: xsuaa
        name: cap-uaa2
        secret: cap-cap-01-uaa2-bind-cf
      - class: service-manager
        name: cap-service-manager
        secret: cap-cap-01-svc-man-bind-cf
  btpAppName: test-cap-01
  domains:
    istioIngressGatewayLabels:
      - name: app
        value: istio-ingressgateway
      - name: istio
        value: ingressgateway
    primary: app-domain.test.local
    secondary:
      - foo.bar.local
  globalAccountId: btp-glo-acc-id

Version Configuration (CAPApplicationVersion)

Create a CAPApplicationVersion in the same namespace as the CAPApplication with service workloads and any content jobs. The serviceExposures section defines how the service is exposed externally, for example, via Istio VirtualService resources.

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  name: cav-ca-app-01-1
  namespace: default # Same namespace as CAPApplication
spec:
  capApplicationInstance: test-ca-01 # Reference to the CAPApplication
  version: "0.0.1"
  registrySecrets:
    - regcred
  workloads:
    - name: cap-backend-service
      consumedBTPServices: # Services used by this workload
        - app-uaa
        - app-service-manager
        - app-saas-registry
      deploymentDefinition:
        type: Service # Defines this as a service workload
        image: app.some.repo.example.com/srv/server:0.0.1
        env:
          - name: CDS_ENV
            value: production
        ports:
          - name: server
            port: 4004
            appProtocol: http
          - name: api
            port: 8000
            appProtocol: http
          - name: metrics
            port: 4005
            appProtocol: http
    - name: service-content # Example content job
      consumedBTPServices:
        - app-uaa
        - app-html5-repo-host
        - app-portal
      jobDefinition:
        type: Content
        image: app.some.repo.example.com/approuter/content:0.0.1
        backoffLimit: 1
  serviceExposures:
    - subDomain: service
      routes:
        - workloadName: cap-backend-service
          port: 4004
    - subDomain: api
      routes:
        - workloadName: cap-backend-service
          port: 8000
          path: /api

Important Considerations

• The CAPApplication reaches a Consistent state, only after
  • the latest CAPApplicationVersion is Ready.
  • the creation of Istio VirtualService, which allows HTTP requests on the serviceExposure subdomain to reach the application.
• The CAPApplication status for services only applications will have a servicesOnly field set to true.
• There is no CAPTenant or other tenant related resources created for such services only scenario. Any tenant-related logic within the service itself is the responsibility of the consuming application.
• You cannot switch a CAPApplicationVersion between services only and tenant-dependent modes after initial creation of overall application. Choose the appropriate mode from the start.
• A successful upgrade of the CAPApplicationVersion will cause any service related VirtualService resources to be modified to route HTTP traffic to the deployments of the newer CAPApplicationVersion. Once a new CAPApplicationVersion is Ready and the service only application is automatically upgraded, the outdated CAPApplicationVersion can be deleted.

5.6 - Version Monitoring

In a continuous delivery environment where newer applications versions may be deployed frequently, monitoring and cleaning up older unused versions becomes important to conserve cluster resources (compute, memory, storage etc.) and operate a clutter free system. The CAP Operator now provides application developers and operations teams to define how an application version can be monitored for usage.

Integration with Prometheus

Prometheus is the industry standard for monitoring application metrics and provides a wide variety of tools for managing and reporting metrics data. The CAP Operator (controller) can be connected to a Prometheus server by setting the PROMETHEUS_ADDRESS environment variable on the controller (see Configuration). The controller is then able to query application related metrics based on the workload specification of CAPApplicationVersions. If no Prometheus address is supplied, the version monitoring function of the controller is not started.

Configure CAPApplication

To avoid incompatible changes, version cleanup monitoring must be enabled for CAP application using the annotation sme.sap.com/enable-cleanup-monitoring. The annotation can have the following values which affects the version cleanup behavior:

Configure CAPApplicationVersion

For each workload of type deployment in a CAPApplicationVersion, it is possible to define:

1. Deletion rules: A criteria based on metrics which when satisfied signifies that the workload can be removed
2. Scrape configuration: Configuration which defines how metrics are scraped from the workload service.

Deletion Rules (Variant 1) based on Metric Type

The following example shows how a workload, named backend, is configured with deletion rules based on multiple metrics.

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  namespace: demo
  name: cav-demo-app-1
spec:
  workloads:
    - name: backend
      deploymentDefinition:
        monitoring:
          deletionRules:  
            metrics:
              - calculationPeriod: 90m
                name: current_sessions
                thresholdValue: "0"
                type: Gauge
              - calculationPeriod: 2h
                name: total_http_requests
                thresholdValue: "0.00005"
                type: Counter

This informs the CAP Operator that workload backend is supplying two metrics which can be monitored for usage.

• Metric current_sessions is of type Gauge which indicates that it is an absolute value at any point of time. When evaluating this metric, the CAP operator queries Prometheus with a PromQL expression which calculates the average value of this metric over a specified calculation period. The average value from each time series is then added together to get the evaluated value. The evaluated value is then compared against the specified threshold value to determine usage (or eligibility for cleanup).

  Evaluation steps for metric type Gauge
  Execute PromQL expression sum(avg_over_time(current_sessions{job="cav-demo-app-1-backend-svc",namespace="demo"}[90m])) to get the evaluated value
  Check whether evaluated value <= 0 (the specified thresholdValue)

• Similarly, metric total_http_requests is of type Counter which indicates that it is a cumulative value which can increment. When evaluating this metric, the CAP operator queries Prometheus with a PromQL expression which calculates the rate (of increase) of this metric over a specified calculation period. The rate of increase from each time series is then added together to get the evaluated value. The evaluated value is then compared against the specified threshold value to determine usage (or eligibility for cleanup).

  Evaluation steps for metric type Counter
  Execute PromQL expression sum(rate(total_http_requests{job="cav-demo-app-1-backend-svc",namespace="demo"}[2h])) to get the evaluated value
  Check whether evaluated value <= 0.00005 (the specified thresholdValue)

All specified metrics of a workload must satisfy the evaluation criteria for the workload to be eligible for cleanup.

Deletion Rules (Variant 2) as PromQL expression

Another way to specify the deletion criteria for a workload is by providing a PromQL expression which results a boolean scalar.

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  namespace: demo
  name: cav-demo-app-1
spec:
  workloads:
    - name: backend
      deploymentDefinition:
        monitoring:
          deletionRules:
            expression: scalar(sum(avg_over_time(current_sessions{job="cav-demo-app-1-backend-svc",namespace="demo"}[2h]))) <= bool 5

The supplied PromQL expression is executed as a Prometheus query by the CAP Operator. The expected result is a scalar boolean (0 or 1). Users may use comparison binary operators with the bool modifier to achieve the expected result. If the evaluation result is true (1), the workload is eligible for removal.

This variant can be useful when:

• the predefined evaluation based on metric types is not enough for determining usage of a workload.
• custom metrics scraping configurations are employed where the job label in the collected time series data does not mach the name of the (Kubernetes) Service created for the workload.

Scrape Configuration

Prometheus Operator is a popular Kubernetes operator for managing Prometheus and related monitoring components. A common way to setup scrape targets for a Prometheus instance is by creating the ServiceMonitor resource which specifies which Services (and ports) that should be scraped for collecting application metrics.

The CAP Operator provides an easy way to create Service Monitors which target the Services created for version workloads. The following sample shows how to configure this.

kind: CAPApplicationVersion
metadata:
  namespace: demo
  name: cav-demo-app-1
spec:
  workloads:
    - name: backend
      deploymentDefinition:
        ports:
          - appProtocol: http
            name: metrics-port
            networkPolicy: Cluster
            port: 9000
        monitoring:
          deletionRules:
            expression: scalar(sum(avg_over_time(current_sessions{job="cav-demo-app-1-backend-svc",namespace="demo"}[2h]))) <= bool 5
          scrapeConfig:
            interval: 15s
            path: /metrics
            port: metrics-port

With this configuration the CAP Operator will create a ServiceMonitor which targets the workload Service. The scrapeConfig.port should match the name of one of the ports specified on the workload.

Evaluating CAPApplicationVersions for cleanup

At specified intervals (dictated by controller environment variable METRICS_EVAL_INTERVAL), the CAP Operator selects versions which are candidates for evaluation.

• Only versions for CAPApplications where annotation sme.sap.com/enable-cleanup-monitoring is set are considered.
• All versions (spec.version) higher than the highest version with Ready status are not considered for evaluation. If there is no version with status Ready, no versions are considered.
• All versions linked to a CAPTenant are excluded from evaluation. This includes versions where the following fields of a CAPTenant point to the version:
  • status.currentCAPApplicationVersionInstance - current version of the tenant.
  • spec.version - the version to which a tenant is upgrading.

Workloads from the identified versions are then evaluated based on the defined deletionRules. Workloads without deletionRules are automatically eligible for cleanup. All workloads (with type deployment) of a version must satisfy the evaluation criteria for the version to be deleted.

5.7 - Operator Metrics

Overview

The CAP Operator includes built-in Prometheus metrics that enable users to effectively monitor and analyze the operator’s performance. These metrics can provide insights into resource usage, potential issues, and overall operator health. The metrics are accessible at the /metrics endpoint on port 9090 of both the Controller and the Subscription Server.

Controller Metrics

The Controller emits several types of metrics, including:

• Standard Go Metrics: These metrics are provided by the Prometheus Go client and include runtime statistics.
• Workqueue Metrics: These metrics are relevant to the resources being reconciled and are based on the MetricsProvider.
• Specific metrics: mentioned below.

Specific Metrics

1. Reconcile Errors – cap_op_reconcile_errors, e.g.:

cap_op_reconcile_errors{kind="CAPApplication",name="my-app",namespace="app"} 11

• Type: Counter
• Description: This metric tracks the total number of resources that failed to reconcile for each resource kind, such as CAPApplication.

2. Tenant Operations – cap_op_tenant_operations, e.g.

cap_op_tenant_operations{app="<hash>",operation="provisioning"} 83

• Type: Counter
• Description: This metric provides insights into overall tenant operations being performed.

Detailed Operational Metrics

To gain deeper insights, you can enable more granular metrics by setting the environment variable DETAILED_OPERATIONAL_METRICS to "true".

1. Failed Tenant Operations – cap_op_tenant_operation_failures, e.g.:

cap_op_tenant_operation_failures{app="<hash>",operation="upgrade",tenant_id="<guid>",namespace="app",name="my-app-tenant-op-xxyyz"} 2

• Type: Counter
• Description: This metric reveals the number of failed tenant operations, categorized by application, tenant ID, and specific operation details.

2. Last Tenant Operation Duration – cap_op_last_tenant_operation_duration_seconds, e.g.:

cap_op_last_tenant_operation_duration_seconds{app="<hash>",tenant_id="<guid>"} 17

• Type: Gauge
• Description: This metric measures the duration (in seconds) of the last tenant operation for a specified application and tenant.

Subscription Server Metrics

The Subscription Server emits the following metrics:

• Standard Go Metrics: These metrics are provided by the Prometheus Go client and include runtime statistics.
• Specific metrics: mentioned below.

Specific Metrics

1. Subscription Requests Total – cap_op_subscription_requests_total, e.g.:

cap_op_subscription_requests_total{code="202",method="POST"} 2024

• Type: Counter
• Description: This metric tracks the total number of subscription requests that were processed, categorized by HTTP method and response code.

2. Inflight Subscription Requests – cap_op_subscription_requests_inflight, e.g.:

cap_op_subscription_requests_inflight{} 4

• Type: Gauge
• Description: This metric indicates the number of subscription requests currently being processed by the server.

Conclusion

The CAP Operator provides a rich set of metrics to facilitate monitoring and operational insights. By effectively leveraging these metrics, you can monitor and ensure the reliability and performance of your applications. For further details, consider exploring the Prometheus documentation and integrating these metrics into your monitoring systems.

5.8 - Resources

5.8.1 - CAPApplication

Here’s an example of a fully configured CAPApplication:

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplication
metadata:
  name: cap-app
  namespace: cap-ns
spec:
  btp:
    services:
      - class: xsuaa
        name: cap-uaa
        secret: cap-uaa-bind
      - class: saas-registry
        name: cap-saas-reg
        secret: cap-saas-reg-bind
      - class: service-manager
        name: cap-service-manager
        secret: cap-svc-man-bind
      - class: destination
        name: cap-destination
        secret: cap-bem-02-dest-bind
      - class: html5-apps-repo
        name: cap-html5-repo-host
        secret: cap-html5-repo-bind
      - class: html5-apps-repo
        name: cap-html5-repo-runtime
        secret: cap-html5-rt-bind
      - class: portal
        name: cap-portal
        secret: cap-portal-bind
      - class: business-logging
        name: cap-business-logging
        secret: cap-business-logging-bind
  btpAppName: cap-app
  domains:
    istioIngressGatewayLabels: # <-- labels used to identify Load Balancer service used by Istio
      - name: app
        value: istio-ingressgateway
      - name: istio
        value: ingressgateway
    primary: cap-app.cluster.project.shoot.url.k8s.example.com
    secondary:
      - alt-cap.cluster.project.shoot.url.k8s.example.com
  globalAccountId: 2dddd48d-b45f-45a5-b861-a80872a0c8a8
  provider: # <-- provider tenant details
    subDomain: cap-app-provider
    tenantId: 7a49218f-c750-4e1f-a248-7f1cefa13010

The overall list of SAP BTP service instances and respective Secrets (credentials) required by the application is specified as an array in btp.services. These service instances are assumed to exist in the provider subaccount. Operators such as cf-service-operator or sap-btp-service-operator can be used to declaratively create these service instances and their credentials as Kubernetes resources.

The provider section specifies details of the provider subaccount linked to this application, while globalAccountId denotes the global account in which the provider subaccount is created. Within a global account, the btpAppName has to be unique as this is equivalent to XSAPPNAME, which is used in various SAP BTP service and application constructs.

The domains section provides details of where the application routes are exposed. Within a “Gardener” cluster, the primary application domain is a subdomain of the cluster domain, and “Gardener” cert-management will be used to request a wildcard TLS certificate for the primary domain. Additional secondary domains may also be specified (for example, for customer-specific domains) for the application.

NOTE: While the same secondary domain can technically be used across applications; the consumers need to ensure that the tenant sub-domains are unique across such applications that share the same domain!

istioIngressGatewayLabels are key-value pairs (string) used to identify the ingress controller component of Istio and the related load balancer service. These values are configured during the installation of Istio service mesh in the cluster.

NOTE: The provider section is omitted for Services Only Applications

5.8.2 - CAPApplicationVersion

The CAPApplicationVersion has the following high level structure:

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  name: cav-cap-app-v1
  namespace: cap-ns
spec:
  version: 3.2.1 # <-- semantic version (must be unique within the versions of a CAP application)
  capApplicationInstance: cap-app
  registrySecrets: # <-- image pull secrets to be used in the workloads
    - regcred
  workloads: # <-- define deployments and jobs used for this application version
    - name: "cap-backend"
      deploymentDefinition: # ...
      consumedBTPServices: # ...
    - name: "app-router"
      deploymentDefinition: # ...
      consumedBTPServices: # ...
    - name: "service-content"
      jobDefinition: # ...
      consumedBTPServices: # ...
    - name: "tenant-operation"
      jobDefinition: # ...
      consumedBTPServices: # ...
  tenantOperations: # ... <-- (optional)

• An instance of CAPApplicationVersion is always related to an instance of CAPApplication in the same namespace. This reference is established using the attribute capApplicationInstance.
• An array of workloads (workloads) must be defined that include the various software components of the SAP Cloud Application Programming Model application. A deployment representing the CAP application server or a job that which is used for tenant operations are examples of such workloads. A workload must have either a deploymentDefinition or a jobDefinition. See the next section for more details.
• An optional attribute tenantOperations can be used to define a sequence of steps (jobs) to be executed during tenant operations (provisioning / upgrade / deprovisioning).

The CAPApplicationVersion resource is meant to be immutable - it’s spec should not be modified once it is deployed. This is also prevented by our web-hooks which we recommend to always keep active (default).

Workloads with deploymentDefinition

name: cap-backend
consumedBTPServices: # <-- an array of service instance names referencing the SAP BTP services defined in the CAPApplication resource
  - cap-uaa
  - cap-saas-reg
deploymentDefinition:
  type: CAP # <-- possible values are CAP / Router / Additional / Service
  image: some.repo.example.com/cap-app/server:3.22.11 # <-- container image
  env: # <-- (optional) same as in core v1 pod.spec.containers.env
    - name: SAY
      value: "I'm GROOT"
  replicas: 3 # <-- (optional) replicas for scaling
  ports:
    - name: app-port
      port: 4004
      routerDestinationName: cap-server-url
    - name: tech-port
      port: 4005
  monitoring:
    scrapeConfig:
      port: tech--port
    deletionRules:
      expression: scalar(sum(avg_over_time(current_sessions{job="cav-cap-app-v1-cap-backend-svc",namespace="cap-ns"}[2h]))) <= bool 5

The type of the deployment is important to indicate how the operator handles this workload (for example, injection of destinations to be used by the approuter). Valid values are:

• CAP to indicate a CAP application server. Only one workload of this type can be used at present.
• Router to indicate a version of AppRouter. Only one workload of this type can be used.
• Additional to indicate supporting components that can be deployed along with the CAP application server.

You can define optional attributes such as replicas, env, resources, probes, securityContext, initContainers and ports to configure the deployment.

Port configuration

It’s possible to define which (and how many) ports exposed by a deployment container are exposed inside the cluster (via services of type ClusterIP). The port definition includes a name in addition to the port number being exposed.

For deploymentDefinition, other than type Router it would be possible to specify a routerDestinationName which would be used as a named destination injected into the approuter.

The port configurations aren’t mandatory and can be omitted. This would mean that the operator will configure services using defaults. The following defaults are applied if port configuration is omitted:

• For workload of type CAP, the default port used by CAP, 4004, will be added to the service and a destination with name srv-api will be added to the approuter referring to this service port (any existing destinations environment configuration for this workload will be taken over by overwriting the URL).
• For workload of type Router, the port 5000 will be exposed in the service. This service will be used as the target for HTTP traffic reaching the application domain (domains are specified within the CAPApplication resource).

NOTE: If multiple ports are configured for a workload of type Router, the first available port will be used to target external traffic to the application domain.

Monitoring configuration

For each workload of type deployment in a CAPApplicationVersion, it is possible to define:

1. Deletion rules: A criteria based on metrics which when satisfied signifies that the workload can be removed
2. Scrape configuration: Configuration which defines how metrics are scraped from the workload service.

Details of how to configure workload monitoring can be found here.

Workloads with jobDefinition

workloads:
  # ... deployment workloads have been omitted in this example
  - name: "content-deployer"
    consumedServices: # ...
    jobDefinition:
      type: Content
      image: some.repo.example.com/cap-app/content:1.0.1
  - name: "tenant-operation"
    consumedServices: # ...
    jobDefinition:
      type: TenantOperation
      image: some.repo.example.com/cap-app/server:3.22.11
      backoffLimit: 2 # <-- determines retry attempts for the job on failure (default is 6)
      ttlSecondsAfterFinished: 300 # <-- the job will be cleaned up after this duration
      env:
        - name: CDS_ENV
          value: production
        - name: CDS_MTX_PROVISIONING_CONTAINER
          value: '{"provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345"}}'
  - name: "notify-upgrade"
    consumedServices: # ...
    jobDefinition:
      type: CustomTenantOperation
      image: # ...
      command: ["npm", "run", "notify:upgrade"] # <-- custom entry point for the container allows reuse of a container image with multiple entry points
      backoffLimit: 1
  - name: "create-test-data"
    consumedServices: # ...
    jobDefinition:
      type: CustomTenantOperation
      image: # ...
      command: ["npm", "run ", "deploy:testdata"]

Workloads with a jobDefinition represent a job execution at a particular point in the lifecycle of the application or tenant. The following values are allowed for type in such workloads:

• Content: A content deployer job that can be used to deploy (SAP BTP) service specific content from the application version. This job is executed as soon as a new CAPApplicationVersion resource is created in the cluster. Multiple workloads of this type may be defined in the CAPApplicationVersion and the order in which they are executed can be specified via ContentJobs.
• TenantOperation: A job executed during provisioning, upgrade, or deprovisioning of a tenant (CAPTenant). These jobs are controlled by the operator and use the cds/mtxs APIs to perform HDI content deployment by default. If a workload of type TenantOperation isn’t provided as part of the CAPApplicationVersion, the workload with deploymentDefinition of type CAP will be used to determine the jobDefinition (image, env, etc.). Also, if cds/mtxs APIs are used, command can be used by applications to trigger tenant operations with custom command.
• CustomTenantOperation: An optional job which runs before or after the TenantOperation where the application can perform tenant-specific tasks (for example, create test data).

Sequencing tenant operations

A tenant operation refers to provisioning, upgrade or deprovisioning which are executed in the context of a CAP application for individual tenants (i.e. using the cds/mtxs or similar modules provided by CAP). Within the workloads, we have already defined two types of jobs that are valid for such operations, namely TenantOperation and CustomTenantOperation.

The TenantOperation is mandatory for all tenant operations.

In addition, you can choose which CustomTenantOperation jobs run for a specific operation and in which order. For example, a CustomTenantOperation deploying test data to the tenant database schema would need to run during provisioning, but must not run during deprovisioning.

The field tenantOperations specifies which jobs are executed during the different tenant operations and the order they are executed in.

spec:
  workloads: # ...
  tenantOperations:
    provisioning:
      - workloadName: "tenant-operation"
      - workloadName: "create-test-data"
    upgrade:
      - workloadName: "notify-upgrade"
        continueOnFailure: true # <-- indicates the overall operation may proceed even if this step fails
      - workloadName: "tenant-operation"
      - workloadName: "create-test-data"
    # <-- as the deprovisioning steps are not specified, only the `TenantOperation` workload (first available) will be executed

In the example above, for each tenant operation, not only are the valid jobs (steps) specified, but also the order in which they are to be executed. Each step in an operation is defined with:

• workloadNamerefers to the job workload executed in this operation step
• continueOnFailure is valid only for CustomTenantOperation steps and indicates whether the overall tenant operation can proceed when this operation step fails.

NOTE:

• Specifying tenantOperations is required only if CustomTenantOperations are to be used. If not specified, each operation will comprise of only the TenantOperation step (the first one available from workloads).
• The tenantOperations and specified sequencing are valid only for tenants provisioned (or deprovisioned) on the corresponding CAPApplicationVersion and for tenants being upgraded to this CAPApplicationVersion.

Sequencing content jobs

When you create a CAPApplicationVersion workload, you can define multiple content jobs. The order in which these jobs are executed is important, as some jobs may depend on the output of others. The ContentJobs property allows you to specify the order in which content jobs are executed.

spec:
  workloads: # ...
  tenantOperations: # ...
  contentJobs:
    - content-deployer-service
    - content-deployer-ui

Services / Tenant independent workloads

spec:
  workloads: 
    - name: cap-backend-service
      consumedBTPServices: # <-- an array of service instance names referencing the SAP BTP services defined in the CAPApplication resource
        - cap-uaa
        - cap-saas-reg
      deploymentDefinition:
        type: Service # <-- possible values are CAP / Router / Additional / Service
        image: some.repo.example.com/cap-app/server:3.22.11 # <-- container image
        env: # <-- (optional) same as in core v1 pod.spec.containers.env
          - name: SAY
            value: "I'm GROOT"
        replicas: 3 # <-- (optional) replicas for scaling
        ports: # Some configs like routerDestinationName are not relevant for ServicesOnly scenario
          - name: app-port
            port: 4004
          - name: tech-port
            port: 4005
          - name: api
            port: 8000
            appProtocol: http
    - name: service-content
      consumedBTPServices:
        - app-uaa
        - app-html5-repo-host
        - app-portal
      jobDefinition:
        type: Content
        image: app.some.repo.example.com/approuter/content:0.0.1
        backoffLimit: 1
  serviceExposures:
    - subDomain: service
      routes:
        - workloadName: cap-backend-service
          port: 4004
    - subDomain: api
      routes:
        - workloadName: cap-backend-service
          port: 8000
          path: /api

Here too the type of the deployment is important to indicate how the operator handles this workload. For Services Only scenario only Service type is valid. You can also have jobs of type Content to deploy application content related to UI, DB as needed.

Other attributes can be configured as documented.

Port configuration

It’s possible to define which (and how many) ports exposed by a deployment container are exposed inside the cluster (via services of type ClusterIP). The port definition includes a name in addition to the port number being exposed.

For service only workloads the routerDestinationName is not relevant.

The port configurations are mandatory and cannot be omitted.

Full Example

apiVersion: sme.sap.com/v1alpha1
kind: CAPApplicationVersion
metadata:
  name: cav-cap-app-v1
  namespace: cap-ns
spec:
  version: 3.2.1
  capApplicationInstance: cap-app
  registrySecrets:
    - regcred
  workloads:
    - name: cap-backend
      consumedBTPServices:
        - cap-uaa
        - cap-service-manager
        - cap-saas-reg
      deploymentDefinition:
        type: CAP
        image: some.repo.example.com/cap-app/server:3.22.11
        env:
          - name: CDS_ENV
            value: production
          - name: CDS_MTX_PROVISIONING_CONTAINER
            value: '{"provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345"}}'
        replicas: 3
        ports:
          - name: app-port
            port: 4004
            routerDestinationName: cap-server-url
          - name: tech-port
            port: 4005
            appProtocol: grpc
        monitoring:
          scrapeConfig:
            port: tech--port
          deletionRules:
            expression: scalar(sum(avg_over_time(current_sessions{job="cav-cap-app-v1-cap-backend-svc",namespace="cap-ns"}[2h]))) <= bool 5
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 4005
          initialDelaySeconds: 20
          periodSeconds: 10
          timeoutSeconds: 2
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 4005
          initialDelaySeconds: 20
          periodSeconds: 10
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 200m
            memory: 500Mi
          requests:
            cpu: 20m
            memory: 50Mi
        securityContext:
          runAsUser: 1000
          runAsGroup: 2000
    - name: "app-router"
      consumedBTPServices:
        - cap-uaa
        - cap-saas-reg
        - cap-html5-repo-rt
      deploymentDefinition:
        type: Router
        image: some.repo.example.com/cap-app/router:4.0.1
        env:
          - name: PORT
            value: "3000"
        ports:
          - name: router-port
            port: 3000
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 3000
          initialDelaySeconds: 20
          periodSeconds: 10
          timeoutSeconds: 2
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 3000
          initialDelaySeconds: 20
          periodSeconds: 10
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 200m
            memory: 500Mi
          requests:
            cpu: 20m
            memory: 50Mi
        podSecurityContext:
          runAsUser: 2000
          fsGroup: 2000
    - name: "service-content"
      consumedServices:
        - cap-uaa
        - cap-portal
        - cap-html5-repo-host
      jobDefinition:
        type: Content
        image: some.repo.example.com/cap-app/content:1.0.1
        securityContext:
          runAsUser: 1000
          runAsGroup: 2000
    - name: "ui-content"
      consumedServices:
        - cap-uaa
        - cap-portal
        - cap-html5-repo-host
      jobDefinition:
        type: Content
        image: some.repo.example.com/cap-app/ui-content:1.0.1
        securityContext:
          runAsUser: 1000
          runAsGroup: 2000
    - name: "tenant-operation"
      consumedServices: # ...
      jobDefinition:
        type: TenantOperation
        image: some.repo.example.com/cap-app/server:3.22.11
        backoffLimit: 2
        ttlSecondsAfterFinished: 300
        env:
          - name: CDS_ENV
            value: production
          - name: CDS_MTX_PROVISIONING_CONTAINER
            value: '{"provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345"}}'
    - name: "notify-upgrade"
      consumedServices: []
      jobDefinition:
        type: CustomTenantOperation
        image: some.repo.example.com/cap-app/server:3.22.11
        command: ["npm", "run", "notify:upgrade"]
        backoffLimit: 1
    - name: "create-test-data"
      consumedServices:
        - cap-service-manager
      jobDefinition:
        type: CustomTenantOperation
        image: some.repo.example.com/cap-app/server:3.22.11
        command: ["npm", "run ", "deploy:testdata"]
  tenantOperations:
    provisioning:
      - workloadName: "tenant-operation"
      - workloadName: "create-test-data"
    upgrade:
      - workloadName: "notify-upgrade"
        continueOnFailure: true
      - workloadName: "tenant-operation"
      - workloadName: "create-test-data"
  contentJobs:
    - service-content
    - ui-content

NOTE: The CAP Operator workloads supports several configurations (present in the kubernetes API), which can be configured by looking into our API reference:

• Container API reference for generic container-specific configuration
• Deployment API reference for deployment-specific configuration
• Job API reference for job-specific configuration

The supported configurations is kept minimal intentionally to keep the overall API simple by considering commonly used configurations.

Note: For initContainers nearly the same environment variables as the main container are made available including VCAP_SERVICES environment.

5.8.3 - CAPTenant

The CAPTenant resource indicates the existence of a tenant in the related application (or one that is current being provisioned). The resource starts with a Provisioning state and moves to Ready when successfully provisioned. Managing tenants as Kubernetes resources allows you not only to control the lifecycle of the entity, but also allows you to control other requirements that must be fulfilled for the application to serve tenant-specific requests (for example, creating of networking resources).

apiVersion: sme.sap.com/v1alpha1
kind: CAPTenant
metadata:
  name: cap-app-consumer-ge455
  namespace: cap-ns
spec:
  capApplicationInstance: cap-app
  subDomain: consumer-x
  tenantId: cb46733-1279-48be-fdf434-aa2bae55d7b5
  version: "1"
  versionUpgradeStrategy: always

The specification contains attributes relevant for SAP BTP, which identifies a tenant such as tenantId and subDomain.

The version field corresponds to the CAPApplicationVersion on which the tenant is provisioned or was upgraded. When a newer CAPApplicationVersion is available, the operator automatically increments the tenant version, which triggers the upgrade process. The versionUpgradeStrategy is by default always, but can be set to never in exceptional cases to prevent an automatic upgrade of the tenant.

5.8.4 - CAPTenantOperation

apiVersion: sme.sap.com/v1alpha1
kind: CAPTenantOperation
metadata:
  name: cap-app-consumer-ge455-77kb9
  namespace: cap-ns
spec:
  capApplicationVersionInstance: cav-cap-app-v2
  operation: upgrade
  steps:
    - continueOnFailure: true
      name: tenant-operation
      type: CustomTenantOperation
    - name: tenant-operation
      type: TenantOperation
    - name: create-test-data
      type: CustomTenantOperation
  subDomain: consumer-x
  tenantId: cb46733-1279-48be-fdf434-aa2bae55d7b5

The example above shows a CAPTenantOperation created to execute an upgrade operation on a tenant. In addition to tenant details, the CAPApplicationVersion to be used for the operation is specified. In case of upgrade or a provisioning operation, this would be the target CAPApplicationVersion whereas for deprovisioning, it would be the current CAPApplicationVersion of the tenant.

The operation is completed by executing a series of steps (jobs) which are specified in or derived from the CAPApplicationVersion. Each step refers to a workload of type TenantOperation or CustomTenantOperation. When CAPTenantOperation is created by CAP Pperator, there must be at least one step of type TenantOperation (which is the job used for the database schema update using CAP provided modules).

CustomTenantOperation jobs are hooks provided to the application, which can be executed before or after the actual TenantOperation. For applications to be able to identify the context of an execution, each job is injected with the following environment variables:

• CAPOP_APP_VERSION: The (semantic) version from the relevant CAPApplicationVersion
• CAPOP_TENANT_ID: Tenant identifier of the tenant for which the operation is executed
• CAPOP_TENANT_OPERATION: The type of operation - provisioning, deprovisioning, or upgrade
• CAPOP_TENANT_SUBDOMAIN: Subdomain (from subaccount) belonging to the tenant for which the operation is executed
• CAPOP_TENANT_TYPE: The type of tenant - provider or consumer
• CAPOP_APP_NAME: The BTP App Name from the corresponding CAPApplication configuration
• CAPOP_GLOBAL_ACCOUNT_ID: The Global Account Identifier from the corresponding CAPApplication configuration
• CAPOP_PROVIDER_TENANT_ID: The provider tenant identifier from the corresponding CAPApplication configuration
• CAPOP_PROVIDER_SUBDOMAIN: The provider tenant subdomain from the corresponding CAPApplication configuration

Note that all of the above environment variables are also made available on the corresponding initContainers (along with other relevant VCAP_SERVICES credentials)

5.8.5 - CAPTenantOutput

The CAPTenantOutput may be used to add additional data to the asynchronous callback parameters from the SaaS provisioning service during tenant onboarding. The resource is not reconciled but just consumed by the subscription server to generate additional data. It has the following structure:

apiVersion: sme.sap.com/v1alpha1
kind: CAPTenantOutput
metadata:
  name: cap-app-consumer-output
  namespace: cap-ns
  labels:
    sme.sap.com/btp-tenant-id: cb46733-1279-48be-fdf434-aa2bae55d7b5
spec:
  subscriptionCallbackData: '{foo: bar}'
  

The example above shows an instance of the resource that is associated with a tenant via the sme.sap.com/btp-tenant-id label (which must be set by consumers).

6 - Troubleshooting

Usage of @sap/cds-mtxs library for multitenancy

The CAP Operator utilizes the @sap/cds-mtxs library. Prior to version 0.7.0 one could disable this by setting the IS_MTXS_ENABLED environment variable to “false” in the TenantOperation workload, in which case the old @sap/cds-mtx library-based wrapper job was used instead. However, this is no longer supported and is removed as support for older CDS version (v6) has ended.

CAP Operator supports the usage of @sap/cds-mtxs (which replaces the former @sap/cds-mtx library) from the SAP Cloud Application Programming Model by default.

This enables us to use built-in (into @sap/cds-mtxs) CLI-based handling for tenant operations during provisioning, deprovisioning, and upgrading tenants.

As of now, for the usage of this new library, you (depending on your k8s cluster hardening setup) need to add additional securityContext for the TenantOperation and also CAP workloads as shown in the sample below.

 - name: tenant-job
    consumedBTPServices:
    - "{{ include "xsuaaInstance" . }}"
    - "{{ include "serviceManagerInstance" . }}"
    - "{{ include "saasRegistryInstance" . }}"
    jobDefinition:
      type: TenantOperation
      env:
      - name: CDS_ENV
        value: production
      - name: CDS_MTX_PROVISIONING_CONTAINER
        value: '{ "provisioning_parameters": { "database_id": "16e25c51-5455-4b17-a4d7-43545345345" } }'
      image: "some.repo.example.com/cap-app/server"
      securityContext: # needed until CAP resolves issue with folder creation in the root dir of the app container at runtime
        runAsUser: 1000

Secret/credential handling for different workloads of the CAP Operator

Libraries like xsenv/cds(CAP) handle credentials differently in different environments (CF, K8s) and on K8s when using credential data directly from secrets, any JSON data type information related to the data values may get lost and lead to inconsistencies.

This issue is now addressed by the SAP Service Binding Specification, which mandates the addition of metadata to these secrets. Both btp-service-operator and cf-service-operator supports the addition of metadata. But, in case this feature is not used in your clusters, CAP Operator avoids inconsistencies by creating VCAP_SERVICES environment variable across all workloads and expects all SAP BTP services credentials to be available in Kubernetes Secrets under a key credentials.

This can be achieved using the secretKey property for a ServiceBinding created using btp-service-operator or cf-service-operator, for example:

apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceBinding
metadata:
  name: uaa
  namespace: demo
spec:
  serviceInstanceName: uaa
  name: app-uaa
  secretKey: credentials

We recommend that you use secretKey, even when credential metadata is available to reduce the overhead of interpreting parsing multiple JSON attributes.

HTTP requests reaching the AppRouter are not getting forwarded to the application server (pods)

The Approuter component maps incoming requests to destinations (applications or services) that have been configured. If you’re using an xs-app.json file with your Approuter to specify route mapping to various destinations, ensure that the destinationName property for the SAP Cloud Application Programming Model back end is specified in the corresponding CAPApplicationVersion configuration. CAP Operator will inject this destination to the Approuter pods (via environment variables).

HTTP Requests Timing Out in the Approuter for Long-Running Operations in Back End Workload

If your back-end service is known to take a long time, configure the destinations environment variable on the Approuter component to set the desired timeout configuration for that destination (destinationName). CAP Operator will overwrite the URL part of that destination to point to the right workload, the remaining settings are taken over exactly as configured.

Supported Approuter Version

Use @sap/approuter version 14.x.x (or higher).

CAP Operator Resources Can’t Be Deleted in the K8S Cluster/Namespace

All custom resource objects (CROs) created by CAP Operator are protected with finalizers to ensure a proper cleanup takes place. For instance, when deleting a CAPApplication CRO, any existing tenants would be deprovisioned automatically to avoid inconsistenties. Once the deprovisioning is successful, the corresponding CROs would be removed automatically. The provider CAPTenant resource can’t be deleted before deleting a consistent CAPApplication. NOTE: CAP operator needs the secrets from service instances/bindings to exist for the entire lifecycle of the SAP Cloud Application Programming Model application. Removing the service instances/bindings i.e. the secrets from the cluster while the CAP application related CROs still exist would cause leftover resources in cluster (and perhaps the db). Recovering from such inconsistent states might not even be possible. Such a situation can easily arise when using helm delete/uninstall as the order of deletion of resouces is not configurable. We recommend that you do this with care. It’s important that you ensure that the secrets from service instance/bindings aren’t deleted before any SAP Cloud Application Programming Model application that consumes those secrets is completely removed.

7 - Support

Bugs

To report a bug, create an issue.

Anything missing? Please let us know or raise a PR.

Communication Channels

Reach out to the project team and the project community via the following communication channels:

• Issues: GitHub
• Email: CAP Operator

8 - Reference

Packages:

• sme.sap.com/v1alpha1

sme.sap.com/v1alpha1

• CAPApplication
• CAPApplicationVersion
• CAPTenant
• CAPTenantOperation
• CAPTenantOutput
• ClusterDomain
• Domain

CAPApplication