This is the multi-page printable view of this section. Click here to print.
cf-service-operator adds Cloud Foundry spaces, service instances, and service bindings as resource types to Kubernetes clusters, allowing to manage Cloud Foundry service instances and bindings in a declarative way.
This website provides the full technical documentation for the project, and can be used as a reference; if you feel that there’s anything missing, please let us know or raise a PR to add it.
cf-service-operator relies on four custom resource types, the according definitions can be found
here.
These definitions must be deployed before the operator can be started, otherwise it will fail.
The core of the cf-service-operator installation is the executable built from this repository
(Docker image ghcr.io/sap/cf-service-operator).
It implements both the controller logic and validating/mutating webhooks for the above custom resource types.
A valid deployment consists of:
Note that it is highly recommended to always activate the webhooks, as they are not only validating, but also adding default values to our custom resource objects. Running without this mutating functionality might lead to unexpected behavior.
The following deployment methods are available (recommended is Helm).
The recommended way to deploy cf-service-operator is to use the Helm chart, also available in packaged form:
The chart does not require any mandatory parameters, so deploying cf-service-operator is as easy as
helm repo add cf-service-operator https://sap.github.io/cf-service-operator-helm
helm -n cf-service-operator upgrade -i cf-service-operator cf-service-operator/cf-service-operator
Below you will find details on configuration options.
cf-service-operator accepts the following command line flags:
Usage of manager:
-cluster-resource-namespace string
The namespace for secrets in which cluster-scoped resources are found.
-health-probe-bind-address string
The address the probe endpoint binds to. (default ":8081")
-kubeconfig string
Paths to a kubeconfig. Only required if out-of-cluster.
-leader-elect
Enable leader election for controller manager.
Enabling this will ensure there is only one active controller manager.
-metrics-bind-address string
The address the metric endpoint binds to. (default ":8080")
-sap-binding-metadata
Enhance binding secrets by SAP binding metadata by default.
-webhook-bind-address string
The address the webhook endpoint binds to. (default ":9443")
-webhook-tls-directory string
The directory containing tls server key and certificate, as tls.key and tls.crt;
defaults to $TMPDIR/k8s-webhook-server/serving-certs
-zap-devel
Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn);
Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) (default: true)
-zap-encoder value
Zap log encoding (one of 'json' or 'console')
-zap-log-level value
Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error',
or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
-zap-stacktrace-level value
Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
-zap-time-encoding value
Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano').
Defaults to 'epoch'.
Notes:
-cluster-resource-namespace defaults to the operator’s namespace; otherwise this flag is mandatory.-kubeconfig (if present)$KUBECONFIG(if present)$HOME/.kube/config (if existing)
Thus when running in-cluster, it is usually not necessary to specify the flag or environment variable, such that the operator just
uses the according service account’s kubeconfig.cf-service-operator honors the following environment variables:
$KUBECONFIG the path to the kubeconfig used by the operator executable; note that this has lower precedence than the command line flag -kubeconfig.cf-service-operator uses logr with zap for logging. Please check the according documentation for details about how to configure logging.
cf-service-operator introduces the following resource types:
Space and ClusterSpace: used to represent or manage Cloud Foundry spaces at namespace or cluster level. A ServiceInstance object always references exactly one Space or ClusterSpace.
ServiceInstance: used to manage (create/update) a Cloud Foundry service instance. A ServiceInstance references a Space or ClusterSpace, the Cloud Foundry service offering/plan, and optionally defines service parameters.
ServiceBinding: used to manage (create/update) a Cloud Foundry service binding. A ServiceBinding references a ServiceInstance Object, and defines the Kubernetes secret used to store the retrieved service key. Optionally binding parameters can be specified.
Objects of type spaces.cf.cs.sap.com represent Cloud Foundry spaces at the scope of a Kubernetes namespace.
Spaces can be defined as managed or unmanaged, and can be referenced by ServiceInstance objects deployed into the same namespace.
A Space object is called unmanaged if it just references an already existing Cloud Foundry space by its GUID via spec.guid.
The Cloud Foundry space will not be touched at all by the controller.
It just serves as a reference for ServiceInstance objects to be linked with the underlying Cloud Foundry space. For example:
apiVersion: cf.cs.sap.com/v1alpha1
kind: Space
metadata:
name: k8s
namespace: demo
spec:
# Cloud Foundry space guid
guid: 0a61a2ea-0326-43b6-bc08-3510bd32c5e8
# Secret containing authentication details (Cloud Foundry API address, username, password)
authSecretName: k8s-space
The referenced secret (expected in the same namespace) contains connection details for the space:
apiVersion: v1
kind: Secret
metadata:
name: k8s-space
namespace: demo
stringData:
url: https://api.cf.sap.hana.ondemand.com
username: "<email>"
password: "<password>"
Here the user specified in username should have at least the space developer role in Cloud Foundry.
A managed Space is not linked with an existing Cloud Foundry space. Instead it contains a reference to the target Cloud Foundry
organization as spec.organizationName.
The controller will then try to maintain a Cloud Foundry space called metadata.name in that organization, and also delete it once the Space object is deleted.
For example:
apiVersion: cf.cs.sap.com/v1alpha1
kind: Space
metadata:
name: k8s
namespace: demo
spec:
# Cloud Foundry organization
organizationName: my-org
# Secret containing authentication details (Cloud Foundry API address, username, password)
authSecretName: k8s-space
The name of the Cloud Foundry space can be overridden by specifying spec.name.
The referenced secret looks the same as in the unmanaged case, but in addition,
it is possible to provide different credentials for the space management, such as:
apiVersion: v1
kind: Secret
metadata:
name: k8s-space
namespace: demo
stringData:
url: https://api.cf.sap.hana.ondemand.com
username: "<email>"
password: "<password>"
org_username: "<email>"
org_password: "<password>"
The user specified as org_username is then expected to have the organization manager role in Cloud Foundry.
If omitted, the user specified in username will be used to create/update/delete the space (and that user of course must be an organization manager in that case).
Finally, the user specified in username will be added as a space manager to the space.
Objects of type clusterspaces.cf.cs.sap.com represent a Cloud Foundry spaces at Kubernetes cluster scope.
Cluster Spaces can be defined as managed or unmanaged, and can be referenced by ServiceInstance objects across the whole cluster.
A ClusterSpace object is called unmanaged if it just references an already existing Cloud Foundry space by its GUID via spec.guid.
The Cloud Foundry space will not be touched at all by the controller.
It just serves as a reference for ServiceInstance objects to be linked with the underlying Cloud Foundry space. For example:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ClusterSpace
metadata:
name: k8s
spec:
# Cloud Foundry space guid
guid: 0a61a2ea-0326-43b6-bc08-3510bd32c5e8
# Secret containing authentication details (Cloud Foundry API address, username, password)
authSecretName: k8s-space
The referenced secret must have the same structure used for Space objects, and by default is looked up in the namespace where the operator is deployed (but this can be overridden by command line flag, see the configuration section for more details).
A managed ClusterSpace is not linked with an existing Cloud Foundry space. Instead it contains a reference to the target Cloud Foundry
organization as spec.organizationName.
The controller will then try to maintain a Cloud Foundry space called metadata.name in that organization, and also delete it once the Space object is deleted.
For example:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ClusterSpace
metadata:
name: k8s
spec:
# Cloud Foundry organization
organizationName: my-org
# Secret containing authentication details (Cloud Foundry API address, username, password)
authSecretName: k8s-space
The name of the Cloud Foundry space can be overridden by specifying spec.name.
The referenced secret must have the same structure used for Space objects, and by default is looked up in the namespace where the operator is deployed (but this can be overridden by command line flag, see the configuration section for more details).
Objects of type serviceinstances.cf.cs.sap.com represent Cloud Foundry service instances. For
example, deploying the following descriptor will let the controller create or update a Cloud Foundry
instance of the service offering ‘xsuaa’ with plan ‘application’, in the Cloud Foundry space
referenced through the Space object given in spec.spaceName:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: uaa
namespace: demo
spec:
# Name of a Space object in the same namespace;
# this defines the Cloud Foundry space where the instance will be created
spaceName: k8s
# Name of service offering to be used
serviceOfferingName: xsuaa
# Name of service plan to be used
servicePlanName: application
In order to reference a ClusterSpace instead of a Space, the definition would change like this:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: uaa
namespace: demo
spec:
# Name of a ClusterSpace object;
# this defines the Cloud Foundry space where the instance will be created
clusterSpaceName: k8s
# Name of service offering to be used
serviceOfferingName: xsuaa
# Name of service plan to be used
servicePlanName: application
Furthermore, instead of specifying service offering and plan by name, it is possible to directly provide the guid of the service plan, such as:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: uaa
namespace: demo
spec:
# Name of a Space object in the same namespace;
# this defines the Cloud Foundry space where the instance will be created
spaceName: k8s
# Guid of service plan to be used
servicePlanGuid: 432bd9db-20e2-4997-825f-e4a937705b87
Instance parameters can be passed like this:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: uaa
namespace: demo
spec:
# Name of a Space object in the same namespace;
# this defines the Cloud Foundry space where the instance will be created
spaceName: k8s
# Name of service offering to be used
serviceOfferingName: xsuaa
# Name of service plan to be used
servicePlanName: application
# Instance parameters (inline)
# Caveat: never specify sensitive parameters here, but use parametersFrom instead!
parameters:
xsappname: myAppName
# Instance parameters (by secret key reference)
parametersFrom:
- name: uaa-params
key: parameters.json
Following the logic implemented by similar controllers (e.g. the K8s service catalog) it is allowed
to specify both parameters and parametersFrom, but it is considered an error if a top level key
occurs in more than one of the sources.
In addition, it is possible to annotate custom instance tags, such as:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: uaa
namespace: demo
spec:
# Name of a Space object in the same namespace;
# this defines the Cloud Foundry space where the instance will be created
spaceName: k8s
# Name of service offering to be used
serviceOfferingName: xsuaa
# Name of service plan to be used
servicePlanName: application
# List of custom tags
tags:
- uaa
- xsuaa
- authentication
Kubernetes annotations provide a flexible way of controlling the behavior of the reconciliation process for custom resources.
The cf-service-operator uses several such annotations for tweaking the behavior of the
reconciliation for service instances.
service-operator.cf.cs.sap.com/recreate-on-creation-failure:
By default, in all kinds of error situations, the controller will send an update request, in
order to trigger a reconciliation of the instance. However some service brokers do not really
support this approach, specifically when the initial creation of the instance has failed.
To overcome this, this annotation can be set on the service instance object. If present, the
controller will drop and recreate instances which are in a failed creation statem, i.e.
the LastOperation reported by the Cloud Foundry API is of type create and state failed.
service-operator.cf.cs.sap.com/max-retries:
This annotation defines the maximum number of retries for a failed operation before considering
the operation as failed permanently. It allows other operators using the custom resources for
service instances to specify how many times the controller should attempt to reconcile the
specific service instance before giving up, providing a mechanism to handle transient errors.
If this annotations is not set the number of retries is unlimited.
service-operator.cf.cs.sap.com/timeout-on-reconcile:
Specifies the timeout for the reconciliation process. If set, this annotation determines how
long the controller should wait before timing out the reconciliation process. This is useful for
operations that are expected to take longer than usual, allowing them to complete without
prematurely terminating.
Here are examples on how these annotations are set in the metadata section of the ServiceInstance
custom resource:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: example-instance
annotations:
service-operator.cf.cs.sap.com/recreate-on-creation-failure: "true"
service-operator.cf.cs.sap.com/max-retries: "3"
service-operator.cf.cs.sap.com/timeout-on-reconcile: "10m"
spec:
spaceName: development
serviceOfferingName: my-service
servicePlanName: standard
In this example, the service instance example-instance is configured to automatically recreate on
initial creation failure, retry up to three times on subsequent failures, and has a timeout of ten
minutes for each reconciliation attempt.
To set these annotations, you can either add them directly in the YAML file when creating or
updating a service instance or use the kubectl command line tool to patch an existing instance:
kubectl annotate serviceinstances example-instance service-operator.cf.cs.sap.com/max-retries=5 --overwrite
This command will set (or update) the max-retries annotation to 5 for the example-instance
service instance.
Objects of type servicebindings.cf.cs.sap.com represent Cloud Foundry service bindings. For example,
deploying the following descriptor will let the controller deploy a Cloud Foundry credentials binding for the
service instanced managed through the ServiceInstance object referenced by spec.serviceInstanceName:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceBinding
metadata:
name: uaa
namespace: demo
spec:
serviceInstanceName: uaa
If the binding is successful, the controller will store the retrieved binding credentials in a Kubernetes secret in the namespace of the ServiceBinding object. By default, the secret will have the same name as the ServiceBinding, and the top-level keys of the credentials object will become secret keys. In the above example, the returned secret would look like this:
apiVersion: v1
kind: Secret
metadata:
name: uaa
namespace: demo
type: Opaque
stringData:
apiurl: https://api.authentication.sap.hana.ondemand.com
clientid: sb-myAppName!t39788
clientsecret: ***
credential-type: instance-secret
identityzone: mysubaccount
identityzoneid: a48fa6e4-df75-4128-abdd-9400d01f3a18
sburl: https://internal-xsuaa.authentication.sap.hana.ondemand.com
subaccountid: 56f1b3a2-dbc2-43b1-8bd9-61e0f8290c27
tenantid: a48fa6e4-df75-4128-abdd-9400d01f3a18
tenantmode: shared
uaadomain: authentication.sap.hana.ondemand.com
url: https://mysubaccount.authentication.sap.hana.ondemand.com
verificationkey: '-----BEGIN PUBLIC KEY-----MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArF3/FjAEJx3LTt+UgM65/5LwyHVYownXmOUriLcEO82PiEPFW2n4438VTj5JTvyk42VW5E97lPuXRuVaialRjVDGBmPC9PX8U4ljYYOL3Wgpkid/PkpNe4H/s/U51xJzGTd/XoyuPw64h4v9B71L7bSjOysD5WPzI32/dIHMI0QbZTX2foB8MZqHjhJmVGll2BlT+E7Q+fwQ6bFXL3Ge7fylPu2EgAhR8rnlvrO7hIGIsIGmbGhRmrp38vdIkFsIhanRgtjh2imPh9tBhsYGxUirgzQqnEWQTvE3QZtlfNJ5fK6rht1oO7orkALHzZ2/azBIAojPk4nNx9hEvoCaVwIDAQAB-----END
PUBLIC KEY-----'
xsappname: myAppName!t39788
zoneid: a48fa6e4-df75-4128-abdd-9400d01f3a18
The name of the secret can be overridden by setting spec.secretName.
Furthermore, it is possible to render the whole service credentials object into a single key of the target secret by specifying spec.secretKey.
Finally, if the binding requires parameters, those can be passed by setting spec.parameters and/or spec.parametersFrom;
here the same logic applies as for ServiceInstance objects.
Updating parameters on the ServiceBinding object has no effect by default (because the Cloud Foundry API does not support such updates). However it is possible to enforce a recreation of the Cloud Foundry binding in that situation by setting the annotation service-operator.cf.cs.sap.com/rotate-on-parameter-change: "true".
In addition to this, setting the annotation service-operator.cf.cs.sap.com/rotate-on-instance-change: "true" triggers a recreation of the Cloud Foundry binding whenever the referenced service instance changes (due to plan or instance parameter changes).
Recently, SAP published a specification to extend binding credentials by additional metadata, to leverage better Kubernetes support in the xsenv library. By default, cf-service-operator will not add these metadata (to remain backwards compatible), but there is a global controller flag --sap-binding-metadata that can be used to enhance all created binding secrets by default. In addition, the default behavior can be overridden on a per service binding basis by setting the annotation service-operator.cf.cs.sap.com/with-sap-binding-metadata: "true", or "false".
This section provides some internal implementation details and architectural concepts abouot cf-service-operator.
Cloud Foundry API version 3 allows storing custom labels and annotations as metadata on all Cloud Foundry objects. Both labels and annotations are key-value string pairs. The difference is that labels can be used in list filters.
cf-service-operator persists the following metadata.labels on Cloud Foundry service instances and bindings:
service-operator.cf.cs.sap.com/owner: the Kubernetes ObjectMeta.uid of the owning ServiceInstance or ServiceBindingcf-service-operator persists the following metadata.annotations on Cloud Foundry service instances and bindings:
service-operator.cf.cs.sap.com/generation: the last applied Kubernetes ObjectMeta.generationservice-operator.cf.cs.sap.com/parameter-hash: a hash of the last applied instance or binding parameters (after merging).On this page you can find informations for developers and contributors.
Prerequisites
HOST_IP.kubeconfig at the root of this repository (can also be a symlink of course).Deployment
kubectl apply -f crds
envsubst < .local/k8s-resources.yaml | kubectl apply -f -
.local/getcerts.sh
Launching
Then the operator can be started by the include .vscode/launch.json file (‘Launch Operator’).
On this page you can find additional information, learnings, solutions to daily problems.
To adopt an existing Cloud Foundry instance, create a Kubernetes ServiceInstance object that specifies the spec.name with the name of the existing Cloud Foundry instance and provides the offering, plans, parameters, and tags matching the current state.
Such as:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: example-instance
namespace: demo
spec:
# Name of a Space object in the same namespace;
# this defines the Cloud Foundry space where the instance will be created
spaceName: k8s
# Name of service offering to be used
serviceOfferingName: xsuaa
# Name of service plan to be used
servicePlanName: standard
# Explicitly specify the name of the Cloud Foundry instance
name: <cf instance name>
# Current paremeters (if any)
parameters:
...
# Current tags (if any)
tags:
...
After deploying this object, it will enter an error state, complaining that an instance with the same name already exists in Cloud Foundry, but is not managed by the controller.
Check the status of the Instance. The following error is expected:
cfclient error (CF-UnprocessableEntity|10008): The service instance name is taken
To solve this, the Cloud Foundry metadata of the existing instance must be updated.
More information about how this controller leverages Cloud Foundry metadata can be found here.
The CF Service Operator provides a way to adopt orphan instances via a Kubernetes Annotation.
An automated way of adopting Cloud Foundry instance is via the Kuberneste annotation service-operator.cf.cs.sap.com/adopt-cf-resources.
During the reconciliation of an orphan ServiceInstance and ServiceBinding custom resource, the controller will check if this annotation is present. If the annotation is found then the controller will try to update the Cloud Foundry instance with label service-operator.cf.cs.sap.com/owner, and the annotations service-operator.cf.cs.sap.com/generation and service-operator.cf.cs.sap.com/parameter-hash
Here’s an example of how to use this annotation in a ServiceInstance and ServiceBinding:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
name: example-instance
namespace: demo
annotations:
service-operator.cf.cs.sap.com/adopt-cf-resources: "adopt"
spec:
spaceName: k8s
serviceOfferingName: xsuaa
servicePlanName: standard
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceBinding
metadata:
name: example-binding-instance
namespace: demo
annotations:
service-operator.cf.cs.sap.com/adopt-cf-resources: "adopt"
spec:
serviceInstanceName: example-instance
After some time the controller will consider the ServiceInstance and ServiceBinding as managed.
The AnnotationPollingIntervalReady annotation is used to specify the duration of the requeue after interval at which the operator polls the status of a Custom Resource after final state ready. It is possible to apply this annotations to Space, ServiceInstance and ServiceBiding CRs.
By using this annotation, the code allows for flexible configuration of the polling interval, making it easier to adjust the readiness checking frequency based on specific requirements or conditions.
The value of the annotation is a string representing a duration, such as “100m” or “5h”.
Usage:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
annotations:
service-operator.cf.cs.sap.com/polling-interval-ready: "3h"
In the example above the custom resource will be reconcile every three hours after reaching the state Ready.
Default Requeue After Interval
If the annotation AnnotationPollingIntervalReady is not set, the interval duration will be set to 10 minutes by default.
The AnnotationPollingIntervalFail annotation is used to specify the duration of the requeue interval at which the operator polls the status of a Custom Resource after the final states Creation Failed and Deletion Failed. Currently it is possible to apply this annotations to ServiceInstance custom resource only.
By using this annotation, the code allows for flexible configuration of the polling interval, making it easier to adjust the re-queue frequency after the failure based on specific requirements or conditions.
The value of the annotation is a string representing a duration, such as “20s” or “10m”.
Usage:
apiVersion: cf.cs.sap.com/v1alpha1
kind: ServiceInstance
metadata:
annotations:
service-operator.cf.cs.sap.com/polling-interval-fail: "5m"
In the example above the custom resource will be reconcile every five minutes after reaching the final state Failed.
Default Requeue After Interval
If the annotation AnnotationPollingIntervalFail is not set, there won’t be an immediate requeue. This means the resource will not be re-reconciled right away. The operator will consider the custom resource to be in a stable state, at least for now.
That means there is no default time duration for it, and it will return an empty result, ctrl.Result{}.