component-operator-runtime

• 1: Getting Started
• 2: Concepts
• 2.1: Components and Generators
• 2.2: Component Reconciler
• 2.3: Dependent Objects
• 3: Generators
• 3.1: Kustomize Generator
• 3.2: Helm Generator
• 3.3: Enhance Existing Generators
• 4: Usage
• 4.1: Overview
• 4.2: Scaffolder

1 - Getting Started

In this short tutorial you will learn how to scaffold a Kubernetes component operator using component-operator-runtime, and how to start with the implementation of the operator.

First of all, you have to download the component-operator-runtime scaffolding tool from the releases page. In the following we assume that the downloaded scaffold executable is somehwere in your path (for example, as /usr/local/bin/scaffold-component-operator).

Then, a git repository for the operator code is needed; in this example, we call it github.com/myorg/mycomponent-operator. We assume that you have cloned the empty repository to your local desktop, and have changed the current directory to the checked out repository.

We assume here that you are implementing a Kyma module operator, and that the managed component shall be represented by a Kubernetes type called MyComponent. Then run:

scaffold-component-operator \
  --group-name operator.kyma-project.io \
  --group-version v1alpha1 \
  --kind MyComponent \
  --operator-name mycomponent-operator.kyma-project.io \
  --go-module github.com/myorg/mycomponent-operator \
  --image mycomponent-operator:latest \
  .

This will give you a syntactically correct Go module. In order to start the operator, you first have to apply the custom resource definition into your development (e.g. kind) cluster:

kubectl apply -f crds/operator.kyma-project.io_mycomponents.yaml

Then, after copying or linking the cluster’s kubeconfig to ./tmp/kubeconfig (no worries, it will not submitted to git because ./tmp is excluded by .gitignore), you can use the generated ./vscode/launch.json to start the operator against your cluster with your Visual Studio Code. Now you are ready to instantiate your component:

kubectl apply -f - <<END
apiVersion: operator.kyma-project.io/v1alpha1
kind: MyComponent
metadata:
  namespace: default
  name: test
END

Now, after having the skeleton generated, we have to breathe life into the controller. The first step is to enhance the spec of the generated custom resource type MyComponentSpec in api/v1alpha1/types.go. In principle, all the attributes parameterizing the deployment of the managed commponent should be modeled there.

Whenever you change the runtime type, you should invoke make generate and make manifests in order to update the generated code artifacts and the custom resource definition; afterwards you should re-apply the custom resource definition to the cluster.

The next step is to implement a meaningful resource generator (the scaffolding just puts a dummy implementation called DummyGenerator into main.go). Writing such a resource generator means to implement the interface

type Generator interface {
  Generate(ctx context.Context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
}

When called by the framework, the namespace and name arguments of the Generate() method will be assigned the component’s namespace and name or, if the component or its spec implements the PlacementConfiguration interface, they will match the return values of the respective GetDeploymentNamespace(), GetDeploymentName() methods. The parameter argument will be set to the return value of the component’s GetSpec() method. In simplistic words, the spec of the component resource will be fed into the resource generator, which will return the concrete manifests of the dependent objects, which will then be applied to the cluster.

In some cases, the best option is to implement your own resource generator from scratch. When doing so, the returned resources []client.Object either have to be of type *unstructured.Unstructured, or the according type must be known to the used scheme.

In many other cases however, it makes more sense to just reuse one of the generic generators shipped with this repository.

2 - Concepts

The framework provided in this repository aims to automate the lifecycle of an arbitrary component in a Kubernetes cluster. Usually (but not necessarily) the managed component contains one or multiple other operators, including extension types, such as custom resource definitions.

Components are described as a set of Kubernetes manifests. How these manifests are produced is up to the consumer of the framework. It is possible to build up the manifests from scratch in code, or to reuse or enhance the included helm generator or kustomize generator generators. The manifest list is then applied to (or removed from) the cluster by an own deployer logic, standing out with the following features:

• apply and delete waves
• configurable status handling
• apply through replace or server-side-apply patch
• smart deletion handling in case the component contains custom types which are still in use
• impersonination
• remote deployment mode via a given kubeconfig

The component-operator-runtime framework plugs into the controller-runtime SDK by implementing controller-runtime’s Reconciler interface.

2.1 - Components and Generators

In the terminology of this project, a Kubernetes cluster component (sometimes called module) consists of a set of dependent objects that are to be deployed consistently into the Kubernetes cluster. The continuous reconciliation of the declared state of these dependent objects is the task of an operator implemented by component-operator-runtime. To achieve this, basically two interfaces have to be implemented by such an operator…

The Component Interface

In the programming model proposed by component-operator-runtime, the declared and observed state of the component is represented by a dedicated custom resource type. The corresponding runtime type has to fulfill the following interface:

package component

// Component is the central interface that component operators have to implement.
// Besides being a conroller-runtime client.Object, the implementing type has to expose accessor
// methods for the components's spec and status, GetSpec() and GetStatus().
type Component interface {
  client.Object
  // Return a read-only accessor to the component's spec.
  // The returned value has to implement the types.Unstructurable interface.
  GetSpec() types.Unstructurable
  // Return a read-write (usually a pointer) accessor to the component's status,
  // resp. to the corresponding substruct if the status extends component.Status.
  GetStatus() *Status
}

Basically, two accessor methods have to be implemented here. First, GetSpec() exposes the parameterization of the component. The only requirement on the returned type is to implement the

package types

// Unstructurable represents objects which can be converted into a string-keyed map.
// All Kubernetes API types, as well as all JSON objects could be modelled as Unstructurable objects.
type Unstructurable interface {
  ToUnstructured() map[string]any
}

interface. In most cases, the returned Unstructurable object is the spec itself, or a deep copy of the spec. In general, the implementation is allowed to return arbitrary content, as long as the receiving generator is able to process it. In particular, it is not expected by the framework that changes applied to the returned Unstructurable reflect in any way in the component; indeed, the framework will never modify the returned Unstructurable.

Finally, GetStatus() allows the framework to access (a part of) the custom resource type’s status, having the following type:

package component

// Component Status. Components must include this into their status.
type Status struct {
  ObservedGeneration int64            `json:"observedGeneration"`
  AppliedGeneration  int64            `json:"appliedGeneration,omitempty"`
  LastObservedAt     *metav1.Time     `json:"lastObservedAt,omitempty"`
  LastAppliedAt      *metav1.Time     `json:"lastAppliedAt,omitempty"`
  Conditions         []Condition      `json:"conditions,omitempty"`
  State              State            `json:"state,omitempty"`
  Inventory          []*InventoryItem `json:"inventory,omitempty"`
}

Note that, other than with the GetSpec() accessor, the framework will make changes to the returned Status structure. Thus, in almost all cases, the returned pointer should just reference the status of the component’s API type (or an according substructure of that status).

The component’s custom resource type is supposed to be namespaced, and by default, dependent objects will be created in that same namespace. To be more precise, the namespace and name parameters of the used generator’s Generate() method will be set to the component’s metadata.namespace and metadata.name, respectively. Sometimes it might be desired to override these defaults, and to render the dependent objects with a different namespace or name. To allow this, the component (or its spec) can implement

package component

// The PlacementConfiguration interface is meant to be implemented by components (or their spec) which allow
// to explicitly specify target namespace and name of the deployment (otherwise this will be defaulted as
// the namespace and name of the component object itself).
type PlacementConfiguration interface {
  // Return target namespace for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generate() as namespace and, in addition, rendered namespaced resources with
  // unspecified namespace will be placed in this namespace.
  GetDeploymentNamespace() string
  // Return target name for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generator() as name.
  GetDeploymentName() string
}

In addition, the component (or its spec) may implement

package component

// The ClientConfiguration interface is meant to be implemented by components (or their spec) which offer
// remote deployments.
type ClientConfiguration interface {
  // Get kubeconfig content. Should return nil if default local client shall be used.
  GetKubeConfig() []byte
}

in order to support remote deployments (that is, to make the deployment of the dependent objects use the specified kubeconfig), and

package component

// The ImpersonationConfiguration interface is meant to be implemented by components (or their spec) which offer
// impersonated deployments.
type ImpersonationConfiguration interface {
  // Return impersonation user. Should return system:serviceaccount:<namespace>:<serviceaccount>
  // if a service account is used for impersonation. Should return an empty string
  // if user shall not be impersonated.
  GetImpersonationUser() string
  // Return impersonation groups. Should return nil if groups shall not be impersonated.
  GetImpersonationGroups() []string
}

to use different user/groups for the deployment of dependent objects. Implementing both ClientConfiguration and ImpersonationConfiguration means that the provided kubeconfig will be impersonated as specified.

Note that, as mentioned above, the interfaces PlacementConfiguration, ClientConfiguration and ImpersonationConfiguration can be implemented by the component itself as well as by its spec type. In the theoretical case that both is the case, the implementation on the component level takes higher precedence.

The Generator interface

While Component (respectively the related custom resource type) models the desired and actual state of the managed component, the Generator interface is about implementing a recipe to render the Kubernetes manifests of the dependent objects, according to the provided parameterization (spec) of the component:

package manifests

// Resource generator interface.
// When called from the reconciler, the arguments namespace and name will match the
// component's namespace and name or, if the component or its spec implement the
// PlacementConfiguration interface, the return values of the GetDeploymentNamespace(), GetDeploymentName()
// methods (if non-empty). The parameters argument will be assigned the return value
// of the component's GetSpec() method.
type Generator interface {
  Generate(ctx context.Context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
}

In addition to namespace, name, parameters, generators can retrieve additional contextual information, such as a client for the deployment target by calling utils.ClientFromContext(), and related functions.

Component controllers can of course implement their own generator. In many cases (for example if there exists a Helm chart or kustomization for the component), one of the generators bundled with this repository can be used.

Generators may optionally implement

package types

// SchemeBuilder interface.
type SchemeBuilder interface {
  AddToScheme(scheme *runtime.Scheme) error
}

in order to enhance the scheme used by the dependent objects deployer.

2.2 - Component Reconciler

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Whenever a component resource (that is, an instance of the component’s custom resource type) is created, udpated, or deleted, the set of dependent object potentially changes, and the cluster state has to be synchronized with that new declared state. This synchronization is the job of the reconciler provided by this framework.

Creating the reconciler instance

Typically, a component operator runs one reconciler which is instantiated by calling the following constructor:

package component

func NewReconciler[T Component](
  name              string,
  resourceGenerator manifests.Generator
  options           ReconcilerOptions
) *Reconciler[T]

The passed type parameter T Component is the concrete runtime type of the component’s custom resource type. Furthermore,

• name is supposed to be a unique name (typically a DNS name) identifying this component operator in the cluster; ìt will be used in annotations, labels, for leader election, …

• resourceGenerator is an implementation of the Generator interface, describing how the dependent objects are rendered from the component’s spec.

• options can be used to tune the behavior of the reconciler:

  package component
  
  // ReconcilerOptions are creation options for a Reconciler.
  type ReconcilerOptions struct {
    // Whether namespaces are auto-created if missing.
    // If unspecified, true is assumed.
    CreateMissingNamespaces *bool
    // How to react if a dependent object exists but has no or a different owner.
    // If unspecified, AdoptionPolicyIfUnowned is assumed.
    // Can be overridden by annotation on object level.
    AdoptionPolicy *reconciler.AdoptionPolicy
    // How to perform updates to dependent objects.
    // If unspecified, UpdatePolicyReplace is assumed.
    // Can be overridden by annotation on object level.
    UpdatePolicy *reconciler.UpdatePolicy
    // How to perform deletion of dependent objects.
    // If unspecified, DeletePolicyDelete is assumed.
    // Can be overridden by annotation on object level.
    DeletePolicy *reconciler.DeletePolicy
    // SchemeBuilder allows to define additional schemes to be made available in the
    // target client.
    SchemeBuilder types.SchemeBuilder
  }
  

The object returned by NewReconciler implements controller-runtime’s Reconciler interface, and can therefore be used as a drop-in in kubebuilder managed projects. After creation, the reconciler can be registered with the responsible controller-runtime manager instance by calling

package component

func (r *Reconciler[T]) SetupWithManager(mgr ctrl.Manager) error

The used manager mgr has to fulfill a few requirements:

• its client must bypass informer caches for the following types:
  • the type T itself
  • the type CustomResourceDefinition from the apiextensions.k8s.io/v1 group
  • the type APIService from the apiregistration.k8s.io/v1 group
• its scheme must recognize at least the following types:
  • the types in the API group defined in this repository
  • the core group (v1)
  • group apiextensions.k8s.io/v1
  • group apiregistration.k8s.io/v1.

Reconciler hooks

Component operators may register hooks to enhance the reconciler logic at certain points, by passing functions of type

package component

// HookFunc is the function signature that can be used to
// establish callbacks at certain points in the reconciliation logic.
// Hooks will be passed the current (potentially unsaved) state of the component.
// Post-hooks will only be called if the according operation (read, reconcile, delete)
// has been successful.
type HookFunc[T Component] func(ctx context.Context, client client.Client, component T) error

to the desired registration functions:

package component

// Register post-read hook with reconciler.
// This hook will be called after the reconciled component object has been retrieved from the Kubernetes API.
func (r *Reconciler[T]) WithPostReadHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right before the reconcilation of the dependent objects starts.
func (r *Reconciler[T]) WithPreReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register post-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right after the reconcilation of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right before the deletion of the dependent objects starts.
func (r *Reconciler[T]) WithPreDeleteHook(hook HookFunc[T]) *Reconciler[T]

// Register post-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right after the deletion of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostDeleteHook(hook HookFunc[T]) *Reconciler[T]

Note that the client passed to the hook functions is the client of the manager that was used when calling SetupWithManager() (that is, the return value of that manager’s GetClient() method). In addition, reconcile and delete hooks (that is, all except the post-read hook) can retrieve a client for the deployment target by calling ClientFromContext().

Tuning the retry behavior

By default, errors returned by the component’s generator or by a registered hook will make the reconciler go into a backoff managed by controller-runtime (which usually is an exponential backoff, capped at 10 minutes). However, if the error is or unwraps to a types.RetriableError, then the retry delay specified at the error will be used instead of the backoff. Implementations should use

pacakge types

func NewRetriableError(err error, retryAfter *time.Duration) RetriableError {
	return RetriableError{err: err, retryAfter: retryAfter}
}

to wrap an error into a RetriableError. It is allowed to pass retryAfter as nil; in that case the retry delay will be determined by calling the component’s GetRetryInterval() method (if the component or its spec implements the

package component

// The RetryConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the retry interval (by default, it would be the value of the requeue interval).
type RetryConfiguration interface {
	// Get retry interval. Should be greater than 1 minute.
	GetRetryInterval() time.Duration
}

interface), or otherwise will be set to the effective requeue interval (see below).

Tuning the requeue behavior

If a component was successfully reconciled, another reconciliation will be scheduled after 10 minutes, by default. This default requeue interval may be overridden by the component by implementing the

package component

// The RequeueConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the requeue interval (by default, it would be 10 minutes).
type RequeueConfiguration interface {
	// Get requeue interval. Should be greater than 1 minute.
	GetRequeueInterval() time.Duration
}

interface.

Tuning the timeout behavior

If the dependent objects of a component do not reach a ready state after a certain time, the component state will switch from Processing to Error. This timeout restarts counting whenever something changed in the component, or in the manifests of the dependent objects, and by default has the value of the effective requeue interval, which in turn defaults to 10 minutes. The timeout may be overridden by the component by implementing the

package component

// The TimeoutConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the processing timeout (by default, it would be the value of the requeue interval).
type TimeoutConfiguration interface {
	// Get timeout. Should be greater than 1 minute.
	GetTimeout() time.Duration
}

interface.

2.3 - Dependent Objects

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Under normal circumstances, the framework manages the lifecycle of the dependent objects in a sufficient way. For example, it will create and delete objects in a meaningful order, trying to avoid corresponding transient or permanent errors.

A more remarkable feature of component-operator-runtime is that it will block deletion of dependent objects as long as non-managed instances of managed extension types (such as custom resource definitions) exist. To be more precise, assume for example, that the managed component contains some custom resource definition, plus the according operator. Then, if the component resource gets deleted, none of the component’s dependent objects will be touched as long as there exist foreign instances of the managed custom resource definition in the cluster.

In some special situations, it is desirable to have even more control on the lifecycle of the dependent objects. To support such cases, the Generator implementation can set the following annotations in the manifests of the dependents:

• mycomponent-operator.mydomain.io/adoption-policy: defines how the reconciler reacts if the object exists but has no or a different owner; can be one of:

  • never: fail if the object exists and has no or a different owner
  • if-unowned (which is the default): adopt the object if it has no owner set
  • always: adopt the object, even if it has a conflicting owner

• mycomponent-operator.mydomain.io/reconcile-policy: defines how the object is reconciled; can be one of:

  • on-object-change (which is the default): the object will be reconciled whenever its generated manifest changes
  • on-object-or-component-change: the object will be reconciled whenever its generated manifest changes, or whenever the responsible component object changes by generation
  • once: the object will be reconciled once, but never be touched again

• mycomponent-operator.mydomain.io/update-policy: defines how the object (if existing) is updated; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • replace (which is the default): a regular update (i.e. PUT) call will be made to the Kubernetes API server
  • ssa-merge: use server side apply to update existing dependents
  • ssa-override: use server side apply to update existing dependents and, in addition, reclaim fields owned by certain field owners, such as kubectl or Helm
  • recreate: if the object would be updated, it will be deleted and recreated instead

• mycomponent-operator.mydomain.io/delete-policy: defines what happens to the object when the compoment is deleted; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • delete (which is the default): a delete call will be sent to the Kubernetes API server
  • orphan: the object will not be deleted, and it will be no longer tracked

  note that the deletion policy has no effect in the case when objects are deleted because they become obsolete by applying a new version of the component manifests

• mycomponent-operator.mydomain.io/apply-order: the wave in which this object will be reconciled; dependents will be reconciled wave by wave; that is, objects of the same wave will be deployed in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous waves are ready; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0

• mycomponent-operator.mydomain.io/purge-order (optional): the wave by which this object will be purged; here, purged means that, while applying the dependents, the object will be deleted from the cluster at the end of the specified wave; the according record in status.Inventory will be set to phase Completed; setting purge orders is useful to spawn ad-hoc objects during the reconcilation, which are not permanently needed; so it’s comparable to Helm hooks, in a certain sense

• mycomponent-operator.mydomain.io/delete-order (optional): the wave by which this object will be deleted; that is, if the dependent is no longer part of the component, or if the whole component is being deleted; dependents will be deleted wave by wave; that is, objects of the same wave will be deleted in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous saves are gone; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0; note that the delete order is completely independent of the apply order

• mycomponent-operator.mydomain.io/status-hint (optional): a comma-separated list of hints that may help the framework to properly identify the state of the annotated dependent object; currently, the following hints are possible:

  • has-observed-generation: tells the framework that the dependent object has a status.observedGeneration field, even if it is not (yet) set by the responsible controller (some controllers are known to set the observed generation lazily, with the consequence that there is a period right after creation of the dependent object, where the field is missing in the dependent’s status)
  • has-ready-condition: tells the framework to count with a ready condition; if it is absent, the condition status will be considered as Unknown
  • conditions: semicolon-separated list of additional conditions that must be present and have a True status in order to make the overall status ready

Note that, in the above paragraph, mycomponent-operator.mydomain.io has to be replaced with whatever was passed as name when calling NewReconciler().

3 - Generators

This section provides documentation about the generic resource generators shipped with component-operator-runtime

3.1 - Kustomize Generator

This generator allows to generate the manifests of the component’s resources from a given kustomization. As a special case, one or more simple Kubernetes manifests (without a kustomization.yaml) are supported as well. In addition, all (or selected; see below) files in the kustomization directory can be templatized in a helm’ish way. That means, they will be considered as a common template group (where all templates are associated with each other), and the same template function set that is available on Helm can be used; so, all the sprig functions, and custom functions such as include, tpl, lookup can be used. In addition:

• parameterless functions namespace and name are defined, which return the corresponding arguments passed to Generate()
• a function kubernetesVersion is available, which returns the version information of the target cluster, as a version.Info structure.

In the generation step, first, all the go templates will be rendered, and the result of this pre-step will be passed to kustomize.

A kustomize generator can be instantiated by calling the following constructor:

package kustomize

func NewKustomizeGenerator(
  fsys fs.FS,
  kustomizationPath string,
  templateSuffix string,
  client client.Client
) (*KustomizeGenerator, error) {

Here:

• fsys must be an implementation of fs.FS, such as embed.FS; or it can be passed as nil; then, all file operations will be executed on the current OS filesystem.
• kustomizationPath is the directory containing the (potentially templatized) kustomatization; if fsys was provided, this has to be a relative path; otherwise, it will be interpreted with respect to the OS filesystem (as an absolute path, or relative to the current working directory of the controller).
• templateSuffx is optional; if empty, all files under kustomizationPath will be subject to go templating; otherwise, only files matching the specified suffix will be considered as templates.
• client should be a client for the local cluster (i.e. the cluster where the component object exists).

As of now, the specified kustomization must not reference files or paths outside kustomizationPath. Remote references are generally not supported.

3.2 - Helm Generator

Sometimes it is desired to write a component operator (using component-operator-runtime) for some cluster component, which already has a productive Helm chart. Then it can make sense to use the HelmGenerator implementation of the Generator interface included in this module:

package helm

func NewHelmGenerator(
  fsys                  fs.FS,
  chartPath             string,
  client                client.Client,
) (*HelmGenerator, error)

Here:

• fsys must be an implementation of fs.FS, such as embed.FS; or it can be passed as nil; then, all file operations will be executed on the current OS filesystem.
• chartPath is the directory containing the used Helm chart; if fsys was provided, this has to be a relative path; otherwise, it will be interpreted with respect to the OS filesystem (as an absolute path, or relative to the current working directory of the controller).
• client should be a client for the local cluster (i.e. the cluster where the component object exists).

It should be noted that HelmGenerator does not use the Helm SDK; instead it tries to emulate the Helm behavior as good as possible. A few differences and restrictions arise from this:

• Not all Helm template functions are supported. To be exact, toToml, fromYamlArray, fromJsonArray are not supported; the functions toYaml, fromYaml, toJson, fromJson are supported, but will behave more strictly in error situtations.
• Not all builtin variables are supported; the following restrictions apply:
  • for the .Release builtin, only .Release.Namespace, .Release.Name, .Release.Service, .Release.IsInstall, .Release.IsUpgrade are supported; note that - since this framework does not really distinguish between installations and upgrades - Release.IsInstall is always set to true, and Release.IsUpgrade is always set to false
  • for the .Chart builtin, only .Chart.Name, .Chart.Version, .Chart.Type, .Chart.AppVersion, .Chart.Dependencies are supported
  • for the .Capabilities builtin, only .Capabilities.KubeVersion and .Capabilities.APIVersions are supported
  • the .Template builtin is fully supported
  • the .Files builtin is not supported at all.
• Regarding hooks, pre-delete and post-delete hooks are not allowed; test and rollback hooks are ignored, and pre-install, post-install, pre-upgrade, post-upgrade hooks might be handled in a sligthly different way; hook weights will be handled in a compatible way; hook deletion policy hook-failed is not allowed, but before-hook-creation and hook-succeeded should work as expected.

3.3 - Enhance Existing Generators

In some cases it is desirable to modify the behaviour of an existing generator by transforming the input parameters passed to the generation step, or by transforming the object manifests returned by the generation step. This can be achieved by wrapping an existing generator into a

package manifests

type TransformableGenerator interface {
  Generator
  WithParameterTransformer(transformer ParameterTransformer) TransformableGenerator
  WithObjectTransformer(transformer ObjectTransformer) TransformableGenerator
}

object tby calling

package manifests

func NewGenerator(generator Generator) TransformableGenerator

The generator obtained this way can now be extended by calling its methods WithParameterTransformer() and WithObjectTransformer(). The actual modification logic happens by implementing the respective interfaces

package manifests

type ParameterTransformer interface {
  TransformParameters(parameters types.Unstructurable) (types.Unstructurable, error)
}

type ObjectTransformer interface {
  TransformObjects(objects []client.Object) ([]client.Object, error)
}

4 - Usage

4.1 - Overview

This framework is based on the controller-runtime project. Therefore one way of consuming it would be to bootstrap a kubebuilder project, such as

kubebuilder init \
  --domain kyma-project.io \
  --repo github.com/myorg/mycomponent-operator \
  --project-name=mycomponent-operator

kubebuilder create api \
  --group operator \
  --version v1alpha1 \
  --kind MyComponent \
  --resource \
  --controller \
  --make

First you will enhance the custom resource type, as generated by kubebuilder in the api/v1alpha1 folder:

// MyComponentSpec defines the desired state of MyComponent
type MyComponentSpec struct {
  // Add fields here ...
}

// MyComponentStatus defines the observed state of MyComponent
type MyComponentStatus struct {
  component.Status `json:",inline"`
}

In many cases, it makes sense to embed one or more of the following structs into the spec:

• component.PlacementSpec if instances should be allowed to specify target namespace and name different from the component’s namespace and name
• component.ClientSpec if (remote) deployments via a specified kubeconfig shall be possible
• component.ImpersonationSpec if you want to support impersonation of the deployment (e.g. via a service account).

Most likely you will add own attributes to the spec, allowing to parameterize the deployment of your component. Including component.Status into the status is mandatory, but you are free to add further fields if needed.

In order to make the custom resource type implement the Component interface, add the following methods:

var _ component.Component = &MyComponent{}

func (s *MyComponentSpec) ToUnstructured() map[string]any {
  result, err := runtime.DefaultUnstructuredConverter.ToUnstructured(s)
  if err != nil {
    panic(err)
  }
  return result
}

func (c *MyComponent) GetSpec() runtimetypes.Unstructurable {
  return &c.Spec
}

func (c *MyComponent) GetStatus() *component.Status {
  return &c.Status.Status
}

Now we are settled to replace the controller generated by kubebuilder with the component-operator-runtime reconciler in the scaffolded main.go:

// Replace this by a real resource generator (e.g. HelmGenerator or KustomizeGenerator, or your own one).
resourceGenerator, err := manifests.NewDummyGenerator()
if err != nil {
  setupLog.Error(err, "error initializing resource generator")
  os.Exit(1)
}

if err := component.NewReconciler[*operatorv1alpha1.MyComponent](
  "mycomponent-operator.kyma-project.io",
  nil,
  nil,
  nil,
  nil,
  resourceGenerator,
).SetupWithManager(mgr); err != nil {
  setupLog.Error(err, "unable to create controller", "controller", "MyComponent")
  os.Exit(1)
}

In addition, you have to add the apiextensions.k8s.io/v1 and apiregistration.k8s.io/v1 groups to the used scheme, such that the kubebuilder generated init() function looks like this:

func init() {
  utilruntime.Must(clientgoscheme.AddToScheme(scheme))
  utilruntime.Must(apiextensionsv1.AddToScheme(scheme))
  utilruntime.Must(apiregistrationv1.AddToScheme(scheme))
  utilruntime.Must(operatorv1alpha1.AddToScheme(scheme))
}

Furthermore, pay attention to bypass informer caching in the client at least for the following types:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  // ...
  Client: client.Options{
    Cache: &client.CacheOptions{
      DisableFor: []client.Object{
        &operatorv1alpha1.MyComponent{},
        &apiextensionsv1.CustomResourceDefinition{},
        &apiregistrationv1.APIService{},
      },
    },
  },
  // ...
})

Now the actual work starts, which means that you will tailor the custom resource type’s spec according to the needs of the managed component, and implement a meaningful resource generator, replacing manifests.NewDummyGenerator().

4.2 - Scaffolder

The recommended way to get started with the implementation of a new component operator is to use the included scaffolding tool, which can be downloaded from the releases page.

After installing the scaffolder, a new project can be created like this:

scaffold \
  --group-name operator.kyma-project.io \
  --kind MyComponent \
  --operator-name mycomponent-operator.kyma-project.io \
  --go-module github.com/myorg/mycomponent-operator \
  --image mycomponent-operator:latest \
  <output-directory>

In this example, some options were left out, using the according default values; the full option list is as follows:

Usage: scaffold [options] [output directory]
  [output directory]: Target directory for the generated scaffold; must exist
  [options]:
      --version                                    Show version
      --owner string                               Owner of this project, as written to the license header (default "SAP SE")
      --spdx-license-headers                       Whether to write license headers in SPDX format
      --group-name string                          API group name
      --group-version string                       API group version (default "v1alpha1")
      --kind string                                API kind for the component
      --resource string                            API resource (plural) for the component; if empty, it will be the pluralized kind
      --operator-name string                       Unique name for this operator, used e.g. for leader election and labels; should be a valid DNS hostname
      --with-validating-webhook                    Whether to scaffold validating webhook
      --with-mutating-webhook                      Whether to scaffold mutating webhook
      --go-version string                          Go version to be used (default "1.21")
      --go-module string                           Name of the Go module, as written to the go.mod file
      --kubernetes-version string                  Kubernetes go-client version to be used (default "v0.28.1")
      --controller-runtime-version string          Controller-runtime version to be used (default "v0.16.0")
      --controller-tools-version string            Controller-tools version to be used (default "v0.13.0")
      --code-generator-version string              Code-generator version to be used (default "v0.28.1")
      --admission-webhook-runtime-version string   Admission-webhook-runtime version to be used (default "v0.1.0")
      --envtest-kubernetes-version string          Kubernetes version to be used by envtest (default "1.27.1")
      --image string                               Name of the Docker/OCI image produced by this project (default "controller:latest")
      --skip-post-processing                       Skip post-processing

After generating the scaffold, the next steps are:

• Enhance the spec type of the generated custom resource type (in api/<group-version>/types.go) according to the needs of your component
• Implement a meaningful resource generator and use it in main.go instead of manifests.NewDummyGenerator(); to do so you can either implement your own generator, or reuse one of the generic generators shipped with this repository.