component-operator-runtime

• 1: Getting Started
• 2: Concepts
• 2.1: Components and Generators
• 2.2: Component Reconciler
• 2.3: Dependent Objects
• 2.4: Kubernetes Clients
• 3: Generators
• 3.1: Kustomize Generator
• 3.2: Helm Generator
• 3.3: Enhance Existing Generators
• 4: Usage
• 4.1: Kubebuilder
• 4.2: Scaffolder

1 - Getting Started

In this short tutorial you will learn how to scaffold a Kubernetes component operator using component-operator-runtime, and how to start with the implementation of the operator.

First of all, you have to download the component-operator-runtime scaffolding tool from the releases page. In the following we assume that the downloaded scaffold executable is somehwere in your path (for example, as /usr/local/bin/scaffold-component-operator).

Then, a git repository for the operator code is needed; in this example, we call it github.com/myorg/mycomponent-operator. We assume that you have cloned the empty repository to your local desktop, and have changed the current directory to the checked out repository.

Then, to scaffold a component operator for a component type MyComponent in group group.my-domain.io, just run:

scaffold-component-operator \
  --group-name group.my-domain.io \
  --group-version v1alpha1 \
  --kind MyComponent \
  --operator-name mycomponent-operator.group.my-domain.io \
  --go-module github.com/myorg/mycomponent-operator \
  --image mycomponent-operator:latest \
  .

This will give you a syntactically correct Go module. In order to start the operator, you first have to apply the custom resource definition into your development (e.g. kind) cluster:

kubectl apply -f crds

Then, after copying or linking the cluster’s kubeconfig to ./tmp/kubeconfig (no worries, it is not submitted to git because ./tmp is excluded by .gitignore), you can use the generated ./vscode/launch.json to start the operator against your cluster with your Visual Studio Code. Now you are ready to instantiate your component:

kubectl apply -f - <<END
apiVersion: group.my-domain.io/v1alpha1
kind: MyComponent
metadata:
  namespace: default
  name: test
END

Now, after having the skeleton generated, we have to breathe life into the controller. The first step is to enhance the spec of the generated custom resource type MyComponentSpec in api/v1alpha1/types.go. In principle, all the attributes parameterizing the deployment of the managed commponent should be modeled there.

Whenever you change the runtime type, you should invoke make generate and make manifests in order to update the generated code artifacts and the custom resource definition; afterwards you should re-apply the custom resource definition to the cluster.

The next step is to implement a meaningful resource generator (the scaffolding just puts a dummy implementation called DummyGenerator into main.go). Writing such a resource generator means to implement the interface

type Generator interface {
  Generate(ctx context.Context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
}

When called by the framework, the namespace and name arguments of the Generate() method is assigned the component’s namespace and name or, if the component or its spec implements the PlacementConfiguration interface, they match the return values of the respective GetDeploymentNamespace(), GetDeploymentName() methods. The parameter argument is set to the return value of the component’s GetSpec() method. In simplified words, the spec of the component resource is fed into the resource generator, which returns the concrete manifests of the dependent objects, which then is applied to the cluster.

In some cases, the best option is to implement your own resource generator from scratch. When doing so, the returned resources []client.Object can of type *unstructured.Unstructured, or the according specific type must be known to the used scheme.

In many other cases however, it makes more sense to just reuse one of the generic generators shipped with this repository.

2 - Concepts

The framework provided in this repository aims to automate the lifecycle of an arbitrary component in a Kubernetes cluster. Usually (but not necessarily) the managed component contains one or multiple other operators, including extension types, such as custom resource definitions.

Components are described as a set of Kubernetes manifests. How these manifests are produced is up to the consumer of the framework. It is possible to build up the manifests from scratch in code, or to reuse or enhance the included helm generator or kustomize generator generators. The manifest list is then applied to (or removed from) the cluster by an own deployer logic, standing out with the following features:

• apply and delete waves
• configurable status handling
• apply through replace or server-side-apply patch
• smart deletion handling in case the component contains custom types which are still in use
• impersonination
• remote deployment mode via a given kubeconfig

The component-operator-runtime framework plugs into the controller-runtime SDK by implementing controller-runtime’s Reconciler interface.

2.1 - Components and Generators

In the terminology of this project, a Kubernetes cluster component (sometimes called module) consists of a set of dependent objects that are to be deployed consistently into the Kubernetes cluster. The continuous reconciliation of the declared state of these dependent objects is the task of an operator implemented by component-operator-runtime. To achieve this, basically two interfaces have to be implemented by such an operator…

The Component Interface

In the programming model proposed by component-operator-runtime, the declared and observed state of the component is represented by a dedicated custom resource type. The corresponding runtime type has to fulfill the following interface:

package component

// Component is the central interface that component operators have to implement.
// Besides being a conroller-runtime client.Object, the implementing type has to expose accessor
// methods for the components's spec and status, GetSpec() and GetStatus().
type Component interface {
  client.Object
  // Return a read-only accessor to the component's spec.
  // The returned value has to implement the types.Unstructurable interface.
  GetSpec() types.Unstructurable
  // Return a read-write (usually a pointer) accessor to the component's status,
  // resp. to the corresponding substruct if the status extends component.Status.
  GetStatus() *Status
}

Basically, two accessor methods have to be implemented here. First, GetSpec() exposes the parameterization of the component. The only requirement on the returned type is to implement the

package types

// Unstructurable represents objects which can be converted into a string-keyed map.
// All Kubernetes API types, as well as all JSON objects could be modelled as Unstructurable objects.
type Unstructurable interface {
  ToUnstructured() map[string]any
}

interface. In most cases, the returned Unstructurable object is the spec itself, or a deep copy of the spec. In general, the implementation is allowed to return arbitrary content, as long as the receiving generator is able to process it. In particular, it is not expected by the framework that changes applied to the returned Unstructurable reflect in any way in the component; indeed, the framework will never modify the returned Unstructurable.

Finally, GetStatus() allows the framework to access (a part of) the custom resource type’s status, having the following type:

package component

// Component Status. Components must include this into their status.
type Status struct {
  ObservedGeneration int64            `json:"observedGeneration"`
  AppliedGeneration  int64            `json:"appliedGeneration,omitempty"`
  LastObservedAt     *metav1.Time     `json:"lastObservedAt,omitempty"`
  LastAppliedAt      *metav1.Time     `json:"lastAppliedAt,omitempty"`
  Conditions         []Condition      `json:"conditions,omitempty"`
  State              State            `json:"state,omitempty"`
  Inventory          []*InventoryItem `json:"inventory,omitempty"`
}

Note that, other than with the GetSpec() accessor, the framework will make changes to the returned Status structure. Thus, in almost all cases, the returned pointer should just reference the status of the component’s API type (or an according substructure of that status).

The component’s custom resource type is supposed to be namespaced, and by default, dependent objects will be created in that same namespace. To be more precise, the namespace and name parameters of the used generator’s Generate() method will be set to the component’s metadata.namespace and metadata.name, respectively. Sometimes it might be desired to override these defaults, and to render the dependent objects with a different namespace or name. To allow this, the component (or its spec) can implement

package component

// The PlacementConfiguration interface is meant to be implemented by components (or their spec) which allow
// to explicitly specify target namespace and name of the deployment (otherwise this will be defaulted as
// the namespace and name of the component object itself).
type PlacementConfiguration interface {
  // Return target namespace for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generate() as namespace and, in addition, rendered namespaced resources with
  // unspecified namespace will be placed in this namespace.
  GetDeploymentNamespace() string
  // Return target name for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generator() as name.
  GetDeploymentName() string
}

In addition, the component (or its spec) may implement

package component

// The ClientConfiguration interface is meant to be implemented by components (or their spec) which offer
// remote deployments.
type ClientConfiguration interface {
  // Get kubeconfig content. Should return nil if default local client shall be used.
  GetKubeConfig() []byte
}

in order to support remote deployments (that is, to make the deployment of the dependent objects use the specified kubeconfig), and

package component

// The ImpersonationConfiguration interface is meant to be implemented by components (or their spec) which offer
// impersonated deployments.
type ImpersonationConfiguration interface {
  // Return impersonation user. Should return system:serviceaccount:<namespace>:<serviceaccount>
  // if a service account is used for impersonation. Should return an empty string
  // if user shall not be impersonated.
  GetImpersonationUser() string
  // Return impersonation groups. Should return nil if groups shall not be impersonated.
  GetImpersonationGroups() []string
}

to use different user/groups for the deployment of dependent objects.

Note that, as mentioned above, the interfaces PlacementConfiguration, ClientConfiguration and ImpersonationConfiguration can be implemented by the component itself as well as by its spec type. In the theoretical case that both is the case, the implementation on the component level takes higher precedence.

The Generator interface

While Component (respectively the related custom resource type) models the desired and actual state of the managed component, the Generator interface is about implementing a recipe to render the Kubernetes manifests of the dependent objects, according to the provided parameterization (spec) of the component:

package manifests

// Resource generator interface.
// When called from the reconciler, the arguments namespace and name will match the
// component's namespace and name or, if the component or its spec implement the
// PlacementConfiguration interface, the return values of the GetDeploymentNamespace(), GetDeploymentName()
// methods (if non-empty). The parameters argument will be assigned the return value
// of the component's GetSpec() method.
type Generator interface {
  Generate(ctx context.Context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
}

In addition to namespace, name, parameters, generators can retrieve additional contextual information, such as a client for the deployment target by calling utils.ClientFromContext(), and related functions.

Component controllers can of course implement their own generator. In many cases (for example if there exists a Helm chart or kustomization for the component), one of the generators bundled with this repository can be used.

Generators may optionally implement

package types

// SchemeBuilder interface.
type SchemeBuilder interface {
  AddToScheme(scheme *runtime.Scheme) error
}

in order to enhance the scheme used by the dependent objects deployer.

2.2 - Component Reconciler

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Whenever a component resource (that is, an instance of the component’s custom resource type) is created, udpated, or deleted, the set of dependent object potentially changes, and the cluster state has to be synchronized with that new declared state. This synchronization is the job of the reconciler provided by this framework.

Creating the reconciler instance

Typically, a component operator runs one reconciler which is instantiated by calling the following constructor:

package component

func NewReconciler[T Component](
  name              string,
  resourceGenerator manifests.Generator
  options           ReconcilerOptions
) *Reconciler[T]

The passed type parameter T Component is the concrete runtime type of the component’s custom resource type (respectively, a pointer to that). Furthermore,

• name is supposed to be a unique name (typically a DNS name) identifying this component operator in the cluster; ìt will be used in annotations, labels, for leader election, …

• resourceGenerator is an implementation of the Generator interface, describing how the dependent objects are rendered from the component’s spec.

• options can be used to tune the behavior of the reconciler:

  package component
  
  // ReconcilerOptions are creation options for a Reconciler.
  type ReconcilerOptions struct {
    // Which field manager to use in API calls.
    // If unspecified, the reconciler name is used.
    FieldOwner *string
    // Which finalizer to use.
    // If unspecified, the reconciler name is used.
    Finalizer *string
    // Default service account used for impersonation of clients.
    // Of course, components can still customize impersonation by implementing the ImpersonationConfiguration interface.
    DefaultServiceAccount *string
    // How to react if a dependent object exists but has no or a different owner.
    // If unspecified, AdoptionPolicyIfUnowned is assumed.
    // Can be overridden by annotation on object level.
    AdoptionPolicy *reconciler.AdoptionPolicy
    // How to perform updates to dependent objects.
    // If unspecified, UpdatePolicyReplace is assumed.
    // Can be overridden by annotation on object level.
    UpdatePolicy *reconciler.UpdatePolicy
    // How to perform deletion of dependent objects.
    // If unspecified, DeletePolicyDelete is assumed.
    // Can be overridden by annotation on object level.
    DeletePolicy *reconciler.DeletePolicy
    // Whether namespaces are auto-created if missing.
    // If unspecified, MissingNamespacesPolicyCreate is assumed.
    MissingNamespacesPolicy *reconciler.MissingNamespacesPolicy
    // SchemeBuilder allows to define additional schemes to be made available in the
    // target client.
    SchemeBuilder types.SchemeBuilder
  }
  

The object returned by NewReconciler implements controller-runtime’s Reconciler interface, and can therefore be used as a drop-in in kubebuilder managed projects. After creation, the reconciler can be registered with the responsible controller-runtime manager instance by calling

package component

func (r *Reconciler[T]) SetupWithManager(mgr ctrl.Manager) error

The used manager mgr has to fulfill a few requirements:

• its client must bypass informer caches for the following types:
  • the type T itself
  • the type CustomResourceDefinition from the apiextensions.k8s.io/v1 group
  • the type APIService from the apiregistration.k8s.io/v1 group
• its scheme must recognize at least the following types:
  • the types in the API group defined in this repository
  • the core group (v1)
  • group apiextensions.k8s.io/v1
  • group apiregistration.k8s.io/v1.

Reconciler hooks

Component operators may register hooks to enhance the reconciler logic at certain points, by passing functions of type

package component

// HookFunc is the function signature that can be used to
// establish callbacks at certain points in the reconciliation logic.
// Hooks will be passed the current (potentially unsaved) state of the component.
// Post-hooks will only be called if the according operation (read, reconcile, delete)
// has been successful.
type HookFunc[T Component] func(ctx context.Context, client client.Client, component T) error

to the desired registration functions:

package component

// Register post-read hook with reconciler.
// This hook will be called after the reconciled component object has been retrieved from the Kubernetes API.
func (r *Reconciler[T]) WithPostReadHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right before the reconcilation of the dependent objects starts.
func (r *Reconciler[T]) WithPreReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register post-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right after the reconcilation of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right before the deletion of the dependent objects starts.
func (r *Reconciler[T]) WithPreDeleteHook(hook HookFunc[T]) *Reconciler[T]

// Register post-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right after the deletion of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostDeleteHook(hook HookFunc[T]) *Reconciler[T]

Note that the client passed to the hook functions is the client of the manager that was used when calling SetupWithManager() (that is, the return value of that manager’s GetClient() method). In addition, reconcile and delete hooks (that is, all except the post-read hook) can retrieve a client for the deployment target by calling ClientFromContext().

Tuning the retry behavior

By default, errors returned by the component’s generator or by a registered hook will make the reconciler go into a backoff managed by controller-runtime (which usually is an exponential backoff, capped at 10 minutes). However, if the error is or unwraps to a types.RetriableError, then the retry delay specified at the error will be used instead of the backoff. Implementations should use

pacakge types

func NewRetriableError(err error, retryAfter *time.Duration) RetriableError {
  return RetriableError{err: err, retryAfter: retryAfter}
}

to wrap an error into a RetriableError. It is allowed to pass retryAfter as nil; in that case the retry delay will be determined by calling the component’s GetRetryInterval() method (if the component or its spec implements the

package component

// The RetryConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the retry interval (by default, it would be the value of the requeue interval).
type RetryConfiguration interface {
  // Get retry interval. Should be greater than 1 minute.
  GetRetryInterval() time.Duration
}

interface), or otherwise will be set to the effective requeue interval (see below).

Tuning the requeue behavior

If a component was successfully reconciled, another reconciliation will be scheduled after 10 minutes, by default. This default requeue interval may be overridden by the component by implementing the

package component

// The RequeueConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the requeue interval (by default, it would be 10 minutes).
type RequeueConfiguration interface {
  // Get requeue interval. Should be greater than 1 minute.
  GetRequeueInterval() time.Duration
}

interface.

Tuning the timeout behavior

If the dependent objects of a component do not reach a ready state after a certain period, the component enters a timeout state. That means:

• if the component was in Processing state, then the state switches to Error, and the reason of the Ready condition is set to Timeout
• if the reconciler encounters a retriable error, then the state is Pending, and the reason of the Ready condition is set to Timeout
• if the reconciler encounters a non-retriable error, then the state is Error, and the reason of the Ready condition is set to Timeout.

This timeout restarts counting down whenever something changed in the component or its references, and by default has the value of the effective requeue interval, which in turn defaults to 10 minutes. The timeout may be overridden by the component by implementing the

package component

// The TimeoutConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the processing timeout (by default, it would be the value of the requeue interval).
type TimeoutConfiguration interface {
  // Get timeout. Should be greater than 1 minute.
  GetTimeout() time.Duration
}

interface.

Tuning the handling of dependent objects

The reconciler allows to tweak how dependent objects are applied to or deleted from the cluster. To change the shipped framework defaults, a component type can implement the

package component

// The PolicyConfiguration interface is meant to be implemented by compoments (or their spec) which offer
// tweaking policies affecting the dependents handling.
type PolicyConfiguration interface {
  // Get adoption policy.
  // Must return a valid AdoptionPolicy, or the empty string (then the reconciler/framework default applies).
  GetAdoptionPolicy() reconciler.AdoptionPolicy
  // Get update policy.
  // Must return a valid UpdatePolicy, or the empty string (then the reconciler/framework default applies).
  GetUpdatePolicy() reconciler.UpdatePolicy
  // Get delete policy.
  // Must return a valid DeletePolicy, or the empty string (then the reconciler/framework default applies).
  GetDeletePolicy() reconciler.DeletePolicy
  // Get namspace auto-creation policy.
  // Must return a valid MissingNamespacesPolicy, or the empty string (then the reconciler/framework default applies).
  GetMissingNamespacesPolicy() reconciler.MissingNamespacesPolicy
}

interface. Note that most of the above policies can be overridden on a per-object level by setting certain annotations, as described here.

Declaring additional (implicit) managed types

One of component-operator-runtime’s core features is the special handling of instances of managed types. Managed types are API extension types (such as custom resource definitions or types added by federation of an aggregated API server). Instances of these types which are part of the component (so-called managed instances) are treated differently. For example, the framework tries to process these instances as late as possible when applying the component, and as early as possible when the component is deleted. Other instances of these types which are not part of the component (so-called foreign instances) block the deletion of the whole component.

Sometimes, components are implicitly adding extension types. That means that the type definition is not part of the component manifest, but the types are just created at runtime by controllers or operators contained in the component. A typical example are crossplane providers. These types are of course not recognized by the framework as managed types. However it is probably desired that (both managed and foreign) instances of these types experience the same special handling like instances of real managed types.

To make this possible, components can implement the

// The TypeConfiguration interface is meant to be implemented by compoments (or their spec) which allow
// to specify additional managed types.
type TypeConfiguration interface {
  // Get additional managed types; instances of these types are handled differently during
  // apply and delete; foreign instances of these types will block deletion of the component.
  // The fields of the returned TypeInfo structs can be concrete api groups, kinds,
  // or wildcards ("*"); in addition, groups can be specified as a pattern of the form "*.<suffix>"",
  // where the wildcard matches one or multiple dns labels.
  GetAdditionalManagedTypes() []reconciler.TypeInfo
}

interface. The types returned by GetAdditionalManagedTypes() contain a group and a kind, such as

// TypeInfo represents a Kubernetes type.
type TypeInfo struct {
	// API group.
	Group string `json:"group"`
	// API kind.
	Kind string `json:"kind"`
}

To match multiple types, the following pattern syntax is supported:

• the Group can be just * or have the form *.domain.suffix; note that the second case, the asterisk matches one or multiple DNS labels
• the Kind can be just *, which matches any kind.

2.3 - Dependent Objects

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Under normal circumstances, the framework manages the lifecycle of the dependent objects in a sufficient way. For example, it will create and delete objects in a meaningful order, trying to avoid corresponding transient or permanent errors.

A more remarkable feature of component-operator-runtime is that it will block deletion of dependent objects as long as non-managed instances of managed extension types (such as custom resource definitions) exist. To be more precise, assume for example, that the managed component contains some custom resource definition, plus the according operator. Then, if the component resource gets deleted, none of the component’s dependent objects will be touched as long as there exist foreign instances of the managed custom resource definition in the cluster.

In some special situations, it is desirable to have even more control on the lifecycle of the dependent objects. To support such cases, the Generator implementation can set the following annotations in the manifests of the dependents:

• mycomponent-operator.mydomain.io/adoption-policy: defines how the reconciler reacts if the object exists but has no or a different owner; can be one of:

  • never: fail if the object exists and has no or a different owner
  • if-unowned (which is the default): adopt the object if it has no owner set
  • always: adopt the object, even if it has a conflicting owner

• mycomponent-operator.mydomain.io/reconcile-policy: defines how the object is reconciled; can be one of:

  • on-object-change (which is the default): the object will be reconciled whenever its generated manifest changes
  • on-object-or-component-change: the object will be reconciled whenever its generated manifest changes, or whenever the responsible component object changes by generation
  • once: the object will be reconciled once, but never be touched again

• mycomponent-operator.mydomain.io/update-policy: defines how the object (if existing) is updated; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • replace (which is the default): a regular update (i.e. PUT) call will be made to the Kubernetes API server
  • ssa-merge: use server side apply to update existing dependents
  • ssa-override: use server side apply to update existing dependents and, in addition, reclaim fields owned by certain field owners, such as kubectl or Helm
  • recreate: if the object would be updated, it will be deleted and recreated instead

• mycomponent-operator.mydomain.io/delete-policy: defines what happens to the object when the compoment is deleted; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • delete (which is the default): a delete call will be sent to the Kubernetes API server
  • orphan: the object will not be deleted, and it will be no longer tracked; always, that is both if the object becomes redundant while applying the component, and if the component itself is deleted
  • orphan-on-apply: the object will not be deleted, and it will be no longer tracked; but only if the object becomes redundant while applying the component
  • orphan-on-delete: the object will not be deleted, and it will be no longer tracked; but only if the component itself is deleted

  note that the deletion policy has no effect in the case when objects are deleted because they become obsolete by applying a new version of the component manifests

• mycomponent-operator.mydomain.io/apply-order: the wave in which this object will be reconciled; dependents will be reconciled wave by wave; that is, objects of the same wave will be deployed in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous waves are ready; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0

• mycomponent-operator.mydomain.io/purge-order (optional): the wave by which this object will be purged; here, purged means that, while applying the dependents, the object will be deleted from the cluster at the end of the specified wave; the according record in status.Inventory will be set to phase Completed; setting purge orders is useful to spawn ad-hoc objects during the reconcilation, which are not permanently needed; so it’s comparable to Helm hooks, in a certain sense

• mycomponent-operator.mydomain.io/delete-order (optional): the wave by which this object will be deleted; that is, if the dependent is no longer part of the component, or if the whole component is being deleted; dependents will be deleted wave by wave; that is, objects of the same wave will be deleted in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous saves are gone; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0; note that the delete order is completely independent of the apply order

• mycomponent-operator.mydomain.io/status-hint (optional): a comma-separated list of hints that may help the framework to properly identify the state of the annotated dependent object; currently, the following hints are possible:

  • has-observed-generation: tells the framework that the dependent object has a status.observedGeneration field, even if it is not (yet) set by the responsible controller (some controllers are known to set the observed generation lazily, with the consequence that there is a period right after creation of the dependent object, where the field is missing in the dependent’s status)
  • has-ready-condition: tells the framework to count with a ready condition; if it is absent, the condition status will be considered as Unknown
  • conditions: semicolon-separated list of additional conditions that must be present and have a True status in order to make the overall status ready

Note that, in the above paragraph, mycomponent-operator.mydomain.io has to be replaced with whatever was passed as name when calling NewReconciler().

2.4 - Kubernetes Clients

When a component resource is reconciled, two Kubernetes API clients are constructed:

• The local client; it always points to the cluster where the component resides. If the component implements impersonation (that is, the component type or its spec implements the ImpersonationConfiguration interface), and an impersonation user or groups are specified by the component resource, then the specified user and groups are used to impersonate the controller’s kubeconfig. Otherwise, if a DefaultServiceAccount is defined in the reconciler’s options, then that service account (relative to the components metadata.namespace ) is used to impersonate the controller’s kubeconfig. Otherwise, the controller’s kubeconfig itself is used to build the local client. The local client is passed to generators via their context. For example, the HelmGenerator and KustomizeGenerator provided by component-operator-runtime use the local client to realize the localLookup and mustLocalLookup template functions.
• The target client; if the component specifies a kubeconfig (by implementing the ClientConfiguration interface), then that kubeconfig is used to build the target client. Otherwise, a local client is used (possibly impersonated), created according the the logic described above. The target client is used to manage dependent objects, and is passed to generators via their context. For example, the HelmGenerator and KustomizeGenerator provided by component-operator-runtime use the target client to realize the lookup and mustLookup template functions.

3 - Generators

This section provides documentation about the generic resource generators shipped with component-operator-runtime

3.1 - Kustomize Generator

This generator allows to generate the manifests of the component’s resources from a given kustomization. As a special case, one or more simple Kubernetes manifests (without a kustomization.yaml) are supported as well. In addition, all (or selected; see below) files in the kustomization directory can be templatized in a helm’ish way. That means, they will be considered as a common template group (where all templates are associated with each other), and the same template function set that is available on Helm can be used; so, all the sprig functions, and custom functions such as include, tpl, lookup can be used. In addition:

• parameterless functions namespace and name are defined, which return the corresponding arguments passed to Generate()
• a function kubernetesVersion is available, which returns the version information of the target cluster, as a version.Info structure.

In the generation step, first, all the go templates will be rendered, and the result of this pre-step will be passed to kustomize.

A kustomize generator can be instantiated by calling the following constructor:

package kustomize

func NewKustomizeGenerator(
  fsys fs.FS,
  kustomizationPath string,
  clnt client.Client,
  options KustomizeGeneratorOptions
) (*KustomizeGenerator, error) {

Here:

• fsys must be an implementation of fs.FS, such as embed.FS; or it can be passed as nil; then, all file operations will be executed on the current OS filesystem.
• kustomizationPath is the directory containing the (potentially templatized) kustomatization; if fsys was provided, this has to be a relative path; otherwise, it will be interpreted with respect to the OS filesystem (as an absolute path, or relative to the current working directory of the controller).
• clnt should be a client for the local cluster (i.e. the cluster where the component object exists).
• options allows to tweak the generator:

  package kustomize
  
  type KustomizeGeneratorOptions struct {
    // If defined, only files with that suffix will be subject to templating.
    TemplateSuffix *string
    // If defined, the given left delimiter will be used to parse go templates;
    // otherwise, defaults to '{{'
    LeftTemplateDelimiter *string
    // If defined, the given right delimiter will be used to parse go templates;
    // otherwise, defaults to '}}'
    RightTemplateDelimiter *string
  }
  

As of now, the specified kustomization must not reference files or paths outside kustomizationPath. Remote references are generally not supported.

3.2 - Helm Generator

Sometimes it is desired to write a component operator (using component-operator-runtime) for some cluster component, which already has a productive Helm chart. Then it can make sense to use the HelmGenerator implementation of the Generator interface included in this module:

package helm

func NewHelmGenerator(
  fsys                  fs.FS,
  chartPath             string,
  clnt                client.Client,
) (*HelmGenerator, error)

Here:

• fsys must be an implementation of fs.FS, such as embed.FS; or it can be passed as nil; then, all file operations will be executed on the current OS filesystem.
• chartPath is the directory containing the used Helm chart; if fsys was provided, this has to be a relative path; otherwise, it will be interpreted with respect to the OS filesystem (as an absolute path, or relative to the current working directory of the controller).
• clnt should be a client for the local cluster (i.e. the cluster where the component object exists).

It should be noted that HelmGenerator does not use the Helm SDK; instead it tries to emulate the Helm behavior as good as possible. A few differences and restrictions arise from this:

• Not all Helm template functions are supported. To be exact, toToml is not supported; all other functions should be supported, but may behave more strictly in error situtations.
• Not all builtin variables are supported; the following restrictions apply:
  • for the .Release builtin, only .Release.Namespace, .Release.Name, .Release.Service, .Release.IsInstall, .Release.IsUpgrade are supported; note that - since this framework does not really distinguish between installations and upgrades - Release.IsInstall is always set to true, and Release.IsUpgrade is always set to false
  • for the .Chart builtin, only .Chart.Name, .Chart.Version, .Chart.Type, .Chart.AppVersion, .Chart.Dependencies are supported
  • for the .Capabilities builtin, only .Capabilities.KubeVersion and .Capabilities.APIVersions are supported
  • the .Template builtin is fully supported
  • the .Files builtin is not supported at all.
• Regarding hooks, pre-delete and post-delete hooks are not allowed; test and rollback hooks are ignored, and pre-install, post-install, pre-upgrade, post-upgrade hooks might be handled in a sligthly different way; hook weights will be handled in a compatible way; hook deletion policy hook-failed is not allowed, but before-hook-creation and hook-succeeded should work as expected.

3.3 - Enhance Existing Generators

In some cases it is desirable to modify the behaviour of an existing generator by transforming the input parameters passed to the generation step, or by transforming the object manifests returned by the generation step. This can be achieved by wrapping an existing generator into a

package manifests

type TransformableGenerator interface {
  Generator
  WithParameterTransformer(transformer ParameterTransformer) TransformableGenerator
  WithObjectTransformer(transformer ObjectTransformer) TransformableGenerator
}

object tby calling

package manifests

func NewGenerator(generator Generator) TransformableGenerator

The generator obtained this way can now be extended by calling its methods WithParameterTransformer() and WithObjectTransformer(). The actual modification logic happens by implementing the respective interfaces

package manifests

type ParameterTransformer interface {
  TransformParameters(parameters types.Unstructurable) (types.Unstructurable, error)
}

type ObjectTransformer interface {
  TransformObjects(objects []client.Object) ([]client.Object, error)
}

4 - Usage

4.1 - Kubebuilder

This framework is based on the controller-runtime project. Therefore one way of consuming it would be to bootstrap a kubebuilder project, such as

kubebuilder init \
  --domain my-domain.io \
  --repo github.com/myorg/mycomponent-operator \
  --project-name=mycomponent-operator

kubebuilder create api \
  --group group \
  --version v1alpha1 \
  --kind MyComponent \
  --resource \
  --controller \
  --make

First you will enhance the custom resource type, as generated by kubebuilder in the api/v1alpha1 folder:

// MyComponentSpec defines the desired state of MyComponent
type MyComponentSpec struct {
  // Add fields here ...
}

// MyComponentStatus defines the observed state of MyComponent
type MyComponentStatus struct {
  component.Status `json:",inline"`
}

In many cases, it makes sense to embed one or more of the following structs into the spec:

• component.PlacementSpec if instances should be allowed to specify target namespace and name different from the component’s namespace and name
• component.ClientSpec if (remote) deployments via a specified kubeconfig shall be possible
• component.ImpersonationSpec if you want to support impersonation of the deployment (e.g. via a service account).

Most likely you will add own attributes to the spec, allowing to parameterize the deployment of your component. Including component.Status into the status is mandatory, but you are free to add further fields if needed.

In order to make the custom resource type implement the Component interface, add the following methods:

var _ component.Component = &MyComponent{}

func (s *MyComponentSpec) ToUnstructured() map[string]any {
  result, err := runtime.DefaultUnstructuredConverter.ToUnstructured(s)
  if err != nil {
    panic(err)
  }
  return result
}

func (c *MyComponent) GetSpec() runtimetypes.Unstructurable {
  return &c.Spec
}

func (c *MyComponent) GetStatus() *component.Status {
  return &c.Status.Status
}

Now we are settled to replace the controller generated by kubebuilder with the component-operator-runtime reconciler in the scaffolded cmd/main.go:

// Replace this by a real resource generator (e.g. HelmGenerator or KustomizeGenerator, or your own one).
resourceGenerator, err := manifests.NewDummyGenerator()
if err != nil {
  setupLog.Error(err, "error initializing resource generator")
  os.Exit(1)
}

if err := component.NewReconciler[*groupv1alpha1.MyComponent](
  "mycomponent-operator.group.my-domain.io",
  resourceGenerator,
  component.ReconcilerOptions{},
).SetupWithManager(mgr); err != nil {
  setupLog.Error(err, "unable to create controller", "controller", "MyComponent")
  os.Exit(1)
}

In addition, you have to add the apiextensions.k8s.io/v1 and apiregistration.k8s.io/v1 groups to the used scheme, such that the kubebuilder generated init() function looks like this:

func init() {
  utilruntime.Must(clientgoscheme.AddToScheme(scheme))
  utilruntime.Must(apiextensionsv1.AddToScheme(scheme))
  utilruntime.Must(apiregistrationv1.AddToScheme(scheme))
  utilruntime.Must(groupv1alpha1.AddToScheme(scheme))
}

Furthermore, pay attention to bypass informer caching in the client at least for the following types:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  // ...
  Client: client.Options{
    Cache: &client.CacheOptions{
      DisableFor: []client.Object{
        &groupv1alpha1.MyComponent{},
        &apiextensionsv1.CustomResourceDefinition{},
        &apiregistrationv1.APIService{},
      },
    },
  },
  // ...
})

Now the actual work starts, which means that you will tailor the custom resource type’s spec according to the needs of the managed component, and implement a meaningful resource generator, replacing manifests.NewDummyGenerator().

4.2 - Scaffolder

The recommended way to get started with the implementation of a new component operator is to use the included scaffolding tool, which can be downloaded from the releases page.

After installing the scaffolder, a new project can be created like this:

scaffold \
  --group-name operator.group.my-domain.io \
  --kind MyComponent \
  --operator-name mycomponent-operator.group.my-domain.io \
  --go-module github.com/myorg/mycomponent-operator \
  --image mycomponent-operator:latest \
  <output-directory>

In this example, some options were left out, using the according default values; the full option list is as follows:

Usage: scaffold [options] [output directory]
  [output directory]: Target directory for the generated scaffold; must exist
  [options]:
      --version                                    Show version
      --owner string                               Owner of this project, as written to the license header (default "SAP SE")
      --spdx-license-headers                       Whether to write license headers in SPDX format
      --group-name string                          API group name
      --group-version string                       API group version (default "v1alpha1")
      --kind string                                API kind for the component
      --resource string                            API resource (plural) for the component; if empty, it will be the pluralized kind
      --operator-name string                       Unique name for this operator, used e.g. for leader election and labels; should be a valid DNS hostname
      --with-validating-webhook                    Whether to scaffold validating webhook
      --with-mutating-webhook                      Whether to scaffold mutating webhook
      --go-version string                          Go version to be used (default "1.23.4")
      --go-module string                           Name of the Go module, as written to the go.mod file
      --kubernetes-version string                  Kubernetes go-client version to be used (default "v0.32.0")
      --controller-runtime-version string          Controller-runtime version to be used (default "v0.19.3")
      --controller-tools-version string            Controller-tools version to be used (default "v0.16.5")
      --code-generator-version string              Code-generator version to be used (default "v0.32.0")
      --admission-webhook-runtime-version string   Admission-webhook-runtime version to be used (default "v0.1.52")
      --envtest-kubernetes-version string          Kubernetes version to be used by envtest (default "1.30.3")
      --image string                               Name of the Docker/OCI image produced by this project (default "controller:latest")
      --skip-post-processing                       Skip post-processing

After generating the scaffold, the next steps are:

• Enhance the spec type of the generated custom resource type (in api/<group-version>/types.go) according to the needs of your component
• Implement a meaningful resource generator and use it in main.go instead of manifests.NewDummyGenerator(); to do so you can either implement your own generator, or reuse one of the generic generators shipped with this repository.