Concepts

• 1: Components and Generators
• 2: Component Reconciler
• 3: Dependent Objects
• 4: Kubernetes Clients

1 - Components and Generators

In the terminology of this project, a Kubernetes cluster component (sometimes called module) consists of a set of dependent objects that are to be deployed consistently into the Kubernetes cluster. The continuous reconciliation of the declared state of these dependent objects is the task of an operator implemented by component-operator-runtime. To achieve this, basically two interfaces have to be implemented by such an operator…

The Component Interface

In the programming model proposed by component-operator-runtime, the declared and observed state of the component is represented by a dedicated custom resource type. The corresponding runtime type has to fulfill the following interface:

package component

// Component is the central interface that component operators have to implement.
// Besides being a conroller-runtime client.Object, the implementing type has to expose accessor
// methods for the components's spec and status, GetSpec() and GetStatus().
type Component interface {
  client.Object
  // Return a read-only accessor to the component's spec.
  // The returned value has to implement the types.Unstructurable interface.
  GetSpec() types.Unstructurable
  // Return a read-write (usually a pointer) accessor to the component's status,
  // resp. to the corresponding substruct if the status extends component.Status.
  GetStatus() *Status
}

Basically, two accessor methods have to be implemented here. First, GetSpec() exposes the parameterization of the component. The only requirement on the returned type is to implement the

package types

// Unstructurable represents objects which can be converted into a string-keyed map.
// All Kubernetes API types, as well as all JSON objects could be modelled as Unstructurable objects.
type Unstructurable interface {
  ToUnstructured() map[string]any
}

interface. In most cases, the returned Unstructurable object is the spec itself, or a deep copy of the spec. In general, the implementation is allowed to return arbitrary content, as long as the receiving generator is able to process it. In particular, it is not expected by the framework that changes applied to the returned Unstructurable reflect in any way in the component; indeed, the framework will never modify the returned Unstructurable.

Finally, GetStatus() allows the framework to access (a part of) the custom resource type’s status, having the following type:

package component

// Component Status. Components must include this into their status.
type Status struct {
  ObservedGeneration int64            `json:"observedGeneration"`
  AppliedGeneration  int64            `json:"appliedGeneration,omitempty"`
  LastObservedAt     *metav1.Time     `json:"lastObservedAt,omitempty"`
  LastAppliedAt      *metav1.Time     `json:"lastAppliedAt,omitempty"`
  Conditions         []Condition      `json:"conditions,omitempty"`
  State              State            `json:"state,omitempty"`
  Inventory          []*InventoryItem `json:"inventory,omitempty"`
}

Note that, other than with the GetSpec() accessor, the framework will make changes to the returned Status structure. Thus, in almost all cases, the returned pointer should just reference the status of the component’s API type (or an according substructure of that status).

The component’s custom resource type is supposed to be namespaced, and by default, dependent objects will be created in that same namespace. To be more precise, the namespace and name parameters of the used generator’s Generate() method will be set to the component’s metadata.namespace and metadata.name, respectively. Sometimes it might be desired to override these defaults, and to render the dependent objects with a different namespace or name. To allow this, the component (or its spec) can implement

package component

// The PlacementConfiguration interface is meant to be implemented by components (or their spec) which allow
// to explicitly specify target namespace and name of the deployment (otherwise this will be defaulted as
// the namespace and name of the component object itself).
type PlacementConfiguration interface {
  // Return target namespace for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generate() as namespace and, in addition, rendered namespaced resources with
  // unspecified namespace will be placed in this namespace.
  GetDeploymentNamespace() string
  // Return target name for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generator() as name.
  GetDeploymentName() string
}

In addition, the component (or its spec) may implement

package component

// The ClientConfiguration interface is meant to be implemented by components (or their spec) which offer
// remote deployments.
type ClientConfiguration interface {
  // Get kubeconfig content. Should return nil if default local client shall be used.
  GetKubeConfig() []byte
}

in order to support remote deployments (that is, to make the deployment of the dependent objects use the specified kubeconfig), and

package component

// The ImpersonationConfiguration interface is meant to be implemented by components (or their spec) which offer
// impersonated deployments.
type ImpersonationConfiguration interface {
  // Return impersonation user. Should return system:serviceaccount:<namespace>:<serviceaccount>
  // if a service account is used for impersonation. Should return an empty string
  // if user shall not be impersonated.
  GetImpersonationUser() string
  // Return impersonation groups. Should return nil if groups shall not be impersonated.
  GetImpersonationGroups() []string
}

to use different user/groups for the deployment of dependent objects.

Note that, as mentioned above, the interfaces PlacementConfiguration, ClientConfiguration and ImpersonationConfiguration can be implemented by the component itself as well as by its spec type. In the theoretical case that both is the case, the implementation on the component level takes higher precedence.

The Generator interface

While Component (respectively the related custom resource type) models the desired and actual state of the managed component, the Generator interface is about implementing a recipe to render the Kubernetes manifests of the dependent objects, according to the provided parameterization (spec) of the component:

package manifests

// Resource generator interface.
// When called from the reconciler, the arguments namespace and name will match the
// component's namespace and name or, if the component or its spec implement the
// PlacementConfiguration interface, the return values of the GetDeploymentNamespace(), GetDeploymentName()
// methods (if non-empty). The parameters argument will be assigned the return value
// of the component's GetSpec() method.
type Generator interface {
  Generate(ctx context.Context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
}

In addition to namespace, name, parameters, generators can retrieve additional contextual information, such as a client for the deployment target by calling utils.ClientFromContext(), and related functions.

Component controllers can of course implement their own generator. In many cases (for example if there exists a Helm chart or kustomization for the component), one of the generators bundled with this repository can be used.

Generators may optionally implement

package types

// SchemeBuilder interface.
type SchemeBuilder interface {
  AddToScheme(scheme *runtime.Scheme) error
}

in order to enhance the scheme used by the dependent objects deployer.

2 - Component Reconciler

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Whenever a component resource (that is, an instance of the component’s custom resource type) is created, udpated, or deleted, the set of dependent object potentially changes, and the cluster state has to be synchronized with that new declared state. This synchronization is the job of the reconciler provided by this framework.

Creating the reconciler instance

Typically, a component operator runs one reconciler which is instantiated by calling the following constructor:

package component

func NewReconciler[T Component](
  name              string,
  resourceGenerator manifests.Generator
  options           ReconcilerOptions
) *Reconciler[T]

The passed type parameter T Component is the concrete runtime type of the component’s custom resource type (respectively, a pointer to that). Furthermore,

• name is supposed to be a unique name (typically a DNS name) identifying this component operator in the cluster; ìt will be used in annotations, labels, for leader election, …

• resourceGenerator is an implementation of the Generator interface, describing how the dependent objects are rendered from the component’s spec.

• options can be used to tune the behavior of the reconciler:

  package component
  
  // ReconcilerOptions are creation options for a Reconciler.
  type ReconcilerOptions struct {
    // Which field manager to use in API calls.
    // If unspecified, the reconciler name is used.
    FieldOwner *string
    // Which finalizer to use.
    // If unspecified, the reconciler name is used.
    Finalizer *string
    // Default service account used for impersonation of clients.
    // Of course, components can still customize impersonation by implementing the ImpersonationConfiguration interface.
    DefaultServiceAccount *string
    // How to react if a dependent object exists but has no or a different owner.
    // If unspecified, AdoptionPolicyIfUnowned is assumed.
    // Can be overridden by annotation on object level.
    AdoptionPolicy *reconciler.AdoptionPolicy
    // How to perform updates to dependent objects.
    // If unspecified, UpdatePolicyReplace is assumed.
    // Can be overridden by annotation on object level.
    UpdatePolicy *reconciler.UpdatePolicy
    // How to perform deletion of dependent objects.
    // If unspecified, DeletePolicyDelete is assumed.
    // Can be overridden by annotation on object level.
    DeletePolicy *reconciler.DeletePolicy
    // Whether namespaces are auto-created if missing.
    // If unspecified, MissingNamespacesPolicyCreate is assumed.
    MissingNamespacesPolicy *reconciler.MissingNamespacesPolicy
    // SchemeBuilder allows to define additional schemes to be made available in the
    // target client.
    SchemeBuilder types.SchemeBuilder
  }
  

The object returned by NewReconciler implements controller-runtime’s Reconciler interface, and can therefore be used as a drop-in in kubebuilder managed projects. After creation, the reconciler can be registered with the responsible controller-runtime manager instance by calling

package component

func (r *Reconciler[T]) SetupWithManager(mgr ctrl.Manager) error

The used manager mgr has to fulfill a few requirements:

• its client must bypass informer caches for the following types:
  • the type T itself
  • the type CustomResourceDefinition from the apiextensions.k8s.io/v1 group
  • the type APIService from the apiregistration.k8s.io/v1 group
• its scheme must recognize at least the following types:
  • the types in the API group defined in this repository
  • the core group (v1)
  • group apiextensions.k8s.io/v1
  • group apiregistration.k8s.io/v1.

Reconciler hooks

Component operators may register hooks to enhance the reconciler logic at certain points, by passing functions of type

package component

// HookFunc is the function signature that can be used to
// establish callbacks at certain points in the reconciliation logic.
// Hooks will be passed the current (potentially unsaved) state of the component.
// Post-hooks will only be called if the according operation (read, reconcile, delete)
// has been successful.
type HookFunc[T Component] func(ctx context.Context, client client.Client, component T) error

to the desired registration functions:

package component

// Register post-read hook with reconciler.
// This hook will be called after the reconciled component object has been retrieved from the Kubernetes API.
func (r *Reconciler[T]) WithPostReadHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right before the reconcilation of the dependent objects starts.
func (r *Reconciler[T]) WithPreReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register post-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right after the reconcilation of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right before the deletion of the dependent objects starts.
func (r *Reconciler[T]) WithPreDeleteHook(hook HookFunc[T]) *Reconciler[T]

// Register post-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right after the deletion of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostDeleteHook(hook HookFunc[T]) *Reconciler[T]

Note that the client passed to the hook functions is the client of the manager that was used when calling SetupWithManager() (that is, the return value of that manager’s GetClient() method). In addition, reconcile and delete hooks (that is, all except the post-read hook) can retrieve a client for the deployment target by calling ClientFromContext().

Tuning the retry behavior

By default, errors returned by the component’s generator or by a registered hook will make the reconciler go into a backoff managed by controller-runtime (which usually is an exponential backoff, capped at 10 minutes). However, if the error is or unwraps to a types.RetriableError, then the retry delay specified at the error will be used instead of the backoff. Implementations should use

pacakge types

func NewRetriableError(err error, retryAfter *time.Duration) RetriableError {
  return RetriableError{err: err, retryAfter: retryAfter}
}

to wrap an error into a RetriableError. It is allowed to pass retryAfter as nil; in that case the retry delay will be determined by calling the component’s GetRetryInterval() method (if the component or its spec implements the

package component

// The RetryConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the retry interval (by default, it would be the value of the requeue interval).
type RetryConfiguration interface {
  // Get retry interval. Should be greater than 1 minute.
  GetRetryInterval() time.Duration
}

interface), or otherwise will be set to the effective requeue interval (see below).

Tuning the requeue behavior

If a component was successfully reconciled, another reconciliation will be scheduled after 10 minutes, by default. This default requeue interval may be overridden by the component by implementing the

package component

// The RequeueConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the requeue interval (by default, it would be 10 minutes).
type RequeueConfiguration interface {
  // Get requeue interval. Should be greater than 1 minute.
  GetRequeueInterval() time.Duration
}

interface.

Tuning the timeout behavior

If the dependent objects of a component do not reach a ready state after a certain time, the component state will switch from Processing to Error. This timeout restarts counting whenever something changed in the component, or in the manifests of the dependent objects, and by default has the value of the effective requeue interval, which in turn defaults to 10 minutes. The timeout may be overridden by the component by implementing the

package component

// The TimeoutConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the processing timeout (by default, it would be the value of the requeue interval).
type TimeoutConfiguration interface {
  // Get timeout. Should be greater than 1 minute.
  GetTimeout() time.Duration
}

interface.

Tuning the handling of dependent objects

The reconciler allows to tweak how dependent objects are applied to or deleted from the cluster. To change the shipped framework defaults, a component type can implement the

package component

// The PolicyConfiguration interface is meant to be implemented by compoments (or their spec) which offer
// tweaking policies affecting the dependents handling.
type PolicyConfiguration interface {
  // Get adoption policy.
  // Must return a valid AdoptionPolicy, or the empty string (then the reconciler/framework default applies).
  GetAdoptionPolicy() reconciler.AdoptionPolicy
  // Get update policy.
  // Must return a valid UpdatePolicy, or the empty string (then the reconciler/framework default applies).
  GetUpdatePolicy() reconciler.UpdatePolicy
  // Get delete policy.
  // Must return a valid DeletePolicy, or the empty string (then the reconciler/framework default applies).
  GetDeletePolicy() reconciler.DeletePolicy
  // Get namspace auto-creation policy.
  // Must return a valid MissingNamespacesPolicy, or the empty string (then the reconciler/framework default applies).
  GetMissingNamespacesPolicy() reconciler.MissingNamespacesPolicy
}

interface. Note that most of the above policies can be overridden on a per-object level by setting certain annotations, as described here.

Declaring additional (implicit) managed types

One of component-operator-runtime’s core features is the special handling of instances of managed types. Managed types are API extension types (such as custom resource definitions or types added by federation of an aggregated API server). Instances of these types which are part of the component (so-called managed instances) are treated differently. For example, the framework tries to process these instances as late as possible when applying the component, and as early as possible when the component is deleted. Other instances of these types which are not part of the component (so-called foreign instances) block the deletion of the whole component.

Sometimes, components are implicitly adding extension types. That means that the type definition is not part of the component manifest, but the types are just created at runtime by controllers or operators contained in the component. A typical example are crossplane providers. These types are of course not recognized by the framework as managed types. However it is probably desired that (both managed and foreign) instances of these types experience the same special handling like instances of real managed types.

To make this possible, components can implement the

// The TypeConfiguration interface is meant to be implemented by compoments (or their spec) which allow
// to specify additional managed types.
type TypeConfiguration interface {
  // Get additional managed types; instances of these types are handled differently during
  // apply and delete; foreign instances of these types will block deletion of the component.
  // The fields of the returned TypeInfo structs can be concrete api groups, kinds,
  // or wildcards ("*"); in addition, groups can be specified as a pattern of the form "*.<suffix>"",
  // where the wildcard matches one or multiple dns labels.
  GetAdditionalManagedTypes() []reconciler.TypeInfo
}

interface. The types returned by GetAdditionalManagedTypes() contain a group and a kind, such as

// TypeInfo represents a Kubernetes type.
type TypeInfo struct {
	// API group.
	Group string `json:"group"`
	// API kind.
	Kind string `json:"kind"`
}

To match multiple types, the following pattern syntax is supported:

• the Group can be just * or have the form *.domain.suffix; note that the second case, the asterisk matches one or multiple DNS labels
• the Kind can be just *, which matches any kind.

3 - Dependent Objects

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Under normal circumstances, the framework manages the lifecycle of the dependent objects in a sufficient way. For example, it will create and delete objects in a meaningful order, trying to avoid corresponding transient or permanent errors.

A more remarkable feature of component-operator-runtime is that it will block deletion of dependent objects as long as non-managed instances of managed extension types (such as custom resource definitions) exist. To be more precise, assume for example, that the managed component contains some custom resource definition, plus the according operator. Then, if the component resource gets deleted, none of the component’s dependent objects will be touched as long as there exist foreign instances of the managed custom resource definition in the cluster.

In some special situations, it is desirable to have even more control on the lifecycle of the dependent objects. To support such cases, the Generator implementation can set the following annotations in the manifests of the dependents:

• mycomponent-operator.mydomain.io/adoption-policy: defines how the reconciler reacts if the object exists but has no or a different owner; can be one of:

  • never: fail if the object exists and has no or a different owner
  • if-unowned (which is the default): adopt the object if it has no owner set
  • always: adopt the object, even if it has a conflicting owner

• mycomponent-operator.mydomain.io/reconcile-policy: defines how the object is reconciled; can be one of:

  • on-object-change (which is the default): the object will be reconciled whenever its generated manifest changes
  • on-object-or-component-change: the object will be reconciled whenever its generated manifest changes, or whenever the responsible component object changes by generation
  • once: the object will be reconciled once, but never be touched again

• mycomponent-operator.mydomain.io/update-policy: defines how the object (if existing) is updated; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • replace (which is the default): a regular update (i.e. PUT) call will be made to the Kubernetes API server
  • ssa-merge: use server side apply to update existing dependents
  • ssa-override: use server side apply to update existing dependents and, in addition, reclaim fields owned by certain field owners, such as kubectl or Helm
  • recreate: if the object would be updated, it will be deleted and recreated instead

• mycomponent-operator.mydomain.io/delete-policy: defines what happens to the object when the compoment is deleted; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • delete (which is the default): a delete call will be sent to the Kubernetes API server
  • orphan: the object will not be deleted, and it will be no longer tracked; always, that is both if the object becomes redundant while applying the component, and if the component itself is deleted
  • orphan-on-apply: the object will not be deleted, and it will be no longer tracked; but only if the object becomes redundant while applying the component
  • orphan-on-delete: the object will not be deleted, and it will be no longer tracked; but only if the component itself is deleted

  note that the deletion policy has no effect in the case when objects are deleted because they become obsolete by applying a new version of the component manifests

• mycomponent-operator.mydomain.io/apply-order: the wave in which this object will be reconciled; dependents will be reconciled wave by wave; that is, objects of the same wave will be deployed in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous waves are ready; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0

• mycomponent-operator.mydomain.io/purge-order (optional): the wave by which this object will be purged; here, purged means that, while applying the dependents, the object will be deleted from the cluster at the end of the specified wave; the according record in status.Inventory will be set to phase Completed; setting purge orders is useful to spawn ad-hoc objects during the reconcilation, which are not permanently needed; so it’s comparable to Helm hooks, in a certain sense

• mycomponent-operator.mydomain.io/delete-order (optional): the wave by which this object will be deleted; that is, if the dependent is no longer part of the component, or if the whole component is being deleted; dependents will be deleted wave by wave; that is, objects of the same wave will be deleted in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous saves are gone; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0; note that the delete order is completely independent of the apply order

• mycomponent-operator.mydomain.io/status-hint (optional): a comma-separated list of hints that may help the framework to properly identify the state of the annotated dependent object; currently, the following hints are possible:

  • has-observed-generation: tells the framework that the dependent object has a status.observedGeneration field, even if it is not (yet) set by the responsible controller (some controllers are known to set the observed generation lazily, with the consequence that there is a period right after creation of the dependent object, where the field is missing in the dependent’s status)
  • has-ready-condition: tells the framework to count with a ready condition; if it is absent, the condition status will be considered as Unknown
  • conditions: semicolon-separated list of additional conditions that must be present and have a True status in order to make the overall status ready

Note that, in the above paragraph, mycomponent-operator.mydomain.io has to be replaced with whatever was passed as name when calling NewReconciler().

4 - Kubernetes Clients

When a component resource is reconciled, two Kubernetes API clients are constructed:

• The local client; it always points to the cluster where the component resides. If the component implements impersonation (that is, the component type or its spec implements the ImpersonationConfiguration interface), and an impersonation user or groups are specified by the component resource, then the specified user and groups are used to impersonate the controller’s kubeconfig. Otherwise, if a DefaultServiceAccount is defined in the reconciler’s options, then that service account (relative to the components metadata.namespace ) is used to impersonate the controller’s kubeconfig. Otherwise, the controller’s kubeconfig itself is used to build the local client. The local client is passed to generators via their context. For example, the HelmGenerator and KustomizeGenerator provided by component-operator-runtime use the local client to realize the localLookup and mustLocalLookup template functions.
• The target client; if the component specifies a kubeconfig (by implementing the ClientConfiguration interface), then that kubeconfig is used to build the target client. Otherwise, a local client is used (possibly impersonated), created according the the logic described above. The target client is used to manage dependent objects, and is passed to generators via their context. For example, the HelmGenerator and KustomizeGenerator provided by component-operator-runtime use the target client to realize the lookup and mustLookup template functions.