Concepts

• 1: Components and Generators
• 2: Component Reconciler
• 3: Dependent Objects

1 - Components and Generators

In the terminology of this project, a Kubernetes cluster component (sometimes called module) consists of a set of dependent objects that are to be deployed consistently into the Kubernetes cluster. The continuous reconciliation of the declared state of these dependent objects is the task of an operator implemented by component-operator-runtime. To achieve this, basically two interfaces have to be implemented by such an operator…

The Component Interface

In the programming model proposed by component-operator-runtime, the declared and observed state of the component is represented by a dedicated custom resource type. The corresponding runtime type has to fulfill the following interface:

package component

// Component is the central interface that component operators have to implement.
// Besides being a conroller-runtime client.Object, the implementing type has to expose accessor
// methods for the components's spec and status, GetSpec() and GetStatus().
type Component interface {
  client.Object
  // Return a read-only accessor to the component's spec.
  // The returned value has to implement the types.Unstructurable interface.
  GetSpec() types.Unstructurable
  // Return a read-write (usually a pointer) accessor to the component's status,
  // resp. to the corresponding substruct if the status extends component.Status.
  GetStatus() *Status
}

Basically, two accessor methods have to be implemented here. First, GetSpec() exposes the parameterization of the component. The only requirement on the returned type is to implement the

package types

// Unstructurable represents objects which can be converted into a string-keyed map.
// All Kubernetes API types, as well as all JSON objects could be modelled as Unstructurable objects.
type Unstructurable interface {
  ToUnstructured() map[string]any
}

interface. In most cases, the returned Unstructurable object is the spec itself, or a deep copy of the spec. In general, the implementation is allowed to return arbitrary content, as long as the receiving generator is able to process it. In particular, it is not expected by the framework that changes applied to the returned Unstructurable reflect in any way in the component; indeed, the framework will never modify the returned Unstructurable.

Finally, GetStatus() allows the framework to access (a part of) the custom resource type’s status, having the following type:

package component

// Component Status. Components must include this into their status.
type Status struct {
  ObservedGeneration int64            `json:"observedGeneration"`
  AppliedGeneration  int64            `json:"appliedGeneration,omitempty"`
  LastObservedAt     *metav1.Time     `json:"lastObservedAt,omitempty"`
  LastAppliedAt      *metav1.Time     `json:"lastAppliedAt,omitempty"`
  Conditions         []Condition      `json:"conditions,omitempty"`
  State              State            `json:"state,omitempty"`
  Inventory          []*InventoryItem `json:"inventory,omitempty"`
}

Note that, other than with the GetSpec() accessor, the framework will make changes to the returned Status structure. Thus, in almost all cases, the returned pointer should just reference the status of the component’s API type (or an according substructure of that status).

The component’s custom resource type is supposed to be namespaced, and by default, dependent objects will be created in that same namespace. To be more precise, the namespace and name parameters of the used generator’s Generate() method will be set to the component’s metadata.namespace and metadata.name, respectively. Sometimes it might be desired to override these defaults, and to render the dependent objects with a different namespace or name. To allow this, the component (or its spec) can implement

package component

// The PlacementConfiguration interface is meant to be implemented by components (or their spec) which allow
// to explicitly specify target namespace and name of the deployment (otherwise this will be defaulted as
// the namespace and name of the component object itself).
type PlacementConfiguration interface {
  // Return target namespace for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generate() as namespace and, in addition, rendered namespaced resources with
  // unspecified namespace will be placed in this namespace.
  GetDeploymentNamespace() string
  // Return target name for the component deployment.
  // If the returned value is not the empty string, then this is the value that will be passed
  // to Generator.Generator() as name.
  GetDeploymentName() string
}

In addition, the component (or its spec) may implement

package component

// The ClientConfiguration interface is meant to be implemented by components (or their spec) which offer
// remote deployments.
type ClientConfiguration interface {
  // Get kubeconfig content. Should return nil if default local client shall be used.
  GetKubeConfig() []byte
}

in order to support remote deployments (that is, to make the deployment of the dependent objects use the specified kubeconfig), and

package component

// The ImpersonationConfiguration interface is meant to be implemented by components (or their spec) which offer
// impersonated deployments.
type ImpersonationConfiguration interface {
  // Return impersonation user. Should return system:serviceaccount:<namespace>:<serviceaccount>
  // if a service account is used for impersonation. Should return an empty string
  // if user shall not be impersonated.
  GetImpersonationUser() string
  // Return impersonation groups. Should return nil if groups shall not be impersonated.
  GetImpersonationGroups() []string
}

to use different user/groups for the deployment of dependent objects. Implementing both ClientConfiguration and ImpersonationConfiguration means that the provided kubeconfig will be impersonated as specified.

Note that, as mentioned above, the interfaces PlacementConfiguration, ClientConfiguration and ImpersonationConfiguration can be implemented by the component itself as well as by its spec type. In the theoretical case that both is the case, the implementation on the component level takes higher precedence.

The Generator interface

While Component (respectively the related custom resource type) models the desired and actual state of the managed component, the Generator interface is about implementing a recipe to render the Kubernetes manifests of the dependent objects, according to the provided parameterization (spec) of the component:

package manifests

// Resource generator interface.
// When called from the reconciler, the arguments namespace and name will match the
// component's namespace and name or, if the component or its spec implement the
// PlacementConfiguration interface, the return values of the GetDeploymentNamespace(), GetDeploymentName()
// methods (if non-empty). The parameters argument will be assigned the return value
// of the component's GetSpec() method.
type Generator interface {
  Generate(ctx context.Context, namespace string, name string, parameters types.Unstructurable) ([]client.Object, error)
}

In addition to namespace, name, parameters, generators can retrieve additional contextual information, such as a client for the deployment target by calling utils.ClientFromContext(), and related functions.

Component controllers can of course implement their own generator. In many cases (for example if there exists a Helm chart or kustomization for the component), one of the generators bundled with this repository can be used.

Generators may optionally implement

package types

// SchemeBuilder interface.
type SchemeBuilder interface {
  AddToScheme(scheme *runtime.Scheme) error
}

in order to enhance the scheme used by the dependent objects deployer.

2 - Component Reconciler

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Whenever a component resource (that is, an instance of the component’s custom resource type) is created, udpated, or deleted, the set of dependent object potentially changes, and the cluster state has to be synchronized with that new declared state. This synchronization is the job of the reconciler provided by this framework.

Creating the reconciler instance

Typically, a component operator runs one reconciler which is instantiated by calling the following constructor:

package component

func NewReconciler[T Component](
  name              string,
  resourceGenerator manifests.Generator
  options           ReconcilerOptions
) *Reconciler[T]

The passed type parameter T Component is the concrete runtime type of the component’s custom resource type. Furthermore,

• name is supposed to be a unique name (typically a DNS name) identifying this component operator in the cluster; ìt will be used in annotations, labels, for leader election, …

• resourceGenerator is an implementation of the Generator interface, describing how the dependent objects are rendered from the component’s spec.

• options can be used to tune the behavior of the reconciler:

  package component
  
  // ReconcilerOptions are creation options for a Reconciler.
  type ReconcilerOptions struct {
    // Whether namespaces are auto-created if missing.
    // If unspecified, true is assumed.
    CreateMissingNamespaces *bool
    // How to react if a dependent object exists but has no or a different owner.
    // If unspecified, AdoptionPolicyIfUnowned is assumed.
    // Can be overridden by annotation on object level.
    AdoptionPolicy *reconciler.AdoptionPolicy
    // How to perform updates to dependent objects.
    // If unspecified, UpdatePolicyReplace is assumed.
    // Can be overridden by annotation on object level.
    UpdatePolicy *reconciler.UpdatePolicy
    // How to perform deletion of dependent objects.
    // If unspecified, DeletePolicyDelete is assumed.
    // Can be overridden by annotation on object level.
    DeletePolicy *reconciler.DeletePolicy
    // SchemeBuilder allows to define additional schemes to be made available in the
    // target client.
    SchemeBuilder types.SchemeBuilder
  }
  

The object returned by NewReconciler implements controller-runtime’s Reconciler interface, and can therefore be used as a drop-in in kubebuilder managed projects. After creation, the reconciler can be registered with the responsible controller-runtime manager instance by calling

package component

func (r *Reconciler[T]) SetupWithManager(mgr ctrl.Manager) error

The used manager mgr has to fulfill a few requirements:

• its client must bypass informer caches for the following types:
  • the type T itself
  • the type CustomResourceDefinition from the apiextensions.k8s.io/v1 group
  • the type APIService from the apiregistration.k8s.io/v1 group
• its scheme must recognize at least the following types:
  • the types in the API group defined in this repository
  • the core group (v1)
  • group apiextensions.k8s.io/v1
  • group apiregistration.k8s.io/v1.

Reconciler hooks

Component operators may register hooks to enhance the reconciler logic at certain points, by passing functions of type

package component

// HookFunc is the function signature that can be used to
// establish callbacks at certain points in the reconciliation logic.
// Hooks will be passed the current (potentially unsaved) state of the component.
// Post-hooks will only be called if the according operation (read, reconcile, delete)
// has been successful.
type HookFunc[T Component] func(ctx context.Context, client client.Client, component T) error

to the desired registration functions:

package component

// Register post-read hook with reconciler.
// This hook will be called after the reconciled component object has been retrieved from the Kubernetes API.
func (r *Reconciler[T]) WithPostReadHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right before the reconcilation of the dependent objects starts.
func (r *Reconciler[T]) WithPreReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register post-reconcile hook with reconciler.
// This hook will be called if the reconciled component is not in deletion (has no deletionTimestamp set),
// right after the reconcilation of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostReconcileHook(hook HookFunc[T]) *Reconciler[T]

// Register pre-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right before the deletion of the dependent objects starts.
func (r *Reconciler[T]) WithPreDeleteHook(hook HookFunc[T]) *Reconciler[T]

// Register post-delete hook with reconciler.
// This hook will be called if the reconciled component is in deletion (has a deletionTimestamp set),
// right after the deletion of the dependent objects happened, and was successful.
func (r *Reconciler[T]) WithPostDeleteHook(hook HookFunc[T]) *Reconciler[T]

Note that the client passed to the hook functions is the client of the manager that was used when calling SetupWithManager() (that is, the return value of that manager’s GetClient() method). In addition, reconcile and delete hooks (that is, all except the post-read hook) can retrieve a client for the deployment target by calling ClientFromContext().

Tuning the retry behavior

By default, errors returned by the component’s generator or by a registered hook will make the reconciler go into a backoff managed by controller-runtime (which usually is an exponential backoff, capped at 10 minutes). However, if the error is or unwraps to a types.RetriableError, then the retry delay specified at the error will be used instead of the backoff. Implementations should use

pacakge types

func NewRetriableError(err error, retryAfter *time.Duration) RetriableError {
	return RetriableError{err: err, retryAfter: retryAfter}
}

to wrap an error into a RetriableError. It is allowed to pass retryAfter as nil; in that case the retry delay will be determined by calling the component’s GetRetryInterval() method (if the component or its spec implements the

package component

// The RetryConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the retry interval (by default, it would be the value of the requeue interval).
type RetryConfiguration interface {
	// Get retry interval. Should be greater than 1 minute.
	GetRetryInterval() time.Duration
}

interface), or otherwise will be set to the effective requeue interval (see below).

Tuning the requeue behavior

If a component was successfully reconciled, another reconciliation will be scheduled after 10 minutes, by default. This default requeue interval may be overridden by the component by implementing the

package component

// The RequeueConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the requeue interval (by default, it would be 10 minutes).
type RequeueConfiguration interface {
	// Get requeue interval. Should be greater than 1 minute.
	GetRequeueInterval() time.Duration
}

interface.

Tuning the timeout behavior

If the dependent objects of a component do not reach a ready state after a certain time, the component state will switch from Processing to Error. This timeout restarts counting whenever something changed in the component, or in the manifests of the dependent objects, and by default has the value of the effective requeue interval, which in turn defaults to 10 minutes. The timeout may be overridden by the component by implementing the

package component

// The TimeoutConfiguration interface is meant to be implemented by components (or their spec) which offer
// tweaking the processing timeout (by default, it would be the value of the requeue interval).
type TimeoutConfiguration interface {
	// Get timeout. Should be greater than 1 minute.
	GetTimeout() time.Duration
}

interface.

3 - Dependent Objects

Dependent objects are - by definition - the resources returned by the Generate() method of the used resource generator. Under normal circumstances, the framework manages the lifecycle of the dependent objects in a sufficient way. For example, it will create and delete objects in a meaningful order, trying to avoid corresponding transient or permanent errors.

A more remarkable feature of component-operator-runtime is that it will block deletion of dependent objects as long as non-managed instances of managed extension types (such as custom resource definitions) exist. To be more precise, assume for example, that the managed component contains some custom resource definition, plus the according operator. Then, if the component resource gets deleted, none of the component’s dependent objects will be touched as long as there exist foreign instances of the managed custom resource definition in the cluster.

In some special situations, it is desirable to have even more control on the lifecycle of the dependent objects. To support such cases, the Generator implementation can set the following annotations in the manifests of the dependents:

• mycomponent-operator.mydomain.io/adoption-policy: defines how the reconciler reacts if the object exists but has no or a different owner; can be one of:

  • never: fail if the object exists and has no or a different owner
  • if-unowned (which is the default): adopt the object if it has no owner set
  • always: adopt the object, even if it has a conflicting owner

• mycomponent-operator.mydomain.io/reconcile-policy: defines how the object is reconciled; can be one of:

  • on-object-change (which is the default): the object will be reconciled whenever its generated manifest changes
  • on-object-or-component-change: the object will be reconciled whenever its generated manifest changes, or whenever the responsible component object changes by generation
  • once: the object will be reconciled once, but never be touched again

• mycomponent-operator.mydomain.io/update-policy: defines how the object (if existing) is updated; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • replace (which is the default): a regular update (i.e. PUT) call will be made to the Kubernetes API server
  • ssa-merge: use server side apply to update existing dependents
  • ssa-override: use server side apply to update existing dependents and, in addition, reclaim fields owned by certain field owners, such as kubectl or Helm
  • recreate: if the object would be updated, it will be deleted and recreated instead

• mycomponent-operator.mydomain.io/delete-policy: defines what happens to the object when the compoment is deleted; can be one of:

  • default (deprecated): equivalent to the annotation being unset (which means that the reconciler default will be used)
  • delete (which is the default): a delete call will be sent to the Kubernetes API server
  • orphan: the object will not be deleted, and it will be no longer tracked

  note that the deletion policy has no effect in the case when objects are deleted because they become obsolete by applying a new version of the component manifests

• mycomponent-operator.mydomain.io/apply-order: the wave in which this object will be reconciled; dependents will be reconciled wave by wave; that is, objects of the same wave will be deployed in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous waves are ready; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0

• mycomponent-operator.mydomain.io/purge-order (optional): the wave by which this object will be purged; here, purged means that, while applying the dependents, the object will be deleted from the cluster at the end of the specified wave; the according record in status.Inventory will be set to phase Completed; setting purge orders is useful to spawn ad-hoc objects during the reconcilation, which are not permanently needed; so it’s comparable to Helm hooks, in a certain sense

• mycomponent-operator.mydomain.io/delete-order (optional): the wave by which this object will be deleted; that is, if the dependent is no longer part of the component, or if the whole component is being deleted; dependents will be deleted wave by wave; that is, objects of the same wave will be deleted in a canonical order, and the reconciler will only proceed to the next wave if all objects of previous saves are gone; specified orders can be negative or positive numbers between -32768 and 32767, objects with no explicit order set are treated as if they would specify order 0; note that the delete order is completely independent of the apply order

• mycomponent-operator.mydomain.io/status-hint (optional): a comma-separated list of hints that may help the framework to properly identify the state of the annotated dependent object; currently, the following hints are possible:

  • has-observed-generation: tells the framework that the dependent object has a status.observedGeneration field, even if it is not (yet) set by the responsible controller (some controllers are known to set the observed generation lazily, with the consequence that there is a period right after creation of the dependent object, where the field is missing in the dependent’s status)
  • has-ready-condition: tells the framework to count with a ready condition; if it is absent, the condition status will be considered as Unknown
  • conditions: semicolon-separated list of additional conditions that must be present and have a True status in order to make the overall status ready

Note that, in the above paragraph, mycomponent-operator.mydomain.io has to be replaced with whatever was passed as name when calling NewReconciler().