# Using the Controller Runtime Client API with Operator SDK

## Overview

The [`controller-runtime`][repo-controller-runtime] library provides various abstractions to watch and reconcile resources in a Kubernetes cluster via CRUD (Create, Update, Delete, as well as Get and List in this case) operations. Operators use at least one controller to perform a coherent set of tasks within a cluster, usually through a combination of CRUD operations. The Operator SDK uses controller-runtime's [Client][doc-client-client] interface, which provides the interface for these operations.

controller-runtime defines several interfaces used for cluster interaction:
- `client.Client`: implementers perform CRUD operations on a Kubernetes cluster.
- `manager.Manager`: manages shared dependencies, such as Caches and Clients.
- `reconcile.Reconciler`: compares provided state with actual cluster state and updates the cluster on finding state differences using a Client.

Clients are the focus of this document. A separate document will discuss Managers.

## Client Usage

### Default Client

The SDK relies on a `manager.Manager` to create a `client.Client` interface that performs Create, Update, Delete, Get, and List operations within a `reconcile.Reconciler`'s Reconcile function. The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller's Reconciler will be populated with the Manager's Client which is a [split-client][doc-split-client].

`pkg/controller/<kind>/<kind>_controller.go`:
```Go
func newReconciler(mgr manager.Manager) reconcile.Reconciler {
	return &ReconcileKind{client: mgr.GetClient(), scheme: mgr.GetScheme()}
}

type ReconcileKind struct {
	// Populated above from a manager.Manager.
	client client.Client
	scheme *runtime.Scheme
}
```

A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.

### Non-default Client

An operator developer may wish to create their own Client that serves read requests(Get List) from the API server instead of the cache, for example. controller-runtime provides a [constructor][doc-client-constr] for Clients:

```Go
// New returns a new Client using the provided config and Options.
func New(config *rest.Config, options client.Options) (client.Client, error)
```

`client.Options` allow the caller to specify how the new Client should communicate with the API server.

```Go
// Options are creation options for a Client
type Options struct {
	// Scheme, if provided, will be used to map go structs to GroupVersionKinds
	Scheme *runtime.Scheme

	// Mapper, if provided, will be used to map GroupVersionKinds to Resources
	Mapper meta.RESTMapper
}
```
Example:
```Go
import (
    "sigs.k8s.io/controller-runtime/pkg/client/config"
    "sigs.k8s.io/controller-runtime/pkg/client"
)

cfg, err := config.GetConfig()
...
c, err := client.New(cfg, client.Options{})
...
```

**Note**: defaults are set by `client.New` when Options are empty. The default [scheme][code-scheme-default] will have the [core][doc-k8s-core] Kubernetes resource types registered. The caller *must* set a scheme that has custom operator types registered for the new Client to recognize these types.

Creating a new Client is not usually necessary nor advised, as the default Client is sufficient for most use cases.

### Reconcile and the Client API

A Reconciler implements the [`reconcile.Reconciler`][doc-reconcile-reconciler] interface, which exposes the Reconcile method. Reconcilers are added to a corresponding Controller for a Kind; Reconcile is called in response to cluster or external Events, with a `reconcile.Request` object argument, to read and write cluster state by the Controller, and returns a `reconcile.Result`. SDK Reconcilers have access to a Client in order to make Kubernetes API calls.

**Note**: For those familiar with the SDK's old project semantics, [Handle][doc-osdk-handle] received resource events and reconciled state for multiple resource types, whereas Reconcile receives resource events and reconciles state for a single resource type.

```Go
// ReconcileKind reconciles a Kind object
type ReconcileKind struct {
	// client, initialized using mgr.Client() above, is a split client
	// that reads objects from the cache and writes to the apiserver
	client client.Client

	// scheme defines methods for serializing and deserializing API objects,
	// a type registry for converting group, version, and kind information
	// to and from Go schemas, and mappings between Go schemas of different
	// versions. A scheme is the foundation for a versioned API and versioned
	// configuration over time.
	scheme *runtime.Scheme
}

// Reconcile watches for Events and reconciles cluster state with desired
// state defined in the method body.
// The Controller will requeue the Request to be processed again if an error
// is non-nil or Result.Requeue is true, otherwise upon completion it will
// remove the work from the queue.
func (r *ReconcileKind) Reconcile(request reconcile.Request) (reconcile.Result, error)
```

Reconcile is where Controller business logic lives, i.e. where Client API calls are made via `ReconcileKind.client`. A `client.Client` implementer performs the following operations:

#### Get

```Go
// Get retrieves an API object for a given object key from the Kubernetes cluster
// and stores it in obj.
func (c Client) Get(ctx context.Context, key ObjectKey, obj runtime.Object) error
```
**Note**: An `ObjectKey` is simply a `client` package alias for [`types.NamespacedName`][doc-types-nsname].

Example:
```Go
import (
	"context"
	"github.com/example-org/app-operator/pkg/apis/cache/v1alpha1"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	app := &v1alpha1.App{}
	ctx := context.TODO()
	err := r.client.Get(ctx, request.NamespacedName, app)

	...
}
```

#### List

```Go
// List retrieves a list of objects for a given namespace and list options
// and stores the list in obj.
func (c Client) List(ctx context.Context, list runtime.Object, opts ...client.ListOption) error
```

A `client.ListOption` is an interface that sets [`client.ListOptions`][list-options] fields. A `client.ListOption` is created by using one of the provided implementations: [`MatchingLabels`][matching-labels], [`MatchingFields`][matching-fields], [`InNamespace`][in-namespace].

Example:

```Go
import (
	"context"
	"fmt"
	"k8s.io/api/core/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	// Return all pods in the request namespace with a label of `app=<name>`
	// and phase `Running`.
	podList := &v1.PodList{}
	opts := []client.ListOption{
		client.InNamespace(request.NamespacedName.Namespace),
		client.MatchingLabels{"app", request.NamespacedName.Name},
		client.MatchingFields{"status.phase": "Running"},
	}
	ctx := context.TODO()
	err := r.client.List(ctx, podList, opts...)

	...
}
```

[list-options]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#ListOptions
[matching-labels]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#MatchingLabels
[matching-fields]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#MatchingFields
[in-namespace]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#InNamespace

#### Create

```Go
// Create saves the object obj in the Kubernetes cluster.
// Returns an error
func (c Client) Create(ctx context.Context, obj runtime.Object, opts ...client.CreateOption) error
```

A `client.CreateOption` is an interface that sets [`client.CreateOptions`][create-options] fields. A `client.CreateOption` is created by using one of the provided implementations: [`DryRunAll`][dry-run-all], [`ForceOwnership`][force-ownership]. Generally these options are not needed.

Example:

```Go
import (
	"context"
	"k8s.io/api/apps/v1"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	app := &v1.Deployment{ // Any cluster object you want to create.
		...
	}
	ctx := context.TODO()
	err := r.client.Create(ctx, app)

	...
}
```

[create-options]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#CreateOptions

#### Update

```Go
// Update updates the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned
// by the API server. Update does *not* update the resource's status
// subresource
func (c Client) Update(ctx context.Context, obj runtime.Object, opts ...client.UpdateOption) error
```

A `client.UpdateOption` is an interface that sets [`client.UpdateOptions`][update-options] fields. A `client.UpdateOption` is created by using one of the provided implementations: [`DryRunAll`][dry-run-all], [`ForceOwnership`][force-ownership]. Generally these options are not needed.

Example:

```Go
import (
	"context"
	"k8s.io/api/apps/v1"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	dep := &v1.Deployment{}
	err := r.client.Get(context.TODO(), request.NamespacedName, dep)

	...

	ctx := context.TODO()
	dep.Spec.Selector.MatchLabels["is_running"] = "true"
	err := r.client.Update(ctx, dep)

	...
}
```

[update-options]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#UpdateOptions

#### Patch

```Go
// Patch patches the given obj in the Kubernetes cluster. obj must be a
// struct pointer so that obj can be updated with the content returned by the Server.
func (c Client) Patch(ctx context.Context, obj runtime.Object, patch client.Patch, opts ...client.UpdateOption) error
```

A `client.PatchOption` is an interface that sets [`client.PatchOptions`][patch-options] fields. A `client.PatchOption` is created by using one of the provided implementations: [`DryRunAll`][dry-run-all], [`ForceOwnership`][force-ownership]. Generally these options are not needed.

Example:

```Go
import (
	"context"
	"k8s.io/api/apps/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	dep := &v1.Deployment{}
	err := r.client.Get(context.TODO(), request.NamespacedName, dep)

	...

	ctx := context.TODO()
	dep.Spec.Selector.MatchLabels["is_running"] = "true"
	// A merge patch will preserve other fields modified at runtime.
	patch := client.MergeFrom(dep)
	err := r.client.Patch(ctx, dep, patch)

	...
}
```

[patch-options]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#PatchOption
[dry-run-all]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#DryRunAll
[force-ownership]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#ForceOwnership

##### Updating Status Subresource

When updating the [status subresource][cr-status-subresource] from the client, the [`StatusWriter`][status-writer] must be used. The status subresource is retrieved with `Status()` and updated with `Update()` or patched with `Patch()`.

`Update()` takes variadic `client.UpdateOption`'s, and `Patch()` takes variadic `client.PatchOption`'s. See [`Client.Update()`](#update) and [`Client.Patch()`](#patch) for more details. Generally these options are not needed.

##### Status

```Go
// Status() returns a StatusWriter object that can be used to update the
// object's status subresource
func (c Client) Status() (client.StatusWriter, error)
```

Example:
```Go
import (
	"context"
	cachev1alpha1 "github.com/example-inc/memcached-operator/pkg/apis/cache/v1alpha1"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	ctx := context.TODO()
	mem := &cachev1alpha1.Memcached{}
	err := r.client.Get(ctx, request.NamespacedName, mem)

	...

	// Update
	mem.Status.Nodes = []string{"pod1", "pod2"}
	err := r.client.Status().Update(ctx, mem)

	...

	// Patch
	patch := client.MergeFrom(mem)
	err := r.client.Status().Patch(ctx, mem, patch)

	...
}
```

[status-writer]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#StatusWriter

#### Delete

```Go
// Delete deletes the given obj from Kubernetes cluster.
func (c Client) Delete(ctx context.Context, obj runtime.Object, opts ...client.DeleteOption) error
```

A `client.DeleteOption` is an interface that sets [`client.DeleteOptions`][delete-opts] fields. A `client.DeleteOption` is created by using one of the provided implementations: [`GracePeriodSeconds`][grace-period-seconds], [`Preconditions`][preconditions], [`PropagationPolicy`][propagation-policy].

Example:

```Go
import (
	"context"
	"k8s.io/api/core/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	pod := &v1.Pod{}
	err := r.client.Get(context.TODO(), request.NamespacedName, pod)

	...

	ctx := context.TODO()
	if pod.Status.Phase == v1.PodUnknown {
		// Delete the pod after 5 seconds.
		err := r.client.Delete(ctx, pod, client.GracePeriodSeconds(5))
		...
	}

	...
}
```

[delete-opts]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#DeleteOptions
[grace-period-seconds]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#GracePeriodSeconds
[preconditions]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#Preconditions
[propagation-policy]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#PropagationPolicy

#### DeleteAllOf

```Go
// DeleteAllOf deletes all objects of the given type matching the given options.
func (c Client) DeleteAllOf(ctx context.Context, obj runtime.Object, opts ...client.DeleteAllOfOption) error
```

A `client.DeleteAllOfOption` is an interface that sets [`client.DeleteAllOfOptions`][deleteallof-opts] fields. A `client.DeleteAllOfOption` wraps a [`client.ListOption`](#list) and [`client.DeleteOption`](#delete).

Example:

```Go
import (
	"context"
	"fmt"
	"k8s.io/api/core/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	...

	// Delete all pods in the request namespace with a label of `app=<name>`
	// and phase `Failed`.
	pod := &v1.Pod{}
	opts := []client.DeleteAllOfOption{
		client.InNamespace(request.NamespacedName.Namespace),
		client.MatchingLabels{"app", request.NamespacedName.Name},
		client.MatchingFields{"status.phase": "Failed"},
		client.GracePeriodSeconds(5),
	}
	ctx := context.TODO()
	err := r.client.DeleteAllOf(ctx, pod, opts...)

	...
}
```

[deleteallof-opts]:https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client#DeleteAllOfOptions

### Example usage

```Go
import (
	"context"
	"reflect"

	appv1alpha1 "github.com/example-org/app-operator/pkg/apis/app/v1alpha1"

	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/labels"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"
)

type ReconcileApp struct {
	client client.Client
	scheme *runtime.Scheme
}

func (r *ReconcileApp) Reconcile(request reconcile.Request) (reconcile.Result, error) {

	// Fetch the App instance.
	app := &appv1alpha1.App{}
	err := r.client.Get(context.TODO(), request.NamespacedName, app)
	if err != nil {
		if errors.IsNotFound(err) {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, err
	}

	// Check if the deployment already exists, if not create a new deployment.
	found := &appsv1.Deployment{}
	err = r.client.Get(context.TODO(), types.NamespacedName{Name: app.Name, Namespace: app.Namespace}, found)
	if err != nil {
	 	if errors.IsNotFound(err) {
			// Define and create a new deployment.
			dep := r.deploymentForApp(app)
			if err = r.client.Create(context.TODO(), dep); err != nil {
				return reconcile.Result{}, err
			}
			return reconcile.Result{Requeue: true}, nil
		} else {
			return reconcile.Result{}, err			
		}
	}

	// Ensure the deployment size is the same as the spec.
	size := app.Spec.Size
	if *found.Spec.Replicas != size {
		found.Spec.Replicas = &size
		if err = r.client.Update(context.TODO(), found); err != nil {
			return reconcile.Result{}, err
		}
		return reconcile.Result{Requeue: true}, nil
	}

	// Update the App status with the pod names.
	// List the pods for this app's deployment.
	podList := &corev1.PodList{}
	listOpts := []client.ListOption{
		client.InNamespace(app.Namespace),
		client.MatchingLabels(labelsForApp(app.Name)),
	}
	if err = r.client.List(context.TODO(), podList, listOpts...); err != nil {
		return reconcile.Result{}, err
	}

	// Update status.Nodes if needed.
	podNames := getPodNames(podList.Items)
	if !reflect.DeepEqual(podNames, app.Status.Nodes) {
		app.Status.Nodes = podNames
		if err := r.client.Status().Update(context.TODO(), app); err != nil {
			return reconcile.Result{}, err
		}
	}

	return reconcile.Result{}, nil
}

// deploymentForApp returns a app Deployment object.
func (r *ReconcileKind) deploymentForApp(m *appv1alpha1.App) *appsv1.Deployment {
	lbls := labelsForApp(m.Name)
	replicas := m.Spec.Size

	dep := &appsv1.Deployment{
		ObjectMeta: metav1.ObjectMeta{
			Name:      m.Name,
			Namespace: m.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: lbls,
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: lbls,
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{{
						Image:   "app:alpine",
						Name:    "app",
						Command: []string{"app", "-a=64", "-b"},
						Ports: []corev1.ContainerPort{{
							ContainerPort: 10000,
							Name:          "app",
						}},
					}},
				},
			},
		},
	}

	// Set App instance as the owner and controller.
	// NOTE: calling SetControllerReference, and setting owner references in
	// general, is important as it allows deleted objects to be garbage collected.
	controllerutil.SetControllerReference(m, dep, r.scheme)
	return dep
}

// labelsForApp creates a simple set of labels for App.
func labelsForApp(name string) map[string]string {
	return map[string]string{"app_name": "app", "app_cr": name}
}
```

[repo-controller-runtime]:https://github.com/kubernetes-sigs/controller-runtime
[doc-client-client]:https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/client#Client
[doc-split-client]:https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/client#DelegatingClient
[doc-client-constr]:https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/client#New
[code-scheme-default]:https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/client/client.go#L51
[doc-k8s-core]:https://godoc.org/k8s.io/api/core/v1
[doc-reconcile-reconciler]:https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/reconcile#Reconciler
[doc-osdk-handle]:https://github.com/operator-framework/operator-sdk/blob/master/doc/design/milestone-0.0.2/action-api.md#handler
[doc-types-nsname]:https://godoc.org/k8s.io/apimachinery/pkg/types#NamespacedName
[cr-status-subresource]:https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#status-subresource
