Kubernetes Operator Study Journey

1. Try using existing operators

2. Understand what is Kubernetes operator.

Kubernetes Controller components.
How Kubernetes Controlloer works.
Custom Resource.

Kubernetes Operator

A Kubernetes operator is an application-specific controller that extends the functionality of the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a Kubernetes user.

From What is a Kubernetes operator?

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

From https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

An Operator is like an automated Site Reliability Engineer for its application.

From Kubernetes Operators ~ Automating the Container Orchestration Platform ~

Operator vs. Controller

Controller（Custom Controller）:Custom Resourceの管理を行うController。Control Loop（Reconciliation Loop）を実行するコンポーネント

Operator: CRDとCustom Controllerのセット。etcd operatorやmysql operatorなどのように、特定のソフトウェアの管理を自動化するためのソフトウェア

From 実践入門Kubernetesカスタムコントローラーへの道

Controllers can act on core resources such as deployments or services, which are typically part of the Kubernetes controller manager in the control plane, or can watch and manipulate user-defined custom resources.

Operators are controllers that encode some operational knowledge, such as application lifecycle management, along with the custom resources defined in Chapter 4.

From Programming Kubernetes

A controller is a loop that reads desired state ("spec), observed cluster state (others' "status"), and external state, and the reconciles cluster state and external state with the desired state, writing any observations down (to our own "status").

All of Kubernetes functions on this model.

An operator is a controller that encodes human operational knowledge: how do I run and manage a specific piece of complex software.

All operators are controllers, but not all controllers are operators.

From Tutorial: Zero to Operator in 90 Minutes! - Solly Ross, Google (YouTube)

For more detail: - CNCF Operator White Paper - Final Version - CNCF White Paper

3. Create a sample operator following a tutorial

There are several ways to create an operator. You can try any of them:

You can also reference example controllers:

4. Understand more detail about each component

Simplified:

Detailed:

More Detailed:

from https://github.com/kubernetes/sample-controller/blob/master/docs/images/client-go-controller-interaction.jpeg

client-go:
1. clientset is a client for the built-in API resources.
2. informer: watch the changes of objects and reflect the changes to the in-memory-cache.
  1. factory: informers.NewSharedInformerFactory
  2. watcher
  3. lister
  4. indexer
  5. event handler
  6. reflector
3. lister: Get data from in-memory cache.
4. indexer: in-memory cache
5. workqueue: A queue to store items that the controller will process.
code-generator:
1. Generate codes for clientset for a custom resource.
apimachinery:
1. Scheme: connects Kubernetes API and Go Types, API version conversion
controller-runtime
1. builder
2. cache
3. client
4. cluster
5. controller
6. eventhandler
7. inject
8. log
9. manager
10. reconciler
11. source
12. webhook

Reference:

5. Create your own operator

After creating a sample operator, you should have deeper understanding of Kubernetes operator. Now you can think about what kind of problem that you want to resolve by utilizing operator pattern.

To clarify a problem to resolve with a new operator, you can reference existing operators:

operator	role	language
prometheus-operator	Manage Prometheus, Alertmanager and their configuration	Golang
mysql-operator	Manage MySQL cluster	Python
postgres-operator	Manage PostgreSQL cluster (version upgrade, live volume resize, ...)	Golang
strimzi-kafka-operator	Manage Kafka cluster, user, and topic	Java
...	...	...

Considerations:

Finalizer
docs: Owners and Dependents: example
Reconciliation Loop
- [operator-sdk] Based on the return value of Reconcile() the reconcile Request may be requeued and the loop may be triggered again: (Building a Go-based Memcached Operator using the Operator SDK)
```
// Reconcile successful - don't requeue
return reconcile.Result{}, nil
// Reconcile failed due to error - requeue
return reconcile.Result{}, err
// Requeue for any reason other than error
return reconcile.Result{Requeue: true}, nil
```
- https://github.com/operator-framework/operator-sdk/issues/4209#issuecomment-729916367
- How can I have separate logic for Create, Update, and Delete events? When reconciling an object can I access its previous state? -> You should not have separate logic. Instead design your reconciler to be idempotent.
  - Q: How do I have different logic in my reconciler for different types of events (e.g. create, update, delete)? in controller-runtime
Testing
- KUbernetes Testing TooL (kuttl): https://kuttl.dev/ KUTTL is built to support some kubernetes integration test scenarios and is most valuable as an end-to-end (e2e) test harness.
- Ginkgo (A Golang BDD Testing Framework): https://onsi.github.io/ginkgo/
- Gomega (Ginkgo's preferred matcher library): https://onsi.github.io/gomega/
- kubetest2: https://github.com/kubernetes-sigs/kubetest2: Kubetest2 is the framework for launching and running end-to-end tests on Kubernetes. It is intended to be the next significant iteration of kubetest.
Managing Errors
- https://cloud.redhat.com/blog/kubernetes-operators-best-practices
- Return the error in the status of the object.
- Generate an event describing the error.
Webhook
- Admission Webhook
  - two types:
    - Mutating Webhook: Make some modifications for a request. e.g. set default value. (Defined with MutatingAdmissionConfiguration)
    - Validating Webhook: Validate a request. (Defined by ValidatingAdmissionConfiguration)
  - Request: AdmissionReview
  - Response: AdmissionReview with response.allowed boolean field.
- Conversion Webhook
  - hub & spoke
  - isConvertible: need to have Hub & all non-Hub types must be able to convert to/from Hub
  - request & response: ConversionReview
Indexing

6. Tools

https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil
https://github.com/spf13/cobra: a library for creating powerful modern CLI applications & a program to generate applications and command files.
- Cobra is used in many Go projects such as Kubernetes, Hugo, and Github CLI to name a few. This list contains a more extensive list of projects using Cobra.

7. Study Golang for better code quality

8. Other Topics

Write Kubernetes Operator in other languages
- kopf for Python
- fabric8io/kubernetes-client for Java
- java-operator-sdk/java-operator-sdk Build Kubernetes Operators in Java Without Hassle
Optimistic Concurrency Control