How Kubernetes Works: One API, Many Controllers

• foundations
• kube-apiserver
• controllers

Kubernetes is a distributed system built around a single HTTP API server. Every other part is a small, independent program called a controller that watches parts of that API and acts on changes. Nothing else is going on inside it.

There is no central “Kubernetes” program

No Kubernetes binary or process exists, and no controller manages the other controllers. “Kubernetes” is the name given to a microservices architecture in which:

• No service is called Kubernetes.
• Every service is decoupled from every other by one central API endpoint.
• That endpoint, kube-apiserver, is not called Kubernetes either.

kube-apiserver is the only interface, and it speaks HTTP

kube-apiserver is a plain RESTful HTTP server. There are no special binary protocols, and it is the only interface in the system. Humans checking pod status, automation deploying workloads and internal components changing a pod’s state (say, from Ready to CrashLoopBackoff) all use it.

Commands like kubectl apply are just details inside the body of an HTTP request. Applying a Deployment sends this:

POST /apis/apps/v1/namespaces/foo/deployments?fieldManager=kubectl-client-side-apply HTTP/1.1
Host: 10.96.0.1:443
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjRPYWQ3NjZ...
Content-Type: application/json
Accept: application/json

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": { "name": "foo", "namespace": "foo", "annotations": { "...": "..." } },
  "spec": {
    "replicas": 1,
    "selector": { "matchLabels": { "app": "foo" } },
    "template": {
      "metadata": { "labels": { "app": "foo" } },
      "spec": { "containers": [ { "name": "foo", "image": "nginx:1.27" } ] }
    }
  }
}

The body is a serialized Deployment data structure . kubectl builds the request from the endpoint and bearer token in the current context, so the same call could be made with cURL. The other kubectl commands that look special are still HTTP:

The kubelet has its own API, but clients never talk to it directly. They only ever reach kube-apiserver.

Because everything passes through this one endpoint, all role-based access control lives in kube-apiserver. A person listing pods in a namespace and a process updating the status of another process in a different namespace are authorized the same way: a request arrives at kube-apiserver, which checks RBAC.

A “resource” is an API endpoint

A resource is a URL on kube-apiserver. In GET /api/v1/namespaces/foo/pods, the request fetches pods from the namespace foo.

The URL structure carries the naming scheme:

• /api vs /apis. /api marks core resources bundled with kube-apiserver. /apis marks resources added as extensions after the fact (custom resources), such as /apis/cert-manager.io/v1/namespaces/foo/certificates.
• Group and version. In cert-manager.io/v1, the group is cert-manager.io and the version is v1. Group, version and the resource kind together form a GroupVersionKind (GVK), which uniquely identifies a resource type.
• YAML names. In YAML manifests the GroupVersion is written as apiVersion: and the kind as kind:.

Pods have no group, in either the URL or the GVK. This is a historical accident. Originally everything was just a version and a kind, which is why the field is called apiVersion. When groups were introduced to avoid name collisions (so confusing.io/v1 pod and stillconfusing.io/v1 pod can coexist), the group name was folded into the existing apiVersion field. Pre-group resources sit in the empty-string group, called the core or legacy group.

A third category also exists: resources added to the built-in set after groups were invented. They ship with kube-apiserver and have proper group names, so they look just like custom resources and can only be told apart by knowing which is which.

CustomResourceDefinitions add new endpoints

A CustomResourceDefinition (CRD) is the manifest used to install a new custom resource. The two are different things: the CRD is the installer, and the custom resource is the endpoint it creates. A CRD is a resource manifest with the schema for the new kind nested in its spec:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: certificates.cert-manager.io
spec:
  group: cert-manager.io
  names:
    kind: Certificate
    plural: certificates
    singular: certificate
    shortNames: [cert, certs]
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [secretName, issuerRef]
              # ...the real one runs several hundred lines longer

apiextensions.k8s.io/v1 CustomResourceDefinition is itself a built-in resource with a named group. Applying a CRD means POSTing it to that endpoint, which is cluster-wide rather than namespaced. kube-apiserver then re-wires itself to serve the new URL, here /apis/cert-manager.io/v1/namespaces/<namespace>/certificates.

State lives in etcd

kube-apiserver stores every modification to every resource in etcd, including copies of all CRDs. That lets it rebuild its API surface after a restart. For a custom resource such as certificates:

• POST a new certificate: kube-apiserver writes it to etcd.
• GET: it reads the matching records from etcd and returns them in the response body. Caching means a GET does not always hit etcd.
• PATCH: it finds the stored instance, applies the change and writes the result back.
• DELETE: it finds the record and removes it.
• Deleting the CRD itself removes the endpoint, and later requests return 404.

An “instance” is a record with the shape defined in the CRD, with required values filled in:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: foo
  namespace: foo
spec:
  secretName: foo-tls
  dnsNames: [foo.example.com]
  issuerRef: { name: foo-issuer, kind: Issuer }
status:
  conditions:
    - { type: Ready, status: "True", reason: Ready }

Internally, each resource is a Go struct serialized into etcd. The YAML that people write and read is only an exported view of that struct.

Typically a Go program must be recompiled to learn new types, but kube-apiserver adds types at runtime. It and its controllers pack custom resources into unstructured.Unstructured values, which implement the runtime.Object interface. Controllers compiled with the specific types can later convert them to strong types. The unstructured envelope is also where fields like metadata.name come from even though the CRD never declares them. etcd itself is untyped by design and needs no schema.

Watch turns the API into an event stream

Adding ?watch=true to any GET, for example GET /api/v1/namespaces/foo/pods?watch=true, makes kube-apiserver switch to streaming mode. The client stays connected, since the server never disconnects it, and receives newline-delimited JSON objects describing every operation on the matching resources as it happens. A single pod’s lifecycle looks like this:

1. ADDED: the pod appears in phase Pending.
2. MODIFIED: it is assigned to node-3, with its container in ContainerCreating.
3. MODIFIED: the phase becomes Running and the pod gets IP 10.244.2.17.
4. MODIFIED: the container fails, and the status shows CrashLoopBackOff with restartCount 4 and exit code 1.
5. BOOKMARK: a progress marker carrying only a resourceVersion.
6. DELETED: the pod is removed.

Each object includes a resourceVersion and a uid. The resourceVersion is a monotonically increasing counter. Clients use it to reconnect at a known point (?watch=true&resourceVersion=10309) or to disambiguate races. Similar mechanisms handle conflicting operations on etcd, such as two clients sending a delete and an update at the same moment.

The full picture of kube-apiserver is therefore: it listens on many resource URLs, holds a connection to etcd, and keeps an in-memory map of open watch sockets keyed by GVK and namespace. When a request arrives, it does the create, read, update or delete against etcd, then tells any matching watchers what happened.

Controllers do all the work

A controller is a program that watches a resource, responds to changes and communicates by writing back to kube-apiserver. Take TLS certificates:

1. A program watches Certificate instances. It can regenerate certs when the authority changes, find certs with a secret but no actual certificate, and wake periodically to check expiry dates.
2. When a certificate lacks a valid cert, it creates a CertificateRequest resource (registered through its own CRD).
3. A second program connected to a certificate authority watches for CertificateRequests. It signs them and PATCHes the result back to the certificates endpoint.
4. The first program, still watching, sees the change and continues.

Controllers never talk to each other directly. They communicate only through HTTP requests to kube-apiserver, and each one typically does one job on one resource. Several controllers usually watch the same resource. The cert-manager.io/v1 group has over 20 controllers working on 6 resources.

kube-apiserver runs controllers of its own to handle CRDs:

• crdController watches CustomResourceDefinitions and registers an HTTP route for each new GVK.
• establishingController sets the CRD’s Established condition once the route is live.
• namingController checks for naming conflicts, since two CRDs can’t claim the same plural name in one group.
• openapiController regenerates the /openapi/v2 and /openapi/v3 docs to include the new schema.
• finalizingController handles cleanup when a CRD is deleted.

Any language that can make an HTTP GET with URL parameters can implement a controller. This bash line is the skeleton of one:

curl -sk -H "Authorization: Bearer $TOKEN" "$SERVER/api/v1/namespaces/foo/pods?watch=true"

Or, with kubectl: kubectl get --raw "/api/v1/namespaces/foo/pods?watch=true". Pipe either into any program.

The reconcile loop is a library convention

Most controllers are written in Go, importing the same type definitions Kubernetes uses internally, such as Pod . The standard library for writing them is controller-runtime , which wraps the client-go Kubernetes client. Many other frameworks build on controller-runtime, and there are several semi-official ways to write controllers. Its contract is:

Tell me what to watch and give me a function to run. I’ll run it when something happens to that thing, and also every few minutes regardless.

The generally accepted pattern for that function is to compare the resource’s spec (the desired state) with its status (the observed state) and fix any difference. A controller that does this is said to reconcile the resource, and doing it repeatedly is the reconcile loop (also called the control loop).

The loop is not enforced by Kubernetes. It depends on a stack of optional assumptions:

• the controller is written in Go, or in a language with equivalent libraries
• it uses controller-runtime and client-go, or libraries with similar contracts
• something else keeps the resource’s status up to date (updating status inside the function is discouraged)
• the kind has a spec and a status at all

Many controllers follow the pattern, but others do work that doesn’t fit it. A controller can be written in any language, ignore these libraries and do anything it likes.

Operators are controllers

The official documentation describes operators as software extensions that use custom resources to manage applications and follow Kubernetes principles, notably the control loop. According to the article, projects that call themselves operators turn out to be collections of controllers. The external-secrets project is an example: its code sits in a controllers directory, in files ending in _controller.go, which import controller-runtime and follow the reconcile pattern voluntarily.

controller-runtime also provides a manager type , ctrl.Manager, which runs groups of controllers inside one process. The 20-plus cert-manager controllers all run in a single pod under such a manager. cert-manager is not branded an operator, while external-secrets is, though some people call both operators. The “operator” label is a naming choice and not a distinct technical mechanism.

The whole system in one pattern

Everything in Kubernetes reduces to controllers listening to API endpoints (resources) and doing whatever they are written to do. The pod controller watches pods, and the deployment controller watches deployments. New kinds of things are added by extending the API with a CRD and writing a controller for them. The complexity beyond that is specific to individual controllers, so learning Kubernetes means learning it one controller at a time.

Next: the kubelet

The kubelet is one of the most complex controllers. It watches v1/pods and reconciles workload scheduling, meaning it runs containers on hardware. This creates a bootstrapping problem: kube-apiserver can’t run without a kubelet, and the kubelet can’t do anything without kube-apiserver. A workaround called static pods solves it. Understanding the kubelet also explains pods (Kubernetes’s process model), nodes (its hardware abstraction) and volumes (its storage abstraction), which are the foundation for workloads and the controllers that manage them.