What is a kubelet?

• kubelet
• controllers
• nodes
• pods

The kubelet is a program that runs on every node. It watches kube-apiserver for Pods assigned to its node, makes the container runtime start and stop containers to match, and writes what it observed back to the Pod’s status. It is the controller in the overview that does the work of running containers on hardware.

It is a client of kube-apiserver

The kubelet has no special channel to the control plane. It authenticates to kube-apiserver and uses the same HTTP API as any other controller, built on client-go. It does not use controller-runtime.

To find its Pods, it opens a list-and-watch on pods, filtered by a field selector on its own node name. The call is in NewSourceApiserver :

cache.NewListWatchFromClient(c.CoreV1().RESTClient(), "pods", metav1.NamespaceAll,
    fields.OneTermEqualSelector("spec.nodeName", string(nodeName)))

On the wire, that is the watch from the overview with a filter added:

GET /api/v1/pods?fieldSelector=spec.nodeName%3Dnode-3&watch=true

You can run the same query yourself:

kubectl get --raw "/api/v1/pods?fieldSelector=spec.nodeName=node-3&watch=true"

Something else must set spec.nodeName first. That is the scheduler’s job: it binds a Pod to a node by writing that field. Until it does, no kubelet sees the Pod.

Pods arrive from three sources

kube-apiserver is only one source. makePodSourceConfig merges up to three:

Pods from the file and HTTP sources are static pods. The kubelet runs them without any instruction from kube-apiserver, which solves the bootstrapping problem from the overview: the control plane itself can run as static pods, so the kubelet can start kube-apiserver before kube-apiserver exists.

The kubelet then creates a read-only mirror pod in the API for each static pod, using mirrorPodClient . You see the static pod in kubectl get pods, but deleting the mirror does not stop it. The manifest directory is the source of truth.

The file source only starts when StaticPodPath is set , so it has no default path. /etc/kubernetes/manifests is the convention that kubeadm sets, so on other distributions the directory differs.

The sync loop

Every source feeds one channel, and one function consumes it. syncLoopIteration is a select over these inputs:

• configCh: pod additions, updates, removals and deletions from the sources above.
• plegCh: events from the pod lifecycle event generator , which polls the container runtime and reports containers that started or died.
• syncCh: a timer that re-syncs pods waiting for work. SyncFrequency defaults to one minute .
• housekeepingCh: a timer that triggers cleanup of pods that no longer exist.
• the health manager: pods whose containers failed a liveness, readiness or startup probe.

The kubelet does not do the work inside this loop. HandlePodAdditions and its siblings admit the Pod, then hand it to podWorkers.UpdatePod , which runs one goroutine per Pod. That goroutine calls Kubelet.SyncPod .

This is the reconcile pattern from the overview, built without controller-runtime. Events and timers both trigger a sync, and each sync compares the Pod’s spec to what is running. The kubelet’s source comments note that select picks among ready channels in no particular order, so the code cannot assume which input comes first.

Running containers through the CRI

The kubelet does not start containers itself. It asks a container runtime such as containerd or CRI-O, over the Container Runtime Interface (CRI). CRI is a gRPC API defined in api.proto , with a RuntimeService (RunPodSandbox, CreateContainer, StartContainer and others) and an ImageService.

kubeGenericRuntimeManager.SyncPod is where the kubelet’s desired state turns into CRI calls. Its doc comment lists the steps:

1. Compute sandbox and container changes.
2. Kill the pod sandbox if necessary.
3. Kill any containers that should not be running.
4. Create the sandbox if necessary.
5. Notify the kubelet to update the pod status.
6. Resize the pod and running containers.
7. Create ephemeral containers.
8. Create init containers.
9. Create normal containers.

The sandbox is the shared environment for a Pod’s containers, which is why they share a network namespace and an IP address.

Reporting back

The kubelet writes to the API in three places.

Pod status. The status manager sends a PATCH to the Pod’s status subresource through PatchPodStatus . That write is what produces the MODIFIED events you saw in the overview’s watch example, such as Running with a pod IP, or CrashLoopBackOff with a restart count.

Node status. On startup the kubelet registers its own Node object with registerWithAPIServer . syncNodeStatus then updates it on a timer. NodeStatusUpdateFrequency defaults to 10 seconds .

A lease. Rewriting a large Node object every few seconds is costly, so the kubelet also renews a small Lease in the kube-node-lease namespace. The lease duration defaults to 40 seconds , and the kubelet renews at a quarter of that , every 10 seconds. The node lifecycle controller monitors both the Node status and the lease, and posts NodeReady==ConditionUnknown when neither is updated for its grace period.

What else it does

• Probes. pkg/kubelet/prober runs liveness, readiness and startup checks and feeds failures back into the sync loop.
• Eviction. pkg/kubelet/eviction watches node resources such as memory and disk and evicts Pods when a threshold is crossed.
• Volumes. The volume manager mounts and unmounts volumes for the Pods on the node.
• Resource limits. The container manager (pkg/kubelet/cm ) sets up cgroups for Pods.
• Its own API. The kubelet serves HTTPS, on port 10250 by default . kube-apiserver calls it to proxy logs, exec and port-forwarding, as described in the overview. Clients do not call it directly.

What this page does not cover

The page traces the structure of the kubelet from its source, but it did not run a kubelet. Behavior at the edges, such as admission rejections after a restart or the exact order in which a Pod’s containers start, is in the code and worth reading directly. The pkg/kubelet package is large, and this page reads kubelet.go, config, status and the kuberuntime manager only in outline.

Next

The kubelet is a client of kube-apiserver, and so is every other controller. The next page follows one controller written with controller-runtime, from a watch event to a Reconcile call and back to the API.