Simple Kubernetes Mutating Admission Webhook

TLDR; Check out this GitHub Repository for a minimal example.

I haven’t found any minimal example for a working Kubernetes admission webhook made with kubebuilder so here is mine.
This example just annotates all created Pods with a nice message.
A lot of code commands and code with comments, as always!


We are going to use Kubebuilder to bootstrap our project.

mkdir pod-webhook
cd pod-webhook
kubebuilder init --domain --repo

I didn’t manage to generate valid configuration files with controller-gen when only using webhooks without writing a controller.
Also I don’t like kustomize, which kubebuilder is using when generating the manifests so let us get rid of all the boilerplate code.
The Makefile won’t make sense either anymore. Dump it and write your own if needed.

rm -rf config hack Makefile

We do not need leader election for a minimal example (you can also remove all kubebuilder comments since we don’t use the generator anyways).

diff --git a/main.go b/new_main.go
index 9052d2a..18780eb 100644
--- a/main.go
+++ b/new_main.go
@@ -46,13 +46,9 @@ func init() {
 func main() {
 	var metricsAddr string
-	var enableLeaderElection bool
 	var probeAddr string
 	flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
 	flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
-	flag.BoolVar(&enableLeaderElection, "leader-elect", false,
-		"Enable leader election for controller manager. "+
-			"Enabling this will ensure there is only one active controller manager.")
 	opts := zap.Options{
 		Development: true,
@@ -66,8 +62,7 @@ func main() {
 		MetricsBindAddress:     metricsAddr,
 		Port:                   9443,
 		HealthProbeBindAddress: probeAddr,
-		LeaderElection:         enableLeaderElection,
-		LeaderElectionID:       "",
+		LeaderElection:         false,
 	if err != nil {
 		setupLog.Error(err, "unable to start manager")

go run . should successfully build and run the project.
Modify the Dockerfile so it respects our new project structure and verify your changes with a docker build -t test ..

diff --git a/old_Dockerfile b/Dockerfile
index 456533d..b53359f 100644
--- a/old_Dockerfile
+++ b/Dockerfile
@@ -10,9 +10,7 @@ COPY go.sum go.sum
 RUN go mod download
 # Copy the go source
-COPY main.go main.go
-COPY api/ api/
-COPY controllers/ controllers/
+COPY *.go .
 # Build
 RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go

Implement the Webhook

The Kubebuilder Book references the following example on GitHub.
We are going to strip these files and integrate them into our bootstraped kubebuilder project.

Create a file called webhook.go:

package main

import (

	corev1 ""

type podAnnotator struct {
	Client  client.Client
	decoder *admission.Decoder

func (a *podAnnotator) Handle(ctx context.Context, req admission.Request) admission.Response {
	pod := &corev1.Pod{}
	if err := a.decoder.Decode(req, pod); err != nil {
		return admission.Errored(http.StatusBadRequest, err)

	// mutating code start
	pod.Annotations["welcome-message"] = "i mutated you but that is okay"
	// mutating code end

	marshaledPod, err := json.Marshal(pod)
	if err != nil {
		return admission.Errored(http.StatusInternalServerError, err)

	return admission.PatchResponseFromRaw(req.Object.Raw, marshaledPod)

func (a *podAnnotator) InjectDecoder(d *admission.Decoder) error {
	a.decoder = d
	return nil

Add the podAnnotator as a webhook to our manager:

diff --git a/old_main.go b/main.go
index 8db76b2..48544b3 100644
--- a/old_main.go
+++ b/main.go
@@ -12,6 +12,7 @@ import (
 	ctrl ""
+	""
 var (
@@ -48,6 +49,8 @@ func main() {
+	mgr.GetWebhookServer().Register("/mutate-pod", &webhook.Admission{Handler: &podAnnotator{Client: mgr.GetClient()}})
 	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
 		setupLog.Error(err, "unable to set up health check")

If you run the project via go run . now, it says that it is missing certificates. CERTIFICATES?? yes … but wait, you won’t see any certificate here, I promise!
Just make sure that it builds without errors and you should be fine.


Kubernetes is not able to call webhooks which are insecure and not protected via HTTPS.
To handle this, we are going to use cert-manager and let it handle all that nasty stuff.
Refer to this guide for the installation of cert-manager, I recommend using Helm.

First of all, let us create a namespace for all our stuff:

kubectl create namespace pod-greeter

Create a cert-manager Issuer that handles self-signed certificates and a Certificate itself:

kind: Issuer
  name: selfsigned
  namespace: pod-greeter
  selfSigned: {}


kind: Certificate
  name: pod-greeter
  namespace: pod-greeter
  # remember the secretName
  secretName: pod-greeter-tls
    # IMPORTANT: format is the following namespace.service-name.svc
    - pod-greeter.pod-greeter.svc
    name: selfsigned

Create a Service that matches the DNS name format in our Certificate:

apiVersion: v1
kind: Service
  # resolves to pod-greeter.pod-greeter.svc
  name: pod-greeter
  namespace: pod-greeter
    - name: https
      port: 9443
      protocol: TCP
    # this has to match the selector in our Deployment later
    app: pod-greeter

Create a Deployment that matches the selector in our Service.
Also make sure that the secretName matches the one in Certificate.
Cert-manager automatically creates a Secret that contains the generated certificates so we can mount them in our pod.

apiVersion: apps/v1
kind: Deployment
  name: pod-greeter
  namespace: pod-greeter
      app: pod-greeter
  replicas: 1
        # IMPORTANT
        app: pod-greeter
      - name: pod-greeter
        imagePullPolicy: Always
        - name: tls
          # the tls certificates automatically get mounted into the correct path
          mountPath: "/tmp/k8s-webhook-server/serving-certs"
          readOnly: true
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
      terminationGracePeriodSeconds: 10
        - name: tls
            # IMPORTANT: has to match from Certificate
            secretName: pod-greeter-tls
            # the pod only gets created if the secret exists
            # so it waits until the cert-manager is done
            optional: false

As the last step we can finally create our MutatingWebhookConfiguration to tell Kubernetes that it should call the correct endpoint of our controller.
Due to the cert-manager annotation, all certificates are going to be injected in this webhook configuration at runtime by cert-manager.
I told you that you won’t see any certs here!

kind: MutatingWebhookConfiguration
  name: pod-greeter
    # IMPORTANT: has to match Certificate pod-greeter/pod-greeter
- admissionReviewVersions:
  - v1
      # has to match the service we created
      namespace: pod-greeter
      name: pod-greeter
      port: 9443
      path: "/mutate-pod"
  failurePolicy: Fail
  - apiGroups:
    - ""
    - v1
    - CREATE
    - UPDATE
    - pods
  sideEffects: None

You are done! Let us test it out by creating a simple pod:

apiVersion: v1
kind: Pod
  name: nginx
  - name: nginx
    image: nginx:1.14.2
    - containerPort: 80

Fetch it and have a look at the annotations. It should have a welcoming message!

kubectl get pods


So far we haven’t really run and test the controller locally. And I haven’t done that myself either, sadly!
I figured out two possible scenarios to do so:

  • start a minikube or kind locally, deploy the controller, and test it
  • use clientConfig.url in the MutatingWebhookConfiguration with ngrok to proxy your local instance into a remote cluster

The last one is probably the easiest but it requires to run the controller without certificates and so far I haven’t figured out how to do so.
The manager expects a tls certificate whenever you register a webhook.
If you got that figured out, please reach out to me! I would love to update my guide and also use this approach for my projects :)