GitOps and a Signed-Image Gate on Three Clusters
Flux reconciling dev, staging and production from one public repo, with Kyverno refusing any image not signed by a GitHub Actions workflow in my org. Every gate was proven able to fail before it was trusted.
Three RKE2 clusters reconciled by Flux from a single repository, every overlay schema-validated in CI, and a keyless cosign policy enforced at admission on all three, including production.
- Role
- Solo: designed, built and operate
- Timeframe
- Oct 2026 to present
- repo created and all three clusters bootstrapped 8 Oct 2026
- Team
- 1
- Problem
- The clusters were built by hand and operated by kubectl. A certificate SAN that was typed into two control planes incorrectly went unnoticed for nine days. Nothing in the clusters had a source of truth, and nothing checked what was allowed to run.
- Approach
- Import the live workloads into git with a kubectl diff of zero, bootstrap Flux per cluster with read-only deploy keys, chain infra, policies and apps with health checks, vendor Kyverno and write one keyless verifyImages policy, and build a separate CI pipeline that signs an image with GitHub OIDC so the policy has something real to verify.
- Result
- Drift repaired in 3 to 4 seconds on an idle cluster. An unsigned image rejected at admission on dev and on production; a signed one admitted and rewritten to its digest. Eight validator red-run cases and nine watchdog red cases, all failing as required. Two Flux behaviours the design did not predict, found by the tests and written down.
Every figure above is measured or read from the running system. Where a figure has a source line, that is how and when it was read.
Technology Stack
What it is
One public repository holds the desired state for all three clusters. Each cluster has a directory of Flux Kustomizations that chain infra, then policies, then apps, each with prune and wait enabled, so a file deleted from git deletes the object and a Kustomization is only Ready when the rollout finished. Each cluster bootstrapped with its own read-only deploy key; Flux can pull and cannot push.
The first commit imported what was already running. kube-state-metrics was exported from the live dev cluster, stripped of server-set fields, and committed only after kubectl diff reported no difference. From that commit on, Flux owns it, and the ownership labels on the live object say so.
A separate repository builds a tiny image in GitHub Actions, signs it with cosign using the workflow's OIDC token (no key to leak), and attaches an SPDX SBOM as an attestation. The Kyverno policy on the clusters accepts only images whose certificate was issued to a workflow under my GitHub org, recorded in Rekor, and rewrites the tag to the verified digest so the thing that was verified is the thing that runs.
Nothing was trusted until it had failed
The CI validator builds every overlay with kustomize and checks it against the Kubernetes 1.36 schema in strict mode, then enforces policy: no latest tags, production images pinned by digest, prune and wait on every Flux object. On its own that is a script that prints PASS. So the same pipeline runs a harness that breaks a copy of the tree eight different ways, an unknown field, broken YAML, a latest tag, an unpinned production image, prune switched off, wait removed, an em dash in prose, and asserts the validator fails each time. Two of the eight did not fail on the first run. Both were bugs in the validator, and the harness is why they were found before the validator was guarding anything.
The admission policy got the same treatment on dev and then on production: a deliberately unsigned copy of the base image, published under the same registry path, is refused with no signatures found; the signed image is admitted and its pod spec comes back carrying the digest. The watchdog signals for all of this have nine red cases of their own, including the quiet one where every Kustomization is Ready but two of them are on different commits.
Two things the tests found that I did not design for
With infra, policies and apps all reconciling every minute, the drift test failed: a deleted Service was not recreated inside two minutes. Reading the controller log, every infra reconcile flipped it to Unknown for about a second while 76 Kyverno objects were server-side applied, policies then saw its dependency not ready and backed off thirty seconds, and apps behind it waited again. A one-minute interval on a chain of three is not a one-minute repair time. Infra and policies now reconcile every ten minutes and apps every minute.
The second one is a Flux behaviour worth knowing before relying on it. A deletion that lands while Flux is inside a health check is not repaired until that check times out. Flux waits on the inventory it just applied and does not re-apply mid-wait. The log says it plainly: health check failed after 3m0s, Service status NotFound, recreated on the following run. So the worst-case repair time is the health-check timeout plus the interval, not the interval. The drift test now waits for the Kustomization to be idle before each case, and the numbers it reports are the idle-cluster figures.
Honest status
The policy covers two namespaces, not the whole cluster, because the clusters also run things I have not signed yet. Kyverno has deprecated the ClusterPolicy kind I used in favour of its CEL-based ImageValidatingPolicy; migrating is a known follow-up. The RKE2 node configuration, where the certificate bug that started all this actually lived, is still outside git. And this is one workload on a lab, not a fleet of services. What is real is that every claim on this page has a script that reproduces it and a log that recorded it.