Self-Hosted Forge and CI
Forgejo with branch protection and Kubernetes-hosted runners, where the first green CI run was a false positive and the pipeline was not trusted until it had failed for the right reason.
A git forge with server-side branch protection and six parallel CI slots on the dev cluster, validated red and green and watched by three output signals.
- Role
- Solo: designed, built and operate
- Timeframe
- Sep 2026 to present
- forge built 19 Sep 2026, runners moved to Kubernetes 20 Sep 2026
- Team
- 1
- Problem
- Code written by automation needs a place where it is judged before it is merged, and a pipeline that only ever passes proves nothing.
- Approach
- Forgejo on its own VM, a runner on a different physical host at first and then three runner pods on the dev Kubernetes cluster, required status checks on main, and a deliberate green, red, green cycle before trusting any of it.
- Result
- A direct push to main as an admin with a valid token is rejected by the pre-receive hook. The red run dies on a sentinel assertion inside the test file, not on a missing binary. Backups of the forge database and repositories are restore-verified.
Every figure above is measured or read from the running system. Where a figure has a source line, that is how and when it was read.
Technology Stack
The first green run was wrong
The first run with a deliberately failing test failed, which looked like success. The log showed it had never run the test at all: the checkout action is a Node action and the Go image had no node binary. Had I stopped there I would have shipped a pipeline that fails everything regardless of correctness and called it working.
The proof came in two runs on the same pipeline: a correct tree passed, then the same tree with the test broken again failed. The fix was to clone with git directly. The lesson I kept is to read every red run, because a failure for the wrong reason looks identical to a failure for the right one.
Runners on Kubernetes, and four convincing false signals
Execution moved from a single VM to three runner pods on the dev cluster, each with a Docker-in-Docker sidecar and a capacity of two. The dev workers had been allocated 32 GB each while measuring one percent used, so right-sizing them to 12 GB freed 60 GB and made the staging and production clusters fit.
Four things went wrong and each looked like something else. A readiness probe that ran docker info hung forever because the runner image has no docker CLI. The entrypoint shell had no /dev/tcp. A ConfigMap was created and never mounted. And cluster DNS could not resolve the internal zone because the node's stub resolver is unreachable from a pod, which registered two of three runners by racing a warm cache and read exactly like a flaky network.
The pipeline was not judging correctness
Registration is not proof. The green, red, green cycle found that CI reported success on a tree containing assert 2 + 2 == 5, because the only test step ran one hardcoded file and nothing under the tests directory was ever executed. Scoping pytest to the repo root then aborted collection on a module-scope sys.exit, which produced a red run that proved nothing.
Only after both fixes did the red run die on the assertion itself, with the sentinel message in the log. A separate bug had been failing one pull request all along: on a pull_request event the ref name is the PR number, so checkout was cloning branch 3. The same commit showed green on its branch and red on the PR ref, which is indistinguishable from bad code unless you read the log.
Honest status
This is a single-user forge on a home network. What I am claiming is the controls and the tests that exercised them, not scale. The runner has full access to its Docker socket, which is root-equivalent on that VM. I accept that only because the VM holds nothing else, and I would rather say so here than leave it out.