Internal CA with Automated Renewal
A step-ca certificate authority issuing 24-hour leaf certificates to four services, where the renewal path, not the CA, is the part that was built and tested.
Four internal services serve TLS from a private CA with password-free renewal every 15 minutes, monitored by eight watchdog signals including hours-to-expiry per certificate.
- Role
- Solo: designed, built and operate
- Timeframe
- Sep 2026 to present
- CA built 19 Sep 2026, renewal bugs fixed 21 Sep 2026
- Team
- 1
- Problem
- Internal services on a .dev domain cannot be reached over plain HTTP at all (the whole TLD is HSTS-preloaded) and a public CA cannot validate names that only resolve internally.
- Approach
- step-ca on its own VM placed away from the hosts it authenticates, a JWK provisioner, a systemd timer running step ca renew (which authenticates with the existing certificate, so no secret sits on disk), and a reload script that restarts a service only when the certificate hash changes.
- Result
- All four services validate without -k. Renewal proven by issuing a 10-minute certificate and watching the serial change. Two self-inflicted renewal bugs found by monitoring and written up below.
Every figure above is measured or read from the running system. Where a figure has a source line, that is how and when it was read.
Technology Stack
Renewal is the whole point
A CA that issues 24-hour certificates without automated renewal is a fleet that breaks every day. The property that makes automation safe is that step ca renew authenticates with the certificate it is renewing: no password, no provisioner secret on disk. I verified it by issuing a 10-minute certificate, renewing it, and watching the serial change with no credential used.
An expired certificate cannot renew itself, so a dead timer means every service loses TLS within 24 hours while the CA reports perfectly healthy. The watchdog therefore checks the timer on each host and the hours remaining on each leaf, which is the signal that actually matters.
Two bugs I caused, and what found them
Fixing an earlier outage, I chowned the certificate directory so the container could read the key. The renewer runs as a different user and rewrites the certificate in place, so every renewal since had failed with permission denied. It was silent for about 17 hours because the certificate was still valid, and it was found with 6 hours 52 minutes to spare. The fix is a shared group with the renewer owning and the container reading; either single-owner answer breaks one side.
The renewer unit also restarted the service after every run. step ca renew exits zero when it decides nothing needs renewing, so an unconditional restart hook bounced the container every 15 minutes: 67 restarts in a day. Most fell between watchdog polls and the rest produced the random blips I had been seeing. Now a script hashes the certificate and restarts only when the hash changes, which I proved three times with an unchanged cert and once with a changed one. The thing I took away is that an exit code tells you a command succeeded, not that anything happened.
Why not the public CA I already pay for
The public zone is on Cloudflare and Universal SSL already covers every public hostname. A public CA can only issue for names it can validate, and every internal service resolves only on the internal DNS and is deliberately unreachable from the internet. Making it work would mean exposing admin interfaces or running DNS validation for names that intentionally do not exist publicly. So the internal names get step-ca and the public ones stay on Cloudflare. They are different problems and I stopped trying to solve them with one tool.
Honest status
The root key is on the same box as the issuing CA, which proper practice keeps offline. There is no CRL or OCSP; revocation relies on 24-hour lifetimes expiring. The CA state directory is in the nightly backup, because losing the root key means re-trusting the CA on every client by hand.