Blog

Technical deep dives, project write-ups, and lessons learned from building systems. I write about infrastructure, backend development, and the engineering process.

Series

Latest Posts

Operations7 min read

The Renewal Timer That Ran 40 Times While the Certificate Sat Expired

Grafana's TLS certificate expired on 22 September. The renewal timer fired twice a day for 20 days, every run logged success, and the certificate on disk never changed. The script ended in || exit 0, the timer was too slow for the renewal window, and the watchdog had never heard of the host.

Operations7 min read

The Monitor That Said the VIP Was Fine Without Using It

The production Kubernetes API was unreachable through its virtual IP for 9 days 8 hours 43 minutes. The signal named 'API via VIP' stayed green the whole time, because it ran on a control plane and talked to localhost.

Automation6 min read

The Runbook That Had Never Worked

An automated restart action passed its guardrail suite for weeks. It had never successfully restarted anything, on any host, because ssh does not preserve argument boundaries.

Operations6 min read

The Backup That Restored an Empty File

A nightly Kubernetes backup reported success for weeks. Every restore produced a zero-byte cluster token, because tar archived a symlink instead of following it.

Operations5 min read

A Failover Test That Passed and Proved Nothing

Stopping the API server on the node holding the virtual IP produced a clean pass in zero seconds. The address never moved, which is exactly why the result was worthless.

Kubernetes in Production

1 post

Log Aggregation Deep Dive

1 post

DevOps Best Practices

2 posts

Checks That Lied

5 posts
1Operations6 min read

The Backup That Restored an Empty File

A nightly Kubernetes backup reported success for weeks. Every restore produced a zero-byte cluster token, because tar archived a symlink instead of following it.

2Operations5 min read

A Failover Test That Passed and Proved Nothing

Stopping the API server on the node holding the virtual IP produced a clean pass in zero seconds. The address never moved, which is exactly why the result was worthless.

3Automation6 min read

The Runbook That Had Never Worked

An automated restart action passed its guardrail suite for weeks. It had never successfully restarted anything, on any host, because ssh does not preserve argument boundaries.

4Operations7 min read

The Monitor That Said the VIP Was Fine Without Using It

The production Kubernetes API was unreachable through its virtual IP for 9 days 8 hours 43 minutes. The signal named 'API via VIP' stayed green the whole time, because it ran on a control plane and talked to localhost.

5Operations7 min read

The Renewal Timer That Ran 40 Times While the Certificate Sat Expired

Grafana's TLS certificate expired on 22 September. The renewal timer fired twice a day for 20 days, every run logged success, and the certificate on disk never changed. The script ended in || exit 0, the timer was too slow for the renewal window, and the watchdog had never heard of the host.

Homelab Journey

2 posts
2Networking11 min read

Network Segmentation in the Homelab

Implementing VLANs, firewall rules, and zero-trust principles to secure a home network with multiple trust zones.

Standalone Posts

Written as I build

Every post here is a finished write-up of something I actually ran. New posts go up when the work behind them is done, not before.