Durable Postgres Task Queue
A PostgreSQL queue that runs my long-running agent work, including a job-application pipeline that has submitted 27 applications with a human approving every one: SKIP LOCKED claims, leases with heartbeats, a reaper, approval gates and capped concurrency.
111 tasks across six workload types (job applications, VM operations, iOS and Roblox builds, code changes, briefings) have moved through claim, heartbeat, approval and completion. 130 human-approval gates, 0 submissions without a human decision, and a reaper bug found in production, fixed, and tested in both directions.
- Role
- Solo: designed, built and operate
- Timeframe
- Sep 2026 to present
- earliest task row 12 Sep 2026; migrations dated 14 and 15 Sep 2026
- Team
- 1
- Problem
- My automated workloads run for minutes to hours, die mid-task when a model call fails or a browser tab closes, and sometimes need a human decision before continuing: submit this job application, approve this VM change. A cron job per workload cannot express any of that.
- Approach
- One tasks table in PostgreSQL. Workers claim with SELECT ... FOR UPDATE SKIP LOCKED, hold a lease they extend by heartbeat, and can park a task as needs_approval with a question attached. An hourly reaper requeues tasks whose lease lapsed. Per-capability caps on running and parked tasks are enforced at claim time.
- Result
- Concurrent workers never double-claim. The job-application workload alone has run 69 tasks through drafting, form filling and a mandatory confirm-before-submit gate, resulting in 27 submitted applications and 0 submitted without a human decision. A task that lost its lease before its first heartbeat was invisible to the reaper for two days because NULL < now() is NULL; the fix matches a NULL lease on a stale row, with a test that a stale orphan is reaped and a fresh one is not.
Every figure above is measured or read from the running system. Where a figure has a source line, that is how and when it was read.
Technology Stack
What a claim is
A claim is one transaction: select the oldest approved-or-pending task for a capability with FOR UPDATE SKIP LOCKED, set it running, record the worker and a lease expiry, and return it. Many workers can poll at once and each row is handed out exactly once. Approved tasks (a human already answered) sort ahead of new ones so a resumed task is never starved by fresh work.
Workers heartbeat to extend the lease. A worker can also release a task back to pending with its state intact, so a long job can be done one slice per tick without holding a lease across the gap.
The reaper bug, and testing it both ways
A task sat in running for two days. The reaper looked for status running and lease_expires_at earlier than now. The task's lease was NULL, because the worker died between claiming and its first heartbeat, and in SQL NULL compared to anything is NULL, not true. The task was permanently invisible to the one process meant to rescue it.
The fix adds a second clause: a NULL lease on a row not updated for an hour is treated as expired. The test creates a stale orphan and a fresh one and asserts the first is requeued and the second is left alone. I wrote the second half because a fix tested only in the direction that was broken can quietly break the other direction, and a reaper that eats live tasks is worse than one that misses stuck ones.
Nothing had been calling the reaper at all. It now runs hourly under system cron. The first wrapper captured its output and printed only when non-empty, which made a working reaper look broken; it now always logs its outcome.
A signal that mixed two owners
A queue-depth alert was red in 369 of 816 runs, 45 percent, and flapping. It was not wrong, it was ambiguous: it counted tasks older than six hours in both pending and approved, which merges nothing is claiming work (the system's fault, an incident) with the worker finished and is waiting on a human (my fault, a nudge).
I split it into two signals with two thresholds: more than five tasks unclaimed for six hours is an incident, and more than five approvals ignored for a day is a nudge. A signal that mixes the system is broken with you have not replied yet trains you to ignore it, and I had started to.
What runs on it
Six workload types share the one table. The largest is a job-application pipeline: a nightly task scrapes postings from two boards, filters and ranks them, and enqueues the top 15. A worker tick every 5 minutes claims one, drafts a CV and cover letter tailored to the posting (compiled from LaTeX, 2 pages, checked), fills the application form in a headless browser, and parks the task with a screenshot for a human to approve. Only after that approval does the next tick click submit. 69 tasks have gone through it, 27 were submitted, 8 were skipped on hard requirements, and 12 expired unanswered.
The others are smaller: VM lifecycle operations on the homelab (36 tasks, every one approved before it touched vCenter), iOS and Roblox build agents, code changes, and a daily briefing. Each has its own caps and its own approval rules, but the claim, lease, heartbeat and reaper code is shared.
Approval gates and caps
A worker can park a task as needs_approval with a structured request attached: the filled application form and a screenshot, or the exact VM change about to be made. Resolving it records who approved and when, and the task becomes claimable again with its state preserved. Parked tasks expire after 48 hours if nobody answers, and a count of expired tasks (12 so far) is itself a signal worth reading.
Caps are enforced at claim time: a per-capability maximum on running tasks (2 for job applications), a maximum on parked tasks (8) that blocks new claims but never blocks resuming an approved one, and a daily ceiling on new work (15). So a workload that is waiting on a human cannot keep pulling new work until the backlog of questions is answered.
Honest status
This serves my own fleet of automated workloads, not a multi-tenant product. The numbers are small on purpose and are read from the database on the date shown. The job-application workload drafts a tailored CV and cover letter per posting and fills the form, but a human reads every one and clicks submit; nothing in the pipeline can submit on its own. There is no dead-letter table yet; failed tasks stay failed and are listed, which is adequate at this volume and would not be at a larger one.