Auditing our own agent runner

# summary

> A Claude Code agent orchestrator we built and operate — Fastify, BullMQ, Postgres, Redis, live output streaming — audited by us, with its command-execution surface described as it actually is and the parts we closed kept separate from the parts we did not.

# the problem

> Most web applications hold data. This one starts processes: it spawns a Claude Code CLI in a chosen working directory, streams the output to a browser as it arrives, and chains runs into pipelines, schedules, webhooks and monitors. That changes which questions matter — who may execute what, and what a process can reach once it has started, are the design, and the dashboard is the easy part. It had also drifted: the deployed service was running code that had never been committed, because it had been maintained by editing files on the server.

# what we built

> Next.js dashboard, Fastify API, a BullMQ worker, PostgreSQL through Drizzle, Redis for both the queue and pub/sub. POST /api/runs writes a run row and enqueues {runId, prompt, model, workdir}; the worker spawns the CLI there, reads --output-format stream-json line by line, writes each line into run_events and publishes it on run:<id>. GET /api/runs/:id/stream replays the persisted events first and only then subscribes to the Redis channel — so a page opened halfway through a run shows the whole run rather than the tail, and a run that has already finished gets a done event and a closed stream instead of a socket held open for nothing.

> A separate websocket gateway carries multi-agent orchestration: we moved its authentication into a Fastify preValidation hook, so an upgrade without a valid session_id cookie is refused as a 401 before the 101 is written and no socket, client record or heartbeat is ever allocated; added per-user (10) and global (500) connection caps and a 60-message-per-10-second budget checked before JSON.parse; and closed a reconnect token that used to be sufficient on its own and carried the admin flag with it, so a token minted for one user could be replayed to inherit that user's identity. Then we audited the whole thing ourselves, shipped the fixes we could verify without changing what the product does, and wrote the rest down with the reasons.

# technical decisions

• The endpoint everyone audits is not the shortest path to execution.

  POST /api/terminal/exec is admin-gated and is the one people look at. POST /api/runs is not — any session that owns the agent can start a run — and POST /api/workspaces accepts an arbitrary workdir string validated only by fs.existsSync, so a user can point a workspace at a home directory or at / and run there. Both are still true today, and both are in the README under known limitations together with the fix we did not make: validate workdir against an allowlist of roots, resolve symlinks, reject anything that escapes after normalisation, and gate /api/runs on admin until that validation exists.

  We left it because it changes the workspace-creation contract the live UI depends on, and breaking the product to secure it is the owner's decision rather than ours. Underneath both endpoints the queue job is {runId, prompt, model, workdir} and the worker's call was spawn('claude', ['-p', prompt], { cwd: workdir, env: { ...process.env } }): the job fields map one for one onto the argument, the working directory and the whole inherited environment.

  Redis had no password, so anything able to open the local Redis port drove the executor without holding a session, without passing authentication and without touching the HTTP surface at all. It requires one now — an unauthenticated PING on this host returns NOAUTH — which moves that path from an open socket to a credential read out of a mode-600 .env.

  Worth stating precisely rather than as a win: every application on this box runs as the same OS user, so that file is readable by all of them.

• An accident that stops execution is not a decision that stops execution.

  At the time of the audit nothing was actually reaching a process, and both reasons were packaging bugs rather than controls. claude lives in ~/.local/bin, which was absent from the worker's pm2 PATH — 122 claude binary not found lines in the worker error log — and the orchestration path built npx tsx <basePath>/<name>.ts from path.resolve(__dirname, …) inside a package declaring "type": "module", so __dirname threw and the failure was caught and logged.

  ▶▼ read the full reasoning

  One line in ecosystem.config.cjs, or a worker restarted from an interactive shell with a fuller PATH, re-armed host-level command execution with no code change and no review. So the change was to turn the accident into a setting. apps/worker/src/exec-policy.ts will not spawn an agent unless ONBERLAB_EXECUTOR=enabled, and then only the binary named by an absolute CLAUDE_BIN; PATH is never consulted for it, so editing PATH cannot arm anything.

  The same gate covers MCP servers, whose name — which arrived off the Redis orchestration:commands channel and was concatenated straight into an executed path — is now matched against a pattern, required to resolve to a file sitting directly in packages/mcp-servers/src, and run with process.execPath --import tsx instead of npx, which can fetch and run a package from the network.

  And buildChildEnv() replaces { ...process.env } with a named allowlist — PATH, HOME, locale, proxy and CA variables, plus the model-provider keys for agent binaries only — with a tripwire that throws rather than spawning if DATABASE_URL, REDIS_URL, SESSION_SECRET or GITHUB_CLIENT_SECRET ever reaches a child by inheritance. disabled is the live setting, and it is visible in the data rather than only in config: since the gate went in, every run that asked the worker to spawn an agent has come back failed, carrying the refusal verbatim as its reason.

  Not a sample of them — all of them. None of this is a sandbox and none of it should be read as one. The worker still runs as the owner's own account with no filesystem restriction, the admin terminal endpoint still hands { ...process.env } to /bin/sh, and pg_hba.conf still authenticates every local connection with trust, so a container around the worker would still be holding a superuser Postgres socket.

  Real containment is a per-run container with exactly one workspace mounted — roughly one function's worth of change in spawnProvider, and worth close to nothing before Postgres authentication is fixed. Producing that ordering, and refusing to describe the worker as sandboxed, was the actual output of the audit; disarming the executor was the part of it that could ship on its own.

• A denylist over a string a shell will re-parse cannot be made sound.

  POST /api/terminal/exec runs a shell command on the host. It is gated on an admin flag requiring both plan === 'admin' and membership of a one-address email allowlist, then filtered through seven "dangerous" regexes. The filter reads like a control and is not one. Two patterns cannot match on this host at all — mkfs is a Linux utility and > /dev/sd a Linux block device, and the host is macOS.

  The fork-bomb pattern has an unescaped |, so the engine reads it as an alternation: it will not match a fork bomb, and it will match innocent strings that happen to end that way. And the filter inspects a JavaScript string that execSync then hands to /bin/sh, which expands it — a command assembled from a shell variable walks past every pattern, and the rm pattern only spells -rf, so rm -fr was never covered.

  None of this is fixable by adding patterns. The honest options are to drop the filter and rely on the admin gate openly, or to stop using a shell: an argv array, no shell: true, an allowlist of commands. We put that in the README under known limitations instead of shipping more regexes, because a filter that looks like protection is worse than an absent one — it moves the endpoint out of the category of things anyone re-examines. It is unchanged today, deliberately.

• We shipped a query that is parameterised in shape and interpolated in fact, and said so.

  apps/api/src/routes/reports.ts builds $1, $2, … placeholders and collects the values into a params array — and then, before executing, substitutes them straight back out again:

  await db.execute(sql.raw(query.replace(/\$(\d+)/g, (_, n) => {
    const val = params[parseInt(n) - 1];
    return typeof val === 'string' ? `'${val.replace(/'/g, "''")}'` : String(val);
  })));

  The comment above it reads "Parameterized query building". It is string interpolation with quote-doubling. We committed it anyway, because the code it replaced interpolated '${workspaceId}' with no escaping at all: reverting to a clean tree would have put the injection back, and rewriting the route properly was a change we were not otherwise making inside a commit already carrying the whole production drift.

  So it went in with the finding written down — make it a real driver-placeholder query, and delete the comment that claims it already is. The comment is the part that does the damage. Code that is visibly wrong gets caught by the next person who reads it; code labelled correct does not get read.

• A run's final state cannot be written by the process whose death you are recording.

  The run table carries a block of rows that still say running with no process behind any of them — one row for each day of an unbroken stretch that ends the day the executor was disarmed, every one of them with a null end time. The block is closed: nothing has been added to it since, and nothing can be.

  The worker is not careless about failure — a missing binary, a spawn error, a refusal from the executor gate and a non-zero exit each write a terminal status and publish a done event, and every failure added since the audit is a run the disarmed executor turned down. That is also why the stalled block stopped growing: a refusal is an ending the worker can see, so a gated run dies as failed instead of being abandoned mid-flight.

  The gap is the one class it cannot observe: the worker dying mid-run. The two counts that can still be taken agree — 61 jobs in BullMQ's failed set, 61 rows stuck at running — and the third that matched them, the job stalled more than allowable limit lines, was in a worker error log that has since rotated, which is its own small lesson about a number whose only evidence is a log file.

  Every attempt writes running at the top of processJob and only an observable ending writes anything else, so a stalled attempt leaves the row asserting a process that is gone, and nothing outside a run's own lifecycle ever revisits it. That places the fix outside the run: on worker boot, mark any row still running with no live job as failed with a stated reason, plus a stalled-job handler that writes the failure back.

  We specified it and did not ship it alongside the rest, because altering worker startup behaviour deserves its own verification — but a system that reports a state it is not in is the same defect as a swallowed exception, one layer down, and the numbers belong in writing either way.

# stack

├─ Next.js 16 (App Router), React 19, TypeScript, Tailwind CSS v4
├─ Fastify 5 with @fastify/websocket, @fastify/cookie, @fastify/cors
├─ PostgreSQL via Drizzle ORM, migrations generated with drizzle-kit
├─ Redis: BullMQ queue plus pub/sub for run output and orchestration,
│  password-authenticated, credential read from .env not the pm2 config
├─ TypeScript executed directly by Node (--import tsx under pm2) — no
│  build step for API or worker, so the checkout is the running code
├─ Server-Sent Events for run output, websockets for orchestration
├─ A zod discriminated union as the websocket wire protocol: nine client
│  message types — auth, ping and seven orchestration verbs
├─ An explicit child-process environment allowlist in the worker, in place
│  of { ...process.env }, with a tripwire on four never-inherited secrets
├─ GitHub OAuth with server-stored sessions, invite-only allowlist
├─ Security headers set by the API process itself — HSTS, nosniff,
│  Referrer-Policy: no-referrer and an enforcing default-src 'none' CSP
└─ pm2 on a Mac mini we own, published through a Cloudflare Tunnel

> None of the nine websocket message types names a stream and asks to be joined to it. A socket is subscribed to a Redis channel only inside the orchestration:start handler, on the channel of a row that same call just created for that user, and every other verb resolves its orchestrationId to a workspace row and compares workspaces.userId against the session before doing anything. The API returns only JSON, and no reverse proxy sits in front that could add the security headers.

# outcome

> Live at lab.onbers.com with the API on api.onbers.com. Re-checked during the independent review of this write-up: the dashboard returns 200, /api/runs returns 401 without a session, and /health returns:

{"status":"ok","db":"connected","redis":"connected","version":"0.1.0"}

> The API's own headers — HSTS, nosniff, Referrer-Policy: no-referrer, and Content-Security-Policy: default-src 'none'; frame-ancestors 'none'; base-uri 'none'; form-action 'none' — are present on the public response.

> The code that was running in production but absent from git is committed: one production-sync commit of 55 files and 8,775 insertions, with two QA-script deletions kept in a separate commit so either can be reverted on its own. Because the API and the worker run TypeScript straight from the checkout under tsx, those commits and the five security commits after them are what is serving.

> For a time they were also the only copy of it: the branch sat 11 commits ahead of origin/main and unpushed, so the host was the one place the work existed, and a deploy from the remote trunk would have silently undone every fix. That is the condition the first commit was written to end, and it is now ended. Local main and origin/main are the same commit, and both carry the hardening — the push was a fast-forward, checked afterwards by cloning the remote fresh and reading the loopback bind and the executor gate out of the clone.

> A deploy from either trunk now preserves the fixes. The rule is in the README — production deploys from git, and a dirty git status on the production checkout is an incident rather than a normal state. The checkout is dirty today. Two stray pnpm-lock.yaml files sit in an npm-workspaces repo whose tracked lockfile is package-lock.json; committing either would enshrine two package managers, and deleting them is a decision about intent that belongs to the owner, so they are left untracked and named rather than quietly resolved.

> Two limitations this write-up originally recorded as live have since been closed, and the check is the same one anybody else would run: the API bound all interfaces and answered on the LAN with Cloudflare out of the path — its listening socket is now 127.0.0.1:4444 and a request to the host's LAN address is refused — and Redis had no password and now answers an unauthenticated client with NOAUTH. The residual on the second stays visible: the credential sits in a mode-600 .env, and every application on this host runs as the same OS user, so the cross-application route to the queue costs a file read rather than an open socket.

> What is still open sits in that README with the reasoning and the recommended fix. POST /api/runs is not admin-gated and workdir is validated only by fs.existsSync. auth.ts still accepts a session id as a ?token= query parameter that no frontend uses, which writes a live credential into every access log and Referer header. The terminal denylist is unchanged, deliberately. And PostgreSQL still authenticates local connections with trust, which is not onberlab's file to change but is the reason the container ranks last on our own list rather than first.

> The run table still reports a closed block of rows as running with nothing behind them — 61 of them when the table was last counted, on 2026-08-24; the reconciliation is specified and not written. It is recorded here because a case study that lists only what was fixed is not a description of a running system.