Skip to main content
The deployment coordinator is the platform’s own deployment authority. GitHub Actions keeps building and publishing; the coordinator decides which build may publish when, records what happened, and gives the Infrastructure dashboard one place to see, promote and roll back deployments. It is the tuturuuu-deploy-coordinator Worker in apps/deploy-coordinator. The Worker holds no Vercel or Cloudflare deploy credentials and never calls a provider or dispatches a workflow. Executors stay in GitHub Actions; the coordinator issues them fenced grants.

Delivery status

Model

  • Projects come from the committed catalog in apps/deploy-coordinator/src/projects.ts. Each entry names the provider (vercel or cloudflare) and, per environment, the workflow files whose jobs may report for it. An unknown project or environment is a 404; adding one is a reviewed code change. The catalog bounds how many Durable Objects can exist. Only workflows that publish the environment are listed: Chat, CMS, Drive, Forms, Teach, Tools and Track publish production natively on Cloudflare after their Vercel production workflows retired (see retiring an app’s Vercel project). That release does not report to the coordinator yet, so they list no production workflow; their retained vercel-preview-<app>.yaml may still report previews. A job inside a reusable workflow reports that reusable file as its job_workflow_ref and the workflow the run started from as its workflow_ref. A shared publisher is therefore listed with its caller, and both claims must match, so a wave1-native-publish.yaml run started from mail-cloudflare.yaml cannot report for calendar. platform (apps/web) staging is not listed: wave1-web-staging.yaml publishes a dispatched source_sha that can differ from the token’s sha, so its record would name the wrong commit until that workflow can state its source commit. The catalog tests check that every listed file exists, that each production file has a job gated on the production ref in a GitHub Environment, that each staging caller runs the shared publisher with its own app and profile: staging, and that preview workflows are dispatch-only.
  • Environments follow the existing branch flow (see environments and release flow): production deploys run on refs/heads/production, staging on refs/heads/main, and previews are dispatched manually from main with a preview_ref input.
  • Lanes are the unit of serialization. Staging and production have one default lane. Every preview token carries ref refs/heads/main, so the preview lane is the preview_ref the caller states. That value is accepted only from an allowlisted preview workflow on main, which is already the trust boundary for preview dispatch.
  • Deployments carry the plan they were built from. The coordinator stores the CI affected-planner’s plan reference (source, sha256 digest, run URL) and never recomputes affected apps.

Statuses

succeeded, failed, superseded, cancelled and unknown release the lane. unknown still accepts one reconciling completion from the same attempt; that record never becomes the current deployment over a newer epoch.

Serialization rules

Each environment Durable Object keeps its lanes in SQLite. A call reads and writes synchronously without awaiting I/O, so a claim’s read and write never interleave with another request.
  1. One publisher per lane. claim grants the lane lock to one deployment; the oldest claimable queued deployment goes first. Others receive wait with a bounded retryAfterSeconds. A deploy queues only behind older deploys: promotion and rollback markers are resolved by operators rather than claimed by CI, so an unresolved marker never holds deploys for its six-hour queue lifetime. Resolving a marker needs the lane lock to be free.
  2. Newest run wins the queue. Deployments are ordered by GitHub run_id, which increases with run creation, so a re-run of an older run cannot overtake a newer one. A new deploy supersedes queued deploy rows with a lower run id. A deploy whose run id is lower than a queued, publishing or succeeded deploy in the lane is created already superseded. Promotions and rollbacks are never superseded by deploys. A superseded job exits successfully without publishing; this is the durable answer superseded-build skipping can consult.
  3. A publish in progress is never aborted. Supersession, stop, hold and rollback never touch a publishing deployment.
  4. Fenced grants. Every claim increments the lane epoch and issues a random attemptId. Heartbeat and completion must present the current epoch and attempt; anything else is STALE_GRANT. The one exception is the reconciling completion of an unknown attempt described above.
  5. Leases. A lease lasts five minutes, heartbeats extend it, and an attempt may publish for at most 90 minutes. The lock stays held for a two-minute grace after the lease lapses, then the deployment becomes unknown and the lane frees. Executors self-fence: they must not start a provider effect after leaseDeadline, the same rule as the offline adapter’s arm deadline. An unknown effect is never replayed automatically.
  6. Lazy evaluation instead of scheduling. Lease lapses and queue expiry are applied at the start of every call to the object, reads included. The coordinator schedules nothing: no alarms, crons, queues or timers. A lane that nobody calls costs nothing and holds its state until the next call.
  7. Idempotency. create is keyed by run id, run attempt and lane from the verified token; repeating it with the same body returns the same deployment and a different body is a 409. Repeating a completion with the same outcome returns the recorded result; a different outcome is a 409.

Promotion and rollback

  • Promote takes a succeeded staging deployment and queues a production promotion for the same commit. Production builds its own artifacts (it never reuses preview or staging builds), so a promotion releases a verified commit rather than copying an artifact. Preview deployments are not promotable.
  • Rollback takes an earlier succeeded deployment in staging or production, queues a rollback carrying that deployment’s commit and artifact identity, supersedes queued deploys and places the lane on hold. While held, deploys still register but claim answers held and the job records a skip; promotions and rollbacks can still claim. Releasing the hold, or a successful promotion, resumes automatic deploys, so a rollback is not immediately overwritten by the next merge. An operator cancelling the rollback that placed the hold releases it; a rollback that fails or expires keeps the hold until an operator releases it, because the lane may still be serving the version that prompted the rollback. Skipped deploys are not replayed: after releasing a hold, an untouched project stays on the rolled-back version until its next deploy or an explicit dispatch.
  • Retries. Repeating a promote or rollback request returns its open or succeeded marker. Once the latest marker for the same request finished without succeeding (failed, cancelled or unknown), the request queues a new marker keyed after it, so a failed promotion or rollback can be retried.
  • Execution. Promotion and rollback rows are markers. The coordinator never dispatches anything. An executor workflow that claims a marker by id, holding provider credentials in its own GitHub Environment, is a separate reviewed change. Until it exists, an operator performs the provider action with the owning release tooling and records the outcome with resolve. Queue admission never authorizes production activation, environment approval or route changes.

API

All responses are JSON with Cache-Control: no-store. Bodies are read as a stream and rejected once they pass 8 KiB, whatever Content-Length says, and every string field has a length bound. Unknown routes return 404. The read endpoints are the stable shape for CLI consumers such as a ttr deploy-queue view.

Authentication

CI sends a GitHub Actions OIDC token. The Worker verifies:
  • RS256 against GitHub’s JWKS, cached per isolate and refetched at most once a minute for an unknown key id;
  • iss, aud (DEPLOY_COORDINATOR_OIDC_AUDIENCE), exp and nbf, and the immutable repository_id (DEPLOY_COORDINATOR_REPOSITORY_ID);
  • event_name is push, workflow_dispatch or schedule; pull request events, including pull_request_target, are rejected, and fork pull requests never receive tokens;
  • job_workflow_ref names a workflow file the catalog allows for the project, at the same protected ref as ref: refs/heads/production for production, refs/heads/main for staging and preview. For a shared reusable publisher, workflow_ref must also name the project’s calling workflow at that ref.
For staging and production the commit SHA comes from the token’s sha claim. The logs URL is built from the token’s repository, run_id and run_attempt, never from the body. Claim, heartbeat, completion and skip must come from the run that created the deployment. Ownership is checked by run_id without run_attempt on purpose: a re-run attempt registers its own deployment, because the create key carries the attempt, yet can still skip or close one its earlier attempt left open. Publishing rights stay fenced by the grant’s attemptId and epoch, which another attempt never holds. Operators are the Infrastructure app’s server routes. They send DEPLOY_COORDINATOR_OPERATOR_TOKEN, compared in constant time, and an X-Deploy-Actor user id. The Worker records that id for audit and trusts it only because the token holder has already authorized the user. The Worker has no public route until release authority approves a hostname; workersDev and preview URLs stay disabled.

GitHub Actions integration

Composite actions have no post step, so the integration is .github/actions/deploy-coordinator (running scripts/ci/deploy-coordinator-client.mjs) used up to three times in a deploy job:
  1. phase: begin, before the publish steps: registers the deployment, waits for a grant, and starts a bounded background heartbeat for the job.
  2. phase: check, immediately before each provider effect: in enforce, fails the step when the heartbeat was refused, so a fenced job never starts an effect after its lease (the self-fence in rule 5).
  3. phase: finish, with if: always(): stops the heartbeat and reports succeeded or failed from the job-status input (job.status).
When the coordinator answers superseded or held, begin records a skip and sets publish=false; later steps check that output. Composite actions cannot read repository variables, so each step passes mode and origin from vars.DEPLOY_COORDINATOR_MODE and vars.DEPLOY_COORDINATOR_ORIGIN, and is itself skipped unless the mode is observe or enforce. The job needs id-token: write; when the deploy workflow runs as a reusable workflow, its caller job must grant it too (the production planner grants it to deploy-colab-cloudflare only). The pilot is the Colab production deploy job in colab-cloudflare.yaml:
The repository variable DEPLOY_COORDINATOR_MODE selects: In enforce, an unreachable coordinator, an exhausted budget or the bounded wait running out still falls open so an outage cannot block a release; GitHub concurrency groups keep their serialization. The consequence is explicit: a rollback hold is advisory to CI while the coordinator is unreachable, and the environment stop fence is advisory to CI both then and once the bounded wait runs out on a reachable stopped answer. A job that publishes without a grant first skips its deployment with PUBLISHED_UNCOORDINATED, so the dashboard never shows it queued. Making the coordinator a required authority is a separate reviewed change.

Dashboard

The Infrastructure app’s Deployments page (/[wsId]/deployments, under Delivery) lists catalog projects, and for the selected project each environment’s lanes, current deployment, queue, statuses and GitHub run links. Viewing requires view_infrastructure. Promote, rollback, cancel, resolve, hold release and stop/resume go through Infrastructure API routes that require the root-workspace operator permission (manage_workspace_roles, via authorizeInfrastructureOperator); the page hides those controls from viewers. Without DEPLOY_COORDINATOR_ORIGIN and DEPLOY_COORDINATOR_OPERATOR_TOKEN the page shows an unconfigured state and the routes make no outbound call. The server client (apps/infrastructure/src/lib/infrastructure/deploy-coordinator.ts) sends the operator token and the signed-in user id as X-Deploy-Actor, and maps records to the dashboard shapes in @tuturuuu/internal-api/infrastructure without grant attempt ids or idempotency material. Environment reads are exempt from the coordinator’s daily budget, so the page reads on demand (open, focus, refresh and after an action) and never polls.

Cost and stop-fence analysis

The coordinator adds Worker requests, Durable Object requests and SQLite row operations. It adds no alarms, queues, crons, WebSockets, outbound provider calls or retries of its own. Its objects do no I/O beyond synchronous SQLite, so they are eligible for hibernation whenever idle and accrue duration only while handling a call (pricing). A busy production environment of 20 deployments a day at the worst case uses about 3,600 calls, inside its budget. The hard ceiling, with every catalog environment pinned at its budget every day:
  • 33 projects × 3 environments × 5,000 calls ≈ 495,000 calls a day, about 14.9 million a month.
  • Worker requests above the 10 million included: about $1.46 a month.
  • Durable Object requests above the 1 million included: about $2.08.
  • SQLite rows written. Cloudflare bills each row a write touches in a table and in every index on it. The schema weighs a deployment write at 5 rows (row, primary key, idempotency key, lane order and status indexes), a lane or budget write at 2 and the sequence counter at 1. Pinned per call by src/cost.test.ts: A call can also apply transitions that belong to earlier calls: lazy lease lapse and queue expiry, supersession, pruning and lane eviction. Each deployment row is terminated and deleted at most once, so those writes are charged to the create that inserted it: at most 25 rows a create (11, its terminal transition 7, its deletion 5, its lane 2), and no other call exceeds 9 once its share is counted. The first counted call of each UTC day also replaces the budget row, 2 more rows once a day per environment.
  • Hard ceiling, every call a create: 495,000 × 25 ≈ 12.4 million rows a day, about 371 million a month against 50 million included: about $321.
  • Realistic ceiling, every environment at budget running the CI client’s worst-case lifecycle (create, 90 wait polls, a grant, 90 heartbeats, completion), which the cost test pins below 5 rows a call: about 74 million a month, about $24.
  • Rows read stay far below the 25 billion included; duration is the millisecond-scale handling time of each call, far below 400,000 GB-s.
The hard ceiling is about $325 a month above the plan, about $28 for the CI client’s own worst case, and both scale linearly with the catalog size. A CI create is idempotent per run attempt and lane, so reaching the hard ceiling needs a distinct GitHub run, or a distinct preview lane, for every call. The catalog test pins the project count and the cost test pins the rows per call, so growing either is a reviewed change to these figures. Requests rejected before reaching an object (bad token, unknown route) cost Worker requests only and are not bounded by the coordinator; that ingress limit belongs to the hostname’s WAF rules when one is assigned. Stop fences, outermost first:
  1. Global switch. DEPLOY_COORDINATOR_ENABLED set to anything other than true makes every CI call return 503 COORDINATOR_OFF before any Durable Object call.
  2. Environment stop. POST …/stop durably sets stopped and increments stopRevision. Claims answer stopped; queued deployments stay queued and publishing ones may still complete. The fence lives in Durable Object storage and survives restarts and redeploys.
  3. Daily budget. The durable per-environment counter caps calls, and with them rows written, even if a client loops. Rejected calls that still reach an object are billed requests; the global switch is the hard cap on call count.
  4. Client. The steps’ wait, heartbeat and retry ceilings; setting DEPLOY_COORDINATOR_MODE to off stops all CI traffic without a code change.
A rate limit or billing alert alone is not a spend cap; the durable budget and the catalog-bounded object count are.

Relationship to existing pieces

  • Offline queue adapter (scripts/ci/release-cf-queue*.mjs): grants use the adapter’s fields (jobId, attemptId, workerId, epoch, targetLockId, stopRevision, leaseDeadline). Statuses map as queued → queued, publishing → running, succeeded, failed and unknown unchanged, and superseded or cancelled → stopped. The adapter’s submit/renew/requestStop port and per-phase receipts have no binding to the coordinator yet; that binding would be a separate adapter.
  • Production planner (.github/workflows/vercel-production.yaml) and per-workflow concurrency groups remain the trigger and first serialization layer. The coordinator adds cross-run supersession, a lane lock and history.
  • CI affected-planner: each deployment stores its plan digest and run URL.
  • Superseded-build skipping: a superseded claim answer is a durable, cross-run reason to skip a build.
  • Cron Control and Devbox Control: the Worker follows their layout (cf config, Vite+ build, SQLite Durable Object, cf-vite-worker.mjs runtime harness).

Verification

Run from the repository root:
  • bun --cwd apps/deploy-coordinator test runs the state machine suite twice, against the in-memory store and against real SQLite (node:sqlite) through the same exec contract as the Durable Object. It covers supersession by run order, lock exclusivity, epochs, lease lapse and reconciliation, idempotent create and completion, hold, marker retries, stop, budget, capacity, queue expiry, lane eviction and retention. The cost suite meters index-weighted rows written per call against SQLite and pins the figures above. The router suite drives the HTTP API with OIDC tokens signed by a generated RSA key; the OIDC suite covers claim, event, workflow and ref checks and the rate-limited key refetch.
  • node --test scripts/ci/deploy-coordinator-client.test.mjs covers the client’s phases, fall-open paths, bounded waits and retries.
  • node --test apps/deploy-coordinator/scripts/verify-worker-runtime.mjs runs the real Worker and Durable Object in workerd through the Vite runtime: the private config, closed routes, and a seeded lane lock, stop fence and exhausted budget surviving restart(). It sends no token that could reach GitHub’s JWKS. The harness warms the first Durable Object route and reads once after a restart before using the Local Explorer, which can reload the runtime on a cold path.
  • Source for the operating limits is LIMITS in apps/deploy-coordinator/src/core.ts; change this page with it.