tuturuuu-deploy-coordinator Worker in apps/deploy-coordinator.
The Worker holds no Vercel or Cloudflare deploy credentials and never calls a
provider or dispatches a workflow. Executors stay in GitHub Actions; the
coordinator issues them fenced grants.
Delivery status
Model
-
Projects come from the committed catalog in
apps/deploy-coordinator/src/projects.ts. Each entry names the provider (vercelorcloudflare) and, per environment, the workflow files whose jobs may report for it. An unknown project or environment is a404; adding one is a reviewed code change. The catalog bounds how many Durable Objects can exist. Only workflows that publish the environment are listed:Chat, CMS, Drive, Forms, Teach, Tools and Track publish production natively on Cloudflare after their Vercel production workflows retired (see retiring an app’s Vercel project). That release does not report to the coordinator yet, so they list no production workflow; their retainedvercel-preview-<app>.yamlmay still report previews. A job inside a reusable workflow reports that reusable file as itsjob_workflow_refand the workflow the run started from as itsworkflow_ref. A shared publisher is therefore listed with its caller, and both claims must match, so awave1-native-publish.yamlrun started frommail-cloudflare.yamlcannot report forcalendar.platform(apps/web) staging is not listed:wave1-web-staging.yamlpublishes a dispatchedsource_shathat can differ from the token’ssha, so its record would name the wrong commit until that workflow can state its source commit. The catalog tests check that every listed file exists, that each production file has a job gated on the production ref in a GitHub Environment, that each staging caller runs the shared publisher with its ownappandprofile: staging, and that preview workflows are dispatch-only. -
Environments follow the existing branch flow (see
environments and release flow):
production deploys run on
refs/heads/production, staging onrefs/heads/main, and previews are dispatched manually frommainwith apreview_refinput. -
Lanes are the unit of serialization. Staging and production have one
defaultlane. Every preview token carriesrefrefs/heads/main, so the preview lane is thepreview_refthe caller states. That value is accepted only from an allowlisted preview workflow onmain, which is already the trust boundary for preview dispatch. -
Deployments carry the plan they were built from. The coordinator stores
the CI affected-planner’s plan reference (
source,sha256digest, run URL) and never recomputes affected apps.
Statuses
succeeded, failed, superseded, cancelled and unknown release the
lane. unknown still accepts one reconciling completion from the same
attempt; that record never becomes the current deployment over a newer epoch.
Serialization rules
Each environment Durable Object keeps its lanes in SQLite. A call reads and writes synchronously without awaiting I/O, so a claim’s read and write never interleave with another request.- One publisher per lane.
claimgrants the lane lock to one deployment; the oldest claimable queued deployment goes first. Others receivewaitwith a boundedretryAfterSeconds. A deploy queues only behind older deploys: promotion and rollback markers are resolved by operators rather than claimed by CI, so an unresolved marker never holds deploys for its six-hour queue lifetime. Resolving a marker needs the lane lock to be free. - Newest run wins the queue. Deployments are ordered by GitHub
run_id, which increases with run creation, so a re-run of an older run cannot overtake a newer one. A newdeploysupersedes queueddeployrows with a lower run id. Adeploywhose run id is lower than a queued, publishing or succeededdeployin the lane is created alreadysuperseded. Promotions and rollbacks are never superseded by deploys. A superseded job exits successfully without publishing; this is the durable answer superseded-build skipping can consult. - A publish in progress is never aborted. Supersession, stop, hold and
rollback never touch a
publishingdeployment. - Fenced grants. Every claim increments the lane epoch and issues a random
attemptId. Heartbeat and completion must present the current epoch and attempt; anything else isSTALE_GRANT. The one exception is the reconciling completion of anunknownattempt described above. - Leases. A lease lasts five minutes, heartbeats extend it, and an attempt
may publish for at most 90 minutes. The lock stays held for a two-minute
grace after the lease lapses, then the deployment becomes
unknownand the lane frees. Executors self-fence: they must not start a provider effect afterleaseDeadline, the same rule as the offline adapter’s arm deadline. An unknown effect is never replayed automatically. - Lazy evaluation instead of scheduling. Lease lapses and queue expiry are applied at the start of every call to the object, reads included. The coordinator schedules nothing: no alarms, crons, queues or timers. A lane that nobody calls costs nothing and holds its state until the next call.
- Idempotency.
createis keyed by run id, run attempt and lane from the verified token; repeating it with the same body returns the same deployment and a different body is a409. Repeating a completion with the same outcome returns the recorded result; a different outcome is a409.
Promotion and rollback
- Promote takes a
succeededstaging deployment and queues a productionpromotionfor the same commit. Production builds its own artifacts (it never reuses preview or staging builds), so a promotion releases a verified commit rather than copying an artifact. Preview deployments are not promotable. - Rollback takes an earlier
succeededdeployment in staging or production, queues arollbackcarrying that deployment’s commit and artifact identity, supersedes queued deploys and places the lane on hold. While held, deploys still register butclaimanswersheldand the job records a skip; promotions and rollbacks can still claim. Releasing the hold, or a successful promotion, resumes automatic deploys, so a rollback is not immediately overwritten by the next merge. An operator cancelling the rollback that placed the hold releases it; a rollback that fails or expires keeps the hold until an operator releases it, because the lane may still be serving the version that prompted the rollback. Skipped deploys are not replayed: after releasing a hold, an untouched project stays on the rolled-back version until its next deploy or an explicit dispatch. - Retries. Repeating a promote or rollback request returns its open or
succeeded marker. Once the latest marker for the same request finished
without succeeding (
failed,cancelledorunknown), the request queues a new marker keyed after it, so a failed promotion or rollback can be retried. - Execution. Promotion and rollback rows are markers. The coordinator never
dispatches anything. An executor workflow that claims a marker by id, holding
provider credentials in its own GitHub Environment, is a separate reviewed
change. Until it exists, an operator performs the provider action with the
owning release tooling and records the outcome with
resolve. Queue admission never authorizes production activation, environment approval or route changes.
API
All responses are JSON withCache-Control: no-store. Bodies are read as a
stream and rejected once they pass 8 KiB, whatever Content-Length says, and
every string field has a length bound. Unknown routes return 404.
The read endpoints are the stable shape for CLI consumers such as a
ttr
deploy-queue view.
Authentication
CI sends a GitHub Actions OIDC token. The Worker verifies:- RS256 against GitHub’s JWKS, cached per isolate and refetched at most once a minute for an unknown key id;
iss,aud(DEPLOY_COORDINATOR_OIDC_AUDIENCE),expandnbf, and the immutablerepository_id(DEPLOY_COORDINATOR_REPOSITORY_ID);event_nameispush,workflow_dispatchorschedule; pull request events, includingpull_request_target, are rejected, and fork pull requests never receive tokens;job_workflow_refnames a workflow file the catalog allows for the project, at the same protected ref asref:refs/heads/productionfor production,refs/heads/mainfor staging and preview. For a shared reusable publisher,workflow_refmust also name the project’s calling workflow at that ref.
sha claim.
The logs URL is built from the token’s repository, run_id and
run_attempt, never from the body. Claim, heartbeat, completion and skip must
come from the run that created the deployment. Ownership is checked by
run_id without run_attempt on purpose: a re-run attempt registers its own
deployment, because the create key carries the attempt, yet can still skip or
close one its earlier attempt left open. Publishing rights stay fenced by the
grant’s attemptId and epoch, which another attempt never holds.
Operators are the Infrastructure app’s server routes. They send
DEPLOY_COORDINATOR_OPERATOR_TOKEN, compared in constant time, and an
X-Deploy-Actor user id. The Worker records that id for audit and trusts it
only because the token holder has already authorized the user.
The Worker has no public route until release authority approves a hostname;
workersDev and preview URLs stay disabled.
GitHub Actions integration
Composite actions have no post step, so the integration is.github/actions/deploy-coordinator (running
scripts/ci/deploy-coordinator-client.mjs) used up to three times in a deploy
job:
phase: begin, before the publish steps: registers the deployment, waits for a grant, and starts a bounded background heartbeat for the job.phase: check, immediately before each provider effect: inenforce, fails the step when the heartbeat was refused, so a fenced job never starts an effect after its lease (the self-fence in rule 5).phase: finish, withif: always(): stops the heartbeat and reportssucceededorfailedfrom thejob-statusinput (job.status).
superseded or held, begin records a skip
and sets publish=false; later steps check that output. Composite actions
cannot read repository variables, so each step passes mode and origin from
vars.DEPLOY_COORDINATOR_MODE and vars.DEPLOY_COORDINATOR_ORIGIN, and is
itself skipped unless the mode is observe or enforce. The job needs
id-token: write; when the deploy workflow runs as a reusable workflow, its
caller job must grant it too (the production planner grants it to
deploy-colab-cloudflare only). The pilot is the Colab production deploy job
in colab-cloudflare.yaml:
DEPLOY_COORDINATOR_MODE selects:
In
enforce, an unreachable coordinator, an exhausted budget or the bounded
wait running out still falls open so an outage cannot block a release; GitHub
concurrency groups keep their serialization. The consequence is explicit: a
rollback hold is advisory to CI while the coordinator is unreachable, and the
environment stop fence is advisory to CI both then and once the bounded wait
runs out on a reachable stopped answer. A job that publishes without a grant
first skips its deployment with PUBLISHED_UNCOORDINATED, so the dashboard
never shows it queued. Making the coordinator a required authority is a
separate reviewed change.
Dashboard
The Infrastructure app’s Deployments page (/[wsId]/deployments, under
Delivery) lists catalog projects, and for the selected project each
environment’s lanes, current deployment, queue, statuses and GitHub run links.
Viewing requires view_infrastructure. Promote, rollback, cancel, resolve,
hold release and stop/resume go through Infrastructure API routes that require
the root-workspace operator permission (manage_workspace_roles, via
authorizeInfrastructureOperator); the page hides those controls from viewers.
Without DEPLOY_COORDINATOR_ORIGIN and DEPLOY_COORDINATOR_OPERATOR_TOKEN the
page shows an unconfigured state and the routes make no outbound call.
The server client (
apps/infrastructure/src/lib/infrastructure/deploy-coordinator.ts)
sends the operator token and the signed-in user id as X-Deploy-Actor, and maps
records to the dashboard shapes in @tuturuuu/internal-api/infrastructure
without grant attempt ids or idempotency material. Environment reads are exempt
from the coordinator’s daily budget, so the page reads on demand (open, focus,
refresh and after an action) and never polls.
Cost and stop-fence analysis
The coordinator adds Worker requests, Durable Object requests and SQLite row operations. It adds no alarms, queues, crons, WebSockets, outbound provider calls or retries of its own. Its objects do no I/O beyond synchronous SQLite, so they are eligible for hibernation whenever idle and accrue duration only while handling a call (pricing).
A busy production environment of 20 deployments a day at the worst case uses
about 3,600 calls, inside its budget. The hard ceiling, with every catalog
environment pinned at its budget every day:
- 33 projects × 3 environments × 5,000 calls ≈ 495,000 calls a day, about 14.9 million a month.
- Worker requests above the 10 million included: about $1.46 a month.
- Durable Object requests above the 1 million included: about $2.08.
-
SQLite rows written. Cloudflare bills each row a write touches in a table
and in every index on it. The schema weighs a deployment write at 5 rows
(row, primary key, idempotency key, lane order and status indexes), a lane
or budget write at 2 and the sequence counter at 1. Pinned per call by
src/cost.test.ts:A call can also apply transitions that belong to earlier calls: lazy lease lapse and queue expiry, supersession, pruning and lane eviction. Each deployment row is terminated and deleted at most once, so those writes are charged to the create that inserted it: at most 25 rows a create (11, its terminal transition 7, its deletion 5, its lane 2), and no other call exceeds 9 once its share is counted. The first counted call of each UTC day also replaces the budget row, 2 more rows once a day per environment. - Hard ceiling, every call a create: 495,000 × 25 ≈ 12.4 million rows a day, about 371 million a month against 50 million included: about $321.
- Realistic ceiling, every environment at budget running the CI client’s worst-case lifecycle (create, 90 wait polls, a grant, 90 heartbeats, completion), which the cost test pins below 5 rows a call: about 74 million a month, about $24.
- Rows read stay far below the 25 billion included; duration is the millisecond-scale handling time of each call, far below 400,000 GB-s.
- Global switch.
DEPLOY_COORDINATOR_ENABLEDset to anything other thantruemakes every CI call return503 COORDINATOR_OFFbefore any Durable Object call. - Environment stop.
POST …/stopdurably setsstoppedand incrementsstopRevision. Claims answerstopped; queued deployments stay queued and publishing ones may still complete. The fence lives in Durable Object storage and survives restarts and redeploys. - Daily budget. The durable per-environment counter caps calls, and with them rows written, even if a client loops. Rejected calls that still reach an object are billed requests; the global switch is the hard cap on call count.
- Client. The steps’ wait, heartbeat and retry ceilings; setting
DEPLOY_COORDINATOR_MODEtooffstops all CI traffic without a code change.
Relationship to existing pieces
- Offline queue adapter (
scripts/ci/release-cf-queue*.mjs): grants use the adapter’s fields (jobId,attemptId,workerId,epoch,targetLockId,stopRevision,leaseDeadline). Statuses map asqueued→queued,publishing→running,succeeded,failedandunknownunchanged, andsupersededorcancelled→stopped. The adapter’ssubmit/renew/requestStopport and per-phase receipts have no binding to the coordinator yet; that binding would be a separate adapter. - Production planner (
.github/workflows/vercel-production.yaml) and per-workflow concurrency groups remain the trigger and first serialization layer. The coordinator adds cross-run supersession, a lane lock and history. - CI affected-planner: each deployment stores its plan digest and run URL.
- Superseded-build skipping: a
supersededclaim answer is a durable, cross-run reason to skip a build. - Cron Control and Devbox Control: the Worker follows their layout (cf
config, Vite+ build, SQLite Durable Object,
cf-vite-worker.mjsruntime harness).
Verification
Run from the repository root:bun --cwd apps/deploy-coordinator testruns the state machine suite twice, against the in-memory store and against real SQLite (node:sqlite) through the sameexeccontract as the Durable Object. It covers supersession by run order, lock exclusivity, epochs, lease lapse and reconciliation, idempotent create and completion, hold, marker retries, stop, budget, capacity, queue expiry, lane eviction and retention. The cost suite meters index-weighted rows written per call against SQLite and pins the figures above. The router suite drives the HTTP API with OIDC tokens signed by a generated RSA key; the OIDC suite covers claim, event, workflow and ref checks and the rate-limited key refetch.node --test scripts/ci/deploy-coordinator-client.test.mjscovers the client’s phases, fall-open paths, bounded waits and retries.node --test apps/deploy-coordinator/scripts/verify-worker-runtime.mjsruns the real Worker and Durable Object in workerd through the Vite runtime: the private config, closed routes, and a seeded lane lock, stop fence and exhausted budget survivingrestart(). It sends no token that could reach GitHub’s JWKS. The harness warms the first Durable Object route and reads once after a restart before using the Local Explorer, which can reload the runtime on a cold path.- Source for the operating limits is
LIMITSinapps/deploy-coordinator/src/core.ts; change this page with it.