> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuturuuu.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment coordinator

> Design of the Tuturuuu-hosted deployment authority: projects, environments and deployments on a Cloudflare Worker with one Durable Object per project environment, serialized publishes, supersession, promotion and rollback markers, a durable stop fence and a bounded cost ceiling.

The deployment coordinator is the platform's own deployment authority. GitHub
Actions keeps building and publishing; the coordinator decides **which** build
may publish **when**, records what happened, and gives the Infrastructure
dashboard one place to see, promote and roll back deployments.

It is the `tuturuuu-deploy-coordinator` Worker in `apps/deploy-coordinator`.
The Worker holds no Vercel or Cloudflare deploy credentials and never calls a
provider or dispatches a workflow. Executors stay in GitHub Actions; the
coordinator issues them fenced grants.

## Delivery status

| Part | State |
| - | - |
| This design | Accepted |
| Worker, Durable Object state machine, API | Source, unit and workerd tests; not published. `deploy-coordinator-cloudflare.yaml` runs its type check, lint, format check, unit suites, the client tests and the workerd harness |
| GitHub Actions begin/check/finish steps | `.github/actions/deploy-coordinator`, piloted on the Colab production deploy; `DEPLOY_COORDINATOR_MODE` unset, so off |
| Infrastructure **Deployments** page | Built under **Delivery → Deployments**; shows the unconfigured state until the origin and operator token are set |
| Executor workflow for promotion and rollback markers | Not built; markers are resolved by operators |
| Hostname, secrets and Worker publication | Release-authority steps |

## Model

```text theme={null}
project (catalog entry, e.g. "calendar")
└── environment: preview | staging | production   ← one Durable Object each
    └── lane: "default", or the preview ref for preview
        └── deployment
            id, kind (deploy | promotion | rollback), commit SHA, ref, run id,
            plan reference, status, epoch, attempt, artifact identity,
            URL, logs URL (GitHub run), actor, timestamps
```

* **Projects** come from the committed catalog in
  `apps/deploy-coordinator/src/projects.ts`. Each entry names the provider
  (`vercel` or `cloudflare`) and, per environment, the workflow files whose
  jobs may report for it. An unknown project or environment is a `404`; adding
  one is a reviewed code change. The catalog bounds how many Durable Objects
  can exist.

  Only workflows that publish the environment are listed:

  | Environment | Vercel apps | Cloudflare Workers |
  | - | - | - |
  | production | `vercel-production-<app>.yaml`; also `meet-cloudflare.yaml` for `meet` and `tasks-cloudflare.yaml` for `tasks`, which publish their native Workers | `<worker>-cloudflare.yaml` |
  | staging | `wave1-native-publish.yaml` called from `<app>-cloudflare.yaml`, for `apps`, `calendar` and `mail`; none for the others, whose `<app>-cloudflare.yaml` only qualifies builds | none |
  | preview | `vercel-preview-<app>.yaml` | none: Worker previews run on pull request events, which are rejected |

  Chat, CMS, Drive, Forms, Teach, Tools and Track publish production natively
  on Cloudflare after their Vercel production workflows retired (see
  [retiring an app's Vercel project](https://docs.tuturuuu.com/build/devops/native-start-migration.md#retiring-an-apps-vercel-project)).
  That release does not report to the coordinator yet, so they list no
  production workflow; their retained `vercel-preview-<app>.yaml` may still
  report previews.

  A job inside a reusable workflow reports that reusable file as its
  `job_workflow_ref` and the workflow the run started from as its
  `workflow_ref`. A shared publisher is therefore listed with its caller, and
  both claims must match, so a `wave1-native-publish.yaml` run started from
  `mail-cloudflare.yaml` cannot report for `calendar`. `platform` (`apps/web`)
  staging is not listed: `wave1-web-staging.yaml` publishes a dispatched
  `source_sha` that can differ from the token's `sha`, so its record would name
  the wrong commit until that workflow can state its source commit. The
  catalog tests check that every listed file exists, that each production
  file has a job gated on the production ref in a GitHub Environment, that
  each staging caller runs the shared publisher with its own `app` and
  `profile: staging`, and that preview workflows are dispatch-only.
* **Environments** follow the existing branch flow (see
  [environments and release flow](https://docs.tuturuuu.com/build/devops/environments-release-flow.md)):
  production deploys run on `refs/heads/production`, staging on
  `refs/heads/main`, and previews are dispatched manually from `main` with a
  `preview_ref` input.
* **Lanes** are the unit of serialization. Staging and production have one
  `default` lane. Every preview token carries `ref` `refs/heads/main`, so the
  preview lane is the `preview_ref` the caller states. That value is accepted
  only from an allowlisted preview workflow on `main`, which is already the
  trust boundary for preview dispatch.
* **Deployments** carry the plan they were built from. The coordinator stores
  the CI affected-planner's plan reference (`source`, `sha256` digest, run URL)
  and never recomputes affected apps.

### Statuses

```text theme={null}
queued ──claim──▶ publishing ──complete──▶ succeeded | failed
  │                   │
  │                   └─lease + grace lapsed──▶ unknown ──late complete, same attempt──▶ succeeded | failed (reconciled)
  ├─newer deploy run in lane, or a rollback──▶ superseded
  └─skip (held or not needed), operator cancel, 6 h queue expiry──▶ cancelled
```

`succeeded`, `failed`, `superseded`, `cancelled` and `unknown` release the
lane. `unknown` still accepts one reconciling completion from the same
attempt; that record never becomes the current deployment over a newer epoch.

## Serialization rules

Each environment Durable Object keeps its lanes in SQLite. A call reads and
writes synchronously without awaiting I/O, so a claim's read and write never
interleave with another request.

1. **One publisher per lane.** `claim` grants the lane lock to one deployment;
   the oldest claimable queued deployment goes first. Others receive `wait`
   with a bounded `retryAfterSeconds`. A deploy queues only behind older
   deploys: promotion and rollback markers are resolved by operators rather
   than claimed by CI, so an unresolved marker never holds deploys for its
   six-hour queue lifetime. Resolving a marker needs the lane lock to be free.
2. **Newest run wins the queue.** Deployments are ordered by GitHub `run_id`,
   which increases with run creation, so a re-run of an older run cannot
   overtake a newer one. A new `deploy` supersedes queued `deploy` rows with a
   lower run id. A `deploy` whose run id is lower than a queued, publishing or
   succeeded `deploy` in the lane is created already `superseded`. Promotions
   and rollbacks are never superseded by deploys. A superseded job exits
   successfully without publishing; this is the durable answer superseded-build
   skipping can consult.
3. **A publish in progress is never aborted.** Supersession, stop, hold and
   rollback never touch a `publishing` deployment.
4. **Fenced grants.** Every claim increments the lane epoch and issues a random
   `attemptId`. Heartbeat and completion must present the current epoch and
   attempt; anything else is `STALE_GRANT`. The one exception is the
   reconciling completion of an `unknown` attempt described above.
5. **Leases.** A lease lasts five minutes, heartbeats extend it, and an attempt
   may publish for at most 90 minutes. The lock stays held for a two-minute
   grace after the lease lapses, then the deployment becomes `unknown` and the
   lane frees. Executors **self-fence**: they must not start a provider effect
   after `leaseDeadline`, the same rule as the offline adapter's arm deadline.
   An unknown effect is never replayed automatically.
6. **Lazy evaluation instead of scheduling.** Lease lapses and queue expiry are
   applied at the start of every call to the object, reads included. The
   coordinator schedules nothing: no alarms, crons, queues or timers. A lane
   that nobody calls costs nothing and holds its state until the next call.
7. **Idempotency.** `create` is keyed by run id, run attempt and lane from the
   verified token; repeating it with the same body returns the same
   deployment and a different body is a `409`.
   Repeating a completion with the same outcome returns the recorded result; a
   different outcome is a `409`.

## Promotion and rollback

* **Promote** takes a `succeeded` staging deployment and queues a production
  `promotion` for the same commit. Production builds its own artifacts (it
  never reuses preview or staging builds), so a promotion releases a verified
  commit rather than copying an artifact. Preview deployments are not
  promotable.
* **Rollback** takes an earlier `succeeded` deployment in staging or
  production, queues a `rollback` carrying that deployment's commit and
  artifact identity, supersedes queued deploys and places the lane on
  **hold**. While held, deploys still register but `claim` answers `held` and
  the job records a skip; promotions and rollbacks can still claim. Releasing
  the hold, or a successful promotion, resumes automatic deploys, so a
  rollback is not immediately overwritten by the next merge. An operator
  cancelling the rollback that placed the hold releases it; a rollback that
  fails or expires keeps the hold until an operator releases it, because the
  lane may still be serving the version that prompted the rollback. Skipped
  deploys are not replayed: after releasing a hold, an untouched project stays
  on the rolled-back version until its next deploy or an explicit dispatch.
* **Retries.** Repeating a promote or rollback request returns its open or
  succeeded marker. Once the latest marker for the same request finished
  without succeeding (`failed`, `cancelled` or `unknown`), the request queues
  a new marker keyed after it, so a failed promotion or rollback can be
  retried.
* **Execution.** Promotion and rollback rows are markers. The coordinator never
  dispatches anything. An executor workflow that claims a marker by id, holding
  provider credentials in its own GitHub Environment, is a separate reviewed
  change. Until it exists, an operator performs the provider action with the
  owning release tooling and records the outcome with `resolve`. Queue
  admission never authorizes production activation, environment approval or
  route changes.

## API

All responses are JSON with `Cache-Control: no-store`. Bodies are read as a
stream and rejected once they pass 8 KiB, whatever `Content-Length` says, and
every string field has a length bound. Unknown routes return `404`.

| Method and path | Caller | Purpose |
| - | - | - |
| `GET /health` | anyone | Liveness only |
| `GET /v1/projects` | operator | Catalog only; no Durable Object calls |
| `GET /v1/projects/:p/environments/:e` | operator | Lanes, locks, hold and stop state, recent deployments |
| `POST /v1/projects/:p/environments/:e/deployments` | CI | Register a deployment |
| `POST …/deployments/:id/claim` | CI, same run | `granted`, `wait`, `held`, `stopped`, `superseded` or `terminal` |
| `POST …/deployments/:id/heartbeat` | CI, same run | Extend the lease |
| `POST …/deployments/:id/complete` | CI, same run | Record `succeeded` or `failed` with URL and artifact |
| `POST …/deployments/:id/skip` | CI, same run | Cancel its own queued deployment |
| `POST …/deployments/:id/cancel` | operator | Cancel a queued deployment |
| `POST …/deployments/:id/resolve` | operator | Record the outcome of a queued promotion or rollback marker |
| `POST …/deployments/:id/promote` | operator | Queue a production promotion from staging |
| `POST …/deployments/:id/rollback` | operator | Queue a rollback marker and hold the lane |
| `POST /v1/projects/:p/environments/:e/hold/release` | operator | Resume automatic deploys in a lane |
| `POST /v1/projects/:p/environments/:e/stop` and `/resume` | operator | Durable stop fence for the environment |

The read endpoints are the stable shape for CLI consumers such as a `ttr`
deploy-queue view.

### Authentication

**CI** sends a GitHub Actions OIDC token. The Worker verifies:

* RS256 against GitHub's JWKS, cached per isolate and refetched at most once a
  minute for an unknown key id;
* `iss`, `aud` (`DEPLOY_COORDINATOR_OIDC_AUDIENCE`), `exp` and `nbf`, and the
  immutable `repository_id` (`DEPLOY_COORDINATOR_REPOSITORY_ID`);
* `event_name` is `push`, `workflow_dispatch` or `schedule`; pull request
  events, including `pull_request_target`, are rejected, and fork pull
  requests never receive tokens;
* `job_workflow_ref` names a workflow file the catalog allows for the project,
  at the same protected ref as `ref`: `refs/heads/production` for production,
  `refs/heads/main` for staging and preview. For a shared reusable publisher,
  `workflow_ref` must also name the project's calling workflow at that ref.

For staging and production the commit SHA comes from the token's `sha` claim.
The logs URL is built from the token's `repository`, `run_id` and
`run_attempt`, never from the body. Claim, heartbeat, completion and skip must
come from the run that created the deployment. Ownership is checked by
`run_id` without `run_attempt` on purpose: a re-run attempt registers its own
deployment, because the create key carries the attempt, yet can still skip or
close one its earlier attempt left open. Publishing rights stay fenced by the
grant's `attemptId` and epoch, which another attempt never holds.

**Operators** are the Infrastructure app's server routes. They send
`DEPLOY_COORDINATOR_OPERATOR_TOKEN`, compared in constant time, and an
`X-Deploy-Actor` user id. The Worker records that id for audit and trusts it
only because the token holder has already authorized the user.

The Worker has no public route until release authority approves a hostname;
`workersDev` and preview URLs stay disabled.

## GitHub Actions integration

Composite actions have no post step, so the integration is
`.github/actions/deploy-coordinator` (running
`scripts/ci/deploy-coordinator-client.mjs`) used up to three times in a deploy
job:

1. `phase: begin`, before the publish steps: registers the deployment, waits
   for a grant, and starts a bounded background heartbeat for the job.
2. `phase: check`, immediately before each provider effect: in `enforce`,
   fails the step when the heartbeat was refused, so a fenced job never starts
   an effect after its lease (the self-fence in rule 5).
3. `phase: finish`, with `if: always()`: stops the heartbeat and reports
   `succeeded` or `failed` from the `job-status` input (`job.status`).

When the coordinator answers `superseded` or `held`, `begin` records a skip
and sets `publish=false`; later steps check that output. Composite actions
cannot read repository variables, so each step passes `mode` and `origin` from
`vars.DEPLOY_COORDINATOR_MODE` and `vars.DEPLOY_COORDINATOR_ORIGIN`, and is
itself skipped unless the mode is `observe` or `enforce`. The job needs
`id-token: write`; when the deploy workflow runs as a reusable workflow, its
caller job must grant it too (the production planner grants it to
`deploy-colab-cloudflare` only). The pilot is the Colab production `deploy` job
in `colab-cloudflare.yaml`:

```yaml theme={null}
- id: coordinator
  if: vars.DEPLOY_COORDINATOR_MODE == 'observe' || vars.DEPLOY_COORDINATOR_MODE == 'enforce'
  uses: ./.github/actions/deploy-coordinator
  with:
    phase: begin
    mode: ${{ vars.DEPLOY_COORDINATOR_MODE }}
    origin: ${{ vars.DEPLOY_COORDINATOR_ORIGIN }}
    project: colab
    environment: production
# publish steps: if: steps.coordinator.outputs.publish != 'false'
# phase: check before `wrangler deploy`; phase: finish under if: always()
```

The repository variable `DEPLOY_COORDINATOR_MODE` selects:

| Mode | Behaviour |
| - | - |
| unset/`off` | Steps skipped; CI behaves exactly as without them. This is the default. |
| `observe` | Register and report; `publish` is always `true` and every error falls open. |
| `enforce` | Wait for a grant, bounded by `max-wait-minutes`; skip publishing when superseded or held. |

In `enforce`, an unreachable coordinator, an exhausted budget or the bounded
wait running out still falls open so an outage cannot block a release; GitHub
concurrency groups keep their serialization. The consequence is explicit: a
rollback hold is advisory to CI while the coordinator is unreachable, and the
environment stop fence is advisory to CI both then and once the bounded wait
runs out on a reachable `stopped` answer. A job that publishes without a grant
first skips its deployment with `PUBLISHED_UNCOORDINATED`, so the dashboard
never shows it queued. Making the coordinator a required authority is a
separate reviewed change.

## Dashboard

The Infrastructure app's **Deployments** page (`/[wsId]/deployments`, under
**Delivery**) lists catalog projects, and for the selected project each
environment's lanes, current deployment, queue, statuses and GitHub run links.
Viewing requires `view_infrastructure`. Promote, rollback, cancel, resolve,
hold release and stop/resume go through Infrastructure API routes that require
the root-workspace operator permission (`manage_workspace_roles`, via
`authorizeInfrastructureOperator`); the page hides those controls from viewers.
Without `DEPLOY_COORDINATOR_ORIGIN` and `DEPLOY_COORDINATOR_OPERATOR_TOKEN` the
page shows an unconfigured state and the routes make no outbound call.

| Route | Permission | Coordinator calls |
| - | - | - |
| `GET /api/v1/infrastructure/deployments` | viewer | `GET /v1/projects` |
| `GET /api/v1/infrastructure/deployments/:project` | viewer | catalog, then the three environment snapshots |
| `POST /api/v1/infrastructure/deployments/:project/:environment` | operator | the one action in the body |

The server client (`apps/infrastructure/src/lib/infrastructure/deploy-coordinator.ts`)
sends the operator token and the signed-in user id as `X-Deploy-Actor`, and maps
records to the dashboard shapes in `@tuturuuu/internal-api/infrastructure`
without grant attempt ids or idempotency material. Environment reads are exempt
from the coordinator's daily budget, so the page reads on demand (open, focus,
refresh and after an action) and never polls.

## Cost and stop-fence analysis

The coordinator adds Worker requests, Durable Object requests and SQLite row
operations. It adds no alarms, queues, crons, WebSockets, outbound provider
calls or retries of its own. Its objects do no I/O beyond synchronous SQLite,
so they are eligible for hibernation whenever idle and accrue duration only
while handling a call ([pricing](https://developers.cloudflare.com/durable-objects/platform/pricing/)).

| Item | Bound |
| - | - |
| Durable Objects | catalog projects × 3 environments; no other names are addressable |
| Calls per deployment | create 1, claim polls every ≥20 s for ≤30 min (≤90), heartbeats every 60 s for ≤90 min (≤90), finish 1: worst case \~182, typical under 10 |
| Client retries | 3 attempts with backoff on network errors and `5xx` only |
| Daily budget | 5,000 calls per environment object per UTC day, counted durably before any other work; beyond it `429 BUDGET_EXHAUSTED`, which writes no rows. The Worker remembers an exhausted environment in isolate memory until UTC midnight and answers without calling the object. Operator stop, resume and reads are neither counted nor rejected, write no rows and cost one Durable Object request each; the dashboard reads on demand and never polls |
| Preview lanes | 50 open lanes per preview environment; more is `429 CAPACITY` |
| Lanes | 200 per environment. A new lane evicts the least recently used idle lane with its rows, except a lane with an `unknown` attempt from the last six hours, which GitHub-hosted jobs can still reconcile; with no evictable lane, `429 CAPACITY` |
| Retention | 100 finished deployments per lane, pruned on write, plus any `unknown` attempt from the last six hours, which can still reconcile (one publisher per lane and the seven-minute lease plus grace keep that under about 52); no log content, only links |
| Payload | 8 KiB request bodies |

A busy production environment of 20 deployments a day at the worst case uses
about 3,600 calls, inside its budget. The hard ceiling, with every catalog
environment pinned at its budget every day:

* 33 projects × 3 environments × 5,000 calls ≈ 495,000 calls a day, about
  14.9 million a month.
* Worker requests above the 10 million included: about \$1.46 a month.
* Durable Object requests above the 1 million included: about \$2.08.
* SQLite rows written. Cloudflare bills each row a write touches in a table
  and in every index on it. The schema weighs a deployment write at 5 rows
  (row, primary key, idempotency key, lane order and status indexes), a lane
  or budget write at 2 and the sequence counter at 1. Pinned per call by
  `src/cost.test.ts`:

  | Call | Rows written |
  | - | - |
  | Budget-rejected call; operator read | 0 |
  | Operator stop or resume | 1 |
  | Resolve a marker | 11, of which its terminal transition (7) belongs to the marker's create |
  | Claim answered `wait`, `held`, `stopped` or `superseded`; repeated create | 2 |
  | Heartbeat | 7 |
  | Granted claim, completion | 9 |
  | Create | 10, 11 for an environment's first |
  | Create that supersedes a queued deploy and prunes past retention | 22 |

  A call can also apply transitions that belong to earlier calls: lazy lease
  lapse and queue expiry, supersession, pruning and lane eviction. Each
  deployment row is terminated and deleted at most once, so those writes are
  charged to the create that inserted it: at most 25 rows a create (11, its
  terminal transition 7, its deletion 5, its lane 2), and no other call
  exceeds 9 once its share is counted. The first counted call of each UTC day
  also replaces the budget row, 2 more rows once a day per environment.
* Hard ceiling, every call a create: 495,000 × 25 ≈ 12.4 million rows a day,
  about 371 million a month against 50 million included: about \$321.
* Realistic ceiling, every environment at budget running the CI client's
  worst-case lifecycle (create, 90 wait polls, a grant, 90 heartbeats,
  completion), which the cost test pins below 5 rows a call: about 74 million
  a month, about \$24.
* Rows read stay far below the 25 billion included; duration is the
  millisecond-scale handling time of each call, far below 400,000 GB-s.

The hard ceiling is about $325 a month above the plan, about $28 for the CI
client's own worst case, and both scale linearly with the catalog size. A CI
create is idempotent per run attempt and lane, so reaching the hard ceiling
needs a distinct GitHub run, or a distinct preview lane, for every call. The
catalog test pins the project count and the cost test pins the rows per call,
so growing either is a reviewed change to these figures. Requests rejected before
reaching an object (bad token, unknown route) cost Worker requests only and
are not bounded by the coordinator; that ingress limit belongs to the hostname's
WAF rules when one is assigned.

Stop fences, outermost first:

1. **Global switch.** `DEPLOY_COORDINATOR_ENABLED` set to anything other than
   `true` makes every CI call return `503 COORDINATOR_OFF` before any Durable
   Object call.
2. **Environment stop.** `POST …/stop` durably sets `stopped` and increments
   `stopRevision`. Claims answer `stopped`; queued deployments stay queued and
   publishing ones may still complete. The fence lives in Durable Object
   storage and survives restarts and redeploys.
3. **Daily budget.** The durable per-environment counter caps calls, and with
   them rows written, even if a client loops. Rejected calls that still reach an object are billed
   requests; the global switch is the hard cap on call count.
4. **Client.** The steps' wait, heartbeat and retry ceilings; setting
   `DEPLOY_COORDINATOR_MODE` to `off` stops all CI traffic without a code
   change.

A rate limit or billing alert alone is not a spend cap; the durable budget and
the catalog-bounded object count are.

## Relationship to existing pieces

* **Offline queue adapter** (`scripts/ci/release-cf-queue*.mjs`): grants use the
  adapter's fields (`jobId`, `attemptId`, `workerId`, `epoch`, `targetLockId`,
  `stopRevision`, `leaseDeadline`). Statuses map as `queued` → `queued`,
  `publishing` → `running`, `succeeded`, `failed` and `unknown` unchanged, and
  `superseded` or `cancelled` → `stopped`. The adapter's
  `submit`/`renew`/`requestStop` port and per-phase receipts have no binding to
  the coordinator yet; that binding would be a separate adapter.
* **Production planner** (`.github/workflows/vercel-production.yaml`) and
  per-workflow concurrency groups remain the trigger and first serialization
  layer. The coordinator adds cross-run supersession, a lane lock and history.
* **CI affected-planner**: each deployment stores its plan digest and run URL.
* **Superseded-build skipping**: a `superseded` claim answer is a durable,
  cross-run reason to skip a build.
* **Cron Control and Devbox Control**: the Worker follows their layout (cf
  config, Vite+ build, SQLite Durable Object, `cf-vite-worker.mjs` runtime
  harness).

## Verification

Run from the repository root:

* `bun --cwd apps/deploy-coordinator test` runs the state machine suite twice,
  against the in-memory store and against real SQLite (`node:sqlite`) through
  the same `exec` contract as the Durable Object. It covers supersession by run
  order, lock exclusivity, epochs, lease lapse and reconciliation, idempotent
  create and completion, hold, marker retries, stop, budget, capacity, queue
  expiry, lane eviction and retention. The cost suite meters index-weighted
  rows written per call against SQLite and pins the figures above. The router suite drives the HTTP API with OIDC tokens signed by a
  generated RSA key; the OIDC suite covers claim, event, workflow and ref
  checks and the rate-limited key refetch.
* `node --test scripts/ci/deploy-coordinator-client.test.mjs` covers the
  client's phases, fall-open paths, bounded waits and retries.
* `node --test apps/deploy-coordinator/scripts/verify-worker-runtime.mjs` runs
  the real Worker and Durable Object in workerd through the Vite runtime: the
  private config, closed routes, and a seeded lane lock, stop fence and
  exhausted budget surviving `restart()`. It sends no token that could reach
  GitHub's JWKS. The harness warms the first Durable Object route and reads
  once after a restart before using the Local Explorer, which can reload the
  runtime on a cold path.
* Source for the operating limits is `LIMITS` in
  `apps/deploy-coordinator/src/core.ts`; change this page with it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.