mastermind-ship job in
Cron Control writes to the
Mastermind store every two minutes. Owners
can also run it with Check now. Web serves the snapshot owner-only at
GET /api/v1/mastermind/fleet/ship from a five-second per-colo cache;
?fresh=1, the read after Check now, skips it and replaces the cached
copy.
What each run reads
The probe is skipped on Check now and when the last full read is four
minutes old, so a read is reused for at most four minutes, inside the
clients’ six-minute stale window. A 304 leaves
okAt at the last full read.
A GraphQL answer with errors or a missing repository fails the source
instead of replacing its facts. When an environment’s first page of
deployments holds no success and more pages exist, the run reads up to two
more pages (100 deployments each, one query for every such environment).
Past those it keeps the previous live release, marked with the newest
attempt’s state, rather than recording none. Only a newest successful
deployment reads as Deployed; a newer failed, inactive or unfinished one
reads as Deploy failed or Deploying. The
catalog of apps, environments, paths and Worker scripts is
apps/cron-control/src/processors/mastermind-ship/catalog.ts. Apps outside this repository default to their known repositories; one
without a known repository stays “not configured” until
MASTERMIND_SHIP_REPOS names it. Each repository is its own source, so an
unreadable repository marks only its own rows.
Each source records checkedAt, okAt (data last confirmed current),
retryAt and a closed-set problem. A failed source keeps its last-known
facts with the old okAt, so a client always shows how old a row is.
The snapshot holds SHAs, times, environment and script names, CI states and
counts only. ETags, installation ids and cached version facts live under a
separate store key that Web never serves.
Configuration
Without GitHub or Cloudflare credentials the job still writes a snapshot in
which those sources read
unconfigured, and makes no provider request.
Bounds and stop fence
-
The job adds no Worker trigger: it is admitted every second minute of the
existing one-minute trigger (
everyMinutesinjobs.ts), so the reviewed trigger set and its parity baseline are unchanged. Odd minutes cost no ledger or network call. - One attempt per slot; the next slot is the retry. A run has a 40-second deadline, 32 provider requests at most and two store requests (one read, one write of two rows).
-
A 403 or 429 backs the source off until
Retry-Afteror the rate-limit reset, clamped to one minute to one hour; a GraphQL budget below 200 points backs off until its reset. Backed-off sources make no request. - Check now goes through the same ledger claim as the schedule. The last claimed slot rate-limits checks to one per 30 seconds across both paths, the lease refuses overlapping runs, and Web adds six requests a minute per owner.
-
A slot in which a provider call failed and no configured source is
okisfailedand counts against the failure budget, so a revoked credential stops the job after five slots. One failing source beside a working one isdegraded. -
Durable stops:
POST /control/jobs/mastermind-ship/stopwith the operator token (survives restarts; resume opens a new generation), five consecutive failed or unknown slots, or the store’s own fence (stoppedends the run without charging the failure budget). The fence is checked before each owner, each Worker read and the write.
apps/cron-control/src/mastermind-ship.test.ts (default-off,
the snapshot and its sanitization, unconfigured sources, the 304 reuse
window, backoff with carried facts, stop across a restart, the store fence,
the failure budget, Check now auth, rate limit and stop) and
src/processors/mastermind-ship/units.test.ts (repository config, the
query, capped pending counts, GraphQL budget backoff, App tokens, version
caching and the request budget).
apps/web/src/app/api/v1/mastermind/fleet/ship/ship-route.test.ts runs the
collector’s output through Web’s schema and covers owner-only reads,
tampered values and the Check now relay.
Operation envelope
At 720 slots a day a normal run makes one or two GitHub requests per repository (one more with App tokens), nine Cloudflare deployment reads, two store requests and about fourteen ledger calls. For three repositories that is about 2,200 to 6,500 GitHub requests, 6,500 Cloudflare API reads and 1,440 store row writes a day (of the store’s 50,000 daily write budget). Check now adds at most one run per 30 seconds. The failure case is capped by the request budget per run and the five-slot failure stop.Enable
- Create the GitHub App (or token) and the Cloudflare token; set the Cron Control secrets above and add the store token to the store’s rotation list.
- Set
MASTERMIND_SHIP_CHECK_TOKENon both Workers andMASTERMIND_SHIP_CHECK_ORIGINon Web. - Add
mastermind-shiptoCRON_CONTROL_ENABLED_JOBSand deploy Cron Control. ConfirmGET /control/jobsshows the job and the Ship screen shows freshokAtstamps.