Current runtime policy: Docker setup and Docker-backed E2E/image CI are paused;
the Docker cron runner is retired. Rust and TanStack Start are not in use and
must not be run, tested, or updated until further notice. Retained Docker and
migration sections below are historical procedures. Current hosted Next.js,
package, Worker, database, and mobile workflows retain their own gates. See the
active runtime map.
Validation and Delivery Evidence
Root Node tests that consume an app-owned package must resolve it withcreateRequire anchored to that app’s real package.json. The satellite cohort
tests resolve Playwright from Web; the Lettin config-loader tests load Next SWC
from Lettin. Keep repository-relative helpers on the test’s original require
and temporary fixtures separate from the owning app. Root links and job-wide
NODE_PATH can mask this boundary, so verify resolution with NODE_PATH unset
in a disposable fixture with only app-level dependency links and no root
node_modules. Resolve entrypoints without loading package code for that control,
then run the focused behavior tests separately. The Lettin SWC test transforms
config source in memory; it does not run a Next release build.
mail-calendar-tasks-builds.yaml builds Mail, Calendar and Tasks, including
their shared dependencies, on relevant PRs and main commits. These hosted CI
jobs check out the immutable PR head (or exact push SHA), use placeholder public
Supabase configuration, and do not deploy or access production credentials.
Require all three matrix builds when changes affect these applications or their
shared packages; type-check results and preview skip statuses do not replace
real app-build evidence. Mail provider OAuth and real delivery/RSVP acceptance
still require separately configured test accounts.
For standalone chat initial-settings qualification, launch the source-owned Vitest entry
directly with verified Node 24.21.0. Require primary exit zero, no signal or timeout,
and the owned group stopped; nine passing assertions cannot override SIGABRT or failure.
The optional process comparison defaults off and never qualifies failure; require the same source/control/run binding, SQL197, Web9, Dart9 and fixture/database
cleanup in a fresh exact-source hosted run. Synthetic ports do not prove runtime.
Focused source-only TypeScript checks must use the source entry corresponding to its package exports artifact and retain the repository compiler flags, including noUncheckedIndexedAccess. When a package contains both a same-named file and a directory index, mapping the directory can hide missing public exports. Regression tests for shared helpers should import through the shipped entry; the Infrastructure desktop public-entry test covers this boundary. Resolve source fixtures relative to import.meta.dirname, so workspace-owned test runs read the same files as root runs.
- Keep cross-app integration tests outside a package’s
srcdirectory when its type-check project usesrootDir: "src". The users-core tenant-boundary tests live intests/approvals; Vitest discovers them separately from the package source type check. Its config maps the two web-specific aliases explicitly while retaining the package’s default@alias. - Run tests from the owning workspace with its actual config, dependencies and providers. Root-hoisted packages or aliases can conceal undeclared dependencies and invalid test boundaries. If a narrow source-alias fixture is needed, record it and retain the owning type/test/build gates; do not substitute built artifacts or mock away the production orchestration being verified.
- Record immutable head and tree, commands, results, and workflow/run/job IDs.
A manual preview workflow may report the workflow branch SHA rather than its
input: verify the decoded checkout matches the requested head. Predecessor green
builds do not prove a corrected head. New findings or commits require new gates.
programming-app-builds.yamlruns nondeployment Next/Turbo builds for Web, Learn, Infrastructure, Mail, Contacts and Tasks on relevant pull requests, including shared dependencies; each app result must name the immutable checked-out head and does not authorize deployment or certify a different parent SHA. - Develop independent scopes in parallel, each with its own worktree and owner.
Admit heavy local validation through the shared
ttr resourcesFIFO after a resource preflight; serialize permitted local installs and tests. Require exact-head CI for builds; independent CI builds need not be serialized. One integration owner serializes main and production mutations. Other owners continue active work and hand off manifests rather than starting competing CI watch loops. - Use native stacks for dependencies: trunk rules and PR workflow filters apply to every layer. Verify each selected head, review, quiet window and CI before merging a contiguous prefix; queued is not merged. Check the resulting trunk SHA separately. Handle absent stack metadata in automation. For fallback chains, retain ancestry. After cascades/retargeting, rerun gates; manual base edits may not trigger CI. Inspect actual runs.
- Ship verified independent scopes in bounded releases. Pin and inspect the exact
main-to-production range, migrations and mobile path triggers before promotion;
stop if the range changes. Verify the exact merged-main gates before normal
authorized
bun git-sync, then verify production deployment and migration results separately. A planner success alone does not prove every downstream deployment. - For mobile, record source SHA, version/build number, signed upload and processing, and tester-group availability. Internal distribution and external beta review are separate states; neither proves installation or live UI behavior. Record supported read-back or device QA, and state any access or verification limits.
Python installer and cache boundaries
Discord workflows pinastral-sh/setup-uv to the immutable commit for stable
v10.2.0. Upstream stopped publishing major-only tags in v8, so future upgrades
must verify and pin the full stable release SHA. Keep uv sync --locked and
the apps/discord/uv.lock cache key together.
Ordinary Python CI retains cache pruning explicitly because setup-uv v9 changed
its default to keep unpruned caches. Modal’s deployment is triggered by
workflow_run and uses enable-cache: auto: setup-uv v10 disables cache access
for that sensitive event to protect against cache poisoning. Manual deployment
can still use the automatic cache policy. Preserve the successful-main-run
condition, exact upstream checkout SHA, and read-only repository permissions
when updating the installer; do not enable caching explicitly for workflow_run.
Workflow Gatekeeping
Every major workflow can be disabled centrally throughtuturuuu.ts.
The reusable workflow
ci-check.yml
reads that config and emits should_run, which downstream jobs use before doing
real work.
For rollback or mixed-version recovery, use $tuturuuu-rollout-recovery from the
Codex plugin and the
critical rollout gates. Audit canonical alias identities
and API upstreams: a Git branch or successful deployment marker can disagree
with the artifact selected by Instant Rollback.
For Vercel deployments, the production planner and platform preview perform
affected-app gating before any install, build, or deploy step starts. They check
the current GitHub event’s changed files against Vercel app metadata and
workspace dependency closures from tuturuuu.ts. Platform preview performs the
check inside its deploy job so protected main pushes create one runner, not a
reusable gate runner followed by a deployment runner.
The changed-file resolver must evaluate the full effective change range:
- Manual
workflow_dispatchruns bypass affected-app gating when the workflow is enabled. - Vercel production push workflows first look for the latest successful GitHub
Deployment marker for the same workflow and branch, then diff that marker SHA
to the current
GITHUB_SHA. - If no marker is available, push events use the GitHub event payload’s complete commit file list instead of falling back to only the latest commit.
- Pull request events diff the PR base SHA to the PR head SHA.
- If the resolver cannot prove the full range, it leaves changed-file state unavailable so Vercel gating defaults open.
apps/<app>/**runs that app’s preview or production Vercel workflow.packages/<package>/**runs every Vercel app whose transitiveworkspace:*dependency closure includes that package.- Storefront UI modules under
packages/ui/src/components/ui/storefront/**are scoped to their verified consumers, Inventory and Storefront, instead of rebuilding every app that imports another part of@tuturuuu/ui. apps/*/package.jsonandpackages/*/package.jsonare dependency changes for their owning workspaces.bun.lock, rootpackage.json,turbo.json,tuturuuu.ts, and.github/workflows/ci-check.ymlrun every Vercel app workflow.- A Vercel workflow file change, such as
.github/workflows/vercel-preview-calendar.yaml, runs that specific workflow. - If changed files cannot be computed, the Vercel gate defaults open.
.github/workflows/vercel-production.yaml once. That
planner evaluates Vercel and Cloudflare targets from their individual successful
deployment markers and calls only affected reusable workflows. Cloudflare
targets are Colab, Coordination, Lettin, Meet, and Parley. Meet can select both
Vercel and Cloudflare for the same source change. Automatic deployments stay
attached to the original push SHA
instead of creating workflow_dispatch runs. Do not restore per-app push
triggers for production; doing so recreates the queued preflight fan-out the
planner replaces. Cloudflare workflows retain their main and pull request
validation triggers and can still be manually dispatched for recovery.
Each reusable Vercel production workflow uses a static per-app concurrency prefix
plus the Git ref. Do not derive this key from reusable-workflow caller context: that
can make sibling app jobs share one group, cancel one another, and surface red X
statuses where unselected jobs should be skipped. Enable cancel-in-progress
only when the inherited event is a push to refs/heads/production; main
commits and manual recovery dispatches must not cancel the active deployment.
bun.lock-only changes intentionally run all Vercel app deploys because ownership
is ambiguous without a manifest or source path.
Coordination native Worker toolchain
Coordination is a Vite+ workspace using the Cloudflare Vite plugin. The inputapps/coordination/wrangler.jsonc remains the identity and binding authority;
Vite writes the deployable configuration to apps/coordination/dist/worker/wrangler.json.
The production workflow builds that artifact and deploys its generated config.
Do not change the Worker name, CoordinationObject export, COORDINATION binding,
or retained Durable Object migrations during a toolchain migration.
Validation first runs the existing node:test/Wrangler lease regression suite
against source. It then runs Vite+ lint and a real Worker build in CI, repeats
that same suite with COORDINATION_TEST_CONFIG pointing to the generated config,
and validates the artifact with Wrangler’s deployment dry run. The fixture injects
a disposable server token and uses local Durable Object storage. It verifies
credentials, concurrent acquisition, fencing, replay and expiry behavior without
production requests. The source and artifact runs must both pass on the exact head.
For focused local runtime verification, run node --test apps/coordination/src/worker.test.mjs from the root through ttr resources run.
It also works as bun run test inside the app. Run builds in CI or on an
authorized verified devbox through the shared resource queue with aggregate
memory, CPU and process caps. Vite+ development uses loopback port 8798 and disables
remote bindings and the inspector by default. Only start it for requested
runtime verification. Node 24 satisfies the pinned Vite+ runtime requirement.
The root Vitest project discovers future TypeScript unit tests separately from
this real Worker fixture; the Node suite stays a mandatory workflow gate.
If temporary transformed modules fail collection or Miniflare reports
SQLITE_IOERR_WRITE, check for actual temporary-file write failures before
changing product tests. df can show free space while an individual /tmp
write fails with Disk quota exceeded. A private writable TMPDIR under the
owning program directory can isolate direct validation from that failure. Set
TMPDIR only in the child payload after resource-broker admission. Do not set it
before bun check: its os.tmpdir()-based check queue would move shared state.
Verify the destination quota and retain the original failure before one bounded
validation attempt. Preserve unknown files and active jobs. Thread-pool retries passed focused Vitest suites,
but a private TMPDIR was also needed for the observed D1 fixture failure;
neither result establishes a universal runtime cause.
Pull Request Run Lifecycle
Closing or merging a pull request does not trigger a workflow that cancels its checks or deletes its caches. Existing workflow concurrency remains responsible for replacing superseded runs on the same branch. It does not cancel a run merely because the pull request closed. Inspect the actual conclusion of each run when reporting validation. A cancelled run is not a successful check and does not by itself establish a repository failure. Cache retention continues to follow GitHub Actions retention settings.Authorized CI Queue Cleanup
Manual cancellation requires explicit authorization for the cleanup scope. Start from a fresh inventory of active run IDs, workflow names, events, repositories, head branches and SHAs, together with currentmain and production refs and
live open PR heads. Pin those current heads before selecting candidates.
Classify protected branches before checking whether a branch appears among
merged PR heads. A historical promotion PR can have main as its head branch;
that does not make current main runs redundant. Never classify main or
production as merged-PR branches. Preserve every run on the current protected
heads and current open PR heads. Older protected-branch validation runs are
eligible only when the explicit cleanup scope includes superseded runs; deployment,
migration, release and signing work needs its own authorization.
A branch name alone is not cancellation evidence. Confirm each candidate’s event,
repository, SHA and current PR state against the authorized scope. Freeze a finite
list of run IDs and recheck protected refs before sending requests; if they moved,
refresh the inventory and remove newly protected candidates. Do not expand the
batch while it is running.
Record a receipt per run with the requested action, API response and observed
status. An accepted cancellation request is not a terminal cancelled result.
Use a bounded follow-up read of the selected IDs to distinguish completed runs,
requests still taking effect, conflicts and runs still active. Report remaining
runs without a polling loop or a claim that queue allocation has improved.
Force cancellation requires explicit authorization and must stay within the
remaining eligible IDs from the same finite batch. A one-time cleanup does not
authorize recurring cancellation.
Main Workflow Groups
Hosted web build and deployment checks
vercel-preview-platform.yamlvercel-preview-apps.yamlvercel-preview-calendar.yamlvercel-preview-chat.yamlvercel-preview-cms.yamlvercel-preview-drive.yamlvercel-preview-finance.yamlvercel-preview-inventory.yamlvercel-preview-infrastructure.yamlvercel-preview-learn.yamlvercel-preview-mail.yamlvercel-preview-meet.yamlvercel-preview-mind.yamlvercel-preview-nova.yamlvercel-preview-qr.yamlvercel-preview-rewise.yamlvercel-preview-shortener.yamlvercel-preview-storefront.yamlvercel-preview-tasks.yamlvercel-preview-teach.yamlvercel-preview-track.yaml- A matching
vercel-production-*.yamlworkflow exists for each app above
.github/workflows/vercel-preview-*.yaml and
.github/workflows/vercel-production-*.yaml as the authoritative source.
CMS uses the same Vercel deployment pattern as the other satellite apps:
- preview workflow:
vercel-preview-cms.yaml - production workflow:
vercel-production-cms.yaml - project secret:
VERCEL_CMS_PROJECT_ID - environments:
vercel-preview-cmsandvercel-production-cms
- preview workflow:
vercel-preview-apps.yaml - production workflow:
vercel-production-apps.yaml - project secret:
VERCEL_APPS_PROJECT_ID - environments:
vercel-preview-appsandvercel-production-apps
- preview workflow:
vercel-preview-qr.yaml - production workflow:
vercel-production-qr.yaml - project secret:
VERCEL_QR_PROJECT_ID - environments:
vercel-preview-qrandvercel-production-qr
- preview workflow:
vercel-preview-infrastructure.yaml - production workflow:
vercel-production-infrastructure.yaml - project secret:
VERCEL_INFRASTRUCTURE_PROJECT_ID - environments:
vercel-preview-infrastructureandvercel-production-infrastructure
- preview workflow:
vercel-preview-mail.yaml - production workflow:
vercel-production-mail.yaml - project secret:
VERCEL_MAIL_PROJECT_ID - environments:
vercel-preview-mailandvercel-production-mail
- run with default
contents: readpermissions - bind deploy jobs to
vercel-preview-<app>orvercel-production-<app>GitHub Environments - run preview workflows only through manual dispatch from
main, with a requiredpreview_refinput and an actor present in theTRUSTED_PREVIEW_DEPLOY_ACTORSrepository variable;vercel-preview-platform.yamlis the exception and also runs on protectedmainpushes so Supabase staging migrations keep their same-SHA prerequisite signal - reject production manual dispatches from non-production branches before install/build/deploy work starts
- call production workflows from the production push planner through
workflow_call; retainworkflow_dispatchonly for deliberate operator reruns - install Bun with
.github/actions/setup-bun-with-retry, pinned to the rootpackageManagerBun version and retried with exponential backoff - install dependencies through
scripts/ci/run-with-backoff.shso transient Bun tarball or cache failures retry after cache cleanup - build selected shared workspace dependencies before Vercel resolves package exports:
@tuturuuu/types,@tuturuuu/supabase, and@tuturuuu/internal-api - verify the package export map when a shared UI subpath fails resolution:
source files and TypeScript source aliases alone do not expose a package API;
node --test scripts/internal-api-calendar-exports.test.jsexercises Contacts’ real ESM resolution for the Calendar series clients without local builds - run
vercel pull - run
vercel build - deploy prebuilt artifacts for satellite apps and the hosted platform production workflow
- treat
vercel-preview-platform.yamlas platform build validation only; self-hosted machines own their own Dockerapps/webruntime deployment - serialize production runs through their existing per-workflow, per-branch
concurrency groups with cancellation disabled and
queue: max; later pushes and recovery dispatches wait rather than interrupting active deployments - record non-blocking GitHub Deployment markers after successful Vercel preview builds and production deploys so the next production push can evaluate every change since the last successful run
Database automation
supabase-staging.yamlsupabase-production.yaml
.github/actions/setup-supabase-cli-with-retry after checkout with Node 20+ and
npm available. The wrapper pins setup-cli v3.0.1 to its reviewed commit and passes
the exact stable CLI version resolved for apps/database in the root bun.lock.
A missing or malformed pin fails before installation; there is no latest
fallback. Explicit overrides must be exact stable versions. The v3 action no
longer accepts github-token, so callers do not pass a token.
The installer gets a runner-temporary home to protect any existing Bun binary.
An always-running restoration step verifies that binary’s version and restores
its PATH priority, while retaining the installed CLI path. Jobs without an
existing Bun runtime keep the installer’s runtime. Four attempts use 5, 10 and
20-second backoff; a fourth failure still fails the job. Real installer and
follow-on runtime behavior must pass the owning exact-commit CI jobs before
integration; this does not authorize a migration dispatch.
Rust dependency verification
The combinedrust-backend.yml workflow is paused in tuturuuu.ci.ts.
Its disabled switch also applies to manual dispatch; skipped verify jobs do not
prove native or wasm compilation, regardless of the aggregate check result.
rust-verify.yml provides a separate manual, main-only CI verification route
once registered on the default branch. Supply the exact 40-character source
commit SHA. It verifies the checkout and records source/workflow SHAs,
manifest and lock hashes, and the Rust toolchain in the run summary. It uses the
repository’s Rust 1.95 minimum to run locked native Clippy/tests, wasm Worker
checking and a native release build. It uses read-only repository permissions,
no deployment environment or secrets, and no shared cache writes.
Native tests run before strict Clippy so an unrelated lint failure cannot hide
actual test results for the requested source. Clippy retains -D warnings and
still fails the verifier; Worker packaging remains dependent on a successful
verifier. A passing test step alongside failed lint is bounded test evidence,
not a successful Rust gate. Preserve the original failed run and inspect the
exact source and test results separately before resolving review findings.
The ordering and strict-failure policy is covered by
scripts/ci/rust-verification-workflow.test.js.
Four private Storage/CMS port modules remain compiled for their regression tests
but deliberately have no dispatcher arms: storage downloads, workspace sharing,
CMS delivery and CMS assets. Their prior direct-download behavior lacks guarded
relay/source parity, so live Next.js retains ownership in route-overrides.json.
Only these module declarations carry reasoned #[expect(dead_code)]; strict
Clippy stays unchanged and handler source is retained. Crate-level
deny(unfulfilled_lint_expectations) rejects an expectation once no diagnostic
fulfills it. Remove the expectation when completing protected parity; the focused
workflow test also rejects reconnecting any of these handlers while its
expectation remains, even if other unused helpers still satisfy that expectation.
Never restore an unsafe dispatcher arm merely to make lint pass.
For dependency drafts, record the requested source SHA and the actual successful
job result. After the existing verifier succeeds, a separate sequential Worker-package job
installs checksum-published stable worker-build 0.8.7
with its locked dependencies into runner-temporary storage and runs the actual
release bundler with locked project dependencies and Worker-only features. It
checks the configured legacy build/worker/shim.mjs compatibility entry point,
its build/index.js target against the configured Wrangler entry, parses the emitted ES modules,
uses isolated pinned Acorn 8.18.0 to follow the default export through aliases
and the vendor Proxy to the same class that owns the emitted fetch function.
Selected class/alias bindings must not be reassigned; the check rejects that
unsupported pattern instead of matching an earlier handler to a later class.
It also validates a real WASM function export named fetch. It records config/artifact hashes. These are
package contract checks without invoking handlers or proving deployed behavior. The pinned vendor tool selects
wasm-bindgen 0.2.129, wasm-opt 132 and esbuild 0.28.2; this is Worker package
construction, without invoking Wrangler or accessing deploy secrets. The native
job keeps its 45-minute timeout. The sequential bundle job is bounded to 60 minutes,
with 25 minutes for bundler installation and 30 for release bundling; these are
limits, not measured durations. Both steps record elapsed seconds on ordinary command exit, including failures;
use hosted step timestamps if runner cancellation prevents the exit trap.
Record actual hosted duration after the first approved exact-source dispatch;
adjust only with evidence if a bound is insufficient. Docker
container packaging and deployed runtime behavior remain separate evidence gaps. Registering this workflow does not resume
the paused migration/deployment workflow or authorize a production cutover.
Docker automation
docker-setup-check.yamlrust-backend.yml
docker-setup-check.yaml is the workflow to watch whenever Docker files,
compose files, or Docker helper scripts change. It also watches the manifests
of apps directly represented in the Docker topology and bun.lock for
workspace dependency changes. Unrelated app or package version-only bumps do
not rebuild the web images. rust-backend.yml owns the
apps/backend Rust service checks and the migration Cloudflare checks:
formatting, locked dependency fetch, Clippy, tests, Cloudflare Worker target
validation, smoke reporter unit tests, native binary build, Docker image build,
apps/tanstack-web type-check/test validation, and the TanStack route-tree
generator/formatter unit test. Keep scripts/generate-tanstack-route-tree.*
in this workflow’s path filters so route-tree generation changes run before
TanStack/Rust deployment handoff.
The TanStack/Rust migration is currently paused. Keep rust-backend.yml,
tanstack-route-manifest.yaml, and
vercel-production-tanstack-web.yaml set to false in tuturuuu.ci.ts; the
shared ci-check.yml switchboard will leave the Rust/TanStack jobs skipped.
Re-enable them only when active maintenance resumes and the full workflow is
again an intentional delivery gate.
The shared e2e-tests.yaml workflow also keeps migration-e2e disabled and
publishes only the next image bundle. Maintained Next.js Playwright coverage
continues to run; the retained TanStack dual-stack and comparison matrix is a
manual source reference until migration work resumes.
Retrying allocated Playwright runs
E2E image bundle tags include the run ID, run attempt, and source SHA. A failed-jobs-only retry increments the attempt without necessarily rerunning the successful image preparation job. Shards can then miss that attempt’s bundle and fall back to local image builds, spending their deadline before assertions finish. After the allocated run’s final cleanup succeeds, retry the full workflow when fresh bundle preparation is needed; never cancel allocated runs, repeatedly retry blindly, or push over an active allocation to recover a missing bundle.E2E runtime budget
The general shard and dedicated invitation runtime commands use GNUtimeout
against the job’s recorded start, allowing up to 37 minutes minus setup
time so eight minutes remain inside the unchanged 45-minute job deadline. Timeout is
an ordinary failed step (exit 124), not a passing or skipped test. TERM is
followed by KILL after 30 seconds if the owned command cannot exit. Existing
failure diagnostics, redaction, artifact upload, and always() cleanup still
run; the reserve is a budget, not proof that runner or Docker failures cannot
prevent them. Setup exhausting the runtime budget fails before starting tests.
Validate this policy with node --test scripts/ci/e2e-runtime-budget.test.js.
Lettin E2E fixture and browser diagnostics
The canonical identity/About persistence check uses the same fixed browser phases and bounded context cleanup as the Markdown/import checks. Each complete phase has a 60-second cap within the existing 180-second test budget; cleanup retains the original action/assertion failure even if closing the browser context fails. Use the last entered profile phase to locate a stalled navigation, banner upload, save, or reload. A context-close error after the overall timeout is insufficient to identify the failing profile operation. Coverage is inlettin-browser-phase.test.ts and lettin-context-cleanup.test.ts; successful
unit checks do not prove authenticated profile persistence in CI.
Lettin fixture phases print fixed names for account creation, D1 migration and
seeding, browser context/cookie installation, and cleanup. Timed-out owned fixture
subprocesses receive SIGKILL at the existing 60-second deadline so a child ignoring
SIGTERM cannot block Playwright’s event loop indefinitely. These diagnostics
identify the last entered phase; an incomplete runner log alone does not prove a
browser, server, or database root cause. Do not include tokens, SQL, account IDs,
response bodies, or authenticated URLs in phase logs.
Lettin workspace GET diagnostics are enabled in CI, or explicitly with
LETTIN_READ_DIAGNOSTICS=true for a bounded investigation. They emit fixed
[lettin-read] stage names and monotonic elapsed milliseconds for actor resolution,
Cloudflare bindings, D1 reads and Supabase member/permission reads. A started stage
without completion identifies an unfinished dependency; a completed duration shows
where time was spent. Failures preserve the original error while logging only the
stage and elapsed time, never actor/workspace IDs, request URLs, data or raw errors.
These measurements diagnose stalls; they do not change authorization, retries or
request timeouts. Outside CI, leave the flag disabled after collecting evidence.
For wrapped native select controls, Playwright’s exact label matching includes
nested option text. Give the import Source select an explicit localized accessible
label so canonical-file selection remains reliable; the focused import component
test checks the explicit accessible label and associated native labels without
relying on private browser bundles. Maintained authenticated import E2E uses
Playwright’s public getByLabel('Source', { exact: true }) locator for browser coverage.
The Markdown persistence helper bounds actions at 15 seconds and navigation/reload
at 60 seconds, identifies opening the project dialog separately from navigation,
and uses the same 10-second context-close boundary as canonical imports. Cleanup
errors preserve the original failed assertion. These are diagnostic safeguards,
not proof that a server or runner stall is fixed.
The Markdown persistence helper also logs page creation before navigation,
using the same fixed phase summaries. This exposes a previously silent browser
boundary without claiming it caused the recurring stall. Canonical import
browser checks also emit fixed phase names and use a 15-second action deadline
(the existing 60-second navigation deadline remains). Import context cleanup
has a ten-second bound and retains the original assertion failure if cleanup
also fails; cleanup failure still fails an otherwise successful test. This
prevents a disposed-context error from obscuring the first failed operation.
Regression coverage lives in lettin-context-cleanup.test.ts.
Markdown and canonical-import browser phases also use public test.step with a
60-second whole-phase cap. Import preview visibility, post-import navigation/content,
and the public privacy response (including its body read) each have an explicit
phase too; they do not sit between capped clicks. It covers the complete awaited phase, including tracing
instrumentation before or after an action; the inner action/navigation/assertion
deadlines and tracing remain unchanged. A phase deadline fails the test and keeps
fixed, safe diagnostics; it never turns the stalled phase into a success or skip.
This contains a possible instrumentation wait and does not prove the cause of a
slow application request or runner shutdown. Regression coverage in
lettin-browser-phase.test.ts verifies the public step timeout option and
propagation of a simulated timeout with safe failure diagnostics. This focused
contract test does not launch a browser or prove a stalled browser operation
exits within the deadline; exact-commit E2E remains required.
E2E runtime heartbeat
For bounded CI stall investigations, the Lettin suite worker and the parent E2E runner emit a heartbeat every ten seconds with only wall-clock, monotonic-clock, and process CPU deltas. It is enabled byCI=true, or explicitly by
E2E_RUNTIME_DIAGNOSTICS=true locally. Worker timers stop after each test; the
parent timer stops when the Playwright command succeeds or fails. Timers are
unreferenced and do not keep either process alive. No identities, fixtures, URLs,
request bodies, or raw errors are included. Timeline page creation, navigation,
and assertions emit fixed operation markers without changing their deadlines.
On Linux, only the parent runner also samples five fixed /proc files:
meminfo, vmstat, and CPU, memory, and I/O pressure. Logs contain only available
memory/free swap in KiB, major fault/swap page counters, cumulative PSI stall
microseconds, and parent RSS bytes. Each asynchronous read allocates at most
32 KiB plus one byte and checks a shared one-second abort signal before and
after I/O. Samples never overlap and suppress late publication after command
cleanup. Unsupported
or unreadable fields emit only { unavailable: 1 }; raw file data and errors
are never logged. These counters can distinguish pressure during delayed ticks;
current readings alone do not prove which process caused it. Regression coverage
in e2e-runtime-resources.test.js checks numeric output, privacy, non-overlap,
and stopped publication.
CI shards use the public Playwright JSON discovery receipt to partition selected
tests into sequential cohorts by their declared satellite dependencies. Only a
cohort’s owned Next.js dev servers remain live during its tests; launcher process
groups, including descendants, stop before the next cohort starts. Multi-app
invitation cohorts keep their services together. A public Reporter onBegin
receipt supplies project dependency metadata omitted by the JSON reporter.
Dependency-connected projects share one invocation so Playwright retains setup
failure and dependent-project skip semantics; independent groups still run.
Selection, including sharding, project filters, and dependencies, happens once;
public --test-list manifests execute that selection without repeating setup.
Retries and
assertion deadlines stay unchanged, as does the outer CI runtime budget.
Each successful cohort must report the selected case count and identities; its blob report
is retained for the standard artifact upload and merged JSON acceptance receipt.
Failures remain failures; cleanup failure stops further service startup. Finite
invocation-wide max-failure/global-timeout policies (including configuration),
project repeat runs, explicit no-dependency execution, last-failed/only-changed or unknown selection modes,
unsupported or missing dependency graphs, teardown projects, commands without
an explicit JSON reporter, and single-cohort selections retain the original
single invocation. Malformed discovery receipts fail closed rather than guessing
which services to start. Dependency-connected projects can retain several
satellites until that whole group completes. The default CI shard uses isolation; local runs keep their
existing lifecycle. This trades additional Playwright/global-setup startup and
occasional satellite restarts for releasing accumulated dev-server heaps earlier.
e2e-satellite-cohorts.test.js verifies shard/filter/setup coverage through public
Playwright commands, multi-service ownership, merged receipts, and failure cleanup;
e2e-owned-process-group.test.js verifies surviving descendant termination. These
focused tests do not establish an exact-commit E2E pass or attribute memory growth
to a particular satellite.
A runner heartbeat continuing while worker ticks stop narrows the problem to the
worker or its output path. Both stopping can also mean host suspension or delayed
log delivery; numeric deltas after recovery help compare those possibilities,
but do not alone establish a root cause. A heartbeat cannot interrupt blocked
JavaScript or guarantee cleanup. Regression coverage in
e2e-runtime-heartbeat.test.js verifies numeric-only logs, preserved command and
failure outcomes, and timer cleanup. Exact-commit E2E remains runtime evidence.
Meet, Lettin, and Parley Cloudflare validation
Meet and Lettin build@tuturuuu/supabase before Vitest because proxy tests import
its Node dist exports. Meet, Lettin and Parley use scripts/ci/release-only-changes.js
to skip PR builds only for version fields, changelogs and release badges; source,
dependencies and missing history still validate. Push/manual behavior is unchanged.
Meet generates build metadata before Turbo/OpenNext: production refs embed
production, other refs embed preview and the checked-out SHA/ref; these inputs
participate in cache keys. Wrangler descriptions do not populate /api/build-info.
Production scripts/ci/verify-meet-build-info.js requires HTTP 200, JSON, appName: meet,
full candidate SHA and production environment/ref before recording success. Fallback,
stale and preview bundles fail closed; main validation never probes production.
Workflow contracts use Bun’s built-in YAML parser via workflow-yaml-test-helper.js,
not undeclared transitive YAML packages. The dependency-free snapshot checks ordering,
identity and caching. Run node --test scripts/ci/meet-build-metadata.test.js scripts/ci/verify-meet-build-info.test.js scripts/ci/generate-build-metadata.test.js.
Meet, Lettin and Parley pin the official OpenNext 1.20.6 registry tarball as a
shared URL dependency. Its URL-keyed Bun patch backports the Cache Components
scheduler and request-scoped module-loading changes from upstream PR 1318 at
6b783939aefa8f9a19d4f002f68bc57c62879098. The older upstream preview rejects Node
middleware; removing that guard does not supply a middleware bundler. Retain the
stable adapter’s real bundleNodeMiddleware path and apply the Cache Components
middleware patch after that bundle. Preserve component names, preview manifests,
and Next 16.4’s CacheSignal(null) constructor arguments.
Keep the registry integrity in bun.lock; packages/meet-core retains its
separate stable-version adapter and patch. Verify installed package isolation
against that independent consumer, not Lettin. Bun’s path-based patch command can
select the package version instead of the URL: verify the generated patchedDependencies key and restore the
stable patch before installing. Set the URL-keyed entry through bun pm pkg set
with JSON input, then verify a frozen install and installed package isolation.
A prepared directory left by bun patch can shadow dependency symlinks; preserve
that owned directory outside the resolver and reinstall before testing.
Meet requires both compatibility fixes: the workerd scheduler prevents the
anonymous login render from hanging, and cache-handler registration must return
before Next’s original dynamic loader accesses removed minified aliases. Otherwise
even the local-metadata /api/build-info handler can fail before it executes.
node --test scripts/ci/meet-cloudflare-render.test.js exercises the installed
Next scheduler transformation, cache-handler method and React resume names.
For retained-artifact diagnosis, preserve the original ZIP and hash; normalize
only CI file imports and Wrangler’s require shim in a private replay. Use the
canonical URL and Host, local bindings, blocked outbound traffic and bounded
requests. Isolate scheduler and cache-loader changes independently. Such a replay
establishes a mechanism; exact-commit CI builds and hosted health remain separate
delivery gates.
When changing the candidate dependency, update both workflow path filters and
tuturuuu.ts selection to its active patch. Selection regressions must derive the
patch from the app manifest and root patchedDependencies rather than an obsolete
filename copied into the test.
Run node --test apps/parley/scripts/opennext-candidate.test.mjs for actual installed
Next CJS/ESM AST-patch compatibility and fail-closed source-shape checks. After the
real CI OpenNext build, Parley runs node --test verify.mjs from
apps/parley/test-fixtures/built-worker. The fixture preserves production compatibility
flags and forwards through the production entry with canonical URL/Host, local assets,
local R2/self bindings and dummy credentials. Global outbound fetch throws; no provider,
production Durable Object or account binding is configured. It requires exact build-info
SHA plus four sequential and four concurrent English/Vietnamese invitation pages,
translated content and complete HTML within bounded deadlines, then closes the harness.
Only hashes and lengths are logged. No release build runs locally. A green fixture
qualifies the anonymous built-page boundary only: authenticated invitation, session,
Live/provider and hosted concurrency acceptance remain required before recovery claims.
Never ship private minified diagnostic patches or infer recovery from HTTP 200 alone.
parley-offline-diagnostic.yaml is manual-only from trusted main, with no deployment
or production secrets. It replays retained artifact 11583133658 from run 37853687763,
fixed to SHA 387caf4872a59cc4944b16324a2554062393d7d1 and its published archive digest.
Expiry, changed provenance, unsafe extraction or missing Linux tool binaries stop it.
It installs frozen tools without lifecycle scripts, never rebuilds the app, and makes
one anonymous /access-denied request inside private mount/network namespaces with
loopback only, local assets/R2/self bindings, dummy credentials and no remote services.
The unit caps aggregate memory/CPU, local state and logs; active work gets 50 seconds
plus 10 seconds for owned process-group cleanup. Missing isolation/caps stop launch.
Only an allowlisted result is retained for one day; request errors or missing HTTP status fail qualification, while a captured HTTP 500 is diagnostic evidence, not health success.
Before retrieval, a finite unit checks memory/CPU/PID caps and private namespaces,
not whole-host boot readiness. Exact digest-bound legacy asset aliases stay inside private extraction without changing ZIP/JS bytes; offline gaps never allow online fallback.
TanStack/Rust Cloudflare deployment
rust-backend.yml is also the manually dispatched Cloudflare preview deployment
workflow for the migration pair:
- Rust backend Worker:
apps/backend/wrangler.jsonc, Worker nametuturuuu-backend. - TanStack Start Worker:
apps/tanstack-web/wrangler.jsonc, Worker nametuturuuu-tanstack-web.
rust-backend.yml entry in
tuturuuu.ts, which is currently disabled while the migration is paused. Pull
request and push runs are validation-only and use
contents: read; they do not require Cloudflare secrets just to type-check,
test, lint config, build the Rust Worker bundle, or build the backend Docker
image.
Manual dispatch inputs:
deploy_target:none,backend,tanstack-web, orall.deployment_mode:dry-runordeploy. Keep the defaultdry-runfor the first run in a new account or environment.
rust-backend.yml from main for Cloudflare deployments. The deployment
preflight requires the dispatch ref to be refs/heads/main and github.actor
to be listed in the comma-delimited TRUSTED_CLOUDFLARE_DEPLOY_ACTORS
repository variable before it loads CLOUDFLARE_API_TOKEN. Keep that allowlist
limited to maintainers who can approve secret-backed Worker deployments.
Cloudflare deploy credentials must not be exposed to arbitrary branch code;
review or merge deployment changes before dispatching a secret-backed run.
The deploy jobs are bound to the cloudflare-workers-preview GitHub
Environment. Configure these GitHub values there before expecting deployment
jobs to run:
If either value is missing, the workflow emits a
Cloudflare deployment skipped warning, records setup guidance in the job
summary, and skips deploy jobs after CI validation has completed. The warning
names only the missing GitHub Environment keys, not their values. A non-empty
but invalid or under-scoped token normally surfaces as a Worker secret preflight
skip or a Wrangler failure before upload.
Worker runtime secrets live in Cloudflare, not GitHub. After the GitHub
credential preflight passes, a dedicated Worker secret preflight checks the
selected Workers with wrangler secret list.
It reads only secret names, never secret values. If Wrangler cannot list names,
or if any required name is
missing, the workflow emits a warning, writes setup commands to the job summary,
and skips the selected deploy job instead of failing later inside a deploy
step. Missing-secret warnings use the Missing Worker secrets wording and list
only secret names plus bun wrangler secret put ... commands; they never print
or infer secret values.
Bootstrap missing Worker secrets from a trusted local shell with Wrangler:
deploy_target is not all.
The TanStack Worker has a BACKEND service binding to tuturuuu-backend, so a
frontend-only deployment assumes the backend Worker and its secrets already
exist. When deploy_target=all, a skipped backend deployment also keeps the
TanStack deployment skipped, even if the TanStack Worker secrets are present.
deployment_mode=dry-run compiles and runs Wrangler checks without uploading,
but it still uses the same Cloudflare account, token, Worker name, and secret
name preflights as deployment_mode=deploy. Treat a dry-run skip as a setup
gap, not as a deploy failure. deployment_mode=deploy uploads the selected
Worker version only after those preflights pass.
When deploy_target=all and deployment_mode=deploy, the
post-deploy-smoke job runs only after both Worker deploy jobs succeed. It
requires BACKEND_WORKER_ORIGIN, TANSTACK_WEB_WORKER_ORIGIN, and
CLOUDFLARE_SMOKE_BACKEND_INTERNAL_TOKEN; missing values fail the full deploy
workflow with a warning, an error, and setup guidance in the step summary
instead of silently skipping runtime verification. The warning title is
Cloudflare smoke inputs missing, and the error title is
Cloudflare smoke verification blocked. The job runs:
tmp/benchmarks/web-migration/<run-id>-<attempt>/cloudflare-smoke.json,
uploaded as the cloudflare-smoke-<run-id>-<attempt> workflow artifact, and
summarized in the GitHub step summary. The report must come from the deployed
backend and TanStack Worker origins; do not point these variables at local
Wrangler or Docker origins in the cloudflare-workers-preview Environment.
Quality and security
type-check.yamlturbo-unit-tests.yamlbiome-check.yamlcodeql.ymlcodecov.yamli18n-check.yamlcheck-migrations.ymlcheck-migration-timestamps.ymlbranch-name-check.yaml
turbo-unit-tests.yaml and codecov.yaml) run bun setup
through scripts/ci/run-with-backoff.sh before executing tests so dependency
installation and workspace package builds match the local setup path while
surviving transient tarball extraction failures. The Codecov coverage test run
also uses the helper with a two-attempt cap so transient runner interruptions
such as exit code 130 retry once without masking deterministic test failures.
Vercel workflows should invoke Turborepo through bun turbo:local ... after the
retried bun install so CI uses the pinned repo dependency instead of resolving
a global or downloaded Turbo binary. Cacheable builds, type checks, and tests
must be executed through .github/actions/run-with-turbo-remote-cache; trusted
jobs pass the dedicated repository TURBO_TOKEN and TURBO_TEAM variable,
while pull-request and Dependabot jobs leave both inputs empty and use the
task-family local fallback cache.
The secretless fallback archives only .turbo/cache under v2 keys partitioned
by OS, architecture, task family, Turbo configuration hash, lockfile hash, and
commit. Restoration tries the exact key, then the same-lock prefix. Only
type-check.yaml opts into a final same-config prefix across lock generations.
Turbo still validates each task hash and both dependency builds and type checks
run; a changed lockfile can invalidate every task even after archive restoration.
Only non-PR, non-Dependabot jobs on the repository default branch save archives.
Feature, production, PR, and Dependabot runs remain restore-only, with GitHub’s
branch cache scope preventing main from restoring feature-produced entries.
The protected remote-token expressions remain unchanged: this does not enable
feature-branch Turbo Remote Cache.
The action exposes cache-hit (exact-key match) and cache-matched-key (empty on
miss). Its job summary distinguishes archive MISS, exact restore, and prefix
restore from actual Turbo task hits. Inspect each command’s Cached/Tasks totals
before claiming reuse. A successful default-branch run must seed a new v2 archive;
deleted archives cannot be recovered by broader prefixes. Before claiming a
speedup, record the exact CI SHA, archive size, matched key, per-task hits, and
elapsed time; archive download cost and retention can outweigh reuse.
Regression coverage: node --test .github/actions/setup-turbo-fallback-cache/action.test.js.
Vercel deploy credentials must stay environment-scoped. Do not put
${{ secrets.* }} values in a Vercel workflow-level env: block, and do not
export production Supabase, encryption, or provider secrets from GitHub
Actions. Store app runtime and build-time configuration in the Vercel project
environment. Remote-cache identity is the narrow exception: it is passed only
to the wrapped vercel build step and never written to GITHUB_ENV or a
workflow/job environment. The regression test
bun test scripts/ci/release-workflows.test.js enforces this for every
vercel-preview-*.yaml and vercel-production-*.yaml workflow.
Cache and artifact resources
actions-storage-report.yaml runs weekly and can be dispatched manually. It
is read-only: it queries the repository’s live Actions cache policy and current
cache/artifact inventories, groups cache bytes by key prefix, and summarizes
artifact count, size, age, and largest workflow families. Cache status is
informational at 80% of the configured limit, warning at 90%, and critical at
100%. The report does not assume a fixed artifact byte entitlement and does not
require an organization billing token.
The July 10, 2026 audit found a 10 GB cache maximum and 7-day retention. Target
steady-state usage below 9 GB so GitHub does not continuously evict and recreate
high-reuse entries. Prefer Bun downloads, package-manager data, task-family
Turbo state, native dependencies, shared Rust state, and service-scoped
BuildKit layers over final application binaries. Leave managed uv and CodeQL
caching to their official actions.
All upload-artifact steps must set retention-days and
if-no-files-found. Optional failure diagnostics may warn; release and deploy
handoffs must fail when absent. E2E diagnostics are failure-only and retained 7
days, package tarballs are retained 1 day, development mobile deliverables 7
days, and production store deliverables 14 days.
Preview deploy credentials must not be exposed to arbitrary branch push code.
Most preview Vercel workflows are manual-only: run the workflow from main,
provide the branch, tag, or SHA in preview_ref, and keep
TRUSTED_PREVIEW_DEPLOY_ACTORS limited to maintainers who can approve
secret-backed preview builds. The workflow still checks out and builds the
requested preview_ref, so reviewers should treat that ref as code that can
execute during install/build.
Automatic CodeQL uses GitHub’s organization-managed
dynamic/github-code-scanning/codeql workflow for JavaScript/TypeScript and
Python. codeql.yml is a manual-only fallback so the repository retains an
explicit workflow file without duplicating managed push or pull-request scans.
Do not add automatic or cron triggers to the fallback; bun git-sync mirrors
the already-scanned main commit to production.
E2E suites reuse only exact same-day success proofs from protected main.
See E2E proof reuse for suite inputs, conservative
fallbacks, diagnostics safety, and inactive database proposal handling.
docker-setup-check.yaml, turbo-unit-tests.yaml, codecov.yaml, and
type-check.yaml use the same gate before their expensive jobs, so version-only
package manifest edits do not rebuild images or repeat unchanged tests and type
checks. Manual dispatch remains a full run.
Supabase migration workflows still consume the required platform workflow-run
signals, but compare the target SHA with the last successful migration marker.
Only database and migration-control changes proceed to Supabase CLI setup and
db push; an unavailable marker fails open. Staging and production each use a
single serialized evaluate-and-migrate job, and production preserves the
same-SHA platform deployment plus staging-success prerequisites.
Cloudflare Worker manual deployments follow the same trust boundary: run
rust-backend.yml from main, keep TRUSTED_CLOUDFLARE_DEPLOY_ACTORS limited
to trusted maintainers, and avoid exposing CLOUDFLARE_API_TOKEN to unreviewed
branch code. The workflow checks the protected dispatch ref and actor allowlist
before loading Cloudflare credentials or running Worker deploy steps.
vercel-preview-platform.yaml also runs on protected main pushes. Keep that
exception narrow: supabase-staging.yaml is triggered by the platform preview
build workflow and production migration requires a successful main staging
migration for the same SHA before it can run supabase db push --include-all.
Other delivery surfaces
discord-modal-deploy.ymlmobile-build-android.yamlmobile-build-ios.yamlmobile-build-macos.yamlmobile-build-windows.yamlmobile-deploy-stores.yamlrelease-*.yamlpackage publishing workflows
Operational Rules
Manual dispatch
Useworkflow_dispatch when:
- you need to rerun a deployment intentionally
- you need to promote a migration outside the normal trigger timing
- you need to recover from a failed but otherwise understood automation path
ci entries, while trusted manual satellite
preview workflows launch their guarded deploy job directly and do not keep an
unused tuturuuu.ts toggle.
For preview Vercel deployments, dispatch the workflow from main, set
preview_ref to the branch, tag, or SHA to deploy, and confirm the actor is in
TRUSTED_PREVIEW_DEPLOY_ACTORS. Do not reintroduce preview push triggers for
secret-backed Vercel workflows, and do not add a standalone check-ci job to a
manual-only preview: the dispatch already selected the build and the extra job
only consumes another runner. Preview concurrency uses the workflow plus
preview_ref, so rerunning the same target cancels stale work without canceling
a different preview target.
Package release workflows that expose production secrets or trusted publishing
authority must add their own ref guard before dependency installation or publish
jobs. Release Please is the only workflow that generates monorepo version and
changelog PRs. It runs from production, uses
secrets.RELEASE_PLEASE_TOKEN so generated PRs and releases can trigger
downstream workflows, and falls back to github.token only to keep the job from
failing when the bot token is not provisioned. The github.token fallback does
not trigger downstream workflow runs from generated release PRs. Release Please
rejects non-production manual dispatches before the write-capable release job.
release-please-transient-retry.yaml retries only the GitHub API disconnect
reported as release-please failed: other side closed. It checks that the run
is still failed on the current production SHA, then reruns failed jobs for at
most three total attempts. Other failures remain visible for investigation;
a manual rerun supersedes a queued automatic retry.
Generated release-note normalization preserves published history and unrelated
feature descriptions. The offline checkout entry from #5734 is a narrow exception:
when both its original commit and merge commit appear in the same unpublished
section, the normalizer retains one description that includes cached images and
all references. Both exact commit identities and titles must match; similar
wording or a lone commit never enables this consolidation. Regression coverage is
in scripts/ci/normalize-release-notes.test.js. Regenerate through the maintained
production Release Please workflow after promoting a generator fix; do not edit
generated changelogs or version fields manually.
Package publish workflows consume release-please version bumps on production;
they must not recreate checksum or PR-title version bump automation. Package
release workflows use npm trusted publishing: build and artifact-pack work runs
before any OIDC permission is granted, then the publish-npm job downloads one
tarball, verifies its package name and version, and runs npm publish without
NPM_TOKEN. Each publish job is bound to its package release environment, and
the matching npm trusted publisher must use this repository, the workflow
filename, and that environment. Manual dispatch is only valid with the branch
selector set to production; non-production refs are rejected before the package
or publish jobs can start. Every workflow-published package manifest must also
declare repository.type: "git",
repository.url: "https://github.com/tutur3u/platform", and
repository.directory matching the package path. npm validates those fields
against GitHub Actions provenance and rejects publishes with E422 when the
packed manifest has an empty or mismatched repository URL. The tarball handoff
uses required-file behavior, one-day retention, and compression level 0 because
.tgz is already compressed. Preparation runs any package lifecycle build
through root Turbo first, then calls npm pack --ignore-scripts so prepack
cannot bypass the shared cache wrapper.
The SDK build embeds only its reviewed portable HTTP-client and task-codec source
closure from the broad internal-api and utils packages. Those packages remain
development dependencies; emitted JavaScript and declarations share rewritten local
dist/_portable imports. dist/portable-sources.json records included source paths.
The build rejects unreviewed modules, private/runtime-specific imports, and
undeclared external libraries. Extend the explicit allowlists only after reviewing
the added closure. Keep the isolated public npm install and compiled CLI probes as
the acceptance gate; source-staging tests do not establish package delivery.
Run node --test packages/sdk/scripts/portable-build.test.cjs for the boundary
regressions, and require exact-commit compilation and install qualification in CI.
SDK qualification has two explicit phases. Only same-repository pull requests from
release-please--branches--production into production select SOURCE. That phase
keeps the serial exact-head compile, portable closure/resource controls, manifest
rewrite and archive inventory/integrity checks. Its receipt has phase: source,
sourceValidated: true and qualified: false: it is source readiness, never public
package acceptance. Ordinary PRs and manual qualification retain the full PUBLIC
phase. This avoids requiring unpublished generated dependency versions before the
release can reach the production-only publishers.
After the existing bounded upstream package readiness gate, SDK production
preparation runs PUBLIC qualification against the exact production SHA: exact
public registry tuples, isolated npm consumer installation and compiled CLI/resource
probes. Runtime, peer and optional workspace dependencies must be publicly available;
embedded development-only dependencies are excluded from registry prerequisites.
Missing dependencies fail closed, including optional dependencies that npm might
otherwise omit. Resolve physical dependency manifests from the installed SDK’s
Node module search paths inside the disposable consumer. npm may place a direct
dependency under node_modules/tuturuuu/node_modules rather than hoist it to the
consumer root. Match the lock entry for that exact physical directory; use the
same verified entry for registry tuple comparison. Reject symlinks in every path
component, missing packages, manifest identity/version mismatches, linked lock
entries and missing or invalid public registry integrity. An invalid nested copy
must not fall back to a valid hoisted copy. The focused
scripts/ci/sdk-installed-dependencies.test.js suite covers these layouts and
failure boundaries without installing packages or evaluating their code.
The preparation uploads the very tarball installed and probed,
along with its receipt. The OIDC job requires a successful PUBLIC receipt for the
same SHA, package name/version and SHA256 bytes before publishing that tarball.
No rebuild or second pack follows qualification. No receipt, a SOURCE receipt,
failed cleanup, changed SHA or changed tarball blocks publication. The existing
bounded publisher defer/dispatch protocol remains the resume mechanism for missing
upstream versions; do not manually publish prospective release versions.
Run node --test scripts/ci/qualify-sdk-package.test.js scripts/ci/package-release-readiness.test.js packages/sdk/scripts/portable-build.test.cjs
for phase, cycle, runtime dependency, artifact binding and portable closure controls.
These injected non-build controls do not replace exact-SHA hosted compilation and
public-consumer acceptance. Release integration and production promotion retain
separate owner approval and exact-commit CI gates.
SDK packed-CLI qualification keeps install failure stream byte counts and SHA256
commitments, without retaining stdout/stderr or the private npm cache. Its receipt
also extracts at most eight package requests from strict npm E404 lines in the
first 4096 bytes of each stream. Only the HTTPS npm public registry origin and
validated package names are allowed; an exact version is included only when a
matching npm package/version line exists. Credentialed URLs, query strings,
fragments, other registries and free-form errors are excluded. A missing identity
or null version remains unknown; a request is not proof the package exists.
Run node --test scripts/ci/qualify-sdk-package.test.js for the diagnostic and
fixture-cleanup regressions. Use the exact-head qualification receipt for new
failures; hashes from older receipts cannot recover discarded npm output.
Test, Coverage, Biome, and Type Check preserve immutable protected concurrency
groups for main, production, and all generated Release Please refs.
A rapid generated branch reset followed by its metadata commit must not cancel
checks for a SHA also used by main or production. Ordinary feature refs retain
branch push supersession; Type Check manual attempts remain independent. Cancellation is not a
successful validation or a valid duplicate skip. Resource-owning E2E and database
workflows retain their own lifecycle policies.
The duplicate-proof runner job is scheduled only for generated Release Please
branch pushes. Ordinary feature, main, production and manual runs proceed directly
from preflight to validation, avoiding three extra runner allocations and checkout/
Node setup stages across Test, Coverage and Biome. A skipped proof job is not a
validation receipt; downstream gates still require the actual checks.
Regression coverage: scripts/ci/generated-proof-scheduling.test.js executes the
real workflow conditions, including skipped/failed proof and successful reuse.
Generated duplicate pushes can cleanly skip these three workflows’ validation
jobs only after scripts/ci/check-completed-validation.js proves a completed
successful main or production push at the exact same SHA. It selects the newest
run per canonical branch from complete history, including pending and failed
runs; a newer non-success run vetoes older success. Test and Coverage
require all four successful shard jobs and their successful aggregate. Biome
requires both successful format and lint jobs. The helper checks the current run
attempt again after reading its jobs. Pending, cancelled, skipped, neutral,
partial, failed, or inaccessible proof never substitutes for validation; checks
run normally when proof is unavailable. A failed proof job (including checkout or
Node setup failure) or missing output also runs validation; dependent gates
bypass the implicit success guard and skip only for a successful proof job with
explicit run_checks=false. Cancellation and shared CI configuration/release
relevance decisions still apply. Main and production always validate
independently. This changes future runs only and does not rewrite historical
cancelled checks attached to a shared SHA.
Biome retains automatic validation on generated release and overflow-notes
branches plus manual dispatch. The CI switchboard checks committed overflow
content before expensive work, and completed exact-commit duplicate proof can
skip validation without creating cancelled checks. Keep generated mutation and
report guards: skipped or cancelled lint/format jobs must not publish a success
report or open a format follow-up. When verifying a release, inspect all check
runs for its SHA as well as branch-filtered workflow runs. Proof queries share a
45-second overall deadline and fail open to normal validation on timeout.
Release Please can move an oversized generated pull request body into
release-notes.md on a companion
release-please--branches--<branch>--release-notes branch. The workflow runs
node scripts/ci/release-please-overflow-recovery.js --target-branch production
before googleapis/release-please-action@v5; when a merged pending release PR
still points at a missing overflow file, the script recreates the branch and
file from the merged manifest bump and current production changelogs, then the
normal Release Please action can create the releases. Missing companion refs are
published only after their notes blob, tree, and final commit exist: recovery
never first creates a ref at the production SHA. This prevents a second notes
push from cancelling duplicate checks on the production commit.
For the exact production overflow-notes ref, CI verifies the committed delta
against production history with scripts/ci/overflow-notes-ci.js. An ancestor
bootstrap with no new work, or a generated notes-only addition/change, skips
heavy downstream jobs through the shared preflight; the inline type-check and
E2E preflight use the same predicate. Source changes, malformed notes, missing
history, ordinary generated release PRs, manual events, main, and production
retain normal validation. Notes refs cannot supersede their own coverage, unit,
E2E, Biome, or baseline runs before this decision. A successful preflight with
skipped downstream jobs means no compilation or runtime validation occurred.
Historical cancelled job conclusions cannot be changed by this fix.
Generated production overflow-notes validation uses a commit-specific concurrency
group. Disabling active-run cancellation alone still lets GitHub replace a pending
run in a shared group; a separate notes group for each SHA preserves the validated
skip result for rapid generated updates. Other branches keep their existing groups.
Regression coverage: overflow-notes-ci.test.js and
release-please-overflow-recovery.test.js cover committed source/content
boundaries and final-ref publication ordering. If the script reports
skipped, inspect the pending labels and merged release PR body before retrying.
The same workflow then runs
node scripts/ci/release-please-auto-approve.js --target-branch production,
which approves the generated release PR while it is still nothing but generated
output. The Protected branches ruleset requires an approving review, so an
untouched release PR otherwise sits at REVIEW_REQUIRED indefinitely. Every one
of these must hold or the script skips and leaves the PR for a human: the head
branch is a release-please--branches-- branch against the target branch, the
PR was opened by a release automation identity, every commit is a
chore(release): release ... commit by that same account, and every changed
file is one release-please itself rewrites. That last allowlist is derived
from release-please-config.json — each package’s changelog, its version file
for its release type, and its extra-files — so adding a package or changing
its release type cannot silently widen what auto-approval accepts.
Nothing needs to un-approve: the ruleset sets dismiss_stale_reviews_on_push,
so any later push to the release branch drops the approval, including one this
workflow never observes. The step deliberately uses github.token rather than
RELEASE_PLEASE_TOKEN, because GitHub refuses a review from the account that
opened the pull request — release-please and the approval have to be different
identities. While RELEASE_PLEASE_TOKEN is unset both are github-actions[bot],
and the script reports blocked and leaves the run green rather than failing.
Use --dry-run to see the decision without posting a review.
Generated release formatting also refreshes Bun workspace-version metadata with
pinned Bun 1.4.1 using --lockfile-only --ignore-scripts. The shared
release-workspace-lock.js validator accepts only workspace version string
changes matching the generated package manifests. It rejects dependency,
resolution, workspace identity, root version and unrelated formatting changes;
every other lockfile byte must remain identical. Auto-approval independently
loads complete lock/package files at immutable base/head SHAs and reruns that
validation; bun.lock is never an unconditional generated-path exception.
The working-tree guard includes nonignored untracked files and rejects authored
spill without staging it. For files above the Contents API inline-body limit,
approval reads object metadata at the immutable commit/path and retrieves the
matching Git blob through
the same repository API. It ignores response-provided download URLs, caps file
size at 16 MiB, and verifies complete canonical base64, byte size, Git blob SHA
and UTF-8 before validating the lock. Missing or mismatched bodies fail closed.
Run this refresh and the merge helper in CI; the merge helper invokes broad
checks and must not run on agent machines where local builds are prohibited.
Formatting amends the original release commit without resetting its author and
pushes with an exact lease. Approval is pinned to the validated head. If refs
move or validation fails, regenerate/review the release instead of publishing
an ordinary authored repair commit. Published notes stay unchanged; ancestry
alone does not prove differently worded features are duplicates.
release-please-auto-merge.yaml closes the loop on the generated release PR. It
runs daily at 00:00 UTC (cron: "0 0 * * *"), which is 07:00 in Vietnam
(UTC+7), and on manual dispatch. It does not reimplement the merge in YAML: it
runs bun git-release-please and then bun git-sync, the same two commands the
release flow uses locally, so the scheduled path and the manual path cannot
drift. Those scripts own the merge rules — platform version sync, conflict
detection, bun check, and fast-forward safety — and are covered by
scripts/git-release-please.test.js and scripts/git-sync.test.js.
The generated PR must have passing checks, no unresolved active review threads,
and a 5-minute quiet period after its latest update or generated commit. This
is checked before dependency setup and again immediately before merging the
pinned generated head. Failed PR checks stop the daily run for investigation.
After local validation, it pushes the release merge to main and waits for all
other workflows on that exact SHA to finish successfully. A failed, missing, or
unfinished check blocks production. The waiter observes the same green run set
twice and rejects a changed main head. Production sync uses
bun git-sync --current-branch --only-branch production, pinning the validated
commit instead of silently adopting newer work. The run summary distinguishes
planned work, a completed local merge, dry runs, and completed promotion.
Both initial validation and advanced-main retries derive their build filters from
package.json’s canonical setup script through scripts/ci/build-release-setup.mjs.
The helper parses arguments without executing setup shell text, and both paths
use the same step-scoped Turbo remote-cache wrapper.
If another PR lands on main during release validation, the push step fetches
and merges the new head, reinstalls its frozen dependency graph and rebuilds the
dist-only setup packages before rerunning bun check. Affected mobile changes
also refresh flutter pub get before mobile checks. The push retries up to four
times. An install or setup failure stops the retry; checks must never validate a
new source commit against dependencies installed for an older checkout. A merge conflict or a push error unrelated to
main advancing fails the run. The exact-SHA CI gate still runs after the push.
Validation runs at two Turbo tasks and two Vitest workers, with the worker limit
preserved through Turbo’s strict environment. SDK resource-admission tests start
real child processes and use bounded integration-test deadlines, rather than the
five-second unit-test default that repeatedly failed on hosted runners. These
limits retain the serialization, nested-command, cancellation, and surviving-child
assertions; they do not skip tests or retry a failed merge blindly.
The workflow resolves what to do before it installs anything. It prefers
origin/release-please--branches--production, ignores the companion
--release-notes overflow branch, and skips the merge entirely when the release
branch is already an ancestor of origin/main — so a run with nothing to merge
is a clean no-op rather than a red run. It separately checks whether
origin/production has drifted from origin/main and syncs even when there was
no merge, which is what makes the schedule self-healing. Every run that pushes
ends by asserting the two refs resolve to the same commit and fails loudly if
they do not.
An empty daily run remains harmless because the workflow exits cleanly when
there is no generated release branch to merge and the protected branches are
already aligned.
The job checks out main with fetch-depth: 0 and
secrets.RELEASE_PLEASE_TOKEN, falling back to github.token. The fallback
cannot complete a release: the Protected branches ruleset covers main and
production and its only bypass actor is OrganizationAdmin, so a run on
github.token merges cleanly and then dies on
GH013: Changes must be made through a pull request. RELEASE_PLEASE_TOKEN
must therefore hold a personal access token (repo + workflow scopes) owned
by an organization admin. Granting the Actions app a ruleset bypass is not
an alternative: pushes made with GITHUB_TOKEN do not trigger push
workflows, so the production push would silently skip
Production Deployment Planner and the next Release Please run and
stall the release chain. A run that would need to push checks for the secret up
front and fails with an actionable error, rather than discovering it after
bun check.
The job also installs Flutter (3.47.x, the same pin as mobile.yaml) and runs
flutter pub get in apps/mobile before the merge. When a release includes
mobile changes, touchesMobile() in scripts/git-release-please.js follows
bun check with bun check:mobile; without a toolchain dart-format,
flutter-analyze and flutter-test all exit 127.
Keep every mobile build/deploy workflow on that same Flutter minor. The mobile
lockfile is resolved with Dart 3.13; Flutter 3.44.9 bundles Dart 3.12.2 and
downgrades 16 locked packages during flutter pub get, which dirties the
checkout before bun git-release-please can begin.
Top-level permissions is {} and
contents: write is granted on the job alone, and concurrency is grouped with
cancel-in-progress: false because cancelling a half-finished release merge
would leave main and production split. Use the dry_run dispatch input to
exercise the merge and run bun git-sync --no-push without publishing anything.
Keep internal Tuturuuu dependencies as workspace:* in source package
manifests so local builds always use the checked-out workspace. Package release
workflows must first run
node scripts/ci/package-release-readiness.js gate-package-release packages/<name>
in a short pre-build gate. The gate checks the package version and publishable
Tuturuuu workspace dependencies once, dispatches missing dependency workflows,
and exits green without build, pack, or publish work while dependencies are
pending. If a dependency workflow for the same production SHA already failed,
completed successfully without npm visibility, or cannot be inspected, the gate
fails immediately. When the gate outputs should_publish == true and
dependencies_ready == true, the workflow can build and rewrite the checked-out
manifest immediately before npm pack with
node scripts/ci/prepare-npm-package-manifest.js packages/<name>. That
temporary rewrite replaces workspace: protocol ranges with the current
workspace package versions so npm consumers can install the tarball. It must
preserve package-included file: tarball dependencies, such as @tuturuuu/ui’s
vendored SheetJS tarball, instead of rewriting them to mutable external tarball
URLs. After
npm publish, the publish job polls npm view for the exact published version
before it reports success, then a separate non-OIDC job dispatches direct
dependent package workflows without checking out the repo or carrying publish
authority. The internal
node scripts/ci/package-release-readiness.js dispatch-dependent-workflows packages/<name>
command is available for direct/manual dispatches that can read the checkout.
If npm returns first-publish or permission errors, fix the npm package access and
trusted publisher setup instead of skipping the package. If a published package
depends on another release-please-managed Tuturuuu package, that dependency also
needs its own
release-*-package.yaml workflow and matching tuturuuu.ts entry.
The platform production Vercel workflow also runs
node scripts/ci/package-release-readiness.js gate-changed-package-versions
before installing dependencies. Release-please package bumps therefore publish
and become visible on npm before the normal production platform deployment
continues. The helper intentionally checks only the checked-out latest commit,
not the whole push event payload, so package changes from earlier commits in a
multi-commit push are not bound to an unrelated newer SHA. If the latest commit
does change package manifests, the deploy job grants actions: write because
the helper may dispatch missing package release workflows for that same
production SHA. When related package releases are queued or running, the helper
sets packages_ready=false; the platform build skips successfully instead of
polling npm until the runner times out. npm publish authority remains isolated to
package publish-npm jobs. A skipped package gate must not count as production
deployment for database migrations. For commit-driven selection, the planner checks database-affecting changes against the independent last successful supabase-production.yaml migration baseline, not the Platform deployment baseline.
When Platform lacks successful explicit coverage of the target SHA, pending migrations or unavailable migration coverage conservatively select enabled Platform for redeployment; disabled configuration remains authoritative. Ordinary manual dispatch still selects all enabled targets.
Selection retains the successful planner, explicit Platform deployment marker, and exact-SHA completed successful staging requirements. Manual production migration recovery may explicitly set allow_cloudflare_verification_failures=true with full matching expected_sha and secret-matched expected_project_ref: only completed planner failures in the three canonical Cloudflare verifier steps qualify, with all three validations and Platform deployment successful and every other job terminal acceptable. This default-off exception never applies to automatic triggers and never bypasses the current Platform marker, staging gate, normal project secrets or supabase db push --include-all; inspect actual application and catalog afterward, without manual SQL.
A registered 15-minute report automation cron is not proof that delivery RPCs are available or email gates are enabled. Missing RPCs remain operational blockers; additive migrations and auto-send toggles do not backfill already-approved reports. Each ready run drains at most five email batches of 20 (100 handled rows) at concurrency four, admitting new batches only within a 30-second budget; already-claimed sends settle before returning, then one generation batch runs. processedEmails counts handled queue rows, not Sent or inbox deliveries; emailBatches and emailDrainStopReason distinguish empty queue, batch limit and admission budget (or missing completion migration). An empty/short claim describes currently eligible rows, not every deferred, locked, blocked or unknown-outcome report.
The marker must explicitly name the platform workflow, deployment kind, and
matching payload SHA, and its latest status must be successful. GitHub
environment records with an empty payload only describe job completion; they
are not deployment evidence. In particular, a package-deferred platform job can
succeed without deploying the API and must never unlock contract-changing
migrations.
production-package-resume.yaml listens to completed package releases and successful planner
completions. A failed package-workflow tail still requires live completion agreement
and every exact package version on npm; its conclusion alone proves no publication.
It resumes the owning planner for the exact current production SHA
only when the prior platform job explicitly deferred for package publishing and
all changed package versions are visible on npm. It checks the live completion,
fully paginates planner runs/jobs, and skips active, failed, promoted, or stale candidates. Planner completion also wakes recovery when the
last package became visible before the original planner finished. The dispatch
carries expected_sha; the planner skips a recovery if production moves before
it starts. Recovery uses commit-driven deployment-marker and affected-path
selection, including the Cron Control and Devbox Control Workers. An exact-SHA successful deployment marker skips its already covered target,
including Platform, only when its payload explicitly identifies the workflow,
deployment kind, and production ref. Automatic job environment records, build
markers, and a bare marker-SHA override cannot establish recovery coverage. An
older Platform marker still permits package recovery. A known marker
with no affected Worker paths skips them; a missing marker still fails open.
Ordinary operator dispatches continue selecting all enabled targets.
Every snapshotted planner’s attempt, status, and conclusion must remain unchanged;
the latest job evidence and run attempt are checked again before reserving an
intent. A completed rerun with the same run ID therefore blocks stale recovery.
Satellite deployments still require the normal verified platform
promotion marker. Completion events use queue: max with cancellation disabled,
so up to 100 completions wait instead of replacing the single pending event;
GitHub cancels overflow beyond that limit. API and npm calls are bounded to
15 seconds each with a four-minute recovery deadline, below the five-minute
job limit. Missing npm visibility and registry read failures retry every two
seconds within that shared deadline, including the final package completion.
Exhaustion fails the recovery job explicitly before creating an intent or
requesting dispatch; it never reports a successful deferred skip. Planner runs
are rechecked after registry reads and immediately before dispatch. GitHub API
timeouts, unreadable/incomplete results, and changed candidates never retry
intent or dispatch writes.
Before dispatch it reserves a durable exact-SHA intent in the dedicated
production-package-resume deployment environment with auto_merge=false,
empty required contexts, production_environment=false, and
transient_environment=true. This is a dispatch lock, not deployment or
migration evidence: no success status is emitted and the real platform marker
still uses vercel-production-platform. Each dispatch passes the returned reservation ID as resume_intent; the planner’s
run-name binds that ID and exact SHA. An existing intent blocks subsequent
completions until its uniquely bound planner is proven completed successfully,
package-deferred, and unpromoted. If npm visibility failed again, a later completion
may reserve a new intent only after all prior intents have that proof, current
versions are visible, and fresh intent/run/attempt/job/ref snapshots agree.
Legacy or unbound intents, absent runs, failed, uncertain, or timed-out requests
remain blocked for operator inspection. Intents are never deleted; the exact-SHA
recovery concurrency group serializes reservation, and a changed intent set blocks
the next write. This
requires deployments: write only in the recovery workflow.
A deferred successful workflow is not deployment evidence.
Validate this guard with node --test scripts/ci/resume-production-package-deployment*.test.js.
The gate queries the planner run because reusable app workflows do not create
standalone Actions runs.
Filtered production Docker installs only include the selected app and the
dependencies that app needs. Workspace packages with direct tsc build scripts
must declare typescript in their own devDependencies; do not
rely on the root devDependency for production Docker builds. Programmatic
compiler API consumers must stay on the active TypeScript 7 toolchain instead
of carrying legacy compiler compatibility packages.
Next.js production apps also declare @typescript/native-preview in their own
devDependencies while the repo uses the TypeScript 7 native compiler package.
The TS7 typescript package does not ship the legacy
typescript/lib/typescript.js API file that Next checks during build-time
TypeScript setup; the native-preview marker makes Next use its supported TS7
native compiler path instead of trying to auto-install classic TypeScript.
Superseded-run guards
Production Vercel workflows use a static per-app concurrency prefix and Git ref with acancel-in-progress predicate restricted to push events on
refs/heads/production. A newer production push cancels only the older run for
that same app instead of cancelling sibling app deployments or allowing an
obsolete run to finish green. A main commit or manual recovery dispatch does
not cancel the running production deploy. The newest production run stays queued
or active and deploys the cumulative branch state.
Deployment markers still let affected-app gating compare every change since
the last successful deployment, including changes accumulated across canceled
intermediate runs. If the marker is missing or cannot be trusted, Vercel gating
defaults open so a later unrelated commit cannot hide an earlier app change.
Prerequisite guards
supabase-staging.yamlrequires a successfulmainplatform preview build trigger unless manually dispatched.- Its deploy step links the staging project and runs
supabase db push --include-all. supabase-production.yamlruns after the production deployment planner and also re-evaluates the current immutableproductioncommit after any successfulmainstaging migration. The staging event’s incoming SHA may be newer and unpromoted, so it does not become the production migration target. Target resolution precedes checkout and the pending-migration range/configuration checks. Manual dispatch remains restricted toproduction; planner completions remain pinned to their successful production run’s SHA. The workflow still requires that target’s successful planner result, production platform deployment marker, and completed successfulmainstaging migration before runningsupabase db push --include-all. Failed, cancelled and other non-success workflow callbacks skip before allocating the evaluation job and use a unique ignored-run concurrency group. Successful callbacks from mismatched workflow/branch pairs and manual dispatches outsiderefs/heads/productionare isolated the same way, so they cannot replace a legitimate pending migration. Only successful production-planner/production and staging/main callbacks, plus manual production branch runs, retain the global migration group with active-run cancellation disabled.
Mobile iOS native assets
Mobile workflows cache the Flutter SDK/pub downloads through the Flutter setup action. Android workflows also use the toolchain-aware Gradle cache, and iOS or macOS workflows cache CocoaPods downloads plus the trunk repository. They do not cacheapps/mobile/build, final APK/AAB/IPA/app bundles, Pods, or
apps/mobile/.dart_tool. Flutter native asset hook outputs can become stale
across simulator/device builds and SDK updates; a stale .dart_tool cache can
leave NativeAssetsManifest.json referencing objective_c while
build/native_assets/ios/ is missing the generated framework.
Every workflow still runs flutter pub get and its platform build on every run.
Development deliverables are retained for 7 days and production store
deliverables for 14 days. Archive directory-based app bundles before upload,
disable redundant artifact compression for already-compressed archives and
mobile packages, and fail if a required deliverable is missing.
The workflow explicitly runs flutter config --no-enable-swift-package-manager
after Flutter setup. Keep the iOS CI path on CocoaPods while image_cropper and
dkimagepickercontroller resolve incompatible TOCropViewController Swift
package ranges; otherwise Flutter 3.44+ can fail before the simulator build
with an Xcode package dependency resolution error.
The iOS/macOS build workflows currently resolve macos-latest to GitHub’s
macOS 26 ARM64 image with Xcode 26+. The former Xcode 16.4 compatibility pins
for connectivity_plus and device_info_plus are no longer required. Before
adding a future Apple-specific dependency pin, verify the active runner image
from a build log and reproduce the native build failure on that toolchain.
Mobile store beta deployment
mobile-deploy-stores.yaml runs automatically on production pushes that touch
apps/mobile/**, scripts/mobile-deployment/**, the workflow file,
ci-check.yml, or tuturuuu.ts. It is a beta-store workflow only:
- Android builds the production flavor AAB from
apps/mobile/.env.githuband publishes it to the Google Playinternaltrack. - iOS builds the production flavor IPA from
apps/mobile/.env.githuband uploads it to TestFlight. - The iOS job verifies the signed local IPA before upload. It also tries to retain an IPA artifact for debugging, but a GitHub artifact storage timeout does not stop the TestFlight upload or beta-group verification.
- CI requests a signed GitHub OIDC token for the
tuturuuu-mobile-deploymentaudience, then fetches an Android or iOS bundle fromhttps://infrastructure.tuturuuu.com/api/v1/mobile-deployment/bundlewith the issuedMOBILE_DEPLOYMENT_CI_TOKEN. - If
MOBILE_DEPLOYMENT_CI_TOKENis not configured in themobile-store-betaGitHub Environment, the credentials preflight fails. Android and iOS publish only after that preflight succeeds. Do not pass a credential-presence flag as a job output: GitHub may redact it as secret-like, silently skipping both jobs. An invalid token fails during the bundle fetch. - The apps/web mobile deployment vault verifies the CI token, GitHub OIDC
issuer, audience, repository,
productionref, workflow file, andmobile-store-betaenvironment before returning any plaintext. - CI hydrates ignored Firebase, signing, store, and
.env.githubfiles into fixed paths underapps/mobileand$RUNNER_TEMPwithumask 077, validates plaintext SHA-256 hashes, masks secret values, and deletes generated files inalways()cleanup steps. - The workflow rejects any Google Play track other than
internal, uploads only the AAB/IPA build artifacts, and uses TestFlight-only iOS upload. - The workflow uses the greater of the
apps/mobile/pubspec.yamlpatch and one patch above matching iOS pre-release versions in App Store Connect. With0.11.0+84and no matching history, the first upload is0.11.0; subsequent uploads advance to0.11.1, then0.11.2. An unused higher source patch is retained. It calculates a separate build number fromGITHUB_RUN_NUMBERandGITHUB_RUN_ATTEMPT; both stores receive the same version name. The iOS publish job resolves that version after its single vault bundle fetch; Android waits for the nonsecret version output. Do not fetch the iOS bundle in preflight as well: the vault rejects a second fetch for the same run attempt as a replay.
mobile-store-beta GitHub Environment. Keep that
environment restricted to the production branch and store only
MOBILE_DEPLOYMENT_CI_TOKEN there. Mobile signing, Firebase, store API, and
build-time secrets live in the root workspace mobile deployment vault at
/internal/mobile-deployment on the infra app and require
manage_mobile_deployment_vault. Manage non-file values in the vault’s
Secrets panel, then use the vault’s Verify action before activating the draft. Legacy
base64 file payload env keys are intentionally blocked; Firebase, signing, and
store credential payloads must be uploaded as file resources.
Local release-build verification should run before relying on the workflow:
xcodebuild -version reports that Command Line Tools are selected, switch to
full Xcode before claiming the iOS archive has been verified locally.
Fast Triage Checklist
- Check whether
ci-check.ymldisabled the workflow throughtuturuuu.ts. - For Vercel workflows, inspect the
ci-check.ymldecision reason and matched paths. - Inspect the changed-file source, base SHA, head SHA, and path count emitted by
resolve-changed-files.ts. - Check whether the workflow was canceled because a newer run superseded it, then follow the newest run in the same workflow-and-branch concurrency group.
- Check whether the prior successful GitHub Deployment marker exists for that Vercel workflow and branch.
- Check path filters to confirm GitHub should have started the lightweight workflow.
- For database workflows, inspect the prerequisite evaluation job before the deploy job.
- Docker setup CI is paused; retained Docker steps are not current release gates.
When You Add New Automation
- Add the workflow file under
.github/workflows/. - Add its key to
tuturuuu.tsunless you intentionally want it always on by default. - Document the workflow in
apps/docs. - If it changes a docs-visible page, add that page to
apps/docs/docs.json.
Calendar retained generation rollout
Thecalendar-generation-contract.yaml workflow applies every migration in a
throwaway local Supabase project, runs strict Calendar pgTAP fixtures, generates
RPC types from that database, and removes its own project. Its artifact names
include the exact source SHA. Run the same finite contract with
node apps/database/scripts/verify-calendar-generation-contract.mjs only when
local Docker is healthy and the resource queue admits the validation.
The compatibility guard migration must reach production before deploying any
Calendar code that consumes the provider operation ledger. First ship the guard
without a ledger or feature activation, verify the automated production migration,
then ship dependent Calendar changes. App deployment precedes production
migrations; an absent RPC or API schema cache is not proof that a ledger is absent.
The guard checks actual PostgreSQL relation presence and fails closed on partial
or incompatible ledger schemas. Never replace a missing guard with an unguarded
provider write.
i18n PR report evidence
The i18n report requires both a completed job result and its explicit validator output. Cancelled, skipped, missing-output, and contradictory results are reported as incomplete; they establish neither a translation defect nor passing validation. Only confirmed validator failures offer repair commands. Reports identify the source head and workflow run. Comment writes are serialized per PR, paginate all existing comments, reject superseded heads, and preserve newer run/attempt reports. A branch can still advance after the final API head check; the embedded source identity makes that evidence boundary visible. Existing cancelled job conclusions are unchanged. Regression coverage:node --test scripts/ci/i18n-report.test.js.
Pending concurrency, skips and exact-head receipts
cancel-in-progress: false protects a running workflow; it does not preserve the
default single pending slot. A new run in the same group replaces the previous
pending run. GitHub now supports queue: max for up to 100 pending runs when
cancellation is disabled. See the official concurrency syntax.
This is an available mechanism, not proof that every repository workflow uses it.
Choose the group from the workload: validation can use workflow/event/commit
identity; deployments that mutate the same environment still need serialization.
Do not make deployment groups unique merely to remove cancellation badges. A
valid duplicate should reach the repository’s proof-based skip decision. Failure,
missing outputs or unavailable proof must run validation rather than silently
skip it. Queue saturation can still cancel new runs; inspect the recorded reason.
For an integration receipt, query the full exact SHA directly:
head_sha, include late fan-out
and separately inspect deployment/migration markers. A queued, cancelled or
missing required workflow is not a successful delivery receipt.
Diagnose image uploads before widening browser deadlines
A small PNG that has a valid signature and dimensions can still contain a bad chunk checksum. Client-side optimization may fail during pixel decoding before an upload-ticket request is sent. Trace the last entered browser operation and whether the ticket endpoint was requested, then decode the exact synthetic fixture with the same image-processing boundary. Do not infer a server upload failure from an absent request or simply widen the browser assertion timeout. The shared fixture inapps/web/e2e/helpers/profile-media-fixture.ts is exercised
by scripts/ci/profile-media-fixture.test.js: pixel decoding and bounded WebP
optimization succeed, while a corrupt checksum fails. These checks establish the
fixture contract, not browser or provider upload success. Confirm the real
request -> upload -> save -> reload path separately.
Preserve causal E2E failures and owned process lifecycle
Public Playwright discovery receipts must preserve project dependencies and the original test selection. Diagnostic stdout may surround JSON; parse a validated report shape, and retain a safe original-selection fallback when discovery is unavailable. Never evaluate private Playwright bundles to reconstruct behavior. Detached satellite process groups require explicit cleanup on timeout and runner signals. Track only groups the current invocation created, remove them only after cleanup is confirmed, and preserve the runner’s signal exit semantics. Childexit alone does not establish that descendants or stdout have closed. Wait for
close when checking terminal output and wait for a fixture’s readiness receipt
before testing its TERM handler under concurrent load.
When report merging also fails after a failed cohort, preserve the original test
failure as the primary cause and attach the merge failure as diagnostics. A
standalone report-merge failure remains a failure. Test these lifecycle boundaries
with real owned child processes and synthetic failures; helper-only source checks
are not proof that a stalled browser process exits.
Preserving protected-branch pending runs
Disablingcancel-in-progress protects the active run, but GitHub’s default
concurrency queue still replaces its single pending run. Validation on main,
production, and generated Release Please refs therefore includes the source SHA,
event name, and run ID in the workflow concurrency group. Existing duplicate-proof
jobs decide whether validated work can be skipped; concurrency cancellation is not
validation evidence. Ordinary PR and feature refs retain latest-run supersession.
Mutating production planners, Vercel and Cloudflare deploys, database migrations,
package publication, Release Please, and store deployments retain their shared
serialization groups with cancel-in-progress: false and queue: max. The
GitHub concurrency reference
allows up to 100 pending runs. Queue overflow can still cancel excess runs; inspect
that condition explicitly instead of interpreting a cancellation as a valid skip.
Queue order follows when each run began waiting, not commit chronology; existing
SHA relevance and deployment-marker guards remain responsible for target validity.
queue: max must never be combined with cancellation enabled.
Regression coverage: scripts/ci/protected-workflow-concurrency.test.js reads the
actual workflow concurrency blocks, checks protected SHA/run separation and PR
supersession, and enforces serialization with expanded queues.
Calendar scheduling startup requests
The Calendar priority panel reads per-user scheduling summaries through the Tasks batch API, using at most 100 distinct task IDs per request and sequential chunks. This replaces one HTTP request per task. The verified actor, calendar workspace, and source-task membership are checked before settings or event data are returned; personal mode counts events only in the personal calendar. Event pages must all succeed before a snapshot is published. Linked and directly associated events are deduplicated by task and event ID. The client retains the last snapshot during transient revalidation, clears visible summaries on definitive access denial, and refreshes stale data on window focus. This scope is scheduling summaries, not the whole Calendar startup waterfall. Regression coverage lives intask-schedule-batch.test.ts and
tasks-scheduling-batch.test.ts. Measure authenticated navigation and meaningful
content after the exact source is deployed; local checks alone are not latency proof.
Devbox Control Vite-native artifact
Devbox Control owns a private workspace with Vite+ and scoped Oxlint/Oxfmt configuration. The workspace pins Wrangler 4.149.0 so local runtime checks support the retained compatibility date; root Wrangler is not the runtime oracle for this scope. The Cloudflare Vite plugin emitsapps/devbox-control/dist/worker
from the existing Wrangler configuration; Worker name, account, compatibility,
RunnerWake class/binding/migration and required secret names remain the source
configuration’s authority. No Supabase auth, runner membership, lease, heartbeat,
notification or WebSocket policy changes accompany tooling adoption.
CI generates Worker types, checks TypeScript and scoped Oxc rules, and runs the
existing 13 authorization/protocol tests through Vite+. Before and after the
real Vite build, node --test apps/devbox-control/scripts/verify-worker-runtime.mjs
checks complete health JSON and rejects unauthenticated runner/internal requests
in a loopback Worker with disposable secrets. Set DEVBOX_TEST_CONFIG to an
absolute emitted dist/worker/wrangler.json path for artifact verification.
Dry-run bundling and the authorized production job consume that emitted config;
secret provisioning still uses the unchanged source target identity.
The fixture closes its temporary runtime and does not contact Supabase or hosted
runners. It proves Worker startup and denial-before-remote-access, not successful
real-account authentication, RunnerWake WebSocket delivery or hosted lease
transitions. Those remain separate acceptance evidence before canonical migration.
Release builds run only in exact-commit CI. Locally, queue lint/types/unit/runtime
checks with ttr resources run; use a private writable TMPDIR when shared temporary
storage rejects writes. The format command checks only owned source and remains
read-only; existing Biome gates remain authoritative until formatter/rule parity.
This tooling unit adds no alarm, queue, retry chain, storage operation or scheduler.
Review background-work and operation-cost bounds separately when those paths change;
health and build success do not establish a spending cap or production promotion.
RunnerWake notification operation envelope
The current RunnerWake class performs no alarm, queue or cron rescheduling. A successful internal POST enumerates all retained WebSockets and sends one wake to each. ForS retained sockets and N duplicate notifications, count N × S send
attempts; failed sends also attempt a close. No durable progress receipt suppresses
later duplicate fanout. A throwing close aborts remaining delivery and propagates
failure rather than returning a complete acknowledgement.
The public notify route authenticates the internal token before one run lookup,
an optional lease lookup and one Durable Object POST. Connect authenticates the
runner before forwarding the WebSocket upgrade. The object trusts these forwarding
boundaries; it does not establish an independent durable stop fence or socket lease.
These per-request counts do not bound retained sockets, repeated ingress, billed
messages, downstream query rows or total account spend.
apps/devbox-control/src/wake.test.ts counts 98,304 sends for 4,096 synthetic sockets
and 24 duplicate/reconstructed notifications, then checks failed-send closure and
incomplete fanout on a throwing close. These tests use the actual class with a
mocked Durable Object base and sockets. They do not verify hosted WebSocket delivery,
provider billing, authenticated Stop or real persisted leases. Before amplifying
this path, require bounded per-wake work, an expiring recurring authority and a
durable authenticated stop fence, preserving recovery and notification semantics.
Native Vite+ test and coverage version alignment
Vite+ 1.1.0 requires Vitest and coverage-v8 5.0.3 together. Pin the root dependency, rootvitest override and provider with Bun; an older root override replaces
Vite+‘s declared runner even when ordinary tests pass. The coverage guard rejects
that mismatch rather than producing unreliable coverage. Shared CI forwards
--maxWorkers=2 and optionally --coverage: keep worker/pool defaults in the
unit config, not a duplicate CLI flag in package scripts. Validate both forwarded
commands under Node 24 in the owning workspace through ttr resources.
Playground metadata lifecycle evidence
The disposable Playground acceptance job opts into the SDK’sTTR_DOCKER_METADATA_DIAGNOSTIC=1 diagnostic. See the
SDK diagnostic contract.
Read each ttr-docker-metadata-v1 record with its exact commit, job and invocation
ordinal. Event times describe JavaScript observations on the same existing
metadata subprocess; null close timing does not prove that the process failed to
close later. Callback lateness measures delay relative to the scheduled deadline.
There is no extra Docker probe, warmup, retry or deadline increase.
An instrumented passing run provides observability only: it cannot explain a
historical timeout. A timer callback proves deadline classification before
settlement, not daemon failure or a Node/Bun adapter defect. Keep causal diagnosis
UNKNOWN until same-invocation evidence distinguishes those hypotheses; source
review and mock tests do not qualify hosted runtime acceptance.