# CI/CD Pipeline Optimization
Source: https://docs.tuturuuu.com/build/development-tools/ci-cd-optimization
Centralized affected-path gating and smart caching to minimize unnecessary CI/CD runs.
**Prerequisite**: Read the [CI/CD
Pipelines](/build/development-tools/ci-cd-pipelines) page to understand the
basic workflow structure.
## Overview
Our CI/CD pipelines avoid wasted builds, tests, and deployments through a
**centralized affected-path gate** plus **build caching**. Instead of every
workflow re-implementing its own change detection, automatic deployments share
the same TypeScript decision function while avoiding a separate configuration
runner when the decision can be made inside an already-required job.
This page documents how that gate works, how the per-app Vercel workflows are
wired, and how to debug skip decisions.
The platform is mid-migration from `apps/web` (Next.js, port `7803`) to
`apps/tanstack-web` (TanStack Start) plus `apps/backend` (Rust, port `7820`).
See [TanStack + Rust
migration](/platform/architecture/tanstack-rust-migration). The optimization
model below applies to both stacks: every Vercel app target is gated the same
way, regardless of framework.
## The Problem
Before centralized gating, workflows ran on every push regardless of:
* Whether files relevant to that specific app or package actually changed
* Whether the change only touched an unrelated app
* Whether build artifacts were already cached
With a large monorepo of apps and shared packages, this meant a single commit to
one app could trigger dozens of unrelated build and deploy workflows, slowing
feedback and congesting the Actions queue.
## How Gating Works Today
Vercel affected-app gating does **not** rely on GitHub's native
`on.push.paths` filters. Path filters are coarse (they cannot understand
workspace dependency graphs), so the Vercel logic stays centralized. A narrow
exception is the external-app compatibility build: its contract is simply
"`apps/external` or any internal package changed," so native trigger paths can
avoid creating a runner at all for unrelated app-only commits.
The shared logic is organized in four pieces:
1. **`tuturuuu.ts`** — the source of truth. It exports a `ci` toggle map (each
workflow can be globally enabled/disabled), the `vercelWorkflowTargets` table
that maps each app to its preview/production workflow and package name, and
the `getWorkflowDecision()` function that decides whether a given workflow is
affected by a set of changed files.
2. **`scripts/ci/resolve-changed-files.ts`** — resolves the changed-file set for
the current event (push payload, pull-request base/head, or git diff against
the last successful deployment marker).
3. **`.github/workflows/ci-check.yml`** — the reusable gate for non-Vercel
workflows that still need an individual `should_run` output.
4. **`.github/workflows/vercel-production.yaml`** — the single production
planner. It resolves every app against that app's last successful deployment
marker in one runner, then calls only affected production workflows as
reusable jobs in the same commit-associated push run.
### Reusable and inline gates
Non-Vercel gated workflows can declare a `check-ci` job that calls
`ci-check.yml`, then gate their real job on the result. Platform preview is
different: it must complete on every protected `main` push so
`supabase-staging.yaml` receives its workflow-run signal. Its deploy job checks
configuration and affected paths inline, then skips install/build/deploy steps
when the app is unaffected. This keeps the workflow signal while using one
runner instead of a configuration runner followed by a deploy runner.
```yaml theme={null}
jobs:
Deploy-Preview:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: actions/setup-node@v7
with:
node-version: 24
- name: Compute changed files
id: changed_files
run: node --experimental-strip-types scripts/ci/resolve-changed-files.ts
- name: Check preview configuration
id: check_config
run: node --experimental-strip-types scripts/ci/check-workflow-config.ts
```
Internally, `ci-check.yml` uses a sparse checkout (only the manifests and
scripts it needs), computes the changed files, and runs the decision script:
```yaml theme={null}
- name: Compute changed files
id: changed_files
env:
GITHUB_TOKEN: ${{ github.token }}
WORKFLOW_NAME: ${{ inputs.workflow_name }}
run: node --experimental-strip-types scripts/ci/resolve-changed-files.ts
- name: Check Configuration
id: check_config
env:
CHANGED_FILES_FILE: ${{ steps.changed_files.outputs.changed_files_path }}
WORKFLOW_NAME: ${{ inputs.workflow_name }}
run: node --experimental-strip-types scripts/ci/check-workflow-config.ts
```
The scripts run with Node's `--experimental-strip-types`, so the `.ts` sources
execute directly without a separate build step. Trusted manual satellite
preview workflows launch their guarded deploy job directly: manual dispatch
already bypasses affected-path gating, so a reusable preflight would only add a
second queued runner. These workflows remain protected by the `main` dispatch
ref, `TRUSTED_PREVIEW_DEPLOY_ACTORS`, and their GitHub Environment. Preview
concurrency is keyed by workflow and `preview_ref`; a repeated request for the
same preview cancels stale work, while different preview refs remain independent.
### The production planner
A `production` push starts one `plan` job instead of starting one
`check-ci` runner per app. `resolve-production-vercel-targets.ts` reads the
workspace graph once, memoizes dependency closures, resolves each target from
its own successful deployment marker, and emits a reviewable Actions summary.
Static reusable jobs call only the selected `vercel-production-.yaml`
workflows, so deployments stay attached to the original push SHA instead of
appearing as detached `workflow_dispatch` runs.
Per-app production workflows remain independently protected by their GitHub
Environment and a static per-app concurrency group. The group must include the
app identifier because reusable workflows inherit caller context; a shared
caller-derived key would cancel sibling deployments and show red X statuses
instead of clean skips. They accept `workflow_call` for commit-driven planner
jobs and retain `workflow_dispatch` for deliberate operator reruns. Their
`cancel-in-progress` predicate is enabled only for a `production` push, so a
`main` commit or manual recovery run cannot cancel an active production deploy.
A missing or untrusted marker fails open for that app without forcing every
other app's gate open.
### The decision function
`getWorkflowDecision()` in `tuturuuu.ts` resolves to `should_run` using this
order of precedence:
1. **Global toggle.** If the workflow is `false` in the `ci` map, it never runs.
2. **`workflow_dispatch` bypass.** Manual runs always proceed (affected-path
gating is skipped), so you can force a deploy from the Actions UI.
3. **Non-Vercel workflows.** Workflows without an entry in
`vercelWorkflowTargets` fall back to their static `ci` toggle.
4. **Unavailable change data.** If the changed-file set cannot be resolved, the
gate stays **open** (`should_run = true`) to fail safe.
5. **Affected-path matching (Vercel workflows).** The workflow runs only if at
least one changed file matches one of:
* a **global affecting path** (`bun.lock`, `package.json`, `turbo.json`,
`tuturuuu.ts`, `.github/workflows/ci-check.yml`, the shared Bun/remote-cache
actions, or the deterministic metadata generator);
* the workflow's **own** workflow file;
* the target app's directory (e.g. `apps/web/**` for the platform target);
* any package inside the target app's **workspace dependency closure**. The
closure is built from `workspace:*` dependencies, so editing a shared
package only triggers the apps that actually depend on it.
If nothing matches, the decision is `should_run = false` and the heavy job is
skipped.
### Vercel target table
Every Vercel app is registered once in `vercelWorkflowTargets`. Adding a new app
means adding a single entry there (app slug, app path, package name, and the
preview/production workflow filenames) — the gating logic and changed-file
matching then apply automatically.
```ts theme={null}
{
app: 'platform',
appPath: 'apps/web',
packageName: '@tuturuuu/web',
previewWorkflow: 'vercel-preview-platform.yaml',
productionWorkflow: 'vercel-production-platform.yaml',
}
```
**Applied to:** the 24-target production planner, the inline platform preview
gate, and other gated CI workflows wired through `ci-check.yml`. The other 23
preview workflows are trusted manual dispatches and therefore do not allocate a
standalone configuration runner.
## Build Caching
Once a workflow decides it must run, cacheable JavaScript work is invoked from
the repository root through
`.github/actions/run-with-turbo-remote-cache/action.yml`:
```yaml theme={null}
- name: Type check
uses: ./.github/actions/run-with-turbo-remote-cache
with:
command: bun turbo:local run type-check --concurrency=4
token: ${{ secrets.TURBO_TOKEN }}
team: ${{ vars.TURBO_TEAM || secrets.TURBO_TEAM }}
```
The composite action exposes credentials only to the wrapped command step. It
runs locally when both token and team are absent, rejects a token without a
team, and keeps Turbo's local rebuild behavior available if the remote service
is unavailable. `TURBO_TOKEN` must never be passed to pull-request or
Dependabot jobs. Those jobs restore a task-family-scoped `.turbo` fallback from
GitHub's cache without writing shared entries; trusted default-branch jobs seed
that fallback.
The shared Bun runtime and package-download caches follow the same ownership
rule: only trusted `main` jobs save entries. Production, pull-request,
Dependabot, and other branch jobs restore the default-branch cache without
creating branch-scoped duplicates. This is especially important for Bun's
package cache, which can approach 1 GB for the full monorepo lockfile.
Turborepo hashes inputs and reuses cached task outputs, so unchanged builds,
type checks, and tests resolve to cache hits instead of recompiling. The task
graph includes transit-only dependency nodes for tests: dependency source
changes invalidate downstream tests without running every dependency's test
suite. Build outputs include framework output, coverage, and TypeScript
incremental state where those artifacts are deterministic. Output-affecting
environment values are hashed, while CPU, heap, and concurrency controls are
pass-through values.
Every active Vercel project has a checked-in `buildCommand` that invokes its
full workspace build through root Turbo. The 48 deployment workflows generate
deterministic source metadata and then wrap `vercel build`; they do not run a
separate dependency prebuild. This lets an unchanged full app build restore as
one task while `.vercel/output` remains an in-job handoff to
`vercel deploy --prebuilt`, not an uploaded artifact or generic cache.
Docker and Rust verification use the same principle at their native cache
layers. Each shared GitHub BuildKit scope has one trusted `main` writer: Docker
setup owns web, TanStack, and storage scopes; Rust CI owns the backend scope;
and E2E shard 1 owns the small leaf-service scopes. Production, other shards,
and migration E2E restore without exporting. Expensive web, TanStack, and
backend images use `mode=max`; validation-only development and small leaf
images use `mode=min` or restore-only caching. Dockerized E2E jobs explicitly
select a `docker-container` Buildx builder so Compose can import and export the
GitHub cache backend. Turbo values enter image builds only as BuildKit secrets.
Rust verify and deploy jobs share one pinned,
toolchain-aware Cargo cache, with writes restricted to protected branches.
Flutter workflows cache dependencies and native tool state (pub, Gradle, and
CocoaPods), never final APK, AAB, IPA, archive, or application-bundle outputs.
Dependency resolution and the platform build always execute so a cache hit
cannot skip validation. Managed uv and CodeQL caches remain owned by their
upstream actions rather than being duplicated.
Automatic CodeQL is owned by GitHub's organization-managed
`dynamic/github-code-scanning/codeql` workflow, which scans
JavaScript/TypeScript and Python. The checked-in `.github/workflows/codeql.yml`
is manual-only: it preserves an explicit fallback and avoids the Security UI's
missing-workflow warning without adding push, pull-request, or cron duplicates.
`bun git-sync` mirrors the already-scanned `main` SHA to `production`.
The full Dockerized E2E graph is created for E2E specs, Playwright and Docker
configuration, database fixtures, package manifests, lockfiles, and its own
runner scripts. Ordinary application source commits do not create six expensive
E2E consumers. There is no cron schedule: automatic E2E runs are commit-driven
and path-scoped, while manual dispatch remains available for deliberate
validation. E2E uses native push paths and launches its image producer directly,
without an extra static switchboard job.
Supabase staging and production migrations use the last successful migration
deployment marker as their change-range baseline. Each workflow resolves the
entire pending range, fails open when no trustworthy marker exists, and runs
`supabase db push --include-all` only when `apps/database/**` or migration
control files changed. Evaluation and deployment share one serialized job per
environment, so no-op workflow-run signals finish quickly while database
migrations cannot race. Production still requires the successful same-SHA
platform deployment marker and staging migration before applying changes.
## Runner And Storage Policy
Tuturuuu is a public repository in a GitHub Free organization. Standard hosted
Linux and Windows jobs are budgeted for 4 CPUs and 16 GB RAM; the organization
can run 20 jobs concurrently, including at most 5 macOS jobs. CPU-bound Turbo
checks therefore default to concurrency 4. Memory-heavy Next.js and Docker
stages limit inner build concurrency to 2 and retain an 8 GB Node heap. Keep
independent jobs parallel, keep macOS matrices below the platform limit, and do
not switch to billable larger runners as a cache optimization.
The live repository cache configuration, queried through GitHub's API, is a
10 GB maximum with 7-day retention. The weekly Actions resource report groups
cache usage by key prefix and reports informational status at 80%, warning at
90%, and critical status at 100%. It also reports artifact count, bytes, age,
and largest workflow families without enforcing a fixed artifact byte cap.
Steady-state cache usage should stay below 9 GB to leave eviction headroom.
The July 10 audit observed active usage temporarily above the configured limit;
that is eviction lag and cache churn, not an additional open-source allowance.
Artifacts are retained only as long as their handoff or diagnostic value
requires:
* E2E and Playwright diagnostics: one failure-only artifact per shard or
migration mode, 7 days.
* Package-release tarballs: 1 day with compression disabled.
* Mobile development deliverables: 7 days; production store deliverables: 14
days. Pre-compressed bundles are archived first and uploaded without
redundant compression.
* Discord JUnit and Cloudflare smoke diagnostics: failure-only or explicit
7-day retention as applicable.
Every artifact upload must declare both retention and missing-file behavior.
Required release outputs fail when absent; optional diagnostics may warn.
Closed trusted pull requests delete cache entries scoped to their merge ref.
Legacy broad Turbo, run-ID Supabase, final Flutter-output, and old Rust cache
prefixes are removed manually only after their replacements demonstrate hits;
default-branch CodeQL caches are never part of automated cleanup.
Earlier versions of these workflows ran an inline Turborepo `--dry-run=json`
step to *decide* whether to skip a build. That per-workflow skip step has been
removed in favor of the centralized `ci-check` gate above. Turborepo is now
used for caching the build itself, not for the run/skip decision.
## Race-Condition Handling
Production deployments serialize each app and branch with GitHub Actions
concurrency:
```yaml theme={null}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'push' && github.ref == 'refs/heads/production' }}
```
When another production commit arrives, GitHub cancels the superseded run and
keeps the newest run deployable. Older runs no longer finish green after
skipping every deployment step, which makes queued and deployed SHAs
observable without allowing stale artifacts to replace newer production code.
Commits that exist only on `main` and manual recovery dispatches leave an active
production deployment running.
Affected-app gating always diffs the newest SHA against the last successful
deployment marker. If no trustworthy marker is available, the gate fails open
and runs the deployment instead of considering only the newest push payload;
this preserves app changes from canceled intermediate commits.
## Edge Case Handling
`resolve-changed-files.ts` and the gate are written to fail safe.
### Missing or unresolvable change data
If the changed-file set cannot be computed (e.g. an unusual event payload or a
missing base commit), `getWorkflowDecision()` returns `should_run = true`. The
guiding principle: **build when uncertain** rather than skip incorrectly.
### Initial commits and shallow clones
When a parent commit is unavailable (the very first commit, or a shallow clone
without history), the changed-file resolver cannot diff against a base, so the
gate opens and a full build runs. The workflow that needs deeper history checks
out with an appropriate `fetch-depth`, and `ci-check.yml` uses `fetch-depth: 0`
with a sparse checkout so it always has enough history to diff.
### Last-successful-deployment markers
For push events, the resolver can diff against the SHA of the last successful
deployment (recorded as a GitHub deployment marker) rather than only the parent
commit. This catches the case where intermediate commits never deployed, so the
full diff since the last live deploy is considered.
## Monitoring & Debugging
### Inspect the gate decision
`check-workflow-config.ts` logs its decision and the matched paths. Preview
`check-ci` logs show the individual decision; production uses the planner's
**Production Vercel deployment plan** job summary, including the per-app
baseline source and deploy/skip result. Individual logs still look like:
```text theme={null}
Workflow: vercel-production-platform.yaml
Should run: false
Reason: vercel-production-platform.yaml is unaffected by the changed paths
```
or, when it does run:
```text theme={null}
Should run: true
Reason: vercel-production-platform.yaml is affected by 3 changed path(s)
Matched paths:
- apps/web/src/app/page.tsx
- packages/ui/src/button.tsx
```
### Reproduce a decision locally
You can run the decision script directly to debug why a workflow did or did not
run:
```bash theme={null}
CHANGED_FILES="apps/web/src/app/page.tsx" \
WORKFLOW_NAME="vercel-production-platform.yaml" \
node --experimental-strip-types scripts/ci/check-workflow-config.ts
```
There are also unit tests covering the gating logic:
```bash theme={null}
node --test scripts/ci/check-workflow-config.test.js
node --test scripts/ci/resolve-changed-files.test.js
node --test scripts/ci/resolve-production-vercel-targets.test.js
```
### Common Issues
#### Issue: A workflow never runs for legitimate changes
**Likely causes:**
* The app is missing (or misconfigured) in `vercelWorkflowTargets`.
* The changed package is not in the target app's workspace dependency closure
(the app does not actually `workspace:*`-depend on it).
* The workflow is toggled off in the `ci` map in `tuturuuu.ts`.
**Fix:** Verify the target entry and the dependency edges, then re-run the local
reproduction above to confirm the decision.
#### Issue: A workflow always runs
**Likely causes:**
* A global affecting path (`bun.lock`, `package.json`, `turbo.json`,
`tuturuuu.ts`) changed — these intentionally trigger every Vercel target.
* The change-data resolver failed, so the gate fell open by design.
This is usually correct behavior; only investigate if it persists for changes
that clearly do not touch shared roots.
#### Issue: Need to force a run
Use the app workflow's manual trigger. A direct `workflow_dispatch` starts that
app without re-running affected-path gating:
```yaml theme={null}
on:
workflow_dispatch:
```
## Performance Impact
The exact savings depend on the size of the app set and how localized each
commit is, so treat the following as **illustrative** rather than a fixed figure.
The monorepo currently registers **24 active Vercel app targets**, each with a
preview and a production workflow (**48 Vercel deploy workflows**), out of 91
workflow files. Production planning prevents a single-app change from creating
23 unrelated production runners. Preview workflows create a runner only for the
inline platform signal or an explicitly requested satellite preview.
For a change that touches only one app:
```text theme={null}
1 production planner workflow
+ only the affected app deployment workflows
+ one inline platform preview gate on protected main
+ direct satellite preview jobs only when explicitly dispatched
```
Each skipped production app now consumes no runner at all; only the shared
planner and selected reusable deployment jobs run under the production push.
Changes to shared roots
(`package.json`, `bun.lock`, `turbo.json`, `tuturuuu.ts`) intentionally fan out
to all targets, so headline skip rates will vary with how often those roots
change.
To measure real-world impact, inspect the production planner's job summary and
compare its selected workflow count with the 24 registered targets. Platform
preview reports its inline decision in the deploy job; manual satellite
previews have no separate gate job.
## Best Practices
### 1. Keep `tuturuuu.ts` as the source of truth
When adding an app or workflow, register Vercel app metadata in
`vercelWorkflowTargets` and register automatic switchboard-controlled workflows
in the `ci` map. Trusted manual satellite previews and intentionally
path-triggered compatibility smokes do not keep unused `ci` toggles. The gate
and the tests key off these boundaries.
### 2. Trust the fail-safe
When change data is missing, the gate opens and the workflow runs. This is
intentional — it is safer to build unnecessarily than to skip a real change.
Don't add ad-hoc skip logic that defeats it.
### 3. Model dependencies with `workspace:*`
The dependency-closure matching only works because shared packages are wired as
`workspace:*` dependencies. If an app consumes shared code without declaring the
dependency, the gate will not know to run it when that package changes.
### 4. Use `workflow_dispatch` to debug
To bypass gating for a one-off investigation, trigger the workflow manually from
the Actions UI rather than touching the gate logic.
### 5. Update tests when changing the gate
`getWorkflowDecision()` and the resolver are covered by unit tests under
`scripts/ci/`. Update them alongside any change to the matching rules so the
behavior stays pinned.
## Further Reading
* [CI/CD Pipelines](/build/development-tools/ci-cd-pipelines)
* [TanStack + Rust migration](/platform/architecture/tanstack-rust-migration)
* [Turborepo: Remote Caching](https://turborepo.com/docs/core-concepts/remote-caching)
* [GitHub Actions: Reusing workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows)
## Summary
Our CI/CD optimization centralizes change detection so each workflow stays thin:
* Reusable `ci-check.yml` calls remain available where a separate gate is
necessary; production planning and platform preview avoid per-app preflight
fan-out.
* `tuturuuu.ts` decides, per workflow, whether the change is relevant — including
workspace dependency-closure awareness for shared packages.
* The remote-cache wrapper keeps Turbo credentials step-scoped and gives
secretless jobs a GitHub-backed local fallback.
* Turborepo caches full app builds, type checks, tests, and release preparation
once a run is warranted.
* The gate fails safe (builds on uncertainty), and trusted manual satellite
previews start their guarded deploy job directly.
The result: a one-app change selects only the necessary production deploys in
the commit-associated push run,
platform preview retains its required staging signal without a second runner,
manual previews never queue a redundant preflight, and superseded previews are
canceled before they occupy the queue indefinitely.
# CI/CD Pipelines
Source: https://docs.tuturuuu.com/build/development-tools/ci-cd-pipelines
High-level CI/CD overview, with current operational runbooks linked from one place.
This page is the entry point for CI/CD. The day-to-day operational detail now lives
in the dedicated DevOps pages below so the docs stay current as workflows change.
## Read Next
* [DevOps & Deployment Overview](/build/devops/overview)
* [Environments & Release Flow](/build/devops/environments-release-flow)
* [Web Docker Deployment](/build/devops/web-docker-deployment)
* [GitHub Actions Runbook](/build/devops/github-actions-runbook)
* [Secrets & Configuration](/build/devops/secrets-and-configuration)
## Current CI/CD Shape
Tuturuuu currently automates:
* Vercel preview deployments for non-`production` branches
* Vercel production deployments from the `production` branch
* Independent Vercel preview and production deployments for satellite apps such as `apps/cms`
* Affected-app gating for Vercel deployments before dependency install, build, or deploy work
* Supabase staging migrations driven from `main` and production migrations driven from `production`
* Docker parity and image validation for the self-hosted web runtime
* Modal deployment for `apps/discord`
* Mobile build artifact generation
* Package publishing to npm
* Quality, lint, type, test, and security checks
## Important Repo Conventions
* `ci-check.yml` reads `tuturuuu.ts` and can disable an individual workflow centrally.
* Vercel workflows use `tuturuuu.ts` app metadata and workspace `workspace:*` dependency closures to run only affected app deploys.
* `bun.lock` changes run all Vercel app deploys because lockfile ownership is ambiguous without a source or manifest path.
* Manual `workflow_dispatch` runs enabled workflows even when changed paths would otherwise be unaffected.
* Production Vercel workflows cancel superseded per-app runs only for newer
`production` pushes. A `main` commit or manual recovery run never cancels an
active production deployment.
* The external-app internal-package build uses the same cancel-superseded policy, so rapid pushes cannot finish as green no-op builds.
* Database migrations are not treated the same as app deploys; staging and production each have explicit prerequisite logic, with staging now sourced from `main`.
* Docker changes are guarded by `docker-setup-check.yaml`, including dev and production image builds.
## When To Use This Page
Use this page for orientation. Use the DevOps pages for the runbook you actually need
while operating the system.
# Cleaning Your Tuturuuu Clone
Source: https://docs.tuturuuu.com/build/development-tools/cleaning-clone
How to reset your local Tuturuuu repository to a clean state.
This guide helps you reset your Tuturuuu repository to a clean state, which is
useful if you encounter dependency issues or want to ensure a fresh
development environment.
## Prerequisites
* Ensure you have [bun](https://bun.sh/) installed. The repository pins a specific
Bun version via the `packageManager` field in the root `package.json` (currently
`bun@1.3.14`) — match that version so your local toolchain stays in sync.
* Node `>=22` is also required (see the `engines` field in the root `package.json`).
## Quick Cleanup (Recommended)
The repository ships a one-shot root script that performs the full cleanup for you:
```bash theme={null}
bun clean
```
This removes `bun.lock`, deletes every `node_modules` directory, and clears all
`.next` and `.turbo` build caches across the monorepo (it uses
`npkill --delete-all` under the hood). After it finishes, reinstall everything:
```bash theme={null}
bun i
```
`bun clean` is the supported shortcut for the manual steps below. Reach for the
manual steps when you want finer control over which folders are removed.
## Manual Cleanup (Granular Alternative)
### 1. Delete the lockfile
Remove the `bun.lock` file at the root of your repository (if it exists):
```bash theme={null}
rm bun.lock
```
### 2. Remove all `node_modules` folders
Use [npkill](https://github.com/voidcosmos/npkill) to find and remove all `node_modules` directories:
```bash theme={null}
bunx npkill
```
* In the `npkill` interface, select and delete all `node_modules` folders found in your repo.
If you don't have `npkill` installed, `bunx` will download and run it
automatically.
### 3. Clear build caches (optional)
Remove stale `.next` and `.turbo` build caches if you suspect they are causing
issues (the `bun clean` shortcut does this automatically):
```bash theme={null}
find . -name node_modules -prune -o \( -name .next -o -name .turbo \) -type d -exec rm -rf {} +
```
### 4. Reinstall dependencies
Install all required dependencies using bun:
```bash theme={null}
bun i
```
This will install fresh dependencies for all apps and packages in the monorepo.
After completing these steps, your repository should be in a clean state. You
can now continue with the usual development workflow as described in
Development.
## Troubleshooting
* If you still encounter issues, try restarting your IDE or terminal.
* Make sure your bun version is up to date: `bun upgrade`.
* If problems persist, check the [Development](/build/development-tools/development) guide or ask for help in the project discussions.
# Codex Plugin
Source: https://docs.tuturuuu.com/build/development-tools/codex-plugin
Use and maintain the local Tuturuuu Codex plugin.
The local Tuturuuu Codex plugin lives at `plugins/tuturuuu`. It packages
repo-specific operating knowledge as Codex skills so future agent sessions can
load focused guidance without copying long instructions into every prompt.
## Discovery
Codex discovers repo-local plugins through
`.agents/plugins/marketplace.json`. The Tuturuuu marketplace is named
`tuturuuu`, displays as `Tuturuuu`, and points its local plugin entry at
`./plugins/tuturuuu`.
After changing marketplace metadata or plugin files, validate the plugin and
refresh the repo-local marketplace from the repo root. Do not bump
`plugins/tuturuuu/.codex-plugin/plugin.json` `version` for ordinary authored
workflow updates unless a release workflow or user request requires it.
The repo-local plugin manifest is currently refreshed to `0.2.23`.
```bash theme={null}
codex plugin marketplace add ./
```
Restart Codex after refreshing the marketplace. Then open the plugin directory,
choose the `Tuturuuu` marketplace, and install or update the `Tuturuuu` plugin.
If Codex does not show the repo marketplace after restart, add the repo root as
a local marketplace source from the CLI:
```bash theme={null}
codex plugin marketplace add ./
```
## skills.sh Discovery
The public skills.sh registry discovers skills from GitHub repositories after
someone installs the repo with the `skills` CLI. The platform repo exposes the
Tuturuuu plugin skills for that flow through two root metadata files:
* `.claude-plugin/marketplace.json` points compatible skill installers at
`plugins/tuturuuu/skills`.
* `skills.sh.json` groups the public repo page so Tuturuuu skills appear ahead
of unrelated installed project skills.
Do not run the public install trigger before the metadata is pushed to GitHub.
After pushing the metadata commit, run the trigger from a disposable directory,
not from this checkout:
```bash theme={null}
npx skills add tutur3u/platform --agent codex --copy -y \
--skill tuturuuu-agent-coordination \
--skill tuturuuu-browser-vercel-debugging \
--skill tuturuuu-ci-docs \
--skill tuturuuu-cli \
--skill tuturuuu-cli-finance \
--skill tuturuuu-cli-tasks \
--skill tuturuuu-cms-studio \
--skill tuturuuu-commit \
--skill tuturuuu-database \
--skill tuturuuu-devbox-ops \
--skill tuturuuu-development-tooling \
--skill tuturuuu-e2e-auth-debugging \
--skill tuturuuu-external-apps \
--skill tuturuuu-mobile-task-board \
--skill tuturuuu-platform \
--skill tuturuuu-pr-merge-sync \
--skill tuturuuu-review-comments \
--skill tuturuuu-satellite-app-ux \
--skill tuturuuu-validation-offload \
--skill tuturuuu-web-release
```
Then verify the cached page after refresh at
`https://skills.sh/tutur3u/platform`.
## Included Skills
* `$tuturuuu-platform`: repo-wide workflow guidance for `apps/web`,
`packages/*`, `apps/database`, `apps/docs`, translations, navigation, and
verification follow-through, including shared-worktree preflight, scoped
staging expectations, and Effect adoption guidance for TypeScript
server/service orchestration.
* `$tuturuuu-browser-vercel-debugging`: authenticated Browser/Chrome and Vercel
CLI guidance for reproducing deployed UI/API bugs, correlating runtime and
build logs, checking satellite routing/auth, auditing responsive and i18n
behavior, investigating performance/cost, and verifying improvements without
mutating customer data.
* `$tuturuuu-cms-studio`: CMS and external-project content studio guidance for
`apps/cms`, branded project adapters, landing-page content editing, content
collections, media workflows, preview delivery, and non-technical editor UX.
* `$tuturuuu-satellite-app-ux`: standalone app shell guidance for satellite apps
such as Mail and CMS, including `@tuturuuu/satellite` auth, app-session
routes, i18n, workspace navigation, permissions, and focused operational UX.
* `$tuturuuu-external-apps`: external and branded sibling app integration
guidance for app-token exchange, refreshable admin sessions, direct signed
storage uploads, external-project mutations, delivery fallbacks, and
sanitized diagnostics.
* `$tuturuuu-e2e-auth-debugging`: local E2E authentication guidance for native
Playwright patch iteration, `dev-session`, guest access, onboarding
redirects, app-session verification, rate-limit tests, and explicit
Docker-parity auth checks.
* `$tuturuuu-commit`: scoped commit workflow guidance for explicit commit,
commit-and-push, atomic staging, Conventional Commit subjects, shared-worktree
path isolation, commit-window claim/wait/release, proof-gated no-verify
commit evidence, hook failure handling, and final commit reporting.
* `$tuturuuu-agent-coordination`: shared-worktree coordination guidance for
dirty checkouts, active ownership notes, archived context, overlapping edits,
stale handoffs, explicit write sets, commit-window coordination, and
path-scoped staging safety.
* `$tuturuuu-cli`: core native `ttr` CLI install, browser login, copy-token
login, workspace discovery, scoped help, SDK-backed command surfaces, task
template command boundaries, version checks, and release verification
guidance. It documents autonomous Bun and CLI installation with
`curl -fsSL https://bun.sh/install | bash` on macOS/Linux,
`powershell -c "irm bun.sh/install.ps1 | iex"` on Windows, then
`bun i -g tuturuuu`.
* `$tuturuuu-cli-tasks`: focused `ttr tasks` and `ttr task-templates` guidance
for task capture, task board/list/label discovery, open-task defaults,
compact task tables, task creation, reusable task-template import/export,
template instantiation, splitting, movement, completion, closure, and task
verification. It documents ttr-first task capture for requests to add, create,
template, track, or split Tuturuuu tasks, including label discovery and
closing superseded combined tasks after replacements are created.
* `$tuturuuu-cli-finance`: focused `ttr finance` guidance for wallet,
transaction, category, budget, and recurring CRUD; analytics reads; finance
pagination footers; explicit-workspace finance read diagnostics; and wrapped
finance response normalization in the SDK client.
* `$tuturuuu-development-tooling`: shared guidance for improving Codex skills,
plugin behavior, validation scripts, docs runbooks, helper scripts, and
durable workflow learnings that should benefit future Tuturuuu contributors
and assistants, including Turborepo/Next cache boundaries, local dev-speed
diagnostics, release-please merge automation, and package release automation
rules.
* `$tuturuuu-pr-merge-sync`: PR closeout guidance for quiet-window review
watching, efficient GitHub polling, merge/admin-merge follow-through, the hard
main-green gate before `bun git-sync`, production sync, and production
workflow verification.
* `$tuturuuu-devbox-ops`: operational guidance for `ttr box` setup, runner
registration, one-shot smoke tests, 24/7 system services, runner-token
cleanup, CLI upgrades, and infrastructure devbox observability.
* `$tuturuuu-validation-offload`: validation strategy guidance for deciding
when to run focused tests, `bun check`, Supabase, Docker, or browser-heavy
workflows through internal devboxes instead of the local agent session.
* `$tuturuuu-web-release`: `apps/web` release badge metadata, blue/green
runtime snapshot fallback, `PLATFORM_BUILD_*` precedence, and
release-please-managed `TUTURUUU_PLATFORM_VERSION` guidance.
* `$tuturuuu-mobile-task-board`: Flutter mobile task-board guidance for task
dates, overdue behavior, task detail routing, description mode, assignee
flows, and release-please-managed mobile version metadata.
* `$tuturuuu-database`: Supabase migration, RLS, workspace-scoped API write,
generated type, storage, and database verification guidance.
* `$tuturuuu-ci-docs`: GitHub Actions, `tuturuuu.ts`, validation script,
docs-page, and docs navigation guidance.
* `$tuturuuu-review-comments`: GitHub PR review-thread guidance for checking,
re-checking, validating, fixing, resolving, and committing unresolved review
comments. Addressed review threads are resolved by default after validation;
commits and pushes still require an explicit user request. Its bundled
`fetch_review_threads.py` helper supports `--active-only --summary` and emits
review-thread counts for final re-check reports.
## Translation Key Workflow
When a platform change introduces a new translation key, prefer the root helper
instead of editing message JSON by hand:
```bash theme={null}
bun i18n:add --app web --key common.save --value en=Save --value vi=Lưu
```
Use `--dir apps//messages` for non-standard message directories and
`--all` only for shared UI keys that must exist in every detected app message
setup. The helper detects locale JSON files in the target setup, requires a
value for each locale, blocks unsafe key paths, prevents accidental overwrites
unless `--overwrite` is passed, and writes sorted JSON. Manual message edits are
reserved for broad copy rewrites or value-only updates; run `bun i18n:sort`
after those edits.
For larger translation updates, use bulk modes instead of repeatedly editing
JSON:
```bash theme={null}
bun i18n:add --app web --mode add --entries '{"common.save":{"en":"Save","vi":"Lưu"}}'
bun i18n:add --app web --mode remove --entries '["common.old"]'
bun i18n:add --app web --mode replace --entries-file ./translations.json
```
Bulk add and replace entries use a JSON object keyed by translation key, with
locale values underneath each key. Bulk remove accepts an array of translation
keys. Replace and remove fail when a key is missing unless `--ignore-missing` is
passed.
## CI Coverage
`.github/workflows/codex-plugin.yaml` runs the plugin-local validator and parses
the docs navigation JSON. The workflow is enabled through `tuturuuu.ts`, matching
the repo's other CI switchboard-controlled checks.
The CI check currently verifies:
* plugin manifest JSON and interface metadata
* natural default prompt examples for the invokable skills, written as short
user requests instead of visible `$skill` commands
* dedicated prompt coverage for CMS studio and landing-page content management
* dedicated prompt coverage for external app integrations and direct uploads
* dedicated prompt coverage for satellite app UX and app-session workflows
* dedicated prompt coverage for native local E2E auth and dev-session debugging
* dedicated prompt coverage for browser and Vercel production troubleshooting
* dedicated prompt coverage for shared-worktree coordination and agent handoff
* dedicated prompt coverage for PR quiet-window merge sync with main-green
gating before production sync
* dedicated prompt coverage for devbox runner operations and validation offload
* dedicated prompt coverage for web release badge metadata
* durable CI tooling guidance for npm package release auto-recovery
* durable CI tooling guidance for Supabase CLI setup retry and GitHub-tokened
release lookup
* durable CI tooling guidance for bounded Codecov coverage test retries
* scoped commit workflow coverage for explicit commit requests
* native `ttr` task-template workflow coverage in the CLI task skill
* proof-gated no-verify commit evidence coverage
* commit-window claim, wait, release, and stale-lock guidance
* repo marketplace metadata at `.agents/plugins/marketplace.json`
* public skills.sh discovery metadata at `.claude-plugin/marketplace.json`
* public skills.sh grouping metadata at `skills.sh.json`
* skill folder and frontmatter naming parity
* `agents/openai.yaml` defaults for every skill
* skill `references/` links
* plugin docs page registration in `apps/docs/docs.json`
* Codex plugin workflow registration in `tuturuuu.ts`
* absence of scaffold `[TODO: ...]` placeholders
* absence of machine-specific local paths in plugin text
## Shared Worktree Coordination
Agents should assume that humans or other agents may have uncommitted work in the
same checkout. The docs-facing protocol lives in the
[Agent Operating Manual](/overview/agent-operating-manual#shared-worktree-coordination),
and the agent-facing workflow lives in `$tuturuuu-agent-coordination`.
Open Tuturuuu pull requests are always handled in isolated worktrees under
`.worktrees/`. Run `bun setup` immediately after creating the PR worktree, keep
the worktree and local task branch while the PR is open, and remove both only
after the merge is confirmed on `main` and post-merge verification is complete.
This lifecycle is documented in the Agent Operating Manual and enforced as a
cross-cutting rule in `AGENTS.md`.
For user-authorized continuous integration, the same lifecycle applies to each
verified checkpoint: integrate it into current `main`, require the exact main
SHA to be fully green, run `bun git-sync`, verify production, then remove only
the completed worktree and local task branch. Rust-heavy retained worktrees use
`bun rust-cache report` plus bounded `prune`/`auto` cleanup rather than broad
filesystem deletion.
Use `tmp/agent-coordination/` for lightweight agent-to-agent notes when work may
overlap, the worktree is dirty, the task is long-running, or the task changes
agent workflow rules. The directory is ignored by git, so it can hold live
intent, ownership, status, verification, risk, and handoff messages without
polluting commits.
Active notes are notes marked `working`, `blocked`, or `handoff`. Treat stale
active notes as ownership signals until you have read them and checked the
current worktree. Keep active notes as direct files under
`tmp/agent-coordination/`; archive completed notes under
`tmp/agent-coordination/archive//.md` after they are
marked `done` and no longer need top-level visibility.
Use `bun git-commit-window` before changing the staged set or creating commits
in a shared checkout. `claim` creates an advisory lock under
`tmp/agent-coordination/git-commit-window.lock.json`; `wait` sleeps until the
current lock is released or expires, then claims it before notifying the waiting
agent that it can commit. Claims default to 10 minutes, accept only 5-10-minute
TTLs, and should be held only while staging, inspecting, committing, amending,
rebasing, or finishing commit-and-push follow-through. Release the lock with
`bun git-commit-window release --token ` after the commit operation
finishes or aborts, and do not record tokens in coordination notes.
Archived notes are historical context for targeted lookup, not locks. Agents
should search them by path, feature, or workflow only when previous decisions or
verification matter to the current task. If another active note claims the same
file set, agents should choose non-overlapping slices, write a response note, or
ask the human partner to arbitrate. They should never format, stage, or fix
unrelated dirty files simply because a repo-wide check or commit hook reports
them.
When coordinating subagents, split work into named lanes with disjoint owned
paths, excluded paths, validation commands, and a no-stage/no-commit boundary
unless the lane explicitly owns a commit. The coordinator owns integration and
shared generated artifacts by default, including TanStack route trees, migration
manifests, route overrides, OpenAPI snapshots, docs navigation, sorted
translation bundles, and Supabase generated types. Workers should report
generated drift and hand off exact follow-up instead of silently committing it.
When a generator can see dirty or untracked files owned by another lane,
regenerate from a clean worktree or an explicit input set and copy back only the
intended artifact.
Write a lane contract before spawning: owner, mode, owned paths, excluded paths,
generated outputs, validation, handoff shape, and commit authority. Read-only
lanes return evidence and risks without taking ownership. In Codex, choose
either a typed worker/explorer role with explicit lane context in the prompt or
a full-history fork without a role override; the harness rejects a spawn request
that asks for both. After integrating a
slice, checkpoint before dispatching more work: refresh status, close completed
subagents, update the parent note with the commit or handoff state, list
validation and unrelated blockers, and choose the next lane from the current
worktree state.
The normal scoped commit path still lets Git hooks run. Agents may use
`git commit --no-verify` only when they can produce a proof packet for their
exact staged paths: reviewed status and staged diff output, path ownership,
separated validation mapped to the relevant `bun check` components, path-based
rationale for skipped components, and `bun check:mobile` coverage when
`apps/mobile` is touched. Incomplete proof, unclear ownership, or uncertain
check mapping means the hook should run or the risk should be reported instead
of bypassed.
## AGENTS.md Split
Root `AGENTS.md` is intentionally a compact hard-policy index. Detailed
implementation gotchas and composable patterns live in focused Tuturuuu skill
references instead:
* web/API/shared UI patterns in `$tuturuuu-platform`
* release badge/version metadata in `$tuturuuu-web-release`
* database/API/storage patterns in `$tuturuuu-database`
* CI/root-script/tooling patterns in `$tuturuuu-development-tooling`
* Docker blue/green watcher patterns in `$tuturuuu-ci-docs`
* mobile task-board and overlay patterns in `$tuturuuu-mobile-task-board`
When a session reveals durable knowledge, update the narrowest matching skill
reference and docs page. Add to root `AGENTS.md` only when the rule is a
cross-cutting hard mandate that agents must see before skill loading.
## Validate The Plugin
Run the local validator after changing the plugin manifest or its skills:
```bash theme={null}
python3 plugins/tuturuuu/scripts/validate_plugin.py
```
The local validator also checks the root skills.sh metadata files against the
actual Tuturuuu skill folders, so add every new public skill to both
`.claude-plugin/marketplace.json` and `skills.sh.json`.
For individual skills, run the Codex skill validator:
```bash theme={null}
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-platform
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-cms-studio
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-satellite-app-ux
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-e2e-auth-debugging
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-external-apps
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-commit
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-agent-coordination
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-cli
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-cli-tasks
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-cli-finance
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-development-tooling
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-devbox-ops
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-validation-offload
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-web-release
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-mobile-task-board
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-database
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-ci-docs
python3 ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py plugins/tuturuuu/skills/tuturuuu-review-comments
```
The Codex skill validator imports `yaml`, so it requires `PyYAML` in the active
Python environment. The plugin-local validator uses only the Python standard
library.
## Maintenance Rules
* Keep `plugins/tuturuuu/.codex-plugin/plugin.json` aligned with the plugin
folder name. Do not bump its `version` for ordinary authored work unless a
release workflow or user request requires it; when a user explicitly asks for
a plugin refresh, bump the manifest and keep this page current.
* Keep each skill folder name aligned with its `SKILL.md` frontmatter `name`.
* Keep plugin and skill default prompts short, natural, and action-oriented.
Marketplace prompt cards should read like user goals, for example "Clear the
unresolved review comments on this PR."
* Keep skill text portable across machines. Prefer repo-relative paths,
``, or `~` over user-specific absolute paths.
* Put detailed guidance in `references/` and keep `SKILL.md` focused on the
workflow that should be loaded immediately.
* Keep shared-worktree and commit-window coordination guidance aligned across
`AGENTS.md`, the Agent Operating Manual, `$tuturuuu-agent-coordination`, and
`$tuturuuu-commit`.
* Keep the open-PR `.worktrees/` lifecycle, immediate `bun setup`, and
post-merge worktree/branch cleanup guidance aligned across `AGENTS.md`, the
Agent Operating Manual, `$tuturuuu-agent-coordination`, the platform
checklist, and `$tuturuuu-pr-merge-sync`. Keep the authorized continuous
integration, main-green, `bun git-sync`, production verification, and bounded
Rust-cache cleanup rules aligned on those same surfaces.
* Keep parallel subagent lane, staged-set ownership, generated-artifact
ownership, and hook-failure guidance aligned across the Agent Operating
Manual, `$tuturuuu-agent-coordination`, `$tuturuuu-commit`, and Git
conventions docs.
* Keep platform release/version guidance aligned across `AGENTS.md`,
`$tuturuuu-web-release`, and the Web Docker Deployment runbook.
* Add marketplace metadata only when deciding where the plugin should be
installed from: repo-local `.agents/plugins/marketplace.json` or home-local
`~/.agents/plugins/marketplace.json`.
* Keep public skills.sh metadata aligned across `.claude-plugin/marketplace.json`
and `skills.sh.json`; do not modify `.agents/skills` just to publish the
Tuturuuu plugin skills.
# Development
Source: https://docs.tuturuuu.com/build/development-tools/development
Learn how to preview changes locally.
**Prerequisite**: You should have installed [Node.js](https://nodejs.org)
(version 22 or higher, matching the repo's `engines.node` of `>=22`).
## Installation
Step 1. Install [bun](https://bun.sh/docs/installation) on your machine (if you don't already have it yet), by running the following command:
**macOS/Linux:**
```bash theme={null}
curl -fsSL https://bun.sh/install | bash
```
**Windows:**
```bash theme={null}
powershell -c "irm bun.sh/install.ps1 | iex"
```
## Why Bun?
We chose bun as our runtime and package manager based on its [design goals](https://bun.sh/docs#design-goals), which align perfectly with our platform's needs:
* **4x Faster Startup**: Bun processes start significantly faster than Node.js, improving development experience and CI/CD performance
* **Built-in TypeScript & JSX Support**: No need for additional transpilation setup - bun natively executes `.ts`, `.tsx`, and `.jsx` files
* **All-in-One Toolkit**: Combines runtime, package manager, bundler, test runner, and script runner in a single executable
* **Web Standards Compatibility**: Implements modern Web APIs like `fetch`, `WebSocket`, and `ReadableStream` out of the box
* **Node.js Compatibility**: Drop-in replacement for Node.js with full compatibility for existing projects
* **Better Performance**: Powered by JavaScriptCore engine with reduced memory usage and faster execution
These benefits make bun an ideal choice for our monorepo architecture and development workflow.
Step 2. **Configure Tiptap Pro Registry:**
This step is no longer needed.
Step 3. After configuring Tiptap Pro registry, you can install all dependencies by running the following command:
```bash standard theme={null}
bun install
```
```bash short-hand theme={null}
bun i
```
This repository's Bun configuration enforces a minimum release age of 1 day for
new npm package versions. If Bun skips a just-published release during install,
wait for the package to age past the gate or choose an older published version.
To complete the initial setup, please restart your IDE so that it can recognize the newly installed dependencies. Additionally, some recommended VS Code Extensions may only work after restarting your IDE.
If you're using VS Code, you can install following the recommended extensions
that will help you with the development process:
[Biome](https://marketplace.visualstudio.com/items?itemName=biomejs.biome)
(this repo standardizes on Biome for linting and formatting via `biome.json`,
not ESLint or Prettier),
[Vitest](https://marketplace.visualstudio.com/items?itemName=vitest.explorer),
[Tailwind CSS
IntelliSense](https://marketplace.visualstudio.com/items?itemName=bradlc.vscode-tailwindcss),
[Version
Lens](https://marketplace.visualstudio.com/items?itemName=pflannery.vscode-versionlens),
[Error
Lens](https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens),
[Pretty TypeScript
Errors](https://marketplace.visualstudio.com/items?itemName=YoavBls.pretty-ts-errors),
[Material Icon
Theme](https://marketplace.visualstudio.com/items?itemName=PKief.material-icon-theme).
### Zed
This monorepo tracks project-level Zed settings in `.zed/settings.json`.
Opening the repository root in Zed can otherwise trigger expensive scans across
generated output and local dependency trees, especially `node_modules`,
`.turbo`, `.next`, package `dist` directories, Flutter build output, Rust
`target`, and Python virtualenv/cache folders.
Keep large generated directories in `file_scan_exclusions`. Zed's
`file_scan_exclusions` setting replaces the default list instead of extending
it, so include Zed's default VCS/system exclusions whenever adding project
exclusions. The project uses TypeScript 7 RC's `tsc` for compiler checks and a
single TypeScript language server path for TypeScript, TSX, and JavaScript;
avoid enabling competing TypeScript language servers for the same language
unless you are deliberately comparing language server behavior.
## Development
### Start Next.js apps
To develop all apps and packages (without requiring a local setup), run the following command:
```bash bun theme={null}
bun dev
```
This command will start all Next.js apps in development mode. You can access
the platform app by visiting the following URL:
[https://tuturuuu.localhost](https://tuturuuu.localhost)
Local app development uses Portless, so each app gets a stable Tuturuuu
subdomain instead of relying on its fallback port. The app package `dev`
scripts run the repo's Portless-safe wrapper, and the underlying framework
command remains available through `dev:app` for direct port-based debugging.
The wrapper only removes stale static aliases for the same app when their
default backend port is closed, then starts Portless with the original app
config. This fixes old worktree aliases such as
`zalo-qr-chat-setup.tuturuuu.localhost` without cleaning certificates, deleting
unrelated routes, or touching other apps' active aliases. Package-local
`bun dev --force` delegates to `portless run --name --force` so Portless
can take over a live route when you explicitly request it.
Root app-specific commands such as `bun dev:chat`, `bun dev:calendar`, and
`bun dev:edu` reuse app servers that are already running. The launcher checks
active Portless routes first, then the app's default localhost port; if it finds
a direct localhost listener, it tries to register a matching Portless alias so
cross-app auth can still use the `*.tuturuuu.localhost` URL. Pass `--force` or
`--no-reuse` when you intentionally want the command to include already-running
app workspaces. Force mode starts the package `dev --force` path so Portless can
take over the matching route instead of leaving a stale static alias in place.
`bun dev:tanstack-web` starts the TanStack Start migration frontend at
`https://tanstack.tuturuuu.localhost`, with fallback port `7824`. It is the
dedicated frontend for the Next.js-to-TanStack migration and should call
Rust-owned APIs through `packages/internal-api` helpers instead of direct
browser access to protected `apps/web` routes. See
`platform/architecture/tanstack-rust-migration` for the route manifest, Docker,
E2E, benchmark gates, and Cloudflare Workers preparation.
For the Cloudflare-compatible preview shape, start the Worker entrypoints
instead of the normal Portless dev wrapper:
```bash theme={null}
bun wrangler dev --config apps/backend/wrangler.jsonc
bun --cwd apps/tanstack-web run preview:cloudflare
```
Use the same environment variable names as deployment:
`BACKEND_INTERNAL_TOKEN`, `BACKEND_PUBLIC_ORIGIN`, and
`BACKEND_INTERNAL_URL`. Keep values in your shell or ignored local env files;
do not add literal tokens or account-specific origins to docs, Wrangler config,
package manifests, or source. After deploying preview Workers, run
`bun smoke:cloudflare` with `BACKEND_WORKER_ORIGIN`,
`TANSTACK_WEB_WORKER_ORIGIN`, and `BACKEND_INTERNAL_TOKEN` to verify backend
health/readiness, protected migration status, missing/invalid token rejection,
and the TanStack root shell.
`bun dev:web` is intentionally lean: by default it starts only `apps/web` and
does not start the `@tuturuuu/types` or `@tuturuuu/supabase` package watch
builds. Use this default for app-route and UI work so the local process tree
stays small. When you are actively editing those package sources and need their
`dist` output rebuilt live, opt in explicitly:
```bash theme={null}
bun dev:web --with-shared-watchers
```
You can also pass `--no-shared-watchers` to other root app launchers when you
want the same low-memory posture for a focused app session.
`apps/web` does not mount React Query Devtools in its default development
provider. Keep the default off for faster cold route compiles and lower
Turbopack memory pressure; add a local, temporary devtools mount only while
debugging query cache behavior, then remove it before committing.
Keep the always-mounted public shell small. The mobile menu drawer, marketing
footer body, report-problem dialog, and authenticated dropdown-only affordances
should stay behind dynamic imports when possible, and shell icons should use
`@tuturuuu/icons/lucide-static` instead of the root `@tuturuuu/icons` entrypoint.
This prevents the public `/login` compile graph from pulling large Radix,
footer, or full icon-package chunks into every `bun dev:web` session.
`bun dev:chat` also manages the local chat realtime sidecar. It reuses
`localhost:7817` when a sidecar is already running; otherwise it starts
`apps/chat-realtime` so `apps/web` can proxy chat SSE traffic for `apps/chat`.
| App | Local URL |
| ------------- | ------------------------------------------ |
| Platform | `https://tuturuuu.localhost` |
| Calendar | `https://calendar.tuturuuu.localhost` |
| Chat | `https://chat.tuturuuu.localhost` |
| CMS | `https://cms.tuturuuu.localhost` |
| Drive | `https://drive.tuturuuu.localhost` |
| External | `https://external.tuturuuu.localhost` |
| Finance | `https://finance.tuturuuu.localhost` |
| Hive | `https://hive.tuturuuu.localhost` |
| Hive Realtime | `https://realtime.hive.tuturuuu.localhost` |
| Infra | `https://infra.tuturuuu.localhost` |
| Inventory | `https://inventory.tuturuuu.localhost` |
| Learn | `https://learn.tuturuuu.localhost` |
| Mail | `https://mail.tuturuuu.localhost` |
| Meet | `https://meet.tuturuuu.localhost` |
| Mind | `https://mind.tuturuuu.localhost` |
| Nova | `https://nova.tuturuuu.localhost` |
| Playground | `https://playground.tuturuuu.localhost` |
| QR | `https://qr.tuturuuu.localhost` |
| Rewise | `https://rewise.tuturuuu.localhost` |
| Shortener | `https://shortener.tuturuuu.localhost` |
| Tasks | `https://tasks.tuturuuu.localhost` |
| Teach | `https://teach.tuturuuu.localhost` |
| Track | `https://track.tuturuuu.localhost` |
When debugging without Portless, run an app's `dev:app` script directly. If
auth redirects or absolute links must stay on the raw listener port, set
`BASE_URL`, `WEB_APP_URL`, or the app-specific URL environment variable to that
legacy fallback port for the direct debug session.
Keep runtime defaults, auth redirects, and cross-app links on the Portless
origins during local development. Use the shared local app URL helpers instead
of hard-coding `localhost` fallbacks in satellite app constants or proxies, so a
browser session that starts on `tasks.tuturuuu.localhost` stays under the
`tuturuuu.localhost` namespace through login and return URLs.
For a native production-mode smoke test of `apps/web`, run the native build and
then start it through the same Portless hostname:
```bash theme={null}
bun run build:web
cd apps/web
bun run start
```
`bun run start` delegates to Portless and runs the built Next.js server through
`start:app`, so the browser URL remains `https://tuturuuu.localhost` while the
server listens on the backend port that Portless injects through `PORT`.
Use `bun dev:edu` when working on the education apps together. It starts Learn,
Teach, their shared packages, and the central web app so cross-app login can
complete locally.
Next.js still prints the internal listener port, for example
`http://localhost:4803`, because Portless assigns a free backend port to the app
process. App `dev:app` scripts also print a `Portless URL:` line from the
`PORTLESS_URL` environment variable; use that URL in the browser:
```text theme={null}
Portless URL: https://tuturuuu.localhost
```
If Turbo cannot prompt for the Portless sudo password while starting app dev
scripts, run the Portless setup helper first:
```bash theme={null}
bun portless:setup
```
The helper starts the Portless HTTPS proxy on port `443` before Turbo launches
package `dev` scripts. It no-ops when the proxy is already responding and skips
in non-interactive shells or CI. `bun setup` runs this helper after dependency
installation so fresh local checkouts are ready before the build step, then
invokes Turborepo through the repo-local `turbo:local` helper instead of any
globally installed Turbo binary. To install Portless as an OS startup service
instead of only starting the current proxy daemon, run:
```bash theme={null}
bun portless:setup -- --service
```
### Diagnose a broken local environment
When a local app suddenly cannot reach another app — for example the Inventory
landing page throws `InternalApiError: 404` because its server-side call to
`https://tuturuuu.localhost/api/...` is not routed — run the doctor:
```bash theme={null}
bun doctor
```
It checks the Node runtime, Docker daemon, local Supabase ports, local Redis
ports, apps/web Redis env wiring, app-local Supabase env consistency, whether
the Portless proxy is responding on port `443`, and the health of every
registered Portless route. The report uses colored `OK`, `WARN`, and `FAIL`
statuses in interactive terminals. Supabase env mismatch warnings are compact
by default so secret-like values stay out of the normal output; use the verbose
view when you need redacted per-file fingerprints:
```bash theme={null}
bun doctor --verbose
```
The most common failure after overlapping or crashed dev sessions is a
**stale routing table**: the proxy still points a live app route at a dead
dev-server port, so cross-app internal API calls return `404`. The doctor flags
those routes as `DEAD` and exits non-zero. Static aliases for apps you simply
are not running are reported as warnings, not failures, and include an exact
cleanup command. An initialized but empty Portless route table is treated as
healthy because no apps may be running yet.
Apply the safe automatic Portless repair (start the proxy, or reset it when
routes are stale) and re-run the checks:
```bash theme={null}
bun doctor --fix
```
`bun setup` deliberately no-ops when a proxy is already responding, so it does
**not** clear a stale routing table on its own. To force a clean restart — stop
the proxy, prune orphaned dev servers from crashed sessions, then start a fresh
proxy that apps re-register against — run:
```bash theme={null}
bun portless:reset
```
After a reset, restart your dev servers (for example `bun dev:inventory`) so
each app re-registers its route with the new proxy.
If `bun doctor` still warns about an inactive static alias after a reset, that
alias was created with `portless alias` and is not deleted by proxy resets.
Start the app if you still use the alias, or remove the alias shown in the
doctor output:
```bash theme={null}
bunx portless alias --remove
```
If the doctor warns that app env files use mismatched Supabase values, choose
the intended source app and keep these three keys together:
`NEXT_PUBLIC_SUPABASE_URL`, `NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY`, and
`SUPABASE_SECRET_KEY`. For quick local development, copy `apps/web/.env.local`
to app-local `.env.local` files:
```bash theme={null}
bun dev:sync:apps
```
Preview the target files first with:
```bash theme={null}
bun dev:sync:apps:dry-run
```
This sync compares app env files only. The root `.env.local` can intentionally
target a different Docker or deployment workflow without keeping the app
mismatch warning alive.
### Diagnose web dev memory and cache pressure
When `apps/web` feels slow or Activity Monitor reports high Node.js usage, start
with the dev diagnostic before moving routes into another app:
```bash theme={null}
bun diagnose:dev:web
```
Read the report by pressure type:
* `Live Next dev process RSS` is the memory held by the running `next dev`
process tree. If this is high, inspect the top processes and the latest trace
spans before changing architecture.
* `apps/web/.next/dev/cache/turbopack` is generated Turbopack filesystem cache.
Multi-GB cache growth can make disk usage look like a live memory leak even
when process RSS is modest.
* `apps/web/.next/cache` can include the Next 16.3 Turbopack build filesystem
cache from local builds. It speeds repeated local `next build` runs, but it is
still local artifact state and must stay out of Turborepo task outputs.
* `.turbo/cache` is Turborepo task cache. It is separate from the live Next.js
dev server and should not be treated as app RSS.
* `Build memory policy` reports Docker and `next build` memory budgets. Those
caps explain build behavior only; do not compare them directly to `next dev`
RSS.
* `Likely pressure` calls out whether the current evidence points to live RSS,
cache growth, watcher limits, or build-only memory settings.
If the report points at Turbopack cache growth, stop the web dev server and
preview the targeted cleanup first:
```bash theme={null}
bun clean:dev:web --dry-run
```
Then remove the web Next dev output:
```bash theme={null}
bun clean:dev:web
```
Use `bun clean:dev:web --all-next-dev --dry-run` when several app dev caches
exist and you want to inspect every `apps/*/.next/dev` target before deleting
them. Add `--include-turbo-cache` only when the report also calls out
`.turbo/cache` growth after branch or dependency graph churn. Avoid the broad
root `bun clean` command for this problem because it deletes dependency and
build state that is unrelated to the current dev-server slowdown.
Then restart the targeted app command you were using, such as `bun dev:web`.
Avoid deleting unrelated app output while other agents or local sessions are
working in the same checkout.
If live RSS is the real pressure, use the latest `.next/dev/trace` spans and
the matching process tree to reduce the compiled import graph first. Favor
narrow package subpath imports and lazy-loaded settings panels before splitting
an app boundary. A dedicated infrastructure app only reduces `apps/web` memory
when infrastructure pages leave the web route graph; running `apps/web` and an
infrastructure app together can increase total RAM.
Native local dev leaves Watchpack polling off unless the environment explicitly
sets `WATCHPACK_POLLING`. This keeps file watching lighter on macOS and Linux.
If the diagnostic reports watcher `EMFILE` errors or route discovery stops
updating on your machine, restart with polling enabled:
```bash theme={null}
WATCHPACK_POLLING=true bun dev:web
```
If the doctor warns that apps/web is missing Redis env, configure the local SRH
bridge and start the Redis stack in one step:
```bash theme={null}
bun redis:setup
```
`bun redis:setup` writes `UPSTASH_REDIS_REST_URL=http://localhost:8079` and the
matching local token into `apps/web/.env.local`, then runs `bun redis:start`.
Restart `bun dev:web` after setup so the app picks up the new env values.
To clear local Redis data while keeping the same local env wiring, run:
```bash theme={null}
bun redis:reset
```
This starts the local Redis stack if needed, then runs `FLUSHALL` inside the
Redis container.
To stop the local dev environment, run:
```bash theme={null}
bun stop
```
This stops this repo's Portless-registered Next.js dev servers, prunes Portless
routes, removes stale static aliases for this project, stops local Supabase,
and finishes by running `bun doctor`. Preview the cleanup first with:
```bash theme={null}
bun stop --dry-run
```
### Start Local Supabase Instance
To start a local supabase instance (database), run the following command:
```bash bun theme={null}
bun sb:start
```
This command will start a local supabase instance on your machine. You can
access the supabase instance by visiting the following URL:
[http://localhost:8003](http://localhost:8003)
You need to have Docker installed and running on your machine to start a local
supabase instance.
### Stop Local Supabase Instance
To stop the local supabase instance, run the following command:
```bash standard theme={null}
bun sb:stop
```
`bun stop` is broader than Supabase. It stops this repo's local dev servers,
cleans stale Portless aliases, stops Supabase, and verifies the result with
`bun doctor`.
### Better Development Experience
In case you want to run all local development servers, run the following command:
```bash bun theme={null}
bun devx
```
Running `devx` will:
1. Stop the currently running supabase instance and save current data as backup (if there is any)
2. Install all dependencies
3. Start a new supabase instance (using backed up data)
4. Start all Next.js apps in development mode
If you want to have the same procedure without the backup, you can run `bun devrs` instead. This will:
1. Stop the currently running supabase instance (if there is any)
2. Install all dependencies
3. Start a new supabase instance (with clean data from seed.sql)
4. Start all Next.js apps in development mode
In case you don't want to run a local supabase instance, you can run `bun dev`
instead.
### Local development
#### Seed accounts
There are 5 seed accounts that are already set up for local development:
1. [local@tuturuuu.com](mailto:local@tuturuuu.com)
2. [user1@tuturuuu.com](mailto:user1@tuturuuu.com)
3. [user2@tuturuuu.com](mailto:user2@tuturuuu.com)
4. [user3@tuturuuu.com](mailto:user3@tuturuuu.com)
5. [user4@tuturuuu.com](mailto:user4@tuturuuu.com)
You can use any of these accounts to log in to the app and quickly test the functionality of the app, since they are already set up with the necessary data.
#### Authentication
A local mail server (**InBucket**) is automatically set up by Supabase to handle authentication emails.
You can access the mail server by visiting the following URL:
[http://localhost:8004](http://localhost:8004)
## Build
To build all apps and packages, run the following command:
```bash bun theme={null}
bun run build
```
## Test
To run all tests, run the following command:
```bash bun theme={null}
bun run test
```
To run the Flutter mobile quality gate, use:
```bash bun theme={null}
bun check:mobile
```
`bun check:mobile` runs Dart format, Flutter analyze, and Flutter tests for
`apps/mobile`. Flutter test output is streamed by default so slow suites still
show progress instead of appearing stuck.
### Mobile release builds
The native build commands verify or hydrate each flavor's ignored Firebase SDK
configuration from the `tuturuuu-mobile` Firebase project before compiling. The
Firebase CLI must be authenticated with access to that project.
```bash theme={null}
# Production Android App Bundle
bun build:android
# Signed production iOS IPA
bun build:ios
```
The default iOS command requires an Apple signing identity and provisioning
profile. `bun build:ios:ipa` is an alias for the same signed release build. To
compile the production app on a machine without signing credentials, use:
```bash theme={null}
bun build:ios:unsigned
```
`bun build:android` only produces signed production bundles. Configure the
protected upload keystore through all four `ANDROID_KEYSTORE_*` environment
variables or `apps/mobile/android/key.properties`; the command fails before
compilation when signing is absent and verifies the resulting AAB with
`jarsigner`. The Mobile Store Deployment workflow hydrates these credentials in
its protected runner and executes this exact command. Development builds remain
available through `bun build:android:dev`.
To hydrate or validate a native Firebase configuration without compiling, run:
```bash theme={null}
bun mobile:firebase:configure --environment production --platform android
bun mobile:firebase:configure --environment production --platform ios
```
Tests are still a work in progress. We're currently working on adding tests to all packages to ensure the best quality possible.
## Git Conventions
Tuturuuu uses standardized conventions for Git commits and branch naming. For more information, see the [Git Conventions](/build/development-tools/git-conventions) guide to learn how to format your commits and branch names to align with our workflow.
# Documenting
Source: https://docs.tuturuuu.com/build/development-tools/documenting
Learn how to contribute to our documentation.
**Prerequisite**: You should have [Docker](https://www.docker.com/) and Bun
installed, and the [Tuturuuu
Monorepo](https://github.com/tutur3u/platform) cloned locally.
Step 1. Start the docs site from the repo root:
```bash theme={null}
bun dev:docs
```
This uses the repo-managed Docker docs stack, which runs Mintlify inside a
Node.js 24 container instead of relying on a globally installed CLI.
The documentation website is now available at `http://localhost:3000`.
To stop the stack:
```bash theme={null}
bun dev:docs:down
```
### Custom Ports
The docs stack exposes port `3000` by default through `${DOCS_PORT:-3000}` in
`docker-compose.docs.yml`. You can use the `--port` flag to customize the
host-side port binding. For example:
```bash theme={null}
bun dev:docs -- --port 3333
```
If the chosen port is already taken, Docker will fail to bind the host port.
## Runtime Details
* The docs image is defined in `apps/docs/Dockerfile`.
* The development stack is defined in `docker-compose.docs.yml`.
* The root command wrapper lives in `scripts/docker-docs.js`.
* The container pins the runtime to Node.js 24 and installs Mintlify inside the
image.
## Writing Documentation
### File Structure
Each documentation file should follow this structure:
```mdx theme={null}
---
title: "Page Title"
description: "Brief description of the page content"
updatedAt: "YYYY-MM-DD"
---
# Page Title
Content goes here...
```
### Content Guidelines
1. **Use clear headings** to structure your content
2. **Include code examples** with proper syntax highlighting
3. **Add cross-references** to related documentation
4. **Keep content up-to-date** by updating the `updatedAt` field
5. **Use consistent formatting** throughout
### Code Examples
When including code examples:
* Use proper syntax highlighting
* Include TypeScript types
* Show both server and client examples
* Include error handling
* Test examples before committing
### Images and Assets
* Store images in the `images/` directory
* Use descriptive filenames
* Optimize images for web
* Include alt text for accessibility
## Contributing
### Before You Start
1. Check if documentation already exists
2. Look for similar patterns in existing docs
3. Follow the established structure
4. Read the [Organization Guide](/overview/organization-guide) for structure
### Making Changes
1. **Create a branch** for your documentation changes
2. **Write clear, concise content** that's easy to understand
3. **Include practical examples** that developers can use
4. **Update related documentation** if needed
5. **Test your changes** locally with `bun dev:docs`
### Review Process
1. **Self-review** your changes for clarity and accuracy
2. **Check links** to ensure they work correctly
3. **Verify code examples** are correct and complete
4. **Update the table of contents** if adding new sections
5. **Submit a pull request** with a clear description
## Best Practices
### Writing Style
* **Be concise** but comprehensive
* **Use active voice** when possible
* **Write for your audience** (developers)
* **Include context** for complex topics
* **Use consistent terminology** throughout
### Organization
* **Group related content** together
* **Use clear section headings**
* **Include a table of contents** for long pages
* **Cross-reference** related documentation
* **Keep examples practical** and realistic
### Maintenance
* **Update documentation** when APIs change
* **Remove outdated information** promptly
* **Keep examples current** with latest versions
* **Review documentation regularly** for accuracy
* **Ask for feedback** from other developers
# Git Conventions
Source: https://docs.tuturuuu.com/build/development-tools/git-conventions
Learn how Tuturuuu uses Conventional Commits and Branch naming to improve development workflow.
**Prerequisite**: You should be familiar with basic Git operations and
understand the [Monorepo Architecture](/build/development-tools/monorepo-architecture) of our
codebase.
## Overview
Tuturuuu follows standardized conventions for Git commits and branch naming to improve collaboration, automate releases, and maintain a clean, navigable repository history. We use two main specifications:
1. [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) for structured commit messages
2. [Conventional Branch](https://conventional-branch.github.io/) for consistent branch naming
These conventions help automate our CI/CD workflows, generate changelogs, and make our development process more efficient.
## Conventional Commits
### What are Conventional Commits?
Conventional Commits is a specification for adding human and machine-readable meaning to commit messages. It provides a set of rules for creating an explicit commit history, making it easier to write automated tools on top of.
The basic structure of a conventional commit is:
```
[optional scope]:
[optional body]
[optional footer(s)]
```
### Types
Tuturuuu uses the following commit types:
| Type | Description | Example |
| ---------- | ----------------------------------------------- | ------------------------------------------------ |
| `feat` | New feature or enhancement | `feat: add dark mode support` |
| `fix` | Bug fix | `fix: prevent crash when user data is undefined` |
| `docs` | Documentation changes | `docs: update installation instructions` |
| `style` | Code style changes (formatting, no code change) | `style: format code with prettier` |
| `refactor` | Code refactoring | `refactor: simplify authentication logic` |
| `perf` | Performance improvements | `perf: optimize database queries` |
| `test` | Add or fix tests | `test: add unit tests for auth middleware` |
| `build` | Changes affecting build system or dependencies | `build: update dependency to fix security issue` |
| `ci` | Changes to CI configuration files and scripts | `ci: simplify workflow branch filters` |
| `chore` | Routine tasks, maintenance | `chore: update package.json metadata` |
### Scopes
Scopes provide additional context about which part of the codebase is affected. In our monorepo structure, we often use package or app names as scopes:
```
feat(web): add new dashboard layout
fix(ui): correct button alignment in mobile view
chore(types): update supabase database types
```
### Breaking Changes
Breaking changes must be indicated by adding a `!` after the type/scope or by using a `BREAKING CHANGE:` footer:
```
feat(api)!: require authentication for all endpoints
BREAKING CHANGE: All API endpoints now require authentication tokens.
```
### Examples
Here are some examples of conventional commits used in our codebase:
```
feat(web): add new calendar integration
fix(supabase): correct permission issue in auth policy
docs(readme): update development setup instructions
style: apply consistent formatting with prettier
refactor(utils): simplify date formatting functions
perf(queries): optimize workspace user loading
test(auth): add tests for token verification
build(deps): update Next.js to v14
ci(vercel): add automatic preview deployments
chore(release): publish packages
```
## Conventional Branches
### What are Conventional Branches?
Conventional Branch refers to a structured naming convention for Git branches that makes it easier to identify branches by type and purpose.
The basic structure is:
```
/
```
### Branch Types
Tuturuuu uses the following branch types:
| Type | Description | Example |
| ------------ | -------------------------------------- | ------------------------------- |
| `main` | Main development branch | `main` |
| `feature` | For new features | `feature/user-dashboard` |
| `feat` | Short feature prefix also accepted | `feat/invite-link-permissions` |
| `fix` | Short bug-fix prefix also accepted | `fix/invite-membership-check` |
| `bugfix` | For bug fixes | `bugfix/login-error` |
| `hotfix` | For urgent fixes | `hotfix/security-vulnerability` |
| `release` | For preparing releases | `release/v1.2.0` |
| `chore` | For maintenance tasks | `chore/update-dependencies` |
| `docs` | For documentation-only work | `docs/git-conventions-refresh` |
| `style` | For formatting or stylistic cleanups | `style/biome-pass` |
| `refactor` | For internal code restructuring | `refactor/task-query-shell` |
| `perf` | For performance-focused changes | `perf/workspace-user-search` |
| `dependabot` | Automated dependency update branches | `dependabot/npm_and_yarn/next` |
| `claude` | AI-assisted scratch or experimental PR | `claude/rate-limit-audit` |
Release Please bot branches are also accepted with the
`release-please--branches--` prefix. Do not create human-authored branches with
that prefix.
### Naming Rules
1. Use lowercase alphanumeric characters and hyphens
2. Keep names concise but descriptive
3. Include ticket/issue numbers when applicable
4. Avoid special characters (except hyphens)
### Examples
```
feature/workspace-sharing
feat/public-course-sharing
fix/file-upload-error
bugfix/auth-vulnerability
release/v2.5.0
chore/update-react-18
docs/git-conventions-refresh
feature/issue-123-user-profile
```
### Merge Commit Exception
Authored commits should follow Conventional Commits. The common exception is a
Git-generated merge commit when syncing a long-lived branch, for example:
```bash theme={null}
Merge branch 'main' into feat/studying-platform
```
Keep that merge summary when the goal is to preserve the branch's merge history.
Use a conventional commit message again for the next authored commit after the
merge.
## How These Conventions Improve Our Workflow
### 1. Automated Changelog Generation
Our conventional commits are used to automatically generate changelogs for releases. Different commit types are categorized accordingly:
```markdown theme={null}
# Changelog
## Features
- add dark mode support (#123)
- add calendar integration (#124)
## Bug Fixes
- prevent crash when user data is undefined (#125)
## Documentation
- update installation instructions (#126)
```
### 2. Semantic Versioning
Conventional commits help determine the next semantic version for packages:
* `fix:` commits trigger a PATCH increment (1.0.0 → 1.0.1)
* `feat:` commits trigger a MINOR increment (1.0.0 → 1.1.0)
* Commits with `BREAKING CHANGE` trigger a MAJOR increment (1.0.0 → 2.0.0)
### 3. Automated Release PRs And Package Publishing
Release Please reads Conventional Commits from `production`, updates
`release-please-config.json` packages through `.release-please-manifest.json`,
and opens one combined release PR with package versions and changelogs. Package
publish workflows do not generate versions; they publish only after a
release-please version bump lands on `production`. Local manifests keep
Tuturuuu workspace dependencies as `workspace:*`; npm release workflows rewrite
the checked-out package manifest with
`scripts/ci/prepare-npm-package-manifest.js` immediately before `npm pack` so
published artifacts contain installable npm version ranges. Package-included
`file:` tarball dependencies, such as `@tuturuuu/ui`'s vendored SheetJS tarball,
must not leak a consumer-relative `file:` range into the npm manifest. Package
preparation expands the checked-in archive into the artifact, redirects the
package export to those immutable bytes, and removes the install-time dependency
edge. Do not rewrite the vetted archive to a mutable external tarball URL.
Before any build or pack work starts, package workflows run
`scripts/ci/package-release-readiness.js gate-package-release packages/`.
The gate checks publishable Tuturuuu workspace dependency versions on npm once,
dispatches missing dependency workflows, and exits green without occupying a
runner while dependencies are pending. If the related dependency workflow for
the same production SHA failed, completed successfully without npm visibility,
or cannot be inspected, the gate fails immediately. Publish jobs still wait for
their own version to become visible after `npm publish`, then a separate
non-OIDC job dispatches direct dependent package workflows. Workflow-published
package manifests must carry
provenance-compatible `repository` metadata for `tutur3u/platform`; otherwise
npm rejects trusted publishes with `E422`. For example,
`release-types-package.yaml` triggers from production changes to
`packages/types/package.json`:
When bringing the generated release PR branch back to `main`, run
`bun git-release-please` from a clean `main` checkout. The helper fetches the
latest `release-please--branches--production` branch, merges it without
committing, syncs `platform-version.txt` into the platform badge constant and
test expectation, runs `bun ff`, stages the resolved merge, then runs `bun check`
directly before the merge commit lands. When the staged release merge includes
`apps/mobile` paths, the helper also runs `bun check:mobile`. If you are already
inside a manual merge, run
`bun release:sync-platform-version` to resolve the recurring
`TUTURUUU_PLATFORM_VERSION` conflict before staging the merge.
```yaml theme={null}
on:
push:
branches: [production]
paths:
- "packages/types/package.json"
# ...
jobs:
check-version-bump:
if: github.ref == 'refs/heads/production' && needs.check-ci.outputs.should_run == 'true'
# ...
```
### 4. Better Code Reviews
With conventional commits and branches, it's easier to understand the purpose of a pull request at a glance:
* A PR from `feature/user-dashboard` with commits like `feat: add user stats widget` clearly indicates a new feature
* A PR from `bugfix/auth-issue` with commits like `fix: prevent token expiration error` indicates a bug fix
### 5. Simplifying Navigation
Conventional branch names make it easier to navigate the repository history and find specific changes. For example:
```bash theme={null}
# Find all feature branches
git branch --list "feature/*"
# Find branches related to authentication
git branch --list "*auth*"
```
## Tools and Enforcement
We use a dedicated CI/CD check to enforce our Git conventions:
### Branch Naming Check
We use a GitHub Action workflow to verify that branch names follow our convention:
```yaml theme={null}
name: Branch Name Check
on:
push:
branches-ignore:
- main
- production
jobs:
check-branch-name:
name: Check branch name
runs-on: ubuntu-latest
steps:
- name: Check branch name
run: |
BRANCH_NAME=${GITHUB_REF#refs/heads/}
if ! [[ $BRANCH_NAME =~ ^(feature|feat|fix|bugfix|hotfix|release|chore|docs|style|refactor|perf|dependabot|claude)/.+|^release-please--branches--.+ ]]; then
echo "❌ Branch name '$BRANCH_NAME' doesn't follow the conventional branch format."
echo "Branch name should be in format: type/description"
echo "Allowed types: feature, feat, fix, bugfix, hotfix, release, chore, docs, style, refactor, perf, dependabot, claude"
echo "Release Please bot branches are also allowed with the release-please--branches-- prefix."
exit 1
else
echo "✅ Branch name follows convention: $BRANCH_NAME"
fi
```
## Best Practices
### Writing Good Commit Messages
1. Use the imperative mood ("add" not "added" or "adds")
2. Keep the description in lowercase to match the repository's observed convention
(for example, `feat(backend): migrate nova team read`)
3. Do not end the description with a period
4. Keep the description under 72 characters
5. Use the body to explain the what and why, not the how
Example of a well-formatted commit:
```
feat(auth): add multi-factor authentication support
Implement TOTP-based multi-factor authentication to improve security.
The implementation follows the RFC 6238 standard.
Closes #123
```
### Coordinating Commits In Shared Checkouts
When multiple agents or humans may commit in the same checkout, use the commit
window before changing the staged set. The lock is advisory and lives under the
ignored `tmp/agent-coordination/` directory, so it protects the Git index and
commit operation without becoming part of a commit. Claims default to 10
minutes and may only be 5-10 minutes, so claim only when ready to stage,
inspect, and commit.
This coordination is **per-checkout** and **harness-agnostic**. The commit
window and the `tmp/agent-coordination/` notes both live in the working
directory, so they coordinate every agent or human sharing one checkout —
parallel Codex or Claude Code sessions, background tasks, and same-directory
subagents — regardless of which tool they run under. A separate `git worktree`
has its own lock file, its own notes, and its own Git index, so the window does
not span worktrees; separate worktrees stay isolated by being on different
branches and integrate through the shared remote instead. In a hot shared
checkout, commit your owned paths promptly in small scoped commits: a large
unstaged set can be discarded by a concurrent rebase, `reset --hard`,
`checkout`, or `stash` from any agent or human. (`bun git-sync` is isolated and
safe; manual destructive Git is not.)
```bash theme={null}
bun git-commit-window status
bun git-commit-window claim --owner "" --scope "type(scope): subject"
git add path/to/file-a path/to/file-b
git diff --cached --stat
git diff --cached --name-only
git commit -m "type(scope): subject"
bun git-commit-window release --token
```
If another agent owns the window and it is appropriate to wait, use `wait`. The
command sleeps until the active lock is released or expires, then claims the
window before reporting that it is safe to proceed. The waiting period can be
longer than the claim TTL, but the claimed window remains capped at 10 minutes:
```bash theme={null}
bun git-commit-window wait --owner "" --scope "type(scope): subject"
```
Use `--allow-staged` only after inspecting existing staged files. The commit
window does not grant ownership of files or permission to stage unrelated paths.
Existing staged files are owned by the staging agent or coordinator until
explicitly reassigned. If a file appears as `MM`, both its staged and unstaged
diffs need owner review before commit.
Let commit hooks run by default. This is a proof-gated no-verify path: use
`git commit --no-verify` only when the current agent can prove its exact staged
paths would pass the checks normally covered by `bun check`. The proof packet
should include reviewed
`git status --short`, `git diff --cached --stat`, and
`git diff --cached --name-only` output; the touched files or narrow path group;
the separated checks that covered each affected `bun check` component; any
skipped components with path-based rationale; unrelated dirty files excluded
from the claim; and `bun check:mobile` coverage when `apps/mobile` is touched.
If ownership is unclear, proof is incomplete, or the check mapping is uncertain,
do not bypass the hook.
Commit hooks and repo checks can read the whole worktree, including files owned
by other agents. If they fail on unrelated dirty files, release the commit
window and report the blocker instead of formatting, fixing, staging, or
committing those files for convenience. Only the staged-set owner may use the
proof-gated no-verify path, and only with exact-path evidence for the staged
files.
### Branch Management
1. Create branches from the latest `main` branch
2. Keep branches focused on a single task or issue
3. Regularly rebase long-lived branches on `main` to avoid merge conflicts
4. Delete branches after they've been merged
### Synchronizing Release Branches
Use the root command below when `production` should point at the same commit as
`main`:
```bash theme={null}
bun git-sync
```
The command fetches `origin`, creates a temporary detached worktree, refreshes
`main`, and fast-forwards `production` to the current `main` commit from that
temporary worktree before pushing those branches. It finishes by fetching again
and verifying that local and remote `main` and `production` both resolve to the
same commit. The checkout you started from is left untouched, so uncommitted
work on another branch can keep running while the release refs are synchronized.
If a branch that needs to move is already checked out in another worktree, Git
will refuse to force-update that branch. Switch that worktree away from the
branch or update it manually, then rerun `bun git-sync`.
`bun git-sync` does not create commits for you. Commit all intended changes on
`main` first; if `production` contains commits that are not already in `main`,
reconcile the branch manually before rerunning the command.
Use `--only-branch` to update one active sync branch while leaving the other
untouched:
```bash theme={null}
bun git-sync --only-branch production
```
`main` is still refreshed locally as the source commit, but only `production` is
fast-forwarded and pushed. Use `--only-branch main` when you only want to pull
and push `main`.
The retired `staging` branch is no longer a supported `--only-branch` target.
The Supabase staging environment still exists, but its workflow is driven from
`main`.
Use `--current-branch` or `-c` when `main` and `production` should move to the
latest commit on the branch you currently have checked out instead of latest
`origin/main`:
```bash theme={null}
bun git-sync --current-branch
bun git-sync -c
```
Current-branch mode still uses a temporary detached worktree, leaves the checkout
you started from untouched, and only moves the selected sync branches. It does
not push the source branch unless the source branch is itself one of the selected
sync branches, such as running from `main`.
Combine `-c` with `--only-branch` when only one long-lived branch should move to
the current checkout's commit:
```bash theme={null}
bun git-sync -c --only-branch production
```
Use `--no-push` for local-only synchronization:
```bash theme={null}
bun git-sync --no-push
bun git-sync --only-branch production --no-push
bun git-sync -c --no-push
```
With `--no-push`, the command fetches, pulls with `--ff-only`, and fast-forwards
the selected local branches, but it never pushes to `origin`. Final verification
checks local refs only.
### Pull Request Workflow
1. Create a branch with the appropriate type based on the work
2. Make commits using conventional commit messages
3. Push the branch and create a pull request
4. Use conventional commit style for the PR title
5. After approval and merge, delete the branch
## Conclusion
Following these Git conventions helps us maintain a clean, understandable repository history, automate release processes, and improve collaboration across the team. By standardizing both commit messages and branch names, we create a more efficient development workflow.
For more information, refer to the official documentation for [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) and [Conventional Branch](https://conventional-branch.github.io/).
# Local Supabase Development
Source: https://docs.tuturuuu.com/build/development-tools/local-supabase-development
Learn how to work with Supabase locally in the Tuturuuu development workflow.
**Prerequisite**: You should have installed
[Docker](https://www.docker.com/products/docker-desktop/) and followed the
[Development](/build/development-tools/development) setup guide.
## Overview
Tuturuuu utilizes Supabase for database management and authentication. This guide explains how to work efficiently with Supabase in the local development workflow.
## Basic Commands
### Starting Supabase
To start a local Supabase instance tailored for Tuturuuu:
```bash theme={null}
bun sb:start
```
This command launches a local Supabase instance on your machine. The local ports
are pinned in `apps/database/supabase/config.toml`:
| Service | URL / Port |
| ---------------------- | -------------------------------------------------------- |
| REST/Auth API | [http://localhost:8001](http://localhost:8001) |
| Postgres database | `postgresql://postgres:postgres@localhost:8002/postgres` |
| Studio (dashboard) | [http://localhost:8003](http://localhost:8003) |
| InBucket (test emails) | [http://localhost:8004](http://localhost:8004) |
Run `bun sb:status` at any time to print the live URLs, keys, and the DB
connection string for your running instance. Prefer reading these values from
`bun sb:status` over hard-coding them, since they are the source of truth for
the current container.
### Stopping Supabase
To stop the local Supabase instance:
```bash theme={null}
bun sb:stop
```
### Checking Status
To view the current URLs and status of your local Supabase instance:
```bash theme={null}
bun sb:status
```
## Development Workflow
### Recommended Startup
When developing for Tuturuuu, you have several options to start your environment:
1. **Standard Approach**: Start Next.js apps and Supabase separately.
```bash theme={null}
bun dev # Starts all Next.js apps
bun sb:start # Starts Supabase
```
2. **Enhanced Development Experience**: Use the `devx` command for a streamlined setup.
```bash theme={null}
bun devx
```
This command:
* Stops any running Supabase instance and saves current data as backup
* Installs all dependencies
* Starts a new Supabase instance (using backed up data)
* Starts all Next.js apps in development mode
3. **Fresh Database Setup**: When switching branches with potential schema changes.
```bash theme={null}
bun devrs
```
This command (`bun sb:stop && bun sb:start && bun sb:reset && bun dev`):
* Stops any running Supabase instance (without backup)
* Starts a new Supabase instance
* Resets the database to use the latest schema and seed data
* Starts all Next.js apps in development mode
App-scoped variants also exist (for example `bun devrs:web`, `bun devrs:tasks`)
that reset Supabase and then start only that app's dev server.
Use `bun devrs` when switching between branches that might have different
database migrations to ensure your local database schema matches the branch
you're working on.
### Syncing With Schema Changes
If you're keeping your Next.js server running but need to reset your database to match the current branch's schema:
```bash theme={null}
bun sb:reset
```
This command will reset your local database to use the latest schema definitions and automatically regenerate TypeScript types. After the reset, the database package also runs the local AI credits bootstrap, which syncs gateway models through the local Supabase REST API and retries the short PostgREST schema-cache warmup window that can happen immediately after containers restart.
## Database Schema Management
### Making Schema Changes
There are two approaches to modifying the database schema:
#### 1. Using the Supabase UI
1. Navigate to your local Supabase Studio at [http://localhost:8003](http://localhost:8003)
2. Make your changes through the UI
3. Generate a migration file:
```bash theme={null}
bun sb:diff
```
This creates a migration file based on the differences between your current schema and the previous state.
Be cautious when using `sb:diff` for schema changes. If you rename columns or
tables, the migration will drop the old ones and create new ones, which can
result in data loss in production environments.
#### 2. Creating Manual Migrations
For more control, you can create empty migration files and populate them manually:
```bash theme={null}
bun sb:new add_custom_function
```
This creates a new empty migration file in `apps/database/supabase/migrations`
using the provided migration name. If you omit the name, the helper uses
`new_migration`.
### Applying Migrations
After creating a migration, apply it to your local database:
```bash theme={null}
bun sb:up
```
This same process is used to keep our production database up-to-date with the
schema defined in the `production` branch.
### Validating an Exact Worktree in Isolation
Database plans pinned to an older commit must not reset or reuse the ordinary
`tuturuuu` stack when its migration history belongs to another checkout. Run the
disposable validator from the repository root instead:
```bash theme={null}
bun --cwd apps/database sb:validate:isolated
```
For a focused pgTAP file, pass its path relative to `apps/database`:
```bash theme={null}
bun --cwd apps/database sb:validate:isolated --test supabase/tests/workspace-creator-membership.sql
```
To replace the checked-in database types only after the disposable reset and
focused pgTAP test both succeed, add the exact approved output path:
```bash theme={null}
bun --cwd apps/database sb:validate:isolated --typegen packages/types/src/supabase.ts --test supabase/tests/habit-tracker-write-rls.sql
```
The type generator uses the same pinned Supabase binary and disposable project,
includes the `public`, `private`, and `storage` schemas, buffers its output, and
atomically replaces `packages/types/src/supabase.ts` only after a successful,
non-empty result. Reset, pgTAP, type generation, or output-write failure keeps
the existing generated file unchanged. No other output path is accepted.
The runner derives a Docker-safe project identity from the exact worktree path
and Git commit, selects an available deterministic eight-port block, and prints
both before starting. It copies only Git-tracked files from
`apps/database/supabase` into a directory named
`tuturuuu-supabase-*` under the operating system's temporary directory. The
temporary config receives the isolated project id and ports; the tracked
`supabase/config.toml` is never edited.
The isolated lifecycle starts that project, resets it from the copied exact-base
migrations, runs the full or focused pgTAP suite, and stops that project with
`--no-backup` in cleanup. It never invokes `supabase stop --all` or the default
`sb:stop`/`sb:reset` aliases. Ordinary development continues to use
`sb:start`, `sb:up`, and `sb:reset` unchanged.
SIGINT and SIGTERM request the same scoped cleanup while preserving the failing
command or signal exit code. A hard process or machine termination can leave the
temporary directory behind; its `.tuturuuu-isolated-supabase.json` records the
project id, worktree commit, ports, and last lifecycle step. Resume or explicitly
clean that recorded directory without guessing its Docker identity:
```bash theme={null}
bun --cwd apps/database sb:validate:isolated --resume /tmp/tuturuuu-supabase-EXAMPLE
bun --cwd apps/database sb:validate:isolated --cleanup /tmp/tuturuuu-supabase-EXAMPLE
```
The exact temporary parent varies by operating system. Use the disposable path
printed when the run starts; cleanup refuses paths outside the owned temporary
directory pattern. If scoped Docker cleanup fails, the metadata directory is
kept so the command can be retried safely.
### Generating TypeScript Types
After schema changes, regenerate the TypeScript types to keep your code in sync with the database schema:
```bash theme={null}
bun sb:typegen
```
The generator includes the `public`, `private`, and `storage` schemas. Keep
server-owned/private tables out of Supabase REST exposure, but still regenerate
types after migrations so server-only helpers and storage paths can import
accurate generated row shapes.
Alternatively, you can use the shorthand:
```bash theme={null}
bun typegen
```
This step is automatically performed when running `bun sb:reset`, making it
useful when catching up with a new branch's schema.
### Using Generated TypeScript Types
The Supabase-generated TypeScript types are available at `packages/types/src/supabase.ts`. These types are accessible to all apps that have the `@tuturuuu/types` package installed.
You can use these types to ensure type safety when working with Supabase data:
With supabase-js v2, table types flow from the client itself rather than a
generic on `.from()`. The Tuturuuu Supabase helpers in `@tuturuuu/supabase` are
already typed with the generated `Database` type, so queries are type-safe out of
the box. Reach for the `Database` type directly when you need a specific row
shape:
```typescript theme={null}
import type { Database } from '@tuturuuu/types/supabase';
type WorkspaceMemberRow =
Database['public']['Tables']['workspace_members']['Row'];
// Type-safe access to tables (the client already knows the row shapes)
const { data, error } = await supabase
.from('workspace_members')
.select('*')
.eq('ws_id', workspaceId);
// data is typed as WorkspaceMemberRow[] | null
// Type-safe access to specific columns
const { data: workspace } = await supabase
.from('workspaces')
.select('id, name, handle')
.eq('id', workspaceId)
.single();
// TypeScript knows the structure of 'workspace' with proper types
const workspaceName: string | undefined = workspace?.name;
```
### Short-hand Type Access
For more convenient access to common table types, Tuturuuu also provides short-hand type definitions in `packages/types/src/db.ts`. These are easier to use and remember than the full database type paths:
```typescript theme={null}
import type { WorkspaceCourse, WorkspaceRole } from '@tuturuuu/types/db';
// Use short-hand types directly
const { data: roles } = await supabase
.from('workspace_roles')
.select('*')
.eq('ws_id', workspaceId);
// Type is now WorkspaceRole[]
roles?.forEach((role: WorkspaceRole) => {
console.log(role.name, role.permissions);
});
// Short-hand types can also include extended properties
const course: WorkspaceCourse = {
id: 'course-id',
ws_id: 'workspace-id',
name: 'Course Name',
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
href: '/courses/course-id', // Extended property not in the database
};
```
You can add your own short-hand types to `db.ts` for tables you frequently work with. This is especially useful for tables that have complex structures or need additional client-side properties.
This ensures that your code correctly interacts with the database schema, reducing runtime errors and improving development experience.
## Migration Files
All migration files are stored in `apps/database/supabase/migrations`. These files:
* Contain SQL commands that create and modify the database schema
* Are executed in order based on the timestamp prefix in their filenames
* Include descriptive names after the timestamp to help developers understand their purpose
When contributing new migrations in a Pull Request, always add them after the
latest migration file from the `main` branch. This maintains the correct
execution order and prevents issues when syncing the production database.
## Local Authentication
A local mail server (**InBucket**) is automatically set up by Supabase to handle authentication emails. You can access it at [http://localhost:8004](http://localhost:8004).
With InBucket, you can:
* Receive all authentication emails sent by your local Supabase instance
* Test any email combination without needing actual mail delivery
* View password reset links, confirmation emails, and other authentication flows
* Troubleshoot email templates and content
This makes it easy to test different authentication scenarios without configuring a real email service or waiting for actual email delivery.
Five seed accounts are pre-configured for local development:
1. [local@tuturuuu.com](mailto:local@tuturuuu.com)
2. [user1@tuturuuu.com](mailto:user1@tuturuuu.com)
3. [user2@tuturuuu.com](mailto:user2@tuturuuu.com)
4. [user3@tuturuuu.com](mailto:user3@tuturuuu.com)
5. [user4@tuturuuu.com](mailto:user4@tuturuuu.com)
These accounts are already set up with the necessary data, allowing you to quickly test the app's functionality. However, you can register any new email address and the authentication emails will be captured by InBucket for you to inspect.
OTP send limits include email-scoped cooldowns in the web process. Playwright
tests that only need to assert the OTP stage should use a dedicated throwaway
email address instead of a seed login account. Reserve seed accounts for tests
that must complete authentication.
### Playwright E2E Safety
Web E2E runs must use the local Supabase stack. For local debugging and patch
iteration, run the web app and Playwright on the native machine so code changes
are picked up quickly and server logs stay visible.
Avoid `bun test:e2e`, `bun --cwd apps/web test:e2e`, and
`bun test:e2e:web:docker` as the first local debugging path. Those commands
route through `scripts/run-web-e2e-docker.js`, which writes
`tmp/e2e/web.env`, resets local Supabase, builds and runs the production-style
Docker web stack, and then runs Playwright. Reserve that Dockerized path for
explicit CI-parity checks or production-Docker runtime bugs.
For the native workflow, start or reset local Supabase with the database package
scripts, start `apps/web` natively with the local E2E environment, and run the
focused Playwright spec directly from `apps/web`. Native web server Supabase
URLs should resolve to `http://127.0.0.1:8001`; `host.docker.internal` is only
for the Dockerized web stack.
Do not point E2E at a cloud Supabase project. The Playwright global setup fails
fast if `BASE_URL` is not local or if `NEXT_PUBLIC_SUPABASE_URL`,
`SUPABASE_SERVER_URL`, or `SUPABASE_URL` resolves outside the local Supabase
origins on port `8001`.
## Further Information
For more details about Supabase CLI usage, refer to the [Supabase CLI documentation](https://supabase.com/docs/guides/local-development/cli/getting-started).
## Row Level Security (RLS)
Row Level Security (RLS) is a powerful Postgres feature that allows you to control access to rows in a database table based on the user making the request. In Tuturuuu, we use RLS extensively to ensure data security.
### Enabling RLS
RLS should be enabled on all tables in exposed schemas (like `public`). When creating tables through the Supabase UI, RLS is enabled by default. For tables created using SQL, you need to explicitly enable RLS:
```sql theme={null}
alter table . enable row level security;
```
### Creating RLS Policies
Policies define the conditions under which users can access or modify data. Here are some common patterns used in Tuturuuu:
#### Organization-based Access
In Tuturuuu, workspace-scoped resource tables use `ws_id`, and membership rows
use `workspace_members.ws_id` plus the `workspace_member_type` enum. Use
`MEMBER` when an example requires a full member rather than a guest. For
application-specific mutations, pair membership with the maintained permission
helper and confirm the permission name in the generated enum. The canonical
[RLS policy reference](/reference/database/rls-policies) explains when a
different membership or permission rule is appropriate.
```sql theme={null}
-- Allow users to select data from their organizations
create policy "Users can view data from their organizations"
on public.table_name
for select
to authenticated
using (
exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = table_name.ws_id
and workspace_member.user_id = (select auth.uid())
and workspace_member.type = 'MEMBER'
)
);
-- Allow members with manage_projects to insert data
create policy "Project managers can insert data"
on public.table_name
for insert
to authenticated
with check (
exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = table_name.ws_id
and workspace_member.user_id = (select auth.uid())
and workspace_member.type = 'MEMBER'
)
and public.has_workspace_permission(
table_name.ws_id,
(select auth.uid()),
'manage_projects'
)
);
-- UPDATE needs USING for the existing row and WITH CHECK for the new row
create policy "Project managers can update data"
on public.table_name
for update
to authenticated
using (
exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = table_name.ws_id
and workspace_member.user_id = (select auth.uid())
and workspace_member.type = 'MEMBER'
)
and public.has_workspace_permission(
table_name.ws_id,
(select auth.uid()),
'manage_projects'
)
)
with check (
exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = table_name.ws_id
and workspace_member.user_id = (select auth.uid())
and workspace_member.type = 'MEMBER'
)
and public.has_workspace_permission(
table_name.ws_id,
(select auth.uid()),
'manage_projects'
)
);
-- DELETE uses USING because there is no new row to check
create policy "Project managers can delete data"
on public.table_name
for delete
to authenticated
using (
exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = table_name.ws_id
and workspace_member.user_id = (select auth.uid())
and workspace_member.type = 'MEMBER'
)
and public.has_workspace_permission(
table_name.ws_id,
(select auth.uid()),
'manage_projects'
)
);
```
#### Role-based Access
Use `public.has_workspace_permission(p_ws_id, p_user_id, p_permission)` rather
than joining retired member-role columns. Permission checks should still be
paired with membership when a workspace-wide default permission must not admit
non-members.
If a policy needs a recursion-free membership lookup, keep the helper in a
non-exposed schema, derive identity from the request, and harden its execution
boundary before referencing it from a policy:
```sql theme={null}
create or replace function private.can_access_workspace(p_ws_id uuid)
returns boolean
language plpgsql
security definer
stable
set search_path = ''
as $$
declare
v_caller_id uuid := (select auth.uid());
begin
if coalesce((select auth.jwt() ->> 'role'), '') = 'service_role' then
return true;
end if;
if v_caller_id is null then
return false;
end if;
return exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = p_ws_id
and workspace_member.user_id = v_caller_id
and workspace_member.type = 'MEMBER'
);
end;
$$;
revoke execute on function private.can_access_workspace(uuid)
from public, anon, authenticated, service_role;
grant execute on function private.can_access_workspace(uuid)
to authenticated, service_role;
```
Because `private` is not a Data API schema, this narrowly granted helper cannot
be invoked as an exposed RPC. A public Data API RPC needs its own internal
workspace permission check; a page or route check is not enough. Revoke
`public` and `anon`, grant only the required caller role, and cover the grant
matrix and allowed/denied calls with pgTAP. See
[Workspace-Scoped Security Definer RPCs](/reference/database/rls-policies#pattern-9-workspace-scoped-security-definer-rpcs).
```sql theme={null}
-- Allow full members with manage_projects to access a resource
create policy "Project managers can view workspace data"
on public.table_name
for select
to authenticated
using (
exists (
select 1
from public.workspace_members workspace_member
where workspace_member.ws_id = table_name.ws_id
and workspace_member.user_id = (select auth.uid())
and workspace_member.type = 'MEMBER'
)
and public.has_workspace_permission(
table_name.ws_id,
(select auth.uid()),
'manage_projects'
)
);
```
### Performance Optimization for RLS
For better performance in your RLS policies:
1. **Wrap function calls in subqueries**:
```sql theme={null}
-- Instead of this
using (auth.uid() = user_id);
-- Use this
using ((select auth.uid()) = user_id);
```
2. **Use hardened security-definer helpers only when needed**. The
`private.can_access_workspace` recipe above shows the required caller,
service-role, search-path, and grant controls. Wrap stable helper calls in a
scalar subquery so Postgres can evaluate them once per statement:
```sql theme={null}
create policy "Users can access workspace data"
on public.table_name
for select
to authenticated
using ((select private.can_access_workspace(table_name.ws_id)));
```
3. **Add explicit filters** in your queries even when you have RLS:
```typescript theme={null}
// Even though RLS will filter by workspace_id, adding the filter explicitly improves performance
const { data } = await supabase
.from('projects')
.select()
.eq('workspace_id', workspaceId);
```
### Testing RLS Policies
Tuturuuu ships a large suite of database tests in
`apps/database/supabase/tests/`. These are [pgTAP](https://pgtap.org/) test files
(each one wraps its assertions in a transaction and rolls back), and they run
against the local Supabase instance through the Supabase CLI.
To run the database test suite locally:
1. Make sure a local Supabase instance is running (`bun sb:start`).
2. Add or edit a `*.sql` test file in `apps/database/supabase/tests/`.
3. Run the tests through the database workspace's Supabase CLI wrapper:
```bash theme={null}
# From the repo root
bun --cwd apps/database scripts/run-supabase.js test db
```
`run-supabase.js` resolves the pinned local Supabase binary and invokes
`supabase test db`, which executes every `tests/*.sql` file against the local
database. There is no dedicated `bun sb:test` alias — the test runner is the
Supabase CLI itself.
Each test file begins by enabling pgTAP, then uses helpers like `plan()`,
`is()`, and `throws_ok()` to assert behavior. A trimmed example:
```sql theme={null}
begin;
create extension if not exists pgtap with schema extensions;
set local search_path = public, extensions;
select plan(2);
-- Impersonate an authenticated user
set local role authenticated;
select set_config(
'request.jwt.claims',
'{"sub":"00000000-0000-0000-0000-000000000001","role":"authenticated"}',
true
);
-- Should return data the user is allowed to see
select isnt_empty(
$$ select 1 from public.workspaces where id = '00000000-0000-0000-0000-000000000000' $$,
'User can see a workspace they belong to'
);
-- Should be blocked by RLS for an unrelated workspace
select is_empty(
$$ select 1 from public.workspaces where id = '11111111-1111-1111-1111-111111111111' $$,
'User cannot see a workspace they do not belong to'
);
select * from finish();
rollback;
```
Look through the existing files in `apps/database/supabase/tests/` (for
example `user-profile-rls.sql` and the `private-schema-*` suites) to follow the
established pgTAP patterns for impersonating roles and asserting RLS behavior.
## Database Triggers
Triggers in Postgres allow you to automatically execute a function when a specified database event occurs (INSERT, UPDATE, DELETE). In Tuturuuu, we use triggers for various purposes like:
* Maintaining audit logs
* Syncing data between tables
* Enforcing complex business rules
### Creating Triggers
Here's how to create a trigger in your Tuturuuu development workflow:
1. First, create a trigger function:
```sql theme={null}
create or replace function public.handle_new_user()
returns trigger
language plpgsql
security definer
set search_path = ''
as $$
declare
v_workspace_id uuid := gen_random_uuid();
begin
-- Create a personal workspace for the new user
insert into public.workspaces (id, name, personal, creator_id)
values (v_workspace_id, new.email || '''s workspace', true, new.id);
-- Record the creator as a full member without re-querying the workspace
insert into public.workspace_members (ws_id, user_id, type)
values (v_workspace_id, new.id, 'MEMBER');
return new;
end;
$$;
revoke execute on function public.handle_new_user()
from public, anon, authenticated, service_role;
grant execute on function public.handle_new_user() to supabase_auth_admin;
```
2. Then, create the trigger:
```sql theme={null}
create trigger on_auth_user_created
after insert on auth.users
for each row
execute function public.handle_new_user();
```
### Common Triggers in Tuturuuu
#### Audit Logging
```sql theme={null}
create or replace function private.audit_log_changes()
returns trigger
language plpgsql
security definer
as $$
begin
insert into public.audit_logs (
table_name,
record_id,
action,
old_data,
new_data,
performed_by
)
values (
TG_TABLE_NAME,
coalesce(new.id, old.id),
TG_OP,
case when TG_OP = 'DELETE' or TG_OP = 'UPDATE' then row_to_json(old) else null end,
case when TG_OP = 'INSERT' or TG_OP = 'UPDATE' then row_to_json(new) else null end,
coalesce(auth.uid(), '00000000-0000-0000-0000-000000000000'::uuid)
);
return coalesce(new, old);
end;
$$;
-- Apply this trigger to a table
create trigger projects_audit_trigger
after insert or update or delete
on public.projects
for each row
execute function private.audit_log_changes();
```
#### Automated Timestamps
```sql theme={null}
create or replace function public.update_timestamp()
returns trigger
language plpgsql
as $$
begin
new.updated_at = now();
return new;
end;
$$;
-- Apply to a table
create trigger update_projects_timestamp
before update
on public.projects
for each row
execute function public.update_timestamp();
```
### Testing Triggers
You can test triggers by running SQL commands in the local Supabase instance and verifying the results:
```sql theme={null}
-- Insert a test user
insert into auth.users (id, email)
values ('test-uuid', 'test@example.com');
-- Verify the trigger created a workspace
select * from public.workspaces where creator_id = 'test-uuid';
-- Verify the creator is a full member of the workspace
select * from public.workspace_members
where user_id = 'test-uuid' and type = 'MEMBER';
```
## Seeding Your Database
Database seeding is the process of populating your database with initial data. In Tuturuuu, we use seeding to:
1. Create test users and workspaces for local development
2. Initialize lookup tables with standard values
3. Ensure a consistent starting point for all developers
### Seed Files Location
In Tuturuuu, seed files are stored in `apps/database/supabase/seed.sql`. This file is automatically executed when you run `bun sb:reset` or start a fresh Supabase instance.
### Real Examples from Tuturuuu's Seed File
Let's look at some real examples from Tuturuuu's seed.sql file:
#### 1. Authentication Users
The seed file creates five default test users with pre-set passwords:
```sql theme={null}
-- Populate auth users
INSERT INTO
"auth"."users" (
"instance_id",
"id",
"aud",
"role",
"email",
"encrypted_password",
"email_confirmed_at",
/* other fields... */
)
VALUES
(
'00000000-0000-0000-0000-000000000000',
'00000000-0000-0000-0000-000000000001',
'authenticated',
'authenticated',
'local@tuturuuu.com',
crypt('password123', gen_salt('bf')),
'2023-02-18 23:31:13.017218+00',
/* other values... */
),
/* additional users... */
```
All seed users have the same password: `password123`, making it easy to log in for testing.
#### 2. Workspaces
The seed creates several workspaces for testing different scenarios:
```sql theme={null}
-- Populate workspaces
insert into
public.workspaces (id, name, handle, creator_id)
values
(
'00000000-0000-0000-0000-000000000000',
'Tuturuuu',
'tuturuuu',
'00000000-0000-0000-0000-000000000001'
),
(
'00000000-0000-0000-0000-000000000001',
'Prototype All',
'prototype-all',
null
),
/* additional workspaces... */
```
#### 3. Workspace Members and Roles
The seed also sets up relationships between users and workspaces with different roles:
```sql theme={null}
-- Populate workspace_members
insert into
public.workspace_members (user_id, ws_id, role)
values
(
'00000000-0000-0000-0000-000000000001',
'00000000-0000-0000-0000-000000000000',
'OWNER'
),
(
'00000000-0000-0000-0000-000000000002',
'00000000-0000-0000-0000-000000000000',
'ADMIN'
),
(
'00000000-0000-0000-0000-000000000003',
'00000000-0000-0000-0000-000000000000',
'MEMBER'
),
/* additional members... */
```
#### 4. Workspace Features Configuration
The seed file configures workspace features using secrets:
```sql theme={null}
-- Populate workspace_secrets
insert into
public.workspace_secrets (ws_id, name, value)
values
(
'00000000-0000-0000-0000-000000000000',
'ENABLE_CHAT',
'true'
),
(
'00000000-0000-0000-0000-000000000000',
'ENABLE_EDUCATION',
'true'
),
/* additional features... */
```
#### 5. Domain-specific Data
The seed includes domain-specific data for different workspace types. For example, user group metric data:
```sql theme={null}
-- Populate user group metrics
insert into
public.user_group_metrics (id, ws_id, name, unit, is_weighted)
values
(
'00000000-0000-0000-0000-000000000001',
'00000000-0000-0000-0000-000000000003',
'Nhiệt độ',
'°C',
true
),
(
'00000000-0000-0000-0000-000000000002',
'00000000-0000-0000-0000-000000000003',
'Chiều cao',
'cm',
false
),
/* additional metrics... */
```
`user_group_metrics.is_weighted = false` keeps a metric visible for entry and category views without counting it toward report totals or averages. Metric categories live in `user_group_metric_categories` and are linked through `user_group_metric_category_links`.
When renaming or recreating these metric tables, keep authenticated CRUD grants and service-role grants in the same migration as the table change. The group indicators dashboard reads the renamed tables through protected API/server paths after workspace permission checks, and missing grants can surface as `permission denied for table user_group_metric_categories`.
When code uses `createAdminClient()` for these protected reads, name the local client `sbAdmin`. Do not name a service-role client `supabase`; reserve that name for request-scoped or user-scoped clients so route authorization is easy to audit.
### Creating Seed Data
Here's how to create and modify seed data:
1. Edit the `apps/database/supabase/seed.sql` file
2. Add SQL statements to insert your data
3. Run `bun sb:reset` to apply the seed data
Example seed data format:
```sql theme={null}
-- Create a new user group
INSERT INTO public.workspace_user_groups (id, name, ws_id)
VALUES
('your-uuid-here', 'New Group', 'workspace-uuid-here')
ON CONFLICT (id) DO NOTHING;
-- Add a new workspace feature
INSERT INTO public.workspace_secrets (ws_id, name, value)
VALUES
('workspace-uuid-here', 'ENABLE_NEW_FEATURE', 'true')
ON CONFLICT (ws_id, name) DO UPDATE
SET value = EXCLUDED.value;
```
### Creating a Custom Seed File
Sometimes you might want to create a custom seed file for specific testing scenarios:
1. Create a new SQL file in the `apps/database/supabase` directory
2. Add your custom seed data
3. Run it with the Supabase CLI:
```bash theme={null}
bun supabase db reset --db-url=postgresql://postgres:postgres@localhost:54322/postgres
psql postgresql://postgres:postgres@localhost:54322/postgres -f apps/database/supabase/my_custom_seed.sql
```
### Exporting Current Data as Seed
You can also export your current database data to use as seed data:
```bash theme={null}
# Export only data (not schema) to seed.sql
bun supabase db dump --db-url=postgresql://postgres:postgres@localhost:54322/postgres --data-only > apps/database/supabase/new_seed.sql
```
This is helpful when you've set up data manually and want to preserve it for future development environments.
### Recommended Seeding Workflow
For Tuturuuu development, we recommend:
1. Start with a fresh database: `bun sb:reset`
2. Make changes through the UI or your app
3. When you're satisfied, export the data: `bun supabase db dump --data-only > apps/database/supabase/new_seed.sql`
4. Edit the generated SQL to keep only what you need
5. Update the main `seed.sql` file with your changes
6. Test by running `bun sb:reset` again
## AI Integration with Vercel AI SDK
Tuturuuu uses Vercel's AI SDK for its AI features, utilizing structured data generation capabilities that integrate with Supabase. This section covers how to work with AI features in the development workflow.
### Overview of AI SDK in Tuturuuu
The AI SDK standardizes integrating various AI models across supported providers into Tuturuuu applications. It enables structured data generation, tool calling, and streaming responses to create rich AI-powered features.
The main libraries used are:
* `ai` - Core Vercel AI SDK package
* `@ai-sdk/google` - Provider-specific integration for Google models
* `@tuturuuu/supabase` - Supabase client with Tuturuuu-specific utilities
### Generating Structured Data
Tuturuuu uses the AI SDK's structured data generation capabilities to create typed responses from AI models. This approach ensures type safety and consistent data structures for features like:
* Flashcards generation
* Quiz generation
* Learning plans
* Task management
#### Example: Flashcard Generation
The structured data pattern used in Tuturuuu follows this workflow:
1. Define a schema using Zod
2. Connect to Supabase for authentication and workspace validation
3. Generate structured data using the AI SDK
4. Stream the response to the client
Here's an example from Tuturuuu's codebase:
```typescript theme={null}
// 1. Define the schema
import { z } from 'zod';
export const flashcardSchema = z.object({
flashcards: z.array(
z.object({
front: z.string().describe('Question. Do not use emojis or links.'),
back: z.string().describe('Answer. Do not use emojis or links.'),
})
),
});
// 2. Setup API endpoint
export async function POST(req: Request) {
const sbAdmin = await createAdminClient();
const { wsId, context } = await req.json();
// Validate user and workspace permissions
const {
data: { user },
} = await supabase.auth.getUser();
if (!user) return new Response('Unauthorized', { status: 401 });
// Check workspace feature flag
const { count, error } = await sbAdmin
.from('workspace_secrets')
.select('*', { count: 'exact', head: true })
.eq('ws_id', wsId)
.eq('name', 'ENABLE_CHAT')
.eq('value', 'true');
if (error) return new Response(error.message, { status: 500 });
if (count === 0)
return new Response('You are not allowed to use this feature.', {
status: 401,
});
// 3. Generate structured data using AI SDK
const result = streamObject({
model: google('gemini-2.0-flash-001', {
safetySettings: [
// Safety settings configuration...
],
}),
prompt: `Generate 10 flashcards with the following context: ${context}`,
schema: flashcardSchema,
});
// 4. Stream the response to the client
return result.toTextStreamResponse();
}
```
### Available Models
Tuturuuu supports multiple AI models through Vercel AI SDK. You can define which models are available in your application by updating the `models.ts` file in the `packages/ai` directory:
```typescript theme={null}
export const models = [
{
value: 'gemini-2.0-flash-001',
label: 'gemini-2.0-flash',
provider: 'Google',
description: 'Gemini 2.0 Flash delivers next-gen features...',
context: 1000000,
},
// Add other models here...
];
export const defaultModel = models.find(
(model) =>
model.value === 'gemini-2.0-flash-001' && model.provider === 'Google Vertex'
);
```
### Creating Custom Schema Types
To create new structured data types for AI generation, add your schema definition to the `packages/ai/src/object/types.ts` file:
```typescript theme={null}
export const myNewSchema = z.object({
items: z.array(
z.object({
name: z.string().describe('Name of the item'),
description: z.string().describe('Description of the item'),
priority: z.enum(['high', 'medium', 'low']).describe('Priority level'),
})
),
});
```
### Integration with Supabase
Tuturuuu's AI features leverage Supabase for:
1. **Authentication** - Validating users before making AI requests
2. **Authorization** - Checking workspace permissions via `workspace_secrets`
3. **Feature Flags** - Using `workspace_secrets` to enable/disable AI features per workspace
4. **Storage** - Storing AI-generated content for later use
To enable AI features for a workspace, ensure the appropriate flags are set in the `workspace_secrets` table:
```sql theme={null}
-- Enable AI chat features for a workspace
INSERT INTO public.workspace_secrets (ws_id, name, value)
VALUES ('your-workspace-id', 'ENABLE_CHAT', 'true');
-- Enable AI document features for a workspace
INSERT INTO public.workspace_secrets (ws_id, name, value)
VALUES ('your-workspace-id', 'ENABLE_DOCS', 'true');
```
### Testing AI Features Locally
When testing AI features in your local environment:
1. Ensure you have the required API keys set in your `.env.local` file:
```
GOOGLE_GENERATIVE_AI_API_KEY=your-api-key
```
2. Verify the workspace has the necessary feature flags enabled in your local database
```sql theme={null}
SELECT * FROM workspace_secrets WHERE ws_id = 'your-workspace-id' AND name = 'ENABLE_CHAT';
```
3. Use the AI-enabled accounts from the seed data (`local@tuturuuu.com`) as they often have additional permissions
### Error Handling
When integrating AI features, implement proper error handling to account for:
1. Missing API keys
2. Model unavailability
3. Invalid user input
4. Exceeded token limits
Example error handling pattern used in Tuturuuu:
```typescript theme={null}
try {
// AI SDK code here
} catch (error) {
console.log(error);
return NextResponse.json(
{
message: `## Edge API Failure\nCould not complete the request. Please view the **Stack trace** below.\n\`\`\`bash\n${(error as Error)?.stack || 'No stack trace available'}`,
},
{
status: 200,
}
);
}
```
# Monorepo Architecture
Source: https://docs.tuturuuu.com/build/development-tools/monorepo-architecture
Understanding Tuturuuu monorepo architecture and the benefits of using Turborepo.
**Prerequisite**: You should have followed the
[Development](/build/development-tools/development) setup guide to understand
the basic structure of the codebase.
## What is a Monorepo?
A monorepo is a single repository containing multiple distinct projects with well-defined relationships. This approach differs from a polyrepo strategy where each project has its own separate repository.
> "A monorepo is a single repository containing multiple distinct projects, with well-defined relationships."
> — [monorepo.tools](https://monorepo.tools/)
It's important to note that a monorepo is not the same as a monolith. A monolith is a single, tightly coupled application, while a monorepo can contain many independent applications, libraries, and tools that can be deployed separately.
## Why Tuturuuu Uses a Monorepo
Tuturuuu's platform is built as a monorepo (`@tutur3u/platform`) for several key reasons:
### 1. Code Sharing Without Overhead
Our monorepo structure makes it easy to share code across different applications and services without the overhead of publishing packages. Core components, utilities, and business logic can be shared across multiple applications without duplicating code.
```
platform/
├── apps/
│ ├── web/ # Main web application
│ ├── tanstack-web/ # TanStack Start migration frontend
│ ├── backend/ # Rust API runtime for migrated web APIs
│ ├── cms/ # Dedicated Tuturuuu CMS satellite app
│ ├── docs/ # Documentation site (Mintlify)
│ ├── rewise/ # Rewise application
│ ├── nova/ # Nova application
│ ├── calendar/ # Standalone calendar application
│ └── ... # Other applications
└── packages/
├── ai/ # Shared AI functions & configurations
├── ui/ # Shared UI components
├── types/ # Shared TypeScript types
├── utils/ # Shared utilities
├── supabase/ # Supabase client and utilities
└── ... # Other shared packages
```
### 2. Atomic Changes Across Projects
When making changes that affect multiple parts of the platform, we can make those changes in a single commit. This ensures that all parts of the system remain compatible with each other, reducing integration issues.
For example, when updating a database schema:
1. We can update the Supabase schema in `apps/database/supabase/migrations`
2. Update the TypeScript types in `packages/types/src/supabase.ts`
3. Update all affected applications in the same pull request
### 3. Consistent Developer Experience
A monorepo allows us to standardize tooling, testing, and deployment processes across all projects. This creates a consistent development experience regardless of which part of the platform a developer is working on.
All developers use the same:
* Package manager (bun)
* Build system (Turborepo)
* Code style guidelines (enforced by Biome)
* Testing framework
* CI/CD pipelines
### 4. Simplified Dependency Management
Managing dependencies in a monorepo is simpler because we can ensure all projects use the same versions of shared dependencies, avoiding version conflicts.
## Why We Chose Turborepo
Tuturuuu uses [Turborepo](https://turborepo.org/) as its build system for several reasons:
### 1. Incremental Builds with Caching
Turborepo provides intelligent caching of build artifacts. This means that if a file hasn't changed, Turborepo will use the cached result instead of rebuilding it, significantly reducing build times.
```bash theme={null}
# Only builds what changed since the last build
bun run build
```
### 2. Parallel Task Execution
Turborepo automatically parallelizes tasks, maximizing the use of available CPU cores.
```bash theme={null}
# Runs lint across all packages in parallel
bun lint
```
### 3. Task Orchestration
Turborepo understands dependencies between packages and ensures tasks are run in the correct order.
```json theme={null}
// In turbo.json (Turbo 2.x uses "tasks", not the older "pipeline")
{
"tasks": {
"build": {
"dependsOn": ["^build"],
"outputs": ["dist/**", ".next/**", "!.next/cache/**", "!.next/dev/**"]
}
}
}
```
Keep local Next development state out of Turbo build outputs. The `.next/dev`
tree belongs to `next dev`; archiving it can capture multi-GB Turbopack caches
and make `bun dev:web` slower.
When local `apps/web` compilation feels slow, run the dev-speed diagnostic
before changing code:
```bash theme={null}
bun diagnose:dev:web
```
It reports generated cache sizes, Next slow-filesystem warnings, and the
slowest `.next/dev/trace` spans from the previous `next dev` session. It also
flags watcher `EMFILE` errors; the web dev launcher raises the child process
open-file limit to `65536` by default, and unusual local shells can override
that with `TUTURUUU_DEV_MAX_OPEN_FILES`. Native local dev leaves
`WATCHPACK_POLLING` unset by default for lower watcher overhead; set
`WATCHPACK_POLLING=true` only when a local filesystem or container workflow
needs polling to keep route discovery reliable.
### 4. Remote Caching
For CI/CD pipelines, Turborepo supports remote caching, which means build artifacts can be shared across different environments.
## Monorepo Structure
Our monorepo is organized into several main directories:
### apps/
Contains all deployable applications. Each application is a standalone project that can be deployed independently.
```
apps/
├── web/ # Main web application and central API host
├── tanstack-web/ # TanStack Start frontend replacing apps/web
├── backend/ # Dedicated Rust API runtime for migrated web APIs
├── cms/ # Dedicated Tuturuuu CMS frontend
├── docs/ # Documentation site (Mintlify)
├── rewise/ # Rewise application
├── nova/ # Nova application
└── database/ # Database migrations and scripts
```
`apps/tanstack-web` and `apps/backend` are the migration pair for replacing the
legacy `apps/web` runtime. The TanStack app owns migrated frontend routes and
Start server functions; the Rust backend owns migrated API, job, cron, and
private/admin call paths. Both apps can run in Docker for local and blue/green
rehearsals, and both have Cloudflare Worker preview entrypoints:
`apps/tanstack-web/wrangler.jsonc` and `apps/backend/wrangler.jsonc`.
Cloudflare preview is incremental, not the production cutover. Bind
`BACKEND_INTERNAL_TOKEN`, `BACKEND_PUBLIC_ORIGIN`, and
`BACKEND_INTERNAL_URL` through Wrangler or Cloudflare secrets, run
`bun check:cloudflare` before preview deployment, and keep `apps/web` serving
production until the TanStack/Rust route manifest, Docker E2E compare report,
benchmark report, and migration gates pass.
### packages/
Contains shared libraries and utilities that are used by multiple applications.
```
packages/
├── ui/ # Shared UI components
├── types/ # Shared TypeScript types including Supabase types
├── utils/ # Shared utilities
├── ai/ # AI-related utilities
└── supabase/ # Supabase client and utilities
```
## Development Workflow
When working in our monorepo, you'll typically:
1. Clone the repository: `git clone https://github.com/tutur3u/platform.git`
2. Install dependencies: `bun install`
3. Start the development servers: `bun dev` or `bun devx` (with Supabase)
Satellite apps can also be started with app-specific scripts when you only need one frontend plus the shared `web` API host. For example, `bun dev:cms` runs the CMS app together with the shared packages and `apps/web`.
For more detailed instructions on development workflow, see the [Development](/build/development-tools/development) and [Local Supabase Development](/build/development-tools/local-supabase-development) guides.
## Best Practices
When working in our monorepo, follow these best practices:
### 1. Keep Projects Modular
Even though all code lives in one repository, maintain clear boundaries between projects. Each package or app should have a well-defined purpose and API.
### 2. Use Workspace References
When one project depends on another within the monorepo, use workspace references:
```json theme={null}
{
"dependencies": {
"@tuturuuu/ui": "workspace:*",
"@tuturuuu/types": "workspace:*"
}
}
```
### 3. Think About Build Order
Be mindful of dependencies between packages. If package B depends on package A, ensure that changes to package A trigger rebuilds of package B.
### 4. Use Correct Scope for Changes
* For changes that affect a single application, focus your changes there
* For changes that affect multiple applications, consider extracting common code to a shared package
* For global changes (like tooling updates), ensure you test across all affected projects
## Common Challenges and Solutions
### Challenge: Long Build Times
**Solution:** Turborepo's caching mechanism helps, but also:
* Be selective about what you build during development
* Use `bun --filter` to focus on specific packages
```bash theme={null}
# Only build the web app and its dependencies
bun --filter web... build
```
### Challenge: Dependency Management
**Solution:**
* Regularly update dependencies
* Bun automatically detects workspace packages from the root `package.json` workspaces field
* Consider using `bun dedupe` to eliminate duplicate dependencies
### Challenge: CI/CD Pipeline Complexity
**Solution:**
* Use GitHub Actions workflows that are specific to the affected parts of the codebase
* Leverage Turborepo's `--filter` and `--since` flags to only build what's changed
## Further Resources
* [monorepo.tools](https://monorepo.tools/): Comprehensive information about monorepos
* [Turborepo Documentation](https://turbo.build/repo/docs): Official Turborepo documentation
* [Bun Workspace](https://bun.sh/docs/install/workspaces): How bun handles monorepos
# Development Tools
Source: https://docs.tuturuuu.com/build/development-tools/overview
Tools and guides for developing on the Tuturuuu platform
This section contains tools, guides, and best practices for developing on the Tuturuuu platform.
## Getting Started
* [Development Setup](/build/development-tools/development) - Set up your development environment
* [Monorepo Architecture](/build/development-tools/monorepo-architecture) - Understand the project structure
* [Git Conventions](/build/development-tools/git-conventions) - Code contribution guidelines
## Development Workflow
* [Local Supabase Development](/build/development-tools/local-supabase-development) - Database development setup
* [CI/CD Pipelines](/build/development-tools/ci-cd-pipelines) - Continuous integration and deployment
* [DevOps & Deployment](/build/devops/overview) - Operational runbooks for environments, workflows, Docker, and secrets
* [Cleaning Clone](/build/development-tools/cleaning-clone) - Reset your local environment
## Documentation
* [Documenting](/build/development-tools/documenting) - How to write and maintain documentation
## Development Guidelines
### Code Quality
* Formatting and linting are handled by [Biome](https://biomejs.dev/) (`biome.json`),
not ESLint or Prettier. Use `bun ff` to format, lint, and check in one pass.
* Run `bun check` before opening a PR — it is the repo's verification gate for
TypeScript, JavaScript, root-script, and config changes (run focused tests first).
* Default to Server Components in `apps/web`; add `'use client'` only for state,
browser APIs, or interactivity.
* Prefer typed expected errors and dependency services via `@tuturuuu/utils/effect`
for new server/service orchestration.
### Git Workflow
* Use [Conventional Commits](/build/development-tools/git-conventions) for commit
messages and branch names.
* The branch checker only accepts these prefixes: `feature/`, `feat/`, `fix/`,
`bugfix/`, `hotfix/`, `release/`, `chore/`, `docs/`, `style/`, `refactor/`,
`perf/`, `dependabot/`, and `claude/`.
* Keep commits atomic and write descriptive PR descriptions.
* Do not manually bump `TUTURUUU_PLATFORM_VERSION`, package versions, or
changelogs — Release Please owns version updates.
### Testing
* Tests run with Vitest. Use `bun test` for the full suite or filter to a single
workspace (for example `bun --filter @tuturuuu/web test`).
* Write unit tests for utilities and cover error scenarios.
* Run focused tests before the full `bun check` gate.
### Performance
* Default to server-side rendering and Server Components where possible.
* Use TanStack Query for client-side fetching and caching; avoid `useEffect` for
data fetching.
* Minimize bundle size and profile performance regularly.
## Tools and Scripts
### Package Management
Dependencies are always installed from inside the owning workspace — never by
hand-editing the root `package.json` and never with a `--workspace` flag.
```bash theme={null}
# Install all dependencies
bun install
# Add a dependency to an app (e.g. apps/web)
cd apps/web && bun add
# Add a dependency to a shared package (e.g. packages/ui)
cd packages/ui && bun add
# Update dependencies across the monorepo
bun update-all
```
### Development Commands
```bash theme={null}
# Start all apps
bun dev
# Start a specific app
bun dev:web
# Run tests (Vitest)
bun test
# Run the full verification gate before a PR
bun check
```
### Database Commands
```bash theme={null}
# Start the local Supabase stack
bun sb:start
# Create a new migration
bun sb:new
# Apply pending migrations locally
bun sb:up
# Generate TypeScript types from the local schema
bun sb:typegen
# Reset the local database (re-runs all migrations + seeds)
bun sb:reset
```
`bun sb:push` and `bun sb:linkpush` push migrations to the **remote**
Supabase project and are reserved for the maintainer applying production
changes. Do not run them — prepare migrations locally with `bun sb:up` and let
the user apply production changes.
## Troubleshooting
### Common Issues
1. **Build / Check Failures**: Run `bun check` to surface TypeScript and Biome errors, and confirm dependencies are installed with `bun install`
2. **Database Issues**: Ensure the local Supabase stack is running (`bun sb:start`) and migrations are applied (`bun sb:up`)
3. **Test Failures**: Check environment setup and re-run the focused Vitest workspace before the full `bun check` gate
4. **Performance Issues**: Use React DevTools and bundle analyzers
### Getting Help
* Check existing documentation
* Search GitHub issues
* Ask in team channels
* Create detailed bug reports
## Contributing
When contributing to the platform:
1. Read the relevant documentation
2. Follow the established patterns
3. Write tests for new features
4. Update documentation
5. Submit a pull request with a clear description
# Remote Devboxes
Source: https://docs.tuturuuu.com/build/development-tools/remote-devboxes
Run heavy Tuturuuu development workflows on containerized self-hosted remote runners.
## Overview
Remote devboxes let internal root workspace members offload expensive local work
to self-hosted runner machines. A runner connects outbound to `tuturuuu.com`,
receives a synced dirty checkout, runs commands in an isolated container, streams
logs, stores artifacts, and releases the lease automatically for one-off work.
V1 is intentionally self-hosted. It does not provision cloud machines. Any
runner host should have Node.js, Bun, Docker, and Git available.
## Quick Start
```bash theme={null}
ttr login
ttr box doctor
ttr box setup
ttr box setup --dir .
ttr box setup --dir . --clone-into ./tuturuuu
ttr box agent register --name "ci-lab-1"
ttr box agent start --token
ttr box repair --dir .
ttr box upgrade --runner
TUTURUUU_DEVBOX_RUNNER_TOKEN= ttr box shutdown
```
`ttr box setup` is the default bootstrap for a runner host. It first checks
whether the current directory is already a Tuturuuu platform checkout. If it is,
setup reuses that checkout. Otherwise it can clone or reuse
`https://github.com/tutur3u/platform.git`, install dependencies with
`bun install --frozen-lockfile`, start local Supabase with `bun sb:start`,
verify `supabase status -o json`, and write local Supabase connection values
into ignored `apps/*/.env.local` files. Secret values are redacted from human
and JSON output.
Pass `--dir ` for a specific checkout. If the target exists but is not a
valid Tuturuuu checkout, interactive setup asks whether it should clone into a
nested `tuturuuu` directory. In non-interactive shells, pass
`--clone-into ` explicitly:
```bash theme={null}
ttr box setup --dir . --clone-into ./tuturuuu
```
Use `--yes` only when the host should install detected missing prerequisites
automatically. Runner registration and service installation are separate
choices.
To register the machine and install a boot-starting runner service in one
non-interactive command after login:
```bash theme={null}
ttr box setup --agent --service --runner-name "$(hostname)-devbox" --yes
```
The setup command stores the runner token in a restricted local env file and
installs a system-level runner service on Linux systemd or macOS launchd. The
service starts after reboot and restarts automatically if the runner exits.
Runner heartbeats report a small observability snapshot, including `ttr`, Bun,
Node, Docker, Git, OS, CPU, RAM, load average, and uptime. Root workspace admins
can inspect those fields from Infrastructure > Devboxes.
Devbox agent poll and heartbeat traffic is blocked by default at the platform
API boundary. Set `TUTURUUU_DEVBOX_AGENT_API_ENABLED=true` on the web app before
starting or repairing runners. When the flag is absent, agent poll and heartbeat
requests return `403` before runner tokens are authenticated or runner state is
updated.
If a host completed `ttr box setup` but does not appear in Infrastructure >
Devboxes, confirm it was registered as a runner. Plain setup prepares the
checkout, dependencies, local Supabase, and ignored env files only; it does not
create a `private.devbox_runners` row. Run setup with `--agent --service`, or
register and start the agent manually, then confirm `ttr whoami` on that host is
pointing at the same production origin and root workspace user.
If the host appears as registered but shows no heartbeat, inspect the installed
service on that host:
```bash theme={null}
sudo systemctl status --no-pager --full tuturuuu-devbox-runner.service
sudo journalctl -u tuturuuu-devbox-runner.service -n 80 --no-pager
```
The runner service reads its token from the local `devbox-runner.env` file. If
the journal shows a missing runner token or stale wrapper, upgrade the CLI and
repair the service. Repair reuses the existing token file; it does not register
a new runner token or create another runner row.
```bash theme={null}
ttr upgrade
ttr box repair --dir .
```
Use dry-run first when you want to confirm the checkout, token file, service
manager, wrapper path, and service path without writing files or calling sudo:
```bash theme={null}
ttr box repair --dir . --dry-run
```
Upgrade a runner's global CLI through the same brokered queue:
```bash theme={null}
ttr box upgrade --runner
```
The upgrade command queues `bun i -g tuturuuu` for the selected runner, waits
for completion, and returns the remote command logs and exit code.
Remove a runner from the cluster from the runner host with its runner token:
```bash theme={null}
TUTURUUU_DEVBOX_RUNNER_TOKEN= ttr box shutdown
```
Shutdown deletes the runner's server-side token rows and marks the runner
revoked. If the host installed a systemd or launchd runner service, stop and
disable that service as well so the OS does not keep restarting an invalid
agent process.
Run heavy commands remotely:
```bash theme={null}
ttr box run -- bun check
ttr box run -- bun sb:reset
ttr box run -- bun test:e2e
ttr box build --cwd apps/web
```
By default, `ttr box run` creates an auto lease, waits for an authenticated
runner to claim the job, streams recorded logs after completion, and releases
the lease after one-off work. Use `--keep` for repeated sync/run work:
```bash theme={null}
ttr box run --keep --preview-port 7803 -- bun test:e2e
ttr box sync --lease --watch
ttr box preview --lease --port 7803
ttr box release
```
## Build, Serve, And Tunnel
Use `ttr box build` for common build workloads without spelling the full remote
command every time:
```bash theme={null}
ttr box build
ttr box build --cwd apps/web
ttr box build --build-command "bun turbo:local run build -F @tuturuuu/web"
```
Use `ttr box serve` when the remote devbox should build an app, start the
server, and keep the run alive. It defaults to `apps/web` on port `7803`, stores
the preview port on the run, and returns immediately unless `--wait` is passed:
```bash theme={null}
ttr box serve --cwd apps/web --port 7803
ttr box preview --lease --port 7803
ttr box logs
ttr box stop
```
`ttr box build`, `ttr box serve`, and `ttr box tunnel` do not set a remote
command timeout by default. Pass `--timeout ` only when the run should
be killed automatically.
To expose a served app through a Cloudflare tunnel, keep the token in a local
environment variable and pass the variable name to the devbox. The queued
command references `$CLOUDFLARED_TOKEN`; the raw token is delivered only as
run-scoped env and is redacted from logs.
```bash theme={null}
export CLOUDFLARED_TOKEN=
ttr box serve --cloudflared --cloudflared-token-env CLOUDFLARED_TOKEN
ttr box tunnel --cloudflared-token-env CLOUDFLARED_TOKEN
```
`ttr box tunnel` runs a dockerized `cloudflare/cloudflared` container with host
networking on the runner. The command policy blocks inline `cloudflared --token ...` arguments, so operators should use `--cloudflared-token-env` or
another local environment variable indirection rather than pasting raw tokens
into commands.
## Container Boundary
Commands run in a per-lease Docker container or Compose project. The runner
mounts only the synced workspace and named cache volumes. Host execution is not
exposed in v1, and arbitrary host path mounts are blocked unless an operator
explicitly allowlists them.
Allowed remote workflows include:
* `bun check`, `bun test`, and package-local Bun commands
* `bun test:e2e` and Docker-backed Playwright workflows
* `bun sb:start`, `bun sb:reset`, `bun sb:up`, and `bun sb:typegen`
Blocked defaults include privileged Docker containers, Docker prune-all style
commands, absolute host mounts, `sudo`, and host-destructive filesystem
operations.
## Env And Secrets
Remote env is explicit. Local `.env*` files are never synced automatically.
The setup command only prepares ignored app env files inside the runner
checkout so local Supabase-backed apps and E2E workflows can use that runner's
own Supabase stack.
```bash theme={null}
ttr box run --env DATABASE_URL=postgres://remote -- bun check
ttr box run --database-url-env DEVBOX_DATABASE_URL -- bun check
ttr box serve --database-url-env DEVBOX_DATABASE_URL
ttr box run --env-file .env.remote -- bun test:e2e
ttr box env set --lease API_TOKEN=value
ttr box env unset --lease API_TOKEN
```
Brokered env values are scoped to the run or lease, delivered only to the
claimed runner, and redacted from command logs.
For split-resource development, run Supabase or another database on one devbox,
publish its reachable URL as `DEVBOX_DATABASE_URL` on the operator machine, and
queue the app devbox with `--database-url-env DEVBOX_DATABASE_URL`. This lets a
database-heavy runner and a Next.js runner share work while the current machine
only brokers commands and opens previews.
## Cache Policy
Runners use named cache volumes for Bun installs, Turbo, Playwright browsers,
Supabase Docker state, package manager cache, and optional `node_modules`.
Cache keys include the repo fingerprint, lockfile hash, runtime image digest,
platform, command profile, Bun/Node versions, and cache schema version.
Cleanup runs on runner startup, after runs, and during heartbeat maintenance.
The runner evicts incompatible caches first, including legacy Bun install caches
when the Bun version, lockfile hash, package profile, or cache schema changes.
It then enforces cache budgets with least-recently-used eviction while keeping a
small set of recent compatible caches to avoid thrashing.
Operators can inspect and prune cache metadata:
```bash theme={null}
ttr box cache list
ttr box cache doctor
ttr box cache prune
```
## Access
Remote devbox API routes require a Tuturuuu CLI/app session and root workspace
membership with `workspace_members.type = 'MEMBER'`. Guests and non-root
workspace users cannot create runs, leases, previews, or runner tokens.
## Local Executable Verification
Use an isolated CLI config when testing local `apps/web` so the production CLI
session is not overwritten:
```bash theme={null}
bun sb:status
docker exec supabase_db_tuturuuu psql -U postgres -d postgres -tAc "select to_regclass('private.devbox_leases'), exists (select 1 from supabase_migrations.schema_migrations where version = '20260603171600')"
TUTURUUU_CONFIG=/tmp/ttr-devbox-local-config.json bun ttr login --base-url http://localhost:7803
TUTURUUU_CONFIG=/tmp/ttr-devbox-local-config.json bun ttr box agent register --name local-devbox
TUTURUUU_CONFIG=/tmp/ttr-devbox-local-config.json bun ttr box agent start --token
TUTURUUU_CONFIG=/tmp/ttr-devbox-local-config.json bun ttr box run -- bun --version
```
The final command should print the remote command logs and exit with the remote
command's exit code. In local Supabase, `private.devbox_runs.status` should end
as `succeeded` with `exit_code = 0`.
## Production Readiness
The production API needs migration
`20260603171600_create_private_devboxes.sql` before any `ttr box` command can
create leases or runs. If the CLI prints a schema-cache error such as
`private.devbox_leases` not being found, first confirm the production migration
workflow ran for the deployed SHA. If the table exists but PostgREST still
returns the schema-cache error, refresh the schema cache from Supabase SQL
Editor:
```sql theme={null}
select to_regclass('private.devbox_leases');
select version from supabase_migrations.schema_migrations
where version = '20260603171600';
notify pgrst, 'reload schema';
```
# Adaptive Abuse Rate Limits
Source: https://docs.tuturuuu.com/build/devops/adaptive-abuse-rate-limits
Operate the reputation layer that distinguishes trusted organic users from risky automation without exposing static scoring rules.
## Overview
Authenticated rate limits now use a server-side reputation layer. The layer
scores users, sessions, API keys, IPs, CIDRs, and user-location pairs from
auditable signals, then returns a coarse risk tier and rate-limit multiplier to
the API and PostgREST guards.
The policy is intentionally server-only. The repository is open source, so do not
move scoring thresholds, bypass rules, or detailed reason codes into client code,
response headers, or public docs. User-agent, missing-header, and request-shape
checks are weak signals; they can raise caution but cannot earn trust by
themselves.
## Trust Tiers
| Tier | Operator meaning | Rate-limit behavior |
| -------------------- | ------------------------------------------------------ | ---------------------------------------------------- |
| `trusted` | Sustained low-risk, organic usage with no recent abuse | Higher authenticated budgets |
| `standard` | Normal authenticated activity or fail-safe default | Current authenticated budgets |
| `watch` | Suspicious but not high-confidence abuse | Standard or slightly lower budgets |
| `challenge_required` | Medium-risk browser mutation activity | Browser mutation routes require Turnstile step-up |
| `restricted` | High-risk activity or manual restriction | Stricter budgets and normal abuse blocks still apply |
Trust never bypasses active user suspension, active IP blocks, sensitive auth
route protections, workspace-secret hard overrides, or severe backend abuse
cascades.
One exception exists for active IP blocks: an authenticated user who passes
Turnstile through `/api/v1/rate-limit-appeals` can receive a short Redis relief
key scoped to the same browser session and `ip:`. The relief only lets
that session continue while admins review the appeal; it does not clear the
global `blocked_ips` row or help unrelated users behind the same IP.
## API Abuse IP Blocks
Session-authenticated routes may defer proxy-side `api_abuse` IP blocks only
long enough to validate route authentication. Do not treat bearer-shaped tokens,
`ttr_` strings, Supabase auth cookie names, or app-session cookie presence as
proof of an authenticated session. If the route auth check fails, return the
original block response before invoking the handler or recording normal auth
failure side effects.
## Emergency Protection Controls
Platform admins with root workspace `manage_workspace_roles` can temporarily
disable edge IP blocking or route rate limits from Infrastructure -> Rate
Limits. The controls are stored in Redis for the proxy runtime.
Redis-backed protection is intentionally availability-sensitive. When Redis is
not configured, cannot initialize, or a Redis command fails, `apps/web` fails
open for route rate limits, abuse counters, and IP-block enforcement. The app
continues to run authentication, authorization, payload-size checks, Turnstile,
user suspension, and normal request validation, but it does not enforce stale
database/manual IP blocks or process-local fallback buckets. Restore Redis to
restore rate-limit and IP-block enforcement.
* Disabling IP blocking stops enforcement of cached `ip:blocked:` entries
and prevents proxy abuse counters from creating new edge IP blocks. It does
not delete `blocked_ips` rows or existing Redis block keys.
* Disabling route rate limits skips the proxy read/mutate buckets while leaving
payload size checks and malformed request validation active.
Use these controls only for incident response or false-positive mitigation.
Re-enable them after the affected traffic pattern is understood and prefer
specific trusted workspace, IP, CIDR, or user overrides for durable fixes.
## Signals
The layer records rolling signals for:
* rate-limit hits, failed auth, repeated `4xx`, `401`, `403`, and `429`
* payload abuse and automation-like clients
* missing or scripted browser headers as weak negative signals
* older accounts, stable sessions, successful organic usage, and passed
challenges as positive signals
* admin overrides and override revocations
If reputation lookup, challenge verification, or signal recording fails, request
handling must fall back to standard limits. Failures must never grant elevated
limits.
## Backend 429 Cascades
Supabase Auth and PostgREST/backend `429` responses are availability signals,
not user-scoped abuse proof. Handlers return `429` to the caller with
`Retry-After`, but a single backend rate limit must not create or extend
`ip:blocked:`, `blocked_ips`, or `user_suspensions`. User suspension
requires manual operator action or repeated, user-attributable abuse counters,
and automated suspensions should be expiry-bound unless a separate policy
explicitly justifies permanence.
## Shared-IP Login Traffic
English centers, classrooms, and offices often put many legitimate teachers
behind one public NAT IP. The proxy guard therefore keeps password login, OTP
send, and OTP verify on separate auth buckets instead of sharing the generic
mutation budget:
| Policy | Default minute/hour/day budget | Override variables |
| ---------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `password-login` | `60 / 600 / 4000` | `API_PROXY_PASSWORD_LOGIN_LIMIT_MINUTE`, `API_PROXY_PASSWORD_LOGIN_LIMIT_HOUR`, `API_PROXY_PASSWORD_LOGIN_LIMIT_DAY` |
| `otp-send` | `30 / 180 / 300` | `API_PROXY_OTP_SEND_LIMIT_MINUTE`, `API_PROXY_OTP_SEND_LIMIT_HOUR`, `API_PROXY_OTP_SEND_LIMIT_DAY` |
| `otp-verify` | `60 / 600 / 4000` | `API_PROXY_OTP_VERIFY_LIMIT_MINUTE`, `API_PROXY_OTP_VERIFY_LIMIT_HOUR`, `API_PROXY_OTP_VERIFY_LIMIT_DAY` |
OTP send also has a shared-IP abuse guard with
`ABUSE_OTP_SEND_IP_LIMIT_MINUTE`, `ABUSE_OTP_SEND_IP_LIMIT_HOUR`, and
`ABUSE_OTP_SEND_IP_LIMIT_DAY`. Keep the per-email cooldown/hour/day limits in
place; they are the primary protection against repeated sends to the same
mailbox. Human auth route `429` responses should return `Retry-After`,
`X-RateLimit-Client-IP`, and `X-RateLimit-Policy` to the caller so the
rate-limit details dialog can show the server-observed public IP and policy
without asking the customer to visit a separate IP lookup page. These auth
`429`s must not be converted into `api_abuse` IP blocks by the proxy escalation
path. Utility-level OTP send, OTP verify, MFA verify, reauth verify, and
password-login failure counters also throttle without writing `blocked_ips`
rows; their abuse events include `hard_block_suppressed` metadata when the old
hard-block threshold would have been crossed. Generic anonymous API route-limit
hits, malformed auth-cookie abuse, scanner-like requests, `api_auth_failed`, and
manual operator blocks can still escalate to hard IP blocks.
## Trusted-Location Read Uplift
High-density centers (many staff behind one office NAT IP) can hit the read-limit
toast ("Bạn đang bị giới hạn tần suất. Thử lại sau 60 giây…") during normal
browsing. Untrusted read traffic keys per `ip:` against the anonymous default
(`60 / 240 / 1200` per minute/hour/day), so everyone behind the shared IP draws
from one bucket. A trusted-location override keyed by CIDR scales that bucket at
the edge (e.g. `60 -> 300/min` at `5x`) and lets trusted sessions key
per-session. The `subject_key` must match the edge `getCidrSubjectKeyEdge` form
(`cidr:.0/24` for IPv4, `cidr:::/64` for IPv6).
1. Confirm the center's public IP via Infrastructure -> Abuse Intelligence or by
asking the center, then derive the `/24`.
2. Insert a time-bound override. An operator runs this against production; the
platform never auto-pushes production SQL:
```sql theme={null}
insert into public.abuse_trust_overrides
(subject_type, subject_key, tier, trust_multiplier, limit_mode, reason, expires_at)
values
('cidr', 'cidr:203.0.113.0/24', 'trusted', 5.00, 'inherit_multiplier',
'Trusted center office NAT — many staff, organic read traffic',
now() + interval '90 days');
```
3. The edge trust cache (`EDGE_TRUST_CACHE_TTL_SECONDS`, \~1h) reconciles via
`list_trusted_subjects_for_cache()` and the `sync-trust-cache` cron, so the
uplift takes effect within the TTL without a deploy. Set
`API_PROXY_EDGE_TRUST_ENABLED=0` as a kill switch to fall back to legacy
per-IP read limiting.
Fix read amplification in the app first (batch per-row table fetches server-side
so one page load costs a few reads, not dozens); the location uplift is
defense-in-depth for genuinely dense, legitimate locations.
## Trusted-Workspace Uplift
Admins can also raise limits for a legitimate high-volume workspace from
Infrastructure -> Rate Limits by creating a `workspace` subject rule:
* `subject_type`: `workspace`
* `subject_key`: `workspace:`
* `tier`: `trusted`
* `limit_mode`: `inherit_multiplier`
* `trust_multiplier`: start at `3`
* `expires_at`: prefer 30-90 days
Workspace rules are enforced at the edge for workspace-scoped API reads and are
visible in the Rate Limits admin center. They are safer than globally unblocking
or uplifting a noisy IP when many unrelated organizations may share that public
address. Keep them time-bound and review live usage before renewing.
## Rate-Limit Appeals
When the details dialog shows `X-Proxy-Block-Reason: ip-already-blocked`, the
request is being rejected by an active hard IP block. Raising route limits alone
will not unblock that user. Admins should first inspect Infrastructure ->
Blocked IPs and Abuse Intelligence, then clear the block if it is a false
positive.
Legitimate authenticated users can submit a review request from the rate-limit
details dialog. The submission route is the only route allowed through an active
IP block, and it still requires:
* an authenticated session cookie
* a valid Turnstile token
* bounded sanitized diagnostics from the dialog
* a low per-user/IP appeal throttle
Submitting an appeal writes `rate_limit_appeals` and grants only temporary
session+IP relief. From Infrastructure -> Rate Limit Appeals, admins can approve,
reject, or close the appeal. Approval clears the active IP block and, by
default, creates a time-bound trusted workspace rule with `3x` multiplier for 30
days. Admins can edit the workspace ID, multiplier, and expiry before approval,
or create more specific IP/CIDR/user rules manually from Infrastructure -> Rate
Limits.
## Step-Up Challenges
Browser mutations with medium risk should require Turnstile through the existing
server verification path. The API returns a generic challenge-required response
without exposing the exact scoring rules. API keys, CLI calls, cron jobs,
webhooks, and native clients do not receive browser challenges; they rely on
token, workspace, and API-key reputation instead.
## Local E2E Isolation
The Playwright rate-limit suite intentionally exhausts budgets and records 429
signals. Its `resetDbRateLimits()` helper must clear both PostgREST rate-limit
counters and generated adaptive abuse state (`abuse_activity_signals`,
`abuse_step_up_challenges`, and `abuse_reputation_subjects`) before each spec.
Specs that intentionally hit authenticated route limits should use Redis and a
fresh local dev-session account per test/retry so Redis-backed user keys and
asynchronous adaptive reputation writes cannot bleed into the next case. Without
Redis, `apps/web` rate-limit and IP-block checks fail open and will not produce
the expected `429` responses.
## Admin Investigation
Use Infrastructure -> Abuse Intelligence to inspect:
* trusted, watched, and restricted subject counts
* recent signals and coarse reason codes
* challenge pass/fail trends
* risky users, sessions, API keys, IPs, and CIDRs
* active manual overrides
When investigating a false positive:
1. Check whether the subject has recent rate-limit, auth-failure, or payload
abuse signals.
2. Compare the subject with related location and user-location entries.
3. Review challenge outcomes and recent route diversity before granting trust.
4. Add a time-bound override only when the activity is clearly organic.
5. Revoke trust immediately if the subject later shows scripted behavior,
account takeover indicators, or noisy API-key traffic.
For a live classroom login incident, ask the customer to open the rate-limit
details dialog and share the copied details. Use the `identity.clientIp`,
`limit.retryAfterSeconds`, and `limit.policy` fields to check Infrastructure ->
Abuse Intelligence for active IP blocks or abuse events with `api_abuse` or
`password_login_failed`. If the activity is confirmed organic, clear the active
false-positive block, reset affected counters, or add a time-bound IP/CIDR trust
or rate-limit uplift. Do not relax the per-email OTP cooldown or failed-attempt
protections.
If the copied details include `limit.proxyBlockReason =
ip-already-blocked`, route-limit tuning is secondary. Clear the active block or
approve the user's Rate Limit Appeal first, then decide whether the root cause
needs a trusted workspace, IP/CIDR uplift, or app-level batching fix.
Manual overrides require a reason and are written to the audit signal stream.
Prefer expiry-bound overrides for trust and watch decisions so stale operational
judgment does not become permanent policy.
# Chat Realtime Runbook
Source: https://docs.tuturuuu.com/build/devops/chat-realtime-runbook
Operating notes for the internal Chat realtime sidecar and web API proxy.
Chat realtime lives in `apps/chat-realtime`. It is an internal Bun SSE service
used by `apps/web` for workspace chat fanout. Browser clients never connect to
the sidecar directly; they subscribe to
`/api/v1/workspaces/:wsId/chat/realtime`, and `apps/web` verifies membership,
mints a short-lived internal token, connects to the sidecar, and forwards SSE
events back through the normal web origin.
## Required Environment
* `CHAT_REALTIME_TOKEN_SECRET`: preferred shared HMAC secret used by `apps/web`
and `apps/chat-realtime`. If unset, both fall back to the platform Supabase
service secret. Production should set this explicitly when rotating realtime
credentials independently.
* `CHAT_REALTIME_INTERNAL_URL`: Docker-internal origin for `apps/web`, normally
`http://chat-realtime:7817`. Local non-Docker `bun dev` defaults to
`http://localhost:7817`.
No Cloudflare tunnel port is required for Chat realtime. The public route is the
existing `apps/web` API path, which keeps authorization, cookies, app-session
auth, and deployment topology owned by the web container.
## Blue/Green Deployment
`docker-compose.web.prod.yml` includes `chat-realtime` as a support sidecar
managed by the blue/green watcher. The sidecar exposes only port `7817` inside
the Compose network. `apps/web` publishes committed chat mutations to
`POST /publish`; clients receive them via the proxied SSE stream.
Every published event must include an explicit `audience`. Channel events use a
workspace audience, while direct, group, and AI conversation events use a
user-list audience derived from the authorized conversation members. The sidecar
keeps workspace rooms for connection management, but it filters each event by
that audience before sending any full conversation or message payload to a
client.
If Chat realtime is unavailable, chat mutations still succeed. The web API logs
the publish failure and clients continue to recover through normal query
refetching and reconnects.
# Environments & Release Flow
Source: https://docs.tuturuuu.com/build/devops/environments-release-flow
How branches map to environments, and what happens between preview, staging, production, and self-hosted deploys.
This page explains how Tuturuuu moves from code to running environments.
## Branch-to-Environment Map
| Branch or event | What deploys | Follow-up automation |
| ----------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Feature branch push | CI only; Vercel preview deployments require trusted manual dispatch from `main` with `preview_ref` | None by default |
| `main` push | Platform Vercel preview build validation | `supabase-staging.yaml` can apply staging migrations after the preview build succeeds |
| `production` push | Platform Vercel production deployment | `supabase-production.yaml` can apply production migrations after the deployment and staging prerequisites pass |
| `main` push for `apps/discord` | Discord Modal deployment after Python CI succeeds | None |
| Manual `workflow_dispatch` | Depends on workflow | Bypasses normal branch trigger timing, but not the workflow logic; production Supabase migration dispatches must select `production` |
| Self-hosted server after `git pull` | Docker deploy from current checkout | Optional blue/green cutover through `bun serve:web:docker:bg` |
## Hosted Web Release Flow
### Preview
* Preview workflows are named `vercel-preview-.yaml`.
* Most preview workflows are manual-only from `main` with a required
`preview_ref`; `vercel-preview-platform.yaml` also triggers on protected
`main` pushes so the staging database workflow receives a same-SHA platform
build signal.
* `e2e-tests.yaml` also runs only on non-`production` pushes; production
promotion relies on the already-green validation before the branch is
promoted plus the production build-validation gates below.
* Production workflows cancel superseded per-app runs only when a newer
`production` push arrives; `main` commits and manual recovery runs leave an
active production deployment untouched.
* Each deploy job is bound to a `vercel-preview-` GitHub Environment and
introduces Vercel credentials only after dependency installation.
* Platform preview is build validation only; on-premise machines own the actual
`apps/web` runtime deployment.
* Satellite preview deploys remain the normal Vercel validation surface for
branch work and merged `main` changes.
* Platform preview can reuse a successful same-SHA production platform build
marker to avoid redundant preview builds. Production does not reuse preview
markers because it must build and deploy the prebuilt production artifacts.
### Staging Database
* `supabase-staging.yaml` is tied to the `Vercel Platform Preview Deployment` workflow on `main`.
* The staging migration runs only when the triggering platform preview build concludes successfully, or when manually dispatched.
* The staging deploy step runs `supabase db push --include-all` after linking
the staging Supabase project, so staging can converge after `main` migration
history changes.
* This means `main` is where app preview and staging schema advancement are
meant to stay aligned.
* If a production commit was promoted before the same-SHA staging migration run
existed, manually dispatch `supabase-staging.yaml` from `main`; production
migration re-evaluates after that staging run succeeds.
### Production
* Satellite production web deploys run through `vercel-production-.yaml`.
* A production push first runs `vercel-production.yaml`, which computes all
affected apps in one runner and dispatches only the necessary per-app
workflows. Each target uses its own last successful deployment marker, so an
app skipped on an earlier commit still receives any undeployed changes.
* Hosted `apps/web` production deploys run through
`vercel-production-platform.yaml`; self-hosted machines still use the Docker
release flow below for their own runtime deployment.
* `apps/apps` production deploys are handled by `vercel-production-apps.yaml`.
* `apps/qr` production deploys are handled by `vercel-production-qr.yaml`.
* Each production app uses a workflow-and-branch concurrency group whose
`cancel-in-progress` predicate matches only `production` push events, so the
newest production SHA remains deployable without letting `main` commits or
manual recovery runs cancel it.
* Production deploy jobs are bound to `vercel-production-` GitHub
Environments and reject manual dispatches unless the selected branch is
`production`.
### Browser App Version Metadata
* Browser apps share one platform version from
`packages/utils/src/platform-release.ts`; do not read visible app versions
from individual app `package.json` files.
* The shared browser app version is the `TUTURUUU_PLATFORM_VERSION` constant
exported from `packages/utils/src/platform-release.ts`. Release Please owns
that value (it carries the `x-release-please-version` annotation), so do not
pin a literal version in docs or bump it by hand. Read the constant directly
when you need the current value.
* Every Vercel browser app workflow runs
`bun run --silent scripts/ci/generate-build-metadata.ts` before
`vercel build`. The generated metadata includes the commit hash, short hash,
commit message, ref, environment, deployment URL, deployment stamp, and build
timestamp.
* The account-scoped version badge uses `public.user_configs` key
`SHOW_VERSION_BADGE`. It is hidden by default and only exact
`@tuturuuu.com` accounts may enable a truthy value.
* Exact-domain enforcement happens on the server through
`isExactTuturuuuDotComEmail`; subdomains such as `@xwf.tuturuuu.com` are not
eligible, and client cookies cannot make the badge render for ineligible
users.
### Production Database
`supabase-production.yaml` is stricter than staging:
1. It re-evaluates automatically after either `Vercel Production Deployment
Planner` completes its selected production deploys on `production` or
`Supabase Staging Migration` completes on `main`.
2. The production planner for the target commit must have concluded
successfully on `production`, and the selected platform deployment must have
recorded the successful
`vercel-production-platform` deployment marker for the same SHA. A workflow
run that skipped because package releases were still publishing is not
enough.
3. The `main` staging migration for the same target commit must have completed
with a `success` conclusion.
4. A manual dispatch can still be used when an operator explicitly wants to run
it, but the dispatch must select the `production` branch and still satisfy
the same production-deployment and staging-migration SHA checks.
The staging-triggered path is an automatic retry for the case where production
production deploy finishes before staging migration does. The production
migration job still checks out the verified target commit explicitly before
running `supabase db push --include-all`. This keeps production schema changes
behind application deployment, staging validation, and branch promotion
for the same commit.
## Self-Hosted Web Release Flow
If a server receives code through `git pull`, use the Docker production commands from
the checked-out commit:
* `bun serve:web:docker` for in-place replacement
* `bun serve:web:docker:bg` for blue/green rebuild-before-cutover deployment
Blue/green is the safer path when uptime matters because the new image is built and
health-checked before the proxy is reloaded.
## Migration Release Path (TanStack + Rust)
`apps/web` (Next.js, port `7803`) is being replaced by `apps/tanstack-web`
(TanStack Start) plus `apps/backend` (the Rust HTTP core, port `7820`). See
[the TanStack/Rust migration overview](/platform/architecture/tanstack-rust-migration)
for the full plan.
While that migration is incomplete, Docker blue/green remains the canonical
production rollout and rollback mechanism for the live hostname. Cloudflare
Workers are used as a separate, incremental **preview** surface to prove Worker
compatibility for `apps/tanstack-web` and `apps/backend` before any cutover:
* Validate the Wrangler deploy configs without contacting Cloudflare with
`bun check:cloudflare`.
* Deploy the Rust backend Worker (`apps/backend/wrangler.jsonc`) and the
TanStack Start Worker (`apps/tanstack-web/wrangler.jsonc`) to preview, then
smoke both returned origins with `bun smoke:cloudflare`.
* Do not route the production hostname to preview Workers until the
TanStack/Rust route manifest, compare-mode Docker E2E, benchmark report, and
cutover gates (`bun migration:tanstack:gates`) all pass.
* Rollback from the preview path is DNS/routing-based (remove the Cloudflare
route or roll back the Worker version); the Docker blue/green `apps/web` stack
keeps serving the canonical hostname throughout.
The full Cloudflare preview, secret-bootstrap, smoke, and cutover-gate runbook
lives in
[Web Docker Deployment → Cloudflare Preview Path](/build/devops/web-docker-deployment).
## Team Release Checklist
1. Merge the change set with green CI.
2. If Docker behavior changed, make sure `docker-setup-check.yaml` is green.
3. If database migrations changed, confirm the `main`-driven staging path first.
4. Promote to `production` only after preview and staging behavior is understood.
5. Confirm the follow-up Supabase workflow after production deployment.
6. For self-hosted rollout, deploy from the intended commit and prefer `bun serve:web:docker:bg`.
## Rollback Guidance
* Vercel-hosted rollback: for satellite apps, redeploy the previous known-good commit or use Vercel rollback tooling.
* Supabase rollback: use a corrective migration instead of editing applied migration history.
* Self-hosted Docker rollback: checkout the previous known-good commit and rerun `bun serve:web:docker:bg`.
## What Not To Do
* Do not push production schema changes directly from a laptop as the normal path.
* Do not assume staging and production migrations are interchangeable; they are gated differently.
* Do not manually dispatch production Supabase migration from `main`; the only
valid `main` path is the automatic staging-migration re-evaluation, and the
production gate still requires the same commit to have both a successful
production platform deployment and a completed successful `main` staging
migration.
* Do not use the in-place Docker path when you specifically need rebuild-before-restart semantics.
# GitHub Actions Runbook
Source: https://docs.tuturuuu.com/build/devops/github-actions-runbook
How the repository’s automation is organized, gated, and operated.
GitHub Actions is the primary automation plane for Tuturuuu.
## Workflow Gatekeeping
Every major workflow can be disabled centrally through
`tuturuuu.ts`.
The reusable workflow
`ci-check.yml`
reads that config and emits `should_run`, which downstream jobs use before doing
real work.
For Vercel deployments, the production planner and platform preview perform
affected-app gating before any install, build, or deploy step starts. They check
the current GitHub event's changed files against Vercel app metadata and
workspace dependency closures from `tuturuuu.ts`. Platform preview performs the
check inside its deploy job so protected `main` pushes create one runner, not a
reusable gate runner followed by a deployment runner.
The changed-file resolver must evaluate the full effective change range:
* Manual `workflow_dispatch` runs bypass affected-app gating when the workflow is
enabled.
* Vercel production push workflows first look for the latest successful GitHub
Deployment marker for the same workflow and branch, then diff that marker SHA
to the current `GITHUB_SHA`.
* If no marker is available, push events use the GitHub event payload's complete
commit file list instead of falling back to only the latest commit.
* Pull request events diff the PR base SHA to the PR head SHA.
* If the resolver cannot prove the full range, it leaves changed-file state
unavailable so Vercel gating defaults open.
Affected Vercel rules:
* `apps//**` runs that app's preview or production Vercel workflow.
* `packages//**` runs every Vercel app whose transitive `workspace:*`
dependency closure includes that package.
* Storefront UI modules under
`packages/ui/src/components/ui/storefront/**` are scoped to their verified
consumers, Inventory and Storefront, instead of rebuilding every app that
imports another part of `@tuturuuu/ui`.
* `apps/*/package.json` and `packages/*/package.json` are dependency changes for
their owning workspaces.
* `bun.lock`, root `package.json`, `turbo.json`, `tuturuuu.ts`, and
`.github/workflows/ci-check.yml` run every Vercel app workflow.
* A Vercel workflow file change, such as
`.github/workflows/vercel-preview-calendar.yaml`, runs that specific workflow.
* If changed files cannot be computed, the Vercel gate defaults open.
Production pushes run `.github/workflows/vercel-production.yaml` once. That
planner evaluates all targets from their individual deployment markers and
calls only the affected `vercel-production-.yaml` workflows as reusable
jobs. This keeps every automatic deployment attached to the original push SHA
instead of creating `workflow_dispatch` runs. Do not restore per-app `push`
triggers or `check-ci` jobs; doing so recreates the queued preflight fan-out the
planner replaces.
Each reusable production workflow uses a static per-app concurrency prefix plus
the Git ref. Do not derive this key from reusable-workflow caller context: that
can make sibling app jobs share one group, cancel one another, and surface red X
statuses where unselected jobs should be skipped. Enable `cancel-in-progress`
only when the inherited event is a push to `refs/heads/production`; `main`
commits and manual recovery dispatches must not cancel the active deployment.
`bun.lock`-only changes intentionally run all Vercel app deploys because ownership
is ambiguous without a manifest or source path.
## Pull Request Close Cancellation
`cancel-pr-runs-on-close.yaml` runs on the `pull_request_target` `closed` event
and cancels active GitHub Actions runs that still belong to the closed pull
request. It exists to stop queued or long-running checks after a PR is closed
without adding `closed` triggers to every CI workflow.
Because `pull_request_target` runs with base-repository privileges, keep this
workflow narrow:
* Do not checkout or execute pull request head code.
* Checkout only the trusted default branch, with persisted credentials disabled.
* Pin GitHub-owned actions to full commit SHAs.
* Grant only `actions: write` and `contents: read` to the cancellation job.
* Do not pass repository secrets other than the short-lived `GITHUB_TOKEN`.
The cancellation script lists active runs by the closed PR head SHA. It cancels
matching PR workflow runs and same-repository branch push runs for the PR branch,
but never cancels `main` or `production` push runs. Runs in contributor forks are
outside the base repository token's authority and are not cancelled by this
workflow. The same trusted job paginates through Actions cache entries and
deletes only entries whose ref equals the closed pull request's merge ref
(`refs/pull//merge`). It never deletes default-branch cache entries.
## Main Workflow Groups
### Hosted web build and deployment checks
* `vercel-preview-platform.yaml`
* `vercel-preview-apps.yaml`
* `vercel-preview-calendar.yaml`
* `vercel-preview-chat.yaml`
* `vercel-preview-cms.yaml`
* `vercel-preview-drive.yaml`
* `vercel-preview-finance.yaml`
* `vercel-preview-inventory.yaml`
* `vercel-preview-infrastructure.yaml`
* `vercel-preview-learn.yaml`
* `vercel-preview-mail.yaml`
* `vercel-preview-meet.yaml`
* `vercel-preview-mind.yaml`
* `vercel-preview-nova.yaml`
* `vercel-preview-qr.yaml`
* `vercel-preview-rewise.yaml`
* `vercel-preview-shortener.yaml`
* `vercel-preview-storefront.yaml`
* `vercel-preview-tasks.yaml`
* `vercel-preview-teach.yaml`
* `vercel-preview-track.yaml`
* A matching `vercel-production-*.yaml` workflow exists for each app above
This list drifts as satellite apps are added; treat
`.github/workflows/vercel-preview-*.yaml` and
`.github/workflows/vercel-production-*.yaml` as the authoritative source.
CMS uses the same Vercel deployment pattern as the other satellite apps:
* preview workflow: `vercel-preview-cms.yaml`
* production workflow: `vercel-production-cms.yaml`
* project secret: `VERCEL_CMS_PROJECT_ID`
* environments: `vercel-preview-cms` and `vercel-production-cms`
The Apps gateway uses the same pattern:
* preview workflow: `vercel-preview-apps.yaml`
* production workflow: `vercel-production-apps.yaml`
* project secret: `VERCEL_APPS_PROJECT_ID`
* environments: `vercel-preview-apps` and `vercel-production-apps`
QR uses the same pattern:
* preview workflow: `vercel-preview-qr.yaml`
* production workflow: `vercel-production-qr.yaml`
* project secret: `VERCEL_QR_PROJECT_ID`
* environments: `vercel-preview-qr` and `vercel-production-qr`
Infrastructure uses the same deployment pattern:
* preview workflow: `vercel-preview-infrastructure.yaml`
* production workflow: `vercel-production-infrastructure.yaml`
* project secret: `VERCEL_INFRASTRUCTURE_PROJECT_ID`
* environments: `vercel-preview-infrastructure` and
`vercel-production-infrastructure`
Mail uses the same deployment pattern:
* preview workflow: `vercel-preview-mail.yaml`
* production workflow: `vercel-production-mail.yaml`
* project secret: `VERCEL_MAIL_PROJECT_ID`
* environments: `vercel-preview-mail` and `vercel-production-mail`
These workflows:
* run with default `contents: read` permissions
* bind deploy jobs to `vercel-preview-` or `vercel-production-`
GitHub Environments
* run preview workflows only through manual dispatch from `main`, with a
required `preview_ref` input and an actor present in the
`TRUSTED_PREVIEW_DEPLOY_ACTORS` repository variable; `vercel-preview-platform.yaml`
is the exception and also runs on protected `main` pushes so Supabase staging
migrations keep their same-SHA prerequisite signal
* reject production manual dispatches from non-production branches before
install/build/deploy work starts
* call production workflows from the production push planner through
`workflow_call`; retain `workflow_dispatch` only for deliberate operator reruns
* install Bun with `.github/actions/setup-bun-with-retry`, pinned to the root
`packageManager` Bun version and retried with exponential backoff
* install dependencies through `scripts/ci/run-with-backoff.sh` so transient Bun
tarball or cache failures retry after cache cleanup
* build selected shared workspace dependencies before Vercel resolves package exports:
`@tuturuuu/types`, `@tuturuuu/supabase`, and `@tuturuuu/internal-api`
* run `vercel pull`
* run `vercel build`
* deploy prebuilt artifacts for satellite apps and the hosted platform
production workflow
* treat `vercel-preview-platform.yaml` as platform build validation only;
self-hosted machines own their own Docker `apps/web` runtime deployment
* cancel superseded production runs through per-workflow, per-branch
concurrency only for newer `production` pushes, so `main` commits and manual
recovery dispatches cannot interrupt the active deployment
* record non-blocking GitHub Deployment markers after successful Vercel preview
builds and production deploys so the next production push can evaluate every
change since the last successful run
The platform preview workflow can cross-credit a successful same-SHA production
platform build marker. Production does not cross-credit preview markers because
it must build and deploy the prebuilt production artifacts. This only applies to
the platform workflows; satellite Vercel workflows still build and deploy
independently.
### Database automation
* `supabase-staging.yaml`
* `supabase-production.yaml`
These cover staging schema promotion and production schema promotion.
Database workflows that install the Supabase CLI use
`.github/actions/setup-supabase-cli-with-retry` after checkout. The local action
passes `github.token` through to `supabase/setup-cli@v2`, leaves `version` empty
so the repo-pinned Supabase CLI version is used instead of the anonymous
`latest` release lookup, and retries failed setup attempts with bounded
exponential backoff.
### Docker automation
* `docker-setup-check.yaml`
* `rust-backend.yml`
`docker-setup-check.yaml` is the workflow to watch whenever Docker files,
compose files, or Docker helper scripts change. `rust-backend.yml` owns the
`apps/backend` Rust service checks and the migration Cloudflare checks:
formatting, locked dependency fetch, Clippy, tests, Cloudflare Worker target
validation, smoke reporter unit tests, native binary build, Docker image build,
`apps/tanstack-web` type-check/test validation, and the TanStack route-tree
generator/formatter unit test. Keep `scripts/generate-tanstack-route-tree.*`
in this workflow's path filters so route-tree generation changes run before
TanStack/Rust deployment handoff.
### TanStack/Rust Cloudflare deployment
`rust-backend.yml` is also the manually dispatched Cloudflare preview deployment
workflow for the migration pair:
* Rust backend Worker: `apps/backend/wrangler.jsonc`, Worker name
`tuturuuu-backend`.
* TanStack Start Worker: `apps/tanstack-web/wrangler.jsonc`, Worker name
`tuturuuu-tanstack-web`.
The workflow stays switchboard-controlled by the `rust-backend.yml` entry in
`tuturuuu.ts`. Pull request and push runs are validation-only and use
`contents: read`; they do not require Cloudflare secrets just to type-check,
test, lint config, build the Rust Worker bundle, or build the backend Docker
image.
Manual dispatch inputs:
* `deploy_target`: `none`, `backend`, `tanstack-web`, or `all`.
* `deployment_mode`: `dry-run` or `deploy`. Keep the default `dry-run` for the
first run in a new account or environment.
Run `rust-backend.yml` from `main` for Cloudflare deployments. The deployment
preflight requires the dispatch ref to be `refs/heads/main` and `github.actor`
to be listed in the comma-delimited `TRUSTED_CLOUDFLARE_DEPLOY_ACTORS`
repository variable before it loads `CLOUDFLARE_API_TOKEN`. Keep that allowlist
limited to maintainers who can approve secret-backed Worker deployments.
Cloudflare deploy credentials must not be exposed to arbitrary branch code;
review or merge deployment changes before dispatching a secret-backed run.
The deploy jobs are bound to the `cloudflare-workers-preview` GitHub
Environment. Configure these GitHub values there before expecting deployment
jobs to run:
| Name | GitHub storage | Purpose |
| ----------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------- |
| `CLOUDFLARE_API_TOKEN` | Environment secret | Cloudflare API token with Workers Scripts edit access for this account. |
| `CLOUDFLARE_ACCOUNT_ID` | Environment variable | Cloudflare account id used by Wrangler. A same-named secret is accepted only as a fallback. |
| `BACKEND_WORKER_ORIGIN` | Environment variable | Deployed Rust backend Worker origin used by the post-deploy smoke gate. |
| `TANSTACK_WEB_WORKER_ORIGIN` | Environment variable | Deployed TanStack Worker origin used by the post-deploy smoke gate. |
| `CLOUDFLARE_SMOKE_BACKEND_INTERNAL_TOKEN` | Environment secret | Dedicated smoke token mapped to `BACKEND_INTERNAL_TOKEN` only while running `bun smoke:cloudflare`. |
If either value is missing, the workflow emits a
`Cloudflare deployment skipped` warning, records setup guidance in the job
summary, and skips deploy jobs after CI validation has completed. The warning
names only the missing GitHub Environment keys, not their values. A non-empty
but invalid or under-scoped token normally surfaces as a Worker secret preflight
skip or a Wrangler failure before upload.
Worker runtime secrets live in Cloudflare, not GitHub. After the GitHub
credential preflight passes, a dedicated Worker secret preflight checks the
selected Workers with `wrangler secret list`.
It reads only secret names, never secret values. If Wrangler cannot list names,
or if any required name is
missing, the workflow emits a warning, writes setup commands to the job summary,
and skips the selected deploy job instead of failing later inside a deploy
step. Missing-secret warnings use the `Missing Worker secrets` wording and list
only secret names plus `bun wrangler secret put ...` commands; they never print
or infer secret values.
| Worker | Required Cloudflare Worker secrets |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tuturuuu-backend` | `BACKEND_INTERNAL_TOKEN`, `TUTURUUU_APP_COORDINATION_SECRET`, `SUPABASE_URL`, `SUPABASE_SERVICE_ROLE_KEY`, `CRON_SECRET`, `DISCORD_APP_DEPLOYMENT_URL`, `AURORA_EXTERNAL_URL`, `AURORA_EXTERNAL_WSID` |
| `tuturuuu-tanstack-web` | `BACKEND_PUBLIC_ORIGIN`, `BACKEND_INTERNAL_TOKEN` |
Bootstrap missing Worker secrets from a trusted local shell with Wrangler:
```sh theme={null}
bun wrangler secret put BACKEND_INTERNAL_TOKEN --config apps/backend/wrangler.jsonc
bun wrangler secret put TUTURUUU_APP_COORDINATION_SECRET --config apps/backend/wrangler.jsonc
bun wrangler secret put SUPABASE_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put SUPABASE_SERVICE_ROLE_KEY --config apps/backend/wrangler.jsonc
bun wrangler secret put CRON_SECRET --config apps/backend/wrangler.jsonc
bun wrangler secret put DISCORD_APP_DEPLOYMENT_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put AURORA_EXTERNAL_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put AURORA_EXTERNAL_WSID --config apps/backend/wrangler.jsonc
bun wrangler secret put BACKEND_PUBLIC_ORIGIN --config apps/tanstack-web/wrangler.jsonc
bun wrangler secret put BACKEND_INTERNAL_TOKEN --config apps/tanstack-web/wrangler.jsonc
```
Deploy the backend before the TanStack Worker when `deploy_target` is not `all`.
The TanStack Worker has a `BACKEND` service binding to `tuturuuu-backend`, so a
frontend-only deployment assumes the backend Worker and its secrets already
exist. When `deploy_target=all`, a skipped backend deployment also keeps the
TanStack deployment skipped, even if the TanStack Worker secrets are present.
`deployment_mode=dry-run` compiles and runs Wrangler checks without uploading,
but it still uses the same Cloudflare account, token, Worker name, and secret
name preflights as `deployment_mode=deploy`. Treat a dry-run skip as a setup
gap, not as a deploy failure. `deployment_mode=deploy` uploads the selected
Worker version only after those preflights pass.
When `deploy_target=all` and `deployment_mode=deploy`, the
`post-deploy-smoke` job runs only after both Worker deploy jobs succeed. It
requires `BACKEND_WORKER_ORIGIN`, `TANSTACK_WEB_WORKER_ORIGIN`, and
`CLOUDFLARE_SMOKE_BACKEND_INTERNAL_TOKEN`; missing values fail the full deploy
workflow with a warning, an error, and setup guidance in the step summary
instead of silently skipping runtime verification. The warning title is
`Cloudflare smoke inputs missing`, and the error title is
`Cloudflare smoke verification blocked`. The job runs:
```sh theme={null}
bun smoke:cloudflare --output "$CLOUDFLARE_SMOKE_REPORT_PATH"
```
The JSON report is written under
`tmp/benchmarks/web-migration/-/cloudflare-smoke.json`,
uploaded as the `cloudflare-smoke--` workflow artifact, and
summarized in the GitHub step summary. The report must come from the deployed
backend and TanStack Worker origins; do not point these variables at local
Wrangler or Docker origins in the `cloudflare-workers-preview` Environment.
### Quality and security
* `type-check.yaml`
* `turbo-unit-tests.yaml`
* `biome-check.yaml`
* `codeql.yml`
* `codecov.yaml`
* `i18n-check.yaml`
* `check-migrations.yml`
* `check-migration-timestamps.yml`
* `branch-name-check.yaml`
The test workflows (`turbo-unit-tests.yaml` and `codecov.yaml`) run `bun setup`
through `scripts/ci/run-with-backoff.sh` before executing tests so dependency
installation and workspace package builds match the local setup path while
surviving transient tarball extraction failures. The Codecov coverage test run
also uses the helper with a two-attempt cap so transient runner interruptions
such as exit code `130` retry once without masking deterministic test failures.
Vercel workflows should invoke Turborepo through `bun turbo:local ...` after the
retried `bun install` so CI uses the pinned repo dependency instead of resolving
a global or downloaded Turbo binary. Cacheable builds, type checks, and tests
must be executed through `.github/actions/run-with-turbo-remote-cache`; trusted
jobs pass the dedicated repository `TURBO_TOKEN` and `TURBO_TEAM` variable,
while pull-request and Dependabot jobs leave both inputs empty and use the
task-family local fallback cache.
Vercel deploy credentials must stay environment-scoped. Do not put
`${{ secrets.* }}` values in a Vercel workflow-level `env:` block, and do not
export production Supabase, encryption, or provider secrets from GitHub
Actions. Store app runtime and build-time configuration in the Vercel project
environment. Remote-cache identity is the narrow exception: it is passed only
to the wrapped `vercel build` step and never written to `GITHUB_ENV` or a
workflow/job environment. The regression test
`bun test scripts/ci/release-workflows.test.js` enforces this for every
`vercel-preview-*.yaml` and `vercel-production-*.yaml` workflow.
### Cache and artifact resources
`actions-storage-report.yaml` runs weekly and can be dispatched manually. It
is read-only: it queries the repository's live Actions cache policy and current
cache/artifact inventories, groups cache bytes by key prefix, and summarizes
artifact count, size, age, and largest workflow families. Cache status is
informational at 80% of the configured limit, warning at 90%, and critical at
100%. The report does not assume a fixed artifact byte entitlement and does not
require an organization billing token.
The July 10, 2026 audit found a 10 GB cache maximum and 7-day retention. Target
steady-state usage below 9 GB so GitHub does not continuously evict and recreate
high-reuse entries. Prefer Bun downloads, package-manager data, task-family
Turbo state, native dependencies, shared Rust state, and service-scoped
BuildKit layers over final application binaries. Leave managed uv and CodeQL
caching to their official actions.
All `upload-artifact` steps must set `retention-days` and
`if-no-files-found`. Optional failure diagnostics may warn; release and deploy
handoffs must fail when absent. E2E diagnostics are failure-only and retained 7
days, package tarballs are retained 1 day, development mobile deliverables 7
days, and production store deliverables 14 days.
Preview deploy credentials must not be exposed to arbitrary branch push code.
Most preview Vercel workflows are manual-only: run the workflow from `main`,
provide the branch, tag, or SHA in `preview_ref`, and keep
`TRUSTED_PREVIEW_DEPLOY_ACTORS` limited to maintainers who can approve
secret-backed preview builds. The workflow still checks out and builds the
requested `preview_ref`, so reviewers should treat that ref as code that can
execute during install/build.
Automatic CodeQL uses GitHub's organization-managed
`dynamic/github-code-scanning/codeql` workflow for JavaScript/TypeScript and
Python. `codeql.yml` is a manual-only fallback so the repository retains an
explicit workflow file without duplicating managed push or pull-request scans.
Do not add automatic or cron triggers to the fallback; `bun git-sync` mirrors
the already-scanned `main` commit to `production`.
`e2e-tests.yaml` uses native push paths for E2E specs, Playwright and Docker
configuration, database fixtures, dependency manifests, lockfiles, and its own
runner scripts. Normal application source-only commits do not create the image
bundle and six downstream consumers. E2E has no cron trigger: automatic runs
are limited to matching commits, and maintainers can still use manual dispatch
for an intentional full run.
Supabase migration workflows still consume the required platform workflow-run
signals, but compare the target SHA with the last successful migration marker.
Only database and migration-control changes proceed to Supabase CLI setup and
`db push`; an unavailable marker fails open. Staging and production each use a
single serialized evaluate-and-migrate job, and production preserves the
same-SHA platform deployment plus staging-success prerequisites.
Cloudflare Worker manual deployments follow the same trust boundary: run
`rust-backend.yml` from `main`, keep `TRUSTED_CLOUDFLARE_DEPLOY_ACTORS` limited
to trusted maintainers, and avoid exposing `CLOUDFLARE_API_TOKEN` to unreviewed
branch code. The workflow checks the protected dispatch ref and actor allowlist
before loading Cloudflare credentials or running Worker deploy steps.
`vercel-preview-platform.yaml` also runs on protected `main` pushes. Keep that
exception narrow: `supabase-staging.yaml` is triggered by the platform preview
build workflow and production migration requires a successful `main` staging
migration for the same SHA before it can run `supabase db push --include-all`.
### Other delivery surfaces
* `discord-modal-deploy.yml`
* `mobile-build-android.yaml`
* `mobile-build-ios.yaml`
* `mobile-build-macos.yaml`
* `mobile-build-windows.yaml`
* `mobile-deploy-stores.yaml`
* `release-*.yaml` package publishing workflows
## Operational Rules
### Manual dispatch
Use `workflow_dispatch` when:
* you need to rerun a deployment intentionally
* you need to promote a migration outside the normal trigger timing
* you need to recover from a failed but otherwise understood automation path
Manual dispatch bypasses affected-app gating. Generic switchboard-controlled
workflows still honor disabled `ci` entries, while trusted manual satellite
preview workflows launch their guarded deploy job directly and do not keep an
unused `tuturuuu.ts` toggle.
For preview Vercel deployments, dispatch the workflow from `main`, set
`preview_ref` to the branch, tag, or SHA to deploy, and confirm the actor is in
`TRUSTED_PREVIEW_DEPLOY_ACTORS`. Do not reintroduce preview `push` triggers for
secret-backed Vercel workflows, and do not add a standalone `check-ci` job to a
manual-only preview: the dispatch already selected the build and the extra job
only consumes another runner. Preview concurrency uses the workflow plus
`preview_ref`, so rerunning the same target cancels stale work without canceling
a different preview target.
Package release workflows that expose production secrets or trusted publishing
authority must add their own ref guard before dependency installation or publish
jobs. Release Please is the only workflow that generates monorepo version and
changelog PRs. It runs from `production`, uses
`secrets.RELEASE_PLEASE_TOKEN` so generated PRs and releases can trigger
downstream workflows, and falls back to `github.token` only to keep the job from
failing when the bot token is not provisioned. The `github.token` fallback does
not trigger downstream workflow runs from generated release PRs. Release Please
rejects non-production manual dispatches before the write-capable release job.
Package publish workflows consume release-please version bumps on `production`;
they must not recreate checksum or PR-title version bump automation. Package
release workflows use npm trusted publishing: build and artifact-pack work runs
before any OIDC permission is granted, then the `publish-npm` job downloads one
tarball, verifies its package name and version, and runs `npm publish` without
`NPM_TOKEN`. Each publish job is bound to its package release environment, and
the matching npm trusted publisher must use this repository, the workflow
filename, and that environment. Manual dispatch is only valid with the branch
selector set to `production`; non-production refs are rejected before the package
or publish jobs can start. Every workflow-published package manifest must also
declare `repository.type: "git"`,
`repository.url: "https://github.com/tutur3u/platform"`, and
`repository.directory` matching the package path. npm validates those fields
against GitHub Actions provenance and rejects publishes with `E422` when the
packed manifest has an empty or mismatched repository URL. The tarball handoff
uses required-file behavior, one-day retention, and compression level 0 because
`.tgz` is already compressed. Preparation runs any package lifecycle build
through root Turbo first, then calls `npm pack --ignore-scripts` so `prepack`
cannot bypass the shared cache wrapper.
Release Please can move an oversized generated pull request body into
`release-notes.md` on a companion
`release-please--branches----release-notes` branch. The workflow runs
`node scripts/ci/release-please-overflow-recovery.js --target-branch production`
before `googleapis/release-please-action@v5`; when a merged pending release PR
still points at a missing overflow file, the script recreates the branch and
file from the merged manifest bump and current `production` changelogs, then the
normal Release Please action can create the releases. If the script reports
`skipped`, inspect the pending labels and merged release PR body before retrying.
The same workflow then runs
`node scripts/ci/release-please-auto-approve.js --target-branch production`,
which approves the generated release PR while it is still nothing but generated
output. The `Protected branches` ruleset requires an approving review, so an
untouched release PR otherwise sits at `REVIEW_REQUIRED` indefinitely. Every one
of these must hold or the script skips and leaves the PR for a human: the head
branch is a `release-please--branches--` branch against the target branch, the
PR was opened by a release automation identity, every commit is a
`chore(release): release ...` commit by that same account, and every changed
file is one release-please itself rewrites. That last allowlist is **derived
from `release-please-config.json`** — each package's changelog, its version file
for its release type, and its `extra-files` — so adding a package or changing
its release type cannot silently widen what auto-approval accepts.
Nothing needs to un-approve: the ruleset sets `dismiss_stale_reviews_on_push`,
so any later push to the release branch drops the approval, including one this
workflow never observes. The step deliberately uses `github.token` rather than
`RELEASE_PLEASE_TOKEN`, because GitHub refuses a review from the account that
opened the pull request — release-please and the approval have to be different
identities. While `RELEASE_PLEASE_TOKEN` is unset both are `github-actions[bot]`,
and the script reports `blocked` and leaves the run green rather than failing.
Use `--dry-run` to see the decision without posting a review.
`release-please-auto-merge.yaml` closes the loop on the generated release PR. It
runs every third day at 06:00 UTC (`cron: "0 6 */3 * *"`) and on manual dispatch,
and it does not reimplement the merge in YAML: it runs `bun git-release-please`
and then `bun git-sync`, the same two commands the release flow uses locally, so
the scheduled path and the manual path cannot drift. Those scripts own the merge
rules — platform version sync, conflict detection, `bun check`, and
fast-forward safety — and are covered by `scripts/git-release-please.test.js`
and `scripts/git-sync.test.js`.
The workflow resolves what to do before it installs anything. It prefers
`origin/release-please--branches--production`, ignores the companion
`--release-notes` overflow branch, and skips the merge entirely when the release
branch is already an ancestor of `origin/main` — so a run with nothing to merge
is a clean no-op rather than a red run. It separately checks whether
`origin/production` has drifted from `origin/main` and syncs even when there was
no merge, which is what makes the schedule self-healing. Every run that pushes
ends by asserting the two refs resolve to the same commit and fails loudly if
they do not.
Day-of-month cron stepping restarts each month, so the 31st and the 1st can land
a day apart. That is harmless here precisely because an empty run does nothing.
The job checks out `main` with `fetch-depth: 0` and
`secrets.RELEASE_PLEASE_TOKEN`, falling back to `github.token`. The fallback
cannot complete a release: the `Protected branches` ruleset covers `main` and
`production` and its only bypass actor is `OrganizationAdmin`, so a run on
`github.token` merges cleanly and then dies on
`GH013: Changes must be made through a pull request`. `RELEASE_PLEASE_TOKEN`
must therefore hold a personal access token (`repo` + `workflow` scopes) owned
by an organization admin. Granting the Actions app a ruleset bypass is **not**
an alternative: pushes made with `GITHUB_TOKEN` do not trigger `push`
workflows, so the `production` push would silently skip
`Vercel Production Deployment Planner` and the next `Release Please` run and
stall the release chain. A run that would need to push checks for the secret up
front and fails with an actionable error, rather than discovering it after
`bun check`.
The job also installs Flutter (`3.44.x`, the same pin as `mobile.yaml`) and runs
`flutter pub get` in `apps/mobile` before the merge. Release Please bumps
`apps/mobile/pubspec.yaml` on every release, so `touchesMobile()` in
`scripts/git-release-please.js` is always true and `bun check` is always
followed by `bun check:mobile`; without a toolchain `dart-format`,
`flutter-analyze` and `flutter-test` all exit `127`.
Top-level `permissions` is `{}` and
`contents: write` is granted on the job alone, and `concurrency` is grouped with
`cancel-in-progress: false` because cancelling a half-finished release merge
would leave `main` and `production` split. Use the `dry_run` dispatch input to
exercise the merge and run `bun git-sync --no-push` without publishing anything.
Keep internal Tuturuuu dependencies as `workspace:*` in source package
manifests so local builds always use the checked-out workspace. Package release
workflows must first run
`node scripts/ci/package-release-readiness.js gate-package-release packages/`
in a short pre-build gate. The gate checks the package version and publishable
Tuturuuu workspace dependencies once, dispatches missing dependency workflows,
and exits green without build, pack, or publish work while dependencies are
pending. If a dependency workflow for the same production SHA already failed,
completed successfully without npm visibility, or cannot be inspected, the gate
fails immediately. When the gate outputs `should_publish == true` and
`dependencies_ready == true`, the workflow can build and rewrite the checked-out
manifest immediately before `npm pack` with
`node scripts/ci/prepare-npm-package-manifest.js packages/`. That
temporary rewrite replaces `workspace:` protocol ranges with the current
workspace package versions so npm consumers can install the tarball. It must
preserve package-included `file:` tarball dependencies, such as `@tuturuuu/ui`'s
vendored SheetJS tarball, instead of rewriting them to mutable external tarball
URLs. After
`npm publish`, the publish job polls `npm view` for the exact published version
before it reports success, then a separate non-OIDC job dispatches direct
dependent package workflows without checking out the repo or carrying publish
authority. The internal
`node scripts/ci/package-release-readiness.js dispatch-dependent-workflows packages/`
command is available for direct/manual dispatches that can read the checkout.
If npm returns first-publish or permission errors, fix the npm package access and
trusted publisher setup instead of skipping the package. If a published package
depends on another release-please-managed Tuturuuu package, that dependency also
needs its own
`release-*-package.yaml` workflow and matching `tuturuuu.ts` entry.
The platform production Vercel workflow also runs
`node scripts/ci/package-release-readiness.js gate-changed-package-versions`
before installing dependencies. Release-please package bumps therefore publish
and become visible on npm before the normal production platform deployment
continues. The helper intentionally checks only the checked-out latest commit,
not the whole push event payload, so package changes from earlier commits in a
multi-commit push are not bound to an unrelated newer SHA. If the latest commit
does change package manifests, the deploy job grants `actions: write` because
the helper may dispatch missing package release workflows for that same
production SHA. When related package releases are queued or running, the helper
sets `packages_ready=false`; the platform build skips successfully instead of
polling npm until the runner times out. npm publish authority remains isolated to
package `publish-npm` jobs. A skipped package gate must not count as production
deployment for database migrations. Database paths therefore select the
platform deployment in the production planner, and `supabase-production.yaml`
requires both the successful planner run and the `vercel-production-platform`
deployment marker for the same SHA before it can run production migrations.
The gate queries the planner run because reusable app workflows do not create
standalone Actions runs.
Filtered production Docker installs only include the selected app and the
dependencies that app needs. Workspace packages with direct `tsc` build scripts
must declare `typescript` in their own `devDependencies`; do not
rely on the root devDependency for production Docker builds. Programmatic
compiler API consumers must stay on the active TypeScript 7 toolchain instead
of carrying legacy compiler compatibility packages.
Next.js production apps also declare `@typescript/native-preview` in their own
`devDependencies` while the repo uses the TypeScript 7 native compiler package.
The TS7 `typescript` package does not ship the legacy
`typescript/lib/typescript.js` API file that Next checks during build-time
TypeScript setup; the native-preview marker makes Next use its supported TS7
native compiler path instead of trying to auto-install classic TypeScript.
### Superseded-run guards
Production Vercel workflows use a static per-app concurrency prefix and Git ref
with a `cancel-in-progress` predicate restricted to push events on
`refs/heads/production`. A newer production push cancels only the older run for
that same app instead of cancelling sibling app deployments or allowing an
obsolete run to finish green. A `main` commit or manual recovery dispatch does
not cancel the running production deploy. The newest production run stays queued
or active and deploys the cumulative branch state.
Deployment markers still let affected-app gating compare every change since
the last successful deployment, including changes accumulated across canceled
intermediate runs. If the marker is missing or cannot be trusted, Vercel gating
defaults open so a later unrelated commit cannot hide an earlier app change.
### Prerequisite guards
* `supabase-staging.yaml` requires a successful `main` platform preview build
trigger unless manually dispatched.
* Its deploy step links the staging project and runs
`supabase db push --include-all`.
* `supabase-production.yaml` runs after the production deployment planner and
also re-evaluates after a matching `main` staging migration succeeds. It still
checks the latest planner result, a successful production platform deployment
marker, and a completed successful staging migration for the same commit
before running
`supabase db push --include-all`.
### Mobile iOS native assets
Mobile workflows cache the Flutter SDK/pub downloads through the Flutter setup
action. Android workflows also use the toolchain-aware Gradle cache, and iOS or
macOS workflows cache CocoaPods downloads plus the trunk repository. They do
not cache `apps/mobile/build`, final APK/AAB/IPA/app bundles, `Pods`, or
`apps/mobile/.dart_tool`. Flutter native asset hook outputs can become stale
across simulator/device builds and SDK updates; a stale `.dart_tool` cache can
leave `NativeAssetsManifest.json` referencing `objective_c` while
`build/native_assets/ios/` is missing the generated framework.
Every workflow still runs `flutter pub get` and its platform build on every run.
Development deliverables are retained for 7 days and production store
deliverables for 14 days. Archive directory-based app bundles before upload,
disable redundant artifact compression for already-compressed archives and
mobile packages, and fail if a required deliverable is missing.
The workflow explicitly runs `flutter config --no-enable-swift-package-manager`
after Flutter setup. Keep the iOS CI path on CocoaPods while `image_cropper` and
`dkimagepickercontroller` resolve incompatible `TOCropViewController` Swift
package ranges; otherwise Flutter 3.44+ can fail before the simulator build
with an Xcode package dependency resolution error.
The iOS/macOS build workflows currently resolve `macos-latest` to GitHub's
macOS 26 ARM64 image with Xcode 26+. The former Xcode 16.4 compatibility pins
for `connectivity_plus` and `device_info_plus` are no longer required. Before
adding a future Apple-specific dependency pin, verify the active runner image
from a build log and reproduce the native build failure on that toolchain.
### Mobile store beta deployment
`mobile-deploy-stores.yaml` runs automatically on `production` pushes that touch
`apps/mobile/**`, `scripts/mobile-deployment/**`, the workflow file,
`ci-check.yml`, or `tuturuuu.ts`. It is a beta-store workflow only:
* Android builds the production flavor AAB from `apps/mobile/.env.github` and
publishes it to the Google Play `internal` track.
* iOS builds the production flavor IPA from `apps/mobile/.env.github` and
uploads it to TestFlight.
* CI requests a signed GitHub OIDC token for the
`tuturuuu-mobile-deployment` audience, then fetches an Android or iOS bundle
from `https://tuturuuu.com/api/v1/mobile-deployment/bundle` with the issued
`MOBILE_DEPLOYMENT_CI_TOKEN`.
* If `MOBILE_DEPLOYMENT_CI_TOKEN` is not configured in the `mobile-store-beta`
GitHub Environment yet, the credentials preflight emits a notice and skips the
Android and iOS publish jobs instead of failing the workflow. A non-empty but
invalid token still fails during the bundle fetch.
* The apps/web mobile deployment vault verifies the CI token, GitHub OIDC
issuer, audience, repository, `production` ref, workflow file, and
`mobile-store-beta` environment before returning any plaintext.
* CI hydrates ignored Firebase, signing, store, and `.env.github` files into
fixed paths under `apps/mobile` and `$RUNNER_TEMP` with `umask 077`, validates
plaintext SHA-256 hashes, masks secret values, and deletes generated files in
`always()` cleanup steps.
* The workflow rejects any Google Play track other than `internal`, uploads only
the AAB/IPA build artifacts, and uses TestFlight-only iOS upload.
* Release Please owns mobile version and build-number bumps. Duplicate store
build numbers should fail in CI instead of being bumped by this workflow.
The workflow is bound to the `mobile-store-beta` GitHub Environment. Keep that
environment restricted to the `production` branch and store only
`MOBILE_DEPLOYMENT_CI_TOKEN` there. Mobile signing, Firebase, store API, and
build-time secrets live in the root workspace mobile deployment vault at
`/internal/mobile-deployment` on the infra app and require
`manage_mobile_deployment_vault`. Manage non-file values in the vault's
**Secrets** panel, then use the vault's Verify action before activating the draft. Legacy
base64 file payload env keys are intentionally blocked; Firebase, signing, and
store credential payloads must be uploaded as file resources.
Local release-build verification should run before relying on the workflow:
```sh theme={null}
bun test scripts/ci/check-workflow-config.test.js
ruby -e "require 'yaml'; YAML.load_file('.github/workflows/mobile-deploy-stores.yaml')"
git diff --check
bun check:mobile
(cd apps/mobile && flutter pub get)
(cd apps/mobile && flutter build appbundle --release --flavor production --target lib/main_production.dart --dart-define-from-file=.env.github)
```
iOS IPA verification requires full Xcode, not only Command Line Tools:
```sh theme={null}
xcodebuild -version
(cd apps/mobile && flutter config --no-enable-swift-package-manager)
(cd apps/mobile && flutter build ipa --release --flavor production --target lib/main_production.dart --dart-define-from-file=.env.github)
```
If `xcodebuild -version` reports that Command Line Tools are selected, switch to
full Xcode before claiming the iOS archive has been verified locally.
## Fast Triage Checklist
1. Check whether `ci-check.yml` disabled the workflow through `tuturuuu.ts`.
2. For Vercel workflows, inspect the `ci-check.yml` decision reason and matched paths.
3. Inspect the changed-file source, base SHA, head SHA, and path count emitted by
`resolve-changed-files.ts`.
4. Check whether the workflow was canceled because a newer run superseded it,
then follow the newest run in the same workflow-and-branch concurrency group.
5. Check whether the prior successful GitHub Deployment marker exists for that
Vercel workflow and branch.
6. Check path filters to confirm GitHub should have started the lightweight workflow.
7. For database workflows, inspect the prerequisite evaluation job before the deploy job.
8. For Docker changes, verify `docker-setup-check.yaml` specifically.
## When You Add New Automation
1. Add the workflow file under `.github/workflows/`.
2. Add its key to `tuturuuu.ts` unless you intentionally want it always on by default.
3. Document the workflow in `apps/docs`.
4. If it changes a docs-visible page, add that page to `apps/docs/docs.json`.
# Hive Realtime Runbook
Source: https://docs.tuturuuu.com/build/devops/hive-realtime-runbook
Operating notes for the Hive WebSocket service and satellite app deployment.
Hive realtime lives in `apps/hive-realtime`. It is a Bun WebSocket service used
only by Hive; do not replace it with Supabase Realtime. The shared protocol and
Yjs helpers live in `@tuturuuu/realtime/hive` so `apps/hive`,
`apps/hive-realtime`, and `apps/web` use the same message schemas.
## Required Environment
* `HIVE_REALTIME_TOKEN_SECRET`: preferred shared HMAC secret used by `apps/web`
to mint short-lived join tokens and by `apps/hive-realtime` to validate them.
If this is not set, both services fall back to the existing platform
Supabase service secret from the shared `apps/web` environment. This keeps
Hive deployable with the same Supabase setup as `apps/web`, while still
allowing operators to rotate Hive realtime signing independently later.
* `HIVE_REALTIME_URL`: internal server URL used by `apps/web` and `apps/hive`
when they need the service-side endpoint.
* `NEXT_PUBLIC_HIVE_REALTIME_URL`: browser-facing WebSocket URL, normally
`wss://hive.tuturuuu.com/realtime` in production.
* `HIVE_DATABASE_URL`: Postgres URL for the dedicated Hive product database.
Realtime uses this database to load compacted CRDT snapshots, append Yjs
update bytes, and persist audit events. Supabase is not the Hive product store
after backfill.
* `INTERNAL_WEB_API_ORIGIN`: Docker-internal web gateway origin used by
`apps/hive` API rewrites. In production Compose this should resolve to
`http://web-proxy:7803` so editor saves do not leave the Docker network and
re-enter through the public web origin.
* Supabase credentials already used by `apps/web`, especially
`NEXT_PUBLIC_SUPABASE_URL` with `SUPABASE_SECRET_KEY`. `SUPABASE_SERVER_URL`
may still override the server-side URL when the deployment needs a
Docker-internal Supabase origin. These credentials are identity/session
infrastructure only for Hive realtime token validation and web auth.
`apps/hive` shares the same Supabase setup as `apps/web`. Docker runtime env
comes from `.env.local` with the same `apps/web/.env.local` fallback, and the
production Hive image receives the `web_env` BuildKit secret during `next build`
so prerendered auth routes can resolve `NEXT_PUBLIC_SUPABASE_URL` and related
platform Supabase variables.
When Hive is exposed as `https://hive.tuturuuu.com` through a Cloudflare tunnel
to `http://localhost:7814`, keep browser-facing realtime URLs secure. Use
`NEXT_PUBLIC_HIVE_REALTIME_URL=wss://hive.tuturuuu.com/realtime` or leave it
unset so the client falls back to the same-origin `/realtime` route. Keep
Docker-internal or host-local forwarding in `HIVE_REALTIME_HTTP_URL` or
`HIVE_REALTIME_URL`; do not expose `ws://...` as the browser URL for an HTTPS
page.
## Local production-style servers (Turbo)
From the repo root, Turbo runs workspace dependency builds first, then starts
each process (see `turbo.json` tasks `@tuturuuu/hive#serve:hive` and
`@tuturuuu/hive-realtime#serve:hive-realtime`):
* `bun serve:hive` — production Next server for Hive on port `7814` (requires a
prior `next build` output; Turbo schedules `build` before `serve:hive`).
* `bun serve:hive-realtime` — Bun WebSocket server (default port `7815` via
`PORT`).
## Blue/Green Deployment
`docker-compose.web.prod.yml` runs `hive-blue`, `hive-green`, and
`hive-realtime` with the same production Supabase environment as `apps/web`.
`scripts/docker-web/blue-green.js` promotes Hive with the same active color as
web and waits for the target Hive color before proxy handoff.
The generated nginx config routes:
* `hive.tuturuuu.com/` to the active `hive-{color}` satellite app on port
`7814`.
* `hive.tuturuuu.com/realtime` to `hive-realtime` on port `7815`.
The production proxy also listens on host port `7814`, so a Cloudflare tunnel
mapping `hive.tuturuuu.com` to `localhost:7814` still goes through the
blue/green proxy instead of bypassing it to a stale single Hive container.
`scripts/watch-blue-green-deploy.js` watches the Hive Dockerfiles, package
manifests, realtime source files, and `docker-compose.web.prod.yml`; changes to
these files trigger the watcher container refresh path.
## Protocol
Clients connect with a short-lived token from
`POST /api/v1/hive/servers/:serverId/realtime-token`. Tokens include user ID,
server ID, role, expiry, and event scopes.
Accepted client messages are CRDT sync, awareness, compatibility, and health
messages:
* `sync.hello`
* `sync.update`
* `sync.diff`
* `sync.compacted`
* `awareness.update`
* `presence`
* `server.status`
* `error`
* `world.event`
* `world.event.applied`
The durable shared world is a Yjs document per server. Clients send encoded Yjs
deltas as base64 in `sync.update`, and the realtime service appends those bytes
to `hive_crdt_updates`. State-vector sync lets a reconnecting client request
only missing updates, while periodic compaction stores merged snapshots in
`hive_world_states.crdt_snapshot`.
`revision` is a compatibility display value backed by `op_seq`; it is not a
write precondition for commutative CRDT world edits. Money, item ownership,
warehouses, trades, LLM spend, and bankruptcy still go through the `apps/web`
Hive APIs and Hive Postgres transactions. Realtime broadcasts only the committed
visual projections for those authoritative writes.
`world.event` and `world.event.applied` remain as temporary compatibility
wrappers for older full-world edit flows. The service converts accepted payloads
into CRDT updates before broadcasting them to the room.
Do not store full `world_data` snapshots in `hive_world_events` for compatibility
events. The current world snapshot belongs in `hive_world_states`; event rows
should keep compact metadata and payloads so high-frequency editor saves do not
double the amount of JSON written per operation.
Awareness is ephemeral and TTL-based. It includes user ID, display name, avatar
URL, role, color, active tool, selected entity, terrain cursor, camera position,
world avatar position, viewport focus, and `lastSeenAt`. Do not persist
awareness state into `hive_world_states`, `hive_crdt_updates`, or audit events.
Operationally, the client should expect token refresh on reconnect,
exponential backoff, heartbeat/TTL presence cleanup, actor echo suppression,
offline edit queue replay, cursor/position throttling, and slow-client
backpressure handling.
# Internal Log Drain and Observability
Source: https://docs.tuturuuu.com/build/devops/log-drain-observability
Operate the Postgres-backed log drain that powers web runtime logs, requests, deployments, analytics, cron, and observability views.
## Overview
`apps/web` stores request, cron, deployment, and runtime-monitoring events in the internal log drain where those wrappers are still wired. Application runtime logging uses the native console method that matches severity (`console.error`, `console.warn`, `console.info`, `console.log`, or `console.debug`); the drain preserves normal stdout/stderr output for deployment and monitoring views instead of requiring a custom app logger.
The monitoring navigation uses compact labels:
* Overview
* Deployments
* Logs
* Analytics
* Observability
* Cron
* Requests
* Resources
* Projects
* Stress Tests
## Runtime
Docker web runs a dedicated `log-drain-postgres` service. It is separate from Supabase and is owned by the Docker web runtime. Compose pins the visible container name to `${COMPOSE_PROJECT_NAME:-tuturuuu}-log-drain-postgres-1` so it stays grouped with the rest of the Tuturuuu stack in Docker Desktop.
Default connection inside Docker:
```txt theme={null}
postgres://platform_log_drain:platform_log_drain@log-drain-postgres:5432/platform_log_drain
```
Relevant environment variables:
* `PLATFORM_LOG_DRAIN_DATABASE_URL`: Postgres connection string.
* `PLATFORM_LOG_DRAIN_ENABLED`: set to `false` to disable persistence while preserving stdout/stderr.
* `PLATFORM_LOG_DRAIN_RAW_RETENTION_DAYS`: raw log retention, default `30`.
* `PLATFORM_LOG_DRAIN_SUMMARY_RETENTION_DAYS`: request, cron, deployment, and usage retention, default `90`.
## Logging Rules
Server runtime code should use the native console method that matches diagnostic severity:
```ts theme={null}
console.info('Processed job', { jobId });
console.error('Job failed', error);
```
Do not add `serverLogger` runtime imports or automatic console log-drain
installation. Prefer a single console call with a short message and a structured
metadata object so Docker, Vercel, and any retained drain views can keep the
event readable.
For route or cron handlers, wrap execution with `withRequestLogDrain(...)` or `withCronLogDrain(...)` when the handler should attach logs to a request or cron run id.
The drain is fail-open: if Postgres is unavailable, requests and cron jobs continue normally and logs still go to stdout/stderr.
## Legacy Compatibility
The monitoring UI reads both the Postgres drain and the pre-drain blue/green
files under `tmp/docker-web`. Before the Postgres log drain existed, the
blue/green watcher captured proxy traffic, watcher logs, and request console
lines as plain files. The Requests and Logs tabs still merge those legacy files
in, and Deployments are enriched from the watcher snapshot so commit subject,
full hash, short hash, stamp, active color, and runtime lane remain visible even
before the Postgres drain has a complete history.
Legacy Docker console capture coalesces timestamped continuation lines from the
same container write before persisting request console logs. This keeps object
dumps from old route console output attached to one function/request row instead
of turning every serialized property into a separate log row.
Request-archive console lenses store and return redacted summaries only. Both the
watcher ingestion path and the web-side archive normalization redact common
bearer tokens, JWTs, sensitive key/value fields, sensitive query parameters, and
email addresses, then cap each attached console message at 500 characters. Raw
app-container console payloads are never persisted in `blue-green-request-logs`.
## Projects
Infrastructure → Monitoring → Projects is the log-drain-owned project registry for Docker web deployments. The built-in project is seeded as:
* `id`: `platform`
* repository: `https://github.com/tutur3u/platform`
* default selected branch: `production`
* app root: `apps/web`
* environment: `production`
* addons: nginx proxy locked on, log drain on, Redis on, cron on
Project definitions live in the log-drain Postgres tables `infrastructure_projects` and `infrastructure_project_branches`, not Supabase. Existing log, request, deployment, cron, and resource records default to `project_id = 'platform'` so the current monitoring views keep showing the deployed platform while new projects are introduced.
V1 GitHub import supports public GitHub repositories only. The UI syncs repository metadata and branches through unauthenticated GitHub REST calls by default; set `GITHUB_TOKEN` only when the deployment host needs a higher GitHub API rate limit. New projects use the Next.js preset, default app root to the repository root, require hostname routing through the central nginx proxy, and start with log drain plus Redis enabled. Cron is disabled by default for imported projects until an operator enables it.
Imported projects can be deleted from the Projects tab. Deletion removes the log-drain project registry entry and its branch cache, but it does not purge retained request, log, deployment, cron, or resource records. The built-in `platform` project is locked and cannot be deleted from the UI or API.
Changing the selected branch queues the project for deployment. For the built-in `platform` project, the blue/green watcher reads the selected branch from the project registry. If the selected branch differs from the current checkout and the worktree is clean, the watcher fetches and checks out that branch before continuing. If the worktree is dirty, the project is marked blocked in watcher status instead of crashing or forcing a checkout. Manual deploy actions also write `deployment_status = 'queued'`; the watcher must consume that queued status whether Git is already up to date or the watcher first fast-forwards to a newer commit, then advance it through `building`, `deploying`, and `ready` or `failed`. Successful platform deploys must update the project row's latest commit fields at the same time as deployment history so the Projects tab cannot remain queued with an older commit after the runtime has advanced.
The project list also reconciles stale queued state for the built-in project from the blue/green runtime snapshot. If the active or latest successful deployment matches the queued commit, or was completed after the queue timestamp, the API clears `queued`, marks the project `ready`, and refreshes the latest commit fields from deployment history.
For imported projects, the same watcher polls enabled projects from the registry, keeps managed source checkouts under `tmp/docker-web/projects//repo`, and writes generated Next.js compose/runtime files under `tmp/docker-web/projects//runtime`. These services run with the shared `tuturuuu` Compose project name, attach to the central Docker network, and receive `PLATFORM_PROJECT_ID`, selected branch, log-drain, Redis, `PORT`, and `HOSTNAME=0.0.0.0` environment wiring. Hostname routes are merged into the central nginx proxy config so nginx remains the mandatory entrypoint. Managed project hostnames must be plain DNS names; the API rejects ports, wildcards, whitespace, nginx syntax, and reserved Tuturuuu platform hostnames, and the watcher defensively drops unsafe persisted values before rendering nginx.
The Logs tab serves grouped rows. Events with the same request id collapse into
one expandable row; legacy standalone logs fall back to route, source,
deployment, and minute bucket grouping. Operators can filter by route, status
family, source, level, request id, and deployment stamp, then expand a row to
see the child event timeline, metadata, error stack, client context, and
deployment context.
Legacy Docker console capture coalesces timestamped continuation lines from the
same container write before persisting request console logs. This keeps object
dumps from old route console output attached to one function/request row instead
of turning every serialized property into a separate log row.
Deployment rows keep historical failed attempts searchable, but failed and
successful attempts for the same commit are not merged into one current failed
state. The global deployment failure banner and current blocked-target summary
come only from the latest active deployment row; older failures remain visible
inside deployment history rows and filters.
The Requests tab intentionally freezes its result window when opened. New traffic is counted in the background and offered as an explicit "show new" action, while older pages are appended automatically as the operator scrolls. This prevents live traffic from shifting the visible rows during investigation. The frozen `since` and `until` cursor bounds must be applied by the Postgres log-drain query or legacy archive reader before any `ORDER BY ... LIMIT`, archive page size, or aggregate row cap so request floods after the freeze cannot evict older rows still inside the operator's investigation window.
Monitoring links preserve the selected `project` query parameter so Overview, Deployments, Logs, Analytics, Observability, Cron, Requests, and Resources stay scoped to the same project while an operator moves between tabs. The project scope card on each tab is the visible source of truth for the active project, branch, hostnames, commit, and addons.
Queued deployment rows only advance while the blue/green deployment watcher is live and connected to log-drain Postgres. If the Projects tab sees a queued project while the watcher is missing, offline, stale, or locked to a different branch than the built-in project, the UI queues a `blue-green-watcher-recovery.request.json` control request. The Docker cron runner reads that request from the shared control directory, clears stale watcher lock/status files, and recreates the `web-blue-green-watcher` service through Docker Compose so recovery does not depend on the stuck watcher process. The production watcher service must receive `PLATFORM_LOG_DRAIN_DATABASE_URL`, `DOCKER_WEB_FRONTEND`, optional `GITHUB_TOKEN`, and any addon env such as Redis from Compose so queued recovery recreates the same frontend family that production is serving.
Each drained request stores the client IP address and user agent when those headers are available. Console logs emitted inside an already-wrapped request or cron AsyncLocalStorage context can be scoped back to the same request id, so Requests can show related console lines next to the originating request without relying on terminal access.
## Cron Control
Infrastructure → Monitoring → Cron exposes the native Docker cron runner state, a global enable/disable switch, per-job enable/disable switches, manual run buttons, retained execution rows, and captured response/console output. Runtime overrides are stored in `tmp/docker-web/watch/control/cron-control.json`; they do not edit `apps/web/cron.config.json`, so Vercel cron config remains source-controlled while local Docker operations can pause individual jobs safely.
Cron snapshot and execution history reads are available to infrastructure
viewers, but cron mutations are operator-only. Manual run requests and global or
per-job runtime enablement changes must go through routes guarded by
`authorizeInfrastructureOperator` so roles with only `view_infrastructure`
cannot change scheduler behavior.
Cron log-drain wrappers must not persist unauthorized attempts. Keep
`withCronLogDrain` configured so 401 and 403 cron responses return to callers
without writing `requests`, `cron_runs`, or buffered `log_events`; attackers must
not be able to fill observability storage with unauthenticated cron probes.
Request-archive console lenses must store and return redacted summaries only.
Watcher ingestion and web-side archive normalization both redact common bearer
tokens, JWTs, sensitive key/value fields, sensitive query parameters, and email
addresses, then cap each attached console message at 500 characters. Do not
persist raw app-container console payloads in `blue-green-request-logs`.
Cron jobs should show both the raw expression and a natural description such as "Every 15 minutes", plus the previous and next scheduled run timestamps when the runtime snapshot provides them. Cron expressions are stored as runtime config, while visible daily schedule descriptions and run timestamps are rendered in the viewer's browser timezone.
The `infrastructure-sample-resources` job runs every minute through `/api/cron/infrastructure/sample-resources`. It is the automated source for retained resource charts; do not rely on opening the Resources page to create samples.
## Resources
Infrastructure → Monitoring → Resources reads the Docker runtime snapshot and displays container health, image/service identity, uptime, CPU, memory, compact network ingress/egress, and aggregate service counts. This is the operator view for Docker Desktop resource pressure without opening Docker Desktop directly.
The same tab also separates Docker build consumption from the generic container
inventory. Build resources are derived from BuildKit and builder containers
captured by `docker stats`; while a watcher deployment is `building` or
`deploying`, the `web-blue-green-watcher` container is also counted as builder
process pressure because it owns the Docker build command. This matters because
Docker Desktop can show an active Buildx record before the watcher captures a
fresh stats sample, and the Resources tab should still show that a build process
is active instead of claiming the build lane is idle. The sampled resource
metrics are charted as `docker.build.*` usage events so operators can
distinguish build pressure from runtime web, proxy, Redis, and sidecar pressure.
The internal resource sampler writes the current Docker snapshot into the log drain `usage_events` table at most once per minute. The UI charts that history across the supported resource windows: 1 hour, 6 hours, 12 hours, 24 hours, 3 days, and 7 days. If the log drain is disabled or unavailable, the tab still shows the live Docker snapshot and falls back to a single current sample.
Chart gaps on the Resources tab mean a retained sampler record is missing for that time bucket; they are not automatically downtime. The Resources tab now shows sampling continuity for runtime and build metrics, including sampled buckets, gap buckets, latest retained sample age, and a live-snapshot marker for the current bucket. If the live snapshot is healthy but historical charts have gaps, inspect the Cron tab and the `infrastructure-sample-resources` job before treating the gap as runtime unavailability.
Resource pressure uses the same thresholds as the dashboard: memory under 200 MB is green, 200-500 MB is amber, 500-1024 MB is orange, and anything above 1024 MB is red. CPU under 5% is green, 5-20% is amber, 20-40% is orange, and anything above 40% is red.
## Stress Tests
Infrastructure → Monitoring → Stress Tests queues controlled native load runs for root workspace infrastructure operators. The web app only validates permissions, writes a control request, and reads status. It must not generate load inside a Next.js route handler.
Allowed targets come from `PLATFORM_STRESS_TEST_TARGETS`, a JSON array of objects with `id`, `label`, `baseUrl`, optional `defaultPath`, and optional `description`. If the variable is absent, the dashboard exposes only the local web target at `http://127.0.0.1:7803`. Do not accept arbitrary operator-entered URLs; add production, staging, or canary targets explicitly to this allowlist.
Run the native worker with:
```bash theme={null}
node scripts/watch-stress-tests.js
```
The web container writes queued run and abort request files only under `PLATFORM_STRESS_TEST_CONTROL_DIR` or the default `tmp/docker-web/watch/control/stress-tests`, which should resolve to the writable blue/green control mount in production. The native worker consumes those control files, writes live run state under `PLATFORM_STRESS_TEST_MONITORING_DIR` or `tmp/docker-web/stress-tests`, and tags synthetic requests with `X-Tuturuuu-Stress-Test-Run` plus a stress-test user agent. Keep the runtime/monitoring tree read-only for web containers; run directories, samples, and result files are worker-owned.
Durable run summaries are stored in private Supabase tables and read only through `/api/v1/infrastructure/monitoring/stress-tests`. Completed runtime files are synced back to those private tables when the dashboard/API reads them, so rollout order stays fail-open if the migration or log-drain database is not available yet.
Per-run resource samples are also copied into log-drain `usage_events` with `stress.*` metrics and `metadata.runId`. This lets operators compare RPS, latency, CPU, memory, and network spikes against the existing Requests, Logs, Deployments, and Resources tabs for the same run window.
## Troubleshooting
If the UI is empty:
1. Confirm `log-drain-postgres` is healthy with Docker Compose.
2. Confirm `PLATFORM_LOG_DRAIN_DATABASE_URL` is present in the web container.
3. Check that the route or cron handler logs with the native console method matching severity and still runs inside a request or cron wrapper when retained correlation is required.
4. Use Infrastructure → Monitoring → Logs to search by message, route, request id, level, or source.
If `bun serve:web:docker:bg` fails with
`dependency failed to start: container ...-log-drain-postgres-1 exited (1)`,
the deploy helper now starts `log-drain-postgres` before promoting the web lane
and retries once after removing only the exited service container. The
`platform-log-drain-postgres` volume is intentionally left intact. Web and
watcher services do not declare Compose `depends_on` for this optional
Postgres service, so a failed retry is not reintroduced as a later dependency
startup failure. In current deploys, a failed retry is not a hard promotion
blocker unless
`DOCKER_WEB_LOG_DRAIN_REQUIRED=1` is set; the rollout continues for that run with
`PLATFORM_LOG_DRAIN_ENABLED=false`, so traffic can serve while persisted
request/server logs and Infrastructure monitoring are degraded. Inspect the
Postgres logs before deciding whether the volume needs migration or a reset:
```bash theme={null}
docker compose -f docker-compose.web.prod.yml --profile redis ps --all log-drain-postgres
docker compose -f docker-compose.web.prod.yml --profile redis logs --tail 200 log-drain-postgres
docker volume ls --filter label=com.docker.compose.volume=platform-log-drain-postgres
```
When the logs mention incompatible database files, a corrupted data directory,
or another persistent storage problem, back up or migrate the
`platform-log-drain-postgres` volume before retrying. Do not remove that volume
unless losing local observability history is an explicit operator decision.
Set `DOCKER_WEB_LOG_DRAIN_REQUIRED=1` only when blocking a rollout is preferable
to serving without persisted log-drain telemetry.
If logs are missing only for cron jobs, verify the cron route is wrapped with `withCronLogDrain(...)` and the cron runner is calling the expected web origin.
# Mobile Store Deployment Vault
Source: https://docs.tuturuuu.com/build/devops/mobile-store-deployment
What every Mobile Deployment vault field is, what it does in the store release workflow, and where to obtain its value.
This runbook explains each field on the **Infrastructure → Mobile Deployment**
settings page (root workspace, `manage_mobile_deployment_vault` permission). The
vault holds the secrets, signing files, and CI tokens consumed by the
`.github/workflows/mobile-deploy-stores.yaml` workflow that builds and publishes
the Flutter mobile app to the Google Play and Apple App stores.
Each field on the settings page has an inline help tooltip with a short summary
and the console URL; this page is the full reference with clickable links.
## How The Vault Is Organized
* **Secrets** — short scalar values (passwords, IDs, names) and preset
environment variables baked into the build.
* **Files** — signing certificates, keystores, and Firebase/Play config files.
* **CI tokens** — bearer tokens the GitHub Actions workflow uses to read the
vault. Each token value is shown only once, at issue time.
* **Overview** — readiness checks plus draft activation and rollback.
## Android Signing
| Field | What it is | Where to get it |
| --------------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ANDROID_KEYSTORE_ALIAS` | Alias of the signing key inside the upload keystore. | Chosen when you run [`keytool -genkeypair -alias `](https://developer.android.com/studio/publish/app-signing). |
| `ANDROID_KEYSTORE_PASSWORD` | Password protecting the keystore file. | Set when generating the keystore with `keytool`. |
| `ANDROID_KEYSTORE_PRIVATE_KEY_PASSWORD` | Password protecting the private key entry (often the same as the keystore password). | Set when generating the keystore with `keytool`. |
| `android_upload_keystore` (file) | The Java keystore (`.jks`) used to sign Android release builds. | Generate with [`keytool -genkeypair -v -keystore upload-keystore.jks ...`](https://developer.android.com/studio/publish/app-signing#generate-key). |
## Google Play
| Field | What it is | Where to get it |
| ----------------------------------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GOOGLE_PLAY_PACKAGE_NAME` | Play application ID (e.g. `com.tuturuuu.app.mobile`); must match the Android `applicationId`. | [Google Play Console](https://play.google.com/console) → App information. |
| `GOOGLE_PLAY_TRACK` | Release track (`internal`, `alpha`, `beta`, `production`). | [Play release tracks](https://support.google.com/googleplay/android-developer/answer/9859348). |
| `google_play_service_account_json` (file) | Service-account JSON with Play Developer API access for automated publishing. | [Google Cloud Console](https://console.cloud.google.com/iam-admin/serviceaccounts) → create JSON key, then grant access in Play Console → Users and permissions. See the [publisher API setup](https://developers.google.com/android-publisher/getting_started). |
## Apple Signing
| Field | What it is | Where to get it |
| --------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `APPLE_BUNDLE_ID` | iOS bundle identifier (e.g. `com.tuturuuu.app.mobile`). | [Apple Developer → Identifiers](https://developer.apple.com/account/resources/identifiers/list). |
| `APPLE_TEAM_ID` | 10-character Apple Developer Team ID. | [Apple Developer → Membership](https://developer.apple.com/account). |
| `APPLE_DISTRIBUTION_CERTIFICATE_PASSWORD` | Password used when exporting the distribution certificate `.p12`. | Chosen by you during Keychain Access export. |
| `apple_distribution_certificate_p12` (file) | Apple distribution certificate (with private key) exported as `.p12`. | [Apple Developer → Certificates](https://developer.apple.com/account/resources/certificates/list); export from Keychain Access. |
| `apple_app_store_provisioning_profile` (file) | App Store provisioning profile (`.mobileprovision`) tying the bundle ID to the certificate. | [Apple Developer → Profiles](https://developer.apple.com/account/resources/profiles/list). |
## App Store Connect API
| Field | What it is | Where to get it |
| ----------------------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| `APP_STORE_CONNECT_API_KEY_ID` | Key ID of the App Store Connect API key. | [App Store Connect → Integrations](https://appstoreconnect.apple.com/access/integrations/api). |
| `APP_STORE_CONNECT_ISSUER_ID` | Issuer ID that pairs with the API key. | Same Integrations page (shown above the keys list). |
| `app_store_connect_private_key_p8` (file) | API private key (`.p8`); downloadable only once. | Same Integrations page → generate key. |
## Firebase
| Field | What it is | Where to get it |
| -------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `android_google_services_json` (file) | Firebase Android config (`google-services.json`). | [Firebase Console](https://console.firebase.google.com) → Project settings → Android app. |
| `ios_google_service_info_plist` (file) | Firebase iOS config (`GoogleService-Info.plist`). | Firebase Console → Project settings → iOS app. |
## App Environment Variables
These preset env vars are baked into the build. Custom env vars can be added but
have no built-in guidance.
| Field | What it is | Where to get it |
| ----------------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `NEXT_PUBLIC_SUPABASE_URL` | Supabase project URL. | [Supabase dashboard](https://supabase.com/dashboard) → Project Settings → Data API. |
| `NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY` | Supabase publishable (anon) key. | Supabase dashboard → Project Settings → API Keys. |
| `API_BASE_URL` | Base URL of the Tuturuuu platform API. | Your production web deployment URL (internal). |
| `TURNSTILE_SITE_KEY` | Cloudflare Turnstile site key for bot protection. | [Cloudflare → Turnstile](https://dash.cloudflare.com/?to=/:account/turnstile). |
| `TURNSTILE_BASE_URL` | URL hosting the Turnstile challenge page. | Your production web app URL (internal). |
| `GOOGLE_WEB_CLIENT_ID` | Google OAuth 2.0 web client ID. | [Google Cloud Console → Credentials](https://console.cloud.google.com/apis/credentials). |
| `GOOGLE_IOS_CLIENT_ID` | Google OAuth 2.0 iOS client ID. | Same Credentials page (iOS client). |
| `MOBILE_TASK_DESCRIPTION_EDITING_ENABLED` | Feature flag for rich task-description editing (`true`/`false`). | Set manually (internal). |
| `MOBILE_CALENDAR_INTEGRATIONS_ENABLED` | Feature flag for calendar integrations (`true`/`false`). | Set manually (internal). |
## CI Tokens
| Field | What it is | Where to get it |
| ---------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Token name | A human-readable label for a CI token the deploy workflow uses to authenticate to the vault. | Chosen by you; the token value is shown once at issue time. |
## Activating Changes
Edits land in a **draft** version. Use **Verify** to re-run readiness checks,
**Activate draft** to promote it to the active bundle the workflow reads, and
**Roll back** to return to the previous active version. Fix any readiness issues
listed on the Overview tab before activating.
# DevOps & Deployment
Source: https://docs.tuturuuu.com/build/devops/overview
The operational map for how Tuturuuu is built, deployed, migrated, and validated.
This section is the runbook for shipping and operating Tuturuuu. It documents the
deployment surfaces that exist in this repository today, the workflows that own
them, and the commands the team should use when something needs to be built,
released, or recovered.
## Deployment Surfaces
| Surface | Source of truth | Delivery target |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Platform web app (`apps/web`) | `.github/workflows/vercel-preview-platform.yaml`, `.github/workflows/vercel-production-platform.yaml` | Vercel preview build validation and production deploy |
| Satellite web apps (every app under `apps/*` with a `vercel-preview-.yaml` / `vercel-production-.yaml` pair: `apps`, `nova`, `rewise`, `calendar`, `finance`, `inventory`, `infrastructure`, `meet`, `tasks`, `track`, `shortener`, `qr`, `cms`, `mail`, `learn`, `teach`, `chat`, `drive`, `mind`, `storefront`) | `.github/workflows/vercel-preview-*.yaml`, `.github/workflows/vercel-production-*.yaml` | Vercel preview and production deployments |
| Database schema (`apps/database`) | `.github/workflows/supabase-staging.yaml`, `.github/workflows/supabase-production.yaml` | Supabase staging and production projects |
| Self-hosted web runtime (`apps/web`) | `apps/web/Dockerfile`, `docker-compose.web.yml`, `docker-compose.web.prod.yml`, `scripts/docker-web.js` | Docker dev stacks and production blue/green deployments |
| Migration web runtime (`apps/tanstack-web`) | `apps/tanstack-web/Dockerfile`, `apps/tanstack-web/wrangler.jsonc`, `.github/workflows/rust-backend.yml`, `.github/workflows/vercel-*-tanstack-web.yaml` | Docker sidecar + Cloudflare Worker (`tuturuuu-tanstack-web`) + opt-in Vercel build validation - see [TanStack/Rust Local And Deployment](/build/devops/tanstack-rust-local-deploy) |
| Rust backend (`apps/backend`) | `apps/backend/Dockerfile`, `apps/backend/wrangler.jsonc`, `.github/workflows/rust-backend.yml` | Docker sidecar on port `7820` + manually dispatched Cloudflare Worker (`tuturuuu-backend`) with preflighted secrets — see [GitHub Actions Runbook](/build/devops/github-actions-runbook) |
| Discord utilities (`apps/discord`) | `.github/workflows/discord-modal-deploy.yml` | Modal |
| Mobile artifacts (`apps/mobile`) | `.github/workflows/mobile-build-*.yaml` | Build artifacts for Android, iOS, macOS, Windows |
| Shared packages (`packages/*`) | `.github/workflows/release-*.yaml` | npm |
## Read This In Order
1. [Environments & Release Flow](/build/devops/environments-release-flow)
2. [Web Docker Deployment](/build/devops/web-docker-deployment)
3. [TanStack/Rust Local And Deployment](/build/devops/tanstack-rust-local-deploy)
4. [GitHub Actions Runbook](/build/devops/github-actions-runbook)
5. [Secrets & Configuration](/build/devops/secrets-and-configuration)
## Core Principles
* GitHub Actions is the canonical automation layer for hosted deployments, platform build validation, and database migrations.
* `tuturuuu.ts` can disable individual workflows; `ci-check.yml` enforces that toggle before a job does real work.
* `bun check` includes path-sensitive Discord Python validation when the local
diff touches `apps/discord/**` or `.github/workflows/discord-python-ci.yml`.
That path runs the same blocking checks as Discord Python CI through
`scripts/check-discord-python.js`.
* Vercel handles hosted satellite web deployments, opt-in TanStack frontend build validation, platform preview build validation, and the hosted platform production deploy. Supabase migrations run as separate workflows; `main` drives staging schema promotion and `production` drives production schema promotion.
* Self-hosted web deployment is Docker-based, and blue/green rollout is the supported rebuild-before-restart path.
* The TanStack/Rust migration runs in parallel with `apps/web`: `apps/tanstack-web` (TanStack Start) and `apps/backend` (Rust, port `7820`) ship as Docker sidecars and Cloudflare Workers, and the TanStack frontend has Vercel build validation when it points at an HTTPS backend origin. See [TanStack/Rust Local And Deployment](/build/devops/tanstack-rust-local-deploy), [Web Docker Deployment](/build/devops/web-docker-deployment), and the [TanStack/Rust migration](/platform/architecture/tanstack-rust-migration) plan.
* Secrets live in GitHub Actions secrets/variables or local env files such as `apps/web/.env.local`. They do not belong in the repo.
## Vercel Cost Controls
Start with measured usage instead of applying cache directives globally. The
following commands identify the expensive projects and inspect a bounded sample
of production traffic without enabling a paid observability add-on:
```bash theme={null}
vercel usage --scope tuturuuu --group-by project --format json
vercel logs --project --scope tuturuuu --environment production --since 1h --json --no-branch
```
`vercel metrics` requires Observability Plus. A `payment_required` response is
an expected limitation when the add-on is disabled; do not enable it solely for
routine cost diagnosis.
Apply these controls in order:
1. Reduce request count and response size before adding cache writes. Prefer
direct-to-storage signed uploads for large payloads and avoid routing public
assets through application functions when a stable CDN URL is available.
2. Cache only public or safely scoped data. Never apply shared CDN caching to
authenticated, user-specific, or workspace-private responses.
3. Shared Satellite and Nova React Query clients use a 30-second stale window
and one retry. Override this only when a surface has a documented freshness
or reliability requirement.
Realtime-backed notification counts use a 15-minute fallback poll and a
5-minute stale window. Active timers render elapsed time locally and use
1-minute running / 5-minute idle reconciliation. Keep mutations and realtime
events invalidating their query keys immediately instead of shortening these
safety-net intervals.
Versioned external-project asset reads return early from the global API proxy
and rely on the asset route's publication/access checks, avoiding the global
guard's cookie validation, suspicious-request checks, and rate-limit work per
cache-busted media redirect. Mutations, unversioned reads, and WebGL delivery
remain behind the global guard.
4. Dashboard sidebar links disable viewport prefetch and prefetch on hover or
keyboard focus. This preserves responsive navigation without invoking every
visible destination merely because the sidebar rendered.
5. After deployment, compare the same billing window and project grouping.
Confirm error rates and latency remain healthy before keeping a cost change.
## Operational Flow
```mermaid theme={null}
flowchart LR
Branch["Git branch or manual dispatch"]
Preview["Platform preview build"]
Staging["Supabase staging migration"]
Production["Platform production deploy"]
ProdDb["Supabase production migration"]
Docker["Self-hosted Docker deploy"]
Branch --> Preview
Preview --> Staging
Branch --> Production
Production --> ProdDb
Branch --> Docker
```
## What Changed Recently
* The TanStack/Rust migration runtimes are now first-class deployment surfaces: `apps/backend` (Rust) builds a native binary, a Docker image, and a Cloudflare Worker bundle via `.github/workflows/rust-backend.yml`; `apps/tanstack-web` validates type-checks/tests, deploys as a Cloudflare Worker (`tuturuuu-tanstack-web`) bound to the backend Worker (`tuturuuu-backend`), and has Vercel preview/production build workflows for frontend compatibility against a separate HTTPS backend origin. Cloudflare uploads are manual-dispatch only and preflight `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, and Worker runtime secret names before deploying.
* `apps/web` now supports both in-place Docker production deploys and blue/green deploys.
* `docker-setup-check.yaml` validates Docker parity, renders both compose files, and builds both the dev and production web images.
* Production Redis in Docker now requires a token, but `scripts/docker-web.js`
satisfies that automatically by generating and injecting the value unless you
explicitly opt out with `--without-redis`. Watcher-managed Infrastructure
projects do not inherit that platform Redis token; they start with Redis
disabled and require project-scoped `MANAGED_PROJECT__UPSTASH_*`
credentials when Redis is intentionally enabled.
If you are changing deployment behavior, update this section and add any new
page to `apps/docs/docs.json`.
# Polar storefront integration
Source: https://docs.tuturuuu.com/build/devops/polar-storefront-integration
Connect a workspace storefront to Polar, wire the payment webhook, and book sales into finance.
This runbook covers connecting an Inventory storefront to [Polar](https://polar.sh)
for real checkout, pointing Polar's webhook at Tuturuuu, and how a paid sale
flows into the workspace finance ledger.
Never paste Polar tokens, client secrets, or webhook secrets into source,
docs, or chat. They are supplied only through environment variables and the
per-workspace integration panel, which stores them encrypted.
## How the flow works
1. A shopper checks out on a `checkoutMode: 'polar'` storefront. Tuturuuu creates
a Polar checkout and stores `polar_checkout_id` on the
`private.inventory_checkout_sessions` row (status `reserved`).
2. The shopper pays on Polar. Polar sends an `order.updated` webhook to Tuturuuu.
3. `syncInventoryPolarOrder` marks the checkout `completed` and, when the order
is `paid`, books the revenue into the workspace finance ledger
(`wallet_transactions`) using the product's finance category and the
workspace default wallet. This is idempotent — a sale books at most one
transaction.
## Money and currency
Inventory commerce money (storefront listing prices, bundle prices,
`compare_at_price`, checkout session/line amounts, settlement ledger entries,
and costing figures) is stored in **integer minor units** of the row's
currency — cents for USD/EUR, but whole units for zero-decimal currencies like
JPY/VND. This matches how Polar represents amounts, so the product/bundle/
checkout sync passes the stored value straight through without scaling.
Conversion is centralized in `@tuturuuu/utils/money`:
* `majorToMinor(amount, currency)` / `minorToMajor(minor, currency)` — convert
at input/storage and read/display boundaries (currency-aware: USD ×100,
JPY/VND ×1).
* `formatMoneyFromMinor(minor, currency)` — the canonical display formatter.
* The shared `MoneyInput` (`@tuturuuu/ui/money-input`) takes and emits minor
units while editing in localized major units.
Two boundaries convert explicitly:
* The finance ledger (`wallet_transactions.amount`) stores **major** units, so
`recordInventorySaleFinanceTransaction` converts the checkout's minor-unit
total back to major units when booking revenue.
* Promotions (`workspace_promotions.value`) remain in major units (shared with
finance invoices); `promotions-polar` multiplies by 100 at the Polar boundary.
> Inventory base prices use exact decimal major units, while Storefront listing,
> bundle, checkout, and Polar amounts use integer minor units. Convert only at
> the explicit provider boundary; never manually multiply a saved Inventory
> price before publishing it.
## Bundles and listings ↔ Polar products
Publishing is automatic: creating or updating a listing/bundle schedules a
best-effort push to Polar (a product with a fixed price per currency). Standalone
bundles with no storefront publish against the workspace-level Polar integration
and are priced in USD. Two-way sync:
* **App → Polar:** name, description, and price sync on every write. Archiving a
listing/bundle (or deleting it) archives its Polar product so it is no longer
buyable; re-publishing un-archives it.
* **Polar → App:** `product.created/updated` webhooks apply name, description,
and price back onto the mapped inventory row.
## Environment variables
Set these on `apps/pay` (reference by name only). Inventory-only deployments
that create Polar products also need the access token and sandbox/currency
configuration in their owning runtime:
| Variable | Purpose |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `POLAR_SANDBOX` | `true` to use Polar's sandbox; unset/`false` for production. |
| `POLAR_ACCESS_TOKEN` | Platform-level Polar organization access token. |
| `POLAR_WEBHOOK_SECRET` | Secret used to verify incoming Polar webhook signatures. Must match the value Polar shows when you create the webhook. |
| `POLAR_CURRENCIES` | Optional allowlist of supported storefront currencies. |
For sandbox testing, create the credentials at
`https://sandbox.polar.sh/dashboard//settings` and the OAuth app /
organization token under **User Settings → Developer**.
## Connect a workspace to Polar
1. Open **Inventory → Overview → Polar settings** in the workspace.
2. Choose the environment (sandbox or production) and paste the Polar
**organization access token**. It is encrypted per workspace before storage;
only the last 4 characters are ever shown again.
3. Save. Tuturuuu validates the token and provisions a private
`inventory_checkout` product in your Polar org.
## Configure the Polar webhook
In the Polar dashboard for your org → **Settings → Webhooks → Add Endpoint**:
1. **URL** — the canonical Pay webhook endpoint:
```
https://pay.tuturuuu.com/api/payment/webhooks
```
For local development, expose `localhost` with a tunnel (e.g. `cloudflared`
or `ngrok`) and use the tunnel's HTTPS URL with the same path.
2. **Format** — `Raw`.
3. **Events** — subscribe at minimum to:
* `checkout.created`
* `checkout.updated`
* `order.created`
* `order.updated`
4. Copy the webhook **secret** Polar generates and set it as
`POLAR_WEBHOOK_SECRET` on `apps/pay`. Signature verification fails closed if
it does not match.
Sandbox changes never touch your live Polar account and never move real money
— the dashboard shows a "Payments are not processed" banner.
## Verify a paid sale books finance
1. Place a sandbox checkout on a `polar` storefront and complete payment.
2. Confirm Polar delivered the `order.updated` event (Webhooks → Deliveries).
3. The checkout session flips to `completed` with `polar_status = paid`.
4. A `wallet_transactions` row appears in the workspace finance ledger for the
sale total, linked back via `inventory_checkout_sessions.finance_transaction_id`.
If no transaction appears, check that the workspace has a **default wallet** set
(finance config `default_wallet_id`) — booking is skipped without one.
# Secrets & Configuration
Source: https://docs.tuturuuu.com/build/devops/secrets-and-configuration
Where deployment configuration lives, and which secrets matter for each delivery surface.
This page documents secret names and configuration boundaries. It does not contain
secret values.
## General Rules
* Store hosted deployment credentials in environment-scoped GitHub Actions
secrets.
* Store Vercel app runtime and build-time configuration in the Vercel project
environment, not workflow-wide GitHub Actions environment variables.
* Store local web runtime configuration in `apps/web/.env.local`.
* Do not commit tokens, keys, passwords, or rendered env dumps.
* Do not edit checked-in compose files to inject per-environment secrets.
## Local And Self-Hosted Web
| Location | Purpose |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `apps/web/.env.local` | Docker build secret input and runtime env file for the web app |
| Shell environment | Optional overrides such as `DOCS_PORT`, `SUPABASE_SERVER_URL`, or `UPSTASH_REDIS_REST_*` when you intentionally override the Docker helpers |
| `tmp/docker-web/prod/` | Generated local deployment state for blue/green rollout |
## GitHub Actions Secrets By Area
### Vercel-hosted web apps
Vercel workflows use GitHub Environments named `vercel-preview-` and
`vercel-production-`. Keep the deployment secrets on those environments,
not as workflow-level `env:` entries. Production environments should require
reviewers and restrict deployment branches to `production`; preview environments
should require review whenever branch pushes can run repository-controlled code
with deployment credentials.
* `VERCEL_TOKEN`
* `VERCEL_ORG_ID`
* App-specific `VERCEL_PROJECT_ID` values such as `VERCEL_PLATFORM_PROJECT_ID`
and `VERCEL_APPS_PROJECT_ID`
* Infrastructure deployments use `VERCEL_INFRASTRUCTURE_PROJECT_ID` in the
`vercel-preview-infrastructure` and `vercel-production-infrastructure` GitHub
Environments
* QR deployments use `VERCEL_QR_PROJECT_ID` in the `vercel-preview-qr` and
`vercel-production-qr` GitHub Environments
### Turborepo remote cache
Remote-cache identity is repository-wide rather than app runtime
configuration:
* Store a dedicated, least-privilege Vercel team cache token as the repository
secret `TURBO_TOKEN`. Rotate it independently from deployment tokens and
revoke the previous token after trusted canaries pass.
* Store the Vercel team slug as the repository variable `TURBO_TEAM`. Workflows
temporarily accept `secrets.TURBO_TEAM` as a migration fallback; remove that
secret after the variable is confirmed.
* Leave `TURBO_API` unset for Vercel-managed remote caching.
* Do not configure `TURBO_REMOTE_CACHE_SIGNATURE_KEY` until artifact signing is
deliberately enabled and rolled out.
* Optional repository variables `ACTIONS_CACHE_NOTICE_PERCENT`,
`ACTIONS_CACHE_WARNING_PERCENT`, and `ACTIONS_CACHE_CRITICAL_PERCENT` override
the weekly report's 80/90/100 thresholds. Cache size and retention are not
variables; the report discovers both from GitHub's repository API.
Only pass these values as inputs to
`.github/actions/run-with-turbo-remote-cache/action.yml`. The action places them
on the wrapped command step, not the workflow, job, `GITHUB_ENV`, image layer,
or deployment artifact. Never pass `TURBO_TOKEN` to pull-request or Dependabot
jobs; secretless jobs use the task-family GitHub cache fallback.
### Production and preview app configuration
These values belong in the Vercel project environment for each app and target.
Do not export them from GitHub Actions workflows. The Vercel deploy workflows
run repository code during install and build, so workflow-wide production
runtime secrets are a supply-chain exposure path.
* `PRODUCTION_SUPABASE_URL`
* `PRODUCTION_SUPABASE_PUBLISHABLE_KEY`
* `PRODUCTION_SUPABASE_SECRET_KEY`
* `ENCRYPTION_MASTER_KEY`
Turborepo cache identity is the repository-level configuration described above,
not a Vercel app runtime secret. Do not duplicate it into every project's
runtime environment.
Individual workflows may also consume other application-specific secrets depending on
the deployed app.
### Integration-specific runtime secrets
* SePay OAuth requires `SEPAY_OAUTH_CLIENT_ID`, `SEPAY_OAUTH_CLIENT_SECRET`,
`SEPAY_OAUTH_TOKEN_ENCRYPTION_SECRET`, and a webhook auth secret such as
`SEPAY_WEBHOOK_API_KEY`.
* SePay web runtimes also require an app origin through `WEB_APP_URL`,
`NEXT_PUBLIC_WEB_APP_URL`, or `NEXT_PUBLIC_APP_URL` so OAuth callbacks and
webhook provisioning can generate stable platform URLs.
* SePay workspace enablement is controlled by the workspace secret
`ENABLE_SEPAY_INTEGRATION=true`.
* SePay runtime overrides may optionally set `SEPAY_OAUTH_AUTHORIZE_URL`,
`SEPAY_OAUTH_BASE_URL`, and `SEPAY_API_BASE_URL` when targeting non-default
environments.
* SePay OAuth state is stored in a short-lived HttpOnly callback cookie and is
HMAC-signed. The signer uses `SEPAY_OAUTH_STATE_SECRET` when set; otherwise it
falls back to the required `SEPAY_OAUTH_CLIENT_SECRET`.
* Polar (inventory storefront checkout + workspace subscriptions) requires
`POLAR_ACCESS_TOKEN` (platform-level org token) and `POLAR_WEBHOOK_SECRET`
(webhook signature verification) on `apps/pay`. Set `POLAR_SANDBOX=true` to
target Polar's sandbox, and the optional `POLAR_CURRENCIES` to allowlist
storefront currencies. Configure Polar to deliver webhooks to
`https://pay.tuturuuu.com/api/payment/webhooks`; `apps/web` no longer owns
payment API or webhook routes. See the Polar storefront integration runbook
for details.
* Square Terminal (Inventory + Storefront pay-at-terminal checkout) stores
Square application credentials, OAuth/manual tokens, webhook signature keys,
locations, terminal ids, and webhook notification URL overrides encrypted from
Inventory settings. Do not configure Square credentials as deployment secrets.
See the Square Terminal integration runbook for the complete setup and smoke
checklist.
### Supabase migrations
* `SUPABASE_ACCESS_TOKEN`
* `STAGING_DB_PASSWORD`
* `STAGING_PROJECT_ID`
* `STAGING_DB_URL`
* `PRODUCTION_DB_PASSWORD`
* `PRODUCTION_PROJECT_ID`
* `PRODUCTION_DB_URL`
### Modal deployment
* `MODAL_TOKEN_ID`
* `MODAL_TOKEN_SECRET`
* `MODAL_ENVIRONMENT` as a GitHub Actions variable
### Mobile store beta deployment
`mobile-deploy-stores.yaml` uses the `mobile-store-beta` GitHub Environment and
runs only from `production` pushes that touch `apps/mobile/**`,
`scripts/mobile-deployment/**`, the workflow file, `ci-check.yml`, or
`tuturuuu.ts`. Keep this environment branch-restricted to `production`.
GitHub Environment secrets:
* `MOBILE_DEPLOYMENT_CI_TOKEN`: issued from the root workspace mobile deployment
vault and scoped to `production` mobile deployment.
If `MOBILE_DEPLOYMENT_CI_TOKEN` is not set yet, the mobile store workflow's
credentials preflight emits a notice and skips the Android and iOS publish jobs.
Set the secret when the deployment vault is ready; invalid non-empty tokens still
fail during bundle fetch so real misconfiguration remains visible.
No Firebase, signing, store API, or mobile build secret values should be stored
in GitHub Environment secrets or variables. Those resources live in the
`apps/infrastructure` mobile deployment vault at `/internal/mobile-deployment`,
which is root-workspace-only and requires `manage_mobile_deployment_vault`.
apps/web deployment requirements:
* `ENCRYPTION_MASTER_KEY`: required before uploads, activation, and CI bundle
fetches can decrypt the per-version deployment data key.
* Root workspace Drive/R2/Supabase storage must be available. File resources are
stored as encrypted ciphertext blobs under the reserved
`.tuturuuu/mobile-deployment-vault` prefix; normal Drive routes deny that
prefix.
Vault resources required before activation:
* Manage all non-file values in the vault's **Secrets** panel. Build-time
secrets are rendered into `apps/mobile/.env.github` during CI; built-in
signing and store secrets are exported separately for the Android and iOS
release steps. The expected production build-time keys are
`NEXT_PUBLIC_SUPABASE_URL`, `NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY`,
`API_BASE_URL`, `TURNSTILE_SITE_KEY`, `TURNSTILE_BASE_URL`,
`GOOGLE_WEB_CLIENT_ID`, `GOOGLE_IOS_CLIENT_ID`,
`MOBILE_TASK_DESCRIPTION_EDITING_ENABLED`, and
`MOBILE_CALENDAR_INTEGRATIONS_ENABLED`. Custom secrets are allowed when they
use uppercase `A-Z0-9_` names, do not collide with built-in secret names, and
have single-line UTF-8 values.
* CI file payload keys such as `MOBILE_ANDROID_GOOGLE_SERVICES_JSON_B64` and
`MOBILE_IOS_GOOGLE_SERVICE_INFO_PLIST_B64` are rejected as secrets; upload
those payloads through the file resources below.
* Android files: production `google-services.json`, upload keystore, and Google
Play service-account JSON.
* Built-in Android secrets: `ANDROID_KEYSTORE_ALIAS`,
`ANDROID_KEYSTORE_PASSWORD`, `ANDROID_KEYSTORE_PRIVATE_KEY_PASSWORD`,
`GOOGLE_PLAY_PACKAGE_NAME=com.tuturuuu.app.mobile`, and
`GOOGLE_PLAY_TRACK=internal`.
* iOS files: production `GoogleService-Info.plist`, Apple distribution `.p12`,
App Store provisioning profile, and App Store Connect `.p8` private key.
* Built-in iOS secrets: `APPLE_BUNDLE_ID=com.tuturuuu.app.mobile`,
`APPLE_DISTRIBUTION_CERTIFICATE_PASSWORD`, `APPLE_TEAM_ID`,
`APP_STORE_CONNECT_API_KEY_ID`, and `APP_STORE_CONNECT_ISSUER_ID`.
After every resource is uploaded, activate the ready draft version and issue a
CI token. Store only that token in the `mobile-store-beta` GitHub Environment as
`MOBILE_DEPLOYMENT_CI_TOKEN`.
### Package publishing
Package publishing uses **npm trusted publishing** (GitHub OIDC), so **no
`NPM_TOKEN` secret is stored**. The `publish-npm` job in each
`release-*-package.yaml` workflow requests `id-token: write`, downloads the
prepared tarball, and runs `npm publish` while npm exchanges the OIDC token for a
short-lived publish credential.
* Each publish job is bound to a per-package GitHub Environment
(`ui-release-production` for `@tuturuuu/ui`, `types-release-production` for
`@tuturuuu/types`), and the matching npm trusted publisher must reference this
repository, the exact workflow filename, and that environment.
* No npm registry token (`NPM_TOKEN` or `NODE_AUTH_TOKEN`) is set on these
environments. Adding one would be unnecessary and would weaken the
build-vs-publish job split.
See the trusted-publishing details in the
[GitHub Actions runbook](/build/devops/github-actions-runbook).
## Docker-Specific Notes
* The Docker web helper auto-generates a stable local Redis token and injects
`UPSTASH_REDIS_REST_TOKEN`, `UPSTASH_REDIS_REST_URL`, and the matching
internal `SRH_TOKEN` value for the bundled `serverless-redis-http` container.
* Watcher-managed Infrastructure projects do not inherit the integrated Docker
Redis runtime. They start with Redis disabled, strip generic
`UPSTASH_REDIS_REST_*` and Docker-specific `DOCKER_UPSTASH_*` values, and
receive Redis only from project-scoped
`MANAGED_PROJECT__UPSTASH_REDIS_REST_URL` and
`MANAGED_PROJECT__UPSTASH_REDIS_REST_TOKEN` variables.
* `SRH_TOKEN` is the container's internal environment variable. The user-facing
override surface is `UPSTASH_REDIS_REST_TOKEN` if you intentionally replace
the helper-generated value.
* The local dev compose file keeps a dev-only fallback token for direct local
use, but production Redis compose requires `UPSTASH_REDIS_REST_TOKEN` during
Compose interpolation. Use the Docker web helper or export a strong token
before enabling the production `redis` profile directly.
* Redis and `serverless-redis-http` host ports must stay loopback-bound. Do not
publish those sidecars on all host interfaces or through Cloudflare Tunnel.
* `apps/web` uses Redis for defense-in-depth one-time state such as CLI
refresh-token replay protection. If `UPSTASH_REDIS_REST_URL` and
`UPSTASH_REDIS_REST_TOKEN` are unavailable, CLI refresh requests continue
after JWT validation and user lookup, while confirmed Redis-backed replays are
still rejected.
* `docker compose config` expands env values; treat its output as sensitive.
## Supabase RPC Authorization Hardening
* Treat every new `SECURITY DEFINER` function in `public` as externally callable until proven otherwise.
* In the same migration that creates or replaces the function, enforce both:
1. in-function authorization checks (`auth.uid()` plus workspace membership/permission gate), and
2. explicit execute privileges (`revoke all ... from public`, then grant only required roles such as `authenticated`/`service_role`).
* Do not rely only on `apps/web` page or API RBAC checks for tenant isolation when an RPC is exposed through the Data API.
* During review, grep the migration for `security definer`, `auth.uid()`, `has_workspace_permission`, `revoke`, and `grant execute` before merge.
## Change Management
When a new deployment flow is added:
1. Add its secret names here.
2. Add the workflow or runtime page to the relevant devops doc.
3. Keep the boundary clear between repo config, GitHub Actions config, and machine-local config.
# SePay Testing Guide
Source: https://docs.tuturuuu.com/build/devops/sepay-testing-guide
End-to-end validation steps for OAuth, provisioning, and webhook ingestion.
## Prerequisites
* The web app is reachable locally.
* You have a target workspace id.
* `ENABLE_SEPAY_INTEGRATION=true` is set in `workspace_secrets`.
* The web runtime is configured with:
* `SEPAY_OAUTH_CLIENT_ID`
* `SEPAY_OAUTH_CLIENT_SECRET`
* `SEPAY_OAUTH_TOKEN_ENCRYPTION_SECRET`
* `SEPAY_WEBHOOK_API_KEY` or `SEPAY_WEBHOOK_SECRET`
* `WEB_APP_URL` or `NEXT_PUBLIC_WEB_APP_URL` or `NEXT_PUBLIC_APP_URL`
* Optional overrides:
* `SEPAY_OAUTH_AUTHORIZE_URL`
* `SEPAY_OAUTH_BASE_URL`
* `SEPAY_API_BASE_URL`
## Enable The Workspace Flag
```sql theme={null}
insert into workspace_secrets (ws_id, name, value)
values ('', 'ENABLE_SEPAY_INTEGRATION', 'true')
on conflict (ws_id, name)
do update set value = excluded.value;
```
## Validate OAuth
1. Start OAuth from the same browser session that will receive the callback.
2. Call:
```bash theme={null}
curl -X POST \
"http://localhost:7803/api/v1/workspaces//integrations/sepay/oauth/start" \
-H "Authorization: Bearer " \
-c /tmp/sepay-oauth.cookies
```
3. Open the returned `authorizeUrl`.
4. Complete SePay consent.
5. Confirm the callback succeeds and the workspace receives a SePay connection.
## Validate Provisioning
```bash theme={null}
curl "http://localhost:7803/api/v1/workspaces//integrations/sepay/endpoints" \
-H "Authorization: Bearer "
```
Expected:
* At least one active endpoint.
* `token_prefix` is present.
* `sepay_webhook_id` is present after provisioning.
Database checks:
```sql theme={null}
select id, ws_id, status, access_token_expires_at, scopes
from sepay_connections
where ws_id = '';
select id, ws_id, sepay_bank_account_id, sepay_sub_account_id, wallet_id, active
from sepay_wallet_links
where ws_id = '';
select id, ws_id, active, deleted_at, token_prefix, sepay_webhook_id
from sepay_webhook_endpoints
where ws_id = ''
order by created_at desc;
```
## Validate Webhook Ingestion
```bash theme={null}
curl -X POST \
"http://localhost:7803/api/v1/webhooks/sepay/" \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"id": "evt_test_001",
"gateway": "VCB",
"transactionDate": "2026-04-09T10:00:00+07:00",
"accountNumber": "0071000888888",
"subAccount": null,
"content": "Thu tien don hang #1001",
"description": "Thanh toan don hang",
"transferType": "in",
"transferAmount": 150000,
"referenceCode": "REF1001",
"code": "PAY1001"
}'
```
Expected:
* The API returns success, or duplicate-safe success on replay.
* A `sepay_webhook_events` row exists for `evt_test_001`.
* A `wallet_transactions` row exists and is linked through
`created_transaction_id`.
* `transactionDate` should always include an explicit timezone such as
`+07:00`. Bare `YYYY-MM-DD HH:mm:ss` strings are interpreted as Vietnam time
(UTC+7) for SePay compatibility; omitting the timezone in test payloads is a
fixture bug, not a supported operator shortcut.
Verification query:
```sql theme={null}
select id, sepay_event_id, status, created_transaction_id, failure_reason
from sepay_webhook_events
where ws_id = ''
order by received_at desc
limit 20;
select id, wallet_id, category_id, amount, creator_id, description, taken_at
from wallet_transactions
where creator_id in (
select id
from workspace_users
where ws_id = ''
and full_name = 'SePay System'
)
order by created_at desc
limit 20;
```
## Idempotency
Resend the exact same payload with the same `id`.
Expected:
* No duplicate transaction is inserted.
* The event is treated as a duplicate/no-op.
## Expense Direction
Send a payload with `transferType: "out"` and a positive `transferAmount`.
Expected:
* `wallet_transactions.amount` is stored as a negative value.
* The transaction resolves against an expense category.
## Endpoint Lifecycle
1. Create an endpoint.
2. Rotate the endpoint.
3. Delete the endpoint.
Expected:
* Deleted endpoints have `active = false`.
* Deleted endpoints have `deleted_at is not null`.
* Listing routes exclude deleted endpoints.
* Token resolution ignores deleted endpoints.
## Disconnect
```bash theme={null}
curl -X POST \
"http://localhost:7803/api/v1/workspaces//integrations/sepay/disconnect" \
-H "Authorization: Bearer "
```
Expected:
* `sepay_connections.status = 'revoked'`
* Active endpoints are marked inactive and soft-deleted.
## Failure Paths
* Invalid webhook auth header is rejected.
* Invalid webhook JSON body returns a validation error.
* Unknown endpoint token is rejected.
* Disabled feature flag blocks SePay workspace APIs.
## Troubleshooting
* OAuth start failures usually mean missing OAuth env vars or app-origin config.
* OAuth callback state failures usually mean the callback did not reuse the
browser session or cookie jar that started OAuth.
* Provisioning failures usually mean SePay has no usable bank account or the
webhook scopes were not granted.
* Missing transactions should be debugged from
`sepay_webhook_events.failure_reason` first.
# Square Terminal integration
Source: https://docs.tuturuuu.com/build/devops/square-terminal-integration
Connect Inventory and Storefront checkout to Square Terminal, OAuth, catalog and stock sync, device pairing, and webhooks.
This runbook covers the Inventory and Storefront integration with
[Square Terminal](https://developer.squareup.com/docs/terminal-api/overview).
Use Square Terminal API for countertop terminal payments; the
[Square POS API](https://developer.squareup.com/docs/pos-api/what-it-does)
opens the mobile Square Point of Sale app and is not the right integration for
driving physical terminals from Tuturuuu.
For a store-owner-friendly walkthrough, use
[Set up a physical Square POS](/platform/applications/inventory-square-pos).
Never paste Square access tokens, OAuth secrets, webhook signature keys, or
device identifiers into source, docs, or chat. Store workspace credentials
only through the Inventory Square settings panel, which encrypts them before
private-schema storage.
## Customer guide map
Roles, credentials, OAuth, webhooks, location, and device setup.
Success, cancel, timeout, offline, expiry, duplicate event, and stock tests.
Direction selection, links, conflicts, prices, counts, and no-delete rules.
Physical pairing, first live sale, go/no-go gate, and customer handoff.
Payments hub observability, lifecycle, reconciliation, and daily checks.
Safe symptom-based recovery and escalation packets.
## How the flow works
1. A shopper checks out on a `checkoutMode: 'square_terminal'` storefront.
Tuturuuu validates Square readiness, creates a local checkout reservation,
stores the provider as `square_terminal`, and returns the buyer to the local
order reference page.
2. A Square Terminal Storefront dispatches the reserved checkout immediately;
an eligible reserved row can also be sent or canceled from the Inventory
Commerce workflow. Tuturuuu creates a Square order, then creates a Terminal
checkout for the selected location and device with the Square order id and
itemized cart display enabled.
3. Square sends terminal checkout and payment webhooks. Tuturuuu verifies the
raw webhook body signature before parsing, reconciles duplicate deliveries
idempotently, and completes the checkout only after a verified paid payment.
4. On Square create failure, cancellation, expiry, or failed terminal checkout,
Tuturuuu releases the local reservation so stock returns to availability. A
five-minute expiry sweep materializes abandoned 15-minute reservations as
`expired`; checkout reads and new checkout creation also reconcile stale rows.
Catalog sync supports Square to Tuturuuu, Tuturuuu to Square, and two-way
comparison. It is intentionally non-destructive on Square: Tuturuuu uses
Catalog Batch Upsert and physical inventory counts, never calls a Square catalog
delete endpoint, preserves Square-only variations when updating an item, and
turns simultaneous edits into conflicts for operator review. A Square deletion
marks the local link for review without deleting local product or stock data.
Inventory base prices use exact decimal major units, so Square Money amounts
round-trip at the currency exponent without rounding to whole dollars. Legacy
fractional-price holds clear on the next Square-to-Tuturuuu import after the
cent-level schema migration is applied.
## Workspace credentials
Square credentials are self-serve per workspace, like Inventory Polar settings.
Do not configure Square app credentials, access tokens, or webhook signature
keys as deployment secrets. Workspace admins save these values in **Inventory →
Payments → Connect & set up → Square POS**, where Tuturuuu encrypts secret
values before private-schema storage.
The Square REST client pins `Square-Version: 2026-05-20` and uses native
`fetch`. OAuth is the recommended connection method because Tuturuuu can refresh
tokens and validate granted scopes. Manual access-token configuration remains
available for controlled deployments and does not require saved Square OAuth app
credentials.
## Configure Square
1. Create or open a Square application in the Square Developer Dashboard.
2. Configure the OAuth redirect URL:
```
https:///api/v1/inventory/square/oauth/callback
```
3. Grant these OAuth scopes:
* `MERCHANT_PROFILE_READ`
* `ORDERS_READ`
* `ORDERS_WRITE`
* `PAYMENTS_READ`
* `PAYMENTS_WRITE`
* `DEVICE_CREDENTIAL_MANAGEMENT`
* `ITEMS_READ`
* `ITEMS_WRITE`
* `INVENTORY_READ`
* `INVENTORY_WRITE`
4. Create a webhook subscription for the Inventory Square endpoint:
```
https:///api/v1/inventory/square/webhook/
```
Subscribe to these required events:
* `device.code.paired`
* `terminal.checkout.created`
* `terminal.checkout.updated`
* `payment.updated`
* `oauth.authorization.revoked`
* `catalog.version.updated`
* `inventory.count.updated`
`payment.created` can be added for additional delivery visibility, but
`payment.updated` is the required payment-state signal in the customer setup
checklist.
5. Copy the webhook signature key into **Inventory → Payments → Connect & set
up → Square POS** for the matching environment.
For local development, expose `apps/web` with an HTTPS tunnel and either use the
tunnel URL in Square or save that exact URL as the workspace webhook
notification URL so signature validation uses the same value.
## Connect a workspace
1. Open **Inventory → Payments → Connect & set up → Square POS**. Configuration
changes open in a three-tab dialog; incomplete checklist steps deep-link to
the required tab.
2. Choose `Sandbox` or `Production`.
3. Save the Square Application ID and Application Secret for that environment.
If Square is configured with a tunnel or canonical URL that differs from the
actual request URL, also save the exact webhook notification URL so HMAC
validation uses the same URL Square signed.
4. Prefer **Connect OAuth**. For manual setup, paste an access token and save it
with the matching environment.
5. Save the webhook signature key for the same environment.
6. Select a Square location.
7. Pair or select a terminal:
* Production: create a device pairing code, enter it on the physical Square
Terminal within five minutes, then select the paired device. The pairing
request uses Square's `TERMINAL_API` product type. Create this code inside
Tuturuuu; device codes from Square Dashboard are not compatible with
Terminal API pairing.
* Sandbox: use Square's sandbox terminal device id when device listing is not
available.
8. Set a storefront's checkout mode to `Square Terminal`.
9. In **Catalog and stock sync**, import from Square first in Sandbox. Review
counts and conflicts, then rehearse publish and two-way sync. Repeat in
Production only after confirming the selected seller and location.
Readiness fails closed until the connection, required scopes, webhook signature
key, location, device, and environment all match. OAuth-backed connections also
require saved workspace Square app credentials so refresh can continue without
deployment secrets. Storefront checkout returns a configuration error instead of
reserving stock when Square is not ready.
The verified Tuturuuu launch path requires reliable connectivity. Square lists
Terminal API offline payments as an opt-in beta capability, but Tuturuuu does
not currently certify or depend on that flow. Operators should restore the
network and reconcile any uncertain checkout before retrying.
## Runtime ownership and endpoints
The Inventory satellite owns the customer-facing Square routes and delegates
provider orchestration to `@tuturuuu/inventory-core`. Client components use
`@tuturuuu/internal-api`; they must not call provider APIs or private database
tables directly.
| Method and route | Purpose |
| -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `GET/PUT /api/v1/workspaces/:wsId/inventory/square-settings` | Read masked readiness state or save the workspace Square configuration |
| `GET /api/v1/workspaces/:wsId/inventory/square/oauth/start` | Start environment-bound OAuth |
| `GET /api/v1/inventory/square/oauth/callback` | Exchange the authorization and return to Inventory |
| `GET /api/v1/workspaces/:wsId/inventory/square/locations` | List locations for the connected seller |
| `GET /api/v1/workspaces/:wsId/inventory/square/devices` | List paired Production Terminal API devices |
| `POST /api/v1/workspaces/:wsId/inventory/square/device-codes` | Create a five-minute `TERMINAL_API` pairing code |
| `GET/POST /api/v1/workspaces/:wsId/inventory/square/catalog-sync` | Inspect links or run import, publish, or two-way sync |
| `POST /api/v1/workspaces/:wsId/inventory/square/terminal-checkouts` | Dispatch one reserved checkout to Square |
| `POST /api/v1/workspaces/:wsId/inventory/square/terminal-checkouts/:checkoutId/cancel` | Cancel the same Square Terminal checkout and reconcile release |
| `POST /api/v1/inventory/square/webhook/:wsId` | Verify and process workspace-scoped Square events |
| `POST /api/v1/inventory/storefronts/:slug/checkouts` | Reserve the cart and dispatch provider checkout when the Storefront uses Square Terminal |
The environment, workspace, seller connection, location, and device are checked
again server-side. A client-visible control is not an authorization boundary.
## State and idempotency boundaries
```mermaid theme={null}
flowchart TD
A["Public cart validation"] --> B["Private reservation RPC"]
B --> C["Square order with idempotency key"]
C --> D["Terminal checkout with idempotency key"]
D --> E["Signed Square webhooks"]
E --> F{"Verified paid payment?"}
F -->|"Yes"| G["Complete once, consume stock, book at most one sale"]
F -->|"Cancel, fail, or expire"| H["Release once"]
```
Provider IDs have unique database indexes, webhook events carry Square event
IDs, and reconciliation routines are safe to call repeatedly. Never weaken
those constraints to make a duplicate test pass.
The local checkout is reserved for 15 minutes. The database expiry function is
service-role only, concurrency-safe, and invoked by a five-minute cron plus lazy
reconciliation on reads and new checkout creation. Terminal cancellation and
Square final-state webhooks use the same release/complete boundaries.
## Focused verification commands
Run the narrow Square contract suites before the wider repository gate:
```bash theme={null}
bun --filter @tuturuuu/inventory-core test
bun x vitest run \
'apps/inventory/src/app/api/v1/inventory/square/webhook/[wsId]/route.test.ts' \
'apps/inventory/src/app/api/v1/inventory/storefronts/[slug]/checkouts/route.test.ts' \
'apps/inventory/src/app/api/v1/workspaces/[wsId]/inventory/square-settings/route.test.ts' \
'apps/inventory/src/app/api/v1/workspaces/[wsId]/inventory/square/catalog-sync/route.test.ts' \
'apps/inventory/src/app/api/v1/workspaces/[wsId]/inventory/square/terminal-checkouts/route.test.ts' \
'apps/inventory/src/app/api/v1/workspaces/[wsId]/inventory/square/terminal-checkouts/[checkoutId]/cancel/route.test.ts' \
'apps/inventory/src/app/api/cron/inventory/checkout-expiry/route.test.ts' \
apps/inventory/src/components/operator/payments-readiness.test.ts \
apps/inventory/src/components/operator/square-setup-progress.test.ts
bun --filter @tuturuuu/inventory type-check
bun check
```
When implementation routes or app dependencies change, also run the real
Inventory build as required by the repository operating manual:
```bash theme={null}
cd apps/inventory
bun run build
```
Do not treat a passing build as physical hardware certification. The customer
must still complete the Production smoke test below.
## Hardware-free verification
No mocked test can prove that a specific production countertop terminal is
online, paired to the right Square seller account, assigned to the selected
location, has a working network path, and can complete a real card-present
payment. Without hardware, treat the automated suite as Square contract and
reconciliation verification, not physical-device certification.
The no-hardware suite must cover these Square contracts before release:
* OAuth authorization URL, token exchange, refresh, scope parsing, encrypted
storage, and secret redaction.
* REST base URLs for sandbox and production, `Square-Version`, bearer auth,
sanitized upstream errors, and idempotency keys.
* Catalog search, additive Batch Upsert, unknown-variation preservation,
two-sided hash conflict detection, deleted-object preservation, and physical
inventory count payloads that never contain a delete instruction.
* Orders API payloads before Terminal checkout creation.
* Terminal checkout payloads with `order_id`,
`device_options.device_id`, and `device_options.show_itemized_cart`.
* Device Code creation with `product_type: TERMINAL_API` and paired-device
webhook reconciliation.
* Raw-body webhook HMAC validation against the exact Square notification URL
before JSON parsing.
* Terminal checkout, payment, OAuth revocation, cancellation, expiry, failure,
duplicate delivery, and event-id reconciliation behavior.
* Reservation-first lifecycle: create local reservation before Square calls,
release on Square create/cancel/failure/expiry, and complete stock/ledger
state only after verified Square payment success.
* Scheduled and lazy checkout-expiry reconciliation, including concurrent
sweeper safety and service-role-only database access.
## Physical-terminal smoke checklist
Complete the simulator matrix in Sandbox first. Square does not pair real
hardware to Sandbox, so physical-device certification requires a separately
approved Production test with a low-value item and real card-present processing.
Do not promise that a refund also refunds every processing fee; the Square owner
must approve the store's refund plan.
1. Pair a terminal for the selected Square location.
2. Create a small Storefront order and confirm Inventory shows it as reserved.
3. Use Inventory commerce actions to send it to the terminal.
4. Complete the payment on the terminal.
5. Confirm Square webhook delivery succeeded and Tuturuuu marked the checkout
completed with Square order, terminal checkout, payment, and receipt URL
metadata.
6. Confirm stock reservations were consumed and the finance ledger sale was
booked once.
7. Repeat cancel, expiry/failure, duplicate webhook delivery, and transient
Square failure cases in Sandbox. Cancel/failure/expiry must release
inventory. Do not repeat destructive Production failure tests unless the
owner explicitly approves their real-money and operational effects.
Use the customer-facing
[Production launch gate](/platform/applications/inventory-square-pos/production-launch)
for the exact owner, operator, receipt, stock, and go/no-go checklist.
## Refund and dispute reconciliation
Square connections created before Inventory–Finance reconciliation may not
include `DISPUTES_READ`. Existing payment capture remains usable without that
scope. Ask the Square owner to reconnect OAuth during a planned maintenance
window before enabling dispute history synchronization.
Subscribe the configured Square webhook endpoint to:
* `refund.created`
* `refund.updated`
* `dispute.created`
* `dispute.state.updated`
Refunds are recorded only when Square reports them completed. Dispute creation
creates a negative chargeback hold; `WON` creates a positive release, while
`LOST` and `ACCEPTED` leave the hold final. Webhook retries update the same
provider source key and must never create duplicate ledger rows.
After deployment, use the Finance reconciliation provider-sync action in
bounded pages. It reads Square refund and dispute history but does not create,
accept, challenge, or otherwise mutate a Square dispute. Validate these flows
in Square Sandbox; do not create or capture Production payments merely to test
reconciliation.
# TanStack/Rust Cutover Runbook
Source: https://docs.tuturuuu.com/build/devops/tanstack-rust-cutover-runbook
Bring up, verify, and gate the dual-stack TanStack frontend and Rust backend before cutover.
This runbook is the discoverable docs copy of the root cutover runbook for the
TanStack Start plus Rust backend migration. It focuses on operational checks and
does not replace the route ownership contract at
`platform/architecture/tanstack-rust-migration`.
For how to start the native services, run the dual-stack Docker stack, deploy
the Workers, validate the Vercel TanStack frontend build, or expose a VPS
through Cloudflare Tunnel, use
[TanStack/Rust Local And Deployment](/build/devops/tanstack-rust-local-deploy).
## Purpose
Use this runbook when proving the new production target is ready:
* `apps/tanstack-web` serves the frontend.
* `apps/backend` serves Rust-owned backend routes.
* the dual stack runs from production Docker artifacts.
* compare mode proves the legacy and new frontend paths can pass the same smoke
suite before traffic moves.
## Local Dual-Stack Commands
Validate the compose file without starting anything:
```bash theme={null}
docker compose -f docker-compose.tanstack-dual.yml config
```
Run the minimal dual-stack TanStack/Rust E2E suite:
```bash theme={null}
bun test:e2e:tanstack:docker -- -- --project=chromium
```
The command starts `docker-compose.tanstack-dual.yml`, waits for `backend-dual`
and `tanstack-web-dual` to become healthy, runs Playwright in
`apps/tanstack-web`, and tears the stack down unless `--keep-up` is passed to
the runner.
Run the focused compare smoke used by CI:
```bash theme={null}
bun test:e2e:web:docker:compare -- public-marketing-routes.noauth.spec.ts --project=chromium-no-auth
```
Run a broader compare rehearsal by omitting the spec filter, but expect a much
longer full-stack run:
```bash theme={null}
bun test:e2e:web:docker:compare -- --project=chromium-no-auth
```
## Services And Ports
`docker-compose.tanstack-dual.yml` exposes only loopback ports by default:
| Service | Container | Default port |
| ----------------- | ------------------- | -----------: |
| Rust backend | `backend-dual` | `7820` |
| TanStack frontend | `tanstack-web-dual` | `7824` |
The frontend healthcheck must prove the root page renders and can reach the
backend. Keep real secret values in ignored env files or CI secrets; docs and
workflow files should reference environment variable names only.
## CI Coverage
The E2E workflow includes a migration matrix:
* `tanstack-dual-stack` runs the minimal dual-stack compose runner against
`apps/tanstack-web/e2e`.
* `compare-smoke` runs `scripts/run-web-e2e-docker.js --frontend compare`
against a focused public no-auth spec.
Both jobs upload Playwright and `tmp/e2e` artifacts for failure triage.
## Evidence For Gates
The final cutover gate consumes fresh evidence reports:
```bash theme={null}
bun migration:tanstack:gates -- \
--benchmark-report \
--e2e-report \
--cloudflare-smoke-report \
--output tmp/tanstack-cutover-gates.json
```
Use diagnostic mode while legacy routes remain:
```bash theme={null}
bun migration:tanstack:gates -- --allow-legacy --skip-benchmark --skip-e2e --skip-cloudflare-smoke
```
For terminal cutover, do not pass any `--skip-*` evidence flags.
## Final Checklist
* `bun migration:tanstack:cutover-check` passes with zero `legacy-next`
artifacts.
* dual-stack Docker E2E passes against `docker-compose.tanstack-dual.yml`.
* compare-mode Docker E2E passes for both Next and TanStack frontends.
* benchmark compare evidence is fresh and within thresholds.
* Cloudflare smoke probes pass for backend and TanStack origins.
* `bun migration:tanstack:gates` exits `0` with all evidence present.
* final `bun check` passes or has only documented unrelated blockers.
Keep `apps/web` available as rollback fallback until the cutover window is
complete and the post-cutover smoke has stayed green.
# TanStack/Rust Local And Deployment
Source: https://docs.tuturuuu.com/build/devops/tanstack-rust-local-deploy
Run apps/backend and apps/tanstack-web locally, deploy to Cloudflare Workers, validate the Vercel build, and self-host with Docker plus Cloudflare Tunnel.
This is the operator runbook for the migration stack:
* `apps/backend`: Rust API runtime, native on port `7820`, Docker sidecar, and
Cloudflare Worker `tuturuuu-backend`.
* `apps/tanstack-web`: TanStack Start frontend, local dev on port `7824`,
Docker sidecar, Cloudflare Worker `tuturuuu-tanstack-web`, and opt-in Vercel
build validation.
This runbook describes future deployment capability. `apps/backend` is not
currently deployed or used by production, and `apps/web` remains the live API
runtime. Do not run the deployment sections or infer a cutover without
explicit migration approval.
Keep the legacy `apps/web` runtime available until the cutover gates pass. This
page explains how to start, validate, and deploy the new stack; route ownership
and cutover evidence stay in the
[TanStack/Rust migration contract](/platform/architecture/tanstack-rust-migration)
and [cutover runbook](/build/devops/tanstack-rust-cutover-runbook).
## Required Environment
Use real values only in ignored env files, shell variables, GitHub Environment
secrets, Cloudflare Worker secrets, Vercel Project Environment Variables, or VPS
secret stores. Do not commit values.
| Variable | Used by | Notes |
| ---------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `BACKEND_INTERNAL_TOKEN` | backend and frontend server runtime | Shared bearer token for internal migration/status calls. |
| `BACKEND_INTERNAL_URL` | `apps/tanstack-web` outside Cloudflare service bindings | Server-only backend origin, such as `http://localhost:7820` or an HTTPS backend Worker/VPS origin. |
| `BACKEND_PUBLIC_ORIGIN` | `apps/tanstack-web` browser-safe backend origin fallback | Public backend origin when the frontend needs a browser-safe base URL. |
| `SUPABASE_URL` | backend Worker | Server-side Supabase REST origin. |
| `SUPABASE_SERVICE_ROLE_KEY` | backend Worker | Server-side Supabase service key. |
| `TUTURUUU_APP_COORDINATION_SECRET` | backend Worker | App coordination token verification. |
| `CRON_SECRET` | backend Worker | Cron proxy parity. |
| `DISCORD_APP_DEPLOYMENT_URL` | backend Worker | Discord app deployment proxy. |
| `AURORA_EXTERNAL_URL` | backend Worker | Aurora health and ingest upstream. |
| `AURORA_EXTERNAL_WSID` | backend Worker | Aurora ingest workspace id. |
| `TANSTACK_WEB_RUNTIME` | frontend build/runtime selection | Use `node` for Docker/node output, `vercel` for Vercel build validation, and unset for Cloudflare Workers. |
## Local Native Run
Native local development is two processes: Rust backend first, then TanStack
Start. `apps/backend/.env.example` is a template; the native Rust binary reads
environment variables from the process environment.
1. From the repo root, export local-only backend values:
```bash theme={null}
export BACKEND_ENV=development
export BACKEND_INTERNAL_TOKEN=""
export PORT=7820
```
2. Start the Rust backend:
```bash theme={null}
cargo run --manifest-path apps/backend/Cargo.toml --features native --bin backend
```
3. In another shell, point TanStack Start at the backend and start the frontend:
```bash theme={null}
BACKEND_INTERNAL_TOKEN="${BACKEND_INTERNAL_TOKEN:?set BACKEND_INTERNAL_TOKEN}" \
BACKEND_INTERNAL_URL=http://localhost:7820 \
BACKEND_PUBLIC_ORIGIN=http://localhost:7820 \
bun dev:tanstack-web
```
4. Verify the backend:
```bash theme={null}
curl -fsS http://127.0.0.1:7820/healthz
curl -fsS http://127.0.0.1:7820/readyz
curl -fsS \
-H "Authorization: Bearer ${BACKEND_INTERNAL_TOKEN:?set BACKEND_INTERNAL_TOKEN}" \
http://127.0.0.1:7820/api/migration/status
```
5. Open the frontend on port `7824`. The root migration shell should report the
backend as reachable when the URL and token match.
Common local failures:
* `readyz` is not ready: `BACKEND_INTERNAL_TOKEN` is missing from the backend
process.
* TanStack shows backend unreachable: the frontend shell does not have
`BACKEND_INTERNAL_URL` or `BACKEND_INTERNAL_TOKEN`, or the backend is running
on a different port.
* Rust starts but route calls fail: the route may require additional Supabase,
Aurora, Discord, cron, or app coordination env values.
## Docker Dual-Stack Rehearsal
Use the minimal dual-stack compose file when you want production artifacts for
only the migration stack:
```bash theme={null}
docker compose -f docker-compose.tanstack-dual.yml config
docker compose -f docker-compose.tanstack-dual.yml up -d --build
```
The default loopback ports are:
| Service | Container | Host URL |
| ----------------- | ------------------- | ----------------------- |
| Rust backend | `backend-dual` | `http://127.0.0.1:7820` |
| TanStack frontend | `tanstack-web-dual` | `http://127.0.0.1:7824` |
Run the minimal Playwright E2E rehearsal:
```bash theme={null}
bun test:e2e:tanstack:docker -- -- --project=chromium
```
Keep the stack up for manual debugging:
```bash theme={null}
bun test:e2e:tanstack:docker -- --keep-up -- --project=chromium
```
Shut it down when finished:
```bash theme={null}
docker compose -f docker-compose.tanstack-dual.yml down
```
Use the broader production web Docker path when the migration stack needs to run
behind `web-proxy`, blue/green cutover, watcher recovery, Redis, cron, or
Cloudflare Tunnel. That path is covered in
[Web Docker Deployment](/build/devops/web-docker-deployment).
## Cloudflare Workers Deployment
Cloudflare Workers is the edge preview path for both runtimes. The frontend
Worker uses the `BACKEND` service binding to call the backend Worker first, with
HTTP env fallback only for local and emergency non-binding runs. This follows
the current TanStack Start Cloudflare guidance for
`@cloudflare/vite-plugin` plus `wrangler`.
1. Authenticate and validate config:
```bash theme={null}
bun wrangler whoami
bun check:cloudflare
```
2. Install Rust Worker prerequisites if the machine has not built Workers
before:
```bash theme={null}
rustup target add wasm32-unknown-unknown
cargo install worker-build --locked
```
3. Bootstrap backend Worker secrets in Cloudflare:
```bash theme={null}
bun wrangler secret put BACKEND_INTERNAL_TOKEN --config apps/backend/wrangler.jsonc
bun wrangler secret put TUTURUUU_APP_COORDINATION_SECRET --config apps/backend/wrangler.jsonc
bun wrangler secret put SUPABASE_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put SUPABASE_SERVICE_ROLE_KEY --config apps/backend/wrangler.jsonc
bun wrangler secret put CRON_SECRET --config apps/backend/wrangler.jsonc
bun wrangler secret put DISCORD_APP_DEPLOYMENT_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put AURORA_EXTERNAL_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put AURORA_EXTERNAL_WSID --config apps/backend/wrangler.jsonc
```
4. Deploy the backend Worker first:
```bash theme={null}
bun wrangler deploy --config apps/backend/wrangler.jsonc
```
5. Bootstrap TanStack Worker secrets:
```bash theme={null}
bun wrangler secret put BACKEND_PUBLIC_ORIGIN --config apps/tanstack-web/wrangler.jsonc
bun wrangler secret put BACKEND_INTERNAL_TOKEN --config apps/tanstack-web/wrangler.jsonc
```
6. Generate Worker types and deploy the TanStack Worker:
```bash theme={null}
bun --cwd apps/tanstack-web run cf-typegen
bun --cwd apps/tanstack-web run deploy:cloudflare
```
7. Smoke both Worker origins:
```bash theme={null}
BACKEND_INTERNAL_TOKEN="${BACKEND_INTERNAL_TOKEN:?set BACKEND_INTERNAL_TOKEN}" \
BACKEND_WORKER_ORIGIN=https:// \
TANSTACK_WEB_WORKER_ORIGIN=https:// \
bun smoke:cloudflare
```
For secret rotation on an already serving Worker, prefer the versions flow:
```bash theme={null}
bun wrangler versions secret put BACKEND_INTERNAL_TOKEN --config apps/backend/wrangler.jsonc
bun wrangler versions deploy --config apps/backend/wrangler.jsonc
```
Rollback is Worker-version or route/DNS based:
```bash theme={null}
bun wrangler deployments list --config apps/backend/wrangler.jsonc
bun wrangler rollback --config apps/backend/wrangler.jsonc
bun wrangler deployments list --config apps/tanstack-web/wrangler.jsonc
bun wrangler rollback --config apps/tanstack-web/wrangler.jsonc
```
External references: [TanStack Start hosting](https://tanstack.com/start/v0/docs/framework/react/guide/hosting),
[Cloudflare TanStack Start](https://developers.cloudflare.com/workers/framework-guides/web-apps/tanstack-start/),
[Wrangler configuration](https://developers.cloudflare.com/workers/wrangler/configuration/),
and [Cloudflare Worker secrets](https://developers.cloudflare.com/workers/configuration/secrets/).
## Vercel Frontend Build Validation
Vercel is used to validate that `apps/tanstack-web` can produce Vercel-compatible
TanStack Start output. This repository does not deploy or publish the TanStack
frontend to Vercel. The Rust backend must still be reachable over HTTPS during
the build, usually through the Cloudflare backend Worker or a self-hosted
backend origin. Vercel mode uses the app-local
`TANSTACK_WEB_RUNTIME=vercel` branch and Nitro, matching the current Vercel
TanStack Start guidance.
Configure the Vercel project:
| Setting | Value |
| ---------------- | ------------------------------------------------------------------------------------ |
| Project root | `apps/tanstack-web` |
| Build command | `bun run build:vercel` |
| Framework preset | Other, unless Vercel auto-detects the TanStack/Nitro output correctly |
| Git integration | Disabled by `apps/tanstack-web/vercel.json`; GitHub Actions runs `vercel build` only |
Required Vercel Project Environment Variables:
| Variable | Value source |
| -------------------------------------- | ------------------------------------------------------------------------ |
| `TANSTACK_WEB_RUNTIME` | `vercel` |
| `BACKEND_PUBLIC_ORIGIN` | HTTPS backend origin |
| `BACKEND_INTERNAL_URL` | Same HTTPS backend origin unless a separate private origin exists |
| `BACKEND_INTERNAL_TOKEN` | Same token configured on the backend |
| `NEXT_PUBLIC_SUPABASE_URL` | Supabase project URL, if the rendered routes need Supabase client config |
| `NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY` | Supabase publishable key, if routes need browser Supabase config |
Manual CLI sequence from the repo root:
```bash theme={null}
bash scripts/ci/run-with-backoff.sh bun install
bash scripts/ci/run-with-backoff.sh bun install --global vercel@latest
VERCEL_ORG_ID="${VERCEL_ORG_ID:?set VERCEL_ORG_ID}" \
VERCEL_PROJECT_ID="${VERCEL_TANSTACK_WEB_PROJECT_ID:?set VERCEL_TANSTACK_WEB_PROJECT_ID}" \
vercel pull --yes --environment=preview --token="${VERCEL_TOKEN:?set VERCEL_TOKEN}"
TANSTACK_WEB_RUNTIME=vercel \
bun turbo:local run build '--filter=@tuturuuu/tanstack-web^...'
bun run --silent scripts/ci/generate-build-metadata.ts
TANSTACK_WEB_RUNTIME=vercel \
VERCEL_ORG_ID="${VERCEL_ORG_ID:?set VERCEL_ORG_ID}" \
VERCEL_PROJECT_ID="${VERCEL_TANSTACK_WEB_PROJECT_ID:?set VERCEL_TANSTACK_WEB_PROJECT_ID}" \
vercel build --token="${VERCEL_TOKEN:?set VERCEL_TOKEN}"
```
Production uses the same sequence with `vercel pull --environment=production`
and `vercel build --prod`. It intentionally does not run `vercel deploy`.
GitHub Actions owns the normal path:
* `.github/workflows/vercel-preview-tanstack-web.yaml`
* `.github/workflows/vercel-production-tanstack-web.yaml`
The workflows use `VERCEL_TANSTACK_WEB_PROJECT_ID`, `VERCEL_ORG_ID`, and
`VERCEL_TOKEN` inside the build jobs only. Preview builds require
`workflow_dispatch` from protected `main` plus a trusted actor. Production builds
run from the `production` branch. They record build markers after `vercel build`
passes and do not create Vercel deployments.
External references: [Vercel TanStack Start](https://vercel.com/docs/frameworks/full-stack/tanstack-start),
[vercel pull](https://vercel.com/docs/cli/pull),
[vercel build](https://vercel.com/docs/cli/build).
## VPS Or Self-Hosting With Cloudflare Tunnel
Self-hosting uses the existing production Docker stack. Use this when the full
platform proxy, blue/green deployment history, watcher recovery, and
Cloudflare Tunnel container should own traffic.
1. Prepare the server:
```bash theme={null}
git clone https://github.com/tutur3u/platform.git tuturuuu
cd tuturuuu
bun install
```
2. Put production env values in root `.env.local` or an explicit deployment env
file. Set the frontend selector and backend token:
```bash theme={null}
DOCKER_WEB_FRONTEND=tanstack
BACKEND_INTERNAL_TOKEN=
```
3. If using Cloudflare Tunnel, create a remotely managed tunnel in Cloudflare
Zero Trust, add a public hostname for the desired domain, and route it to:
```text theme={null}
http://localhost:7803
```
Store the tunnel token as `CF_TUNNEL_TOKEN`, `CLOUDFLARED_TOKEN`, or
`DOCKER_CLOUDFLARED_TOKEN` in the server env file. The Docker helper maps
`CF_TUNNEL_TOKEN` to the Compose `CLOUDFLARED_TOKEN`.
4. Start the TanStack production stack without a tunnel:
```bash theme={null}
DOCKER_WEB_FRONTEND=tanstack bun serve:web:docker:bg
```
Start it with the bundled Cloudflare Tunnel sidecar:
```bash theme={null}
DOCKER_WEB_FRONTEND=tanstack bun serve:web:docker:bg -- --with-cloudflared
```
A non-empty `CF_TUNNEL_TOKEN` in root `.env.local` also auto-enables the
tunnel sidecar unless the Docker helper is explicitly opted out.
5. Verify through the local proxy:
```bash theme={null}
curl -fsS http://127.0.0.1:7803/__platform/drain-status
```
6. Verify through the public hostname after Tunnel reports healthy:
```bash theme={null}
curl -fsS https:///
```
7. Stop the stack:
```bash theme={null}
DOCKER_WEB_FRONTEND=tanstack bun serve:web:docker:bg:down
```
Rollback options:
* Blue/green: use the existing cached rollback path documented in
[Web Docker Deployment](/build/devops/web-docker-deployment).
* Git: check out the previous known-good commit and rerun
`DOCKER_WEB_FRONTEND=tanstack bun serve:web:docker:bg`.
* Tunnel: remove or change the Cloudflare public hostname route while the local
stack is repaired.
External references: [Cloudflare Tunnel setup](https://developers.cloudflare.com/tunnel/setup/)
and [Cloudflare remote tunnel creation](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/get-started/create-remote-tunnel/).
## Verification Checklist
Run these before handing off a local or deployment-config change:
```bash theme={null}
python3 -m json.tool apps/docs/docs.json apps/tanstack-web/vercel.json
node --test scripts/check-cloudflare-workers.test.js scripts/ci/check-workflow-config.test.js scripts/ci/release-workflows.test.js
bun check:cloudflare
bun type-check:tanstack-web
git diff --check
bun check
```
Do not run deploy commands, production Supabase pushes, or long build commands
from an implementation session unless the user explicitly requested live
deployment or build execution.
# Web Docker Deployment
Source: https://docs.tuturuuu.com/build/devops/web-docker-deployment
Run the web app in Docker for development, production, and blue/green self-hosted rollout.
This is the operational guide for the Docker-based `apps/web` runtime and the
parallel `apps/tanstack-web` migration runtime.
## Files That Define The Stack
* `apps/web/Dockerfile`
* `apps/web/docker/blue-green-watcher.Dockerfile`
* `apps/web/docker/cron-runner.Dockerfile`
* `apps/web/cron.config.json`
* `apps/tanstack-web/Dockerfile`
* `apps/backend/Dockerfile`
* `apps/meet-realtime/Dockerfile`
* `apps/supermemory/Dockerfile`
* `docker-compose.web.yml`
* `docker-compose.web.prod.yml` (Compose `include` entry that merges `docker-compose/compose.web.prod.*.yml` fragments plus shared `secrets` / `volumes`)
* `scripts/sync-web-crons.js`
* `scripts/watch-web-crons.js`
* `scripts/docker-web.js`
* `scripts/check-docker-web.js`
## Supported Commands
| Command | Purpose |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bun dev:web:docker` | Run the web dev workflow inside Docker |
| `bun devx:web:docker` | Explicitly start local Supabase, then the Docker dev workflow |
| `bun devrs:web:docker` | Explicitly start and reset local Supabase, then the Docker dev workflow |
| `bun dev:web:docker:down` | Stop the Docker dev workflow |
| `bun serve:web:docker` | Build and run the production web image in-place |
| `bun serve:web:docker:bg` | Blue/green production deploy with health-checked cutover |
| `bun serve:web:docker:bg:watch` | Recreate the watcher container, then tail its live logs while it polls the tracked branch and auto-runs blue/green after a successful fast-forward pull |
| `bun serve:web:docker:down` | Stop the production Docker stack |
| `bun serve:web:docker:bg:down` | Stop the blue/green stack and clear local runtime state |
| `bun test:e2e` / `bun test:e2e:web:docker` | Start local Supabase, reset it, run the production blue/green Docker web stack, then run Playwright |
| `bun benchmark:web-setups` | Compare reachable Next.js, TanStack Start, and Rust backend routes and write reports under `tmp/benchmarks/web-migration/` |
| `bun migration:tanstack:gates` | Validate route parity, Docker E2E evidence, and benchmark evidence before TanStack/Rust cutover |
| `bun check:docker` | Validate Dockerfile and compose parity rules |
| `bun check:cloudflare` | Validate TanStack Start and Rust backend Wrangler deploy configs without contacting Cloudflare |
## Flags And Implicit Mappings
| Flag | Meaning |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `--without-redis` | Disable the bundled Redis profile and skip Docker-injected Redis env |
| `--with-cloudflared` | Enable the bundled Cloudflare Tunnel container profile |
| `--with-supabase` | Start local Supabase before the Docker web flow |
| `--reset-supabase` | Start and reset local Supabase before the Docker web flow |
| `--env-file tmp/e2e/web.env` | Use an explicit Docker web env file for build secrets and runtime env files |
| `--mode prod` | Use the production compose file instead of the dev stack |
| `--strategy blue-green` | Use blue/green production deployment instead of in-place replacement |
| `--profile redis` | Explicitly enable the Redis profile when calling the helper directly |
| `--profile cloudflared` | Explicitly enable the Cloudflare Tunnel profile when calling the helper directly |
| `--build-memory 4g` | Run builds through a capped Buildx builder with a memory ceiling |
| `--build-cpus 4` | Run builds through a capped Buildx builder with an approximate CPU limit |
| `--build-max-parallelism 2` | Limit concurrent BuildKit solve steps for lower build pressure |
| `--build-builder-name tuturuuu` | Override the throttled Buildx builder name |
| `--resume-if-running` | If another watcher PID already holds the lock, mirror its live dashboard instead of failing |
| `--replace-existing` | If another watcher PID already holds the lock, stop it and take over |
| `--if-locked <fail|resume|replace>` | Explicit lock-conflict policy for the watcher |
| Command | Implicit flags |
| --------------------------------------- | ----------------------------------- |
| `bun dev:web:docker` | none |
| `bun devx:web:docker` | `--with-supabase` |
| `bun devrs:web:docker` | `--reset-supabase` |
| `bun serve:web:docker` | `--mode prod` |
| `bun serve:web:docker:bg` | `--mode prod --strategy blue-green` |
| `bun dev:web:docker -- --without-redis` | `--without-redis` |
## Runtime Requirements
* `.env.local` should be the primary Docker env file. The helper still falls back to `apps/web/.env.local` for older hosts that have not moved their env yet.
* When `--env-file` is provided, the Docker helper uses that file for the
Dockerfile secret and for the Compose runtime `env_file` entries. This keeps
special-purpose runs such as E2E from accidentally inheriting a developer's
cloud Supabase `.env.local`.
* Production Compose fragments live under `docker-compose/`, so their relative
host paths must be written from that directory. Use `..` to reach the repo
root for build contexts, env files, and bind mounts; otherwise Docker Compose
resolves paths like `apps/...` as `docker-compose/apps/...` and the watcher
image fails before the deployment loop starts.
* Docker BuildKit must be available. The helper sets
`COMPOSE_DOCKER_CLI_BUILD=1`, `DOCKER_BUILDKIT=1`, and
`BUILDX_NO_DEFAULT_ATTESTATIONS=1` so local blue/green image exports do not
stall while resolving default provenance metadata.
* Web, Hive, and TanStack Docker builder stages mount `/workspace/.turbo` so
local Turbo hits survive across Docker builds. They also read optional
BuildKit secrets named `turbo_token`, `turbo_team`, `turbo_api`, and
`turbo_remote_cache_signature_key` only for build `RUN` steps. When those
secrets are absent, builds fall back to the mounted local cache.
* GitHub-hosted Docker verification and E2E use service-specific `type=gha`
BuildKit scopes so the same backend or frontend layers can be reused across
workflows. Every shard may restore. Only shard 1 on the default branch may
export a true miss, with a bounded timeout and `ignore-error=true`, preventing
parallel shards from competing to save the same scope. Use `mode=max` for the
expensive web, TanStack, and backend scopes; use `mode=min` or restore-only
caching for small leaf services.
* Manual `docker buildx` and Compose invocations do not receive the GitHub cache
runtime variables automatically. Every CI job that uses `type=gha` outside
`docker/build-push-action` must first run the pinned
`crazy-max/ghaction-github-runtime` step so `ACTIONS_RUNTIME_TOKEN` and
`ACTIONS_RESULTS_URL` are available to BuildKit. Keep the action pinned to a
full commit SHA and covered by the CI cache-policy tests.
* Forward GitHub remote-cache identity to Docker only with BuildKit `--secret`
mounts. Never pass Turbo tokens as build arguments, Compose environment baked
into an image, labels, or files copied into a layer. Pull requests and
Dependabot builds receive no remote token and continue with local BuildKit and
Turbo fallback behavior.
* The production and TanStack-dual Compose files accept restore entries through
`DOCKER_WEB_CACHE__FROM` and optional exporter entries through the
matching `_TO` variable. Supported service tokens are `WEB`, `TANSTACK`,
`BACKEND`, `HIVE`, `SUPERMEMORY`, `MARKITDOWN`, `STORAGE_UNZIP`,
`CHAT_REALTIME`, `HIVE_REALTIME`, and `MEET_REALTIME`. CI configures shared
`type=gha` scopes; local operators normally leave these variables unset.
* `DOCKER_WEB_TURBO_TEAM_SECRET_FILE` and
`DOCKER_WEB_TURBO_TOKEN_SECRET_FILE` point Compose at short-lived files used
as BuildKit secrets. With no configured credentials they resolve to the
committed zero-byte `docker-compose/empty-secret`, so local and cross-platform
Compose parsing remains inert. Never put a token in that placeholder.
* Rust backend cache cleanup is local maintenance, not a separate cache service.
`bun rust-cache report` prints current `apps/backend/target` usage,
`bun rust-cache prune --apply` removes stale or oversized target entries, and
`bun rust-cache auto` runs at most once every 24 hours by default using
`tmp/rust-cache/state.json`. The default policy is local-only, skips CI,
keeps target entries newer than 14 days when possible, and starts pruning when
the repo-owned target cache exceeds 20 GiB.
* The dependency stages in `apps/web/Dockerfile`, `apps/hive/Dockerfile`,
`apps/hive-realtime/Dockerfile`, `apps/meet-realtime/Dockerfile`,
`apps/chat-realtime/Dockerfile`, and `apps/supermemory/Dockerfile` must copy
every `apps/*/package.json` and `packages/*/package.json` manifest before
running any frozen Bun install. Filtered Bun installs in these production
images are retry-wrapped and clear the Bun install cache before retry; keep
those snippets in sync with `scripts/check-docker-web.js`. Adding a new
workspace app or package without updating those lists makes Docker-only
installs try to rewrite `bun.lock`. `bun check:docker` validates this manifest
parity. `apps/backend` is an independent Rust crate, and `apps/meet-realtime`
is intentionally not a workspace package; their Dockerfiles do not require
`bun.lock` changes when service source changes.
* The Docker web flow does not start local Supabase unless you explicitly choose `bun devx:web:docker` or `bun devrs:web:docker`.
* Production Docker serving commands (`bun serve:web:docker`, `bun serve:web:docker:bg`, and `bun serve:web:docker:bg:watch`) prefer root `.env.local` when it exists, even if the shell inherited stale `DOCKER_WEB_ENV_FILE` or `DOCKER_WEB_COMPOSE_*ENV_FILE` values. Passing `--env-file ` is the explicit way to use another deployment env file.
* `ttr box setup` intentionally writes local Supabase values into app-local env files such as `apps/web/.env.local`. Those files are valid for devbox/local work but must not be the effective production watcher env. Keep root `.env.local` or the explicit deployment env file pointed at the cloud Supabase project.
* Production Docker serving refuses local Supabase origins (`localhost`, `127.0.0.1`, `::1`, `host.docker.internal`, and local Supabase ports) unless `DOCKER_WEB_ALLOW_LOCAL_SUPABASE=1` is set for a local production-image rehearsal.
* `log-drain-postgres` stores deployment telemetry only. Production startup
explicitly starts it from `scripts/docker-web.js` and retries once after
removing only the service container, but web and watcher services do not
declare a Compose `depends_on` relationship to it. An unhealthy log-drain
database no longer blocks web promotion by default: the helper continues with
`PLATFORM_LOG_DRAIN_ENABLED=false`, prints recent service state/log
diagnostics, and leaves the `platform-log-drain-postgres` volume intact. Set
`DOCKER_WEB_LOG_DRAIN_REQUIRED=1` only when telemetry storage must be a hard
promotion gate. If diagnostics mention incompatible database files or data
directory corruption, back up or migrate the Compose volume first; do not run
`docker compose down --volumes` or remove the volume without explicit
operator approval.
* Non-production Docker helpers still rewrite an explicitly local server-side Supabase URL to `host.docker.internal` while leaving `NEXT_PUBLIC_SUPABASE_URL` alone for browsers.
* Dockerized web services set `__NEXT_PRIVATE_ORIGIN` from
`DOCKER_WEB_NEXT_PRIVATE_ORIGIN`, defaulting to
`http://127.0.0.1:7803`. This keeps Next.js Server Action forwarding on the
in-container web listener even when nginx preserves an external `Host`.
If logs show `failed to forward action response` or
`UND_ERR_HEADERS_TIMEOUT`, verify the running web container has
`__NEXT_PRIVATE_ORIGIN=http://127.0.0.1:7803` or an intentional internal
override. Do not use `serverActions.allowedOrigins` as the primary fix for
this symptom; that setting controls Server Action origin/host validation, not
the forwarded-action fetch URL.
* If logs still show `Error checking if workspace is personal` with `[locale]`,
verify the running image includes commit `b30d7e2b07` or newer plus the shared
`@tuturuuu/utils` UUID guard.
* Dockerized web commands auto-enable the local Redis companion stack and inject `UPSTASH_REDIS_REST_URL` plus a generated `UPSTASH_REDIS_REST_TOKEN` into the web container.
* Dockerized production commands generate `BACKEND_INTERNAL_TOKEN` when one is
not provided and inject `BACKEND_INTERNAL_URL=http://backend:7820` for the
Rust backend service. Dev Compose uses the same internal URL with a local
fallback token. The same Rust HTTP core is prepared for later Cloudflare
Workers deployment through `apps/backend/wrangler.jsonc`; validate the Worker
deploy contract with `bun check:cloudflare`.
* `apps/backend/Dockerfile` copies
`apps/tanstack-web/migration/route-manifest.json` before `cargo build`
because the Rust migration endpoints include that checked manifest at compile
time. Regenerate the manifest before Docker validation when legacy route
inventory changes.
* The TanStack migration services use `apps/tanstack-web/Dockerfile`, direct
host port `7824`, and the Portless browser origin
`https://tanstack.tuturuuu.localhost:1355`.
Production Compose defines `tanstack-web`, `tanstack-web-blue`, and
`tanstack-web-green` so benchmark and cutover checks can run beside the
legacy Next.js lanes. Docker web E2E aliases the TanStack Portless route to
the `web-proxy` host port `7803` in `DOCKER_WEB_FRONTEND=tanstack` mode,
because the blue/green TanStack lanes are exposed through nginx rather than
the standalone `tanstack-web` host port. Production TanStack services wait on
the Rust
backend `service_healthy` check before starting, because the Start runtime
uses `BACKEND_INTERNAL_URL=http://backend:7820` for server-owned API calls.
The TanStack image and production Compose healthchecks run
`apps/tanstack-web/docker/healthcheck.mjs`, which requires the local
TanStack HTTP runner to answer below `500` and the configured
`BACKEND_INTERNAL_URL` to pass `/healthz`. The TanStack Node runner also
answers `/__platform/drain-status` directly so the shared nginx
`web-proxy` healthcheck can use the same internal readiness path in
`DOCKER_WEB_FRONTEND=tanstack` mode. Host-side Docker E2E readiness probes
the TanStack root route instead, because nginx denies external access to the
internal drain-status path.
Use `DOCKER_WEB_FRONTEND=next|tanstack` to select the default web frontend in
migration-aware scripts. The default is still `next`; setting `tanstack`
keeps nginx on the public `web-proxy:7803` listener but routes the active
upstream to `tanstack-web-blue:7824` or `tanstack-web-green:7824`. Set the
variable in the host/root deployment env before creating
`web-blue-green-watcher`, `web-docker-control`, or `web-cron-runner`, because
those containers inherit it for status probes, cached recovery, and service
recreation. Cloudflare
preview deploys use
`apps/tanstack-web/wrangler.jsonc` plus the app-local
`deploy:cloudflare`, `deploy:cloudflare:dry-run`, `preview:cloudflare`, and
`cf-typegen` scripts, so incremental TanStack route ports can be validated on
Workers before the full cutover gate passes.
* Cloudflare preview deployment is separate from the Docker blue/green
production stack. Use it to prove Worker compatibility for
`apps/tanstack-web` and `apps/backend`; do not route the production hostname
to preview Workers until the TanStack/Rust manifest, compare-mode Docker E2E,
benchmark report, and cutover gates pass.
* The production stack runs the first-party AI memory sidecar as an internal
support service at `http://supermemory:8787`. The service name and
`SUPERMEMORY_*` env names stay compatible with existing web runtime wiring,
but `apps/supermemory/Dockerfile` builds Tuturuuu-owned pgvector memory code.
* Dockerized production commands auto-configure the memory sidecar unless explicitly
disabled. `scripts/docker-web/env.js` generates and persists the internal
`SUPERMEMORY_API_KEY`, `SUPERMEMORY_POSTGRES_PASSWORD`, and
`SUPERMEMORY_DATABASE_URL`, and defaults `SUPERMEMORY_ENABLED=true`,
`SUPERMEMORY_FAIL_OPEN=true`, and
`SUPERMEMORY_TIMEOUT_MS=1500`.
* Operators can override generated values with `DOCKER_SUPERMEMORY_API_KEY`,
`DOCKER_SUPERMEMORY_POSTGRES_PASSWORD`,
`DOCKER_SUPERMEMORY_DATABASE_URL`,
or `DOCKER_SUPERMEMORY_ENABLED`; standard `SUPERMEMORY_*` env still works.
* Blue/green promotion health-gates `supermemory` with the rest of the support
services. Changing `apps/supermemory/`, the production Compose fragments, or
the Docker bake file refreshes the support service set.
Explicit `SUPERMEMORY_ENABLED=false` or `DOCKER_SUPERMEMORY_ENABLED=false`
removes that support service from blue/green builds, starts, and health gates
for local-only runs.
## Cloudflare Preview Path
Docker remains the production rollout and rollback mechanism while the
TanStack/Rust migration is incomplete. Cloudflare Workers are available now for
incremental preview validation:
1. Run the config preflight without contacting Cloudflare:
```bash theme={null}
bun check:cloudflare
```
Keep local Worker secret values in `apps/backend/.dev.vars` and
`apps/tanstack-web/.dev.vars`; those files are ignored by `.gitignore`.
`wrangler.jsonc` may name required secrets, but it must not contain secret
values or account-specific private origins.
2. Deploy the Rust backend Worker first:
```bash theme={null}
rustup target add wasm32-unknown-unknown
cargo install worker-build --locked
bun wrangler secret put BACKEND_INTERNAL_TOKEN --config apps/backend/wrangler.jsonc
bun wrangler secret put TUTURUUU_APP_COORDINATION_SECRET --config apps/backend/wrangler.jsonc
bun wrangler secret put SUPABASE_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put SUPABASE_SERVICE_ROLE_KEY --config apps/backend/wrangler.jsonc
bun wrangler secret put CRON_SECRET --config apps/backend/wrangler.jsonc
bun wrangler secret put DISCORD_APP_DEPLOYMENT_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put AURORA_EXTERNAL_URL --config apps/backend/wrangler.jsonc
bun wrangler secret put AURORA_EXTERNAL_WSID --config apps/backend/wrangler.jsonc
bun wrangler deploy --config apps/backend/wrangler.jsonc
```
`SUPABASE_URL` and `SUPABASE_SERVICE_ROLE_KEY` are required for the
Rust-owned contact/profile APIs. The backend uses them only server-side to
read `users` and `user_private_details` and insert `support_inquiries`
through Supabase REST.
`CRON_SECRET` and `DISCORD_APP_DEPLOYMENT_URL` are required for Rust-owned
Discord cron proxy preview readiness, and `AURORA_EXTERNAL_URL` with
`AURORA_EXTERNAL_WSID` is required for the Rust-owned Aurora health and
ingest probes; configure them even when the first smoke target is only
`/healthz`.
Use `wrangler secret put` for first preview bootstrap only. It creates and
deploys a new active Worker version when the secret changes. For rotations or
canary traffic, use:
```bash theme={null}
bun wrangler versions secret put BACKEND_INTERNAL_TOKEN --config apps/backend/wrangler.jsonc
bun wrangler versions secret put TUTURUUU_APP_COORDINATION_SECRET --config apps/backend/wrangler.jsonc
bun wrangler versions secret put SUPABASE_URL --config apps/backend/wrangler.jsonc
bun wrangler versions secret put SUPABASE_SERVICE_ROLE_KEY --config apps/backend/wrangler.jsonc
bun wrangler versions secret put CRON_SECRET --config apps/backend/wrangler.jsonc
bun wrangler versions secret put DISCORD_APP_DEPLOYMENT_URL --config apps/backend/wrangler.jsonc
bun wrangler versions secret put AURORA_EXTERNAL_URL --config apps/backend/wrangler.jsonc
bun wrangler versions secret put AURORA_EXTERNAL_WSID --config apps/backend/wrangler.jsonc
bun wrangler versions deploy --config apps/backend/wrangler.jsonc
```
3. Bind the TanStack Worker to the backend Worker by service binding and keep
secret values out of source. `apps/tanstack-web/wrangler.jsonc` declares the
`BACKEND` service binding to `tuturuuu-backend`; `BACKEND_INTERNAL_URL` is
only an optional HTTP fallback for local or emergency non-binding runs:
```bash theme={null}
bun wrangler secret put BACKEND_PUBLIC_ORIGIN --config apps/tanstack-web/wrangler.jsonc
bun wrangler secret put BACKEND_INTERNAL_TOKEN --config apps/tanstack-web/wrangler.jsonc
```
Use the versions flow for TanStack Worker secret rotations too:
```bash theme={null}
bun wrangler versions secret put BACKEND_PUBLIC_ORIGIN --config apps/tanstack-web/wrangler.jsonc
bun wrangler versions secret put BACKEND_INTERNAL_TOKEN --config apps/tanstack-web/wrangler.jsonc
bun wrangler versions deploy --config apps/tanstack-web/wrangler.jsonc
```
4. Deploy the TanStack Start Worker:
```bash theme={null}
bun --cwd apps/tanstack-web run cf-typegen
bun --cwd apps/tanstack-web run deploy:cloudflare
```
5. Smoke both returned Worker origins:
```bash theme={null}
BACKEND_INTERNAL_TOKEN="${BACKEND_INTERNAL_TOKEN:?set BACKEND_INTERNAL_TOKEN}" \
BACKEND_WORKER_ORIGIN=https:// \
TANSTACK_WEB_WORKER_ORIGIN=https:// \
bun smoke:cloudflare
```
Keep `BACKEND_INTERNAL_TOKEN` in the shell, ignored local env, or Wrangler
secret storage only. The smoke report must include the positive authenticated
migration-status probe and the missing/invalid-token rejection probes before
the Cloudflare cutover gate can pass.
Security expectations for this path:
* Keep `BACKEND_INTERNAL_TOKEN`, backend origins, and future service credentials
in Wrangler/Cloudflare bindings or ignored local env only. `wrangler.jsonc`
may list variable names and non-secret preview defaults, never secret values.
* Browser code must not receive backend bearer tokens. Protected workspace,
private schema, cron, job, and admin routes stay server-owned through Rust
endpoints and `packages/internal-api` / TanStack server functions.
* Re-check CORS, cookie domain, secure-cookie, `SameSite`, and session-origin
behavior before adding a custom hostname; `workers.dev`, local Portless, and
production hosts are different browser origins.
* Keep Cloudflare preview in `BACKEND_ENV=preview`; development-only migration
bypasses are local-only.
Rollback from this preview path is DNS/routing-based: remove the Cloudflare
route or custom domain that points at the preview Worker, or redeploy the last
known-good Worker version, while the Docker blue/green `apps/web` production
stack keeps serving the canonical hostname. Do not delete Worker secrets unless
the secret value is compromised. Use Wrangler to inspect and roll back preview
Worker versions:
```bash theme={null}
bun wrangler deployments list --config apps/backend/wrangler.jsonc
bun wrangler deployments status --config apps/backend/wrangler.jsonc
bun wrangler rollback --config apps/backend/wrangler.jsonc
bun wrangler deployments list --config apps/tanstack-web/wrangler.jsonc
bun wrangler deployments status --config apps/tanstack-web/wrangler.jsonc
bun wrangler rollback --config apps/tanstack-web/wrangler.jsonc
```
Worker rollback does not revert external resources, bindings, routes, custom
domains, or secret values. Re-run `bun smoke:cloudflare` against the remaining
preview origins before resuming canary traffic.
## Dockerized E2E
`bun test:e2e` from the repo root and `bun test:e2e` in `apps/web` run through
`scripts/run-web-e2e-docker.js` instead of starting `next dev`. The runner:
1. writes `tmp/e2e/web.env` with local-only Supabase, local app-origin
variables, app-session JWT values, and a local-only E2E auth bypass for
Turnstile/dev-session,
2. starts and resets the Dockerized local Supabase stack,
3. boots `apps/web` through the production blue/green Docker flow,
4. starts Portless on unprivileged HTTPS port `1355` and registers the
`https://tuturuuu.localhost:1355` route only after the direct Docker proxy is
healthy,
5. waits for `https://tuturuuu.localhost:1355/login`, then runs Playwright
against that shared-cookie origin, and
6. tears down Docker web plus local Supabase unless `E2E_KEEP_DOCKER_STACK=1`.
Full web E2E runs also start satellite apps on their native ports when the
selected Playwright tests exercise routes those apps own. The runner currently
registers Tasks, Forms, and Infrastructure as
`https://.tuturuuu.localhost:1355` routes through the shared Portless proxy
and waits for each app on its direct loopback port before Playwright begins.
The direct readiness probe avoids trusting local TLS in the Node runner, while
the browser still exercises the shared HTTPS origins. This lets cross-app
scenarios use the same Apps picker, token exchange, shared-session cookie, and
satellite UI that users operate, while direct API coverage reaches the app that
actually owns the route.
Before starting a satellite, the runner builds its compiled workspace
dependencies so a fresh CI checkout matches a prepared local checkout. For
sharded runs, it asks Playwright which tests belong to the current shard and
starts only the satellites required by that shard, so suite redistribution
cannot strand a spec or make unrelated shards pay the startup cost. Set the
matching `E2E_TASKS_SATELLITE_ENABLED`, `E2E_FORMS_SATELLITE_ENABLED`, or
`E2E_INFRASTRUCTURE_SATELLITE_ENABLED` variable to `0` only when a focused local
run cannot reach that satellite and does not exercise its routes.
Pass `--frontend next`, `--frontend tanstack`, or `--frontend compare` to select
the legacy Next.js host, the TanStack host, or both sequentially:
```bash theme={null}
bun test:e2e:web:docker -- --frontend compare
```
The TanStack mode reuses the existing `apps/web/e2e` suite and points Playwright
at `https://tanstack.tuturuuu.localhost:1355`. In Docker web E2E, that Portless
host points at the `web-proxy` host port because the runner validates the
production blue/green TanStack lane selected by `DOCKER_WEB_FRONTEND=tanstack`.
Standalone TanStack checks such as `docker-compose.tanstack-dual.yml` still use
the direct TanStack Docker port `7824`.
Compare mode writes `tmp/e2e/web-migration/compare-report.json` after both
frontend runs complete. Set `E2E_COMPARE_REPORT_PATH` to redirect that evidence
file under another ignored `tmp/` location. The report records the normalized
Next and TanStack frontend origins used by the run plus per-frontend Playwright
test counts from JSON reporter output. The cutover gate rejects missing,
credentialed, invalid, or same-origin compare evidence, and it also rejects
reports that do not prove nonzero Playwright execution for both frontends before
they can satisfy terminal migration gates.
Before cutover, pair the compare-mode E2E evidence with a full benchmark report
and Cloudflare smoke report:
```bash theme={null}
BACKEND_INTERNAL_TOKEN="${BACKEND_INTERNAL_TOKEN:?set BACKEND_INTERNAL_TOKEN}" \
bun benchmark:web-setups -- \
--setup compare \
--profile full \
--require-all \
--next-origin https:// \
--tanstack-origin https:// \
--backend-origin https://
```
The benchmark report is cutover evidence, so the origins are part of the gate.
The benchmark command rejects same-origin Next/TanStack compare runs, and the
cutover gate rejects reports with missing origins or route/sample URLs that do
not match the recorded setup origin. This catches accidental proxy reuse before
the report can satisfy migration gates.
```bash theme={null}
bun migration:tanstack:gates -- \
--e2e-report tmp/e2e/web-migration/compare-report.json \
--benchmark-report tmp/benchmarks/web-migration//report.json \
--cloudflare-smoke-report tmp/benchmarks/web-migration//cloudflare-smoke.json \
--output tmp/benchmarks/web-migration//cutover-gates.json
```
The gate command does not start Docker. It only validates explicit report files
so generated evidence remains under ignored `tmp/` paths. The output JSON is the
review/handoff artifact that links the Docker E2E, benchmark, and Cloudflare
smoke evidence for sign-off without committing generated reports.
Normal teardown passes `--volumes --rmi local` to Docker Compose and then removes
custom image tags for the current `ttr-e2e-*` project, so per-run containers,
Compose volumes, and baked blue/green images do not accumulate. E2E also sets
`DOCKER_WEB_BUILDKIT_PRUNE_AFTER_BUILD=1` and
`DOCKER_WEB_BUILDKIT_PRUNE_MODE=all` by default because the per-run BuildKit
cache/state is disposable; set `E2E_DOCKER_BUILDKIT_PRUNE_AFTER_BUILD=0` only
when debugging a local E2E build and you intentionally want to keep BuildKit
cache.
Local E2E also starts Supabase with `edge-runtime` excluded by default through
`DOCKER_WEB_SUPABASE_START_EXCLUDE=edge-runtime`. The platform E2E suite does not
serve local Edge Functions, and excluding that service keeps local runs from
failing when the Supabase Edge Runtime tries to resolve external JSR packages.
Set `E2E_SUPABASE_START_EXCLUDE=` when you intentionally need a full local
Supabase stack for debugging.
Local E2E also pins `DOCKER_SUPERMEMORY_ENABLED=false` and
`SUPERMEMORY_ENABLED=false`; the memory integration is not under test there.
E2E build caps default to `auto` for memory, CPU, and BuildKit max parallelism.
The runner reads Docker's current `MemTotal` before booting the stack, forwards
that value as `DOCKER_WEB_DOCKER_MEMORY_LIMIT`, and resolves the BuildKit memory
cap just under the active Docker Desktop allocation. On allocations below 10 GB,
the E2E runner keeps the inner Next build at one CPU, static generation
concurrency one, and a 4 GB Node heap so BuildKit keeps enough container
headroom. The Next build engine remains Turbopack. Do not switch local E2E runs
to the Webpack build path; the production and local Docker build paths are
expected to exercise the same Turbopack compiler.
For blue/green web deploys, the helper defaults to the Docker/BuildKit web image
build path so production builds keep their container isolation boundary. Set
`DOCKER_WEB_NATIVE_BUILD=1` only for an explicit operator-approved native host
build. That opt-in runs `bun run build:web:docker` on the host with
`DOCKER_WEB_STANDALONE=1`, packages `apps/web/.next/standalone` plus static
assets into the same Node runtime image shape, and continues with Docker Compose
startup and Playwright. Native builds derive their host-side build memory budget
from the machine's total memory, while leaving Next/Turbopack CPU and static
generation worker counts unset so the toolchain can auto-configure from the real
host. Set `DOCKER_WEB_NATIVE_BUILD_MEMORY=16g` or another explicit value only
when the host needs a different Node heap bucket. Native runner packaging uses
plain `docker build` by default so a remote BuildKit transport failure does not
block the host-built artifact path. It also strips builder-routing env such as
`BUILDX_BUILDER` for that packaging subprocess; set
`DOCKER_WEB_NATIVE_RUNNER_BUILDX=1` only when that packaging step must use the
configured buildx builder. Native mode skips support-service image builds by
default and reuses the existing support images; set
`DOCKER_WEB_NATIVE_SUPPORT_BUILD=1` to build support images locally with
`docker compose build`, or `DOCKER_WEB_NATIVE_SUPPORT_BUILDX=1` when those
support builds should use the configured buildx builder.
When a production blue/green web image was just built locally with
`bun serve:web:docker:bg`, Dockerized E2E can reuse that image instead of
building `apps/web` again:
```bash theme={null}
E2E_DOCKER_REUSE_WEB_IMAGE=1 bun test:e2e:web:docker -- e2e/multi-account.noauth.spec.ts --project=chromium-no-auth
```
The runner reads `tmp/docker-web/prod/active-color` and retags the matching
`tuturuuu-web-` image into the isolated `ttr-e2e-*` Compose project for
both `web-blue` and `web-green`, then skips the web build stage. When the same
local blue/green build also left the support images behind, the runner retags
`hive-`, `hive-realtime`, `backend`, `meet-realtime`, `markitdown`,
`storage-unzip-proxy`, `supermemory`, `web-docker-control`, and
`web-cron-runner` into the E2E project and skips those support builds too. If
any support image is missing, E2E falls back to the normal support-service build
path while still reusing the web image. Use this only when the source image was
built from the current checkout.
Set `E2E_DOCKER_REUSE_WEB_IMAGE_SOURCE=` to reuse a specific web image
tag, or `E2E_DOCKER_REUSE_WEB_IMAGE_COLOR=blue|green|auto` when the active-color
file is not the lane you want.
GitHub Actions builds the same image set once per E2E workflow run and shares it
through the private `ghcr.io/tutur3u/platform-e2e` package. The producer starts
after the CI switchboard check, and the Playwright and migration matrices wait
for that job to finish before GitHub allocates their runners. The producer
publishes immutable tags under
`---` and pushes the `-ready` tag only
after every planned blue/green and TanStack image exists and the package has
been verified private. Consumers wait up to
`E2E_IMAGE_BUNDLE_WAIT_SECONDS` (ten seconds by default), pull the complete
frontend plan, retag it for their isolated Compose project, and skip web/support
image builds. Playwright shards request only Next plus support images, the
TanStack dual-stack mode requests only TanStack plus support images, and compare
mode requests both frontends. Consumer jobs use `always()` at the job boundary,
so a failed best-effort producer still starts the existing local cache-backed
build path.
Missing or partial bundles emit a notice and fall back without leaving matrix
runners idle for minutes.
The workflow exposes only three bundle controls:
* `E2E_IMAGE_BUNDLE_REPOSITORY` is the private two-segment GHCR repository.
* `E2E_IMAGE_BUNDLE_TAG_PREFIX` identifies one workflow run and attempt.
* `E2E_IMAGE_BUNDLE_WAIT_SECONDS` bounds how long a consumer may wait before
falling back.
The producer uses deterministic E2E-only `PLATFORM_BUILD_*` metadata so commit
timestamps and messages do not invalidate otherwise reusable build work;
production release metadata remains unchanged. Published cache images carry the
official `org.opencontainers.image.source` config label, and run indexes retain
the corresponding OCI annotation, so GHCR links the package to this repository
and grants its workflow token package administration. The
privacy check retries briefly while GitHub's package metadata catches up. An
`always()` cleanup job deletes the exact run prefix after all shards and
migration modes finish, and the next producer removes abandoned bundle versions
older than 24 hours to cover canceled workflows. Before publishing run-scoped
tags, the producer creates a permanent `sentinel` image version when one does
not already exist. Later runs reuse it instead of producing untagged sentinel
versions. GHCR rejects deleting a package's final tagged version individually,
so the sentinel lets
exact-prefix cleanup remove every run tag without deleting the package or
touching another concurrent workflow. Bootstrap cleanup preserves one final
version when no sentinel exists yet; the next successful publish creates the
sentinel so a later stale sweep can remove it. Bundle publication is
best-effort, but cleanup is a required storage-safety gate. Investigate package
visibility, `packages` token permissions, or GHCR deletion errors when cleanup
fails; do not make the package public or replace these images with
long-retention Actions artifacts.
The producer also maintains one mutable `cache-` tag per planned image.
These tags anchor the latest runnable layers in private GHCR after run-scoped
tags are deleted, so the next commit does not have to upload unchanged final
image layers again. Superseded cache versions become untagged and remain covered
by the 24-hour stale sweep. Buildx writes cache images directly to GHCR instead
of exporting them through the runner's local Docker image store. Each successful
build records its immutable digest; registry-side manifest promotion creates a
distinct annotated run index from that digest, even if another workflow updates
the mutable cache tag concurrently. Exact-prefix cleanup can therefore delete
run indexes without deleting persistent cache tags. The ready index remains
last, so consumers never observe a partial bundle. These registry-layer anchors
complement Turbo and BuildKit caches without adding Actions cache entries or
artifact storage.
Trusted `main` producers also expose the Turbo team and token secret files to
the native web artifact child process. This lets both Docker/BuildKit stages and
the host-side Next build reuse Turborepo remote cache entries. The values remain
file-scoped until that child starts, are not passed as Docker build arguments,
and are never persisted in an image layer. Pull requests and other untrusted
runs continue without remote-cache credentials.
Each GitHub-hosted E2E runner starts from an empty Supabase volume. In CI,
`E2E_DOCKER_SUPABASE_RESET=0` therefore asks the runner to start Supabase once
and trust that initial bootstrap, which already applies migrations and seed
data. This avoids immediately repeating the same reset while preserving a
separate database stack for every shard and migration mode. Local E2E keeps the
safer reset-by-default behavior because a developer may already have mutable
Supabase volumes.
On GitHub-hosted runners, the E2E workflow frees disk before restoring or
loading cached Supabase Docker images. Keep that cleanup ahead of the cache load:
running `docker system prune -af --volumes` after cached images are loaded would
remove the images the shard is about to use, while skipping the cleanup can leave
too little space for the web Docker image dependency layer.
The default push workflow keeps the existing `cache` transport. To benchmark
the cache archive against direct registry pulls without changing healthy main
runs, dispatch **E2E Tests** manually with **Supabase image transport benchmark
mode** set to `registry`. Compare the `Restore cached Docker images`, `Load
cached Docker images`, and `Run Playwright shard` step durations with an
otherwise equivalent `cache` dispatch. Do not replace the default until three
successful paired runs show a consistent critical-path improvement; migration
replay itself is only a small part of Supabase startup time.
When an E2E shard fails, the runner prints diagnostics before teardown while the
containers still exist. The job log includes the primary error, blue/green stage
state, the Playwright `.last-run.json` file when available, Docker containers
for the shard Compose project, production Compose status, recent logs for web,
Hive, proxy, and support services, the Portless route list, a probe against the
configured E2E `BASE_URL`, and `bun sb:status`. The workflow also has a
failure-only diagnostic step after `Run Playwright shard` as a backstop, so the
job output should show the failing service or stage even when Playwright report
artifacts are incomplete. Before upload, the workflow rewrites the diagnostics
directory to redact secret-shaped key/value pairs, bearer tokens, sensitive
query parameters, JWTs, and secret-like runner environment values. The workflow
uploads diagnostics, Playwright reports, and `apps/web/test-results` for every
non-cancelled shard so traces and screenshots stay available when the job output
is too short.
The Playwright global setup refuses non-local web origins and refuses Supabase
origins outside `localhost`, `127.0.0.1`, or `host.docker.internal` on port
`8001`. CI shards E2E with `--shard=x/4`; each shard gets its own Compose
project name, but all shards still use ephemeral local Supabase rather than any
cloud Supabase project. The generated E2E env sets
`DOCKER_WEB_ALLOW_LOCAL_SUPABASE=1` so the production-image rehearsal can use
that local Supabase origin without weakening production serving defaults.
Because the Docker web app runs with
`NODE_ENV=production`, the generated env file and Playwright process env also
pin `WEB_APP_URL`, `NEXT_PUBLIC_WEB_APP_URL`, and `NEXT_PUBLIC_APP_URL` to
the local shared-cookie origin; otherwise central-auth redirects can escape to
the real `tuturuuu.com` origin during setup. The auth bypass is guarded by the
local E2E web origin, the incoming request `Host` / forwarded host / `Origin`
headers, and both the public and server-side Supabase origins before
server-side auth code honors it, so it must not be used as a general production
configuration.
The blue/green proxy and `apps/web` runtime both allow 64 KB request headers.
That headroom lets the browser reach `/~recover-browser-state` or the normal
login flow when duplicated Supabase cookies make the default header limit too
small. If a request is still too large for the proxy to forward, nginx handles
431/494 directly with `Clear-Site-Data` and redirects to
`/login?browserStateReset=1`; this recovery must stay in the proxy because
Next.js middleware cannot run after nginx rejects the header.
The blue/green nginx proxy must forward the original `Host` header with its
port intact via `$http_host`. Local E2E auth setup posts to
`http://localhost:7803/api/auth/dev-session`, and the production-mode app
accepts the setup route only when the public request origin stays local. The
guard also tolerates production standalone/proxy normalization where `request.url`
or `Host` becomes an internal Docker web upstream, but only when the forwarded
public host is still the local E2E origin.
## Coolify
Coolify can provide enough default deployment metadata for Tuturuuu's Dockerfile
setup to derive the app origin even when you do not manually define the usual
app URL variables.
* During Dockerfile builds, `scripts/build-web-docker.js` now derives missing
`WEB_APP_URL`, `NEXT_PUBLIC_WEB_APP_URL`, and `NEXT_PUBLIC_APP_URL` values
from Coolify's `COOLIFY_URL` or `COOLIFY_FQDN` defaults before running
`bun run build:web`.
* During production container startup, `apps/web/docker/prod-entrypoint.js`
applies the same Coolify fallback so server-side runtime code sees the same
derived values.
* The runtime URL resolvers used by the web proxy, internal API client, and
drive export/auto-extract flows also fall back to `COOLIFY_URL` and
`COOLIFY_FQDN`.
Recommended setup in Coolify:
* Still set explicit Tuturuuu env like `NEXT_PUBLIC_SUPABASE_URL`,
`NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY`, `SUPABASE_SECRET_KEY`, and any email
or storage secrets yourself.
* You can omit `WEB_APP_URL`, `NEXT_PUBLIC_WEB_APP_URL`, and
`NEXT_PUBLIC_APP_URL` if Coolify already injects `COOLIFY_URL` or
`COOLIFY_FQDN` for the deployment.
* If you need one specific canonical domain while Coolify exposes multiple
domains, set the Tuturuuu app URL variables explicitly instead of relying on
the automatic fallback.
## Development Mode
Development mode exists to preserve the normal root script contract while moving the
web runtime into containers.
* Container-managed `node_modules` are isolated from the host.
* Package-local `node_modules` and `dist` directories are also isolated so host installs do not shadow container artifacts.
* The root Docker context excludes generated app artifacts such as `.next`,
`.turbo`, coverage output, and Flutter build directories. Keep these excludes
intact so production builds do not stream multi-gigabyte local artifacts into
BuildKit.
* A host `bun install` is not required just to boot the Dockerized web stack.
## Production Mode
The production compose file uses the `runner` target from `apps/web/Dockerfile`.
### In-Place
```bash theme={null}
bun serve:web:docker
```
Use this when a short restart is acceptable.
### Blue/Green
```bash theme={null}
bun serve:web:docker:bg
```
Blue/green deploy does this:
1. Reads the last active color from `tmp/docker-web/prod/active-color`.
2. Ignores that state if the corresponding container no longer exists.
3. Builds the target web image through Docker Buildx Bake using Compose-derived
targets, then stops/removes only the old target web lane and starts the fresh
replacement.
4. Starts the target web lane after its healthcheck passes and records
`web-promote` as a staged target in
`tmp/docker-web/prod/target-state.json`. The deploy does not reload
`web-proxy` or write `tmp/docker-web/prod/active-color` yet.
5. Builds and runs Hive separately. `hive-db-migrate`, the target
`hive-blue`/`hive-green` service, `hive-realtime`, and the Hive proxy check
must pass before web can be publicly promoted. A migration or Hive health
failure marks the Hive stage failed, leaves `active-color` on the previous
web lane, and keeps the staged target web lane out of public routing.
6. Refreshes support services (`backend`, `meet-realtime`, `markitdown`,
`storage-unzip-proxy`, `web-docker-control`, and `web-cron-runner`) after
web/Hive target work.
A support build or health failure also blocks `web-proxy` reload and leaves
the previous active web lane serving.
Their build step is scoped:
ordinary web commits build only `web-blue` or `web-green`, while Hive and
helper images rebuild only when their source, Dockerfile, compose wiring, or
shared dependency inputs changed. Image-only services such as `redis`,
`serverless-redis-http`, `web-proxy`, and `cloudflared` are never passed to
Bake.
7. Injects Docker-internal helper URLs into `apps/web`:
`BACKEND_INTERNAL_URL=http://backend:7820`,
`MARKITDOWN_ENDPOINT_URL=http://markitdown:8000/markitdown`,
`DISCORD_APP_DEPLOYMENT_URL=http://markitdown:8000`,
`DRIVE_AUTO_EXTRACT_PROXY_URL=http://storage-unzip-proxy:8788/extract`, and
`INTERNAL_WEB_API_ORIGIN=http://web-proxy:7803`.
8. Keeps the stable `web-proxy` container running in place during ordinary
promotions instead of re-running `compose up` against the public `:7803`
listener. If the running proxy is missing required host ports or its
container image no longer matches the resolved Compose image, the deploy
defers the forced proxy recreate until after the target web, Hive, and
support gates have passed.
9. Validates the generated nginx config with `nginx -t`, then reloads or
recreates the proxy only after every staging gate has passed.
10. Immediately verifies the proxy can serve the internal
`/__platform/drain-status` endpoint through the newly routed color before
writing `active-color` and marking the staged web target healthy. This
avoids false deployment failures from public API middleware or rate limits.
11. Polls an internal drain-status endpoint on the old color and waits until it
has no in-flight HTTP work left before demoting it to standby. This keeps
long-running server actions, route handlers, and other open requests from
being cut off mid-flight.
12. Falls back to the short fixed drain window only when the old image predates
the drain-status endpoint and cannot report its active requests yet.
13. Keeps the demoted color online as a warm nginx backup target instead of
removing it immediately, so stale keepalive workers and Cloudflare Tunnel
connections can still fail over cleanly during the post-promotion window.
14. If the demoted standby color is still on the previous revision after 15
minutes, the watcher automatically rebuilds that stale standby in place so
both colors converge on the latest checked-out code without flipping the
active port or promoting traffic again.
During blue/green deploys, the watcher supplies the version badge metadata via
`PLATFORM_BUILD_*` variables for both Docker image builds and runtime
containers. It infers `PLATFORM_BUILD_COMMIT_HASH`,
`PLATFORM_BUILD_COMMIT_SHORT_HASH`, `PLATFORM_BUILD_COMMIT_MESSAGE`,
`PLATFORM_BUILD_REF_NAME`, `PLATFORM_BUILD_ENVIRONMENT`,
`PLATFORM_BUILD_BUILT_AT`, `PLATFORM_BUILD_DEPLOYMENT_URL`, and
`PLATFORM_BUILD_DEPLOYMENT_STAMP` from the current checkout plus the deployment
context. `PLATFORM_BUILD_BUILT_AT` is the checked-out commit's source timestamp,
so rebuilding the same commit and environment produces identical metadata; it
is not the image-build or rollout time. Deployment stamps and the watcher's
`startedAt`, `finishedAt`, and `activatedAt` records remain the source of actual
deployment timing. The build helper strips deployment stamps and deployment
URLs from Compose, Bake, and native build environments, then supplies them only
to the runtime container; different rollouts of the same source therefore reuse
the same Turbo and image layers. The account-gated badge reads those runtime
values before falling back to generated Vercel/GitHub metadata, so on-prem
watcher deployments show the served commit instead of `local` / `Unknown`.
If those `PLATFORM_BUILD_*` values are missing or blank in a self-hosted
runtime, `apps/web` falls back to the mounted blue/green snapshot before using
generated/local defaults. The resolver reads only lightweight snapshot files:
`prod/target-state.json`, `prod/active-color`, `prod/deployment-stamp`,
`watch/blue-green-auto-deploy.status.json`, and
`watch/blue-green-auto-deploy.history.json` under
`PLATFORM_BLUE_GREEN_MONITORING_DIR`, with local `tmp/docker-web` candidates for
development. Selection prefers `targets.web` for the active color, then an
active deployment row, then the latest successful row for the active color, then
the latest successful row overall. `commitSubject` becomes the badge commit
message, `committedAt` (or an explicit source timestamp) becomes `builtAt`, and
deployment timestamps are used only to order deployment candidates. Legacy rows
without a source timestamp never relabel rollout time as source time. The
runtime deployment stamp file supplies the displayed deployment stamp. The
resolver does not invent deployment URL, ref, or environment from color or
commit data alone.
The helper writes support-image input hashes to
`tmp/docker-web/prod/build-input-hashes.json` and keeps recent decisions in
`tmp/docker-web/prod/build-input-hashes.history.json`. Infrastructure
monitoring reads that history so deployment rows can show which helper images
were rebuilt and which ones were served from the cached build inputs.
### Meet Realtime
`apps/meet-realtime` is the internal control-plane service for Meet calls,
webinars, and low-latency broadcast coordination. It is a Bun WebSocket service
started by production Compose as `meet-realtime` on container port `7816`.
`web-proxy` exposes `/realtime` for `meet.tuturuuu.com` and
forwards WebSocket upgrades to that service.
Production meeting logic stays on Tuturuuu infrastructure:
* `apps/web` owns protected meeting APIs, verifies workspace access, and mints
short-lived `MEET_REALTIME_TOKEN_SECRET` join tokens.
* `apps/web` and `apps/meet-realtime` must share the same
`MEET_REALTIME_TOKEN_SECRET`. Production Compose exposes
`MEET_REALTIME_URL` and `NEXT_PUBLIC_MEET_REALTIME_URL` to Web, defaulting
both to `wss://meet.tuturuuu.com/realtime` for browser join-token payloads.
* Browsers connect to `wss://meet.tuturuuu.com/realtime?token=...`.
* `apps/meet-realtime` validates the token, manages ephemeral room presence,
chat, stage state, and reconnect resync, then calls Cloudflare Realtime SFU
APIs with server-only `CLOUDFLARE_REALTIME_APP_ID` and
`CLOUDFLARE_REALTIME_APP_SECRET`.
* Browser media flows to Cloudflare Realtime SFU. The control WebSocket can
reconnect during watcher-managed service refreshes without creating a new
meeting record.
* Broadcast streaming stays API-owned by `apps/web`: the meeting host calls
`/api/v1/workspaces/:wsId/meetings/:meetingId/stream`, `apps/web` creates or
resumes a Cloudflare Stream live input with server-only
`CLOUDFLARE_ACCOUNT_ID` plus `CLOUDFLARE_STREAM_API_TOKEN` (or
`CLOUDFLARE_API_TOKEN`), stores the live input UID and WHIP/WHEP URLs in
`private.meet_stream_live_inputs`, and returns the WHIP publish URL only to
the host response. Workspace viewers receive only the WHEP playback URL.
Cost controls are part of the signed token contract: camera defaults off,
video is capped at 720p/24fps, room limits are explicit, and webinar viewers do
not receive publish scope. Cloudflare Stream live inputs are created with
recording mode `off` and hidden viewer counts by default; set
`CLOUDFLARE_STREAM_ALLOWED_ORIGINS` to a comma-separated allowlist when Stream
playback should be origin-restricted. Do not add Cloudflare Workers or Durable
Objects for production Meet room logic; use the internal service and
blue/green watcher instead.
`scripts/docker-web/env.js` persists generated helper tokens under
`tmp/docker-web/markitdown-token`, `tmp/docker-web/storage-unzip-token`,
`tmp/docker-web/supermemory-api-key`,
and `tmp/docker-web/supermemory-postgres-password`. Override them with
`DOCKER_MARKITDOWN_ENDPOINT_SECRET`,
`DOCKER_DRIVE_UNZIP_PROXY_SHARED_TOKEN`, or the `DOCKER_SUPERMEMORY_*` env when
an operator needs fixed values.
Workspace ZIP auto-extract is enabled by the workspace-level
`DRIVE_AUTO_EXTRACT_ZIP` secret. Workspaces with `EXTERNAL_PROJECT_ENABLED=true`
also opt in automatically so CMS/WebGL workspaces can reuse the unzipper without
duplicating storage automation setup. The Docker-internal URL and token are
fallbacks for workspaces that have not supplied custom proxy secrets. If a
workspace supplies a custom `DRIVE_AUTO_EXTRACT_PROXY_URL`, it must also supply
its own `DRIVE_AUTO_EXTRACT_PROXY_TOKEN`; the process-wide fallback token must
not be sent to a workspace-controlled proxy URL.
CMS WebGL package uploads also use the `storage-unzip-proxy`, but they are a
first-class CMS upload path rather than generic Drive automation. They require a
configured unzip proxy URL and token, but they do not require the
`DRIVE_AUTO_EXTRACT_ZIP` workspace opt-in secret. The CMS finalize route unpacks
the ZIP into workspace Drive, detects the playable `index.html`, and stores the
same-origin artifact map on the CMS `webgl-package` asset. Browser uploads go
directly to the signed storage URL returned by the self-hosted web app's WebGL
upload-url route, so large ZIPs do not pass through the Vercel-hosted CMS app or
the web app proxy before reaching Supabase Storage or R2. The CMS client reports
per-file upload progress during the signed upload, then calls the WebGL finalize
route so the backend handles extraction and artifact-map persistence.
The unzip proxy fans out backend callbacks for extracted folders and asks the
callback route for per-file upload URLs. Before uploading extracted bytes, the
proxy verifies the callback response names a trusted provider and that the
signed upload URL belongs to hosted Supabase, Cloudflare R2, or an exact
operator-configured upload origin. It forwards only content type and generated
bearer-token headers to the upload URL. The storage auto-extract and CMS WebGL
extract callback routes still pass through the central API proxy guard before
they validate the shared unzip token, so malformed, rate-limited, or oversized
callback requests are rejected at the same cheap boundary as other API
mutations. Direct `file` callbacks are legacy/small-file only and enforce the
same 512 KiB body budget locally; large extracted files must use the
`file-upload-url` callback flow. The proxy currently buffers the downloaded
archive and each extracted file in memory, so the default caps stay
conservative: 100 MiB ZIP downloads, 50 MiB per extracted file, and 250 MiB
total extracted output.
Operators can tune those caps with
`DRIVE_UNZIP_PROXY_MAX_ARCHIVE_BYTES`,
`DRIVE_UNZIP_PROXY_MAX_ENTRY_BYTES`,
`DRIVE_UNZIP_PROXY_MAX_TOTAL_EXTRACTED_BYTES`, and
`DRIVE_UNZIP_PROXY_MAX_ARCHIVE_ENTRIES`; workspace Drive quota must still be
large enough for the uploaded archive and extracted files. Set
`DRIVE_UNZIP_PROXY_ALLOWED_UPLOAD_ORIGINS` for self-hosted Supabase or custom
R2/S3-compatible origins, and reserve
`DRIVE_UNZIP_PROXY_ALLOW_LOCAL_UPLOAD_ORIGINS=true` for local Supabase testing.
The MarkItDown endpoint is the conversion path for uploaded workspace files.
Do not route YouTube summaries through MarkItDown or Google Search. Google
Gemini chat requests attach one public or unlisted YouTube URL directly as a
native `video/mp4` file input, so the model can summarize the video through the
provider-supported video path. Playlist/query parameters are stripped before
the URL is attached so each request references only one video. Any legacy direct
URL conversion path that still reaches MarkItDown must reserve and commit the
fixed MarkItDown credit charge before the sidecar request is sent.
Interrupted Docker Compose recreates can leave temporary container names such
as `_platform-markitdown-1`. The Docker helper treats those as recoverable
only when the suffix matches one of the services in the current `compose up`
request, removes that stale temp container, and retries the same narrow up
operation.
Compose can also briefly report `dependency failed to start` with `No such
container: ` when a dependency was recreated between Docker's dependency
resolution and health wait. The helper treats that as a stale dependency
reference and retries the same narrow `compose up` without deleting unrelated
containers. Tune that retry budget with
`DOCKER_WEB_COMPOSE_UP_STALE_DEPENDENCY_RETRY_MAX_ATTEMPTS`.
The production `web-proxy` service is pinned to the official mainline Alpine
image `nginx:1.31.0-alpine`, and `scripts/check-docker-web.js` verifies that
pin in the merged production Compose config.
The long-lived nginx proxy also raises its request-header buffer limits so larger
session/auth cookies do not fail at the proxy layer with `400 Request Header Or
Cookie Too Large` before the active web container sees the request. It now also
raises its upstream response-header buffers (`proxy_buffer_size`,
`proxy_buffers`, and `proxy_busy_buffers_size`) so larger Supabase auth
responses with multiple `Set-Cookie` headers do not fail with `upstream sent too
big header while reading response header from upstream`. The proxy uses Docker
DNS re-resolution plus a shorter keepalive timeout so promotions are less
likely to produce transient `502 Host Error` responses for existing Cloudflare
Tunnel connections, while the previous color remains alive as a warm standby.
The proxy keeps both blue and green in the nginx upstream group during steady
state, with the active color as the primary upstream and the standby color as a
backup. The runtime DNS resolver is defined at the nginx include/http scope,
not just inside `server`, so Docker service-name resolution continues to work
for the blue/green upstream block at reload time.
Both the production web image healthcheck and the `web-proxy` compose
healthcheck now use the internal `/__platform/drain-status` endpoint too, so
raw `bun serve:web:docker:bg` waits on the same non-rate-limited readiness path
as the blue/green promotion gate. The proxy exposes that path as an exact
loopback-only nginx location and forwards a private internal probe header to
the active web lane, because the web request tracker intentionally answers the
drain-status endpoint only for local or explicitly trusted Docker-network
requests.
Every blue/green deployment also stamps the runtime with
`PLATFORM_DEPLOYMENT_STAMP` and `PLATFORM_BLUE_GREEN_COLOR`. Those values are
surfaced through both nginx response headers and the web process itself, and
the web layout appends the deployment stamp to the service-worker URL with
`updateViaCache: 'none'` so new deployments push browsers toward the latest
worker instead of lingering on stale cached state.
The local runtime state lives in:
* `tmp/docker-web/prod/active-color`
* `tmp/docker-web/prod/deployment-stamp`
* `tmp/docker-web/prod/nginx.conf`
* `tmp/docker-web/prod/target-state.json`
These files are intentionally local-only and safe to regenerate.
Infrastructure Monitoring → Deployments reads `target-state.json`, the watcher
deployment history, and the latest deployment stage handoff together, so
operators can see staged target work such as a prepared web color while Hive or
support gates still block public promotion. `active-color` and the generated
proxy config remain on the previous serving web lane until the final
`proxy-reload` stage passes. Watcher-managed deployments persist the
`web-build`, `web-promote`, `hive-migrate`, `hive-promote`,
`support-refresh`, and `proxy-reload` stage results into deployment history.
Modern rows that were recorded without a stage array are inferred from final
deployment status and build-cache metadata; truly pre-tracking rows still show
stage chips as not applicable. Active watcher deployments that only have
pending build/deploy status are surfaced with a synthetic current stage so
operators can see the build is in progress before full stage history is written.
When `TUTURUUU_CI_CHECKS_ENABLED=1`, the watcher also publishes one sanitized
GitHub Check Run per watched commit. The default check name is `Tuturuuu CI`;
override it with `TUTURUUU_CI_CHECK_NAME`. The managed production path is the
Infrastructure → GitHub Bot page in the root workspace: create and install a
Tuturuuu-owned GitHub App with repository `Checks: write` permission, then save
the App ID, installation ID, repository owner/name, and private key there. The
private key stays server-side and encrypted in the private-schema vault. After
the configuration validates, use **Enable watcher auto-pickup** on the same
page. apps/web issues a dedicated watcher-only client token, writes a small
credential request into the blue/green control directory, and the watcher moves
that credential into its local runtime directory on the next Check Run publish.
The watcher then discovers the apps/web installation-token endpoint without
requiring you to copy GitHub-related env into the watcher process. The queued
runtime credential is the Check Run opt-in signal for the watcher.
Manual generated-token setup remains available for local or emergency use:
```bash theme={null}
TUTURUUU_CI_CHECKS_ENABLED=1
TUTURUUU_CI_GITHUB_TOKEN_URL=https:///api/v1/infrastructure/github-bot/installation-token
TUTURUUU_CI_GITHUB_TOKEN_CLIENT_TOKEN=
```
The watcher exchanges that client token for repository-scoped GitHub App
installation tokens and refreshes them before expiry. Revoke and reissue the
watcher client from Infrastructure → GitHub Bot when rotating access. The
browser UI never displays generated GitHub installation tokens, and the
auto-pickup action does not display the watcher client token. Manual and
static-token paths still require `TUTURUUU_CI_CHECKS_ENABLED=1`; when enabled,
the publisher uses `TUTURUUU_CI_GITHUB_TOKEN` first, then an explicitly
configured generated token endpoint, then the watcher auto-pickup runtime
credential, then `GITHUB_TOKEN`. Static tokens must be able to create and
update Check Runs for the repository. The watcher stores the latest check-run
id per commit in
`tmp/docker-web/watch/blue-green-github-checks.json` so restarts update the
same GitHub row instead of creating duplicates. `TUTURUUU_CI_CHECK_DETAILS_URL`
is optional and omitted unless explicitly configured.
The GitHub-facing Check Run is intentionally not a log export. It includes only
allowlisted rollout metadata: commit SHA/short SHA, branch/upstream, deployment
kind, watcher status, current stage, aggregate stage counts, and safe
timestamps/durations. It must not include raw watcher logs, raw error messages,
environment values, local host paths, hostnames, emails, user ids, tokens, or
secret-shaped key/value text.
The watcher also uses GitHub workflow-run validation to avoid repeatedly
building a commit whose CI has already failed. When Check Run publishing is
enabled, `GITHUB_REPOSITORY` is present, or
`DOCKER_WEB_WATCHER_GITHUB_VALIDATION=1` is set, the watcher reads the latest
Actions workflow runs for the candidate commit's exact `head_sha`. If the
latest run for any workflow completed with `failure`, `cancelled`,
`timed_out`, `startup_failure`, or `action_required`, automatic deploy,
recovery handoff, reconciliation, and standby-refresh builds are suppressed
with watcher status `validation-blocked`. That state does not add a failed
deployment row or consume the retry budget; fix CI and let the watcher see a
new successful/latest run, or set
`DOCKER_WEB_WATCHER_GITHUB_VALIDATION_DISABLED=1` for an operator-approved
manual override.
After each Hive migration pass, the deploy helper runs
`docker compose rm --stop -f hive-db-migrate` so the completed one-shot
migration service is stopped if necessary and removed. This keeps
`hive-db-migrate` from lingering after `depends_on` starts it while Hive
services come up.
## Native Cron Runner
Self-hosted production cron jobs use `apps/web/cron.config.json` as the shared
source of truth. `apps/web/vercel.json.crons` should stay generated from that
file with:
```bash theme={null}
node scripts/sync-web-crons.js --check
node scripts/sync-web-crons.js
```
Use `--check` in CI and local verification when cron definitions change. The
sync script preserves Vercel behavior by copying each enabled job's `path` and
`schedule` from the shared config into `apps/web/vercel.json`.
In Docker production, the `web-cron-runner` service runs
`scripts/watch-web-crons.js` against `INTERNAL_WEB_API_ORIGIN`, defaulting to
`http://web-proxy:7803`. Requests include
`Authorization: Bearer ${CRON_SECRET || VERCEL_CRON_SECRET}` so the same route
auth gate can protect Vercel and native Docker executions.
The cron-runner image bundles the `cron-parser` dependency used by that script,
so runtime execution does not depend on `node_modules` existing in the mounted
host checkout.
When neither `CRON_SECRET` nor `VERCEL_CRON_SECRET` is set on the host, the
Docker environment generator creates a persisted internal secret at
`tmp/docker-web/cron-token` and injects it into both the web containers and the
`web-cron-runner` service as `CRON_SECRET`. This keeps native Docker cron auth
self-contained while preserving explicit host-provided secrets when present.
Watcher startup also keeps the Docker control sidecar and cron runner present.
`bun serve:web:docker:bg`, `bun serve:web:docker:bg:watch`, and watcher
recovery recreate or resume `web-blue-green-watcher` first, refresh
`web-docker-control`, then ensure `web-cron-runner` exists with a no-recreate
Compose start. A healthy existing cron runner is left running; a missing control
sidecar or runner fails watcher bootstrap instead of leaving the stack without
native cron recovery or execution.
`web-cron-runner` is the executor. The blue/green watcher is one recovery
consumer, not the owner of cron execution. Each watcher poll reconciles
cron-runner health before normal deploy work by checking the
`web-cron-runner` container state, Docker healthcheck result, and
`tmp/docker-web/cron/status.json.updatedAt`. If the container is missing,
unhealthy, or the heartbeat is stale, the watcher writes the existing
`tmp/docker-web/watch/control/cron-runner-recovery.request.json` request and
processes it immediately with a force-recreate of `web-cron-runner`. It then
waits for a fresh heartbeat before reporting `cron-runner-recovered`; if the
Compose restart fails, the request remains on disk with `lastError` and the
watcher backs off before retrying.
`web-docker-control` also owns an independent cron-runner watchdog. The watchdog
defaults on, reads the same cron runner heartbeat, inspects the
`web-cron-runner` container, ensures `web-blue-green-watcher`, and
force-recreates only `web-cron-runner` when the heartbeat is stale/missing or
the container is missing/unhealthy. This keeps cron recovery available even when
the blue/green watcher heartbeat is stale. The watchdog does not recover the
whole serving stack; route-origin failures such as an unreachable
`INTERNAL_WEB_API_ORIGIN` / `web-proxy` are reported in monitoring diagnostics
for an operator to fix separately.
Cron runner heartbeats refresh both runner liveness and schedule metadata. When
the runner starts a cycle or keeps a long execution alive, it recomputes each
enabled job's next future run from `apps/web/cron.config.json`, current runtime
control overrides, and UTC time. The monitoring API also derives future
`nextRunAt` values from config/control if an older persisted status file still
contains stale schedule fields, while preserving the separate stale/live runner
health signal from `status.json.updatedAt`.
`web-cron-runner` also protects itself. The entrypoint restarts the child
`scripts/watch-web-crons.js` process when the status heartbeat is missing,
invalid, or stale after startup grace, and the Docker healthcheck calls the same
entrypoint heartbeat check. This prevents a hung cron child from looking
healthy just because the process still exists.
Useful cron-runner recovery knobs:
* `DOCKER_WEB_WATCHER_CRON_RUNNER_STALE_AFTER_MS` controls watcher heartbeat
staleness detection. Default: `120000`.
* `DOCKER_WEB_WATCHER_CRON_RUNNER_RECOVERY_WAIT_MS` controls how long the
watcher waits for a fresh heartbeat after restart. Default: `90000`.
* `DOCKER_WEB_WATCHER_CRON_RUNNER_RECOVERY_POLL_MS` controls the heartbeat wait
poll interval. Default: `2000`.
* `PLATFORM_CRON_RUNNER_STATUS_STALE_AFTER_MS` controls the entrypoint child
watchdog. Default: `120000`.
* `PLATFORM_DOCKER_CONTROL_CRON_WATCHDOG_DISABLED` disables the
`web-docker-control` cron-runner watchdog when set to `true`. Default:
`false`.
* `PLATFORM_DOCKER_CONTROL_CRON_WATCHDOG_INTERVAL_MS` controls the
`web-docker-control` watchdog polling interval. Default: `30000`.
* `PLATFORM_DOCKER_CONTROL_CRON_RUNNER_STALE_AFTER_MS` controls
`web-docker-control` heartbeat staleness detection. Default: `120000`.
* `PLATFORM_DOCKER_CONTROL_CRON_RECOVERY_COOLDOWN_MS` controls
`web-docker-control` recovery cooldown after a restart attempt. Default:
`60000`.
* `PLATFORM_CRON_DOCKER_TELEMETRY_TIMEOUT_MS` bounds Docker `ps` / `logs`
probes used for cron telemetry. Default: `10000`.
`web-docker-control` is an internal-only sidecar with the Docker CLI, Docker
socket, and host worktree mount. `apps/web` reaches it through
`PLATFORM_DOCKER_CONTROL_URL` and `PLATFORM_DOCKER_CONTROL_TOKEN` for hardcoded
watcher/cron-runner recovery actions only; it does not expose a general Docker
or shell interface. Its status file at
`tmp/docker-web/docker-control/status.json` includes both the latest manual or
watchdog recovery attempt and the latest watchdog check, which the cron
monitoring dashboard surfaces alongside watcher and runner heartbeat warnings.
When the runner heartbeat is stale, future cron times in the dashboard are
scheduled estimates derived from config/control, not proof that the runner will
execute them.
Runtime cron telemetry is intentionally file-based and local to the host:
* `tmp/docker-web/cron/status.json` for runner health, the current cycle, and
the latest manual run lifecycle records. Manual runs move through `queued`,
`processing`, and a final `success` / `failed` / `timeout` / `skipped` state.
While any run is processing, the runner refreshes this heartbeat and, for
manual runs, captured route console logs so the monitoring UI can show
near-realtime status and log updates. Heartbeat refreshes also recompute
future per-job and aggregate `nextRunAt` values so a recovered runner does not
keep serving stale schedule badges.
* `tmp/docker-web/cron/state.json` for restart-safe last-run markers.
* `tmp/docker-web/cron/executions/*.jsonl` for per-run route response,
duration, status, and captured web-container console logs.
* `tmp/docker-web/watch/control/cron-control.json` for the global enabled
switch.
* `tmp/docker-web/watch/control/cron-run-requests/*.json` for queued manual
runs created by the monitoring UI.
Disabling cron execution blocks scheduled jobs and leaves manual run requests
queued until cron execution is enabled again. The runner also supports
`--once`, which is used by script tests to verify due-run detection, queued
manual runs, restart-safe state, and log persistence without starting the
long-running loop.
Calendar cron routes should call workspace calendar APIs through
`INTERNAL_WEB_API_ORIGIN` when it is present. In Docker production this keeps
provider sync and smart scheduling traffic on the internal `web-proxy` origin
instead of accidentally depending on a public app URL from inside the container.
`calendar-provider-sync` intentionally calls the same
`/api/v1/workspaces/:wsId/calendar/sync` route that the calendar page uses, but
with cron auth and `source: "cron"` so dashboard runs stay manual and scheduled
runs stay auditable. The workspace sync route is responsible for provider
fan-out: Google connections are selected from active Google auth-token rows, and
Microsoft connections are selected from active Microsoft auth-token rows. Do not
reimplement provider-specific calendar fetching inside the cron wrapper.
The job should run every 15 minutes from `apps/web/cron.config.json`; Calendar
provider sync should not be scheduled through Trigger.dev in production.
## Auto-Deploy Watcher
`bun serve:web:docker:bg:watch` locks the current branch/upstream at startup,
polls every second, fast-forwards when GitHub has a newer commit, and runs the
blue/green deploy flow automatically.
When the watcher container starts on a host with no active blue/green runtime,
it treats that idle state as a missing active deployment and bootstraps the
current commit. This first-run recovery creates `web-proxy`, the active and
standby lanes, and `cloudflared` when the tunnel profile is enabled.
Run this command from a host-level process manager, not only from inside Docker.
The command starts and tails the `web-blue-green-watcher` container, but the
host process is the part that can recover after the Docker engine itself dies.
When Docker is unavailable or Docker CLI probes stop returning, the host
supervisor polls Docker with bounded probes; after
`DOCKER_WEB_WATCHER_DOCKER_RESTART_AFTER_MS` milliseconds of continuous failure
(default 30000), it attempts to restart Docker, waits for Docker probes to pass,
runs any configured host-level post-restart commands, then recreates the watcher
container. The recreated watcher reuses the existing cached blue/green recovery
path to bring `web-proxy` and the active/standby web lanes back to health.
During normal steady-state polling, after active build-lock and revert checks,
the watcher also reconciles the production Compose services that should already
be serving. It inspects the expected services with stopped containers included:
`web-proxy`, the active web or TanStack lane, Redis profile services, enabled
health-gated sidecars, active-color Hive/realtime services, and `cloudflared`
when its profile is enabled. Missing, exited, dead, or unhealthy services are
recovered with bounded no-build Compose operations: stopped or missing services
use `docker compose ... up --detach --no-build --remove-orphans`, unhealthy
services are force-recreated, and `starting` services are left alone until the
next poll. Image rebuilds remain owned by the normal blue/green deploy flow and
the cached/full active-runtime recovery fallback.
Before recreating the watcher after a suspected devbox/local Supabase mix-up,
inspect only origin classifications, not secrets. Root `.env.local` should
classify as cloud for `NEXT_PUBLIC_SUPABASE_URL` and `SUPABASE_SERVER_URL`;
`apps/web/.env.local` may classify as local after `ttr box setup`, but the
watcher must not select it while root `.env.local` exists. If local Supabase was
started accidentally on a production host and no other local workflow is using
it, stop it with `bun sb:stop`, then recreate the watcher with
`bun serve:web:docker:bg:watch`.
Docker restart command defaults:
* Linux: `systemctl restart docker`
* macOS: `open -ga Docker`
* Windows: `powershell.exe -NoProfile -Command Start-Process "Docker Desktop"`
Override the command with a JSON array when the host needs a different service
manager or a narrow sudo rule:
```bash theme={null}
DOCKER_WEB_WATCHER_DOCKER_RESTART_COMMAND='["sudo","systemctl","restart","docker"]' \
bun serve:web:docker:bg:watch -- --if-locked replace
```
Useful host-supervisor knobs:
* `DOCKER_WEB_WATCHER_DOCKER_RESTART_AFTER_MS`: delay before the first Docker
restart attempt while `docker info` is failing; set `0` to disable attempts.
* `DOCKER_WEB_WATCHER_DOCKER_RESTART_COOLDOWN_MS`: minimum time between restart
attempts; default 300000.
* `DOCKER_WEB_WATCHER_DOCKER_RESTART_COMMAND`: command used to restart or open
Docker. Prefer JSON array syntax for commands with quoted arguments.
* `DOCKER_WEB_WATCHER_DOCKER_RESTART_DISABLED=1`: hard-disable daemon restart
attempts while still waiting for Docker to recover externally.
* `DOCKER_WEB_WATCHER_DOCKER_RECOVERY_TIMEOUT_MS`: optional maximum wait time
for Docker recovery; unset or `0` means wait indefinitely.
* `DOCKER_WEB_WATCHER_DOCKER_PROBE_TIMEOUT_MS`: timeout for quick Docker CLI
probes such as `docker info`, `docker compose version`, and watcher container
state checks; default 10000.
* `DOCKER_WEB_WATCHER_LOG_STREAM_RECONNECT_MS`: maximum time to let the host
wrapper follow watcher logs before reconnecting and checking watcher health;
default 60000.
* `DOCKER_WEB_WATCHER_DOCKER_POST_RESTART_COMMAND_TIMEOUT_MS`: timeout for each
additional host-level recovery command; default 600000.
* `DOCKER_WEB_WATCHER_DOCKER_POST_RESTART_COMMANDS`: JSON array of host-level
commands to run after Docker is reachable again and before Tuturuuu recreates
its watcher container. Each entry is an object with `command`, `args`, and an
optional `cwd`.
* `DOCKER_WEB_WATCHER_MAX_REQUEST_LOG_BYTES`: maximum durable proxy
request-log ledger size before the watcher rotates and prunes older JSONL
chunks before appending new entries; default 268435456 bytes.
Timing, disable, and email alert values can be updated from Infrastructure
Monitoring in the web dashboard. The dashboard writes
`tmp/docker-web/watch/control/blue-green-docker-recovery-settings.json`, and
the host supervisor reads that file before each Docker recovery wait. Dashboard
settings override the environment defaults without restarting the supervisor.
Host-level executable commands are different: configure
`DOCKER_WEB_WATCHER_DOCKER_RESTART_COMMAND` and
`DOCKER_WEB_WATCHER_DOCKER_POST_RESTART_COMMANDS` only in the host supervisor
environment. The supervisor intentionally ignores command fields from
`blue-green-docker-recovery-settings.json` so dashboard viewers cannot persist
host commands for a later recovery event.
That settings file also owns Docker crash email alerts:
* `emailAlertsEnabled`: enables SES-backed Docker recovery alert emails from the
web cron worker and watcher-side first-failure build/deploy incident emails.
* `emailAlertRecipients`: explicit recipient list. If this is empty, the cron
and watcher fall back to `PLATFORM_DOCKER_RECOVERY_ALERT_EMAILS`, then the
last operator email that saved the settings.
* `emailAlertCooldownMs`: minimum time between alert emails; default 1800000.
The host supervisor persists Docker crash/recovery events to the watcher log
archive as soon as it detects failed or timed-out Docker probes. If Docker had
to be restarted before services recovered, the watcher sends an immediate
force-restart recovery email to the configured recovery recipients and records
that incident as notified. If the watcher-side email is disabled or fails, the
infra app cron job `/api/cron/infrastructure/docker-recovery-alerts` can still
send a fallback SES email after Docker and the infra app are reachable again. Both paths
deduplicate by Docker recovery incident id using
`tmp/docker-web/watch/control/blue-green-docker-recovery-alert-state.json`.
The blue/green watcher sends its own build/deploy incident email when an
`apps/web` deployment attempt first fails for a commit. This runs from the
watcher process rather than the current web image, uses the same recipient
resolution as Docker recovery alerts, and only sends for the first failed
history row per `commitHash`. Later retries for that same commit still append
full failure history but do not spam operators. The incident email includes the
full and short commit hash, commit subject, branch/upstream, deployment kind,
host, timing, exit code or signal when available, the recorded `failureReason`,
and debugging pointers for watcher history/log files plus commands like
`git show --stat --oneline ` and watcher container logs. Notification
send failures are logged and never block the watcher loop.
Watcher incident email code runs from the repo root inside the watcher
container, so any workspace package imported by `scripts/watch-blue-green/*`
must be root-resolvable through `package.json` and covered by a root-runtime
import test.
Example post-restart commands for colocated projects:
```json theme={null}
[
{
"command": "docker",
"args": ["compose", "-f", "/srv/zeus/docker-compose.yml", "up", "-d"],
"cwd": "/srv/zeus"
},
{
"command": "docker",
"args": ["compose", "-f", "/srv/upskii/docker-compose.yml", "up", "-d"],
"cwd": "/srv/upskii"
}
]
```
For Linux production hosts, install the command as a root-owned `systemd`
service or run it as an operator account with permission to execute only the
configured Docker restart command and the explicit post-restart commands needed
by colocated projects. Use `Restart=always` so the host supervisor itself comes
back after reboots or process crashes.
Example unit:
```ini theme={null}
[Unit]
Description=Tuturuuu blue/green watcher
After=docker.service network-online.target
Wants=docker.service network-online.target
[Service]
Type=simple
WorkingDirectory=/srv/tuturuuu
Environment=NODE_ENV=production
ExecStart=/usr/local/bin/bun serve:web:docker:bg:watch -- --if-locked resume
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
```
Install it with the production checkout path and Bun path for the host, then
run `systemctl enable --now tuturuuu-blue-green-watcher.service`. Keep
deployment secrets in the checkout's root `.env.local` or an explicit env file,
not in the unit file.
Additional behavior:
* If the watcher script itself changed in the pulled revision, the current
watcher process restarts first and the replacement process performs the
deploy.
* If blue/green is already live and the standby color remains on an older
revision for 15 minutes, the watcher rebuilds only the standby color in
place. The active color remains primary for new traffic the whole time.
* If the watcher sees a degraded blue/green runtime with a proxy or runtime
marker present but no active web color serving traffic, it immediately
retags the latest retained successful image into the active web color and
starts it with `--no-build`. It prefers a retained image for the current
`main` commit, then falls back to the newest retained successful image so
the runtime can recover first and reconcile to `main` afterward. It then
retags the same cached image into the opposite color and starts that as the
warm standby, creating two ready copies without waiting for a fresh build.
* Blue/green active and standby discovery uses Docker health, not just
container presence. If the persisted active color is unhealthy but the
opposite color is healthy, the watcher rewrites the active marker and proxy
to the healthy color before building or refreshing another lane.
* Cached recoveries write a fresh nginx proxy config before the proxy is
started, so recovery never boots nginx with a stale upstream that points at a
missing or unhealthy color.
* That standby catch-up path also stops and removes the stale standby
container before rebuilding it, so health checks target the fresh
replacement container rather than an outdated standby instance.
* Standby catch-up rebuilds reuse the current deployment stamp so the warm
backup matches the latest deployment state instead of serving an older build
if nginx needs to fail over.
* The watcher dashboard surfaces the top 3 most relevant deployments from the
recent history, prioritizing in-progress rollouts first, then the live
promoted color, then the warm standby. Direct manual
`bun serve:web:docker:bg` runs are written into that same history too.
* Cached recoveries write both the active recovery and the standby refresh into
the same retained ledger, preserving the current two warm copies plus the
prior successful deployment as the fastest rollback reference. If no retained
image exists, the watcher falls back to the normal recovery build path.
* The infrastructure monitoring rollback controls show the latest retained
cached recovery images separately from the general deployment history, so an
operator can quickly select a known cache-backed commit before pinning it for
rollback or smoke testing.
* Successful active and standby builds tag the service image as
`{compose-project}-web-cache:{commit}` and prune older retained cache tags
beyond the three newest successful deployments. Pruning is idempotent:
already-removed cache tags are ignored instead of warning in the live watcher
log.
### Deployment build lock
Blue/green deploys coordinate on a JSON lock file under
`tmp/docker-web/watch/blue-green-deployment-build.lock` (owner PID, command,
deployment kind, and a re-entrant token for nested helper calls).
* On **Linux**, the helper compares `/proc//cmdline` to the recorded lock
so a reused PID after a crash cannot masquerade as an in-flight deploy. The
recorded command is the package script name (`bun serve:web:docker:bg`), but
production deploys usually run as `node scripts/docker-web.js ...`; the
matcher treats those as the same holder so a live `node` deploy is not cleared
as a stale PID reuse. On **macOS and Windows**, the same age-based stale window
(`DOCKER_WEB_DEPLOYMENT_LOCK_STALE_AFTER_MS`, default eight hours) still clears
abandoned locks because `/proc` validation is unavailable. When no `web-proxy`
/ `web-blue` / `web-green` / `tanstack-web-blue` / `tanstack-web-green`
containers exist, the auto-deploy watcher also runs the same stale-lock sweep
before cached recovery.
* **`DOCKER_WEB_DEPLOYMENT_LOCK_STALE_AFTER_MS`**: optional override for the
default eight-hour window used when `/proc` is unreadable but `kill(pid, 0)`
still reports a process (for example permission quirks). Set to `0` to
disable age-based assists.
* **`DOCKER_WEB_CANCEL_ACTIVE_BUILD=1`** or **`--cancel-active-build`** on a
manual `bun serve:web:docker:bg` run stops the watcher/buildkit services,
clears the lock, and records a canceled history row before starting fresh.
* The auto-deploy watcher treats an active deployment lock as a wait state, not
a failed deploy attempt. Recovered pending handoffs, reconcile builds, standby
refreshes, platform promotions, and imported Infrastructure project builds all
defer behind the same lock so only one deployment build runs across the stack.
* The watcher also treats a build lock older than 30 minutes as a timed-out
build. For another live deployment PID, it sends `SIGTERM` to the recorded
owner. If the lock is owned by the watcher process itself and the watcher has
already returned to the polling loop, the lock is treated as leaked
`cached-recovery` state and cleared without signaling the watcher. In both
cases, the watcher records a failed deployment history row with the timeout
reason and waits until the next polling cycle before retrying. Override the
window with `DOCKER_WEB_WATCHER_BUILD_TIMEOUT_MS`; set it to `0` only when an
operator explicitly wants to disable watcher-side build termination.
* The **`apps/web` Dockerfile `deps` stage** retries `bun install --frozen-lockfile` up to three times with a Bun cache scrub between attempts.
If the build still exits with `bun install --frozen-lockfile` **exit code 1**
after a `git pull`, regenerate `bun.lock` in a development checkout, commit
the reviewed lockfile update, and deploy that commit. Do not let the
production host rewrite `bun.lock` as part of the auto-deploy path. If tarball
extraction still fails (for example
`@biomejs/cli-linux-x64`), the blue/green helper prunes BuildKit exec cache
mounts, recreates the Compose-owned `buildkit` service, and retries once with
`docker compose build --no-cache` so a cached failed deps stage is not reused.
If BuildKit has already lost transport and the exec-cache prune command exits
with EOF or `code = Unavailable`, that prune is treated as best-effort and the
service is still recreated before retrying. The same one-time fresh retry is
used for `CACHED ERROR ... COPY --from=deps` and for the build watchdog
timeout.
### Monitoring Surfaces
The infrastructure monitoring UI in `apps/infrastructure` is intentionally split
into smaller pages instead of one oversized dashboard:
* `/{wsId}/monitoring` for the operator overview, runtime
snapshot, cron health summary, and jump points into deeper surfaces.
* `/{wsId}/monitoring/cron` for cron job schedules, global
enable/disable control, manual run requests, recent execution status, route
responses, and captured web-container console logs.
* `/{wsId}/monitoring/rollouts` for rollout controls, deployment
charts, event streams, and ledger history.
* `/{wsId}/monitoring/requests` for paginated proxy request
history backed by the durable JSONL request store under
`tmp/docker-web/watch/blue-green-request-logs/`.
* `/{wsId}/monitoring/watcher-logs` for paginated watcher log
browsing backed by `tmp/docker-web/watch/blue-green-auto-deploy.logs.json`.
Operationally, keep the overview route lightweight and treat request/log
archives as dedicated drill-down pages. The summary snapshot is for quick
operator context; durable history should be paged from the persisted ledgers.
## Build Resource Caps
When build and serve run on the same machine, use the Docker web helper's
Buildx throttling options instead of letting BuildKit consume the full host.
Example:
```bash theme={null}
bun serve:web:docker:bg -- --build-memory auto --build-cpus 4 --build-max-parallelism 1
```
Current root-script defaults:
* `bun serve:web:docker` defaults to `--build-memory auto --build-cpus 4 --build-max-parallelism 1`
* `bun serve:web:docker:bg` defaults to `--build-memory auto --build-cpus 4 --build-max-parallelism 1`
The helper resolves `auto` from Docker's reported memory limit before starting,
restarting, or recreating the Compose-owned BuildKit service. On a Docker
Desktop allocation of 28 GiB, that means the BuildKit memory cap resolves to
about 21 GiB: the helper keeps a real reserve for the Docker VM and other
containers instead of assigning nearly the full Docker allocation to BuildKit.
Direct Compose use still has a concrete `mem_limit` fallback of **12g** and
`cpus` **4** when `DOCKER_WEB_BUILD_MEMORY` and `DOCKER_WEB_BUILD_CPUS` are
unset, because Compose cannot resolve the helper's `auto` value by itself. Raise
or lower the caps with env vars or helper flags when your machine is tighter or
has spare capacity.
For blue/green runs that use the root-script defaults, the helper also keeps a
machine-local adaptive profile at
`tmp/docker-web/buildkit/resource-profile.json`. If BuildKit fails with a
transport or resource-pressure signature such as `code = Unavailable`,
`closing transport`, `error reading from server: EOF`, `received prior goaway`,
`ResourceExhausted`, `cannot allocate memory`, `context deadline exceeded`, or
`[internal] waiting for connection`, the deploy keeps retrying lower profiles in
the same command until the build succeeds or the budget-aware retry ladder is
exhausted. Each retry persists the selected profile for later runs, skips fixed
profiles that exceed the effective Docker memory budget during normal
selection, and can use a larger hard-limit rescue profile after conservative
profiles are exhausted if Docker's reported memory limit still has headroom.
For explicit memory-exhaustion signatures such as `cannot allocate memory` or
exit code 137, the helper can prefer that hard-limit rescue before smaller
profiles that are unlikely to help.
When a later default run sees stale persisted fallback state, it promotes that
state back to the largest Docker-hard-limit rescue profile, preferring lower CPU
at equal memory, before starting BuildKit.
The helper recreates the Compose-owned BuildKit service even if the cleanup
prune command itself fails with the same transport signature, then recreates the
remote Buildx builder if `docker buildx inspect tuturuuu` reports
`Status: inactive`. The profile ladder is:
* `default`: `auto`, 4 CPUs, max parallelism 1
* `stable`: `16g`, 2 CPUs, max parallelism 1
* `low`: `10g`, 2 CPUs, max parallelism 1
* `serial`: `10g`, 1 CPU, max parallelism 1
* `minimal`: `8g`, 1 CPU, max parallelism 1
* `floor`: `6g`, 1 CPU, max parallelism 1
The `serial` profile intentionally keeps the `low` memory cap while reducing
Next/Turbo concurrency. Prefer that retry before shrinking the BuildKit memory
cap when `low` gets through compilation or page-data work but exits with code
137\.
If the floor profile still fails with the same BuildKit infrastructure
signature, the helper resets the machine-local profile back to the
budget-derived `default` profile. If that profile also fails and Docker's
reported memory limit can safely fit a larger fixed profile, the helper can
retry `low` (`10g`) before surfacing the failure. This prevents a machine from
getting stuck starting every future deploy at `floor` while still allowing
memory-starved Next builds to recover on Docker VMs with enough real capacity.
Delete `tmp/docker-web/buildkit/resource-profile.json` only as an emergency
manual reset back to the default profile. Explicit build cap flags or
`DOCKER_WEB_BUILD_MEMORY`, `DOCKER_WEB_BUILD_CPUS`, or
`DOCKER_WEB_BUILD_MAX_PARALLELISM` opt out of the adaptive profile for that run.
You can still override those defaults per run by appending your own flags after
`--`, for example:
```bash theme={null}
bun serve:web:docker:bg -- --build-memory 16g --build-cpus 4 --build-max-parallelism 2
```
Equivalent environment variables:
* `DOCKER_WEB_BUILD_MEMORY=16g`
* `DOCKER_WEB_BUILD_CPUS=4`
* `DOCKER_WEB_BUILD_MAX_PARALLELISM=2`
* `DOCKER_WEB_BUILD_BUILDER_NAME=tuturuuu`
* `DOCKER_WEB_BUILDKIT_PORT=7914`
* `DOCKER_WEB_BUILDKIT_ENDPOINT=tcp://127.0.0.1:7914`
* `DOCKER_WEB_BUILDKIT_PRUNE_AFTER_BUILD=0` for blue/green watcher handoffs
* `DOCKER_WEB_BUILDKIT_PRUNE_MODE=bounded|all|off` (`bounded` is the default)
* `DOCKER_WEB_BUILDKIT_PRUNE_UNTIL=168h`
* `DOCKER_WEB_BUILDKIT_PRUNE_KEEP_STORAGE=50gb`
* `DOCKER_WEB_BUILDKIT_STOP_AFTER_BUILD=0` to keep the `buildkit` container
warm after a build
* `DOCKER_WEB_DOCKER_MEMORY_LIMIT=`
* `DOCKER_WEB_STATIC_PAGE_GENERATION_TIMEOUT=180`
* `DOCKER_WEB_STATIC_GENERATION_MAX_CONCURRENCY=auto`
* `DOCKER_WEB_NEXT_BUILD_CPUS=auto`
* `DOCKER_WEB_NEXT_APP_ONLY=1`
* `DOCKER_WEB_NODE_MAX_OLD_SPACE_SIZE=auto`
* `DOCKER_WEB_NEXT_BUILD_ENGINE=turbopack`
* `DOCKER_WEB_REACT_COMPILER=1`
How it works:
* The helper starts the Compose-owned `buildkit` service and then creates or
reuses the remote Buildx builder named by `DOCKER_WEB_BUILD_BUILDER_NAME`.
The container is named `${COMPOSE_PROJECT_NAME:-tuturuuu}-buildkit-1`, so it
stays visually grouped under the `tuturuuu` Docker Desktop stack.
* The BuildKit caps accept `auto`. Auto memory uses Docker's reported memory
limit minus a small host overhead buffer, rounded down to MiB precision; auto
CPU uses 1 CPU below 10 GB, 2 CPUs below 16 GB, and 4 CPUs on larger Docker
allocations; auto max parallelism uses 1 below 16 GB and 2 above that. The
E2E runner and production web serve scripts use auto memory by default so
local Playwright verification and watcher builds adapt to the current Docker
Desktop setting without requiring one-off env overrides.
* `DOCKER_WEB_BUILD_MEMORY` caps the Compose-owned BuildKit service's memory
budget.
* `DOCKER_WEB_BUILD_CPUS` sets the BuildKit service CPU budget.
* `DOCKER_WEB_BUILD_MAX_PARALLELISM` writes a BuildKit config that limits
concurrent solve steps, which is often the most effective way to reduce
CPU spikes on smaller machines.
* Host-side helper runs point Buildx at
`DOCKER_WEB_BUILDKIT_ENDPOINT` (default
`tcp://127.0.0.1:${DOCKER_WEB_BUILDKIT_PORT:-7914}`). The watcher container
uses `tcp://buildkit:1234` on the Compose network.
* Blue/green watcher handoffs preserve the Compose-owned BuildKit cache volume
by default (`DOCKER_WEB_BUILDKIT_PRUNE_AFTER_BUILD=0`), but stop and remove
the `buildkit` container after the build/deploy phase
(`DOCKER_WEB_BUILDKIT_STOP_AFTER_BUILD=1`). This frees idle CPU and memory
while keeping layer state for the next deployment. Set
`DOCKER_WEB_BUILDKIT_STOP_AFTER_BUILD=0` only when an operator intentionally
wants BuildKit to stay warm after a handoff.
* When BuildKit pruning is enabled, the default mode is bounded: the helper runs
`docker buildx prune --filter until=168h --keep-storage 50gb` for the active
builder. Use `DOCKER_WEB_BUILDKIT_PRUNE_MODE=all` only for disposable build
state, or `DOCKER_WEB_BUILDKIT_PRUNE_MODE=off` to skip pruning without
changing older `DOCKER_WEB_BUILDKIT_PRUNE_AFTER_BUILD` callers.
* Dockerized E2E is the exception: its per-run BuildKit state is disposable and
`scripts/run-web-e2e-docker.js` sets
`DOCKER_WEB_BUILDKIT_PRUNE_AFTER_BUILD=1` and
`DOCKER_WEB_BUILDKIT_PRUNE_MODE=all` unless explicitly overridden.
* The same max-parallelism value is also forwarded as `COMPOSE_PARALLEL_LIMIT`
when that variable is not already set. When the limit is `1`, the blue/green
workflow builds each Bake target group separately so image export and web
compilation do not overlap on memory-constrained hosts.
* When Docker reports less than 10 GB of total memory for a blue/green run, the
helper also restarts the Compose-owned `buildkit` service immediately before
the build batch. This clears long-lived BuildKit RSS before the replacement
web image builds while the active lane is still running. Set
`DOCKER_WEB_BUILDKIT_RESTART_BEFORE_BUILD=0` to skip that low-memory
restart, or `1` to force it on a larger host.
* Docker web builds use `bun run build:web:docker`, which keeps the normal web
build dependency graph, sets `NODE_OPTIONS=--max-old-space-size` to at least
4 GB on Docker allocations below 10 GB, then scales to 8 GB, 12 GB, or 16 GB
based on the lower of Docker's reported memory limit and the selected BuildKit
memory cap, falling back to `DOCKER_WEB_BUILD_MEMORY` for environments where
Docker memory cannot be detected. The helper reads Docker's `MemTotal` and
forwards it as `DOCKER_WEB_DOCKER_MEMORY_LIMIT`; auto buckets reserve 1 GB of
effective Docker build memory for BuildKit, the active runtime lane, and
sidecar overhead before selecting the Node heap bucket on larger allocations.
Docker production builds use Turbopack under the real Node 24 runtime with App
Router-only compilation and React Compiler enabled. This avoids Bun runtime
crashes while loading native Next SWC modules and keeps local E2E aligned with
production Docker builds.
The Docker `builder` stage is based on
`node:24-bookworm-slim` and copies the Bun binary in only for workspace script
orchestration, so the actual `next build` process runs under real Node instead
of Bun's `node` shim. The `@tuturuuu/web` `build:docker` script delegates to
`scripts/run-web-docker-next-build.js`, which spawns `DOCKER_WEB_NODE_BINARY`
(pinned by the Dockerfile to `/usr/local/bin/node`) for the Next CLI and
honors `DOCKER_WEB_NODE_MAX_OLD_SPACE_SIZE=auto` by default. Set a numeric
heap value only when you need to override the bucket selection; values below
4096 MB are rejected. Set `DOCKER_WEB_NEXT_APP_ONLY=0` only when you need to
compare a full-app build.
Keep `DOCKER_WEB_NEXT_BUILD_ENGINE` on its default Turbopack value for
production watcher hosts and local E2E runs. The wrapper also passes
Turbo's `--concurrency` flag inside the Dockerfile build. It uses
`DOCKER_WEB_TURBO_CONCURRENCY` when explicitly set, otherwise it follows the
current `DOCKER_WEB_BUILD_MAX_PARALLELISM` profile value and defaults to `1`.
BuildKit max parallelism limits Docker build graph execution; this inner
Turbo cap limits concurrent workspace package builds such as
`@tuturuuu/types`, `@tuturuuu/devbox`, and `@tuturuuu/masonry`.
* Docker standalone Next builds default static page generation to a 180 second
timeout. Legacy all-in-Docker builds auto-scale the inner Next build CPU count
plus static generation concurrency from Docker memory. Docker allocations
below 10 GB use 1 Next build CPU and static generation concurrency 1; 10-16 GB
allocations use 2 for both; 16 GB and larger allocations use 4 for both. The
Compose-owned BuildKit service still defaults to a 4 CPU budget, while the
inner Next workers stay lower on smaller hosts to avoid OOM kills when the
same machine is also running the active blue/green lane and sidecars. Native
host builds do not set `DOCKER_WEB_STATIC_GENERATION_MAX_CONCURRENCY` or
`DOCKER_WEB_NEXT_BUILD_CPUS` unless the operator explicitly provides them.
Override those with `DOCKER_WEB_STATIC_PAGE_GENERATION_TIMEOUT`,
`DOCKER_WEB_STATIC_GENERATION_MAX_CONCURRENCY`, and
`DOCKER_WEB_NEXT_BUILD_CPUS` only when a host needs a fixed worker count.
* Hive Docker images use a filtered workspace install and a Next standalone
runner. Before `apps/hive` runs `next build`, the image must build
`@tuturuuu/types`, `@tuturuuu/internal-api`, and `@tuturuuu/supabase` because
those packages expose production `dist/*` subpath exports that Turbopack
resolves during the standalone build. Hive realtime installs its filtered
production workspace with Bun's hoisted linker so the direct
`bun apps/hive-realtime/src/index.ts` runtime can resolve top-level production
packages such as `postgres` and `@tuturuuu/realtime`. Keep `.dockerignore`
explicit about recursive generated directories such as `**/.next/**`,
`tmp/**`, and `apps/mobile/build/**`; otherwise previous local builds can be
copied into the next Docker context and inflate small sidecar images by
several gigabytes.
Operational notes:
* These caps affect image builds, not the runtime `apps/web` container after it
has started.
* If no build caps are configured, the helper continues using Docker's default
builder behavior.
* Do not switch capped builds back to the Buildx `docker-container` driver. It
creates Docker-managed containers named like `buildx_buildkit_*` outside the
Compose project, which makes Docker Desktop grouping and health reporting
confusing.
* During capped-build setup, the helper removes known legacy Buildx builders
such as `platform-web-capped-builder` before creating or reusing the
Compose-owned remote `tuturuuu` builder. If `docker buildx ls` still shows
that legacy builder, run the capped web deploy helper once so it can clean up
the stale Buildx record.
* A lower parallelism setting usually trades build speed for host stability.
* If BuildKit fails with `ResourceExhausted`, `cannot allocate memory`, or exit
code 137 while host memory still appears available, Docker's VM/cgroup budget
is the relevant build budget. Prefer the default `--build-memory auto` path:
it now uses a conservative budget and still lets the adaptive profile lower
pressure automatically. Raise `DOCKER_WEB_BUILD_MEMORY`/`--build-memory` only
when `docker info` shows enough Docker memory and host swap is not saturated.
* Docker Desktop resource graphs can show memory near the configured maximum
because the Docker VM and file cache are preallocated or cached. Treat
`docker info --format '{{json .MemTotal}}'` as the build budget source of
truth, then use process/container RSS and BuildKit errors to decide whether
the build is truly exhausting memory.
* If `docker compose` reports `services[buildkit].mem_limit invalid size:
'auto'`, the command path is bypassing the Docker web helper's Compose env
resolver or the host is running an older commit. The helper-only `auto` value
must be converted to a concrete MiB value before any `docker compose up`,
`restart`, `stop`, `rm`, or health-check call reads the BuildKit service.
* If host memory or swap is saturated, lower `--build-max-parallelism` first
and stop unrelated containers before raising the builder memory cap.
* If the build loops on exit code 137 or `SIGSEGV (Address boundary error)`
inside `@tuturuuu/*:build` package tasks, treat it as inner Turbo concurrency
pressure in addition to BuildKit pressure. Keep the default Docker web helper
profile so `DOCKER_WEB_BUILD_MAX_PARALLELISM=1` reaches the Dockerfile build,
or set `DOCKER_WEB_TURBO_CONCURRENCY=1` explicitly for a one-off recovery.
* If the default blue/green deploy keeps failing with BuildKit transport EOFs,
`context deadline exceeded`, `[internal] waiting for connection`, or
graceful-stop messages, leave the root-script defaults in place and let the
adaptive profile step down through the lower profiles automatically. Use
`docker buildx ls` or
`docker buildx inspect tuturuuu` to confirm whether the remote builder is
inactive. Use explicit caps only when you want to bypass the remembered local
profile for a one-off run.
* If Bun fails during an image install with a tarball extraction error such as
`Fail extracting tarball for "@biomejs/cli-linux-x64"`, the blue/green helper
treats it as BuildKit exec-cache corruption once per deployment attempt. It
prunes BuildKit exec cache mounts, recreates the Compose-owned `buildkit`
service, and retries the build once with `--no-cache`. If the prune command
fails because BuildKit has already dropped transport, recovery still proceeds
to service recreation. A second failure is recorded as a real deployment
failure with the original command context preserved in logs.
* If BuildKit reports `CACHED ERROR` after a failed deps stage, or if the
compose build exceeds `DOCKER_WEB_BUILD_TIMEOUT_MS` (default 45 minutes), the
helper uses the same one-time cache recovery and fresh `--no-cache` retry.
* If a deployment fails during the build, the watcher captures the actionable
failure lines into the retained deployment history as `failureReason`. The
Infrastructure → Monitoring → Deployments page and rollout ledger display that
reason inline so operators do not need to reconstruct failures from a terminal
scrollback.
## Redis Profile
Redis is enabled by default in both dev and production-style Docker web stacks.
The Redis and `serverless-redis-http` host ports bind to `127.0.0.1` only; do
not expose them through Cloudflare Tunnel, public firewall rules, or
all-interface Docker port mappings.
The helper persists the generated token in:
* `tmp/docker-web/redis-token`
and injects these values into `apps/web` automatically:
* `UPSTASH_REDIS_REST_URL=http://serverless-redis-http:80`
* `UPSTASH_REDIS_REST_TOKEN=<generated local token>`
The production Redis compose fragment requires `UPSTASH_REDIS_REST_TOKEN` during
Compose interpolation. Use the Docker web helper, which injects the generated
token automatically, or export a strong token before running direct
`docker compose --profile redis ...` commands. Service `env_file` entries do
not satisfy Compose interpolation for the Redis HTTP bridge token.
Docker Redis mode intentionally ignores generic `UPSTASH_REDIS_REST_URL` and
`UPSTASH_REDIS_REST_TOKEN` values from the host shell. This prevents old Upstash
REST URLs from leaking into self-hosted Docker containers after the Upstash
instance is shut down. If a Docker host must override the bundled Redis sidecar,
use the Docker-specific `DOCKER_UPSTASH_REDIS_REST_URL` and
`DOCKER_UPSTASH_REDIS_REST_TOKEN` variables.
Watcher-managed Infrastructure projects do not receive the integrated platform
Docker Redis token. New managed projects start with `redis_enabled=false`, and a
project with Redis enabled receives `UPSTASH_REDIS_REST_URL`,
`UPSTASH_REDIS_REST_TOKEN`, and `SRH_TOKEN` only when project-scoped credentials
are set with `MANAGED_PROJECT__UPSTASH_REDIS_REST_URL` and
`MANAGED_PROJECT__UPSTASH_REDIS_REST_TOKEN`, where `` is
the normalized project id uppercased with dashes converted to underscores.
Generic host `UPSTASH_REDIS_REST_*` and Docker-specific `DOCKER_UPSTASH_*`
values are stripped from managed project compose runs so untrusted project code
cannot access platform Redis credentials.
If you intentionally want to exercise Redis-unavailable fail-open behavior, opt
out:
```bash theme={null}
bun dev:web:docker -- --without-redis
```
That opt-out disables both the bundled Redis companion services and the
Docker-injected `UPSTASH_REDIS_REST_URL` / `UPSTASH_REDIS_REST_TOKEN`
variables. `apps/web` treats Redis-backed route rate limits, abuse counters, and
IP blocks as fail-open availability guardrails in this mode: requests continue
through normal auth, authorization, payload-size, Turnstile, suspension, and
validation checks, but rate-limit buckets and IP-block enforcement are skipped.
Defense-in-depth one-time state such as CLI refresh-token replay protection also
continues without Redis after JWT validation and user lookup. Confirmed
Redis-backed replay attempts are still rejected when Redis is available.
Vercel-hosted satellite apps such as CMS, Calendar, Finance, Learn, Teach, and
Tasks cannot reach Docker-private Redis hosts such as `serverless-redis-http`.
Do not point their Vercel `UPSTASH_REDIS_REST_URL` at the Docker sidecar or
expose Redis through Cloudflare Tunnel. Satellite proxy guards should run
without Redis when Upstash is retired; protected product APIs continue to flow
through `apps/web`, where Docker Redis is available.
## Cloudflare Tunnel Profile
The Docker compose files include an optional `cloudflared` service. Enable it
when the same host should publish the Dockerized web proxy through Cloudflare
Tunnel:
```bash theme={null}
bun serve:web:docker:bg -- --with-cloudflared
```
Required env:
* `CF_TUNNEL_TOKEN`, `CLOUDFLARED_TOKEN`, or `DOCKER_CLOUDFLARED_TOKEN`
When root `.env.local` contains a non-empty `CF_TUNNEL_TOKEN`, Docker web
helpers automatically enable the `cloudflared` profile and pass the value to
Compose as `CLOUDFLARED_TOKEN`. If `--env-file` is passed, the helper applies
the same auto-detection to that explicit file. Set
`DOCKER_WEB_WITH_CLOUDFLARED=0` to keep a configured tunnel token available
without starting the `cloudflared` container; an explicit `--with-cloudflared`
or `--profile cloudflared` still enables the profile.
For a remotely managed Cloudflare Tunnel, configure the public hostname route in
Cloudflare to point at the Docker service or the local proxy loopback:
* Production blue/green: `https://tuturuuu.com` -> `http://localhost:7803` or
`http://web-proxy:7803`
* Dev stack: `https://dev.tuturuuu.com` or a temporary hostname ->
`http://localhost:7803` or `http://web:7803`
The tunnel container shares the web/proxy network namespace, so existing
Cloudflare routes that use `localhost:7803` resolve to the Docker web service
instead of the tunnel container itself. Keep `cms.tuturuuu.com` and other
satellite app hostnames on Vercel unless those apps are explicitly moved into
this Docker stack.
Production compose binds host-published web, Hive, Meet, and Redis ports to
`127.0.0.1` only. Do not remove that loopback prefix during blue/green
migration; public exposure should go through Cloudflare Tunnel or another
controlled frontend, not the staged Docker host ports.
When blue/green is deployed with `--with-cloudflared`, the watcher receives
`DOCKER_WEB_WITH_CLOUDFLARED=1` so future auto-deploys keep the tunnel profile
active and do not remove the `cloudflared` container as an orphan.
## Auto-Pull Blue/Green Watcher
For simple self-hosted boxes that deploy directly from a Git branch, the repo
also provides a long-running auto-deploy watcher:
```bash theme={null}
bun serve:web:docker:bg:watch
```
That command now bootstraps Docker instead of running the watcher loop as a
host PID. Each invocation:
1. Writes the forwarded watcher CLI args to `tmp/docker-web/watch/blue-green-auto-deploy.args.json`.
2. Rebuilds and force-recreates the dedicated `web-blue-green-watcher` service.
3. Builds and force-recreates `web-docker-control` so direct admin recovery
uses the current sidecar code.
4. Builds and starts `web-cron-runner` with `--no-recreate` so native Docker
cron stays available without interrupting a healthy runner.
5. Tails the watcher container's live logs so the terminal still shows the
watcher dashboard.
The watcher container mounts:
* the repo worktree at `/workspace`
* the same repo again at the real host checkout path via
`PLATFORM_HOST_WORKSPACE_DIR`, so host Docker bind mounts resolve against
the host filesystem when the watcher shells into `docker compose ...`
* the linked-worktree common Git directory through `DOCKER_WEB_GIT_COMMON_DIR`
when `.git` is a file, so in-container Git commands can resolve
`.git/worktrees/...` metadata from Docker-mounted checkouts
* `/var/run/docker.sock` so it can manage the blue/green compose stack itself
* the shared Bun install cache volume
* a dedicated watcher `node_modules` volume so the frozen dependency install
stays container-local
Behavior:
1. Reads the built-in `platform` project from the log-drain Postgres project registry. The production watcher service is wired with `PLATFORM_LOG_DRAIN_DATABASE_URL` so a live watcher can consume queued Infrastructure project deployments instead of falling back to the legacy single-branch loop.
2. The seeded branch is `production`, but operators can change it from Infrastructure → Monitoring → Projects. If the selected project branch differs from the current checkout, the watcher restarts its child process, resets tracked changes, removes untracked files, fetches, and checks out that branch. Set `DOCKER_WEB_WATCHER_WORKTREE_RESET_DISABLED=1` to restore the protective dirty-worktree block instead. If the watcher is already stuck or missing, the monitoring UI asks `web-docker-control` to ensure the watcher before ensuring or restarting `web-cron-runner`. If direct control is unavailable, the UI keeps the stalled request visible so host supervisor recovery can be diagnosed instead of appearing queued forever.
3. Locks the selected local branch and tracked upstream at startup.
4. Writes a PID-backed lock file at `tmp/docker-web/watch/blue-green-auto-deploy.lock`.
5. Renders a live terminal dashboard with the locked branch, tracked upstream,
latest local commit, relative commit age, last check time, next poll time,
current blue/green runtime state, and recent watcher events.
6. Polls the tracked upstream every `1000ms` by default.
7. Auto-clears and redraws the dashboard in place on each state change when
attached to a TTY.
8. Runs the Git and deploy subprocesses quietly so the dashboard is not
disrupted by `git fetch`, `git reset`, or Docker build output during normal
watcher operation.
9. Treats the watcher-managed checkout as disposable by default. Before each
upstream comparison it runs `git reset --hard HEAD`, `git clean -fd`,
fetches the locked upstream, and hard-resets to the tracked upstream when
local `HEAD` is behind, ahead, or diverged. Ignored files are left alone.
10. Set `DOCKER_WEB_WATCHER_WORKTREE_RESET_DISABLED=1` only when you need to
preserve manual edits in a deployment clone. With that escape hatch, dirty
worktrees block polling, ahead/diverged branches are skipped, and only
fast-forward pulls are attempted.
11. Runs `bun install --frozen-lockfile` automatically after every successful
upstream sync so installed dependencies match the reviewed `bun.lock`
before the deploy handoff continues. The watcher does not run `bun upgrade`
or a non-frozen install on the production host.
12. Resets dirty `bun.lock` changes by default with the rest of the disposable
checkout. When `DOCKER_WEB_WATCHER_WORKTREE_RESET_DISABLED=1` is set,
dirty `bun.lock` remains a blocking worktree change.
13. Runs `bun serve:web:docker:bg` automatically after a successful upstream
sync.
14. Polls imported Infrastructure projects from log-drain Postgres, synchronizes enabled public GitHub projects into `tmp/docker-web/projects//repo`, deploys them through generated Next.js compose files under the shared `tuturuuu` Compose project, and merges hostname routes into the central nginx proxy.
The imported-project and manual deployment queue cadence is independent
from the normal Git polling interval, so a watcher configured with a long
Git interval such as 1000 seconds still wakes on the shorter project queue
interval to advance queued Deploy actions.
Platform project state is updated on both queue-only deploys and normal
upstream deploys, so a successful sync/deploy clears `queued` and
refreshes the latest commit columns instead of relying only on deployment
history.
Imported project builds share the same deployment build lock as platform
blue/green builds. If platform, standby, recovery, or another imported
project build is already active, the project poll is deferred instead of
starting a second Docker build.
15. The watcher no longer prebuilds `main` or advances `production` on its own.
Advance production outside the watcher through the release process, then
let the watcher deploy the locked branch that is already checked out on the
host.
16. Rollback pins intentionally pause normal upstream sync for the pinned
deployment state. Remove the pin when the locked branch contains the
corrective commit that should resume normal deployment.
17. Infrastructure operators can queue
`tmp/docker-web/watch/control/blue-green-deployment-revert.request.json`
to revert production to a retained successful deployment. The watcher keeps
the 5 newest unique successful deployed image tags for instant revert; a
cached revert verifies the cached target first, cancels any active
blue/green build,
retags the selected image, starts active/standby with `--no-build`,
health-checks through the normal proxy path, records deployment kind
`instant-revert`, and writes a deployment pin so normal upstream
deployment does not immediately overwrite the rollback. Older retained deployments remain
revertable through the existing rollback pin path, which may rebuild
because no cached image is available and still respects the active build
lock.
18. After forward database migrations finish, blue/green workflows remove
completed `hive-db-migrate` and `supermemory-db-migrate` containers by
Compose service labels. This catches stopped one-off `docker compose run`
containers such as `tuturuuu-hive-db-migrate-*` that can otherwise make the
Docker cluster look unhealthy even after the migration succeeded.
19. If watcher runtime code such as `scripts/watch-blue-green-deploy.js`,
`scripts/docker-web/blue-green.js`, or `scripts/docker-web/env.js` changed
in the pulled revision, the current watcher does not deploy from the old
process. It releases its lock, spawns a replacement watcher with the same
CLI args, and exits first.
20. The replacement watcher refreshes the live `web-proxy` nginx config and
workers in place if blue/green is already serving traffic, verifies proxy
routing through `/__platform/drain-status`, and only then starts the new
blue/green build/promotion.
21. If compose or helper-image wiring changed, including
`docker-compose.web.prod.yml`, Hive service files, MarkItDown service files,
`apps/storage-unzip-proxy` package/source files, or
`apps/web/docker/cron-runner*`, the containerized watcher recreates its own
compose service before the pending deploy handoff. The deploy then includes
only the affected buildable helper images in the blue/green build command
instead of rebuilding every service on every commit.
22. Retries recoverable Git command failures instead of exiting. The first retry
waits 1 minute, then the watcher backs off exponentially on consecutive Git
failures up to a 15 minute ceiling.
23. Caps deployment attempts at 3 failures per commit. A recovered pending
handoff failure is recorded, the pending request is cleared, and the
watcher keeps polling; once the cap is reached, that commit reports
`retry-limited` until a new commit is available or an operator pins a
different deployment.
24. Stops immediately if the checked-out branch changes while the watcher is
running.
25. If another watcher already owns the lock, a new invocation can fail with
guidance, mirror the active watcher with `--resume-if-running`, or replace
it with `--replace-existing`.
Operational notes for the containerized watcher:
* Manual `bun serve:web:docker:bg` and watcher-triggered deploys share a
deployment-build lock at
`tmp/docker-web/watch/blue-green-deployment-build.lock`. This lock is
separate from `blue-green-auto-deploy.lock`: the watcher may remain alive, but
only one build/deploy phase can be active across manual deploys, watcher
upstream sync, standby refreshes, rollback pins, cached recovery, and
reconcile deploys.
* If a manual deploy sees that lock or a live watcher status of `building` or
`deploying`, an interactive terminal prompts before it interrupts the active
deployment. Confirming stops `web-blue-green-watcher`, stops/resets the
Compose-owned BuildKit work, clears the active build lock/status, records the
interrupted entry as `canceled`, then starts the requested deployment alone.
* Non-interactive manual automation fails fast on an active deployment unless
`--cancel-active-build` or `DOCKER_WEB_CANCEL_ACTIVE_BUILD=1` is provided.
Use that override only when it is acceptable to interrupt all BuildKit work
owned by the platform deployment stack.
* Re-running `bun serve:web:docker:bg:watch` intentionally recreates the
watcher container so it picks up local repo changes, new CLI args, and
watcher-image updates in one path.
* The host log follower treats Docker's `143` exit from an intentionally
recreated watcher container as a reconnect signal, then reattaches to the
replacement service instead of leaving the terminal dark.
* If the followed watcher logs explicitly request host-supervised watcher
service recreation, the host wrapper force-recreates `web-blue-green-watcher`
before reattaching. Do not rely only on Docker's restart policy in this path:
the old container can briefly report healthy while still running the stale
image/runtime.
* Git fetch/pull credentials now need to be usable inside the watcher
container because the watcher no longer runs directly on the host.
* Full Docker daemon or Docker Desktop crashes cannot be recovered by a watcher
that is itself running inside Docker. Keep `bun serve:web:docker:bg:watch`
running from the host, ideally under `systemd`, `launchd`, or another host
process supervisor. That host command waits for the Docker daemon to respond
again, reruns the watcher compose `up --build --detach --force-recreate`, and
then resumes tailing logs. Every hosted project with its own Docker watcher
needs its own host-side watch process; container `restart:` policies only help
after Docker is already healthy again.
* The host Docker recovery loop polls bounded Docker CLI probes every 5 seconds
by default. Override with `DOCKER_WEB_WATCHER_DOCKER_RECOVERY_POLL_MS`; tune
each quick probe with `DOCKER_WEB_WATCHER_DOCKER_PROBE_TIMEOUT_MS`. By default
it waits indefinitely because a host process manager is expected to own the
terminal process; set `DOCKER_WEB_WATCHER_DOCKER_RECOVERY_TIMEOUT_MS` to a
positive value to fail after a bounded recovery window.
* The watcher image lives at `apps/web/docker/blue-green-watcher.Dockerfile`.
* Its entrypoint wrapper relaunches the watcher in-place when
`scripts/watch-blue-green-deploy.js` requests a self-restart after pulling a
new watcher revision.
* The entrypoint is also the watcher supervisor. It restarts the child process
after crashes, after the status snapshot fails to appear during startup, or
after `blue-green-auto-deploy.status.json` becomes stale. The compose service
uses `restart: unless-stopped` so Docker also brings the watcher back after a
daemon or container failure.
* A stale status snapshot is tolerated while the snapshot already shows an
active `building` or `deploying` deployment. During a long `docker compose
build`, the watcher child is intentionally busy inside the deploy command and
may not rewrite the status file until the command exits. The wrapper keeps the
child alive until `DOCKER_WEB_WATCHER_BUILD_TIMEOUT_MS` plus a short grace
window, then treats the stale snapshot as unhealthy.
* `bun serve:web:docker:bg:down` also stops the watcher service because it is
part of the production compose stack now.
Dashboard details:
* Shows the current active blue/green color when `web-proxy` is serving live
traffic.
* Docker resource rows use the running containers directly as a fallback when
`docker compose ps` cannot inspect the prod stack because of env
interpolation issues, so watcher metrics can still appear on an already-live
deployment.
* Docker stats are read with an explicit field format instead of Docker's
version-dependent JSON object shape, which avoids bogus `0` CPU/memory
readings when the watcher is running against a different Docker release.
* The watcher parser also normalizes locale-style decimal commas from
`docker stats`, so hosts that emit values like `0,10%` or `24,0MiB` no
longer collapse into zeroed metrics.
* Each watcher snapshot now includes `docker ps` metadata for every running
container visible through the host Docker socket, plus compose service health
for containers in the production project. The monitoring overview uses that
persisted snapshot to show service health and a full running-container
inventory without mounting the Docker socket into `apps/web`.
* The request archive view computes route summaries, status totals, RSC counts,
and error totals across the selected timeframe instead of only the visible
page. The default timeframe is seven days, the API rejects unbounded or
oversized windows, and operators can query at most 30 days of retained request
logs at a time. The web API keeps a short in-process aggregate cache keyed by
bounded timeframe plus telemetry log file stats, but the cache stores only
aggregate analytics so request rows are not retained in memory between page
reads.
* Drive ZIP extraction does not stream extracted file bytes back through the web
app proxy. The unzip worker requests a per-entry signed upload URL from the
callback route and uploads extracted files directly to trusted storage
origins only, which avoids nginx body-size limits for large WebGL artifacts
while keeping folder creation and auth checks in the backend callback.
* Hive is promoted with the web blue/green color: `hive-blue` and `hive-green`
are routed from `hive.tuturuuu.com`, and `hive-realtime` serves `/realtime` with
`HIVE_REALTIME_TOKEN_SECRET`, `HIVE_REALTIME_URL`, and
`NEXT_PUBLIC_HIVE_REALTIME_URL` configured in the same production stack.
Hive product data is stored in the Docker-managed `hive-postgres` service via
`HIVE_DATABASE_URL`; Supabase remains the identity/session source only. The
`web`, `web-cron-runner`, `hive-{color}`, and `hive-realtime` services must
all receive that URL so API routes, disabled-by-default simulation cron, the
editor, and the CRDT realtime service share the same Hive product database.
Optional local LLM support runs behind the `hive-ollama` profile and is
disabled unless operators enable the profile and Hive settings enable the
exact `gemma4` model.
Production compose publishes **`127.0.0.1:7814:7814`** from `web-proxy`, not
from a direct Hive container, so host-local or Cloudflare tunnel traffic to
`localhost:7814` always reaches the currently promoted Hive color without
exposing staged migration ports on every host interface.
Deploys verify that the running `web-proxy` container has the required
loopback host bindings (`7803`, `7814`, and `7816`) and that its running image
matches the resolved Compose image before reusing it; if an older proxy was
created before Hive moved behind blue/green or before the nginx image pin
changed, the next deploy force-recreates the proxy so the host-level
Cloudflare Tunnel route can reach Hive on the expected proxy runtime.
The Hive color services use the same Supabase env source as `apps/web`:
runtime env files are shared, and production image builds mount the `web_env`
BuildKit secret so hidden-locale auth pages can prerender with the platform
Supabase URL.
**Deploy coordination:** `scripts/docker-web/blue-green.js` still scopes prod
builds by changed service group, but runtime promotion happens in staged
order: first the target `web-{color}`, then `hive-{color}` and
`hive-realtime`, then refreshed support services such as backend,
MarkItDown, storage-unzip-proxy, `web-cron-runner`, and optional Redis-backed
helpers. If `web-proxy` or `cloudflared` must be bootstrapped or recreated for
host-port changes, they start only after target web, Hive, and support
services are healthy, so `hive.tuturuuu.com` is not exposed with an empty
`hive_app_upstream` and web is not publicly switched before dependent gates
finish. Promotion waits for the final proxy route check before writing
`active-color`.
* Every service owned by `docker-compose.web.prod.yml` should declare a
healthcheck, either directly in compose or in the image. The resources
inventory treats an `Up` container without Docker health metadata as healthy
for cross-project runtime visibility, but first-party prod services and
sidecars still need explicit probes so deploy gates can fail before promotion.
* The MarkItDown sidecar needs `SUPABASE_URL` set to the same Docker-internal
Supabase URL used by the web container. The service validates signed Storage
URLs before downloading attachments, and local Docker runs may use
`host.docker.internal` over HTTP.
* MarkItDown source changes and storage-unzip-proxy package/source changes are
part of the watcher refresh globs. Keep those globs in sync with any future
sidecar entrypoints so a running watcher refreshes helper containers during
the next deploy handoff, not just the web app container.
* Host dependency refreshes must not rewrite `bun.lock` on the production host.
The watcher resets a dirty lockfile by default with the rest of the disposable
deployment checkout. With `DOCKER_WEB_WATCHER_WORKTREE_RESET_DISABLED=1`, a
dirty lockfile remains a blocking worktree change. Automatic dependency sync
uses `bun install --frozen-lockfile` only.
* Runtime upgrades are an explicit operator action. The watcher does not run
`bun upgrade`; update the host Bun runtime only after reviewing the pinned
version in the repository and the watcher image.
* Recoverable Git poll failures stay visible in the dashboard as a retrying
watcher state instead of terminating the process, and the next-check timer
reflects the active backoff delay.
* If `git reset`, `git fetch`, `git checkout`, or the reset-disabled
`git pull --ff-only` path fails only because a Git lock already exists, the
watcher inspects the lock age. Fresh locks are polled inside the watcher
process, then removed automatically only when they are stale (older than 2
minutes). This covers linked-worktree `index.lock` files,
`.git/packed-refs.lock`, and remote-ref locks such as
`.git/refs/remotes/origin/staging.lock`.
* Build/deploy failures also stay inside the watcher loop. The watcher records
failed attempts in deployment history, clears stale pending handoff files
after recovery failures, and stops retrying the same commit after the third
failed deployment attempt. Once that cap is reached, it reports the
retry-limited state once for that commit instead of logging the same skip on
every poll.
* Normal promotions keep the long-lived `web-proxy` container and bound port
stable, which avoids transient listener drops for upstreams such as
Cloudflare Tunnel that are connected to `:7803`. Proxy container recreates are
reserved for required host-port or image drift and happen only after the
replacement web/Hive lane is healthy.
* Persists recent deployment history, including manual
`bun serve:web:docker:bg` runs, and renders the top 3 most operationally
relevant entries as stacked terminal cards that favor vertical scanability
over very wide lines.
* Each deployment card now uses a stronger header with status/color badges plus
grouped metric bands, so active traffic state, rollout intent, and
request-rate data are easier to scan while multiple cards are stacked.
* As soon as a new commit starts rolling out, the recent deployment section
shows it immediately as `DEPLOYING` instead of waiting for the rollout to
finish.
* Each deployment block includes:
* deploy status (`ACTIVE`, `ENDED`, or `FAILED`)
* build time
* activation/finish time
* deployment lifetime while it served traffic
* total requests served during that deployment window
* average requests per minute
* peak requests per minute
* `day`: requests served on the current day for the active deployment, or the
final active day for an ended deployment
* `davg`: average requests per day across that deployment's serving lifetime
* `dpeak`: busiest single-day request count across that deployment's serving
lifetime
* The live blue/green summary uses the same traffic metrics as the deployment
history cards, with consistent color coding for build/lifetime/traffic/age
metrics so the dashboard is easier to scan quickly.
* A dedicated Docker resources row summarizes aggregate CPU, memory, and
network usage across the live blue/green containers, followed by a
per-container row for `proxy`, `green`, and `blue` when those services are
running. This is sampled from `docker stats --no-stream`, so it stays local
to the host and is appropriate for self-hosted operator monitoring.
* The infrastructure dashboard's Docker Runtime Inventory uses the watcher
snapshot as the source of truth for every running Compose container and
derives total CPU and memory from those rows when present, so the summary
cards stay aligned with the detailed container inventory.
* The bundled `serverless-redis-http` companion uses an in-container `wget`
health check that posts `["PING"]` to `/` with the generated `SRH_TOKEN`;
do not use a Node-based probe for that image because it is an Erlang release
image, and do not probe `/ping` because SRH does not expose that route.
* Production Redis compose requires `UPSTASH_REDIS_REST_TOKEN` and binds Redis
host ports to `127.0.0.1`. Do not reintroduce the
`platform-local-redis-token` fallback in production fragments or remove the
loopback host bind; direct Compose users must export a strong token before
enabling the `redis` profile.
* After a successful host-triggered `serve:web:docker:bg` rollout, the Docker
helper starts or resumes the containerized `web-blue-green-watcher` with
`--resume-if-running`. Deploys that are already running inside the watcher
skip that handoff via `PLATFORM_BLUE_GREEN_WATCHER_CONTAINER=1`, which avoids
recursive watcher starts while still leaving a poller alive for future Git
commits.
* The watcher wrapper and child process must agree on runtime files. If
`PLATFORM_BLUE_GREEN_WATCH_ARGS_FILE`,
`PLATFORM_BLUE_GREEN_WATCH_RUNTIME_DIR`, or
`PLATFORM_BLUE_GREEN_WATCH_STATUS_FILE` are set, both the wrapper and child
use those paths so the wrapper does not restart a healthy child for a missing
status snapshot.
* Request counters now come from a persisted local proxy-log drain under
`tmp/docker-web/watch/blue-green-request-telemetry.*`, not from one-off
`docker logs` scrapes in the dashboard. The watcher continuously drains
structured `web-proxy` access logs into a local ledger, so request metrics
survive watcher restarts and do not require any external analytics service.
* Internal proxy health checks for `/api/health` and
`/__platform/drain-status` are excluded from the request totals so the
numbers reflect real served traffic more closely.
* The proxy now emits structured JSON access logs that include the upstream
deployment stamp and blue/green color. That lets the watcher link requests
back to the correct deployment instead of only estimating by time window.
* For each newly drained proxy request, the watcher also reads recent
stdout/stderr from the selected frontend lanes (`web-blue` / `web-green` for
`DOCKER_WEB_FRONTEND=next`, or `tanstack-web-blue` / `tanstack-web-green` for
`DOCKER_WEB_FRONTEND=tanstack`) and stores up to 20 route console lines that
fall inside the request latency window. Those captured lines are persisted on
the request-log record itself so the request explorer can show request-scoped
server console output after the live Docker logs have moved on.
* The watcher retains up to 10,000 deployment history entries and up to
100,000,000 recent drained request-log records on disk, bounded by a 256 MiB
durable request-log byte cap by default. When the next request record would
exceed the byte cap, the watcher rotates the current JSONL chunk if needed
and prunes older chunks before appending, so public request URIs cannot grow
the host-backed ledger without an aggregate limit. Rolling
daily/weekly/monthly/yearly metric buckets plus a recent-request excerpt still
feed the monitoring dashboard.
* The watcher also persists a separate latest-log ledger under
`tmp/docker-web/watch/blue-green-auto-deploy.logs.json`, which captures the
high-level poll/pull/build/deploy watcher messages with deployment stamps and
commit hashes when available. The infrastructure dashboard uses that ledger
for a deployment-scoped latest-log view without needing live `docker logs`
access.
* The watcher uses the same Docker runtime env resolution as the real deploy
flow, so blue/green status probes still work when the Redis profile is part
of the production compose file.
* The active watcher also persists a live status snapshot under
`tmp/docker-web/watch/`, which is what `--resume-if-running` uses to mirror
the dashboard without taking over the PID lock.
* The infrastructure dashboard at
`/{ROOT_WORKSPACE_ID}/infrastructure/monitoring` reads the same watcher
status snapshot and renders it as a Next.js control room with rollout,
request-rate, container-resource, and event-feed views.
* The monitoring dashboard now exposes paginated request and watcher-log
explorers. Route filters come from normalized request paths, the raw request
URI still surfaces query signatures, and `?_rsc=*` requests are called out so
React Server Component traffic is inspectable separately from document hits.
* Deployment-facing dashboard surfaces deduplicate successful blue/green rows
for the same commit so active and standby colors do not appear as separate
rollouts. Failed attempts remain separate because the retry cap and recovery
debugging depend on seeing each failed build/deploy attempt. Large deployment,
rollback-candidate, Docker-service, and container lists are paginated in the
UI instead of rendering every retained row at once.
* Production `web`, `web-blue`, and `web-green` containers mount
`./tmp/docker-web` read-only at `/app/runtime/docker-web` and use
`PLATFORM_BLUE_GREEN_MONITORING_DIR` to find the watcher snapshot. Keep that
mount/env pair in sync if the runtime path changes, or the dashboard will
degrade to an empty offline state even while blue/green deployments still
work.
* Production `web`, `web-blue`, and `web-green` also mount the narrower
`./tmp/docker-web/watch/control` path read-write at
`/app/runtime/docker-web-control` via `PLATFORM_BLUE_GREEN_CONTROL_DIR`.
Keep operator command files in that control directory so the broader watcher
runtime and telemetry mount can stay read-only.
* The monitoring dashboard's "Sync Standby Now" action writes
`tmp/docker-web/watch/control/blue-green-instant-rollout.request.json`. The
watcher consumes that file on its next poll, clears it after a success,
failure, or no-op, and uses it to rebuild the standby color immediately so
blue and green can converge on the same commit without waiting for the stale
standby window.
* The dashboard reads that pending instant-rollout request back from the
watcher control directory. While the request is queued, or while the latest
standby refresh is building/deploying, the sync button stays disabled and
shows a queued/building status instead of allowing duplicate control files.
* The monitoring dashboard's rollback pin action writes
`tmp/docker-web/watch/control/blue-green-deployment-pin.json`. The watcher
treats that file as authoritative: it skips normal fetch/pull work, checks out
the pinned commit in detached mode, deploys it if the latest successful
deployment is different, and keeps production on that commit until the pin is
removed from the dashboard. Removing the pin lets the watcher check out its
locked branch again and resume normal fast-forward polling.
* In-container watcher child restarts preserve the locked branch/upstream
metadata even when the child is killed while Git is detached for a rollback or
parent-fallback build. The replacement child can recover `production` from the
target-only lock instead of trying to poll detached `HEAD`. If an older child
already removed the lock, a clean detached startup falls back to the selected
platform branch (`production` by default) before locking and polling.
* When the watcher receives a shutdown signal while it is temporarily detached,
it attempts to check out the locked branch again before exiting, as long as
the worktree is clean. This keeps manual operator commands such as
`git pull && bun serve:web:docker:bg` from inheriting a detached checkout
after a stopped watcher.
* The watcher image must also include both the Docker Compose and Buildx CLI
plugins (`docker-cli-compose` and `docker-cli-buildx` on Alpine) because the
rollout handoff shells into `docker compose ...`, and capped production
builds create/use the remote `tuturuuu` Buildx builder from inside the watcher
container. The watcher reaches the Compose-owned BuildKit daemon at
`tcp://buildkit:1234`.
* When the watcher drives Docker Desktop through `/var/run/docker.sock`, it
must run the deploy handoff from the mirrored host-path mount, not a
container-only path like `/workspace`. Otherwise Docker Desktop rejects bind
mounts such as `./tmp/docker-web/prod/nginx.conf` with "mounts denied"
because `/workspace/...` is not a real shared host path.
* The watcher compose environment preserves `PLATFORM_HOST_WORKSPACE_DIR` and
pins `COMPOSE_PROJECT_NAME` from that host checkout path unless
`DOCKER_WEB_COMPOSE_PROJECT_NAME` is explicitly set. For linked worktrees, the
same compose env injects `DOCKER_WEB_GIT_COMMON_DIR` and the watcher service
mounts that directory at the same absolute path inside the container. This
prevents `fatal: not a git repository` failures for `.git/worktrees/`
paths when `.git` points outside the worktree mount. A canonical checkout
directory named `platform` maps to the `tuturuuu` Compose project so Docker
Desktop groups the stack under the product name on clean startups. During
self-refresh from an already-running legacy `platform` Compose project, the
legacy watcher starts a staged `tuturuuu` watcher with non-conflicting host
ports, then stops only the old watcher service. The target watcher builds or
recovers the `tuturuuu` stack before it touches the public proxy port. When
the target proxy is healthy on the staged port, it stops the legacy
`platform` proxy, recreates the `tuturuuu` proxy on port `7803`, and verifies
the internal drain-status route within 3 seconds. If that handoff exceeds
3 seconds or the target proxy health check fails, the watcher stops the
target proxy, restores the legacy `platform` proxy, and leaves the legacy
project intact for another retry. After a successful handoff, it removes the
old `platform` Compose project with `docker compose down --remove-orphans`.
Once the legacy project is absent, a watcher that lacks inherited Compose
project env is treated as the fully migrated `tuturuuu` watcher and remains in
the normal Git poll/build loop instead of starting another migration handoff.
Do not inherit arbitrary container-scoped Compose project names from the
watcher container; doing so can create duplicate service names such as nested
`tuturuuu-markitdown-1` containers during self-refresh.
* If Docker reports `container name ... is already in use` for a requested
production service, the helper removes only the exact expected container name
for the current Compose project, then retries `docker compose up`. This handles
stale names left by interrupted rollouts without pruning unrelated containers.
* If `docker compose up` hits a transient Docker registry or Docker Hub auth
timeout while pulling support images, the helper retries the Compose start
without treating it as a deployment failure. Tune the bounded backoff with
`DOCKER_WEB_COMPOSE_UP_RETRY_MAX_ATTEMPTS`,
`DOCKER_WEB_COMPOSE_UP_RETRY_INITIAL_DELAY_MS`, and
`DOCKER_WEB_COMPOSE_UP_RETRY_MAX_DELAY_MS`. Stale dependency container
references use `DOCKER_WEB_COMPOSE_UP_STALE_DEPENDENCY_RETRY_MAX_ATTEMPTS`,
so registry retry tuning cannot accidentally disable recovery from Compose
referencing a removed dependency container. Non-transient Compose errors still
fail immediately.
* If `log-drain-postgres` starts but remains unhealthy before promotion, the
helper removes and recreates only that service container once, collects
`docker compose ps`, container inspect state, recent service logs, and
matching Compose volume names, then continues the rollout with
`PLATFORM_LOG_DRAIN_ENABLED=false` for that run. The production `web` and
`web-blue-green-watcher` services intentionally do not use Compose
`depends_on` for log-drain; the script-owned preflight is the only log-drain
gate. Platform traffic can promote while telemetry and Infrastructure project
automation are degraded.
* Set `DOCKER_WEB_LOG_DRAIN_REQUIRED=1` when an operator wants
`log-drain-postgres` to remain a hard deployment gate. Use that only when the
log-drain database is part of the deployment objective and a blocked rollout
is preferable to serving without persisted request/server logs.
* To inspect a degraded log-drain startup without changing data, run:
```bash theme={null}
docker compose -f docker-compose.web.prod.yml --profile redis ps --all log-drain-postgres
docker compose -f docker-compose.web.prod.yml --profile redis logs --tail 200 log-drain-postgres
docker volume ls --filter label=com.docker.compose.volume=platform-log-drain-postgres
```
If those logs mention incompatible database files, data-directory corruption,
or an invalid checkpoint, back up or migrate the matching Compose volume
before retrying. Do not run `docker compose down --volumes`,
`docker volume rm`, or any other volume-clearing command for log-drain data
unless an operator has explicitly approved a backed-up reset.
* Starting `bun serve:web:docker:bg:watch` now clears the persisted watcher
status snapshot and active PID before the watcher service is force-recreated,
but preserves any complete branch/upstream target metadata. A stale lock from
the previous container cannot block the replacement watcher, and a detached
checkout still has enough metadata to reattach to `production`.
* If the watcher pulls a revision that changes its own Dockerfile or baked
entrypoint, it now rebuilds and recreates the `web-blue-green-watcher`
service automatically before handing off to the next deployment cycle.
* When that container-refresh request is emitted from inside the followed
watcher logs, `bun serve:web:docker:bg:watch` treats the log text itself as
the recreate signal, then rebuilds/recreates and resumes tailing. This keeps
the service from getting stuck in a Docker-restarted but operationally offline
state.
* Recovery handoffs now persist a pending-deploy request under
`tmp/docker-web/watch/` and reconcile it against the latest successful
deployment history entry on startup. If `HEAD` is newer than the last
successful built/deployed commit after a watcher restart or container
recreate, the watcher builds the current `HEAD` before settling back into
normal polling.
* The same reconciliation now also runs during steady-state polling: if Git is
already up to date but the latest successful deployment record still points
at an older commit, the watcher rebuilds/deploys the current `HEAD` instead
of incorrectly reporting `up-to-date`.
* Blue/green deploys build the replacement lane before stopping or removing any
existing blue/green container. A failed `docker compose build` must leave the
currently serving lane and any warm standby untouched; only after the build
succeeds may the target lane be recreated with `--no-build`.
* After a watcher-owned child deploy command fails, the watcher prunes
failed-build residue before returning to the poll loop: it prunes the
configured Buildx builder when `BUILDX_BUILDER` or
`DOCKER_WEB_BUILD_BUILDER_NAME` is available, then runs
`docker image prune --force --filter dangling=true`. Set
`DOCKER_WEB_WATCHER_PRUNE_FAILED_BUILD_RESIDUE=0` only when an operator needs
to preserve failed build layers for debugging. When the child deploy failed
with a BuildKit transport/resource signature, the watcher also runs the
best-effort exec-cache cleanup and recreates the Compose-owned `buildkit`
service so the next poll does not inherit the dead builder endpoint.
* If `tmp/docker-web/prod/active-color` is missing or stale, the deployer and
watcher recover the serving lane from the generated nginx proxy config before
deciding what to rebuild. Treat the proxy config as the runtime source of
truth during drift so a failed reconciliation cannot misclassify and clear a
stable deployment.
## Browser-State 502 Recovery
If some normal browsers still return Cloudflare `502 Host Error`, `431`, or
Chrome `ERR_INVALID_RESPONSE` while incognito works, treat it as stale client
state or an auth-cookie/header-size problem before assuming the tunnel itself is
broken. Normal browser-state recovery should use the app route: GET is a
no-store confirmation page only, and the destructive `Clear-Site-Data` response
happens only after a same-origin POST.
Oversized request headers are different: the request may never reach Next.js.
The production `web-proxy` therefore gives web, Hive, and Meet 64 KB request
header headroom, maps Nginx `431`/`494` oversized-header failures to a local
browser-state recovery response, and the Docker web app server starts with
`--max-http-header-size=65536` so ordinary Supabase auth-cookie chunking has
matching headroom after proxying.
How to recognize each failure mode:
* `upstream sent too big header while reading response header from upstream`
means nginx response-header buffers were too small for the auth response.
* `web-green could not be resolved` or `web-blue could not be resolved` means a
stale nginx worker or keepalive connection still tried to reach a color that
no longer existed. The current warm-standby model is designed to avoid that.
* A browser that fails only in regular mode but works in incognito usually has
stale Supabase auth cookies, stale service-worker state, or both.
Recovery path:
* Send affected users to the recovery route on the affected origin:
`https://tuturuuu.com/~recover-browser-state` for the main app, or
`https://hive.tuturuuu.com/~recover-browser-state` for Hive.
* That route is public and bypasses auth/onboarding middleware. When the
request can reach the app, use the confirmation form so the cookie-clearing
POST explicitly expires Supabase auth cookie variants for the current host.
* If the browser is already sending too many stale cookies, the proxy catches
the `431`/`494` before Next.js and returns
`Clear-Site-Data: "cache", "cookies", "storage", "executionContexts"` while
redirecting the browser back to `/login?browserStateReset=1`.
Operational signals:
* Inspect `X-Platform-Deployment-Stamp`, `X-Platform-Blue-Green-Primary`, and
`X-Platform-Blue-Green-Color` response headers to confirm which rollout is
currently serving a request.
* If the recovery URL fixes the issue for a user, the likely root cause was
stale browser state rather than an active deploy outage.
* If recovery does not help and proxy logs still show `too big header`, focus
on auth redirect size or additional cookie bloat.
Operational notes:
* This is intended for disposable deployment clones on a server, not for active
developer worktrees or manual hotfix edits.
* By default the watcher resets tracked changes, deletes untracked files and
directories, fetches, and hard-resets to the locked upstream when the local
checkout is behind, ahead, or diverged.
* Set `DOCKER_WEB_WATCHER_WORKTREE_RESET_DISABLED=1` before starting the
watcher to preserve the old protective behavior: dirty worktrees block,
ahead/diverged branches are skipped, and only fast-forward pulls are
attempted.
* The self-restart path only triggers when the watcher script itself changed in
the fetched revision; normal app-code deploys keep the current watcher
process alive.
* During that self-restart path, nginx keeps the assigned proxy port up the
whole time because the replacement watcher refreshes the existing proxy
container in place before it starts the new build.
* The watcher inherits the default blue/green build caps from
`bun serve:web:docker:bg`, so the current defaults still apply during
auto-deploys.
* Deployment history is watcher-managed. Manual blue/green rollouts still show
up in the live runtime status if the stack is active, but they do not
backfill the watcher’s last-3 deployment list unless they were performed
through the watcher itself.
* Rollback pins are intended for bad latest deployments or failed reconciliation
builds. Pin only a known successful deployment from the retained ledger, then
remove the pin after `main` contains the corrective commit you want the
watcher to resume deploying.
## Validation And CI
`docker-setup-check.yaml` now validates all of the following:
* `node scripts/check-docker-web.js`
* `node --test scripts/check-docker-web.test.js scripts/docker-web.test.js scripts/run-tanstack-e2e-docker.test.js`
* `docker compose -f docker-compose.web.yml config`
* `docker compose -f docker-compose.web.yml --profile redis config`
* `docker compose -f docker-compose.web.yml --profile cloudflared config`
* `docker compose -f docker-compose.web.prod.yml config`
* `docker compose -f docker-compose.web.prod.yml --profile redis config`
* `docker compose -f docker-compose.web.prod.yml --profile cloudflared config`
* `docker compose -f docker-compose.tanstack-dual.yml config`
* `docker buildx build --load --cache-from type=gha,scope=docker-backend --cache-to type=gha,scope=docker-backend,mode=max -f apps/backend/Dockerfile .`
* `docker buildx build --load --target dev --cache-from type=gha,scope=docker-web-dev --cache-to type=gha,scope=docker-web-dev,mode=max -f apps/web/Dockerfile .`
* `docker buildx build --load --target runner --secret id=web_env,src=apps/web/.env.local --cache-from type=gha,scope=docker-web-prod --cache-to type=gha,scope=docker-web-prod,mode=max -f apps/web/Dockerfile .`
* `docker buildx build --load --target runner --cache-from type=gha,scope=docker-tanstack-web-prod --cache-to type=gha,scope=docker-tanstack-web-prod,mode=max -f apps/tanstack-web/Dockerfile .`
That means Docker CI now covers the dev image, the Next.js production image,
the TanStack production image, the dual-stack compose file, and the optional
Cloudflare Tunnel profile rendering.
For focused watcher-script checks, run the root Node test file directly, for
example `node --test --test-name-pattern "pullTrackedBranch"
scripts/watch-blue-green-deploy.test.js`. Running `bun test scripts/...` from
the repo root invokes the package test script and can expand into the full Turbo
test suite.
When `scripts/check-docker-web.js` or similar root validators need to assert a
literal Dockerfile template placeholder like `${process.env.PORT || 7803}`,
prefer a regex or another explicitly escaped matcher instead of a plain string
literal. Biome treats raw `${...}` text inside normal strings as
`lint/suspicious/noTemplateCurlyInString`, which can break CI even when the
runtime behavior is unchanged.
## Operator Notes
* Do not paste `docker compose config` output into chat or tickets; it expands env values.
* If you need rebuild-before-restart on a server, use `bun serve:web:docker:bg`.
* If the latest blue/green deployment is bad, use the infrastructure monitoring dashboard to pin a previous successful deployment before debugging forward.
* If a blue/green deploy is interrupted, rerunning the same command from the intended commit is the normal recovery path.
# Build & Ship
Source: https://docs.tuturuuu.com/build/overview
Set up your environment, work with the monorepo, and ship confidently.
This section aggregates everything you need to contribute to Tuturuuu—from local setup to CI/CD.
## Start Here
1. **Environment Setup** – Follow the [Development Guide](/build/development-tools/development) to install Bun, Supabase, and required tooling.
2. **Understand the Monorepo** – Dive into our [Monorepo Architecture](/build/development-tools/monorepo-architecture) for workspace conventions and dependency flows.
3. **Ship and Operate** – Review [DevOps & Deployment](/build/devops/overview) before touching infrastructure, workflows, or release paths.
## Build Toolkit
```mermaid theme={null}
flowchart LR
DevSetup[Local Development]
Monorepo[Monorepo Architecture]
Supabase[Local Supabase]
CICD[CI/CD Pipelines]
Docs[Documentation Standards]
DevSetup --> Monorepo
DevSetup --> Supabase
Monorepo --> CICD
Supabase --> CICD
CICD --> Docs
Docs --> DevSetup
```
* **Clean Environments** – Use the [Cleaning Clone](/build/development-tools/cleaning-clone) guide when local builds misbehave.
* **Deployment Runbooks** – Use the [DevOps & Deployment](/build/devops/overview) section for GitHub Actions, Docker rollout, environments, and secrets.
* **Documenting** – Keep docs current with the [Documenting Workflow](/build/development-tools/documenting).
When you are ready to explore the product experience, head over to the [Platform section](/platform/overview). To sharpen AI and experimentation skills, continue to [Learn](/learn/overview).
# Skills Installation
Source: https://docs.tuturuuu.com/build/skills/installation
Install Tuturuuu skills from the public platform repository.
## Install A Focused Skill
Use the `skills` CLI from the project where the agent should load the skill.
For Codex, install one focused skill like this:
```bash theme={null}
npx skills add tutur3u/platform --agent codex --copy -y \
--skill tuturuuu-platform
```
Replace `tuturuuu-platform` with any skill from the catalog. Common starting
points:
* `tuturuuu-platform` for general platform repo work.
* `tuturuuu-agent-coordination` for shared or dirty worktrees and short
5-10-minute commit-window coordination.
* `tuturuuu-commit` for scoped commit follow-through with exact staging and
commit-window claim/wait/release.
* `tuturuuu-pr-merge-sync` for quiet-window PR merge follow-through, main-green
verification before `bun git-sync`, and production checks.
* `tuturuuu-cli` for `ttr` and SDK workflows.
* `tuturuuu-database` for Supabase migrations, RLS, and protected data access.
## Install The Full Tuturuuu Set
Install all public Tuturuuu skills for Codex with explicit skill names:
```bash theme={null}
npx skills add tutur3u/platform --agent codex --copy -y \
--skill tuturuuu-agent-coordination \
--skill tuturuuu-ci-docs \
--skill tuturuuu-cli \
--skill tuturuuu-cli-finance \
--skill tuturuuu-cli-tasks \
--skill tuturuuu-cms-studio \
--skill tuturuuu-commit \
--skill tuturuuu-database \
--skill tuturuuu-devbox-ops \
--skill tuturuuu-development-tooling \
--skill tuturuuu-e2e-auth-debugging \
--skill tuturuuu-external-apps \
--skill tuturuuu-mobile-task-board \
--skill tuturuuu-platform \
--skill tuturuuu-pr-merge-sync \
--skill tuturuuu-review-comments \
--skill tuturuuu-satellite-app-ux \
--skill tuturuuu-validation-offload \
--skill tuturuuu-web-release
```
Run the command from the target project checkout. For a one-time skills.sh
discovery trigger, run it from a disposable directory after the metadata commit
has been pushed to GitHub.
## Verify Discovery
Before pushing metadata changes, validate local repo-root discovery:
```bash theme={null}
npx skills add . --list
```
The output should include all `tuturuuu-*` skills. After the metadata is pushed
and the public install trigger has run, check the cached public page after it
refreshes:
```text theme={null}
https://skills.sh/tutur3u/platform
```
# Skills
Source: https://docs.tuturuuu.com/build/skills/overview
Install and publish Tuturuuu agent skills for platform work.
## Overview
Tuturuuu agent skills package repo-specific operating knowledge for coding
agents. They help agents load focused guidance for platform work without
copying long instructions into every prompt.
The source skills live in `plugins/tuturuuu/skills`. The platform repo exposes
them to public skill installers through `.claude-plugin/marketplace.json`, and
the public skills.sh page is grouped with `skills.sh.json`.
## Skill Groups
The public catalog groups the Tuturuuu skills by workflow:
* Platform: monorepo, Supabase, CI/docs, release metadata, and mobile task-board
work.
* Agent Workflow: shared-worktree coordination, short 5-10-minute commit-window
claim/wait/release, scoped commits, review comments, quiet-window PR merge
sync, validation offload, and tooling improvements.
* CLI And Ops: core `ttr` workflows, task capture, finance commands, and remote
devbox operations.
* Product Surfaces: CMS studio, satellite app UX, and local E2E authentication
debugging.
## Maintenance
When a skill is added, renamed, or removed, keep these files aligned:
* `plugins/tuturuuu/skills//SKILL.md`
* `.claude-plugin/marketplace.json`
* `skills.sh.json`
* `apps/docs/build/skills/installation.mdx`
* `apps/docs/build/development-tools/codex-plugin.mdx`
Run the plugin validator after changing skill metadata:
```bash theme={null}
python3 plugins/tuturuuu/scripts/validate_plugin.py
```
Use the [installation guide](/build/skills/installation) to install skills from
the public repo or trigger skills.sh discovery after a metadata push.
# Examples
Source: https://docs.tuturuuu.com/learn/examples/overview
Practical examples and code samples for the Tuturuuu platform
This section contains practical examples and code samples demonstrating how to use various components and features of the Tuturuuu platform.
## Component Examples
### WorkspaceWrapper
Examples showing how to use the WorkspaceWrapper component in different scenarios.
The main [WorkspaceWrapper documentation](/platform/components/workspace-wrapper) includes comprehensive code examples demonstrating various usage patterns.
## Usage Patterns
### Server Components
```tsx theme={null}
// Basic server component with WorkspaceWrapper
export default async function MyPage({
params
}: {
params: Promise<{ wsId: string }>
}) {
return (
{({ workspace, wsId }) => (
;
}
```
### API Routes
```tsx theme={null}
// API route with workspace validation
export async function GET(
request: Request,
{ params }: { params: Promise<{ wsId: string }> }
) {
const { wsId } = await params;
const workspace = await getWorkspace(wsId);
if (!workspace) {
return Response.json({ error: 'Workspace not found' }, { status: 404 });
}
return Response.json({ workspace });
}
```
## Best Practices
### Error Handling
```tsx theme={null}
// Proper error handling in components
export default async function MyPage({ params }) {
return (
}
>
{({ workspace, wsId }) => {
try {
return ;
} catch (error) {
return ;
}
}}
);
}
```
### Loading States
```tsx theme={null}
// Proper loading state handling
export default async function MyPage({ params }) {
return (
}
>
{({ workspace, wsId }) => (
}>
)}
);
}
```
### Type Safety
```tsx theme={null}
// Proper TypeScript usage
interface PageProps {
params: Promise<{ wsId: string }>;
}
interface WorkspaceData {
workspace: Workspace & { joined: boolean };
wsId: string;
}
export default async function MyPage({ params }: PageProps) {
return (
{({ workspace, wsId }: WorkspaceData) => (
)}
);
}
```
## Common Patterns
### Dashboard Pages
Most dashboard pages follow this pattern:
1. Use WorkspaceWrapper for workspace resolution
2. Fetch additional data based on workspace
3. Render components with proper loading states
4. Handle permissions and access control
### Settings Pages
Settings pages typically:
1. Validate user permissions
2. Show different UI based on user role
3. Handle form submissions with proper validation
4. Provide clear feedback to users
### API Endpoints
API endpoints should:
1. Validate workspace access
2. Handle errors gracefully
3. Return appropriate HTTP status codes
4. Include proper error messages
## Contributing Examples
When adding new examples:
1. Use realistic, practical scenarios
2. Include proper TypeScript types
3. Show error handling
4. Include both server and client examples
5. Update this index page
# AI Chat
Source: https://docs.tuturuuu.com/learn/experiments/ai-chat
Talk to an AI-powered assistant that understands your workspace and helps you get things done, faster and easier.
## Mobile live mode
The mobile assistant now has two distinct interaction modes:
* Standard chat is the default assistant surface. Typed turns continue to use the normal text-model selection in the mobile shell.
* Live mode is a dedicated fullscreen Gemini Live experience. It is entered from the assistant mic control or the assistant chrome action and binds the session to `gemini-3.1-flash-live-preview`.
### Mobile cache warmup
The mobile assistant shell restores core metadata from `AssistantRepository`
cache helpers before revalidating: personal workspace resolution, Mira soul,
task and calendar insight, workspace credits, model catalog, recent chat
history, and restored chat detail. The `assistant_metadata` warmup task should
remain the entry point for preloading this data after boot, home resume, or
workspace changes.
### What fullscreen live mode does
* Immediately switches the assistant route into a live-first surface instead of silently preparing a background session.
* Auto-starts the microphone when the user enters live mode, so the transition feels responsive.
* Shows large voice-activity blobs for both the user and the assistant, plus a live transcript lane for drafts and synced turns.
* Shows the current live-session status detail inline in the fullscreen header, plus a contextual status panel with recovery actions for reconnecting and failure states.
* Keeps voice, camera, and typed turns inside the same assistant thread, so persisted live turns still appear in the standard chat history after leaving fullscreen live mode.
### Model split
* Mobile typed chat keeps the normal text-model path managed by `AssistantChatCubit` and the workspace-selected shell model.
* Mobile live mode always uses Gemini 3.1 Flash Live. Restoring a live-backed chat must not overwrite the normal text-model selection.
### Backend dependencies
The mobile app still uses the existing internal APIs for Live sessions:
* `/api/v1/assistant/live/token`
* `/api/v1/live/session`
* `/api/v1/live/tools/execute`
* `/api/v1/assistant/live/turns`
These routes mint constrained ephemeral Gemini Live tokens, persist resumable session handles, execute synchronous tool calls, and sync completed live turns back into the chat thread.
The token route now returns the latest resumable session handle together with the ephemeral token. Mobile uses that bundled handle to skip a follow-up read before opening the socket, and only falls back to loading seed chat history when there is no resumable handle or the user explicitly starts a fresh live conversation.
Gemini 3.1 Flash Live should stay on a single `AUDIO` response modality for the live session config. The mobile UI relies on input/output audio transcription streams for inline text instead of requesting simultaneous `TEXT` and `AUDIO` outputs.
The ephemeral token request should match Google’s current `v1alpha` auth-token shape: send `httpOptions.apiVersion = 'v1alpha'` on the token request itself, keep `uses` aligned with the default single-session flow, and prefer `lockAdditionalFields: []` so only the explicitly constrained setup fields are locked while client-side setup details like history/session resumption can still merge at connect time. When bootstrapping a fresh live assistant chat, omit `ai_chats.id` entirely unless a specific chat id is being resumed so Postgres can apply the default UUID.
Browser AI chat surfaces that use the shared `/api/ai/chat` streaming route must
create a durable `ai_chats` row through `/api/ai/chat/new` before the first
streaming turn. Chat identifiers are UUID database identifiers, not local UI
session keys; the streaming route verifies requested chat ownership before model
invocation so persistence, observability, and AI-credit deduction stay coupled.
## Web Chat and Live modes
The web dashboard presents Chat and Live as one assistant surface. Chat is the
default and remains mounted while Live is selected so draft text is preserved.
Entering Live requests microphone access and connects automatically; switching
back to Chat or pressing Escape closes media and returns focus to the composer.
Web Live access is credit-gated on every product tier. `/api/v1/live/token`
reserves the active Chat credit source and returns a single-use Gemini token
that expires after five minutes. The browser reports cumulative provider usage
to `/api/v1/live/usage`; private database functions reject regressing snapshots,
price each modality against the effective Gemini rate, and release unused held
credits when the session closes or expires. Keep provider prices versioned so
historical transactions remain auditable when Google changes its rates.
### Verification
For mobile-only live-mode changes, the required verification path is:
1. Update both ARB files when live-mode copy changes.
2. Run `flutter gen-l10n`.
3. Run `bun check:mobile`.
# Calendar
Source: https://docs.tuturuuu.com/learn/experiments/calendar
Manage your schedule and events without leaving your workspace.
This page is under construction.
# Finance
Source: https://docs.tuturuuu.com/learn/experiments/finance
Manage your workspace finances with ease.
This page is under construction.
# Form Builder
Source: https://docs.tuturuuu.com/learn/experiments/form-builder
Build forms with ease, integrated with all Tuturuuu services.
This page is under construction.
# Project Management
Source: https://docs.tuturuuu.com/learn/experiments/project-management
Manage tasks and monitor progress without any hassle.
This page is under construction.
# Introduction to Prompt Engineering
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/introduction
A foundational guide for NEO LEAGUE: Prompt the Future — empowering students to master the art of prompting and unlock AI’s full creative potential.
## What is a prompt
A **prompt** is a natural language input that instructs a generative AI on the task at hand. These AI models can create a variety of content like stories, conversations, videos, and more. The quality of the output depends on the prompt's clarity and context, as AI models need accurate details to produce meaningful and precise responses.
### Prompt content types
A prompt can include one or more of the following elements:
* [**Input** (required)](#input)
* [**Context**](#context)
* [**Examples**](#examples)
### Input
The input is the core part of a prompt. It tells the model what task to perform or what question to answer. Inputs come in different forms:
#### Question input
A **question input** asks the model for an answer, usually in the form of a direct question.
**Prompt Example:**
```prompt theme={null}
What's a good name for a flower shop that specializes in selling bouquets of dried flowers? Create a list of 10 options with just the names.
```
**Response:**
```response theme={null}
Here are 10 names for a flower shop specializing in dried flowers:
1. Everlasting Blooms
2. Dried & Delightful
3. The Petal Preserve
4. Whispers of Wildflowers
5. Forever Blooms
6. The Botanical Muse
7. Timeless Blooms
8. Dried Flower Co.
9. Bloom & Co.
10. The Sun-Kissed Petals
```
#### Task Input
A **task input** asks the model to perform a specific action—like generating ideas, making a list, writing a summary, or offering suggestions.
**Prompt Example:**
```prompt theme={null}
Give me a simple list of just the things that I must bring on a camping trip. The list should have 10 items.
```
**Response:**
```response theme={null}
Here's a list of 10 essential items for a camping trip:
1. **Tent:** Your shelter from the elements.
2. **Sleeping Bag:** Stay warm and comfortable.
3. **Sleeping Pad:** Insulation and cushioning for a good night's sleep.
4. **Headlamp or Flashlight:** Light for nighttime activities.
5. **First Aid Kit:** Be prepared for minor injuries.
6. **Water Bottle/Hydration Pack:** Stay hydrated.
7. **Food and Snacks:** Fuel for your adventures.
8. **Cooking Gear:** Camp stove, fuel, utensils, etc.
9. **Knife and Multitool:** Essential for various tasks.
10. **Camp Chair:** Enjoy your campfire and meals comfortably.
```
#### Entity Input
An **entity input** is when the prompt gives the model a specific piece of content to act on—like summarizing a paragraph, classifying a sentence, or translating a passage. It’s often paired with clear instructions on what to do with that content.
**Prompt Example:**
```prompt theme={null}
Classify the following items as [large, small].
Elephant
Mouse
Snail
```
**Response:**
```response theme={null}
Here's the classification:
* **Elephant:** Large
* **Mouse:** Small
* **Snail:** Small
```
#### Completion Input
**Prompt Example:**
```prompt theme={null}
Complete the following prompt: Some simple strategies for overcoming writer's block include...
```
**Response:**
```response theme={null}
Some simple strategies for overcoming writer's block include:
* **Freewriting:** Write non-stop for a set amount of time (5-10 minutes) without worrying about grammar, spelling, or even making sense. Just keep your pen moving and let your thoughts flow freely. This can help you loosen up and access ideas you might not have considered otherwise.
* **Brainstorming:** Make a list of everything that comes to mind related to your topic, no matter how silly or unrelated it seems. You can also use mind mapping or other visual brainstorming techniques.
* **Changing your environment:** Sometimes a change of scenery can be all you need to get the creative juices flowing. Go for a walk, work in a coffee shop, or even just move to a different room in your house.
* **Reading:** Immerse yourself in the work of other writers, especially those who inspire you. Pay attention to their style, their structure, and their use of language.
* **Taking a break:** Sometimes the best way to overcome writer's block is to simply step away from the project for a while. Go for a walk, watch a movie, listen to music, or do something else that you enjoy. You might be surprised at how much clearer your head feels when you return to your writing.
* **Focusing on a single element:** Instead of tackling the entire project, focus on a single sentence, paragraph, or scene. Once you get something down on paper, it can be easier to build from there.
* **Starting with the ending:** If you're stuck on the beginning, try writing the ending first. This can help you to clarify your overall vision for the project and provide a framework for your ideas.
* **Talking it out:** Share your ideas and challenges with a friend, a writing group, or a writing coach. Discussing your project with someone else can help you to see it from a new perspective and find solutions to your writing problems.
Remember, writer's block is a common experience, and it's not a sign that you're a bad writer. Just be patient with yourself, try some of these strategies, and keep writing!
```
### Context
The **context** section gives the model helpful background or specific instructions that shape how it responds. You can use context to:
* Direct the model’s behavior.
* Supply relevant information it should refer to.
* Limit the model’s answers to specific data or rules.
Use context when you want to guide the model more clearly or restrict it to the facts you've provided.
**Prompt Example:**
```prompt theme={null}
Marbles:
Color: red
Number: 12
Color: blue
Number: 28
Color: yellow
Number: 15
Color: green
Number: 17
How many green marbles are there?
```
**Response:**
```response theme={null}
There are 17 green marbles.
```
In this case, the context is the list of marble colors and quantities, and the model uses that information to answer accurately.
### Examples
**Examples** are pairs of inputs and expected outputs you include in your prompt to show the model what kind of response you're looking for. This technique is especially useful when you want the output to follow a certain format or logic.
**Prompt Example:**
```prompt theme={null}
Classify the following.
Options:
- red wine
- white wine
Text: Chardonnay
The answer is: white wine
Text: Cabernet
The answer is: red wine
Text: Moscato
The answer is: white wine
Text: Riesling
The answer is:
```
**Response:**
```response theme={null}
The answer is: white wine
```
By including a few classification examples before the final input, you're teaching the model how to respond.
***
### References
* Some sections in this guide are paraphrased or inspired by Google’s [Gemini API documentation on prompting](https://ai.google.dev/gemini-api/docs/prompting-intro#prompt-content-types).
# Clear Instructions
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/clear-instructions
Learn how to design prompts with clear tasks, constraints, and formatting guidance for more accurate AI responses.
## Explain the Task Clearly
To effectively guide the model's behavior, it's important to provide instructions that are both direct and specific. Whether you're asking for a simple summary or building a more complex user experience, the clearer your guidance, the better the model can deliver the result you want.
**Prompt Example:**
```prompt theme={null}
Create a summary of the following content:
Text: A quantum computer operates using principles of quantum physics, enabling it to process data at speeds far beyond that of conventional machines. At microscopic levels, particles exhibit both wave and particle characteristics, and quantum systems are engineered to take advantage of this. Their inner workings defy traditional physics. When developed at scale, quantum machines could have vast applications—from decoding modern encryption to boosting weather prediction accuracy. However, today’s quantum tech remains mostly experimental and not yet suitable for real-world deployment.
```
**Response:**
```response theme={null}
Quantum computers tap into the dual nature of matter—both wave and particle—at microscopic levels to execute tasks far faster than conventional machines. By utilizing quantum effects that classical physics can’t explain, they gain a significant speed advantage. Although they show strong potential in areas like cybersecurity and meteorology, these systems are still in early development and not ready for mainstream deployment.
```
## Specify any constraints
Define any rules or limitations the model should follow while interpreting the prompt or creating a response. For instance, if you want a more concise version of the original text, make that clear by stating a word or sentence limit. In the earlier example, the summary was shorter than the source content—but you can explicitly direct the model to keep it brief by setting a specific length constraint.
**Prompt Example:**
```prompt theme={null}
Summarize this text in two sentences:
Text: A quantum computer operates using principles of quantum physics, enabling it to process data at speeds far beyond that of conventional machines. At microscopic levels, particles exhibit both wave and particle characteristics, and quantum systems are engineered to take advantage of this. Their inner workings defy traditional physics. When developed at scale, quantum machines could have vast applications—from decoding modern encryption to boosting weather prediction accuracy. However, today’s quantum tech remains mostly experimental and not yet suitable for real-world deployment.
```
**Response:**
```response theme={null}
Quantum machines utilize strange particle behaviors at the quantum level to achieve processing speeds that leave regular computers behind. Despite their huge promise in areas like cybersecurity and climate modeling, they’re still in early development and not yet viable for everyday tasks.
```
## Set the Desired Output Format
You can guide how the model should present its answer by specifying a format. Whether you want the response in bullet points, a short paragraph, a table, a list of keywords, or even as an elevator pitch—you can simply include that in your instructions. For instance, you might ask the model to respond in a more casual, conversational tone by stating that directly in your prompt.
**System message:**
```message theme={null}
Respond to all prompts using clear and structured bullet points, unless the user explicitly requests a different format. Each bullet should be concise yet informative, helping break down complex information into easily digestible parts.
```
**Example Prompt:**
```prompt theme={null}
What’s an effective way to run a DVD-selling business in 2025?
```
**Response:**
```response theme={null}
Running a DVD-focused business in 2025 is challenging due to the rise of streaming and digital formats. However, it's not entirely out of reach. Here's a strategic way to approach it:
- Niche Targeting: Focus on collectors, retro lovers, or areas with low internet access.
- Product Differentiation: Sell special edition DVDs, box sets, or director’s cuts not available digitally.
- Experience-Based Selling: Open pop-up nostalgia stores or host DVD movie nights to create a community experience.
- Bundle Offers: Combine DVDs with themed merchandise (posters, T-shirts) to increase perceived value.
```
Another example about setting desired format.
**System message:**
```message theme={null}
Respond using a numbered list format by default. Each number should represent a main idea or step. Keep the explanations concise and informative. Only switch to another format if the user requests it.
```
**Example Prompt:**
```prompt theme={null}
How can a student better manage their time while studying for exams?
```
**Response:**
```response theme={null}
1. Set clear goals
Define what you need to study each day based on your exam schedule.
2. Prioritize tasks
Focus first on subjects or topics you're weakest in or those that carry the most weight in the exam.
3. Use time blocks
Study in focused intervals (e.g., Pomodoro method: 25 minutes study, 5 minutes break).
4. Eliminate distractions
Turn off notifications, put your phone away, and create a quiet study environment.
5. Review consistently
Instead of cramming, revisit key topics regularly to strengthen memory.
6. Track your progress
Use a checklist or planner to mark completed topics and stay motivated.
```
**📝 Summary:**
* Clearly tell the model what task you want it to perform.
* Make sure your instructions are detailed and easy to understand.
* If there are any limits or formatting preferences for the response, be sure to include them.
# Context Strategies
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/context-strategies
Add background details, references, or brand-specific data to help the model generate more targeted responses.
Don’t assume the model knows everything it needs. Instead, add helpful background details or instructions directly in the prompt. Supplying context gives the model a better understanding of your goals, limitations, and the specific problem it needs to solve.
For example, the following prompt asks the model to provide router troubleshooting advice:
**Example Prompt:**
```prompt theme={null}
My printer won’t print even though it’s connected. The display says “Paper Jam”, but I checked and there’s no paper stuck inside. What should I do?
```
**Response:**
```response theme={null}
If your printer shows a “Paper Jam” error but there’s no visible jam, it could be due to a false sensor trigger or small bits of paper stuck inside. Here are steps you can take to troubleshoot:
1. Power off and unplug the printer.
2. Open all accessible panels and trays.
- Remove any paper and carefully check inside using a flashlight.
- Look for tiny scraps of paper or dust near rollers or sensors.
3. Clean the rollers.
- Use a lint-free cloth slightly dampened with water to clean the feed rollers.
4. Let it rest for a few minutes, then restart.
- Plug the printer back in and turn it on.
5. Try a test print.
- If the issue persists, consult the printer’s manual or support site for advanced help.
```
The response provided sounds like general advice and doesn’t directly address this specific printer model or its error message behavior. To make the response more accurate and relevant, you could include details from the printer’s official troubleshooting guide—such as how it handles “Paper Jam” alerts or what might trigger them falsely. Providing that context helps the model give targeted, brand-specific instructions rather than just standard tips.
**Example Prompt:**
```prompt theme={null}
Answer the question using the text below. Respond with only the text provided.
Question: What should I do if my Epson printer shows an error with blinking red and orange lights?
Text:
Light Pattern: Blinking red and orange
What it means: A fatal printer error has occurred.
What to do:
- Turn off the printer.
- Unplug the power cord and wait for at least 60 seconds.
- Plug the printer back in and power it on.
- If the error persists, contact Epson support for service.
Light Pattern: Blinking green
What it means: The printer is receiving data or printing.
What to do: No action needed.
Light Pattern: Solid red
What it means: The printer is out of paper or has a paper jam.
What to do: Check for jammed paper and reload the paper tray properly.
```
**Response:**
```response theme={null}
Turn off the printer. Unplug the power cord and wait for at least 60 seconds. Plug the printer back in and power it on. If the error persists, contact Epson support for service.
```
**📝 Summary:**
* Add any relevant background or reference material directly into the prompt to guide the model’s response.
* Also, be sure to clearly explain how the model should apply that context when generating its answer.
# Fallback and Caution
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/fallback-caution
Handle fallback messages and understand the model’s limitations with factual content, logic, and math.
## Fallback Responses
A fallback response is what the model returns when the prompt or generated content activates a safety mechanism. A common fallback message might be: "I'm sorry, but I can't help with that."
If you encounter a fallback response, try raising the temperature setting to encourage a different outcome.
## Cautionary Notes
* Avoid using the model as a reliable source for factual information.
* Be cautious when using it for math or logic-related tasks, as the output may not always be accurate.
# Few-Shot Examples
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/few-shot-examples
Explore zero-shot vs few-shot prompting, and how to use examples to teach the model desired output patterns and styles.
You can guide the model by including examples in your prompt that demonstrate the ideal response. The model learns from these examples by recognizing patterns and structures, then applies that understanding to generate its own output. Prompts with a few examples are known as few-shot prompts, while those with none are called zero-shot prompts. Few-shot prompts are particularly useful for shaping the tone, format, scope, and overall structure of the model’s replies. Using clear, diverse examples helps the model stay focused and deliver more accurate results.
It’s generally best to include a few examples in your prompt. Without them, the model’s responses may be less precise. In fact, if your examples are well-chosen and illustrative, you may not even need additional instructions at all.
## Zero-shot vs few-shot prompts
The zero-shot prompt below instructs the model to select the most suitable explanation.
**Example Prompt:**
```prompt theme={null}
Select the most accurate explanation for the following question:
Question: What causes earthquakes?
Option 1: Earthquakes occur when tectonic plates shift suddenly along fault lines, releasing energy in the form of seismic waves.
Option 2: Earthquakes happen due to strong winds and storms shaking the ground.
Your Answer:
```
**Response:**
```response theme={null}
Option 1 is the most accurate explanation.
```
If your goal is to get brief responses from the model, you can add examples in your prompt that highlight concise answers.
In the next prompt, two sample explanations are given, showing a clear preference for shorter responses. As a result, the model is influenced by these examples and selects the more concise explanation (Explanation 2), unlike the earlier case where it chose the longer one (Explanation 1).
**Example Prompt:**
```prompt theme={null}
Below are some examples showing a question, explanation, and answer format:
Question: Why do leaves change color in the fall?
Explanation1: Leaves change color due to a combination of factors like temperature, light, and the reduction of chlorophyll production, which causes other pigments like carotenoids and anthocyanins to become more visible.
Explanation2: Because chlorophyll fades, other colors in the leaves show through.
Answer: Explanation2
Question: Why do tides occur?
Explanation1: Tides are caused primarily by the gravitational pull of the moon and the sun on Earth’s oceans.
Explanation2: The moon’s gravity pulls ocean water, causing tides.
Answer: Explanation2
Now, using the same format and logic, answer the following:
Question: Why do we see lightning before we hear thunder?
Explanation1: Lightning travels at the speed of light, which is much faster than the speed of sound. That’s why we see the flash before we hear the thunderclap, even though they happen at the same time.
Explanation2: Light moves faster than sound, so we see it first.
Answer:
```
**Response:**
```response theme={null}
Answer: Explanation2
```
## Determine the Right Number of Examples
To get the best outcome, try different numbers of examples in your prompt. Models like Gemini can usually recognize patterns with just a few examples, but you might need to test how many are needed to achieve the result you want. Be cautious though—adding too many can cause the model to mimic the examples too closely instead of generalizing properly.
**Use examples to show patterns instead of antipatterns:**
It’s more effective to include examples that demonstrate the kind of response you want, rather than showing what not to do. Highlighting good examples helps the model follow the intended pattern more accurately.
**🚫 Negative pattern:**
```prompt theme={null}
Don’t start formal emails too casually:
Hey there!
Just checking in on that report.
Let me know, okay?
```
**✅ Positive pattern:**
```prompt theme={null}
Begin formal emails with a professional greeting:
Dear Team,
I’m writing to follow up on the report.
Please let me know your updates by Friday.
```
## Keep Formatting Consistent Across Examples
To ensure the model produces the desired output, your few-shot examples should all follow the same structure and formatting. Since the purpose of few-shot prompting is to guide the model's response format, it’s important to maintain uniformity in elements like XML tags, line breaks, whitespace, and separators between examples. Inconsistencies might lead to unpredictable results.
**📝 Summary:**
Use prompt-response examples to teach the model how to respond.
Focus on showing clear examples of what you want, not what to avoid.
Test different numbers of examples to see what works best—too few may not help, while too many could limit the model’s flexibility.
Always keep the formatting consistent across all examples to guide the model accurately.
# Modular Prompts
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/modular-prompts
Simplify complex tasks using modular instructions, chaining prompts step-by-step, or aggregating outputs.
When dealing with complex tasks, simplify them by dividing the prompt into smaller, more focused parts. This makes it easier for the model to follow and process your instructions accurately.
## Simplify Instruction
Rather than packing multiple instructions into a single prompt, create separate prompts for each one. Choose the appropriate prompt to run based on the user’s input or situation.
## Use Chained Prompts
For tasks that follow a specific order of steps, use one prompt per step and pass the output from one step into the next. This sequence allows the model to build on its responses and reach a final result.
## Aggregate Outputs
When you need to perform several tasks on different parts of your data, handle them separately and then combine the outputs. For instance, analyze the beginning of a document in one way, process the rest differently, and then merge the results for a complete answer.
**📝 Summary:**
* Break down complex prompts into individual instructions and select the right one based on the user’s needs.
* For multi-step workflows, treat each step as its own prompt and pass outputs forward in sequence.
* For parallel tasks, process each part independently and combine the results to produce a complete answer.
# Prompt Design Strategies
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/overview
Master the art of prompt crafting with strategic techniques to guide AI responses, structure output, and enhance clarity.
Prompt design is more than just asking a question — it’s about structuring information, setting the right context, and guiding AI to respond exactly how you want.
In this section, you'll explore tactical strategies for writing effective prompts, such as:
* Giving clear and specific instructions
* Including few-shot examples
* Providing relevant context
* Using smart prefixes and formats
* Structuring complex prompts into manageable parts
* Tuning model behavior through parameters
* Handling fallback cases and avoiding common pitfalls
Each strategy is broken down into its own focused lesson. Ready to level up your prompting game?
👇 Start exploring the strategies below:
* [Give Clear and Specific Instructions](./clear-instructions)
* [Use Few-Shot Examples Effectively](./few-shot-examples)
* [Include Relevant Context](./context-strategies)
* [Guide with Prefixes and Partial Patterns](./prefixes-patterns)
* [Break Down Prompts into Manageable Parts](./modular-prompts)
* [Tune Parameters and Iterate on Prompts](./parameters-iteration)
* [Fallback Responses and Cautionary Tips](./fallback-caution)
# Parameters and Iteration
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/parameters-iteration
Understand temperature, top-p, and max token settings. Learn how to refine your prompt through experimentation.
## Experiment with different parameter values
Each time you send a prompt to a model, it uses certain parameters that influence how the response is generated. Adjusting these values can help you fine-tune the output to better fit your task. Different models may offer different parameter options, but here are the most commonly used ones:
### Max Output Tokens
This limits the maximum length of the response. One token is roughly 4 characters or ¾ of a word.
* Use a lower value for short answers
* Use a higher value for longer, more detailed responses
### Temperature
Controls how creative or random the response is.
* **Lower values (e.g., 0.2)** make outputs more focused and consistent
* **Higher values (e.g., 0.7+)** lead to more varied, creative responses
* A temperature of **0** always picks the most likely next word (deterministic)
Start with **0.2** and increase if the output feels generic, repetitive, or too short.
### Top-K
Limits the model’s choices for the next word to the top K most likely tokens.
* **Top-K = 1** gives the most predictable result
* **Top-K = 3** or more allows more randomness and variation Top-K is often used with temperature and top-P together.
### Top-P
Instead of picking from a fixed number of top options like Top-K, Top-P chooses from the smallest set of words whose total probability exceeds the specified threshold.
* **Lower top-P (e.g., 0.5)** = safer, more focused responses
* **Higher top-P (e.g., 0.95)** = more variety and creativity
## Prompt Iteration Strategies
Designing effective prompts is a trial-and-error process, and it often takes several attempts to consistently get the results you want. This section offers tips on how to improve your prompts through iteration.
### Use different phrasing
Designing effective prompts is a trial-and-error process, and it often takes several attempts to consistently get the results you want. This section offers tips on how to improve your prompts through iteration.
```prompt theme={null}
Version 1:
How can I improve my sleep quality?
Version 2:
Give me tips for getting better sleep.
Version 3:
What are some effective ways to sleep better?
```
### Switch to an analogous task
If the model isn’t responding well to your original instructions, consider reframing the task as a similar one that leads to the same outcome.
For example, instead of directly asking the model to label a book, you might instruct it to sort the book into one of several predefined categories.
**Example Prompt:**
```prompt theme={null}
Which category does `1984` belong to:
- romance
- dystopian
- fantasy
- biography
```
**Response:**
```response theme={null}
1984 belongs to the category of dystopian.
Here’s why:
- Dystopian: 1984 is set in a totalitarian future society where the government controls every aspect of life, including thoughts and language. It explores themes of surveillance, propaganda, and loss of freedom—hallmarks of dystopian fiction.
```
The answer is accurate, but the model went beyond the list of provided choices and responded in full sentences. If you want the model to stick strictly to the given options and respond with just one of them, you can reword the prompt as a multiple-choice question and instruct the model to select only one answer.
**Example Prompt:**
```prompt theme={null}
Multiple choice problem: Which of the following options describes the book `1984`?
Options:
- romance
- dystopian
- fantasy
- biography
```
**Response:**
```response theme={null}
The correct answer is dystopian.
```
### Change the order of prompt content
Rearranging the elements in your prompt can influence how the model responds. Try altering the sequence of the content to see if it changes or improves the outcome.
```prompt theme={null}
Format Option 1:
- Sample responses
- Background information
- User input
Format Option 2:
- User input
- Sample responses
- Background information
Format Option 3:
- Sample responses
- User input
- Background details
```
# Prefixes and Patterns
Source: https://docs.tuturuuu.com/learn/guides/prompt-engineering/prompt-design-strategies/prefixes-patterns
Use labeled prefixes and incomplete structures to guide the model’s response format and intent.
## Add prefixes
Prefixes are short phrases or labels you can attach to parts of a prompt to guide the model’s understanding and output. They serve different roles based on where they’re placed:
* **Input prefix**: This helps the model interpret sections of the input by labeling them. For instance, using “English:” and “French:” can indicate that different languages are being used.
* **Output prefix**: Even though the model generates the output, you can lead with a prefix like “JSON:” to show that the response should follow a specific format.
* **Example prefix**: In few-shot prompts, labeling examples with prefixes makes it easier for the model to recognize patterns and generate consistent, easy-to-understand results.
For example, in the prompt:
Text: signals the input section, and
The answer is: signals the beginning of the expected output.
**Example Prompt:**
```prompt theme={null}
Classify the text as one of the following categories:
- fruit
- vegetable
Text: Apple
The answer is: fruit
Text: Carrot
The answer is: vegetable
Text: Banana
The answer is: fruit
Text: Broccoli
The answer is:
```
**Response:**
```response theme={null}
The answer is: vegetable
```
## Let the model complete partial input
Language models function similarly to powerful autocomplete systems. When you supply the beginning of a sentence, structure, or task, the model can predict and continue the rest based on that starting point. If your prompt includes examples or additional context, the model can use that information to generate a more relevant and accurate continuation.
**Example Prompt:**
```prompt theme={null}
Convert the following temperature report into a JSON object with fields for city, temperature (in Celsius), and condition.
Report: It’s 18 degrees and cloudy in Hanoi.
```
**Response:**
```response theme={null}
{
"city": "Hanoi",
"temperature": 18,
"condition": "cloudy"
}
```
Although the model followed the instructions, describing the task in plain language can sometimes be tricky and may lead to varied interpretations. For instance, if someone says “Can I get two bananas and an orange?”, the model should generate a JSON object that includes only those items—banana and orange—and exclude fields like apple or grapes that weren’t mentioned. To make the task clearer and guide the output structure, it’s more effective to include a few-shot example with a response prefix like Output: so the model can complete the response by following the pattern shown.
**Example Prompt:**
```prompt theme={null}
Valid fields are apple, banana, orange, and grapes.
Order: I'd like an apple and some grapes.
Output:
{
"apple": 1,
"grapes": 1
}
Order: Can I get two bananas and an orange?
Output:
```
**Response:**
```response theme={null}
{
"banana": 2,
"orange": 1
}
```
Notice how "grapes" was excluded from the output because it wasn't a part of the order.
## **Prompt the model to format its response**
You can also guide the model to structure its output by using a completion-based approach. For example, the prompt below asks the model to generate an outline for an essay.
**Example Prompt:**
```prompt theme={null}
Create an outline for an essay about volcanoes.
```
**Response:**
```response theme={null}
Volcanoes: Earth’s Fiery Power
I. Introduction
A. What volcanoes are and how they form
B. Thesis: Volcanoes shape the Earth through both destruction and creation
II. Types of Volcanoes
A. Stratovolcanoes, shield, and cinder cones
B. Key features and eruption styles
III. Effects on People and Nature
A. Hazards: lava, ash, tsunamis
B. Benefits: fertile soil, geothermal energy
IV. Monitoring and Safety
A. How eruptions are predicted
B. Importance of early warning systems
V. Conclusion
A. Volcanoes’ role in shaping Earth and human history
```
The prompt didn’t include instructions about how the outline should be structured, so the model selected a format on its own. If you want the model to follow a specific structure, you can provide the beginning of the outline in your desired format. The model will then continue generating the rest to match the pattern you've started.
**Example Prompt:**
```prompt theme={null}
Create an outline for an essay about volcanoes.
I. Introduction
*
```
**Response:**
```response theme={null}
Volcanoes: Earth’s Fiery Sculptors
I. Introduction
- Hook: Start with a dramatic description of a volcanic eruption, capturing the raw power and impact on the landscape.
- Background: Introduce what volcanoes are and how they form as openings in the Earth's crust.
- Thesis: Present the central idea of the essay. Examples:
- Volcanoes are both destructive and creative forces that shape ecosystems and human life.
- Understanding volcanic activity is key to reducing disaster risks and appreciating Earth’s dynamic nature.
- Despite their dangers, volcanoes offer valuable scientific, ecological, and cultural insights.
```
**📝 Summary:**
* When you provide the model with an incomplete input, it can finish it by recognizing patterns from any examples or context you've included.
* In many cases, letting the model complete a partially written prompt is simpler and more effective than explaining the task in full.
* By starting the answer yourself, you can guide the model to generate responses in the specific structure or style you want.
# Learn & Experiment
Source: https://docs.tuturuuu.com/learn/overview
Examples, experiments, and prompt engineering techniques for Tuturuuu.
Level up your Tuturuuu expertise through curated examples, experiments, and prompt engineering playbooks.
## Learning Paths
* **Examples** – Browse practical snippets and starter code in the [Examples Library](/learn/examples/overview).
* **Experiments** – Explore in-progress ideas in [Experiments](/learn/experiments/form-builder) to understand what is coming next.
* **Prompt Engineering** – Master AI collaboration with the [Prompt Engineering Guides](/learn/guides/prompt-engineering/introduction).
## Experimentation Flow
```mermaid theme={null}
graph LR
A[Idea] -->|Rapid build| B[Prototype]
B -->|Share| C[Feedback]
C -->|Promote| D[Product]
D -->|Document| A
style A fill:#4f46e5,stroke:#3730a3,color:#fff
style D fill:#4f46e5,stroke:#3730a3,color:#fff
```
Stay curious—share findings, improve docs, and help us expand what the Tuturuuu ecosystem can do. For detailed platform architecture, visit the [Platform section](/platform/overview). Need local tooling help? Return to [Build](/build/overview).
# Part 1: Web Fundamentals
Source: https://docs.tuturuuu.com/learn/workshops/nextjs-workshop/01-web-fundamentals
HTML, CSS, and JavaScript - The building blocks of the web
⏱️ **Duration**: 15 minutes | This section covers the absolute basics of web development.
## What is a Website?
Every website you visit—from Netflix to TikTok to your university portal—is built with just **three technologies**:
```mermaid theme={null}
graph LR
A["🦴 HTML"] --> D["🌐 Website"]
B["🎨 CSS"] --> D
C["🧠 JavaScript"] --> D
style A fill:#e34c26,color:#fff
style B fill:#264de4,color:#fff
style C fill:#f7df1e,color:#000
style D fill:#22c55e,color:#fff
```
**Think of a website like a human body:**
| Technology | What it does | Human analogy |
| ----------------- | ------------------- | ------------------- |
| 🦴 **HTML** | Structure & content | Skeleton + organs |
| 🎨 **CSS** | Appearance & style | Skin, hair, clothes |
| 🧠 **JavaScript** | Behavior & logic | Brain + muscles |
**🤯 Did you know?** The first website ever created is still online! Visit [info.cern.ch](http://info.cern.ch/hypertext/WWW/TheProject.html)—it was made in 1991 by Tim Berners-Lee. No CSS, no JavaScript, just pure HTML!
***
## HTML: The Structure
HTML (HyperText Markup Language) defines the content and structure of web pages.
### Your First HTML Page
```html theme={null}
My First Page
Hello, World!
Welcome to web development!
```
**Beginner Tip**: Every HTML element has an opening tag `` and a closing tag ``. The content goes between them.
### Common HTML Elements
| Element | Purpose | Example |
| ---------------- | --------- | ----------------------------------------- |
| `