Skip to main content

​
Overview

Remote devboxes let internal root workspace members offload expensive local work to self-hosted runner machines. A runner connects outbound to tuturuuu.com, receives queued work, executes it on its host, streams logs, stores artifacts, and releases the lease automatically for one-off work. The current ttr box agent executes claimed commands as child processes on its host. Its command policy and environment filtering are operational controls, not an isolation boundary for untrusted learner code. Do not route Learn coding submissions to ordinary Devbox runs. Judge needs a separate sandbox executor and readiness checks before its capability can be enabled. V1 is intentionally self-hosted. It does not provision cloud machines. Any runner host should have Node.js, Bun, Docker, and Git available.

​
Quick Start

ttr box setup is the default bootstrap for a runner host. It first checks whether the current directory is already a Tuturuuu platform checkout. If it is, setup reuses that checkout. Otherwise it can clone or reuse https://github.com/tutur3u/platform.git, install dependencies with bun install --frozen-lockfile, start local Supabase with bun sb:start, verify supabase status -o json, and write local Supabase connection values into ignored apps/*/.env.local files. Secret values are redacted from human and JSON output. Pass --dir <path> for a specific checkout. If the target exists but is not a valid Tuturuuu checkout, interactive setup asks whether it should clone into a nested tuturuuu directory. In non-interactive shells, pass --clone-into <path> explicitly:
Use --yes only when the host should install detected missing prerequisites automatically. Runner registration and service installation are separate choices. To register the machine and install a boot-starting runner service in one non-interactive command after login:
For a platform runner that does not need a Tuturuuu checkout or local Supabase, use ttr box connect --service --runner-name "$(hostname)-devbox" instead. This registers the machine, writes its runner token to a mode-0600 file, and installs the same boot-starting service. The CLI prints a direct Infrastructure dashboard link for that runner. The dashboard shows the heartbeat’s CLI version and machine identity so an administrator can verify the connection before enabling workloads. The setup command stores the runner token in a restricted local env file and installs a system-level runner service on Linux systemd or macOS launchd. The service starts after reboot and restarts automatically if the runner exits. Runner heartbeats report a small observability snapshot, including ttr, Bun, Node, Docker, Git, OS, CPU, RAM, load average, and uptime. Root workspace admins can inspect those fields from Infrastructure > Devboxes. Devbox agent poll and heartbeat traffic is blocked by default at the platform API boundary. Set TUTURUUU_DEVBOX_AGENT_API_ENABLED=true on the web app before starting or repairing runners. When the flag is absent, agent poll and heartbeat requests return 403 before runner tokens are authenticated or runner state is updated.

​
Cloudflare control plane

ttr box agent start uses the existing web API until TUTURUUU_DEVBOX_CONTROL_URL is set on the runner host. Point this variable at the deployed tuturuuu-devbox-control Worker to send heartbeats, job claims, and an authenticated wake connection through Cloudflare. The agent still sends job events and completion to the web API. It polls the Worker every 30 seconds when idle, even with a wake connection, so a missed signal cannot strand work. The Worker checks the runner token, token revocation, runner status, and root workspace membership before each heartbeat or claim. Disabling heartbeats on a runner continues to reject heartbeat requests. Worker traffic requires TLS. The production workflow installs Worker secrets from the repository’s PRODUCTION_SUPABASE_URL, PRODUCTION_SUPABASE_SECRET_KEY, and DEVBOX_CONTROL_INTERNAL_TOKEN secrets. It uses COLAB_CLOUDFLARE_API_TOKEN for Cloudflare deployment. Set TUTURUUU_DEVBOX_CONTROL_URL and TUTURUUU_DEVBOX_CONTROL_TOKEN on web and Learn to wake an assigned runner immediately after enqueuing work. Use the same internal token as the Worker secret; never put it in the runner’s env file. Missing or failed wake calls do not fail enqueue, and the fallback poll catches the job. Verify /health, then a real runner heartbeat, claim, completion, revocation rejection, and Learn submission before moving additional runners. The old web API remains available until all runners are migrated. Without a Worker URL, idle claims back off from 5 to 10, 20, and then 30 seconds. A claimed job or completion resets that delay; heartbeats remain independent at 20 seconds to preserve online detection and leases. Worker wake messages still trigger immediate claims, with a 30-second reconciliation poll for missed wakes. If a host completed ttr box setup but does not appear in Infrastructure > Devboxes, confirm it was registered as a runner. Plain setup prepares the checkout, dependencies, local Supabase, and ignored env files only; it does not create a private.devbox_runners row. Run setup with --agent --service, or register and start the agent manually, then confirm ttr whoami on that host is pointing at the same production origin and root workspace user. If the host appears as registered but shows no heartbeat, inspect the installed service on that host:
The runner service reads its token from the local devbox-runner.env file. If the journal shows a missing runner token or stale wrapper, upgrade the CLI and repair the service. Repair reuses the existing token file; it does not register a new runner token or create another runner row.
Use dry-run first when you want to confirm the checkout, token file, service manager, wrapper path, and service path without writing files or calling sudo:
Upgrade a runner’s global CLI through the same brokered queue:
The upgrade command queues bun i -g tuturuuu for the selected runner, waits for completion, and returns the remote command logs and exit code. After a successful CLI update, the agent exits; a runner installed with --service restarts under systemd or launchd and reports its new CLI version on the next heartbeat. A manually started agent must be restarted by its operator. The run completion and the subsequent heartbeat are separate verification steps.

​
Platform-owned fleet controls

Root members can use the existing Infrastructure authorization boundary through client.devboxes.fleet.snapshot() or the CLI:
Fleet maintenance defaults to a dry run. --apply performs sequential, allowlisted maintenance, rechecks fresh heartbeats and unfinished work before each operation, stops on failure, and waits for a subsequent healthy heartbeat. Upgrades additionally verify the advertised CLI version against the registry release. Offline, revoked, busy, or truncated snapshots are skipped. Snapshot limits mean this is a conservative operator convenience, not an atomic host-wide reservation; coordinate a maintenance window when service users share an install. Capabilities, feature switches, and resource limits are available as structured JSON for tooling extensions. These commands do not enable arbitrary shell access or change runner workload permissions. Custom SDK deployments use their own origin rather than forwarding credentials to hosted Infrastructure.

​
Protecting hosts with existing critical files

Use --execution-mode judge-only with box connect, box setup, or box repair. Supplying Judge images during registration defaults to Judge-only; general development runners retain trusted execution. Manual Judge-only setup restricts accepted jobs but does not receive protected-service host isolation; use the protected Linux service setup for hosts with critical files. --control-url persists the HTTPS Cloudflare Worker URL during registration. Protected service setup requires Linux systemd, a pre-existing non-login service user with its private home under /var/lib, a private service-owned token directory, and checkout and CLI outside personal homes. The dedicated root-owned Unix Docker socket must be mode 0660 and restricted to that user’s group; the user must not belong to the general docker group. Preflight verifies the daemon defaults to runsc with cgroup limits before registering a credential. It does not create OS users, move host files, or reconfigure a shared Docker daemon automatically.
Run protected setup and repair as root from the prepared runner directory, using an authenticated operator CLI context and a CLI installed outside personal homes. The installer deliberately fails before registration when invoked by a non-root operator: it does not broaden private directory permissions or implicitly copy operator credentials into the service account. Prepare the token directory with mode 0700; group and other access are rejected. Protected systemd units hide personal homes, make the host filesystem read-only except runner directories, remove capabilities, restrict devices, namespaces and kernel access, and set a private umask. Repair preserves an existing Judge-only policy and refuses to weaken it; --dry-run performs the same preflight before service writes. Setup persists the dedicated service user for subsequent repair. Older private token files need --service-user once; repair then retains it without rotating the runner token. The wrapper prefers the service user’s global ttr executable so upgrades affect the running service. Before starting a protected agent, it requires box agent policy to succeed on that executable; an older CLI lacking policy support fails closed rather than launching. Launchd lacks these Linux protections; use a dedicated Linux VM for protected Judge setup on macOS. Local policy enforcement requires the CLI release containing this change. Judge-only servers must keep Run, Build, Preview, and Tunnel disabled in Infrastructure. Set TUTURUUU_DEVBOX_EXECUTION_MODE=judge-only in the private runner environment after installing the release containing this local policy. It rejects host commands, environment files, and environment overrides on maintenance jobs even if a remote workload switch is accidentally enabled. Unknown modes fail closed. The exact CLI-update and supervised-restart commands remain root-member-authorized maintenance operations. Judge submissions use digest-pinned images, gVisor runsc, an unprivileged container user, no network, a read-only root filesystem, no host bind mounts, dropped capabilities, no privilege escalation, and CPU/memory/PID/time limits. Never expose the Docker socket to submissions. The runner and dedicated Docker daemon remain trusted infrastructure: access to a rootful Docker socket is a privileged control plane, even when its default runtime is gVisor. Keep the service user out of the general Docker group, use a dedicated Unix socket, hide personal home directories, and do not use this host for arbitrary trusted devbox commands. Stronger isolation for additional workloads requires a separate VM or host; a shell-command denylist is not a security boundary.

​
Operator-owned automatic upgrades

For a fleet whose service users do not have CLI user sessions, schedule upgrades from an authenticated operator machine with an explicit runner allowlist:
The installer supports macOS launchd, checks at login and every 24 hours while the operator machine is running, and stores its portable script, lock helper, and private logs under ~/.tuturuuu/devbox-upgrades. It uses the existing CLI session and never copies credentials into its plist. launchctl print gui/$(id -u)/com.tuturuuu.devbox-cli-upgrades shows scheduler status. A full snapshot, an active lease, stale heartbeat, or unfinished job prevents upgrade; an unsuccessful run or version heartbeat stops the remaining operations. Keep runner self-upgrades disabled when this scheduler owns maintenance. Ensure the service’s executable points at the same global installation that bun i -g tuturuuu updates for its OS user. A successful install into a different path does not update a service launching an older private copy. After changing a systemd path, drain work, reload the unit, restart sequentially, and verify each runner’s heartbeat and CLI version.

​
Automatic CLI upgrades

New or repaired managed services default TUTURUUU_DEVBOX_AUTO_UPGRADE=false. Enable TUTURUUU_DEVBOX_AUTO_UPGRADE=true in the private runner environment only after selecting an upgrade owner and maintenance policy. Existing services need one repair after the release containing this capability is installed:
An idle agent checks the official npm registry at most once per day, installs only a newer stable tuturuuu release pinned to its exact version, verifies --version, and exits for its service manager to restart. It pauses new claims during installation and keeps heartbeating. A private host lock serializes installs from runners sharing an OS user home directory, and persisted check state prevents restart retry loops. Cached install state is scoped to the verified CLI command, and a failed cached-version verification resumes its registry check. Other agents using that shared install restart after verifying the installed version. Lock contention schedules another attempt after 30 seconds, at the next idle claim, rather than delaying a stale process for a full day. Registry checks remain limited by the shared daily state. Each agent drains its own jobs; for hosts with different OS service users or unrelated CLI consumers, coordinate a host maintenance window or opt out instead of assuming this lock covers them. Installer subprocesses receive host path/runtime variables, never the runner token. An unverified install attempts to restore the previous release. A failed rollback stops the agent with a repair error; inspect service logs and repair the host installation. Locks record process ownership and identity; dead owners are reclaimed under a separate reclamation guard, while live installers are never evicted by age. An unknown legacy lock requires operator inspection. Reclaim guards also record process ownership; an interrupted guard reports a repair-required error rather than being deleted automatically or retrying silently. Never remove a lock held by a running installer. Set TUTURUUU_DEVBOX_AUTO_UPGRADE=false in devbox-runner.env to opt out. Manually started and source-checkout agents do not upgrade automatically unless explicitly configured, and source .ts entry points never mutate the global CLI. Infrastructure’s Update CLI remains the bootstrap for existing runners; confirm both its successful run and the subsequent version heartbeat. Source changes take effect only after the normal tested CLI release is published. Regression coverage: devbox-auto-upgrade.test.ts covers stable version validation, pinned installation, shared-host locking, restart throttling, verification and rollback. devbox-agent.test.ts covers claim backoff, independent heartbeats, concurrency, draining and supervised restart; devbox-setup.test.ts verifies the service wrapper preserves the explicit opt-in. Infrastructure > Devboxes lets root administrators enable or disable command, build, preview, and tunnel workloads per runner. New runners default to those four workload types enabled. Disabling a type stops new matching jobs from being claimed on that runner; it does not interrupt a running job or act as a security permission for root workspace members who can submit generic commands. The dashboard keeps switches unavailable until the 20260928130000_devbox_runner_features.sql migration is applied. Its Update CLI action queues an update for an online runner, then the agent restarts as above. The Restart agent action and ttr box restart queue a typed maintenance job that exits the agent after successful acknowledgement; the installed service manager starts it again. These actions restart the agent process, not the operating system. Judge defaults off. Enabling it requires a recent heartbeat reporting a verified sandbox runtime and at least one pinned, cached language image. Existing Devbox host commands remain separate from Judge submissions. The same dashboard stores a runner’s Judge capacity budget. Defaults reserve at most 50% of host CPU and 25% of host memory, with one sandbox and one instance at a time. Each sandbox defaults to 1 GB, 30 seconds, and 64 processes. Administrators can adjust these limits per machine. These settings are enforced on each Judge container by Docker cgroups, with a conservative free-memory check before each case. The platform claim function caps concurrent Judge runs on a runner at the lower of its sandbox and instance maxima. Each container receives a share of the host CPU and memory budget divided by that ceiling. A managed agent can dispatch independent Judge jobs concurrently within the configured process-wide sandbox budget. These settings do not limit the current trusted host commands, builds, previews, or tunnels.

​
Judge setup

The Judge executor requires a Docker engine with the runsc gVisor runtime registered, working CPU/memory/process cgroup limits, and locally cached OCI images pinned to SHA-256 digests. The CLI fails readiness when these are absent or the Docker engine is rootless with resource enforcement not verified. gVisor currently requires a Linux host or Linux VM. Do not enable Judge on a machine that has only the default Docker runtime. Configure the languages you want with a JSON map of language names to pinned image references. Supported adapters are Python, JavaScript, TypeScript, C, C++, Java, Rust, Go, Ruby, and PHP. The chosen image must contain that language’s runtime or compiler plus a POSIX shell and base64 for compiled languages. Pull each digest on the host before connecting. Then run:
Replace the placeholder digests with full verified 64-character values. The connect command saves the map with the runner token in the restricted local environment file used by the service. Its dashboard link opens the exact runner row in Infrastructure. After the heartbeat reports Judge readiness, review the host budget and enable Judge. Workspaces consume the platform Judge capacity through Learn; they do not register or control the physical runner. The executor runs one case at a time without network access or host mounts, with a read-only image, dropped capabilities, no new privileges, a non-root user, bounded temporary storage, and per-case CPU, memory, process, time, and output limits. Compiled languages compile inside the same sandbox. Source and hidden cases are stored in the private platform schema; learner results expose only the aggregate pass count and visible case outcomes. A failed or missing sandbox readiness check leaves Judge unavailable.

​
Recover a Lima Judge VM after disk errors

On macOS, a full host disk can leave a Lima guest’s ext4 root filesystem read-only. If writes fail with Read-only file system or the VM stops before SSH is ready, restore host free space first. Keep the VM disk and runner token files. Check df -h on the host and remove only known rebuildable caches or clean, merged worktrees. If the guest remains read-only, stop the VM before checking its disk. On an APFS host, make a copy-on-write backup, then attach the raw disk without mounting it:
Find the ext4 root partition in the diskutil output (for example, /dev/disk4s1). Install the bottled e2fsprogs package if needed. Run the read-only check first, then repair that partition with the block device path, not the raw /dev/rdisk* path:
Replace the example disk identifiers with those from your attach output. The final check must report a clean filesystem. Verify guest writes and the runner service before deleting the backup. If the host fills again, repair the host space problem before restarting the VM; repeated restarts can corrupt more guest data. See Lima’s offline fsck guidance for the underlying disk-image procedure. Remove a runner from the cluster from the runner host with its runner token:
Shutdown deletes the runner’s server-side token rows and marks the runner revoked. If the host installed a systemd or launchd runner service, stop and disable that service as well so the OS does not keep restarting an invalid agent process. Run heavy commands remotely:
By default, ttr box run creates an auto lease, waits for an authenticated runner to claim the job, streams recorded logs after completion, and releases the lease after one-off work. Use --keep for repeated sync/run work:

​
Build, Serve, And Tunnel

Use ttr box build for common build workloads without spelling the full remote command every time:
Use ttr box serve when the remote devbox should build an app, start the server, and keep the run alive. It defaults to apps/web on port 7803, stores the preview port on the run, and returns immediately unless --wait is passed:
ttr box build, ttr box serve, and ttr box tunnel do not set a remote command timeout by default. Pass --timeout <duration> only when the run should be killed automatically. To expose a served app through a Cloudflare tunnel, keep the token in a local environment variable and pass the variable name to the devbox. The queued command references $CLOUDFLARED_TOKEN; the raw token is delivered only as run-scoped env and is redacted from logs.
ttr box tunnel runs a dockerized cloudflare/cloudflared container with host networking on the runner. The command policy blocks inline cloudflared --token ... arguments, so operators should use --cloudflared-token-env or another local environment variable indirection rather than pasting raw tokens into commands.

​
Container Boundary

The current agent starts brokered commands directly on the host. The command policy blocks several destructive command forms, but is not a sandbox. These machines must be operated as trusted infrastructure. Docker-backed commands create their own containers only when the requested command does so. Allowed remote workflows include:
  • bun check, bun test, and package-local Bun commands
  • bun test:e2e and Docker-backed Playwright workflows
  • bun sb:start, bun sb:reset, bun sb:up, and bun sb:typegen
Blocked defaults include privileged Docker containers, Docker prune-all style commands, absolute host mounts, sudo, and host-destructive filesystem operations.

​
Env And Secrets

Remote env is explicit. Local .env* files are never synced automatically. The setup command only prepares ignored app env files inside the runner checkout so local Supabase-backed apps and E2E workflows can use that runner’s own Supabase stack.
Brokered env values are scoped to the run or lease, delivered only to the claimed runner, and redacted from command logs. For split-resource development, run Supabase or another database on one devbox, publish its reachable URL as DEVBOX_DATABASE_URL on the operator machine, and queue the app devbox with --database-url-env DEVBOX_DATABASE_URL. This lets a database-heavy runner and a Next.js runner share work while the current machine only brokers commands and opens previews.

​
Runner scheduling and hosted environments

Agents keep heartbeats on a 20-second cadence while dispatching independent jobs concurrently, with at most eight active jobs. Judge cases share a process-wide sandbox budget and always use fresh isolated containers. Each case reserves 1 / parallelLimit of the host budget, so mixed jobs cannot each claim the whole memory or CPU allocation. Eligible smaller reservations can pass a waiting full-budget case. Infrastructure failures cancel queued cases and wait for already admitted commands and container cleanup before reporting failure. Regression coverage lives in devbox-sandbox-slots.test.ts and devbox-judge-case-batch.test.ts. Maintenance and CLI restart jobs drain active work before changing the agent. An execution failure stops new dispatch and drains already active work before the agent exits. Heartbeats continue during that drain, and local completions wake idle claim polling. The parent scheduling client omits Playground capabilities until both heartbeat contracts support them; devbox-agent-capabilities.test.ts exercises its emitted body against both current handlers. Judge readiness probes are cached for ten minutes and refreshed in the background; execution still fails closed if required isolation or language images are unavailable. Platform-managed playground workers opt in through operator configuration. They require gVisor and pinned language image digests, run unprivileged without host credentials, and disable network access by default. Project environments retain dependencies while warm. A configured TUTURUUU_PLAYGROUND_POOL_ID dedicates the agent process to Playground: Judge readiness is false and Judge execution fails before Docker admission, so warm environments cannot double the Judge host budget. Idle/max-age eviction waits for an active project run and holds the pool lock; near-expiry containers are replaced before admitting another bounded run. Failed starts remove their exact owned name, retaining quarantined capacity when Docker cannot confirm removal. Environments expire after fifteen idle minutes and have a two-hour maximum lifetime. Environment count is bounded by the runner’s resource limits. A failed container removal retains its capacity reservation until a successful removal or healthy Docker inventory confirms the container is absent. Failed startup orphan cleanup blocks new environments and is retried on the next admission. Creation and stop removal share the same pool lock. Same-project runs serialize synchronization, execution and saves; stop invalidates queued runs without waiting for a running command. Stopping waits for admitted creation, with a fifteen-second wait bound; a timeout reports an unconfirmed stop and retains capacity ownership. It does not wait for the arbitrary project command to finish. Python helpers use the pinned image PATH, including images that install Python under /usr/local/bin. A failed background save emits a fixed diagnostic and makes the run report incomplete persistence rather than silently claiming success. These lifecycle rules are covered by devbox-playground-sandbox.test.ts. File inventories and changed bytes are exported separately; credentials and runner metadata never enter project files. File/directory conflicts are checked against every path ancestor, and preview paths reject traversal before URL normalization. Export transport permits the worst-case JSON escaping of a valid two-MiB text project; unsupported paths or oversized files report the offending file instead of failing with an opaque schema error. The schema and Python export contracts cover these rules. The platform must provide its signed file-save callback before enabling the hosted playground workload. The scheduling primitives alone reject the reserved hosted Playground envelope without running a host command; its HTTP adapter ships together with the canonical file-save API. The typed-adapter regression in devbox-runner-judge.test.ts verifies hosted execution uses the sandbox and canonical authenticated callback, while malformed envelopes never fall through to host execution. Use focused SDK agent, Judge and sandbox-slot tests for scheduling changes. A passing mock test does not establish real sandbox startup latency; measure cold and warm execution on a configured gVisor runner before advertising timings. Do not fall back to ordinary Docker isolation when gVisor is missing.

​
Cache Policy

Runners use named cache volumes for Bun installs, Turbo, Playwright browsers, Supabase Docker state, package manager cache, and optional node_modules. Cache keys include the repo fingerprint, lockfile hash, runtime image digest, platform, command profile, Bun/Node versions, and cache schema version. Cleanup runs on runner startup, after runs, and during heartbeat maintenance. The runner evicts incompatible caches first, including legacy Bun install caches when the Bun version, lockfile hash, package profile, or cache schema changes. It then enforces cache budgets with least-recently-used eviction while keeping a small set of recent compatible caches to avoid thrashing. Operators can inspect and prune cache metadata:

​
Access

Remote devbox API routes require a Tuturuuu CLI/app session and root workspace membership with workspace_members.type = 'MEMBER'. Guests and non-root workspace users cannot create runs, leases, previews, or runner tokens.

​
Local Executable Verification

Use an isolated CLI config when testing local apps/web so the production CLI session is not overwritten:
The final command should print the remote command logs and exit with the remote command’s exit code. In local Supabase, private.devbox_runs.status should end as succeeded with exit_code = 0.

​
Production Readiness

The production API needs migration 20260603171600_create_private_devboxes.sql before any ttr box command can create leases or runs. If the CLI prints a schema-cache error such as private.devbox_leases not being found, first confirm the production migration workflow ran for the deployed SHA. If the table exists but PostgREST still returns the schema-cache error, refresh the schema cache from Supabase SQL Editor:
Fleet selection rejects combining --all with explicit runner IDs. Maintenance is deferred when a runner does not report its hostname. Fleet upgrades honor --timeout; a timed-out run requests cancellation and reports whether that request succeeded. Verify its run and lease before retrying or touching another runner, since a stop request does not prove immediate process termination.

​
Runtime Acceptance Diagnostics

Judge capacity discovery runs a strict four-second docker info metadata probe before allocating a sandbox. A failure reports a fixed timeout, spawn (not found, permission denied, or other), numeric exit, signal, or missing-exit classification. These diagnostics omit Docker hosts, command arguments, raw output, and provider error messages. A timeout is not proof that the engine is absent: inspect the owning CI job and runner lifecycle before repairing or retrying. Successful discovery still requires registered gVisor, enforced cgroups, and reported CPU/memory capacity. Mocked process tests verify the classification and failure boundary; they do not verify a runner’s Docker engine. The Lettin persistence browser regression separately reports project-navigation confirmation and the notebook-button action, retaining their existing deadlines and assertions. Phase failures emit an allowlisted error name and a fixed message summary rather than authenticated URLs or response bodies. A runner shutdown without a final assertion report leaves the underlying cause unproven. Preserve allocated fixtures until successful final cleanup or independently verified resource absence, and use the next exact-source CI result for runtime evidence.