> ## Documentation Index > Fetch the complete documentation index at: https://docs.tuturuuu.com/llms.txt > Use this file to discover all available pages before exploring further. # Machine Storage Health > Measure, triage and safely reclaim disk space on Macs, Linux devboxes and self-hosted runners with ttr storage. Every Tuturuuu machine (the M4 Mac, Linux devboxes and VMs, self-hosted CI runners) uses one tool for storage triage: `ttr storage`. It reports headroom, ranks consumers by the bytes deleting them would actually free, suggests exact cleanup commands, and applies only safe, bounded cleanup on request. ## Daily triage ```bash theme={null} ttr storage status # free/used/pressure and inodes per volume (exit 2 when critical) ttr storage analyze # ranked consumers by category (add --json) ttr storage suggest # actions: reclaimable bytes, risk, exact command ttr storage prune # dry run of the default safe categories ttr storage prune --apply # re-verifies every fence, then applies the plan ttr storage reclaim # automatic-reclaim plan, lease holder and last run on this host ttr storage fleet # disk headroom reported by every devbox runner ``` `analyze` and `suggest` walk large trees (Bun cache, stale `node_modules`, every worktree) and take a few minutes on a busy Mac; run them through `ttr resources run --` when other lanes are validating. `suggest` caches its top actions in `~/.tuturuuu/storage/last-suggest.json`. ## Measure real bytes, not apparent bytes | Platform | Method | Why | | - | - | - | | macOS (APFS) | `getattrlist` `ATTR_CMNEXT_PRIVATESIZE` per file, through the system Python | Bun's `node_modules` are APFS clones of `~/.bun/install/cache`; `du` counts shared blocks once per clone. | | Linux | Allocated blocks (`st_blocks`), hard links counted once per run | Bun's Linux backend hard-links from its cache. | Never use `du -L`: it follows Bun's workspace symlinks and reported 300 GiB for `apps/web` alone. A fresh worktree costs about 0.3 GiB unique (source only) and about 250k inodes, not the 4–9 GiB `du` shows. `bun worktree:footprint` reports the per-worktree breakdown. ## Thresholds | Pressure | Rule (any one) | | - | - | | `critical` | free \< `TTR_STORAGE_MIN_FREE` (15 GiB), free \< `TTR_STORAGE_MIN_FREE_PERCENT` (5%), or free inodes \< `TTR_STORAGE_MIN_FREE_INODE_PERCENT` (5%) | | `low` | within twice any threshold | | `ok` | otherwise | Free space is what an unprivileged user can allocate (`statfs` `bavail`). On APFS every volume shares its container's free space and inode tables grow on demand, so the inode percentage only matters on ext4/xfs. Invalid threshold values fall back to the default with a warning. Heavy admission uses hysteresis on top of these thresholds. A volume enters `low` below the `critical` thresholds above and leaves it only at the separate recover thresholds: * `TTR_STORAGE_RECOVER_FREE` * `TTR_STORAGE_RECOVER_FREE_PERCENT` * `TTR_STORAGE_RECOVER_FREE_INODE_PERCENT` Each defaults to twice its low threshold (30 GiB / 10% / 10%). A recover value below its low value is raised to the low value. These are configurable starting points, not measured fleet values; tune them per host. ## Disk-aware admission `ttr resources` checks the volume holding a heavy command's working directory before queueing it. Once queued, the lane queue re-checks free disk and inodes at dequeue (`TTR_STORAGE_MIN_FREE`, `TTR_STORAGE_MIN_FREE_PERCENT`, and `TTR_STORAGE_MIN_FREE_INODE_PERCENT`) and holds the job, failing closed, until headroom returns. * **`low`:** while the volume is in the hysteresis `low` state, automatic reclaim runs first (see below). If that does not bring the volume back to the recover thresholds, the command is refused with exit code 75 and a blocker report. The report gives: * what reclaim removed and its receipt file; * the eligible items it skipped, with their owners and reasons; * the largest non-high manual actions from the last `ttr storage suggest`, with their hold owners; * the exact free space needed to resume. * **`ok`:** when the volume is `ok` but below the recover thresholds, the command warns and continues. * **Alerts:** warn, low and recovered alerts print once per storage authority (this OS user on this host), volume and level. The state lives in `~/.tuturuuu/storage/pressure-state.json`. * **Probe failure:** a failed probe is UNKNOWN. It never triggers reclaim and the command is refused with the probe error; only the override below admits it. * `--ignore-disk` (or `TTR_RESOURCES_IGNORE_DISK=1`) overrides both checks once. Prefer reclaiming space. * `--hosts` dispatch skips the queue-time refusal; it applies its own local disk gate per placement, including inode pressure. Both placement and dequeue probe the command directory selected by `--cwd`, `--dir` or `-C`. * With `--offload auto`, a critical local disk counts as saturated, so the exact pushed commit runs on a devbox runner when one is eligible. * Commands nested inside an admitted job are not checked again. ## Fleet headroom Each devbox agent adds a bucketed disk report to its existing heartbeat: `resources.disk = { freeGiB, freePct, inodeFreePct? }`. Values are conservative lower bounds (GiB steps 0, 1, 2, 5, 10, 15, 20, 30, 50, …; percentages in 5% steps), about 35 bytes against the 4 KiB heartbeat cap. The probed path is the agent's working directory, or `TUTURUUU_DEVBOX_STORAGE_PATH`. The web route (zod `.strict()`) and the devbox-control Worker (`typeof` checks, no coercion) both reject malformed reports. ```bash theme={null} ttr storage fleet # every registered runner, critical hosts first (exit 2 if any) ttr resources offload # blockedBy=low_disk when every trusted runner is critical ``` Offload selection skips a trusted runner that reports critical disk. Runners that do not report disk yet (older agents) stay eligible and show as `unreported`. No new timer or controller is involved: the heartbeat is the transport. The authenticated heartbeat of a registered runner is the only fleet input. `ttr storage fleet` reports a runner as `unknown` when any of these hold: * it is offline; * its heartbeat is disabled; * it has never reported; * its last heartbeat is older than the 60-second freshness window. `unknown` is never healthy, and `fleet` exits 2 when any runner is `unknown`. Nothing connects to a host to measure or clean it; a host's own heartbeat or admission triggers its own reclaim. ## Automatic reclaim Automatic reclaim is **off by default**. It runs only after an explicit opt-in for this storage authority: `TTR_STORAGE_AUTO_RECLAIM=1` (or `true`, `on`, `yes`), or the sentinel file `~/.tuturuuu/storage/auto-reclaim` containing one of those values. Once opted in, when a trigger finds a volume `low`, reclaim removes explicitly regenerable space on its own. A human can always run `ttr storage reclaim --apply` without opting in. There are two triggers, and neither is a schedule: * heavy `ttr resources` admission; * a devbox agent heartbeat, which starts one detached `ttr storage reclaim --apply --auto --trigger heartbeat`. This follows the shared [cross-orchestrator quota and storage coordination](#cross-orchestrator-contract) contract. **Scope: one storage authority.** Every guarantee below holds for one storage authority: one OS user's storage home (`TTR_STORAGE_HOME`, default `~/.tuturuuu/storage`) on one host. Every orchestrator and project that user runs (Tuturuuu, Upskii, Mastermind) shares it. Another OS user on the same host has a separate home, lease and receipts; nothing here coordinates across OS users, so this is not a host-wide guarantee. * **One lease per storage authority.** The lease is `~/.tuturuuu/storage/reclaim//cleanup.lease`, keyed by host identity (`TTR_STORAGE_HOST_ID`, default the hostname) inside the user's home. It records: * a monotonic fence number (from a durable counter, `fence`); * TTL (15 minutes, renewed before each item); * owner pid and start time; * trigger; * the caller's `TTR_ORCHESTRATOR_ID`. Acquisition, takeover, renewal and release all run under an O\_EXCL mutation lock, so they are serialized. A second caller does not reclaim in parallel; its admission refusal names the holder. An expired or dead-pid lease is taken over with a larger fence. Holders record their process start time, so a reused pid counts as a dead holder. Breaking a dead holder's mutation lock is itself serialized behind a separate O\_EXCL break lock, and the lock is re-read under it before removal, so a fresh lock is never broken. * **Fenced at the destructive boundary.** Each removal runs inside the lease's guard: under the mutation lock, the holder's token, fence and expiry are re-validated, and the removal runs while the lock is still held. A holder that paused past its TTL and was taken over never removes anything, and no takeover can overlap a removal. * **Eligible items only.** Only path-bounded rules, removed as one verified directory tree, so the deleted set is exactly the checked set: * idle Xcode DerivedData folders, never while Xcode or `xcodebuild` runs; * ignored, untracked build artifacts inside merged, clean, unheld worktrees idle at least a day. Command-based cleanups (Homebrew, npm cache, Docker, simulators) are never automatic; they stay manual suggestions. `TTR_STORAGE_AUTO_RECLAIM_RULES` narrows the list (comma-separated rule ids). The idle age is `TTR_STORAGE_RECLAIM_OLDER_THAN_DAYS` (7), measured over the whole tree. Only candidates on the low volume's device count, checked when planning and again right before each item. A volume reading that is not usable (no device, non-positive total, non-finite or inconsistent values) stops the run with nothing removed. * **Re-checked right before each item, fail closed.** In this order: 1. the directory identity (device and inode chain from the rule root, no symlinks, same device) is captured; 2. holds from every file in every holds directory (an invalid line stops); 3. the rule's own re-check (for worktree artifacts, a fresh merged-clean classification at the same `HEAD`); 4. a process `cwd` scan covering every live process of this user; 5. `lsof +D` open files. A check that fails or errors skips the item. Hidden processes of other users (including root) are tolerated only with [private-tree evidence](#process-visibility-on-macos-private-tree-evidence) for the item, proven at or above its rule root. Receipts record the evidence level (`full-visibility` or `private-tree`). * **Durable pre-receipt.** Before acting, an `intent` line is appended and fsynced (new directories and the new file's directory entry are fsynced too) to `~/.tuturuuu/storage/reclaim//receipts/*.jsonl` (mode 0600). If the append or any fsync fails, the item is not removed. After acting, an `outcome` line records the result. Each line records path and bytes, rule, the checks that passed, the lease fence, outcome and freed bytes, host, orchestrator, trigger and time. * **Bounded by the goal.** Free space is re-measured under the lease before each item, and reclaim stops at the recover thresholds. If space recovered meanwhile, nothing is removed. * **Not enough safe reclaim.** Heavy admission stays closed and prints the blocker report. Reclaim never widens its scope to close the gap. After an unsuccessful run, triggers wait `TTR_STORAGE_RECLAIM_COOLDOWN_MINUTES` (30) before planning again. `ttr storage reclaim --apply` ignores only the cooldown. * **Hard kill switch.** A false value (`0`, `false`, `off`, `no`) in `TTR_STORAGE_AUTO_RECLAIM` (process or passed environment) or in the `auto-reclaim` sentinel file disables both triggers and the manual `ttr storage reclaim --apply`, and wins over any opt-in. A sentinel with any other content, or one that cannot be read, also kills. The engine checks it itself before the lease, before every item and inside the lease guard right before each removal, so no caller or flag bypasses it. * **Trusted re-exec.** The heartbeat starts the child only through a verified absolute path: the absolute runtime plus the real path of an existing script entry, or a packaged executable named `ttr`. It never resolves `ttr` through `PATH`; otherwise it spawns nothing. Set `TTR_ORCHESTRATOR_ID` in each orchestrator's environment so leases and receipts name it. Without it they record an unidentified caller. T3 does not inject a thread id into child processes today. ### What stays manual Automatic reclaim never touches any of these: * worktrees themselves; * Homebrew, npm, Bun and pub caches, Docker and simulators; * `node_modules`; * tmp, evidence and scratch; * held paths; * databases and Supabase or Docker volumes; * Docker images; * VM disks, Trash, archives and AVDs; * production assets; * files another process has open. A human owns those decisions: run `ttr storage suggest` and `ttr storage prune`, and ask hold owners to release their holds. The blocker report names the largest such items and their owners. ### Activation These hooks ship in the `ttr` CLI. Until a host runs a released CLI that contains them, they are not active there. A heartbeat trigger also needs that host's devbox agent to be upgraded. Record per-host activation evidence (version, a receipt or a refusal) before claiming coverage. ## Categories | Category | What | Risk | `prune` | | - | - | - | - | | `worktrees` | Linked worktrees, classified `dirty`, `unmerged`, `merged-clean`, `orphaned`, with owner and hold status | safe for merged-clean | named only | | `node-modules-stale` | Bun stores with far more `.bun` entries than `bun.lock` packages (diverged private clones) | moderate | named, one `--path` | | `build-artifacts` | `.turbo`, `.next`, `.open-next`, `dist`, `build`, `.dart_tool` in checkouts | safe | default, via `bun worktree:prune-artifacts` | | `rust-target` | `apps/backend/target` | safe | default, via `bun rust-cache prune` | | `bun-cache`, `npm-cache`, `pub-cache` | Package manager caches. The npm cache root is resolved once (`npm_config_cache`, then `npm config get cache`, then `~/.npm`); `prune` measures that root's `_cacache` and runs `npm cache clean --force --cache ` only on a directory owned by you | moderate / safe | named only | | `xcode-derived-data` | DerivedData project folders idle past `--older-than` | safe | default | | `xcode-simulators` | Simulators for uninstalled runtimes, each deleted by its own UDID (`xcrun simctl delete `, never `delete unavailable`) | safe | default | | `homebrew-cache` | `brew cleanup --prune=` (also removes old formula versions) | safe | named only | | `docker` | Build cache older than the age bound; unused images. Verified local daemon only | safe / moderate | named only | | `supabase-volumes` | Supabase CLI local volumes | high | only `--name `, never attached | | `tmp-old` | Identified tool scratch dirs (mktemp, `tmp-*`, Vitest, Turbo, Bun, `ttr-*` and similar) directly under a tmp root, idle past the age bound, at least 25 MiB, with no git checkout, linked worktree or bare repository inside (checked again by the deleting walk itself, which refuses the whole removal if one appears) | moderate | named only | | `xcode-device-support`, `gradle`, `playwright` | Rebuildable but costly | moderate | manual command only | | `trash`, `xcode-archives`, `android-sdk` AVDs, `vm-disks` | User data, signed builds, emulator state, VM images | high | never | Defaults: `--older-than 7` days, `--max-remove 20G` total per run. ## What is never removed automatically * Dirty, unmerged, locked, busy (a process `cwd` inside) or current worktrees, and the primary checkout. * Worktrees active in the last 24 hours. * Any path covered by a hold file. Every file in `~/.t3/cleanup/holds.d` counts, except dotfiles and editor swap or backup files (`~`, `.swp`), which are skipped with a warning. That includes each project's file (`tuturuuu.txt`, `upskii.txt`, `mastermind.txt`, …) whatever its extension. `--holds-dir` and `TTR_STORAGE_HOLDS_DIRS` add directories; the shared default is always read. Each file holds one absolute path or glob per line, with `#` comments. A hold covers its path, everything below it and everything above it. An unreadable holds directory, or any non-comment line that is not an absolute path or glob, stops the run with the file and line; fix or comment out the line. * Any path a process may be using. Right before each removal, every process's working directory (`lsof -d cwd`, checked against the full `ps` table) and every open file under the path (`lsof +D`) are re-checked. A failed scan, any `lsof` diagnostic, or an uninspectable live process of this user means use is not proven absent, and the step is skipped. Uninspectable processes of other users (including root) are tolerated only under the private-tree rule below. `ps` and `lsof` run by absolute path and scripts by the absolute runtime, so a shim directory on `PATH` cannot answer for them. * Worktrees whose ignored files are not rebuildable. `git worktree remove` deletes ignored files without `--force`, so a worktree is kept when `git ls-files -o -i --exclude-standard --directory` lists anything outside build artifacts, `node_modules` and `target` (for example `.env.local` or `tmp/agent-coordination/`); the dry run names those paths. * Trees with entries this user does not own, or with a group- or world-writable directory (`mode & 0o022`), checked in the pre-scan and again before every unlink. Running as root removes only fully root-owned trees. * Anything reached through a symlink or across a mount. Trees are removed by directory identity (device and inode) captured before the checks and re-verified before every unlink; a swapped ancestor or a nested mount stops the removal, and entries created after the pre-scan are left in place. * Docker on a remote or unknown engine. Docker steps run only when `DOCKER_HOST` is unset and the current context's endpoint is an existing local unix socket; the command is pinned to that socket with `-H`. * `preserve/*` branches and every other ref. Merged-clean worktrees are removed with plain `git worktree remove` (no `--force`) and their branch with `git branch -d`; git keeps a squash-merged branch, which is reported for a human to delete after checking. * Repositories or linked worktrees nested inside a reclaim candidate, including ignored output and bare repositories. Planning rejects the candidate; the deletion walk rechecks repository markers before every unlink and rmdir. * Source, Trash, VM disk images, Xcode archives, AVDs and Supabase volumes not named explicitly. "Merged" means the exact local `HEAD` reached the base: either it is an ancestor of `origin/main`, or a merged pull request's head is that `HEAD` (squash merges). An open PR, a closed-unmerged PR or local commits after the merged head keep the worktree. Every apply step re-reads holds, process working directories, open files and worktree state immediately before acting. Idle ages use the newest modification or change time anywhere in a tree (a deep, recently written file keeps the whole tree); trees over 500,000 entries or unreadable ones count as recent. ## Process visibility on macOS: private-tree evidence Decision (orchestrator, 2026-10-23): a non-root user's `lsof` cannot inspect other users' or root's processes, so strict visibility would skip every removal on macOS. Removal (manual `prune` and automatic reclaim alike) may proceed with `private-tree` evidence only when all of these hold: 1. Every live process of the effective user was inspected; only processes of other uids may be hidden. 2. The item has a private-tree proof: some directory at or above the rule root is owned by the effective user, has `mode & 0o077 == 0` and no ACL; every directory from there down to the item is owned by the effective user and not group- or other-writable; all on one device. 3. The open-file scan is scoped to the effective user (`lsof -a -u `). 4. Never when running as root, and only on macOS. Linux stays fail-closed until a container-mount check exists. Receipts and step checks record the evidence level (`private-tree` or `full-visibility`). Root's processes remain uninspectable under this rule; that residual is accepted only for proven-private trees. Idle Xcode DerivedData usually qualifies through `~/Library/Developer` (0700, no ACL). Its repository-refusal exemption applies only to the literal `~/Library/Developer/Xcode/DerivedData` root when its canonical path matches and every component through the user's home is a real directory owned by the effective user. Symlinked roots or ancestors and nonstandard report roots are skipped, and the chain is checked again immediately before removal. This keeps checkouts reached through a redirected DerivedData path intact; see `packages/sdk/src/cli/storage-derived-data.test.ts`. Worktrees and their artifacts under `~/.t3` do not qualify unless that tree is 0700; ttr never changes permissions to make something qualify. Regression coverage: `packages/sdk/src/cli/storage-privacy.test.ts` (scan rules per platform and uid, the proof's failure modes, scoped `lsof`, manual prune wiring), `storage-liveness.test.ts` (strict scan parsing) and `storage-reclaim.test.ts` (automatic reclaim records `private-tree` evidence in its receipts). ## Stale `node_modules` `bun install` never removes old `node_modules/.bun/` entries. On APFS the old entries become private copies once the cache is refreshed; the main checkout held about 10 GiB this way. The fix is a reinstall, queued through `ttr resources`: ```bash theme={null} ttr storage prune --category node-modules-stale --path # dry run ttr storage prune --category node-modules-stale --path --apply ``` This removes `node_modules` and runs `ttr resources run -- bun install --frozen-lockfile`. It refuses while another process works inside the checkout. A shared primary checkout needs its owner's approval first. ## Per-OS commands ### macOS (APFS) ```bash theme={null} diskutil apfs list | grep -E 'Capacity (In Use|Not Allocated)' df -h /System/Volumes/Data # the Data volume holds user files xcrun simctl list devices booted xcrun simctl shutdown all # a booted simulator mounts ~220 GB of phantom DeviceFS size tmutil listlocalsnapshots / # local snapshots pin deleted blocks ``` ### Linux devbox or runner ```bash theme={null} df -h / /var/lib/docker df -i / # inodes; ext4 tables are fixed at mkfs docker system df journalctl --disk-usage ``` Self-hosted runner jobs remove their own `start-source--*` checkout in an `if: always()` step; sweep leftovers only from completed runs. ## Ownership Run `ttr storage prune --apply` only on a machine you own. Devbox VMs and CI runner hosts belong to their host owner; report suggestions to that owner instead. On the M4 Mac the Mastermind daily cleanup task uses the same hold files and busy/PR-state rules, so a hold written for one tool protects the path from both. Automatic reclaim acts only for the storage authority (OS user and host) that triggered it, under that authority's lease. ### Cross-orchestrator contract Every orchestrator (Tuturuuu, Upskii, Mastermind) follows the shared contract in `~/.codex/skills/pstack-t3-workflow/references/t3-orchestration.md#cross-orchestrator-quota-and-storage-coordination`. Under that contract: * missing telemetry stays UNKNOWN; * one cleanup lease owns each host's reclaim (implemented here per storage authority, one OS user's home; a host-wide authority shared across OS users is not implemented yet); * automatic reclaim is limited to eligible regenerable caches and completed, unowned outputs, with receipts; * insufficient reclaim pauses heavy admission or offloads it rather than deleting more. The Tuturuuu `ttr storage` lane is the single implementation owner of these hooks; other projects supply requirements and receipts. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.