Machine Storage Health

​

Daily triage

ttr storage status             # free/used/pressure and inodes per volume (exit 2 when critical)
ttr storage analyze            # ranked consumers by category (add --json)
ttr storage suggest            # actions: reclaimable bytes, risk, exact command
ttr storage prune              # dry run of the default safe categories
ttr storage prune --apply      # re-verifies every fence, then applies the plan
ttr storage reclaim            # automatic-reclaim plan, lease holder and last run on this host
ttr storage fleet              # disk headroom reported by every devbox runner

​

Measure real bytes, not apparent bytes

​

Thresholds

• TTR_STORAGE_RECOVER_FREE
• TTR_STORAGE_RECOVER_FREE_PERCENT
• TTR_STORAGE_RECOVER_FREE_INODE_PERCENT

​

Disk-aware admission

• low: while the volume is in the hysteresis low state, automatic reclaim runs first (see below). If that does not bring the volume back to the recover thresholds, the command is refused with exit code 75 and a blocker report. The report gives:
  • what reclaim removed and its receipt file;
  • the eligible items it skipped, with their owners and reasons;
  • the largest non-high manual actions from the last ttr storage suggest, with their hold owners;
  • the exact free space needed to resume.
• ok: when the volume is ok but below the recover thresholds, the command warns and continues.
• Alerts: warn, low and recovered alerts print once per storage authority (this OS user on this host), volume and level. The state lives in ~/.tuturuuu/storage/pressure-state.json.
• Probe failure: a failed probe is UNKNOWN. It never triggers reclaim and the command is refused with the probe error; only the override below admits it.
• --ignore-disk (or TTR_RESOURCES_IGNORE_DISK=1) overrides both checks once. Prefer reclaiming space.
• --hosts dispatch skips the queue-time refusal; it applies its own local disk gate per placement, including inode pressure. Both placement and dequeue probe the command directory selected by --cwd, --dir or -C.
• With --offload auto, a critical local disk counts as saturated, so the exact pushed commit runs on a devbox runner when one is eligible.
• Commands nested inside an admitted job are not checked again.

​

Fleet headroom

ttr storage fleet          # every registered runner, critical hosts first (exit 2 if any)
ttr resources offload      # blockedBy=low_disk when every trusted runner is critical

• it is offline;
• its heartbeat is disabled;
• it has never reported;
• its last heartbeat is older than the 60-second freshness window.

​

Automatic reclaim

• heavy ttr resources admission;
• a devbox agent heartbeat, which starts one detached ttr storage reclaim --apply --auto --trigger heartbeat.

• One lease per storage authority. The lease is ~/.tuturuuu/storage/reclaim/<host>/cleanup.lease, keyed by host identity (TTR_STORAGE_HOST_ID, default the hostname) inside the user’s home. It records:
  • a monotonic fence number (from a durable counter, fence);
  • TTL (15 minutes, renewed before each item);
  • owner pid and start time;
  • trigger;
  • the caller’s TTR_ORCHESTRATOR_ID.
  Acquisition, takeover, renewal and release all run under an O_EXCL mutation lock, so they are serialized. A second caller does not reclaim in parallel; its admission refusal names the holder. An expired or dead-pid lease is taken over with a larger fence. Holders record their process start time, so a reused pid counts as a dead holder. Breaking a dead holder’s mutation lock is itself serialized behind a separate O_EXCL break lock, and the lock is re-read under it before removal, so a fresh lock is never broken.
• Fenced at the destructive boundary. Each removal runs inside the lease’s guard: under the mutation lock, the holder’s token, fence and expiry are re-validated, and the removal runs while the lock is still held. A holder that paused past its TTL and was taken over never removes anything, and no takeover can overlap a removal.
• Eligible items only. Only path-bounded rules, removed as one verified directory tree, so the deleted set is exactly the checked set:
  • idle Xcode DerivedData folders, never while Xcode or xcodebuild runs;
  • ignored, untracked build artifacts inside merged, clean, unheld worktrees idle at least a day.
  Command-based cleanups (Homebrew, npm cache, Docker, simulators) are never automatic; they stay manual suggestions. TTR_STORAGE_AUTO_RECLAIM_RULES narrows the list (comma-separated rule ids). The idle age is TTR_STORAGE_RECLAIM_OLDER_THAN_DAYS (7), measured over the whole tree. Only candidates on the low volume’s device count, checked when planning and again right before each item. A volume reading that is not usable (no device, non-positive total, non-finite or inconsistent values) stops the run with nothing removed.
• Re-checked right before each item, fail closed. In this order:
  1. the directory identity (device and inode chain from the rule root, no symlinks, same device) is captured;
  2. holds from every file in every holds directory (an invalid line stops);
  3. the rule’s own re-check (for worktree artifacts, a fresh merged-clean classification at the same HEAD);
  4. a process cwd scan covering every live process of this user;
  5. lsof +D open files.
  A check that fails or errors skips the item. Hidden processes of other users (including root) are tolerated only with private-tree evidence for the item, proven at or above its rule root. Receipts record the evidence level (full-visibility or private-tree).
• Durable pre-receipt. Before acting, an intent line is appended and fsynced (new directories and the new file’s directory entry are fsynced too) to ~/.tuturuuu/storage/reclaim/<host>/receipts/*.jsonl (mode 0600). If the append or any fsync fails, the item is not removed. After acting, an outcome line records the result. Each line records path and bytes, rule, the checks that passed, the lease fence, outcome and freed bytes, host, orchestrator, trigger and time.
• Bounded by the goal. Free space is re-measured under the lease before each item, and reclaim stops at the recover thresholds. If space recovered meanwhile, nothing is removed.
• Not enough safe reclaim. Heavy admission stays closed and prints the blocker report. Reclaim never widens its scope to close the gap. After an unsuccessful run, triggers wait TTR_STORAGE_RECLAIM_COOLDOWN_MINUTES (30) before planning again. ttr storage reclaim --apply ignores only the cooldown.
• Hard kill switch. A false value (0, false, off, no) in TTR_STORAGE_AUTO_RECLAIM (process or passed environment) or in the auto-reclaim sentinel file disables both triggers and the manual ttr storage reclaim --apply, and wins over any opt-in. A sentinel with any other content, or one that cannot be read, also kills. The engine checks it itself before the lease, before every item and inside the lease guard right before each removal, so no caller or flag bypasses it.
• Trusted re-exec. The heartbeat starts the child only through a verified absolute path: the absolute runtime plus the real path of an existing script entry, or a packaged executable named ttr. It never resolves ttr through PATH; otherwise it spawns nothing.

​

What stays manual

• worktrees themselves;
• Homebrew, npm, Bun and pub caches, Docker and simulators;
• node_modules;
• tmp, evidence and scratch;
• held paths;
• databases and Supabase or Docker volumes;
• Docker images;
• VM disks, Trash, archives and AVDs;
• production assets;
• files another process has open.

​

Activation

​

Categories

​

What is never removed automatically

• Dirty, unmerged, locked, busy (a process cwd inside) or current worktrees, and the primary checkout.
• Worktrees active in the last 24 hours.
• Any path covered by a hold file. Every file in ~/.t3/cleanup/holds.d counts, except dotfiles and editor swap or backup files (~, .swp), which are skipped with a warning. That includes each project’s file (tuturuuu.txt, upskii.txt, mastermind.txt, …) whatever its extension. --holds-dir and TTR_STORAGE_HOLDS_DIRS add directories; the shared default is always read. Each file holds one absolute path or glob per line, with # comments. A hold covers its path, everything below it and everything above it. An unreadable holds directory, or any non-comment line that is not an absolute path or glob, stops the run with the file and line; fix or comment out the line.
• Any path a process may be using. Right before each removal, every process’s working directory (lsof -d cwd, checked against the full ps table) and every open file under the path (lsof +D) are re-checked. A failed scan, any lsof diagnostic, or an uninspectable live process of this user means use is not proven absent, and the step is skipped. Uninspectable processes of other users (including root) are tolerated only under the private-tree rule below. ps and lsof run by absolute path and scripts by the absolute runtime, so a shim directory on PATH cannot answer for them.
• Worktrees whose ignored files are not rebuildable. git worktree remove deletes ignored files without --force, so a worktree is kept when git ls-files -o -i --exclude-standard --directory lists anything outside build artifacts, node_modules and target (for example .env.local or tmp/agent-coordination/); the dry run names those paths.
• Trees with entries this user does not own, or with a group- or world-writable directory (mode & 0o022), checked in the pre-scan and again before every unlink. Running as root removes only fully root-owned trees.
• Anything reached through a symlink or across a mount. Trees are removed by directory identity (device and inode) captured before the checks and re-verified before every unlink; a swapped ancestor or a nested mount stops the removal, and entries created after the pre-scan are left in place.
• Docker on a remote or unknown engine. Docker steps run only when DOCKER_HOST is unset and the current context’s endpoint is an existing local unix socket; the command is pinned to that socket with -H.
• preserve/* branches and every other ref. Merged-clean worktrees are removed with plain git worktree remove (no --force) and their branch with git branch -d; git keeps a squash-merged branch, which is reported for a human to delete after checking.
• Repositories or linked worktrees nested inside a reclaim candidate, including ignored output and bare repositories. Planning rejects the candidate; the deletion walk rechecks repository markers before every unlink and rmdir.
• Source, Trash, VM disk images, Xcode archives, AVDs and Supabase volumes not named explicitly.

​

Process visibility on macOS: private-tree evidence

1. Every live process of the effective user was inspected; only processes of other uids may be hidden.
2. The item has a private-tree proof: some directory at or above the rule root is owned by the effective user, has mode & 0o077 == 0 and no ACL; every directory from there down to the item is owned by the effective user and not group- or other-writable; all on one device.
3. The open-file scan is scoped to the effective user (lsof -a -u <euid>).
4. Never when running as root, and only on macOS. Linux stays fail-closed until a container-mount check exists.

​

Stale node_modules

ttr storage prune --category node-modules-stale --path <checkout>        # dry run
ttr storage prune --category node-modules-stale --path <checkout> --apply

​

Per-OS commands

​

macOS (APFS)

diskutil apfs list | grep -E 'Capacity (In Use|Not Allocated)'
df -h /System/Volumes/Data      # the Data volume holds user files
xcrun simctl list devices booted
xcrun simctl shutdown all       # a booted simulator mounts ~220 GB of phantom DeviceFS size
tmutil listlocalsnapshots /     # local snapshots pin deleted blocks

​

Linux devbox or runner

df -h / /var/lib/docker
df -i /                         # inodes; ext4 tables are fixed at mkfs
docker system df
journalctl --disk-usage

​

Ownership

​

Cross-orchestrator contract

• missing telemetry stays UNKNOWN;
• one cleanup lease owns each host’s reclaim (implemented here per storage authority, one OS user’s home; a host-wide authority shared across OS users is not implemented yet);
• automatic reclaim is limited to eligible regenerable caches and completed, unowned outputs, with receipts;
• insufficient reclaim pauses heavy admission or offloads it rather than deleting more.