Skip to main content
This page is the inventory and design for platform observability: golden signals per app and Worker, privacy-preserving product analytics, the Admin (/admin) Health view, and SLO burn alerts. The observability patterns page explains the concepts; this page records the sources, schema, bounds and delivery plan.

Inventory

Golden signals

Product analytics

Stores that hold personal data and must stay out of analytics: the log drain requests/log_events (IP, user agent, user id, email), meet_attendance_sessions (display name, user and guest ids), get_power_users (user id, username) and external_provider_costs (actor_id).

Alerting

There are no Discord, email or Mira operational alerts. Mastermind receives signed hub events at /api/v1/mastermind/push-events; the contract carries no free text and only hub-issued item ids. The Perf Factory has no HTTP intake: a T3 scheduled task runs the detector every 30 minutes and turns JSON regression records into perf: cards on the Tasks Fleet board.

Design

Principles

  • Aggregate on read from the provider, persist only rollups. Ingestion stores counts, sums and percentiles per service and time bucket. It never stores request rows, URLs, IPs, user agents or log text.
  • One scheduler. Every collector is a Cron Control Worker processor. It inherits the ledger lease, generation fence, deadline and durable failure budget. No new Worker, cron trigger or queue.
  • Fail closed. A source runs only when its job is listed in CRON_CONTROL_ENABLED_JOBS and its database control row says on or dry_run. A missing row, an unknown mode or a stale generation means off.
  • Source-aware reads. Admin shows unavailable, partial and suppressed distinctly from a measured zero, following the executive view.

Ingestion

Durable Object storage (durableObjectsStorageGroups) is not read yet. Workflow deploy markers say when a deploy finished; Worker version annotations, which the Perf Factory detector reads, remain the source for per-version suspects. Every token is read-only and scoped to the account or project it reads. Values are never logged, stored or returned; logs carry the source id, counts and the HTTP status class only. Bounds per run. Each source row sets max_requests_per_run (default 4, hard cap 10). Responses are capped at 1 MiB and writes at 5,000 rollup rows and 200 deploy markers. A Cloudflare run reads at most four 15-minute buckets starting two buckets before the watermark, so it commits at most two new buckets per 15-minute slot and gains one bucket per slot after an outage. Four buckets of about 100 scripts at 12 metrics are about 4,800 rows, inside the 5,000-row write cap. A GitHub run reads up to twelve 15-minute buckets per 30-minute slot, one request per page of 100 runs, and moves the watermark only past buckets it read whole. When the first bucket alone needs more pages than the run’s budget, or more than the listing’s 1,000 results, it is stored as a sample (runs from the listing total, sampled_runs from what was read) and the watermark moves on, so one busy bucket cannot stall the source. Reads stop at seven days back; anything older stays a recorded gap. A run that reaches a cap records budget_exhausted and keeps its watermark at the last complete bucket, and the next slot continues. There is no in-run retry. The Cloudflare GraphQL API allows 300 queries per 5 minutes; these sources issue at most 2 per 15-minute slot. Idempotence. Rollup rows are keyed on (source, service, metric, bucket_seconds, bucket_start) and upserted with the newest observation, so a repeated or overlapping slot converges. Adaptive analytics can revise recent buckets; the Cloudflare collector re-reads the last two closed buckets each run. Stop fence. Each source has its own Cron Control job, so /control/jobs/{id}/stop stops exactly that source. The database adds a second fence: private.observability_source_control holds one row per source with mode (off, dry_run, on), max_requests_per_run, a daily row budget and a generation that any mode or budget change increments. This generation is separate from the Cron Control ledger generation. Rows start off. The processor reads the row at start, checks the Cron Control fence before every provider request and before the write, and passes the database generation to the write RPC, which refuses a stale generation, a source that is not on, and a write past the day’s row budget. dry_run performs the reads and records counts without writing rollups. A failed or unknown run counts toward Cron Control’s consecutive-failure budget, which stops the job after five until an operator resumes it. Cost envelope. Each source writes at most its daily row budget: 150,000 rows for cloudflare-workers and 40,000 for github-actions. Expected volume is about 115,000 and 12,000 rows a day: about 100 scripts × 12 metrics × 96 buckets (10 Worker metrics plus Durable Object requests and errors), and about 20 active workflows per bucket × 6 metrics (runs, failed_runs, duration and queue p50/p95) × 96 buckets. Both sources write 15-minute rollups, kept for 90 days (hourly rollups, which no source writes yet, would be kept for 400 days), so the store is capped at about 17.1 M rows ((150,000 + 40,000) × 90) and expected near 11.4 M. Collectors run 144 times a day, plus 48 product analytics and 144 retention runs: about 340 Worker invocations and under 700 PostgREST calls a day. prune_observability_rollups runs in its own observability-retention job, at most 1,000 rows per table every 10 minutes (144,000 a day, above the expected daily expiry of about 127,000). GraphQL Analytics and the GitHub read API are not billed per call.

Schema

All tables live in private and have RLS enabled with no policies. Clients have no table privileges; Cron Control reaches them only through SECURITY DEFINER RPCs executable by service_role. Admin reads through /api/v1/infrastructure/ routes that check view_infrastructure. The cost discovery work shares the scheduler, the source-control and ingest-run tables, and the Cloudflare read client. Its billing facts (minor units, currency, statement identities) stay in their own tables.

Product analytics and privacy

Product analytics follows the data protection standard:
  • Aggregates only. private.rollup_product_analytics_daily(p_limit) runs inside Postgres as a Cron Control maintenance RPC every 30 minutes and writes (day, app, metric, dimension, value) for the current and previous UTC day. Session-based counts keep the highest value seen for the day, because auth.sessions rows move forward or disappear after the day ends. It never copies a user, guest, workspace or meeting id, a name or free text into the analytics store.
  • Small cells are suppressed whole. A cell whose cohort has 1 to 9 people is stored as suppressed = true with a null value for every metric in it, including sums such as minutes, tokens and credits. Zero stays a measured zero.
  • Complements are suppressed. AI features with fewer than 10 people are merged into an other cell. When other itself has 1 to 9 people, it and the all-feature total are both suppressed, so no small cell can be derived by subtraction. Per-app activity counts will follow the same rule against the platform totals.
  • Retention. Daily aggregates are kept for 25 months. They are treated as personal data with that bounded schedule until anonymization is verified; they are not claimed as anonymized aggregates.
  • Inputs keep their own clocks. The rollup reads source tables inside their existing retention (for example Meet attendance at 90 days) and does not extend it.
Purpose and basis. product_analytics_daily serves one purpose: aggregate product health for platform operators with view_infrastructure. The intended basis is legitimate interest. It holds no subject-level rows, so a subject export has nothing to return from it, and an erasure does not recompute closed days, whose counts cannot be traced to a person. The observability-product-analytics job stays out of CRON_CONTROL_ENABLED_JOBS until the legal owner confirms this record. Per-app activity. No store records which app a person used. The planned writer marks activity once per person, app and UTC day from the satellite session resolver. It stores HMAC(key(day, app), user_id), with one key per day and app that lives until that UTC day closes, so marks cannot be joined across apps or days. Runs during the day only count marks. The first run after the day closes writes the final count, then deletes that day’s marks and key in the same transaction. Before inserting, the writer checks the account-level usage analytics preference and pending erasure; turning the preference off deletes that person’s marks for the open day. The writer needs the legal owner’s confirmation of its basis and its own privacy review, because a keyed hash is personal data while the key exists.

Admin Health view

/admin → Monitoring → Health (/monitoring/health) shows one row per app and Worker: request rate, error rate, p95 latency, CPU p99, DO errors, and the latest deploy, each with a status against its SLO. Selecting a service opens a drill-down with 15-minute series for 24 hours, 7 days or 30 days, deploy markers drawn on every chart, and the product analytics for that app. Every value carries its source state, and nothing outside view_infrastructure can read it.

SLOs and alerts

private.observability_slos names a service (a Worker script, or * for every script with traffic in the last six hours), an indicator and an objective. The defaults are workers-availability (99.5% of requests without errors) and workers-latency (99% of requests in 15-minute buckets whose p95 stays at or under 1,000 ms). Bucket quantiles approximate the request-level latency indicator. The observability-slo-burn job runs private.evaluate_observability_slos every 15 minutes, at most 500 SLO and service pairs per pass, and evaluates multi-window burn rates (bad share divided by the error budget): A window with fewer than 50 requests, or with more than a quarter of its buckets missing, is unavailable: it neither fires, extends nor resolves an alert. An open alert resolves after two consecutive healthy evaluations. Alerts are keyed by SLO, severity, service and start bucket, and resolved alerts are kept for 90 days. Each transition (fired, resolved) is delivered once, at most 20 per pass:
  • Owner notification through the shared private.send_root_operator_alert RPC (shared with cost alerts), which dedupes per key and calls public.create_notification for root workspace members holding view_infrastructure. This reaches the web inbox, email and push, honoring each person’s notification preferences; page burns are urgent. A delivery that creates no notification is retried on the next pass. Mira sees the same notification; there is no separate Mira channel.
  • Admin through GET /api/v1/infrastructure/observability/alerts, which returns firing and recently resolved alerts to a root member with view_infrastructure (session or bearer token). The alert type and parser are shared from @tuturuuu/utils/observability/alerts.
  • Mastermind fleet sync (see Fleet sync): Web reads firing alerts with their SLO indicator server-side with the admin client (filtered, ordered and limited in the query), memoized per isolate for 60 seconds (15 seconds after a failure), and serves firing alerts only (at most 20, pages first, then newest) to the owner’s phone on GET /api/v1/mastermind/fleet/state and to the orchestrator Mac on the HMAC-signed GET /api/v1/mastermind/fleet/alerts. The Mac never holds an infrastructure token. The orchestrator relays each new page burn to the owner’s ask channel once per alert id; tickets and resolved alerts raise no ask. The push-event contract stays unchanged.
  • Perf Factory through the same route’s regressions: firing alerts in the detector’s record shape (id, app, worker, metric, startTime, detectedAt, baseline, trigger, observed, windows, baselineWindows, rule, suspects, versionSummary, queries) with source: "observability-slo". scripts/perf/perf-observability-alerts.mjs writes them for the dispatcher, which fills suspects and versionSummary from Worker version annotations with the detector’s rerun before intake.

Delivery plan

The cloudflare-gateway, Vercel and Supabase collectors and the per-app activity writer follow once their tokens are provisioned and their reviews pass. The Perf Factory detector covers gateway mounts until then.

Operating

  • Enable a source: provision its token, set its control row to dry_run, add the job to CRON_CONTROL_ENABLED_JOBS, check the run log, then set on.
  • Stop a source now: stop its job with /control/jobs/{id}/stop. The collector checks that fence before every provider request and before the write. Setting the control row to off also stops it: a running collector finishes its in-flight read, and the write is refused.
  • Investigate a missing series: read observability_ingest_runs for the source (budget_exhausted, failed, unknown) before reading provider dashboards.