/admin) Health view, and SLO burn alerts. The
observability patterns page
explains the concepts; this page records the sources, schema, bounds and
delivery plan.
Inventory
Golden signals
Product analytics
Stores that hold personal data and must stay out of analytics: the log drain
requests/log_events (IP, user agent, user id, email),
meet_attendance_sessions (display name, user and guest ids),
get_power_users (user id, username) and external_provider_costs
(actor_id).
Alerting
There are no Discord, email or Mira operational alerts. Mastermind receives signed hub events at/api/v1/mastermind/push-events; the contract carries no
free text and only hub-issued item ids. The Perf Factory has no HTTP intake:
a T3 scheduled task runs the detector every 30 minutes and turns JSON
regression records into perf: cards on the Tasks Fleet board.
Design
Principles
- Aggregate on read from the provider, persist only rollups. Ingestion stores counts, sums and percentiles per service and time bucket. It never stores request rows, URLs, IPs, user agents or log text.
- One scheduler. Every collector is a Cron Control Worker processor. It inherits the ledger lease, generation fence, deadline and durable failure budget. No new Worker, cron trigger or queue.
- Fail closed. A source runs only when its job is listed in
CRON_CONTROL_ENABLED_JOBSand its database control row saysonordry_run. A missing row, an unknown mode or a stale generation means off. - Source-aware reads. Admin shows
unavailable,partialandsuppresseddistinctly from a measured zero, following the executive view.
Ingestion
Durable Object storage (
durableObjectsStorageGroups) is not read yet.
Workflow deploy markers say when a deploy finished; Worker version
annotations, which the Perf Factory detector reads, remain the source for
per-version suspects.
Every token is read-only and scoped to the account or project it reads. Values
are never logged, stored or returned; logs carry the source id, counts and the
HTTP status class only.
Bounds per run. Each source row sets max_requests_per_run (default 4,
hard cap 10). Responses are capped at 1 MiB and writes at 5,000 rollup rows
and 200 deploy markers. A Cloudflare run reads at most four 15-minute
buckets starting two buckets before the watermark, so it commits at most two
new buckets per 15-minute slot and gains one bucket per slot after an outage.
Four buckets of about 100 scripts at 12 metrics are about 4,800 rows, inside
the 5,000-row write cap. A GitHub run reads up to twelve 15-minute buckets
per 30-minute slot, one request per page of 100 runs, and moves the watermark
only past buckets it read whole. When the first bucket alone needs more pages
than the run’s budget, or more than the listing’s 1,000 results, it is stored
as a sample (runs from the listing total, sampled_runs from what was
read) and the watermark moves on, so one busy bucket cannot stall the
source. Reads stop at seven days back; anything older stays a
recorded gap. A run that reaches a cap records budget_exhausted and keeps
its watermark at the last complete bucket, and the next slot continues. There is no in-run retry. The Cloudflare
GraphQL API allows 300 queries per 5 minutes; these sources issue at most 2
per 15-minute slot.
Idempotence. Rollup rows are keyed on
(source, service, metric, bucket_seconds, bucket_start) and upserted with
the newest observation, so a repeated or overlapping slot converges. Adaptive
analytics can revise recent buckets; the Cloudflare collector re-reads the
last two closed buckets each run.
Stop fence. Each source has its own Cron Control job, so
/control/jobs/{id}/stop stops exactly that source. The database adds a
second fence: private.observability_source_control holds one row per source
with mode (off, dry_run, on), max_requests_per_run, a daily row
budget and a generation that any mode or budget change increments. This
generation is separate from the Cron Control ledger generation. Rows start
off. The processor reads the row at start, checks the Cron Control fence
before every provider request and before the write, and passes the database
generation to the write RPC, which refuses a stale generation, a source that
is not on, and a write past the day’s row budget. dry_run performs the
reads and records counts without writing rollups. A failed or unknown run
counts toward Cron Control’s consecutive-failure budget, which stops the job
after five until an operator resumes it.
Cost envelope. Each source writes at most its daily row budget:
150,000 rows for cloudflare-workers and 40,000 for github-actions.
Expected volume is about 115,000 and 12,000 rows a day: about 100 scripts ×
12 metrics × 96 buckets (10 Worker metrics plus Durable Object requests and
errors), and about 20 active workflows per bucket × 6 metrics (runs,
failed_runs, duration and queue p50/p95) × 96 buckets. Both sources write
15-minute rollups, kept for 90 days (hourly rollups, which no source writes
yet, would be kept for 400 days), so the store is capped at about 17.1 M rows
((150,000 + 40,000) × 90) and expected near 11.4 M. Collectors run 144 times a day, plus 48 product analytics
and 144 retention runs: about 340 Worker invocations and under 700 PostgREST
calls a day. prune_observability_rollups runs in its own
observability-retention job, at most 1,000 rows per table every 10 minutes
(144,000 a day, above the expected daily expiry of about 127,000). GraphQL Analytics and the
GitHub read API are not billed per call.
Schema
All tables live inprivate and have RLS enabled with no policies. Clients
have no table privileges; Cron Control reaches them only through SECURITY DEFINER RPCs executable by service_role. Admin reads through
/api/v1/infrastructure/ routes that check view_infrastructure.
The cost discovery work shares the scheduler, the source-control and ingest-run
tables, and the Cloudflare read client. Its billing facts (minor units,
currency, statement identities) stay in their own tables.
Product analytics and privacy
Product analytics follows the data protection standard:- Aggregates only.
private.rollup_product_analytics_daily(p_limit)runs inside Postgres as a Cron Control maintenance RPC every 30 minutes and writes(day, app, metric, dimension, value)for the current and previous UTC day. Session-based counts keep the highest value seen for the day, becauseauth.sessionsrows move forward or disappear after the day ends. It never copies a user, guest, workspace or meeting id, a name or free text into the analytics store. - Small cells are suppressed whole. A cell whose cohort has 1 to 9
people is stored as
suppressed = truewith a null value for every metric in it, including sums such as minutes, tokens and credits. Zero stays a measured zero. - Complements are suppressed. AI features with fewer than 10 people are
merged into an
othercell. Whenotheritself has 1 to 9 people, it and the all-feature total are both suppressed, so no small cell can be derived by subtraction. Per-app activity counts will follow the same rule against the platform totals. - Retention. Daily aggregates are kept for 25 months. They are treated as personal data with that bounded schedule until anonymization is verified; they are not claimed as anonymized aggregates.
- Inputs keep their own clocks. The rollup reads source tables inside their existing retention (for example Meet attendance at 90 days) and does not extend it.
Purpose and basis.
product_analytics_daily serves one purpose:
aggregate product health for platform operators with view_infrastructure.
The intended basis is legitimate interest. It holds no subject-level rows, so
a subject export has nothing to return from it, and an erasure does not
recompute closed days, whose counts cannot be traced to a person. The
observability-product-analytics job stays out of CRON_CONTROL_ENABLED_JOBS
until the legal owner confirms this record.
Per-app activity. No store records which app a person used. The planned
writer marks activity once per person, app and UTC day from the satellite
session resolver. It stores HMAC(key(day, app), user_id), with one key per
day and app that lives until that UTC day closes, so marks cannot be joined
across apps or days. Runs during the day only count marks. The first run
after the day closes writes the final count, then deletes that day’s marks
and key in the same transaction. Before inserting, the writer checks the
account-level usage analytics preference and pending erasure; turning the
preference off deletes that person’s marks for the open day. The writer needs
the legal owner’s confirmation of its basis and its own privacy review,
because a keyed hash is personal data while the key exists.
Admin Health view
/admin → Monitoring → Health (/monitoring/health) shows one row per app
and Worker: request rate, error rate, p95 latency, CPU p99, DO errors, and the
latest deploy, each with a status against its SLO. Selecting a service opens a
drill-down with 15-minute series for 24 hours, 7 days or 30 days, deploy
markers drawn on every chart, and the product analytics for that app. Every
value carries its source state, and nothing outside view_infrastructure can
read it.
SLOs and alerts
private.observability_slos names a service (a Worker script, or * for
every script with traffic in the last six hours), an indicator and an
objective. The defaults are workers-availability (99.5% of requests
without errors) and workers-latency (99% of requests in 15-minute buckets
whose p95 stays at or under 1,000 ms). Bucket quantiles approximate the
request-level latency indicator. The observability-slo-burn job runs
private.evaluate_observability_slos every 15 minutes, at most 500 SLO and
service pairs per pass, and evaluates multi-window burn rates (bad share
divided by the error budget):
A window with fewer than 50 requests, or with more than a quarter of its
buckets missing, is
unavailable: it neither fires, extends nor resolves an
alert. An open alert resolves after two consecutive healthy evaluations.
Alerts are keyed by SLO, severity, service and start bucket, and resolved
alerts are kept for 90 days. Each transition (fired, resolved) is delivered
once, at most 20 per pass:
- Owner notification through the shared
private.send_root_operator_alertRPC (shared with cost alerts), which dedupes per key and callspublic.create_notificationfor root workspace members holdingview_infrastructure. This reaches the web inbox, email and push, honoring each person’s notification preferences; page burns areurgent. A delivery that creates no notification is retried on the next pass. Mira sees the same notification; there is no separate Mira channel. - Admin through
GET /api/v1/infrastructure/observability/alerts, which returns firing and recently resolved alerts to a root member withview_infrastructure(session or bearer token). The alert type and parser are shared from@tuturuuu/utils/observability/alerts. - Mastermind fleet sync (see
Fleet sync): Web reads
firing alerts with their SLO indicator server-side with the admin client
(filtered, ordered and limited in the query), memoized per isolate for
60 seconds (15 seconds after a failure), and serves firing alerts only
(at most 20, pages first, then newest) to the owner’s phone on
GET /api/v1/mastermind/fleet/stateand to the orchestrator Mac on the HMAC-signedGET /api/v1/mastermind/fleet/alerts. The Mac never holds an infrastructure token. The orchestrator relays each new page burn to the owner’s ask channel once per alert id; tickets and resolved alerts raise no ask. The push-event contract stays unchanged. - Perf Factory through the same route’s
regressions: firing alerts in the detector’s record shape (id,app,worker,metric,startTime,detectedAt,baseline,trigger,observed,windows,baselineWindows,rule,suspects,versionSummary,queries) withsource: "observability-slo".scripts/perf/perf-observability-alerts.mjswrites them for the dispatcher, which fillssuspectsandversionSummaryfrom Worker version annotations with the detector’srerunbefore intake.
Delivery plan
The
cloudflare-gateway, Vercel and Supabase collectors and the per-app
activity writer follow once their tokens are provisioned and their reviews
pass. The Perf Factory detector covers gateway mounts until then.
Operating
- Enable a source: provision its token, set its control row to
dry_run, add the job toCRON_CONTROL_ENABLED_JOBS, check the run log, then seton. - Stop a source now: stop its job with
/control/jobs/{id}/stop. The collector checks that fence before every provider request and before the write. Setting the control row tooffalso stops it: a running collector finishes its in-flight read, and the write is refused. - Investigate a missing series: read
observability_ingest_runsfor the source (budget_exhausted,failed,unknown) before reading provider dashboards.