Abuse-guard store (Durable Objects)

​

Model

​

Commands

​

Replay safety

​

Bounds

​

Partitioning

• Generic records, windows, clearance and replay fences get one partition per hashed subject key, so there is no hotspot and little storage per object.
• Password, Meet guest and Nova reservations use one partition per limiter scope, so the source and target attempt budgets reserve atomically. Before subject execution, GuardControl checks durable source/principal command quotas and charges the family in one transaction. A fairness denial never reaches the subject or allocates a dedupe row. A subject denial still spends its admitted command. The target window is anchored to its first reservation: after a successful login refunds the target to zero, the next attempt keeps the existing expiry. The legacy Lua script instead restarts a full window on that 0→1 increment. The store keeps the anchored window, which can return a fresh target allowance up to one window sooner after a refund. That is bounded and accepted, and it is covered by an engine expiry test.
• A sequential RPC runs its steps in order and skips the rest after the first denial. Callers do not rely on snapshot consistency across partitions.

​

Selecting the backend

1. Add { binding: 'ABUSE_GUARD', service: 'tuturuuu-abuse-guard', entrypoint: 'AbuseGuardEntrypoint' } to services in apps/meet/void.config.ts, or in the staging config with the staging Worker’s name.
2. Set the ABUSE_GUARD_KEY_PEPPER secret to the environment’s pepper.
3. Set ABUSE_GUARD_FAMILIES=password.

​

Cost and denial shortcuts

• Proxy: all of a caller’s windows are charged in one win.hit, which is one command. A denial is remembered per instance until its window resets, so blocked repeats cost no store operations. While the store errors, is stopped or is off, the same limits apply per instance as fixed windows, which is degraded enforcement rather than the store’s sliding windows. Healthy traffic does not seed those counters, so a store outage starts them at zero.
• Offline: on the guard store, the session windows and clearance are spent only for an actor the proxy verified (useVerifiedBearerActor). Unverified or rotating bearers cost one IP operation and get 401. Daily windows of 50,000 reads per IP and per session bound each source. A read costs at most four commands:
  • IP and session win.hit;
  • the clearance read;
  • a clearance write when a Turnstile proof is granted.
  So one source can spend up to 200,000 commands a day. That is one third of the default 600,000 offline share, not a quarter, before denial and cache-miss overhead. Size the offline share for the expected number of concurrent sources. A denied subject is remembered per instance until its reset.
• Password: a subject reservation denial is cached by source and target for at most one second in a bounded 10,000-entry map. This suppresses immediate repeated res.reserve calls without treating a target denial as immutable: another successful login can refund the target. Expiry, eviction, concurrent requests and caller restarts can cause another probe; each such admitted probe spends the durable identity allowance below.
• Replay: once replay is assigned, a token proven replayed outside its grace period is answered from memory until it expires. After a lost concurrent claim, the route rereads the winner’s timestamp and applies grace without caching. Each instance pre-limits refresh attempts to 20 per minute per principal and 60 per minute per IP before any store call; excess attempts get 429 with Retry-After: 60.
• Skipped steps: every RPC conservatively reserves the whole requested batch, including skipped steps and interrupted execution. Family and identity command usage is never refunded. Reservations commit before subject execution and survive isolate replacement; subject operation IDs independently deduplicate mutations.
• Cache limits: every in-memory map (replay, denials, outage counters) holds at most 10,000 entries and evicts the oldest first. All of them are per instance, so restarts, new instances and eviction under pressure reset them. The zero-cost claims hold only while an entry stays resident; the store, or the fixed-window outage limiter, still enforces the limits.

​

Durable command fairness

​

Activation checklist (per environment)

1. Publish the Worker and run its staging canary (separate admission).
2. Proxy: measure staging and production API volume first. Size the proxy share of ABUSE_GUARD_FAMILY_BUDGETS so a normal day uses at most 50% of it. Do not assign proxy without that measurement.
3. Offline — gate (unmet): verified admission is not yet wired on every marked-read path. Today the opt-in has gaps:
   • Calendar and Infrastructure pass a verified actor only on their calendar-preference routes. HEAD preference requests miss the opt-in.
   • Web marked reads have no verified-actor callback.
   • Cookie-only and app-coordination-token sessions never produce a verified actor.
   Under the guard store, every such request gets 401 even with valid credentials. Before assigning offline:
   • enumerate every hop that serves marked reads, and every credential type it accepts;
   • wire or implement the verified-actor path for each;
   • prove end to end that authenticated reads succeed and that forged or opaque bearers are denied;
   • run a staging drill showing that admitted traffic from realistic source fan-out cannot exhaust the offline share. The per-source daily windows bound one source, not many.
4. Replay — R1-REPLAY-DRAIN CLOSED AS SCOPED: one verified principal or one source cannot drain replay’s family command share across instances, isolates, scopes or token keys. Atomic durable admission and the concurrency, restart and cache-pressure evidence support this closure. It does not prove that a family stop cannot deny every caller. HIGH activation hold FAIR-R1-SYBIL-DRAIN remains OPEN: 20 accounts plus 20 sources can consume the share. Do not assign replay until the successor fairness holds are resolved or receive an authorized scoped risk disposition.
5. Password — PW-R1-DRAIN CLOSED AS SCOPED: one source or one verified Nova user cannot drain password’s family command share. Denied subject probes and successful refunds count; the bounded one-second denial cache suppresses repeated reservations. Anonymous Meet has only the source axis. HIGH activation hold FAIR-R1-SYBIL-DRAIN remains OPEN: rotating source prefixes can still drain the share and deny all Meet and Nova password checks. The closure does not establish one-actor fairness or family-wide availability. Both families remain unassigned. The successor HIGH/P2 holds below and the existing hosted stop/resume, ingress identity, endpoint-to-store end-to-end admission, request-billing and full-day hosted-drill holds remain open. Local tests and independent source review do not establish hosted acceptance.
6. Replay cutover: fence keys consumed before cutover live only in the legacy store, and CLI refresh tokens live 90 days. Choose one:
   • Read-through. Stop every writer to the legacy fence keys first, so the legacy store is frozen. Keep UPSTASH_REDIS_REST_* configured and set ABUSE_GUARD_REPLAY_LEGACY_UNTIL to the cutover time plus 90 days, as a full ISO-8601 timestamp. A key present in the legacy store counts as used, and a legacy read error fails closed. A malformed deadline, or a future deadline with either credential missing, makes the replay store unavailable, so the routes return 503 instead of skipping the check.
   • Generation bump. If no legacy store is configured, rotate the refresh-token signing generation at cutover, so tokens issued before it no longer verify. Production had no Upstash, so its fences were effectively off and this applies there. Users re-authenticate once.
   • Rollback: unassigning replay returns the routes to the legacy path, which does not know fences recorded in the guard store. Either bump the generation on rollback, or keep the guard store readable until its tokens expire.
7. Alerting is live (see Cost safety) and the operator stop has been exercised in staging.

​

Fairness activation holds

• HIGH — FAIR-R1-SYBIL-DRAIN (OPEN). Each identity receives budget / 20, so 20 independent identities can exhaust a family. Anonymous Meet password admission is source-only: one actor rotating 20 IPv6 /64s within one /48 can spend 5,000 commands per /64. At approximately 5 requests/second in aggregate, the default 100,000-command password share can be drained in about 5.6 hours, then its family stop denies every Meet and Nova password check until rollover or operator intervention. Replay needs 20 verified accounts plus 20 sources. This is source arithmetic, not a measured attack or hosted load result. The planned follow-up is aggregate IPv6 /48 or /56 and IPv4 /24 tiers checked atomically with the existing quotas/family spend, and/or a graceful family stop with a known-good reserve. Prove a prefix-rotation drill before activation. Neither aggregate tiers nor a reserve is implemented by this change.
• P2 — FAIR-R1-SHARED-SOURCE-LOCKOUT (OPEN). The fixed 5% daily cap also locks out legitimate shared sources: an office NAT, mobile CGNAT address or every caller when ingress resolves to unknown. At the default replay share, 5,000 commands allow about 2,500 two-command GET/claim refreshes per day across all users behind that source; actual costs vary by route and grace/reread path. A caller sharing the source can spend this allowance and deny its neighbors until UTC midnight. Operator resume preserves identity quotas, and there is currently no audited fairness reset or configurable per-axis share. Before activation, add configurable source/principal shares (including a larger source allowance for verified-principal families), an audited per-identity/per-family fairness reset and fairness-denial alerts. Measure real per-source peaks and the unknown-source fraction; unresolved ingress coverage remains a blocker.
• P2 — FAIR-R1-CONTROL-HOTSPOT (OPEN capacity hold). Every guarded request now goes through the singleton global control:v1 Durable Object for durable admission; fair requests also check per-identity quotas there. The fence.ts refresh path was removed, so every request reaches the global control object and its serialized storage work. All families share this capacity; a high request rate or fresh-identity flood can make every family unavailable with 503s and concentrates storage cost on one object. Separate family budgets do not isolate this capacity. Keep every family unassigned until a load test characterizes the global hotspot and cost, a sharding design preserves cross-family stop and quota atomicity, and the HIGH Sybil drain hold above is addressed. Kill-switch authority stays with designated operational roles.
• P3 — FAIR-R1-PEPPER-SCOPE (OPEN). Quotas are scoped by ABUSE_GUARD_KEY_PEPPER. Different peppers on Web, Meet or Nova create separate allowances for the same source/principal. All callers sharing a family and environment must use the same pepper; a dedicated shared fairness pepper would require follow-up implementation. Rotating the current pepper resets all quotas immediately, including mid-day, as well as the other keyed state. Treat rotation as an explicit reset requiring acceptance, not a routine lockout remedy.

• A batch larger than the remaining family allowance can return guard_stopped while the durable family is not stopped. Do not infer the stored stop flag from that response alone; inspect control state. This is a telemetry limitation.
• Control cleanup purges at most 1,000 quota rows per wake and re-arms after one second when expired rows remain. The loop is bounded and terminal, but alarm requests, row deletes and scheduling operations belong in the request-cost envelope below.
• A stolen, spent but unexpired refresh token can charge the victim principal’s quota before a local replay-cache hit suppresses another lookup. For a token proven replayed outside grace, this is normally one command per instance; eviction/restart can add probes. This is a low-impact residual, not a token-mint bypass or evidence of family-wide availability.

​

Cost safety

• Bounds: the limits above are enforced on the client and again in the Worker before any Durable Object call. Each RPC makes at most one subject call per step. There are no retries anywhere.
• Stop fence: POST /control/stop with Authorization: Bearer $ABUSE_GUARD_OPERATOR_TOKEN persists a stop. Every RPC then returns 503 guard_stopped before any subject call. Stopped requests spend no subject commands, but Worker/control requests remain billable. Every admitted RPC consults the durable fence before executing subjects. POST /control/resume clears it, and GET /control/state reads it. Stored data is preserved.
• Budget: each family has its own daily allowance and budget-stop flag. Exhausting one allowance does not set another family’s flag, but the shared control DO can still make other families’ admissions unavailable under overload (FAIR-R1-CONTROL-HOTSPOT). Unauthenticated marked-bulk traffic spends only offline’s share. All RPCs reserve commands durably before subject execution, without isolate buffering or overshoot. At 80% of a family’s budget the control object logs a warning once. When a family exhausts its budget, the control object persists a stop for that family only. The stop survives restarts and clears at the next UTC day, so the daily command cap holds without one flood becoming an open-ended outage. The cap applies per calendar UTC day, not per rolling 24 hours, so a burst across UTC midnight can admit up to about twice a family’s budget within a short span. An operator resume clears it earlier. Admission rejections spend no commands; admitted subject denials do count. No RPC can reserve beyond the remaining family command allowance. Each caller of a stopped family applies its own fail policy. An invalid budget configuration makes every RPC return 503 guard_unavailable, the same as a bad secret. That covers a malformed total, an unknown or duplicate family, a malformed value, or entries above the total.
• Alerting: alert on the daily operation budget at warning and the daily operation budget exhausted error, filtered per family. A stop at 10:00 GMT+7 lasts until 07:00 GMT+7, the next UTC midnight, unless an operator first checks GET /control/state and then resumes or stops globally.
• Subject alarm failures: a failed cleanup is logged and re-armed with capped exponential backoff (1, 2, 4, 8 minutes). After five consecutive failures it stops until the next write; lazy expiry keeps reads correct meanwhile.
• Alarms: each partition keeps one alarm at its latest expiry, moved only when a write extends it by more than 60 seconds. Cleanup purges at most 1,000 rows per wake. An empty partition is wiped with deleteAll() and stores nothing. A still-warm instance recreates its tables before its next use, and no abort() is needed. It is never re-armed; otherwise it re-arms once at a future expiry. Cleanup makes no network calls, and it logs errors instead of throwing so automatic alarm retries cannot loop. Without new writes, a partition’s alarms end within its longest TTL. Control quota cleanup instead purges at most 1,000 rows per wake and re-arms after one second while expired rows remain. Count every such wake, delete and scheduling operation in the cost envelope; a bounded terminal loop is not free.
• Ingress: when the owner attaches a hostname, add a Cloudflare WAF rate-limit rule for it, such as per-IP requests to /v1/rpc and /control/*. It sheds unauthenticated floods before Worker billing. Family budgets cap admitted subject commands; total request billing remains an activation hold.
• Accounting limits / open request-cost hold: the budget counts subject commands, not all billed operations. A fair admission cache miss makes a control DO request, even when rejected. Cache eviction, restarts and multiple instances multiply those probes; their request billing has no demonstrated total cap. Denied probes spend no subject commands, but must not be described as free or protected by a hard dollar cap. A family-command quota cannot cap an unbounded stream of Worker or control-DO requests. Admission/WAF sizing and a separately accepted request-cost envelope remain activation holds. Each accepted fair RPC writes up to two quota rows plus the control state, in addition to subject/dedupe rows, deletes and alarm operations. Each accepted non-fair RPC also makes one control request and durably writes the reservation before executing subjects. Stops are checked on every admission. Storage is bounded by 2 KiB × 120 days × the daily budget. See the adoption guidance.

​

Pricing envelope

​

Local verification

cd apps/abuse-guard
ttr resources run -- bunx vitest run --maxWorkers=2

• p50/p95/p99 latency from sin1, measured with a bounded number of RPCs;
• a hosted stop and resume;
• the observed request and row-write counts.