> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tuturuuu.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability & Monitoring

> How Tuturuuu observes the platform today, and the patterns that guide where it is heading

Observability gives you the ability to understand system behavior by examining
its outputs—logs, request records, and metrics. This page explains the core
observability patterns and, for each one, describes **what Tuturuuu actually
implements today** versus what is conceptual or aspirational.

<Note>
  **Observability vs Monitoring**: Monitoring tells you when something is wrong.
  Observability tells you why it's wrong and how to fix it. Tuturuuu leans on a
  self-hosted, Postgres-backed log drain rather than a third-party observability
  vendor.
</Note>

<Warning>
  **Ground-truth scope.** Tuturuuu's apps are conventional Next.js App Router and
  TanStack Start apps plus a Rust backend—not a hexagonal microservice mesh.
  There is no OpenTelemetry pipeline, no `@vercel/otel`, no
  `@tuturuuu/logging`/`@tuturuuu/observability` package, and no event-sourcing
  store. Where this page describes a pattern Tuturuuu has not adopted, it says so
  plainly and keeps the pattern only for its teaching value.
</Warning>

## Architecture in Transition

The legacy `apps/web` Next.js runtime (port `7803`) is being replaced by
`apps/tanstack-web` (TanStack Start) plus `apps/backend` (Rust, port `7820`).
The observability surface described below lives mostly in `apps/web` today and
is migrating alongside the rest of the platform. See
[TanStack Start And Rust Migration](/platform/architecture/tanstack-rust-migration)
for the contract, runtime shape, and ports.

***

## 1. Request-Scoped Correlation (Not Distributed Tracing)

### The pattern

Distributed tracing assigns each request a **trace ID** that flows through every
service, with each service recording **spans** (units of work) linked by that
trace ID. This is invaluable in a deep microservice mesh where one user action
fans out across dozens of services.

### What Tuturuuu actually does

Tuturuuu does **not** run OpenTelemetry or `@vercel/otel`, and there is no
`instrumentation.ts` that registers an OTel exporter. The real
`apps/web/src/instrumentation.ts` does not install a console drain:

```typescript theme={null}
// apps/web/src/instrumentation.ts (real)
export async function register() {
  return;
}
```

Instead of cross-service spans, Tuturuuu correlates work **within a single
request** using an `AsyncLocalStorage` context that carries a generated
`requestId`. API and cron handlers run inside `withRequestLogDrain` /
`withCronLogDrain`, and every log emitted during that request is stamped with
the same `requestId`, route, deployment color/stamp, and (once known)
`userId`/`userEmail`:

```typescript theme={null}
// apps/web/src/lib/infrastructure/log-drain.ts (real shape, abridged)
export async function withRequestLogDrain<T extends Response>(
  options: RequestDrainOptions,
  handler: () => Promise<T>
) {
  // Runs handler inside an AsyncLocalStorage context.
  // Retained request/cron events are persisted with a shared requestId.
}
```

When the handler finishes, the request and its captured `log_events` are written
to Postgres in one transaction, so you can later query every log line for a
single `requestId`. This gives per-request correlation and timing without an OTel
backend—but it does **not** propagate a trace across the `apps/web` → Rust
backend boundary. True cross-service tracing remains aspirational.

<Note>
  **Takeaway:** Correlate by `requestId`, not by OTel spans. There is no
  `getActiveTrace()`, `trace.span()`, or `io.getSpan()` API in this codebase.
</Note>

***

## 2. Centralized Logging via the Log Drain

### The pattern

In a distributed system, scattered per-instance log files are impractical.
Centralized logging streams structured logs to a unified store where they can be
searched, filtered, and correlated across boundaries.

### What Tuturuuu actually does

This pattern **is partially implemented**. `apps/web/src/lib/infrastructure/log-drain.ts`
is a self-hosted Postgres store for retained request, cron, deployment, and
usage events. Server runtime logs use the native console method that matches
severity; do not add `serverLogger` runtime imports or automatic console
log-drain installation.
Captured retained events are persisted to dedicated tables:

| Table          | Holds                                                        |
| -------------- | ------------------------------------------------------------ |
| `requests`     | One row per request/cron run: method, path, status, duration |
| `log_events`   | Individual log lines, linked to a `request_id`               |
| `cron_runs`    | Cron execution outcomes (success/failed, http status)        |
| `usage_events` | Numeric usage/metric samples (`metric`, `value`, `unit`)     |
| `deployments`  | Blue/green deployment metadata used to stamp logs            |

Each event carries `level`, `message`, `route`, `requestId`, deployment
color/stamp, optional `userId`/`userEmail`, and a JSON `metadata` bag. The drain
redacts bearer tokens and `token`/`key`/`secret`/`password` query params before
persisting, and is designed to **never throw into the caller**—logging failures
are swallowed.

```typescript theme={null}
console.info('Payment processing started', { workspaceId });
console.error('Payment failed', error);
```

Configuration is environment-driven (referenced by name, not value):

* `PLATFORM_LOG_DRAIN_ENABLED` — set to `0`/`false`/`off` to disable.
* `PLATFORM_LOG_DRAIN_DATABASE_URL` — Postgres connection for the drain.
* `PLATFORM_LOG_DRAIN_RAW_RETENTION_DAYS` — raw `log_events` retention
  (default 30 days).
* `PLATFORM_LOG_DRAIN_SUMMARY_RETENTION_DAYS` — summary table retention
  (default 90 days).
* `PLATFORM_BLUE_GREEN_COLOR`, `PLATFORM_DEPLOYMENT_STAMP`,
  `PLATFORM_PROJECT_ID` — deployment stamping.

The drain also prunes old rows on write, so retention is enforced without an
external job.

### Querying logs

Logs are read back through the internal observability API under
`apps/infrastructure/src/app/api/v1/infrastructure/observability/*` (for example `logs`,
`requests`, `cron-runs`, `deployments`, `overview`, `resources`, `analytics`),
which power the in-app infrastructure/observability dashboard. There is no
third-party log platform (no Datadog/ELK) in the loop.

***

## 3. Metrics and Usage Telemetry

### The pattern

Per-service metrics—request rates, error rates, latency percentiles, resource
utilization—let you detect and localize failures quickly and scale individual
services on their own load.

### What Tuturuuu actually does

Tuturuuu does **not** run Prometheus, a `metrics.counter()/histogram()/gauge()`
client, or a `@tuturuuu/observability` package. Instead, metrics are derived
from the same Postgres-backed drain:

* The `requests` table already encodes per-request **status** and
  **duration\_ms**, so request rate, error rate, and latency are computed by
  aggregating that table over a time window.
* The `usage_events` table stores arbitrary numeric samples
  (`metric`, `value`, `unit`)—the place to record product/business telemetry
  such as tokens consumed or workspaces created.
* `apps/web/src/lib/infrastructure/observability.ts` aggregates these (and
  container resource metrics like CPU percent / memory bytes) for the
  observability `overview`, `analytics`, and `resources` endpoints.

So per-service Prometheus metrics are **conceptual/aspirational** here; the
implemented reality is SQL aggregation over request, cron, and usage rows. If you
need a new metric, write `usage_events` rows through the infrastructure layer
rather than reaching for a Prometheus client that does not exist.

***

## 4. Health Checks

### The pattern

Services expose standardized health endpoints. Orchestrators probe them and stop
routing to—or replace—instances that fail, enabling self-healing deployments.
A common refinement splits **liveness** ("is the process running?") from
**readiness** ("is it ready to serve traffic?").

### What Tuturuuu actually does

The `apps/web` health endpoint is intentionally trivial—it does **not** probe
the database, Supabase, or Trigger.dev, and there are no `liveness`/`readiness`
sub-routes:

```typescript theme={null}
// apps/web/src/app/api/health/route.ts (real, complete)
import { connection, NextResponse } from 'next/server';

export async function GET() {
  await connection();

  return NextResponse.json(
    { status: 'ok' },
    {
      headers: {
        'Cache-Control': 'no-store',
      },
    }
  );
}
```

The Rust backend (`apps/backend`) goes further and **does** separate liveness
from readiness:

* `GET /api/health` → `{ "status": "ok" }` (parity with the web app, `no-store`).
* `GET /healthz` → liveness, returning service name, runtime (`"rust"`), and
  environment.
* `GET /readyz` → readiness; returns `200` only when the backend is fully
  configured (for example, `BACKEND_INTERNAL_TOKEN` is present), otherwise `503`.

These are simple HTTP probes suitable for container orchestration and the
blue/green deployment watcher. The Kubernetes `livenessProbe`/`readinessProbe`
YAML shown in earlier revisions of this page is illustrative only—Tuturuuu's
deployment model is the blue/green container watcher, not raw Kubernetes pods.

***

## 5. Scheduled Background Work (Not Event Sourcing)

### The pattern

Event sourcing persists an immutable, ordered history of domain events and
rebuilds state by replaying them—enabling audit trails, time-travel debugging,
and projection rebuilds.

### What Tuturuuu actually does

Tuturuuu does **not** implement event sourcing. There is no event store,
no `eventStore.append/query`, and no `rebuildProjection`. Background work runs on
**Trigger.dev v4** (`@trigger.dev/sdk` `^4.4.5`) using the `task()` API—not the
removed v2 `client.defineJob` / `eventTrigger` / `io.runTask` APIs.

```typescript theme={null}
// packages/trigger/src/schedule-tasks.ts (real)
import { task } from '@trigger.dev/sdk/v3';
import { schedulableTasksHelper } from './schedule-tasks-helper';

export const scheduleTask = task({
  id: 'schedule-task',
  queue: {
    concurrencyLimit: 10,
  },
  run: async (payload: { ws_id: string }) => {
    try {
      const result = await schedulableTasksHelper(payload.ws_id);

      if (!result.success) {
        throw new Error(result.error || 'Schedule tasks failed');
      }

      return { ws_id: payload.ws_id, ...result, success: true };
    } catch (error) {
      console.error(`[${payload.ws_id}] Error in schedule task:`, error);
      return {
        ws_id: payload.ws_id,
        success: false,
        error: error instanceof Error ? error.message : 'Unknown error',
      };
    }
  },
});
```

The historical-insight need that event sourcing addresses is partially met by
the log drain's `requests`, `cron_runs`, and `log_events` tables: cron and
request history is queryable for a retention window, and cron outcomes are
recorded as success/failure with duration and HTTP status. That is operational
history, **not** a replayable domain-event log.

***

## What Is Implemented vs. Conceptual

| Capability                 | Status in Tuturuuu | Reality                                                                |
| -------------------------- | ------------------ | ---------------------------------------------------------------------- |
| Request-scoped correlation | Implemented        | `AsyncLocalStorage` `requestId` in the log drain                       |
| Distributed tracing (OTel) | Aspirational       | No OTel / `@vercel/otel`; no cross-service spans                       |
| Centralized logging        | Partial            | Native `console.*` plus retained Postgres request/cron/deployment data |
| Per-service metrics        | Partial            | SQL aggregation over `requests`/`usage_events`; no Prometheus          |
| Health checks              | Implemented        | Web `/api/health`; backend `/healthz` + `/readyz`                      |
| Self-healing orchestration | Partial            | Blue/green watcher, not Kubernetes pods                                |
| Event sourcing / replay    | Not implemented    | Trigger.dev v4 `task()` scheduled jobs + cron history                  |

## Observability Best Practices Here

1. **Use native server logs.** In API, cron, and infrastructure runtime code,
   log diagnostics with the native console method that matches severity. Include
   structured metadata in the same call so retained platform views and deployment
   logs stay readable.
2. **Correlate by `requestId`.** When debugging, find the request row, then pull
   its `log_events`—that is the unit of correlation, not a trace ID.
3. **Record metrics as `usage_events`.** New product/business metrics belong in
   the usage table via the infrastructure layer, surfaced through the
   observability API.
4. **Never let logging break the caller.** The drain swallows its own errors by
   design; preserve that property in any extension.
5. **Reference config by name.** Tune retention and enablement through the
   `PLATFORM_LOG_DRAIN_*` environment variables; never commit connection values.

## Related Documentation

* [TanStack Start And Rust Migration](/platform/architecture/tanstack-rust-migration) - Frontend/backend runtime split and ports
* [Microservices Patterns](/platform/architecture/system-design/microservices-patterns) - Service design and deployment
* [Event-Driven Architecture](/platform/architecture/system-design/event-driven-architecture) - Event/queue patterns
* [Extensibility, Resilience & Scalability](/platform/architecture/system-design/extensibility-resilience-scalability) - System quality attributes
* [Security Architecture](/platform/architecture/system-design/security-architecture) - Security observability and audit trails