Observability reference¶

Metrics¶

Coxswain exposes the Prometheus endpoint at http://<admin-address>:<admin-port>/metrics (default port 8082). Series are emitted under one of three prefixes:

coxswain_proxy_* — emitted by serve proxy --shared and serve proxy --dedicated.
coxswain_controller_* — emitted by serve controller.
coxswain_discovery_* — the control-plane gRPC channel. Server-side series appear on the controller /metrics (it serves the stream); client-side series appear on each proxy /metrics (it consumes the stream).

The route Prometheus label is a stable rule identifier, not a request path. Operators reading coxswain_proxy_requests_total{route="httproute/checkout/api:0"} see the same label value on the matching access-log line's route_id field, so a Grafana → Loki/Tempo pivot is an exact join (no fuzzy host/path matching). Path patterns stay on the access log for human-readable display.

Route id formats:

HTTPRoute / GRPCRoute: httproute/<namespace>/<name>:<rule-id> / grpcroute/<namespace>/<name>:<rule-id> — rule-id is the rule's .name (HTTPRouteNamedRouteRule / GRPCRouteNamedRouteRule, GEP-995) when the operator sets one; otherwise the position in spec.rules[]. A rule's id is stable across reorders only when named — an unnamed rule's index-based id shifts if a preceding rule is added or removed.
Ingress per-rule: ingress/<namespace>/<name>:<r>.<p> — nested (rules, paths) index, mirroring the YAML structure.
Ingress spec.defaultBackend: ingress/<namespace>/<name>:default.
The controller-wide --ingress-default-backend fallback: ingress-default-backend/<service-namespace>/<service-name>.

Proxy-pod metrics (`coxswain_proxy_*`)¶

Metric	Type	Labels
`coxswain_proxy_requests_total`	Counter	`listener`, `route`, `method`, `status_code`
`coxswain_proxy_request_duration_seconds`	Histogram	`listener`, `route`
`coxswain_proxy_upstream_errors_total`	Counter	`listener`, `route`, `upstream`, `error_type` (`connect`/`timeout`/`refused`/`tls`/`5xx`/`other`)
`coxswain_proxy_upstream_retries_total`	Counter	`listener`, `route`, `upstream`, `condition` (`connect-failure`/`timeout`/`5xx`)
`coxswain_proxy_upstream_connections_total`	Counter	`state` (`new`/`reused`) — incremented once per request; `reused` climbs when upstream keepalive is effective
`coxswain_proxy_mirror_requests_total`	Counter	`route`, `upstream` — incremented once per mirror dispatch (fire-and-forget shadow requests from `RequestMirror` filters); counts attempts, not successes
`coxswain_proxy_active_upstreams`	Gauge	`upstream`
`coxswain_proxy_routing_table_hosts`	Gauge	—
`coxswain_proxy_routing_table_routes`	Gauge	`kind` (`ingress`/`gateway`)
`coxswain_proxy_routing_table_rebuilds_total`	Counter	`result` (`ok`/`error`)
`coxswain_proxy_routing_table_rebuild_duration_seconds`	Histogram	—
`coxswain_proxy_tls_certs_loaded`	Gauge	`bucket` (`exact`/`wildcard`/`default`)
`coxswain_proxy_tls_cert_expiry_seconds`	Gauge	`sni`
`coxswain_proxy_tls_handshakes_total`	Counter	`result` (`ok`/`fail`), `version`
`coxswain_proxy_connections_active`	Gauge	`listener`
`coxswain_proxy_connections_total`	Counter	`listener`
`coxswain_proxy_connection_duration_seconds`	Histogram	`listener`

The following listener-lifecycle metrics are also exposed: coxswain_proxy_listeners_active, coxswain_proxy_listener_lifecycle_total, coxswain_proxy_listener_drain_duration_seconds, coxswain_proxy_requests_force_closed_total.

coxswain_proxy_listener_lifecycle_total carries an event label ∈ {added, removed, drain_completed, drain_exceeded, bind_failed}. bind_failed counts a listener whose socket bind() failed — that local port serves no traffic until a later reconcile retries the bind. A sustained non-zero rate is a data-plane outage signal worth alerting on (a port conflict, or a stale process still holding the port); it should otherwise stay flat at zero.

The listener label is the local port the proxy accepted the connection on. For Ingress and dedicated-mode Gateway traffic this is the advertised port (e.g. 80/443). For shared-mode Gateway listeners it is the internal accept port the controller allocates per Gateway (in the 30000–32767 range), not the advertised port: a per-Gateway VIP Service maps the advertised port onto that internal port, and the proxy keys routing, TLS, and metrics on the port it accepts on so cross-Gateway hostname namespaces stay isolated (see Deployment models → per-Gateway addressing). To slice a shared-mode Gateway's series by its advertised port, join on the VIP Service rather than reading it from the listener label.

Controller-pod metrics (`coxswain_controller_*`)¶

Metric	Type	Labels
`coxswain_controller_leader`	Gauge	— (1 when this replica holds the lease)
`coxswain_controller_leader_transitions_total`	Counter	—
`coxswain_controller_reconcile_total`	Counter	`controller`, `result` (`ok`/`error`)
`coxswain_controller_reconcile_duration_seconds`	Histogram	`controller`
`coxswain_controller_reconcile_errors_total`	Counter	`controller`
`coxswain_controller_status_patch_total`	Counter	`kind`, `result` (`ok`/`error`/`conflict`)
`coxswain_controller_status_patch_duration_seconds`	Histogram	`kind`
`coxswain_controller_watch_events_total`	Counter	`kind`, `event` (`init_done`/`apply`/`delete`/`restart`)
`coxswain_controller_watch_errors_total`	Counter	`kind`
`coxswain_controller_routing_table_hosts`	Gauge	— (mirrors the proxy view; drift indicates a stale snapshot)
`coxswain_controller_routing_table_routes`	Gauge	`kind`
`coxswain_controller_routing_table_rebuilds_total`	Counter	`result`
`coxswain_controller_routing_table_rebuild_duration_seconds`	Histogram	—
`coxswain_controller_tls_certs_loaded`	Gauge	`bucket`

Discovery channel metrics (`coxswain_discovery_*`)¶

The control-plane gRPC channel between controller (server) and proxies (clients). Server-side series are scraped from the controller /metrics; client-side series from each proxy /metrics.

Metric	Side	Type	Labels
`coxswain_discovery_connected_proxies`	Server	Gauge	— (proxy streams live now; a drop to `0` means the fleet lost its control-plane link)
`coxswain_discovery_streams_total`	Server	Counter	`result` (`accepted`/`rejected`) — `rejected` covers wire-version mismatch, malformed scope, and SVID/scope-binding denial
`coxswain_discovery_acks_total`	Server	Counter	— (snapshot Acks received; tracks fleet convergence throughput)
`coxswain_discovery_client_reconnects_total`	Client	Counter	— (reconnect attempts; excludes the first connect — a climbing rate flags an unstable link)
`coxswain_discovery_client_state`	Client	Gauge	— (`0`=pending, `1`=ready, `2`=degraded; scalar mirror of the proxy health subsystem)
`coxswain_discovery_bootstrap_total`	Server	Counter	`result` (`accepted`/`rejected`), `reason` (`ok` for accepts; `wire_version`/`sa_token`/`token_review_error`/`invalid_principal`/`ca_not_ready`/`malformed_csr`/`signing_error`/`internal` for rejects) — `sum(result="rejected")` answers "is the fleet failing to bootstrap"; the `reason` split separates a misconfigured client from a controller-side fault
`coxswain_discovery_svid_issued_total`	Server	Counter	— (SVIDs the CA signed on the Bootstrap path; a flat line while SVIDs expire flags a stuck issuance path)
`coxswain_discovery_client_bootstrap_total`	Client	Counter	`result` (`success`/`failure`) — proxy-side rotation outcome; sustained `failure` means the proxy is serving its last-good SVID toward expiry
`coxswain_discovery_client_svid_expiry_seconds`	Client	Gauge	— (seconds until the proxy's current SVID `notAfter`; approaching `0` means rotation is not keeping up with the TTL)

Metric labels per Gateway¶

When you run a dedicated proxy pool, you usually want to slice its metrics by Gateway. Coxswain does not bake the Gateway identity into the emitted series — a gateway_name / gateway_namespace label on every counter would multiply request-counter cardinality across every dedicated Gateway, and the shared pool has no single Gateway to name.

Instead the identity rides in as a scrape-time target label. Each proxy pod the controller provisions carries the Gateway it serves as two pod labels:

gateway.networking.k8s.io/gateway-name — the Gateway name (the GEP-1762 well-known label).
gateway.coxswain-labs.dev/gateway-namespace — the Gateway namespace (a Coxswain label; the upstream group defines no namespace label).

The chart's PodMonitor then copies those pod labels onto every scraped sample via relabelings, so they appear on the metrics without ever touching the series the proxy emits:

relabelings:
  - sourceLabels:
      - __meta_kubernetes_pod_label_gateway_networking_k8s_io_gateway_name
    targetLabel: gateway_name
  - sourceLabels:
      - __meta_kubernetes_pod_label_gateway_coxswain_labs_dev_gateway_namespace
    targetLabel: gateway_namespace

(The __meta_kubernetes_pod_label_* source names are Prometheus's mangling of the pod-label keys — dots and slashes become underscores.) Pods in the shared pool don't carry those labels, so the relabel leaves gateway_name / gateway_namespace empty on shared pool samples — which is how you tell the two apart in a query.

Health endpoints¶

Endpoint	Port	Returns
`/healthz`	`8081`	Always `200 ok` while the process is running
`/readyz`	`8081`	`200` once all subsystems are Ready or Degraded; `503` otherwise

/readyz returns 503 during startup until:

All Kubernetes reflectors emit their first InitDone event (CRDs must be installed)
The routing table is built for the first time

Inspect the per-subsystem detail via the admin port (open kubectl -n coxswain-system port-forward svc/coxswain-shared-proxy-internal 8082:8082 in a separate terminal first; a non-default Helm release name <rel> prefixes the Service as <rel>-coxswain-shared-proxy-internal):

curl -s http://localhost:8082/api/v1/health | jq .

Example output:

{
  "version": "0.3.0",
  "subsystems": {
    "controller": {
      "status": "Ready",
      "checks": {
        "httproute": "Ready",
        "ingress": "Ready",
        "gateway": "Ready",
        "routing_table_built": "Ready"
      }
    },
    "proxy": {
      "status": "Ready",
      "checks": {
        "routing_table_loaded": "Ready"
      }
    }
  }
}

Routes endpoint¶

Proxy pods carry no query surface beyond /metrics and /api/v1/health — a proxy's compiled routing table is served from the controller's own local routing snapshot, not fetched from the pod:

curl -s http://<controller-admin-address>:8082/api/v1/fleet/proxies/<pod-name>/routes | jq .

Returns the named pod's routing table as JSON, nested under routes: all hostname entries, their rules, and resolved upstream addresses. Useful for debugging routing decisions without reading raw Kubernetes objects. List pod names with curl -s http://<controller-admin-address>:8082/api/v1/fleet/proxies | jq ..

Access logs¶

Coxswain emits one structured log event per proxied request at INFO level on the coxswain_proxy::access target. Access logs are active by default; set --access-log=false (or COXSWAIN_ACCESS_LOG=false) to disable them entirely.

Log fields¶

Field	Type	Description
`host`	string	`Host` header value
`method`	string	HTTP method
`path`	string	Request path — see `--access-log-path-mode` below
`status`	integer	Response HTTP status code
`route_id`	string	Canonical rule identifier; same value as the `route` Prometheus label. Empty for requests that bypass routing (e.g. 404 with no matching host)
`upstream`	string	Name of the matched upstream service
`upstream_addr`	string	Selected endpoint `ip:port`
`duration_ms`	integer	Total request duration in milliseconds
`bytes_sent`	integer	Response body bytes sent to the client
`error`	string	Error message if the request failed (omitted on success)

The route_id field is the join key for pivoting from a Grafana alert into a log slice. Copy the metric label value verbatim into a Loki / CloudWatch / Splunk filter to land on the exact rule's traffic.

The timestamp field is written automatically by the logging subscriber in RFC 3339 format.

Path redaction¶

The --access-log-path-mode flag (or COXSWAIN_ACCESS_LOG_PATH_MODE) controls what the path field contains:

Value	`path` field content	Use case
`full` (default)	The concrete normalised request path, e.g. `/users/42/orders/7`	Standard traffic analysis; required for security log analysis
`pattern`	The matched rule's registered path pattern, e.g. `/users/`	Cardinality reduction — see security note below
`none`	Field omitted entirely	Strict path redaction — see security note below

Security impact of pattern and none

Both pattern and none suppress the actual request path in the access log. Any WAF, SIEM, or log-based alerting that scans the path field for path-traversal attempts (.., %2e, %2f), encoded payloads, or other abnormal input will miss those requests entirely when either mode is active.

Deployments where access logs feed a security pipeline must use full.

Prefer pipeline-side redaction

Redacting at the log-collection pipeline is the architecturally correct default — it keeps the proxy emitting ground truth while centralising PII policy. Use pattern or none only when the pipeline genuinely cannot handle high-cardinality path fields.

Per-class suppression¶

Operators who want to silence access logs for a specific traffic segment (for example, health-check noise) without turning off logging globally can configure it on the CoxswainIngressClassParameters CR:

apiVersion: ingress.coxswain-labs.dev/v1alpha1
kind: CoxswainIngressClassParameters
metadata:
  name: health-check-class
  namespace: coxswain
spec:
  accessLog: false   # suppress access-log lines for this class's routes

Set the matching IngressClass to reference this CR via spec.parameters, then route your health-check traffic through Ingresses claiming that class. Only the coxswain_proxy::access log lines are suppressed — error logs and Prometheus metrics continue unaffected.

accessLog: false is a downward-only override: it never force-enables logging when --access-log is already off globally. This mirrors Istio's Telemetry.spec.accessLogging[].disabled + spec.selector model, where suppression is workload-scoped rather than per-route.

Filtering access logs¶

Access logs are emitted on the coxswain_proxy::access target, so they can be silenced independently of other logs:

# Silence access logs, keep controller logs at INFO
--log=info,coxswain_proxy::access=off

# Or via environment variable
COXSWAIN_LOG=info,coxswain_proxy::access=off

Kubernetes Events¶

The controller emits Kubernetes Warning Events for diagnostic conditions — on the affected Ingress object for the Ingress-routing conditions below, and on the affected Gateway object for the shared-mode addressing conditions. Events appear in kubectl describe <kind> <name> and kubectl get events — no log-aggregation pipeline needed.

RouteConflict¶

Triggered when two or more Ingresses claim the same (host, path) combination. The earliest-applied rule wins; every losing Ingress receives one Warning Event:

Type     Reason         Age   From                 Message
----     ------         ---   ----                 -------
Warning  RouteConflict  12s   coxswain-controller  Route on host api.example.com path /v1 is shadowed by default/primary-ingress

The Message field names the winning Ingress as <namespace>/<name>. No Event is emitted on the winning Ingress. See Ingress annotation reference for how to resolve conflicts (e.g. deduplicate rules or adjust IngressClass assignments).

InvalidAnnotation¶

Triggered when an ingress.coxswain-labs.dev/* annotation value cannot be parsed. The affected feature is disabled or falls back to its default; the Ingress itself continues to serve traffic from its valid rules.

Type     Reason             Age   From                 Message
----     ------             ---   ----                 -------
Warning  InvalidAnnotation  5s    coxswain-controller  ingress.coxswain-labs.dev/read-timeout: invalid duration — using default

InternalPortRemapped (shared-mode Gateway)¶

Emitted on a shared-mode Gateway if the controller ever reallocates a live listener's internal accept port (the 30000–32767 port its VIP Service maps the advertised port onto). The controller reads existing allocations authoritatively from the apiserver each pass and keeps every in-range assignment, so this is expected to never fire; if it does, it means kube-proxy's NAT was remapped while the proxy may still be bound to the old port (a brief data-plane blip for that Gateway) and points at a genuine anomaly — a persisted targetPort outside the range, or a duplicate.

Type     Reason                Age   From                 Message
----     ------                ---   ----                 -------
Warning  InternalPortRemapped  3s    coxswain-controller  listener 443 internal port moved 30001 -> 30004 while the Gateway is live

InternalPortRangeExhausted (shared-mode Gateway)¶

Emitted on a shared-mode Gateway when the 30000–32767 internal-port band is exhausted; the un-allocated listeners get no VIP port and are not addressed until capacity frees up.

Deduplication¶

Both event types are deduplicated by the kube Recorder's built-in per-process cache. A resync storm does not produce duplicate Events in kubectl describe output — the controller updates the existing Event's count rather than creating a new one. If the condition is corrected and then reintroduced (e.g. the annotation is fixed and then broken again), the Event re-appears.

Operator queries¶

# All Warning events on Ingresses in the default namespace
kubectl get events -n default --field-selector reason=RouteConflict,type=Warning

# All Invalid annotation events cluster-wide (requires cluster-admin)
kubectl get events -A --field-selector reason=InvalidAnnotation,type=Warning

# Describe a specific Ingress to see its Events inline
kubectl describe ingress <name> -n <namespace>

Logging¶

Coxswain uses structured logging via tracing. Configure the level with --log (or COXSWAIN_LOG):

Value	Effect
`error`	Only errors
`warn`	Errors and warnings
`info`	Normal operational messages (default)
`debug`	Detailed reconciler and routing events
`trace`	Very verbose; includes per-request proxy events

Use RUST_LOG directive syntax for per-crate control:

--log=info,coxswain_controller=debug,coxswain_proxy=warn

Log formats¶

`--log-format`	Description
`json`	Structured JSON; one line per event. Use in production for log aggregation.
`console`	Human-readable; colourised in a terminal. Use for local development.

Prometheus scrape configuration¶

Prometheus operator (PodMonitor)Prometheus scrape_configs

The chart ships a PodMonitor template gated on .Values.podMonitor.enabled. Enable it with --set podMonitor.enabled=true (or podMonitor.enabled: true in your values file). One selector matches both the shared proxy pool and the operator-rendered dedicated proxies; the relabel block injects gateway_name / gateway_namespace on dedicated metrics only (see Metric labels per Gateway above).

Why PodMonitor and not ServiceMonitor? Dedicated proxy Services don't expose port 8082 — adding it would leak /metrics onto the LoadBalancer IP. PodMonitor scrapes the pod directly (port admin, :8082) and skips that issue entirely. Shared pool pods are also discovered the same way, so one resource covers every coxswain proxy in the cluster.

Hardened installs that pin podMonitorSelector on the Prometheus resource must update it to include the chart's labels (app.kubernetes.io/name: coxswain) — by default kube-prometheus-stack matches both ServiceMonitor and PodMonitor broadly.

The Service names below are for the default coxswain release and for raw-manifest installs; a non-default Helm release name <rel> prefixes them as <rel>-coxswain-shared-proxy-internal and <rel>-coxswain-controller. Dedicated proxy metrics aren't reachable through this Service surface — use the PodMonitor path for full coverage.

scrape_configs:
  - job_name: coxswain-shared
    static_configs:
      - targets: ['coxswain-shared-proxy-internal.coxswain-system.svc:8082']
    metrics_path: /metrics
  - job_name: coxswain-controller
    static_configs:
      - targets: ['coxswain-controller.coxswain-system.svc:8082']
    metrics_path: /metrics

Grafana dashboard¶

A community Grafana dashboard for Coxswain is planned for a future release. In the meantime, the metrics above are compatible with standard Kubernetes proxy dashboards (e.g. kubernetes-nginx-ingress panels adapted for coxswain_ prefix).