In Terraform quoted strings $var is literal — only ${var} triggers
interpolation. The $$ was passing through as literal $$kube_* to
Fluent Bit, causing a record accessor syntax error on startup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds two targeted observability improvements across all homelab services.
pkg/logger/access.go (new)
HTTP access log middleware that logs one structured line per request:
method, path, status, ms, trace_id
The trace_id comes from the OTel span already in context (created by
trace.Middleware which runs outside this one), so each log entry in
Loki has a clickable link into Jaeger. Health/metrics endpoints are
excluded to avoid noise. Level is ERROR for 5xx, WARN for 4xx, INFO
otherwise.
pkg/setup/setup.go
Wire the new middleware between trace.Middleware (which creates the
span) and metrics.Middleware:
trace → AccessMiddleware → metrics → mux
Order matters: span must exist before AccessMiddleware reads it.
infrastructure/terraform/monitoring.tf
Fluent Bit was shipping all container logs to Loki with a single
static label (job=fluent-bit), making it impossible to filter logs
by service. Added a `nest/lift` filter that flattens the kubernetes
metadata block to top-level fields (kube_namespace_name,
kube_container_name, …), then promoted those as Loki label_keys.
After this change you can query:
{kube_namespace_name="finance"} |= "trace_id"
and LogQL will only return finance-api logs.
Co-authored-by: Gonçalo Rodrigues <guga@Goncalos-MacBook-Pro.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
