Where do the Astronomy Shop's RED metrics come from, if the services never explicitly count requests or errors?

They are derived from the traces: the Collector's spanmetrics connector counts the spans passing through it and measures their durations and error status, producing request rate, error rate and duration per service and operation.

Every span already contains everything RED needs: a service name, an operation name (such as GET /api/cart), a duration, and a status that says whether it failed. The spanmetrics connector sits between the Collector's trace pipeline and its metrics pipeline. For each span it increments a counter and records the duration in a histogram, labelled by service and span name. Those metrics go to Prometheus, and Grafana draws the three RED panels from them.

This has two consequences worth knowing:

• Instrument once, get both. A service with tracing gets RED dashboards for free, with no separate metrics code.
• The metrics are only as complete as the spans that reach the connector. If traces were sampled away before it, request counts and error rates would come out too low, so the connector has to be fed the full, unsampled span stream.

It also explains a detail on the dashboard: panels can take a few minutes after start-up to show data, because enough spans have to flow through first.

Go deeper:

• spanmetrics connector README — configuration and the metrics it emits.