Trace Town reads OpenTelemetry. What matters is that your services ship OTel metrics somewhere with a Prometheus query API — not who stores them.
This is a young document and it will grow as more systems get connected. If yours is not here and you get it working, we would like to know.
Open the town, then Data → Connect a source. Two steps: which provider, then where it is. Everything can be changed afterwards.
Signed in, the credential is encrypted and stored against your account, and our server does the fetching — so it survives a reload and a team can share a town without sharing a key with a person. Some providers, Grafana Cloud among them, only work this way: they send no CORS headers, so a browser cannot call them directly.
Signed out, the key is held in that tab's memory and nowhere else. You retype it after a reload. That is not a crippled mode — for someone pointing at a Prometheus on their own laptop it is the right one — but it only works where the provider allows cross-origin requests.
Also Mimir, Thanos, Victoria Metrics — anything speaking the Prometheus query API.
Trace Town/api/v1/query hangs off, usually http://your-prometheus:9090
Browsing signed out, Prometheus needs --web.cors.origin
set or it will refuse the browser. Signed in, our server does the
fetching and this does not arise.
Grafana Cloud sends no CORS headers, so it cannot be read from a browser directly.
instanceID:token — with the colon, which is what sends HTTP Basicmetrics:read
scope. Copy it — it is shown once. If you lose it, delete it
under Security → Access Policies and make another; the value
cannot be retrieved.
instanceID:token as the credential. Use the
Show button to check it before connecting.
Self-hosted or cloud. Brings its own logs, so there is nothing else to point at.
8080SigNoz accepts two authentication schemes and which one a route wants varies by version. A 403 is more often the key’s account lacking the admin role than a wrong key.
A companion, not a source of its own — pair it with a metrics provider that keeps no logs.
Standard OpenTelemetry, and nothing bespoke. Trace Town reads whichever of these it finds:
http.server.request.duration the service itself — rate, errors, latency
http.client.request.duration service to service, and to third parties
db.client.operation.duration service to datastore
messaging.client.sent.messages service to broker
messaging.client.consumed.messages broker to worker
If your collector runs the spanmetrics connector, that is
found automatically and preferred — it covers every service that emits a
span rather than only the ones serving HTTP, which is the difference
between seeing your queue workers and not.
Set service.namespace on your services and the town zones
itself into neighbourhoods with a signpost at the edge of each. It is the
single highest-value attribute you can add.
A service's shape is guessed from its name and from how it is used, and you can correct it — select a building and change its kind. Height is request rate on a log scale, so five a second still gets you a skyline.
Every figure on the roads is a call. Blue is a clean one and red is a failed one, and how fast they move is the p99 of whatever they are going to see. What they are carrying says something the roads on their own never could: whether anybody is waiting for an answer.
Two of those go to the same building and mean opposite things, which is why the depot has both a counter and loading bays, and why a factory has a door for people and a shutter for vans. You can watch which one a given call uses.
Every signal has one visual meaning, and nothing else shares it.
Requests, errors and p99 are the town's totals, and they mean what you would expect. Two do not:
Population is an accumulator, not a measurement. It counts the traffic this town has served while healthy, and nothing else. Serve a lot, well, for a long time, and it grows. An incident stops it growing rather than knocking it down — the repair sites carry the bad news instead. It is a session's worth: a reload resets it, and so does moving to a different town, because it belongs to the town that earned it.
Level is the same number on a gentler curve, so it keeps moving once population has grown too large for a change in it to show.
Repairs counts the services currently degraded or under repair, out of the total.
An empty town. Usually the query window: a provider only reports what had traffic, and a quiet service reports nothing at all. Widen the window in the connection settings before assuming anything is broken.
Services that come and go. Same cause. Trace Town remembers a service that has gone quiet rather than demolishing its building, but a window narrower than your traffic will still make the town restless.
Nothing but buildings, no roads. Roads come from client metrics —
http.client, db.client,
messaging.client. Without them we know your services exist
but not what they call.
Missing workers. A queue worker never serves an HTTP request, so
it has no http.server metrics and appears only as the far
end of somebody else's. Running the collector's
spanmetrics connector fixes this properly.
A 401 or 403. Check whether the provider wants the credential in a different header, and whether the key's account has the role it needs.