Trace Funnels: step-overview `latency_type: "p50"` silently returns the p99 value (only p90/p95 are implemented)

[wiz-abhi]

Bug description

POST /api/v1/trace-funnels/{funnel_id}/analytics/steps/overview returns a latency field whose percentile is taken from the end step's latency_type. Only p90 and p95 are actually implemented — every other value, including p50, falls through to default and produces a p99 quantile, which is then returned as if it were the requested percentile.

In pkg/modules/tracefunnel/clickhouse_queries.go, inside BuildFunnelStepOverviewQuery:

	// Determine latency quantile for the end step
	latencyQuantile := "0.99"
	if steps[endIdx].LatencyType != "" {
		latency := strings.ToLower(steps[endIdx].LatencyType)
		switch latency {
		case "p90":
			latencyQuantile = "0.90"
		case "p95":
			latencyQuantile = "0.95"
		default:
			latencyQuantile = "0.99"
		}
	}

latencyQuantile is then interpolated straight into the SQL as quantileIf(%s)(...) AS latency.

latency_type reaches this switch unvalidated: pkg/types/tracefunneltypes/tracefunnel.go declares it as a plain LatencyType string with json:"latency_type,omitempty", and pkg/modules/tracefunnel/query.go copies it through verbatim (LatencyType: step.LatencyType). Nothing anywhere rejects an unsupported value.

Impact

This is a silently wrong number, not an error — arguably the worst failure mode for an analytics endpoint:

• A p50 request returns a p99. The user reads a "median" that is actually a tail number, and concludes the typical request through that funnel step is far slower than it is. On a long-tailed latency distribution — the normal case — the gap is large.
• No error, no warning, no log line, HTTP 200. There is no way for a client to tell that its request was not honoured.
• It produces results that are internally contradictory: in the measured run below, the "p50" comes back higher than the p90 and p95 of the same distribution, which is arithmetically impossible for a real median.
• Any percentile a user might reasonably send (p50, p75, p99.9) has the same behaviour.

SigNoz's own docs and marketing consistently talk about latency in terms of P50/P95/P99, so p50 is a value users will naturally reach for. (The frontend enum LatencyOptions in frontend/src/types/api/traceFunnels/index.ts currently exposes only P99/P95/P90, so the UI happens not to hit this — but the API is public and does.)

Expected behavior

latency_type: "p50" should produce quantileIf(0.50)(...), i.e. the returned latency should be the median of the step transition duration. Generally, the returned value should be the percentile that was asked for — or the request should be rejected — but never a different percentile reported under the requested label.

How to reproduce

Reproduced live on self-hosted v0.132.2: a 2-step funnel agent.plan → agent.respond over service fot-spike2, 60 traces in the window.

Note the two time units, which are easy to get wrong: funnel create/update timestamp is in milliseconds, while the analytics start_time/end_time are in nanoseconds. step_start/step_end are 1-based integers. latency_type lives on the step object inside steps/update, and the value that matters is the one on the end step of the transition being queried.

JWT=<your token>
BASE=http://localhost:8080

# 1. Create a funnel (timestamp in MILLISECONDS)
curl -X POST $BASE/api/v1/trace-funnels/new \
  -H "Authorization: Bearer $JWT" -H 'Content-Type: application/json' \
  -d '{"funnel_name":"p50-repro","timestamp":1784219000000}'
# -> {"data":{"funnel_id":"<id>", ...}}

# 2. Configure two steps that real traces satisfy.
#    latency_type is per-step; the END step's value drives the reported latency.
curl -X PUT $BASE/api/v1/trace-funnels/steps/update \
  -H "Authorization: Bearer $JWT" -H 'Content-Type: application/json' \
  -d '{"funnel_id":"<id>","timestamp":1784219000000,"steps":[
    {"step_order":1,"service_name":"fot-spike2","span_name":"agent.plan",
     "filters":{"items":[],"op":"AND"},"latency_pointer":"start","latency_type":"p50","has_errors":false},
    {"step_order":2,"service_name":"fot-spike2","span_name":"agent.respond",
     "filters":{"items":[],"op":"AND"},"latency_pointer":"start","latency_type":"p50","has_errors":false}]}'

# 3. Ask for the step overview (start_time/end_time in NANOSECONDS, step_start/step_end 1-based)
curl -X POST $BASE/api/v1/trace-funnels/<id>/analytics/steps/overview \
  -H "Authorization: Bearer $JWT" -H 'Content-Type: application/json' \
  -d '{"start_time":1781627000000000000,"end_time":1784219000000000000,"step_start":1,"step_end":2}'

Then re-run step 2 varying only the end step's latency_type, re-running step 3 each time. Actual latency values returned (ms), same funnel, same window, same data:

latency_type	returned latency
p50	18.673343
p90	17.963510
p95	18.011850
p99	18.673343
(unset)	18.673343

Three things this shows, none of which require reading the code:

1. p50 returns exactly the p99 value — 18.673343, identical also to the unset default. That is the default: branch firing.
2. p90 and p95 return distinct values, so this is not a degenerate distribution where every quantile happens to coincide, and those two cases are genuinely implemented.
3. The reported "p50" is larger than both the p90 and the p95 of the same distribution. A median cannot exceed the 90th and 95th percentiles of the data it came from. The response is self-evidently wrong on its face — and it arrives as HTTP 200 with no warning and no validation error.

Expected: the p50 response returns the median of the step transition duration, which — by definition of a percentile — must sit at or below every one of the p90/p95/p99 values measured on the same data.

The mechanism behind the numbers is visible without a running stack, straight from the query builder — this is where I first found it:

steps := []struct {
	ServiceName    string
	SpanName       string
	ContainsError  int
	LatencyPointer string
	LatencyType    string
	Clause         string
}{
	{ServiceName: "svc", SpanName: "span-a", LatencyPointer: "start", LatencyType: "p50"},
	{ServiceName: "svc", SpanName: "span-b", LatencyPointer: "start", LatencyType: "p50"},
}

query := BuildFunnelStepOverviewQuery(steps, 1000000000, 2000000000, 1, 2)
// query contains "quantileIf(0.99)" — the p50 request produced a p99 aggregate.

Suggested fix

In BuildFunnelStepOverviewQuery, replace the two-case switch with a generic pNN parse — take the numeric suffix and convert it to a 0–1 quantile — so p50, p75, p99.9 and anything else a client sends all map correctly, while an empty or unparseable latency_type keeps today's 0.99 default. Alternatively, return an explicit validation error for unsupported values; either is fine, as long as the endpoint stops relabelling one percentile as another.

I have a patch with unit tests ready and will open a PR against this issue.

Related (not fixed by the above)

BuildFunnelOverviewQuery — the sibling builder used by /analytics/overview — hardcodes quantileIf(0.99) with no latency_type knob at all, so the whole-funnel overview is always p99 regardless of step configuration. Worth deciding whether that endpoint should honour latency_type too; flagging it here as context rather than folding it into this bug.

Version information

• SigNoz version: v0.132.2 (/api/v1/version → {"version":"v0.132.2","ee":"Y","setupCompleted":true}), self-hosted via Foundry (docker compose flavor)
• OS: Windows 11, Docker Desktop (WSL2 backend)
• Data: 60 traces in the queried window, emitted through the OpenTelemetry Python SDK by a test agent and stored in ClickHouse normally. The workload is synthetic, but the spans, the storage path, and the returned latency values are all real — and the defect is data-independent in any case, since it is a constant chosen before the query runs.
• Code verified against: main

Thank you for your bug report – we love squashing them!

[wiz-abhi]

Opened a PR for this: #12221

It replaces the two-case switch in BuildFunnelStepOverviewQuery with a generic pNN parse, so a requested percentile maps to the quantile it names. p90/p95/p99 and unset latency_type produce byte-identical SQL to today (0.90/0.95/0.99), so nothing that currently works changes — only the values that were previously mismapped. Table-driven tests cover p50 → quantileIf(0.50), p75, fractional p99.9, uppercase P50, and the three fallback paths.

Happy to adjust the approach if you'd prefer an explicit validation error over a permissive parse.

[kunalrawat425]

Hi, I've been looking at this bug and opened a fix in #12230.

The root cause is the BuildFunnelStepOverviewQuery switch in pkg/modules/tracefunnel/clickhouse_queries.go only handles p90 and p95 — everything else including p50 falls through to default and produces a p99 quantile silently.

The PR adds p50, p75, and explicit p99 cases with regression tests covering each new case and the unknown-value fallback. Happy to adjust based on review feedback.