docs(rfc-0001): CLI auth & client provisioning — accepted design-of-record

[saadqbal]

Merges RFC 0001 into develop as the accepted design-of-record for CLI auth & one-command client provisioning.

Tracking epic: #54 (closed).

Status: ACCEPTED — implemented

This started as a draft for discussion. The design it proposed has since been built and shipped in CLI v0.4.0 (#107), so this PR closes it out: the status header is flipped from DRAFT → ACCEPTED and reconciled with what actually landed. The branch has been rebased onto current develop.

What shipped against this design:

• Browser device-flow login / logout / auth status — feat(cli): auth scaffold — login/logout/auth status + config + backend client (cli#83) #85, feat(cli): authenticate with Bearer + verify token on login (cli#83) #86
• client create / list / use (silent, idempotent, never-show credential) — feat(cli): client create / list / use commands (#84) #92, feat(cli): client create reads the cluster anchor — idempotent get-or-create + 409 (#84) #102, feat(cli): client create --credential-file — write the credential for the installer (#84) #104
• env-scoped config v2 — feat(cli): env-scoped config v2 — fix cross-env active-client clobber (cli#100) #120; cluster doctor + --verbose — feat(#88): tracebloc cluster doctor — live-cluster health checks (WS3) #89, feat(#90): cluster doctor — node-fit + image-pull checks (WS3 follow-up) #91, feat(cli): --verbose + cluster doctor auth checks + resume hint & install log (cli#101) #121
• User-Agent version header + 426 handling — feat(auth): User-Agent version header + 426 Upgrade Required handling (cli#98) #115; local logout — feat(auth): logout revokes the token server-side (cli#112) #116

Not in this PR (Phase 2, still tracked)

Server-side token revoke (backend#887), short-lived auto-refreshing tokens, client rotate, atomic fleet enrollment — see §11. The revision history in the doc is retained as the record of how the design converged.

🤖 Generated with Claude Code

---

Note

Low Risk
Documentation-only addition with no runtime or config changes; main caveat is readers may treat pending Phase 2 items (e.g. server-side revoke) as already shipped unless they read the status header.

Overview
Adds docs/rfcs/0001-cli-auth-and-client-provisioning.md (~1,266 lines) as the accepted design-of-record for browser-based CLI auth and one-command client provisioning, marked implemented in CLI v0.4.0 with Phase 2 items called out explicitly.

The document replaces copy-paste Client ID/password onboarding with RFC 8628 device flow plus silent client create, and centers the hard problems on client lifecycle: kube-system UID as cluster_id for idempotent get-or-create, slug-based use/delete, selected vs connected, and installer reorder (auth/provision before Helm). It also pins API/data contracts (Appendix C), migration risks (R1–R12), and reconciles Rev 6–7 accuracy (CLI/installer—not heartbeat—sets cluster_id; logout is local-only until POST /auth/revoke / backend#887).

^{Reviewed by Cursor Bugbot for commit 10608ba. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 16 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#102 — chore(kanban): declare staging as prod-truth branch via .kanban.yml · author: @LukasWodka · no reviewer assigned
• averaging-service#103 — fix(averaging): correctness foundations — DP, shape guard, sklearn weighting, bounded ensembles (dataset push: tabular pre-flight silently collapses duplicate header columns and accepts zero-data-row CSVs #73) · author: @LukasWodka · reviewer: @shujaatTracebloc
• averaging-service#98 — ci: add fr-gate caller workflow · author: @LukasWodka · no reviewer assigned
• backend#732 — feat(experiments): add per-experiment aggregation_strategy field (#730) · author: @LukasWodka · reviewer: @saqlainsyed007, @shujaatTracebloc
• data-ingestors#146 — test(schema): lock category enum to the engine's TaskCategory (anti-drift) · author: @LukasWodka · reviewer: @saadqbal
• data-ingestors#148 — Develop to prod · author: @divyasinghds · reviewer: @saadqbal
• design-system#19 — fix: un-track coverage/ and node_modules/ from git · author: @LukasWodka · no reviewer assigned
• design-system#22 — ci: add Vitest test workflow · author: @LukasWodka · reviewer: @aptracebloc
• design-system#23 — ci: add fr-gate caller workflow · author: @LukasWodka · no reviewer assigned
• docs#47 — docs(environment-setup): fix(client): hide client create from the user surface (installer-internal) #192 namespace/workspace flow (hold for client#192) · author: @LukasWodka · reviewer: @saadqbal

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[LukasWodka]

Stood up a cross-repo tracking epic so this RFC is tracked as the leading CLI plan: tracebloc/backend#830 (North Stars, Epic, owner @saadqbal). It carries the phased plan (Phase 0/1/2), the cross-repo work breakdown (backend / cli / client), and the open questions for the discussion. This RFC doc stays the source of truth — the epic is just the tracker.

[LukasWodka]

Worked the §11 open questions to resolution with a code-grounded sweep (backend / client / cli) and laid out the full cross-repo plan + device-grant design in the tracking epic tracebloc/backend#830. Headlines (most are dictated by the code, not opinion):

• Q1 air-gap → out of scope. Preflight hard-fails on no egress; data ingest: silent wait after submit, undisclosed 1h watch cap, and k8s jargon in the run output #172 is corporate-proxy support, not air-gap. Real segment = egress-restricted-but-online (must work behind a TLS-inspecting proxy).
• Q2 namespace → name→slug→BOTH EdgeDevice.namespace + TB_NAMESPACE. The heartbeat overwrites namespace every ping, so the provisioned slug must equal the install-time namespace or it drifts.
• Q3 location change → future-only, already clean. gCO₂ is a frozen per-experiment snapshot — never re-derived from current location.
• Q4 RBAC → split read from write + "pick existing / ask admin" (today one perm gates both list and create → bare 403).
• Q5 → multi-client per account/host is free; re-parenting deferred (the viewset force-stamps account to the requester).
• Q6 IdP → reuse the existing web login as-is. Auth is token-based; Google/GitHub already mint the same token, so /activate is just a token-authed endpoint binding to request.user — no new IdP wiring.

Two need your product nod — Q1 (air-gap out) and Q5 (re-parenting deferred); the rest follow from the code. Full detail, the DeviceGrant design, and the phased per-repo breakdown are in backend#830.

[LukasWodka]

Review — aligned on the model, +1 on the cluster_id anchor; gaps to close before final

Read Rev 2 end-to-end against an independent user-flow teardown: the two-context model, the nine lifecycle loopholes, and their resolutions all match. Appendix A's slug rule is byte-identical to what's already shipped + parity-tested in cli/internal/slug. Strong shape.

Decisions locked (Lukas):

• cluster-id = the kube-system namespace UID — go with it (closes the §12 open item).
• Q1 (air-gap out of scope) + Q5 (one-client-per-cluster, re-parenting deferred) — confirmed (already worked on backend#830); recording the owner nod here.

Adopting the three refinements over our draft spec: the cluster_id anchor (stronger than a post-install Helm-clientId), location-unset-not-0, slug-as-handle.

Now the part that doesn't feel final yet — issues by priority.

🔴 High — correctness / security

1. cluster_id uniqueness scope + cross-account. §6.3/§7.2 make cluster_id unique with get-or-create on it, but the kube-system UID is not a secret (anyone with cluster access reads it). If it's globally unique and get-or-create blindly returns the match, re-targeting a cluster from a different account would silently adopt the first account's client. Spec it: globally unique, but get-or-create returns the client only if it's in the requester's account — a cluster bound to another account is an explicit conflict, never a silent adoption.

2. Existing installs have no cluster_id → re-run double-provisions. The anchor is net-new, so every current client has a null cluster_id. get-or-create-by-cluster won't match them, so the next installer re-run on an already-connected box mints a duplicate and orphans the live client. This needs an explicit backfill before it can ship (ties to Promote kubeconfig flags (--kubeconfig/--context/--namespace) to persistent on root #3) — otherwise every existing customer double-provisions on their next upgrade.

3. Heartbeat must carry cluster_id. The "connected" check (§7.3 / §12 = cluster-id match + heartbeat recency) needs the runtime to report cluster_id, but §4.5 has the heartbeat reporting only namespace. Add cluster_id to the heartbeat contract — it both powers the connected-state check and backfills Phase 2: kubeconfig discovery + parent release detection + SA token #2 for live clients.

🟠 Medium — edge cases

4. Cluster rebuild orphans the old client. The documented "switch client" path (k3d cluster delete → reinstall) produces a new kube-system UID → new cluster_id → a freshly minted client, leaving the old EdgeDevice orphaned forever (its cluster_id never returns). Needs a reaper or a teardown hook, or orphans accumulate per rebuild.

5. Orphan password-reset assumption (§7.9). "An orphan never connected, so resetting its password is safe" doesn't hold for a client that connected once then lost its local values file — the reset would break its running pod. Gate the reset on heartbeat recency (offline / truly-never-connected), not just "no values file."

6. client create requires a live cluster (ordering). The anchor is only readable once a cluster exists, so create can't precede cluster bootstrap. State that create operates against an existing reachable cluster (clear error if none) and who bootstraps it (installer vs a cluster verb) — §8.1 implies it, but the command contract (§6.2) doesn't.

7. logout should revoke server-side. §6.2/§7.5 clear the local token, but the ClientAccessToken stays valid for 30 days — a leaked/copied token survives logout. Have logout call the backend revoke (backend#845 already supports it); otherwise "logout" is a false sense of security.

🟡 Low — clarity / polish

8. Rename vs the frozen handle. The handle is the frozen slug (§7.1), but §8.1 offers "rename later." Rename changes first_name (display), not the slug — so the handle the user types stays the old slug, mismatching the new name. Say explicitly that rename is cosmetic; the handle remains the frozen slug.

9. Reconcile the "authoritative" framing. Appendix A calls the namespace UniqueConstraint "authoritative"; §7.2 makes cluster_id the real one-per-cluster guard. With cluster-id confirmed, cluster_id is authoritative for idempotency and namespace-unique is cosmetic cross-cluster dedup (demotes backend#863 — noting it there).

10. Device-code phishing (§9). For the regulated on-prem sell, spell the RFC 8628 mitigations out a bit more: bind user_code → account, short TTL (impl has 10 min), rate-limit, and an approval page that names the device + "only approve if you started this."

11. Credential at-rest footprint. §9 covers the host values.yaml (0600); also note the credential lands base64 in the k8s / Helm release Secret (etcd). Fine for single-tenant on-prem — worth stating as a conscious call.

12. Picker at fleet scale. The arrow-key picker is great for a handful of clients; an account with dozens needs filter/search (or the phase-2 enrollment-key path). Note the N-scaling.

---

#1–#3 are the ones I'd treat as blocking — together they're the difference between "idempotent" working for new boxes only vs. the existing fleet too. Happy to draft the cluster_id field + heartbeat + backfill on the backend side if useful.

[saadqbal]

Thanks for the thorough review — all of it is folded into the RFC in 58bdd06 (rev 3). Map of where each point landed:

🔴 Blocking

• Cross-account adoption → §6.3 + §7.2 (R6): account-scoped get-or-create; a cluster_id bound to another account is a 409, never a silent adoption.
• Existing fleet has null cluster_id → §10 backfill (R7) + a §7.2 step-2a "never mint over a live in-namespace release" guard for the pre-backfill window. This was the real hole — idempotency now covers the existing fleet, not just new boxes.
• Heartbeat must carry cluster_id → §4.5 + §6.3 + §10 (powers the connected check and backfills live clients).

🟠 Medium

• Rebuild orphans → R3 (reaper / teardown hook).
• Orphan password-reset → §7.9, now gated on heartbeat recency, not "no values file".
• create needs a live cluster → stated in the §6.2 command contract.
• logout server-revoke → §7.5 + §9 (R2), now done at logout rather than deferred.

🟡 Polish

• Rename is cosmetic / handle stays the frozen slug → §7.1 + §8.1.
• The "authoritative" contradiction → fixed: cluster_id is authoritative for idempotency; the namespace UniqueConstraint is demoted to cosmetic cross-cluster dedup (Appendix A + §12).
• Device-code phishing detail + etcd-at-rest note → §9. Picker at scale → §7.1.

Decisions recorded: cluster-id = kube-system UID, Q1 (air-gap out) and Q5 (one-client-per-cluster, re-parenting deferred) all confirmed (§0 / §12).

The remaining blocking work is backend — the cluster_id field + account-scoped get-or-create, the heartbeat cluster_id + existing-fleet backfill, and wiring logout to the revoke endpoint — reflected in the §13 breakdown. Happy to take you up on the offer to draft those on the backend side.

[LukasWodka]

Review round 2 — Rev 3 is a strong revision; one foundational soft spot left (the cluster_id anchor)

Ran a multi-perspective pass (security / UX / correctness / SRE / completeness) over the full Rev 3 + adversarial verification of every finding against the actual code (22 verified real, 10 misreads filtered). The round-1 blockers (R6/R7/R2) are genuinely mechanized now, not just named — nice. The findings below all converge on one place: the cluster_id anchor. The concept (idempotency keyed on cluster identity) is right; the mechanism as specified doesn't hold yet. Everything else is second-order.

🔴 The anchor cluster — recommend a design pass before Phase 1

These interlock (one problem, several facets), each grounded in the code this will ship against:

1. Wrong trust model (root cause). The anchor's premise is "the kube-system UID is not a secret", yet cluster_id is self-reported by the client over the machine-credential-authed heartbeat, and common/utils/edge_device_utils.py:113 already trusts self-reported client_info verbatim (edge_device.namespace = client_info.get('namespace', ...), no residency check). So a compromised box — or anyone who read the non-secret UID off a shared cluster — can assert any cluster_id, poisoning the binding so a victim's next create adopts the attacker's client. A self-asserted, non-secret, every-ping-mutable value can't be a cross-account security boundary.

2. The backfill (R7) is mechanically impossible as written. It backfills cluster_id "from the heartbeat", but the heartbeat sender (jobs-manager, send_heartbeat) reads its namespace from its own pod mount and its ClusterRole (client/client/templates/rbac.yaml) grants jobs/pods/secrets but not namespaces — it can't GET kube-system to read the UID. The only component that can (the resource-monitor DaemonSet, resource-monitor-rbac.yaml) doesn't send client_info and is disableable. The named carrier can't carry it — on exactly the existing fleet R7 must protect.

3. §7.2 step-2a bypasses R6 and discloses credentials cross-account. Step-2a adopts the in-cluster TB_CLIENT_ID client-side, before any backend call. So account B pointed at account A's cluster reads A's machine credential straight out of the Secret and adopts it — the server-side 409 is never reached. That's cross-account credential disclosure, not a duplicate-client nuisance. (And the 409 is structurally unreachable even on the mint path: edge_device_views.py:17-19 hard-filters every query to request.user.account and :23 force-stamps the account, so account B can't even see A's row — get-or-create falls through and mints/500s instead of returning 409.)

4. unique=True (global) contradicts the account-scoped 409. Specified literally as global-unique (§6.3 line 287, §13 line 716), but the 409 contract needs account scoping. Global-unique → the second account's INSERT is a DB IntegrityError/500, never the documented 409. → UniqueConstraint(account, cluster_id) + an app-level 409 check.

5. step-2a's guard is keyed on the unstable namespace. §4.2 notes namespace isn't set at provisioning today — legacy clients' namespace wasn't installer-chosen, so a re-run's derived slug may not match the live release's namespace → the "never mint over a live release" guard misses → duplicate. The fleet-wide safety net keys on the one value that isn't stable for that fleet. (Fix: probe by Helm/Deployment label like DiscoverParentRelease, not by guessed namespace.)

6. 1:1 client↔cluster contradicts the shipped chart. client/client/values.yaml (~L372–375) explicitly supports multiple tracebloc namespaces per cluster (BYO) — they share one kube-system UID, so unique(cluster_id) would merge or hard-collide them. §3.1/Q5 and the chart disagree on the product's core topology. → either anchor on (cluster_id, namespace), or declare BYO multi-namespace unsupported and reverse that values.yaml decision.

Suggested fix shape: cluster_id is an idempotency hint, not a security boundary. Bind it authoritatively at create under the user token (account-scoped); treat the heartbeat value as write-once advisory (reject divergent, flag for the reaper). Constraint = UniqueConstraint(account, cluster_id) (or (cluster_id, namespace) for BYO) + app-level 409; route 2a/2b/3 through one server-side get-or-create so the client-side adopt can't bypass the cross-account check. Pick the backfill carrier explicitly (and state it's impossible under clusterScope:false).

🟠 Second-order findings (RFC otherwise solid)

• UX: /activate failure paths (tab closed / code expired / wrong account) unspecified — the Phase 1: embed ingest.v1.json + tracebloc ingest validate #1 beginner stuck-state. §8.1 never renders the 409/403 mid-install outcomes. Picker empty/single/remote-only states unspecified, and it invites selecting a client §7.3 then refuses. client list (§8.4) shows slug-only → "rename later" is invisible + handle coherence silently breaks. Silent wrong-account enrollment in the zero-prompt flow has no course-correct.
• Correctness: §7.9 concurrent-mint "loser adopts the winner" has no credential to adopt (backend stores only the hash) — a dead end; define the loser's path (server mints-and-returns once, loser gets a deterministic 409→resume). The orphan password-reset gated on "heartbeat recency" is itself a TOCTOU — needs a compare-and-set fence or an explicit operator rotate.
• Operability / security: cluster-rebuild orphans are unbounded live, un-revoked machine credentials (delete is the only revocation) — promote the reaper from watch-item to a security requirement; it also has no signal to distinguish a rebuild-orphan from a recoverable interrupted-install orphan.
• Contracts to pin: the device-token lifetime/scope/"refresh" is self-contradictory across §2/§4.3/R2/§11; the heartbeat cluster_id field contract (name, first-send, namespace-overwrite interaction) and the /activate user_code→request.user binding are left to the implementer.

✅ Solid

Two-identity model, the §7 lifecycle resolutions, the auth/device flow, dual-mode, and the §14 risk framing are all strong — and the round-1 blockers are closed. This is a mature RFC with one concentrated area to rework.

---

Net: the cluster_id anchor needs a focused pass (trust model → backfill carrier → constraint shape → BYO topology); the rest is buildable. Happy to draft the anchor-redesign as a concrete starting point if useful, @saadqbal — over to you.

_{Findings from a 5-lens review with per-finding adversarial verification against the backend/runtime/chart code; file:line refs above are the canonical repos.}

[saadqbal]

Re: your Rev 3 round-2 review — the cluster_id anchor design pass.

Did the focused anchor pass you recommended; it landed across Rev 4→6 (58bdd06…ab0ee88) and shipped on the backend. No new diff in this PR because it's already folded in — mapping each finding to where it closed:

🔴 Anchor cluster — trust model inverted exactly as suggested (idempotency hint, not a security boundary):

1. Wrong trust model → cluster_id is now set only on a Bearer-authed create / adopt-backfill by the CLI/installer (the kubeconfig-holder), and the heartbeat never sets it (§6.3, §9). Shipped in backend#893 — incl. a test asserting the heartbeat ignores a spoofed cluster_id.
2. Backfill carrier impossible via the heartbeat → backfill moved to the CLI/installer, which can read the kube-system UID; the heartbeat carrier is gone (§7.2, §10/R7).
3. step-2a cross-account disclosure → the account-scoped check now gates adoption itself, not just the mint, so reading a live TB_CLIENT_ID off-cluster can't sidestep the 409 (§7.2).
4. global unique vs account-scoped 409 → globally-unique cluster_id + an app-level account-scoped 409 (never a DB 500). The partial UniqueConstraint shipped in backend#898 (also closes feat(dataset): stage Pod + tar-over-exec stream (PR-b of #151) #9's authoritative-vs-cosmetic framing: cluster_id is authoritative, namespace-unique demoted to backend#863).
5. step-2a guard on the unstable namespace → re-run reconcile keys on the cluster anchor + live-release probe, not the guessed namespace (§7.2).
6. 1:1 vs BYO multi-namespace → declared one client per cluster explicitly; tenancy boundary = one account per cluster, multi-namespace handling stated (§3.1 / Q5 §12).

🟠 Second-order — also closed: logout revokes server-side (backend#845, §7.5) · §7.9 concurrent-mint loser now "fetches and adopts the winner," serialized by the constraint — mechanized at the DB in backend#898 this morning · /activate expiry / denied / wrong-account + RFC 8628 phishing mitigations specced (§ device-flow, §0, Appendix C) · device-token lifetime/scope pinned in §0 + Appendix C.

Net: the anchor was the part you flagged as not-final — it's both specced (Rev 6) and shipped now. If any of the above still reads as open, point me at it and I'll reopen.

🤖 Addressed by Claude Code

[saadqbal]

Correction to my resolution-map above — re: §7.5 logout server-side revoke.

FR'ing the connect flow on dev today (#877), this point was wrong in my earlier reply. tracebloc logout clears the local token but does not revoke server-side: I backed up the token → logout (auth status then shows "not signed in") → restored the copied token → client list still authenticated. A leaked/copied token survives logout (Lukas's original finding #7 stands).

So "logout revokes server-side (backend#845, §7.5)" is aspirational, not shipped: #845 added the revoke() primitive, but the POST /auth/revoke endpoint (#887) and the CLI logout→revoke call aren't wired. §7.5 overclaims and should be softened until #887 + the CLI half land. Evidence logged on #887.

🤖 Addressed by Claude Code

[LukasWodka]

👋 Heads-up — Code review queue is at 32 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#143 — chore(release): promote develop → staging (week of 2026-07-06) · author: @aptracebloc · no reviewer assigned
• averaging-service#156 — feat(strategies): FedMedian Byzantine-robust aggregation (CLI: a mistyped subcommand (e.g. dataset pus) prints help and exits 0 instead of erroring #75) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#157 — feat(strategies): FedTrimmedAvg Byzantine-robust aggregation (CLI: a mistyped subcommand (e.g. dataset pus) prints help and exits 0 instead of erroring #75) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#158 — feat(strategies): Krum / MultiKrum Byzantine-robust selection (CLI: a mistyped subcommand (e.g. dataset pus) prints help and exits 0 instead of erroring #75) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#159 — feat(strategies): FedProx aggregation strategy (dataset push: --category help lists 5 of 9 categories, and unknown-vs-empty category errors give two different lists (one leaks unsupported segmentation) #74) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#160 — feat(strategies): FedOpt family — FedAdam / FedYogi / FedAdagrad (dataset push: --category help lists 5 of 9 categories, and unknown-vs-empty category errors give two different lists (one leaks unsupported segmentation) #74) · author: @shujaatTracebloc · reviewer: @divyasinghds
• averaging-service#161 — feat(strategies): resolve per-experiment aggregation hyperparameters (dataset push: --category help lists 5 of 9 categories, and unknown-vs-empty category errors give two different lists (one leaks unsupported segmentation) #74) · author: @shujaatTracebloc · no reviewer assigned
• backend#926 — chore(release): promote develop → staging (week of 2026-07-06) · author: @aptracebloc · reviewer: @saadqbal
• backend#927 — feat(validators): schema check for tabular/text/token classification (#924 G2) · author: @divyasinghds · reviewer: @saqlainsyed007
• backend#931 — feat(validators): generic all-category checks + validation matrix (#924 Phase 0/G12) · author: @divyasinghds · reviewer: @saqlainsyed007

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)