Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The operator runs on a single cluster and reaches remote clusters via kubeconfig
- **Multi-cluster WireGuard mesh** — declarative `ClusterMesh` CRD bridges any number of clusters
- **Fork-aware Kilo support** — accepts WireGuard IP annotations in both upstream (`<host>/32`) and Cozystack-patched (`<host>/<subnet-mask>`) form; normalises to host routes automatically
- **Endpoint resolution chain** — per-node endpoint determined by priority: `clustermesh-endpoint` annotation → `force-endpoint` annotation → Node `ExternalIP` combined with `wireguardPort`; nodes with no resolvable endpoint are skipped cleanly
- **Anchor peers** — a single per-cluster anchor `Peer` advertises `serviceCIDR` and `additionalCIDRs` so service and host-network ranges are reachable across clusters
- **Anchor peers** — a single per-cluster anchor `Peer` advertises the `allowedNetworks` entries that no individual node already carries (e.g. service and host-network ranges) so they are reachable across clusters
- **Embedded CRD bootstrap** — the operator self-applies its CRD at startup; no separate CRD pre-install step required
- **Safe cluster reconfiguration** — a change-watcher triggers a controlled pod restart when cluster topology or kubeconfig Secrets change, rebuilding the client registry from scratch
- **Finalizer-based cleanup** — removing a `ClusterMesh` CR triggers deletion of all managed `Peer` objects on every cluster before the resource is released
Expand Down Expand Up @@ -190,18 +190,20 @@ spec:
clusters:
- name: cluster-a
local: true
podCIDRs: ["10.1.0.0/16"]
wireguardCIDR: "10.200.0.0/24"
allowedNetworks: # pod, WireGuard and service CIDRs, all in one flat list
- "10.1.0.0/16"
- "10.200.0.0/24"
- "10.96.0.0/12"
wireguardPort: 51820 # default; set explicitly if your cluster uses a different port
serviceCIDR: "10.96.0.0/12"
- name: cluster-b
kubeconfigSecretRef:
name: cluster-b-kubeconfig
key: kubeconfig
podCIDRs: ["10.2.0.0/16"]
wireguardCIDR: "10.200.1.0/24"
allowedNetworks:
- "10.2.0.0/16"
- "10.200.1.0/24"
- "10.112.0.0/12"
wireguardPort: 51820
serviceCIDR: "10.112.0.0/12"
```

> **Warning:** Pod CIDRs, WireGuard CIDRs, and service CIDRs must not overlap between any two clusters in the same namespace. Overlapping CIDRs block reconciliation for all affected meshes.
Expand All @@ -210,7 +212,7 @@ spec:

## How It Works

On each reconcile cycle, the operator connects to every cluster in the `ClusterMesh` spec, lists all `Node` objects, validates each node's pod CIDR and WireGuard IP against the declared spec, and creates or updates Kilo `Peer` objects accordingly. Nodes that fail validation or have no resolvable endpoint are skipped. For each cluster that declares a `serviceCIDR` or `additionalCIDRs`, an anchor `Peer` carrying those CIDRs is also created on every other cluster. The operator uses a finalizer to clean up all managed peers when a `ClusterMesh` resource is deleted.
On each reconcile cycle, the operator connects to every cluster in the `ClusterMesh` spec, lists all `Node` objects, validates that each node's pod CIDR and WireGuard IP fall within the cluster's declared `allowedNetworks`, and creates or updates Kilo `Peer` objects accordingly. Nodes that fail validation or have no resolvable endpoint are skipped. Any `allowedNetworks` entry that no individual node already advertises (e.g. the service CIDR or host-network ranges) is folded into a single anchor `Peer` on every other cluster. The operator uses a finalizer to clean up all managed peers when a `ClusterMesh` resource is deleted.

See [./docs/architecture.md](./docs/architecture.md) for the full reconciliation flow and component details.

Expand Down
47 changes: 13 additions & 34 deletions api/v1alpha1/clustermesh_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,17 @@ type ClusterEntry struct {
// +optional
KubeconfigSecretRef *SecretKeyRef `json:"kubeconfigSecretRef,omitempty"`

// PodCIDRs is the list of pod network CIDRs for this cluster.
// Node.Spec.PodCIDRs must be a subset of these CIDRs.
// AllowedNetworks is the flat list of every CIDR this cluster contributes
// to the mesh: pod CIDRs, the WireGuard (kilo0) CIDR, the service CIDR,
// host-network ranges, external subnets, and so on. There is no typed
// distinction between them — both validation and Peer construction treat
// every entry uniformly. A node is eligible when its PodCIDR is a subset
// of some entry and its kilo.squat.ai/wireguard-ip host IP falls within
// some entry; entries that have no per-node representative (e.g. the
// service CIDR or host-network ranges) are advertised via the anchor Peer.
// Multiple entries support dual-stack (IPv4 + IPv6).
// +kubebuilder:validation:MinItems=1
PodCIDRs []string `json:"podCIDRs"` //nolint:tagliatelle // "podCIDRs" is the canonical field name; "CIDR" is a well-known acronym

// WireguardCIDR is the CIDR for Kilo's WireGuard interface (kilo0) addresses.
// Each node's kilo.squat.ai/wireguard-ip must have its host IP within this CIDR.
// The annotation may carry any prefix length (e.g. "10.4.0.1/32" upstream Kilo
// or "10.4.0.1/16" cozystack-patched Kilo); only the host portion is validated.
WireguardCIDR string `json:"wireguardCIDR"`
AllowedNetworks []string `json:"allowedNetworks"`

// WireguardPort is the UDP port of Kilo's WireGuard endpoint on each node in
// this cluster. Used as a fallback when the operator synthesises the
Expand All @@ -69,17 +69,6 @@ type ClusterEntry struct {
// +optional
WireguardPort uint16 `json:"wireguardPort,omitempty"`

// ServiceCIDR is the Kubernetes service network CIDR for this cluster.
// If set, it will be advertised via an anchor Peer so that services
// in this cluster are reachable from other mesh members.
// +optional
ServiceCIDR string `json:"serviceCIDR,omitempty"`

// AdditionalCIDRs are extra CIDRs to advertise into the mesh
// (e.g., host-network ranges, external subnets).
// +optional
AdditionalCIDRs []string `json:"additionalCIDRs,omitempty"` //nolint:tagliatelle // "additionalCIDRs" is the canonical field name; "CIDR" is a well-known acronym

// PersistentKeepalive is the interval in seconds at which WireGuard
// sends keepalive packets to peers in this cluster. Set to a non-zero
// value (e.g. 25) for clusters behind NAT so that the stateful NAT
Expand All @@ -92,21 +81,11 @@ type ClusterEntry struct {
PersistentKeepalive int `json:"persistentKeepalive,omitempty"`
}

// AllCIDRs returns the union of all CIDRs declared by this cluster entry.
// The order is: podCIDRs, wireguardCIDR, serviceCIDR (if set), additionalCIDRs.
// AllCIDRs returns every CIDR declared by this cluster entry. With the flat
// model this is simply the AllowedNetworks list; it is consumed by the
// cross-mesh overlap validation, whose behaviour is unchanged.
func (c *ClusterEntry) AllCIDRs() []string {
cidrs := make([]string, 0, len(c.PodCIDRs)+1+1+len(c.AdditionalCIDRs))
cidrs = append(cidrs, c.PodCIDRs...)

cidrs = append(cidrs, c.WireguardCIDR)

if c.ServiceCIDR != "" {
cidrs = append(cidrs, c.ServiceCIDR)
}

cidrs = append(cidrs, c.AdditionalCIDRs...)

return cidrs
return c.AllowedNetworks
}

// SecretKeyRef identifies a key within a Kubernetes Secret.
Expand Down
64 changes: 17 additions & 47 deletions api/v1alpha1/clustermesh_types_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,13 @@ func TestClusterMeshJSONRoundTrip(t *testing.T) {
{
Name: "local-cluster",
Local: true,
PodCIDRs: []string{"10.0.0.0/16", "fd00::/48"},
WireguardCIDR: "172.30.0.0/24",
AllowedNetworks: []string{"10.0.0.0/16", "fd00::/48", "172.30.0.0/24", "10.96.0.0/12", "192.168.100.0/24"},
WireguardPort: 51820,
ServiceCIDR: "10.96.0.0/12",
AdditionalCIDRs: []string{"192.168.100.0/24"},
},
{
Name: "remote-cluster",
KubeconfigSecretRef: secretRef,
PodCIDRs: []string{"10.1.0.0/16"},
WireguardCIDR: "172.30.1.0/24",
AllowedNetworks: []string{"10.1.0.0/16", "172.30.1.0/24"},
WireguardPort: 52000,
},
},
Expand Down Expand Up @@ -119,9 +115,7 @@ func TestClusterMeshDeepCopy(t *testing.T) {
{
Name: "c1",
Local: true,
PodCIDRs: []string{"10.0.0.0/16"},
WireguardCIDR: "172.30.0.0/24",
AdditionalCIDRs: []string{"192.168.0.0/24"},
AllowedNetworks: []string{"10.0.0.0/16", "172.30.0.0/24", "192.168.0.0/24"},
},
},
},
Expand All @@ -132,17 +126,17 @@ func TestClusterMeshDeepCopy(t *testing.T) {

// Mutate the original after copying.
original.Spec.Clusters[0].Name = "mutated"
original.Spec.Clusters[0].PodCIDRs[0] = "99.99.99.0/24"
original.Spec.Clusters[0].AdditionalCIDRs[0] = "1.2.3.0/24"
original.Spec.Clusters[0].AllowedNetworks[0] = "99.99.99.0/24"
original.Spec.Clusters[0].AllowedNetworks[2] = "1.2.3.0/24"

// The copy must be unchanged.
assert.Equal(t, "c1", copied.Spec.Clusters[0].Name)
assert.Equal(t, "10.0.0.0/16", copied.Spec.Clusters[0].PodCIDRs[0])
assert.Equal(t, "192.168.0.0/24", copied.Spec.Clusters[0].AdditionalCIDRs[0])
assert.Equal(t, "10.0.0.0/16", copied.Spec.Clusters[0].AllowedNetworks[0])
assert.Equal(t, "192.168.0.0/24", copied.Spec.Clusters[0].AllowedNetworks[2])
}

// TestClusterEntryAllCIDRs verifies that AllCIDRs returns the correct union
// for various combinations of optional fields.
// TestClusterEntryAllCIDRs verifies that AllCIDRs returns the flat
// AllowedNetworks list verbatim, preserving order.
func TestClusterEntryAllCIDRs(t *testing.T) {
t.Parallel()

Expand All @@ -152,50 +146,26 @@ func TestClusterEntryAllCIDRs(t *testing.T) {
want []string
}{
{
name: "required fields only",
name: "single entry",
entry: v1alpha1.ClusterEntry{
PodCIDRs: []string{"10.0.0.0/16"},
WireguardCIDR: "172.30.0.0/24",
AllowedNetworks: []string{"10.0.0.0/16"},
},
want: []string{"10.0.0.0/16", "172.30.0.0/24"},
},
{
name: "with service CIDR",
entry: v1alpha1.ClusterEntry{
PodCIDRs: []string{"10.0.0.0/16"},
WireguardCIDR: "172.30.0.0/24",
ServiceCIDR: "10.96.0.0/12",
},
want: []string{"10.0.0.0/16", "172.30.0.0/24", "10.96.0.0/12"},
want: []string{"10.0.0.0/16"},
},
{
name: "with additional CIDRs",
name: "pod and wireguard CIDRs",
entry: v1alpha1.ClusterEntry{
PodCIDRs: []string{"10.0.0.0/16"},
WireguardCIDR: "172.30.0.0/24",
AdditionalCIDRs: []string{"192.168.1.0/24", "192.168.2.0/24"},
AllowedNetworks: []string{"10.0.0.0/16", "172.30.0.0/24"},
},
want: []string{"10.0.0.0/16", "172.30.0.0/24", "192.168.1.0/24", "192.168.2.0/24"},
want: []string{"10.0.0.0/16", "172.30.0.0/24"},
},
{
name: "all fields set",
name: "with service and additional CIDRs",
entry: v1alpha1.ClusterEntry{
PodCIDRs: []string{"10.0.0.0/16", "fd00::/48"},
WireguardCIDR: "172.30.0.0/24",
ServiceCIDR: "10.96.0.0/12",
AdditionalCIDRs: []string{"192.168.0.0/24"},
AllowedNetworks: []string{"10.0.0.0/16", "fd00::/48", "172.30.0.0/24", "10.96.0.0/12", "192.168.0.0/24"},
},
want: []string{"10.0.0.0/16", "fd00::/48", "172.30.0.0/24", "10.96.0.0/12", "192.168.0.0/24"},
},
{
name: "empty service CIDR not included",
entry: v1alpha1.ClusterEntry{
PodCIDRs: []string{"10.0.0.0/16"},
WireguardCIDR: "172.30.0.0/24",
ServiceCIDR: "",
},
want: []string{"10.0.0.0/16", "172.30.0.0/24"},
},
}

for _, tc := range tests {
Expand Down
9 changes: 2 additions & 7 deletions api/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 2 additions & 3 deletions cmd/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,8 @@ func sourceFromEntry(meshNamespace string, e kilov1alpha1.ClusterEntry) multiclu

func entry(name string, podCIDR string) kilov1alpha1.ClusterEntry {
return kilov1alpha1.ClusterEntry{
Name: name,
PodCIDRs: []string{podCIDR},
WireguardCIDR: "10.4.0.0/16",
Name: name,
AllowedNetworks: []string{podCIDR, "10.4.0.0/16"},
}
}

Expand Down
39 changes: 12 additions & 27 deletions config/crd/bases/kilo.squat.ai_clustermeshes.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 10 additions & 9 deletions internal/controller/clustermesh_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -795,18 +795,19 @@ func (r *ClusterMeshReconciler) updateStatus(ctx context.Context, mesh *v1alpha1
}

// buildDesiredPeers constructs the desired Peer slice for all valid nodes.
// The first valid node carries the cluster-wide CIDRs (serviceCIDR and any
// AdditionalCIDRs) folded into its Peer.AllowedIPs. The older design emitted
// a separate anchor Peer that reused the anchor node's WireGuard public key;
// WireGuard's per-pubkey dedup made the second `wg setconf` call to apply
// either the node or the anchor entry clobber the AllowedIPs of the other,
// silently losing pod-CIDR or service-CIDR routing in a racy way. Folding
// the anchor CIDRs into the first node Peer keeps a single WG peer entry
// per pubkey with the full union of AllowedIPs.
// The first valid node carries the cluster-wide CIDRs from AllowedNetworks
// that are not covered by any per-node value (e.g. the service CIDR or
// host-network ranges) folded into its Peer.AllowedIPs. The older design
// emitted a separate anchor Peer that reused the anchor node's WireGuard
// public key; WireGuard's per-pubkey dedup made the second `wg setconf` call
// to apply either the node or the anchor entry clobber the AllowedIPs of the
// other, silently losing pod-CIDR or service-CIDR routing in a racy way.
// Folding the anchor CIDRs into the first node Peer keeps a single WG peer
// entry per pubkey with the full union of AllowedIPs.
func buildDesiredPeers(meshName string, entry *v1alpha1.ClusterEntry, nodes []*corev1.Node) ([]*kilov1alpha1.Peer, error) {
peers := make([]*kilov1alpha1.Peer, 0, len(nodes))

anchorExtras := peer.CollectAnchorCIDRs(entry)
anchorExtras := peer.CollectAnchorCIDRs(entry, nodes)

for i, node := range nodes {
var extras []string
Expand Down
Loading
Loading