HA Lighthouse Agent support (Leader Election)

[msft-paddy14]

lighthouse-agent runs as a single replica. If the pod is evicted or a node fails/OOM issues etc. Service discovery synchronization stops until a replacement pod starts and re-lists all resources.

Proposal

Add leader election using coordination/v1 Leases so multiple replicas can run simultaneously. The leader processes resources; standby replicas keep informer caches warm. On failover, the new leader starts processing immediately with no cold-start delay.

Approach:

Syncers in Admiral already expose ShouldProcess hook, we can use that to make a decision on Leader to process the event and skip it on Replica. Ensuring the informer caches stay warm but we don't process events on both instances.
Probably can just add support for passive replica initially and build on top of that for hot standby replica as optimization.

Configuration:
The replica count should probably come from the operator itself as it configures this. Currently the count seems hardcoded?
https://github.com/submariner-io/submariner-operator/blob/928b75e213138a0daec74c32683a0253905de1e6/internal/controllers/servicediscovery/servicediscovery_controller.go#L241

Open Challenges:

1. If we have a standby replica that gets the events and skips processing and we fail over, the events might stay in the informer cache and never be updated to API server. Might need to force drain post failover somehow

[dfarrell07]

We talked about this quite a bit on the scrub call today, @tpantelis and @vthapar in particular. They have some questions about potential new bugs this could introduce, I'll let them raise/discuss. Basically if the hot standby is necessary and if it introduces new issues vs what issues you're trying to solve. Also if a re-sync is needed. Are you planning on working on this? Thanks!

[tpantelis]

Implementing leader election with passive replica is reasonable and is the standard, proven pattern for HA in Kubernetes controllers. The "hot standby with warm informer caches" idea is interesting but would add complexity and potential risks and race conditions for probably marginal benefit over a cold start. On fail-over, we'd have to essentially trigger a re-sync in the informers (at least re-queue all items) to ensure no events are missed, which is effectively a cold start. The "warm cache" approach would only save the network overhead of the initial API server LIST operation (typically on the order of 1-2 seconds) - but all resources would still need to be re-processed for consistency. Also hot standby replica(s) would increase resource consumption in terms of API server watch load.

[msft-paddy14]

Hi, sorry for the late response.
The standby will be watching for the events so it should in theory have all the data locally, I'm not sure if there are cases which might need full sync but that would guarantee correctness for sure. It might be better to validate the network overhead before we think about the hot standby since there might be other issues as you pointed out.
To start with, it does sound good to have passive replica to ensure availability incase there are scheduling issues/eviction issues