fixed unability to run more than one commit proxy in small configurations

[MarkSh1]

Rebase PR #10411
Co-authored-by: @oleg68 Oleg Samarin [osamarin@openintegration.inc]

Fixed unability to run more than one commit proxy in small configurations

Problem statement

By default, the desired nubrer of commit proxies is 3. But in small configurations with 3 stateless processes (and any number of transaction and storage processes) foundationdb runs only one commit proxy process, that may cause some performance issues with high write volume.

Any attempts to change the number of proxies with configure commit_proxies=2 and so on do not help: it change only the Desired commit proxies value shown by the status fdbcli command, but do not change the actual number of commit proxies.

Steps to reproduce

1. Create an FDB cluster
2. Setup three stateless server, processes and, for example, three transaction and three storage processes
3. Execute fdbcli --exec "status json" | grep commit_proxy

Expected result

Three lines

                        "role" : "commit_proxy"
                        "role" : "commit_proxy"
                        "role" : "commit_proxy"

Actual result

Only one line

                        "role" : "commit_proxy"

Findings

As far as I understand a list of worker candidates is determined by the ClusterControllerData::getWorkersForRoleInDatacenter method.

The reason why it doesn't return more than one worker is the condition id_used[it.first] <= minWorker.get().used that is never satisfied with small number of stateless processes.

Proposal

1. Fill out the worker candidate list fitness_workers regardless how much they are used.
2. Fill out the result worker list with desired number of workers with the lowerst usage count.

Proposed PR implements these ideas. It

• introduces the used_fitness local variable that is filled by the lowest fitness value of availdable processes
• addes filtering by used_fitness when the result list is filled to use only workers with the smallest fitness value
• eliminates the condition id_used[it.first] <= minWorker.get().used

After applying this PR all desired number of commit proxies is recruited as long as there is a sufficient number number of stateless processes.