add new commit statistics metrics

[MarkSh1]

Rebase #10993
Co-authored-by: @oleg68 Oleg Samarin osamarin@openintegration.inc

Problem statement

Now it is hard to tune a fdb cluster for a write-intensive workload.

Description

While tuning a fdb cluster with a write-intensive application often the bottleneck is the commit latency: when trying to parallel degree of transactions payload, the commit latency grows and prevents increasing the transaction throughput.

There are lots of conditions and knobs influencing the commit latency: number of commit proxies, number of resolvers, number of tlog processes, commit batching knobs: MAX_COMMIT_BATCH_INTERVAL, COMMIT_TRANSACTION_BATCH_INTERVAL_MAX, COMMIT_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA and others.

But for now, there is no any information, where is the root cause of the high commit latency, so it is unclear, what is to be changed.

Proposal

• To collect and to log latency statistics from parts of commit workflow:
  • Waiting for a batch
  • Preresolution (allocating a commit version)
  • Resolution
  • Postresolution
  • Pushing to TLog
  • Replying
• To collect and log the batch size statistics
  • The number of transactions in one batch
  • The total bytes in the transaction batch

PR content

This PR implements this proposal: the following new metrics are logged and exposed in status json:

• CommitBatchTransactions (commit_batch_transactions) - the number of transactions in one batch
• CommitBatchBytes (commit_batch_bytes) - the total size of one commit batch in bytes
• CommitBatchingWaiting (commit_batching_waiting) - the time while the transaction is waiting for the batch becomes ready
• CommitPreresolutionLatency (commit_preresolution_latency) - the time of the Preresolution phase
• CommitResolutionLatency (commit_resolution_latency) - the time of the Resolution phase
• CommitPostResolutionLatency (commit_resolution_latency) - the time of the Postresolution phase
• CommitTLogLoggingLatency (commit_tlog_logging_latency) - the time of the TlogLogging phase
• CommitReplyLatency (commit_reply_latency) - the time of Reply phase

The sum of the added *_latency_mean metrics should be equal to the commit_latency_mean metrics that already exists

[MarkSh1]

@jzhou77, could you please take a look at the PR?

[ploxiln]

@jzhou77 does not work on FoundationDB anymore (but he is still welcome to, of course!)

Should you perhaps add to the commit message(s):

Co-Authored-By: Oleg Samarin <osamarin@openintegration.inc>

[MarkSh1]

@ploxiln , I don't have permission to add reviewers myself. Could someone from the FoundationDB team please take a look at this PR when possible? Thank you!

[Ronitsabhaya75]

@ploxiln , I don't have permission to add reviewers myself. Could someone from the FoundationDB team please take a look at this PR when possible? Thank you!

@MarkSh1 you dont need any permission for adding co-author to your commit message.

git commit -m "XYZ" -m "Co-authored-by: xyz <xyz@example.com>"

you can basically amend the co-author name + email associate with their github account and then rebase it and force push.

[MarkSh1]

@Ronitsabhaya75, Thanks! I was actually asking about adding reviewers, not co-authors. I've already added the Co-authored-by line as suggested by @ploxiln. My question was about how to request a formal code review since I don't have permission to assign reviewers myself.

[Ronitsabhaya75]

@Ronitsabhaya75, Thanks! I was actually asking about adding reviewers, not co-authors. I've already added the Co-authored-by line as suggested by @ploxiln. My question was about how to request a formal code review since I don't have permission to assign reviewers myself.

it would be picked by apple employees as they are the ones who has write access so I'd recommend wait haha :)