HDDS-11063. TestSnapshotDiffManager#testThreadPoolIsFull is flaky without wait between batches

[rhalm]

What changes were proposed in this pull request?

Previously in HDDS-10604 there were changes made to OzoneConfiguration initialization that briefly increased the cost of constructing a new instance, which caused TestSnapshotDiffManager#testThreadPoolIsFull to fail in the no-wait-between-batches scenario. That change was reverted later, but it exposed two issues that are addressed by this PR:

• testThreadPoolIsFull relied on timing (using Thread.sleep) to verify pool rejection behavior under load, making the test fragile to any potential latency in the snapshot diff submission path (such as ozone config initialization). The test was rewritten to use a CountDownLatch, making the IN_PROGRESS/REJECTED split deterministic.
• getSnapshotRootPath constructed a new OzoneConfiguration() on every call just to build a path. It now uses ozoneManager.getConfiguration() instead.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11063

How was this patch tested?

• Reproduced the fragility by adding a 10 ms delay to the OzoneConfiguration constructor: the old test fails on master, the fixed test consistently passes with the same setup.
• CI run: https://github.com/rhalm/ozone/actions/runs/27956879008
• flaky-test-check on this branch: https://github.com/rhalm/ozone/actions/runs/27959048712 (all passed)

[SaketaChalamchala]

Thanks for the patch @rhalm.
nit: Use full_thread_pool_size = 2 * OZONE_OM_SNAPSHOT_DIFF_THREAD_POOL_SIZE and 45 - full_thread_pool_size for the expected accepted and rejected jobs here.
Otherwise, LGTM.

[rhalm]

Thanks for the review @SaketaChalamchala, done.

[SaketaChalamchala]

Not in scope for the current PR to fix flakiness but may be considered as a follow up test improvement:
In the drainBetweenBatches scenario would a better check be to

1. Keep the latch closed until full_thread_pool_size jobs are submitted
2. Open the latch
3. Submit new jobs when totalSubmitted - completedJobs.get() < full_thread_pool_size

[rhalm]

Great idea, I opened HDDS-15648 as a follow-up.

[adoroszlai]

Thanks @SaketaChalamchala for the review. When requesting changes, CI workflow should not be approved on the PR, since it will anyway need to be re-run after addressing review comments.