Skip to content

[#695] Fix race in TraditionalWorkQueue.isIdle()#688

Open
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:features/workqueue-isidle-race
Open

[#695] Fix race in TraditionalWorkQueue.isIdle()#688
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:features/workqueue-isidle-race

Conversation

@vharseko

@vharseko vharseko commented Jul 4, 2026

Copy link
Copy Markdown
Member

Fixes #695

Problem

TraditionalWorkQueue.isIdle() checked that the operation queue is empty and that no worker thread is active (operation != null). A worker thread is invisible to both checks in the window between taking an operation from the queue (opQueue.poll() inside retryNextOperation) and assigning it to its operation field in TraditionalWorkerThread.run(). waitUntilIdle() polls isIdle() every millisecond, so it could return immediately while a just-submitted operation was still being handed off to a worker.

This made TraditionalWorkQueueTestCase.testWaitUntilIdleSlowOpInProgress (from the slow group, excluded from CI) flaky: the test submits an operation delayed by 5 seconds and expects waitUntilIdle(10000) to block for at least 4 seconds, but it occasionally returned right away.

Fix

Track the number of accepted-but-not-fully-processed operations in a dedicated AtomicInteger:

  • incremented in submitOperation() before the operation is enqueued, and rolled back if the operation is rejected (queue full, shutdown, interrupt);
  • decremented via the new package-private operationDone(), called from a try/finally around the processing block in TraditionalWorkerThread (covers the normal path, the transaction paths and exceptions);
  • decremented for operations cancelled while the queue is being resized (interrupted re-submission path);
  • reset in initializeWorkQueue().

isIdle() now simply checks that this counter is zero, covering queued operations, in-flight operations and the hand-off window in between.

The four waitUntilIdle* tests are moved out of the slow group into the default build (~16s for the class).

Verification

TraditionalWorkQueueTestCase run 5 times in a row under the default CI group filter: 6/6 green each time.

isIdle() checked that the operation queue is empty and that no worker
thread is active, but a worker is invisible to both checks in the
window between taking an operation from the queue and assigning it to
its operation field. waitUntilIdle() could therefore return
immediately while the just-submitted operation was still being handed
off to a worker, which made
TraditionalWorkQueueTestCase.testWaitUntilIdleSlowOpInProgress flaky.

Track the number of accepted-but-not-fully-processed operations in a
dedicated counter: incremented before an operation is enqueued (and
rolled back if it is rejected), decremented when a worker thread is
done with it, and adjusted for operations cancelled while the queue is
being resized. isIdle() now simply checks that this counter is zero.

Move the four waitUntilIdle tests out of the "slow" group into the
default build.
@vharseko vharseko changed the title Fix race in TraditionalWorkQueue.isIdle() [#695] Fix race in TraditionalWorkQueue.isIdle() Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TraditionalWorkQueue.isIdle() misses operations in the queue-to-worker hand-off window

2 participants