Bug Description

During secret rotation the driver removes the old Docker Swarm secret immediately after triggering the service update. Since Docker Swarm performs rolling updates asynchronously, some tasks may still be running with the old secret when it gets removed.

Relevant code:

createResponse, err := d.dockerClient.SecretCreate(ctx, newSecretSpec)

if err := d.updateServicesSecretReference(secretName, newSecretName, createResponse.ID); err != nil {
    return fmt.Errorf("failed to update services: %v", err)
}

if err := d.dockerClient.SecretRemove(ctx, existingSecret.ID); err != nil {
    log.Warnf("Failed to remove old secret version %s: %v", existingSecret.ID, err)
}

Steps to Reproduce

1. Create a service that uses a secret.
2. Scale the service to multiple replicas.
3. Trigger secret rotation.

Expected Behavior

The old secret should only be removed after services have fully switched to the new secret.

Actual Behavior

The old secret is removed immediately after the service update is triggered.

Reproducibility

• Sometimes
• Always
• Rarely
• Unable to reproduce