Skip to content

HDDS-15639. Adjust Upgrade status command to call OM intead of SCM #10579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sodonnel wants to merge 6 commits into
base: HDDS-14496-zdu
Choose a base branch
from
+287 −26
Open

HDDS-15639. Adjust Upgrade status command to call OM intead of SCM #10579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
13a79f8
Initial changes
Jun 22, 2026
7c8d1ca
Add new command to the request processing
Jun 22, 2026
76240d5
Add more tests
Jun 22, 2026
c17f3bd
Add server version check
Jun 22, 2026
87c5298
Add deprecated labels to legacy query command protobuf
Jun 22, 2026
d225723
Fix pmd
Jun 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 36 additions & 14 deletions ...ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/upgrade/StatusSubCommand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,32 +17,54 @@

package org.apache.hadoop.ozone.admin.upgrade;

import java.io.IOException;
import java.util.concurrent.Callable;
import org.apache.hadoop.hdds.cli.AbstractSubcommand;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
import org.apache.hadoop.hdds.scm.client.ScmClient;
import org.apache.hadoop.ozone.OzoneManagerVersion;
import org.apache.hadoop.ozone.admin.om.OmAddressOptions;
import org.apache.hadoop.ozone.client.rpc.RpcClient;
import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import picocli.CommandLine;

/**
* Sub command to query the overall upgrade status of the cluster, returning information about the finalization
* status of SCM, the datanodes and OM.
* status of SCM, the datanodes and OM. The command makes a single call to OM which returns the status of the other
* components as well as itself.
*/
@CommandLine.Command(
name = "status",
description = "Show status of the cluster upgrade",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
public class StatusSubCommand extends ScmSubcommand {
public class StatusSubCommand extends AbstractSubcommand implements Callable<Integer> {

@CommandLine.Mixin
private OmAddressOptions.OptionalServiceIdOrHostMixin omAddressOptions;

@Override
public void execute(ScmClient client) throws IOException {
HddsProtos.UpgradeStatus status = client.queryUpgradeStatus();

out().println("Upgrade status:");
out().println(" SCM Finalized: " + status.getScmFinalized());
out().println(" Datanodes finalized: " + status.getNumDatanodesFinalized());
out().println(" Total Datanodes: " + status.getNumDatanodesTotal());
out().println(" Should Finalize: " + status.getShouldFinalize());
public Integer call() throws Exception {
try (OzoneManagerProtocol client = getClient()) {
OzoneManagerVersion omVersion = RpcClient.getOmVersion(client.getServiceInfo());
if (!OzoneManagerVersion.ZDU.isSupportedBy(omVersion)) {
err().println("OM does not support ZDU. The cluster upgrade status should be queried with the pre ZDU " +
"commands, eg `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
Comment on lines +50 to +51

@errose28 errose28 Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
err().println("OM does not support ZDU. The cluster upgrade status should be queried with the pre ZDU " +
"commands, eg `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
err().println("OM does not support zero downtime upgrade. The cluster upgrade status should be queried with `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");

return 1;
}
OzoneManagerProtocolProtos.QueryUpgradeStatusResponse status = client.queryUpgradeStatus();

out().println("Upgrade status:");
out().println(" OM Finalized: " + status.getOmFinalized());
out().println(" SCM Finalized: " + status.getHddsStatus().getScmFinalized());
out().println(" Datanodes finalized: " + status.getHddsStatus().getNumDatanodesFinalized());

@dombizita dombizita Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's worth to phrase it differently, as the first two line are true/false and this is the number of finalized datanodes, right?

@errose28 errose28 Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, maybe just using a question mark for the boolean and colon for the count? There will be a json option added next.

OM Finalized? false
SCM Finalized? true
Datanodes Finalized: 3/10

out().println(" Total Datanodes: " + status.getHddsStatus().getNumDatanodesTotal());
out().println(" Should Finalize: " + status.getHddsStatus().getShouldFinalize());

@errose28 errose28 Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just an indicator to OM from SCM. I don't think we should display it to the user. Maybe if we add a --verbose flag later we can display this along with the software and apparent versions of the components.

}
return 0;
}

protected OzoneManagerProtocol getClient() throws Exception {
return omAddressOptions.newClient();
}

}
74 changes: 64 additions & 10 deletions ...e/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/upgrade/TestStatusSubCommand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,25 @@

package org.apache.hadoop.ozone.admin.upgrade;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.PrintStream;
import java.io.UnsupportedEncodingException;
import java.nio.charset.StandardCharsets;
import java.util.Collections;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
import org.apache.hadoop.hdds.scm.client.ScmClient;
import org.apache.hadoop.ozone.OzoneManagerVersion;
import org.apache.hadoop.ozone.om.helpers.ServiceInfo;
import org.apache.hadoop.ozone.om.helpers.ServiceInfoEx;
import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
Expand All @@ -42,40 +49,87 @@ public class TestStatusSubCommand {
private static final String DEFAULT_ENCODING = StandardCharsets.UTF_8.name();

private final ByteArrayOutputStream outContent = new ByteArrayOutputStream();
private final ByteArrayOutputStream errContent = new ByteArrayOutputStream();
private final PrintStream originalOut = System.out;
private final PrintStream originalErr = System.err;
private OzoneManagerProtocol omClient;
private StatusSubCommand cmd;

@BeforeEach
public void setup() throws UnsupportedEncodingException {
cmd = new StatusSubCommand();
public void setup() throws IOException {
omClient = mock(OzoneManagerProtocol.class);
when(omClient.getServiceInfo()).thenReturn(serviceInfoWithVersion(OzoneManagerVersion.ZDU));

cmd = new StatusSubCommand() {
@Override
protected OzoneManagerProtocol getClient() throws Exception {
return omClient;
}
};
System.setOut(new PrintStream(outContent, false, DEFAULT_ENCODING));
System.setErr(new PrintStream(errContent, false, DEFAULT_ENCODING));
}

@AfterEach
public void tearDown() {
System.setOut(originalOut);
System.setErr(originalErr);
}

@Test
public void testStatusCommandPrintsUpgradeStatus() throws IOException {
ScmClient scmClient = mock(ScmClient.class);
HddsProtos.UpgradeStatus status = HddsProtos.UpgradeStatus.newBuilder()
public void testStatusCommandPrintsUpgradeStatus() throws Exception {
HddsProtos.UpgradeStatus hddsStatus = HddsProtos.UpgradeStatus.newBuilder()
.setScmFinalized(false)
.setNumDatanodesFinalized(1)
.setNumDatanodesTotal(3)
.setShouldFinalize(true)
.build();
when(scmClient.queryUpgradeStatus()).thenReturn(status);

OzoneManagerProtocolProtos.QueryUpgradeStatusResponse response =
OzoneManagerProtocolProtos.QueryUpgradeStatusResponse.newBuilder()
.setOmFinalized(false)
.setHddsStatus(hddsStatus)
.build();

when(omClient.queryUpgradeStatus()).thenReturn(response);
new CommandLine(cmd).parseArgs();
cmd.execute(scmClient);
cmd.call();

String output = outContent.toString(DEFAULT_ENCODING);
assertTrue(output.contains("Upgrade status:"));
assertTrue(output.contains("OM Finalized: false"));
assertTrue(output.contains("SCM Finalized: false"));
assertTrue(output.contains("Datanodes finalized: 1"));
assertTrue(output.contains("Total Datanodes: 3"));
assertTrue(output.contains("Should Finalize: true"));
verify(scmClient).queryUpgradeStatus();
verify(omClient).queryUpgradeStatus();
}

@Test
public void testStatusCommandPropagatesException() throws Exception {
when(omClient.queryUpgradeStatus()).thenThrow(new IOException("OM unavailable"));
new CommandLine(cmd).parseArgs();
assertThrows(IOException.class, () -> cmd.call());
}

@Test
public void testNonZduServerPrintsErrorAndReturnsNonZero() throws Exception {
when(omClient.getServiceInfo()).thenReturn(serviceInfoWithVersion(OzoneManagerVersion.DEFAULT_VERSION));

new CommandLine(cmd).parseArgs();
assertEquals(1, cmd.call());

String errOutput = errContent.toString(DEFAULT_ENCODING);
assertTrue(errOutput.contains("OM does not support ZDU"));
verify(omClient, never()).finalizeUpgrade();

@dombizita dombizita Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this check queryUpgradeStatus()?

Suggested change
verify(omClient, never()).finalizeUpgrade();
verify(omClient, never()).queryUpgradeStatus();

}

private ServiceInfoEx serviceInfoWithVersion(OzoneManagerVersion version) {
ServiceInfo serviceInfo = new ServiceInfo.Builder()
.setNodeType(HddsProtos.NodeType.OM)
.setHostname("localhost")
.setOmVersion(version)
.build();
return new ServiceInfoEx(Collections.singletonList(serviceInfo), "", Collections.emptyList());
}
}
2 changes: 2 additions & 0 deletions hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,7 @@ public static boolean isReadOnly(OMRequest omRequest) {
case GetObjectTagging:
case GetQuotaRepairStatus:
case StartQuotaRepair:
case QueryUpgradeStatus:
return true;
case CreateVolume:
case SetVolumeProperty:
Expand Down Expand Up @@ -455,6 +456,7 @@ public static boolean shouldSendToFollower(OMRequest omRequest) {
// Quota repair lifecycle request should be initiated by the leader
case DBUpdates: // We are currently only interested on the leader DB info
case UnknownCommand:
case QueryUpgradeStatus:
return false;
case EchoRPC:
return omRequest.getEchoRPCRequest().getReadOnly();
Expand Down
8 changes: 8 additions & 0 deletions ...-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,14 @@ ListOpenFilesResult listOpenFiles(String path, int maxKeys, String contToken)
*/
void finalizeUpgrade() throws IOException;

/**
* Returns the upgrade status of the cluster. This call is received by OM which will in turn query SCM to get the
* status of it and the datanodes, and return the combined status to the caller.
* @return QueryUpgradeStatusResponse containing details of the overall cluster state
* @throws IOException If any error occurs.
*/
OzoneManagerProtocolProtos.QueryUpgradeStatusResponse queryUpgradeStatus() throws IOException;

/**
* Queries the current status of finalization.
* This method when called, returns the status messages from the finalization
Expand Down
12 changes: 12 additions & 0 deletions ...ava/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2073,6 +2073,18 @@ public StatusAndMessages queryUpgradeFinalizationProgress(
);
}

@Override
public OzoneManagerProtocolProtos.QueryUpgradeStatusResponse queryUpgradeStatus() throws IOException {
OzoneManagerProtocolProtos.QueryUpgradeStatusRequest
req = OzoneManagerProtocolProtos.QueryUpgradeStatusRequest.newBuilder().build();

OMRequest omRequest = createOMRequest(Type.QueryUpgradeStatus)
.setQueryUpgradeStatusRequest(req)
.build();

return handleError(submitRequest(omRequest)).getQueryUpgradeStatusResponse();
}

/**
* Get a valid Delegation Token.
*
Expand Down
16 changes: 14 additions & 2 deletions hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ enum Type {
DeleteObjectTagging = 142;
SubmitSnapshotDiff = 143;
StartFinalizeUpgrade = 144;
QueryUpgradeStatus = 145;
}

enum SafeMode {
Expand Down Expand Up @@ -217,7 +218,7 @@ message OMRequest {
optional ServiceListRequest serviceListRequest = 51;
optional DBUpdatesRequest dbUpdatesRequest = 53;
optional FinalizeUpgradeRequest finalizeUpgradeRequest = 54 [deprecated = true];
optional FinalizeUpgradeProgressRequest finalizeUpgradeProgressRequest = 55;
optional FinalizeUpgradeProgressRequest finalizeUpgradeProgressRequest = 55 [deprecated = true];
optional PrepareRequest prepareRequest = 56;
optional PrepareStatusRequest prepareStatusRequest = 57;
optional CancelPrepareRequest cancelPrepareRequest = 58;
Expand Down Expand Up @@ -311,6 +312,7 @@ message OMRequest {

optional SubmitSnapshotDiffRequest submitSnapshotDiffRequest = 144;
optional StartFinalizeUpgradeRequest startFinalizeUpgradeRequest = 145;
optional QueryUpgradeStatusRequest queryUpgradeStatusRequest = 146;
}

message OMResponse {
Expand Down Expand Up @@ -361,7 +363,7 @@ message OMResponse {
optional ServiceListResponse ServiceListResponse = 51;
optional DBUpdatesResponse dbUpdatesResponse = 52;
optional FinalizeUpgradeResponse finalizeUpgradeResponse = 54 [deprecated = true];
optional FinalizeUpgradeProgressResponse finalizeUpgradeProgressResponse = 55;
optional FinalizeUpgradeProgressResponse finalizeUpgradeProgressResponse = 55 [deprecated = true];
optional PrepareResponse prepareResponse = 56;
optional PrepareStatusResponse prepareStatusResponse = 57;
optional CancelPrepareResponse cancelPrepareResponse = 58;
Expand Down Expand Up @@ -447,6 +449,7 @@ message OMResponse {

optional SubmitSnapshotDiffResponse submitSnapshotDiffResponse = 143;
optional StartFinalizeUpgradeResponse startFinalizeUpgradeResponse = 144;
optional QueryUpgradeStatusResponse queryUpgradeStatusResponse = 145;
}

enum Status {
Expand Down Expand Up @@ -1640,6 +1643,15 @@ message StartFinalizeUpgradeRequest {
message StartFinalizeUpgradeResponse {
}

message QueryUpgradeStatusRequest {
}

message QueryUpgradeStatusResponse {
required bool omFinalized = 1;

@errose28 errose28 Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and its SCM equivalent will need the software and apparent versions as well for two use cases:

  • Showing with a --verbose option to the CLI and maybe by default with the --json option.
  • Querying by leader to peers to ensure their versions match before finalizing.

required hadoop.hdds.UpgradeStatus hddsStatus = 2;
Comment on lines +1650 to +1651

@errose28 errose28 Jun 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For new messages we have been making everything optional since there is no required in proto3. It is up to the server to enforce the contract.

}

// deprecated, use QueryUpgradeStatusRequest/Response instead. Retained for wire compatibility.
message FinalizeUpgradeProgressRequest {
// Ignored by OM; retained for wire compatibility.
required string upgradeClientId = 1;
Expand Down
11 changes: 11 additions & 0 deletions hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,7 @@
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ExtendedUserAccessIdInfo;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.KeyArgs;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.OMRoleInfo;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.QueryUpgradeStatusResponse;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.S3Authentication;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ServicePort;
import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.TenantState;
Expand Down Expand Up @@ -3648,6 +3649,16 @@ public void finalizeUpgrade() throws IOException {
throw new UnsupportedOperationException();
}

@Override
public QueryUpgradeStatusResponse queryUpgradeStatus() throws IOException {
HddsProtos.UpgradeStatus scmStatus = scmClient.getContainerClient().queryUpgradeStatus();

return QueryUpgradeStatusResponse.newBuilder()
.setOmFinalized(!versionManager.needsFinalization())
.setHddsStatus(scmStatus)
.build();
}

@Override
public StatusAndMessages queryUpgradeFinalizationProgress(
String unusedUpgradeClientId, boolean unusedTakeover, boolean unusedReadonly)
Expand Down
5 changes: 5 additions & 0 deletions ...-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,11 @@ public OMResponse handleReadRequest(OMRequest request) {
getObjectTagging(request.getGetObjectTaggingRequest());
responseBuilder.setGetObjectTaggingResponse(getObjectTaggingResponse);
break;
case QueryUpgradeStatus:
OzoneManagerProtocolProtos.QueryUpgradeStatusResponse queryUpgradeStatusResponse =
getOzoneManager().queryUpgradeStatus();
responseBuilder.setQueryUpgradeStatusResponse(queryUpgradeStatusResponse);
break;
default:
responseBuilder.setSuccess(false);
responseBuilder.setMessage("Unrecognized Command Type: " + cmdType);
Expand Down
Loading