[AI Generated] BugFix: iperf3 force source build for versions >= 3.14 to avoid UDP segfault by Gnandeep99 · Pull Request #4445 · microsoft/lisa

Gnandeep99 · 2026-04-29T15:07:58Z

Summary

Force source build of iperf3 from tag 3.10.1 when the distro-installed version is >= 3.14, which has a known segfault in UDP mode with high parallelism (-P 64+). Also passes ref to git.clone() and replaces bare assert with LisaException for better diagnostics.

Validation Results

Image	Result
canonical ubuntu-24_04-lts server 24.04.202404230	PASSED

Copilot

Pull request overview

Updates LISA’s iperf3 tool installation logic to avoid a known UDP segfault in newer distro-packaged iperf3 versions, and adjusts nested VM connection setup to respect the effective host connection address (honoring use_public_address).

Changes:

Detect installed iperf3 version and force a source build from a known-good ref when the installed version is in the reported UDP-segfault range.
Pin git clone to the intended iperf3 source ref and improve JSON parsing failure handling with a LisaException.
Use host.connection_info["address"] (effective address) instead of host.public_address when connecting to nested VMs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
lisa/tools/iperf3.py	Adds iperf3 version detection and conditional source rebuild; pins clone ref; improves JSON parse failure error handling.
lisa/microsoft/testsuites/nested/common.py	Uses the host’s effective connection address for nested VM connectivity to respect `use_public_address`.

Copilot · 2026-04-29T15:27:06Z

+    # iperf3 versions 3.14+ have a known segfault in UDP mode with high
+    # parallelism (-P 64+). Force a source build from _branch when the
+    # distro-installed version falls in this buggy range.


The PR template description is empty. Please add a description explaining what this PR does and why, and (if applicable) link the related issue for traceability.

Copilot · 2026-04-29T15:27:07Z

+    # iperf3 versions 3.14+ have a known segfault in UDP mode with high
+    # parallelism (-P 64+). Force a source build from _branch when the
+    # distro-installed version falls in this buggy range.


The comment about the "known UDP segfault" would be more actionable if it referenced an upstream issue/commit (or internal tracking link) and, if known, the version where it is fixed. This helps future maintainers validate whether the forced downgrade is still necessary.

Suggested change

# iperf3 versions 3.14+ have a known segfault in UDP mode with high

# parallelism (-P 64+). Force a source build from _branch when the

# distro-installed version falls in this buggy range.

# iperf3 versions >= 3.14.0 have been observed to segfault in UDP mode

# with high parallelism (-P 64+), so versions in that range are replaced

# with a source build from _branch. When the upstream issue/commit and the

# first fixed release are confirmed, record them here and remove or narrow

# this workaround only after validating that fixed version in LISA.

Copilot · 2026-04-29T15:27:07Z

+                f"Failed to parse iperf3 JSON output. "
+                f"Raw output (first 500 chars): {result[:500]}"


This exception message explains what happened, but it doesn’t give a next step to investigate/resolve. Consider mentioning common causes (e.g., iperf3 crash/segfault or non-JSON output because -J wasn’t used) and what to check (exit code, iperf3 version, full stdout/stderr/logfile).

Suggested change

f"Failed to parse iperf3 JSON output. "

f"Raw output (first 500 chars): {result[:500]}"

"Failed to parse iperf3 JSON output. "

"This usually means iperf3 produced non-JSON output, such as "

"running without JSON mode enabled, a crash/segfault, or truncated "

f"output. Verify the iperf3 command used JSON output and inspect the "

f"full stdout/stderr or logfile for errors. Raw output (first 500 "

f"chars): {result[:500]}"

Copilot · 2026-04-29T15:27:08Z

+    # Use the host's connection address which respects use_public_address setting.
+    host_address = host.connection_info["address"]
    connection_info = schema.ConnectionInfo(
-        address=host.public_address,
+        address=host_address,
        port=guest_port,


To match the established pattern elsewhere in the repo, consider indexing connection_info with constants.ENVIRONMENTS_NODES_REMOTE_ADDRESS instead of the string literal "address". This avoids typos and keeps key usage consistent.

Copilot · 2026-04-29T15:27:08Z

    # setup connection to nested vm
+    host_address = host.connection_info["address"]
    connection_info = schema.ConnectionInfo(
-        address=host.public_address,
+        address=host_address,
        port=port,


To match the established pattern elsewhere in the repo, consider indexing connection_info with constants.ENVIRONMENTS_NODES_REMOTE_ADDRESS instead of the string literal "address". This avoids typos and keeps key usage consistent.

github-actions · 2026-04-29T15:30:46Z

✅ AI Test Selection — PASSED

1 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

	Count
✅ Passed	1
❌ Failed	0
⏭️ Skipped	0
Total	1

Test case details

Test Case	Status	Time (s)	Message
verify_nested_kvm_basic (lisa_0_0)	✅ PASSED	270.072

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

github-actions · 2026-04-29T16:18:02Z

✅ AI Test Selection — PASSED

4 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

	Count
✅ Passed	3
❌ Failed	0
⏭️ Skipped	1
Total	4

Test case details

Test Case	Status	Time (s)	Message
verify_sriov_interrupts_change (lisa_0_3)	✅ PASSED	223.167
verify_sriov_ethtool_offload_setting (lisa_0_1)	⏭️ SKIPPED	0.000	deployment skipped: awaitable resource 'vm size' is not enough. . AllocationFailed: Allocation failed. We do not have su
verify_irqbalance (lisa_0_2)	✅ PASSED	326.303
verify_device_statistics (lisa_0_0)	✅ PASSED	326.443

github-actions · 2026-04-30T00:45:35Z

✅ AI Test Selection — PASSED

4 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

	Count
✅ Passed	3
❌ Failed	0
⏭️ Skipped	1
Total	4

Test case details

Test Case	Status	Time (s)	Message
verify_sriov_interrupts_change (lisa_0_3)	✅ PASSED	217.526
verify_irqbalance (lisa_0_2)	✅ PASSED	317.981
verify_sriov_ethtool_offload_setting (lisa_0_1)	⏭️ SKIPPED	0.000	deployment skipped: awaitable resource 'vm size' is not enough. . AllocationFailed: Allocation failed. We do not have su
verify_device_statistics (lisa_0_0)	✅ PASSED	319.760

LiliDeng · 2026-05-06T13:43:50Z

@Gnandeep99 please double check the logic, I ran case perf_tcp_iperf_sriov + canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest, I saw the iperf version is 3.9 and the iperf was not upgraded.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

+                # iperf3 >= 3.14 segfaults in UDP mode with high parallelism.
+                # Force a source build from a known-good version.
+                version = self.get_version()
+                if version and version >= self._buggy_version_min:
+                    self._log.info(
+                        f"iperf3 {version} has known UDP segfault, "
+                        f"rebuilding from source ({self._branch})"
+                    )
+                    install_from_src = True


… to avoid UDP segfault

Gnandeep99 · 2026-05-11T02:14:57Z

Hi @LiliDeng, thanks for testing this!

The behavior you saw is actually expected, iperf3 3.9 is below the 3.14 threshold, so the fix intentionally leaves it as-is.
I realized there was no log message explaining this decision, which made it look like the version check never ran. I've just pushed an update that adds a debug log when the installed version is checked and found safe.

Could you give it another run when you get a chance? The logs should now make it clear the check is working.
Thanks!

github-actions · 2026-05-11T02:22:10Z

✅ AI Test Selection — PASSED

4 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

	Count
✅ Passed	4
❌ Failed	0
⏭️ Skipped	0
Total	4

Test case details

Test Case	Status	Time (s)
verify_device_statistics (lisa_0_0)	✅ PASSED	323.061
verify_sriov_interrupts_change (lisa_0_3)	✅ PASSED	237.569
verify_sriov_ethtool_offload_setting (lisa_0_1)	✅ PASSED	289.509
verify_irqbalance (lisa_0_2)	✅ PASSED	338.619

…around - Treat unparseable iperf3 -v output as rebuild trigger so the segfault mitigation cannot be silently bypassed by an unexpected version output format - Expand _buggy_version_min comment with upstream tracking placeholder and first-fixed-release guidance - Enrich _pre_handle LisaException with actionable diagnostic hints (missing -J, crash/segfault, truncated output)

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

lisa/tools/iperf3.py:167

The workaround is implemented inside Iperf3.install(), but Tools.get() only calls tool.install() when tool.exists is False (see lisa/executable.py:660-666). If iperf3 is already present on the image, this new version gate won’t run and a buggy distro iperf3 (>= 3.14) will still be used, defeating the purpose of the change.

Consider moving the version/feature checks into an override of _check_exists() (or exists) so it returns False when the installed iperf3 is >= _buggy_version_min (or lacks --logfile), which will cause Tools.get() to invoke install() and rebuild from source as intended.

    def install(self) -> bool:
        posix_os: Posix = cast(Posix, self.node.os)
        try:
            posix_os.install_packages("iperf3")
        except Exception as e:
            self._log.debug(f"Failed to install iperf3 from package manager: {e}")
        install_from_src = False
        if self._check_exists():
            if "--logfile" not in self.help().stdout:
                install_from_src = True
            else:
                # iperf3 >= 3.14 segfaults in UDP mode with high parallelism.
                # Force a source build from a known-good version. If the
                # version cannot be parsed, fail safe and rebuild so the
                # mitigation cannot be silently bypassed by an unexpected
                # `iperf3 -v` output format.
                version = self.get_version()
                if version is None:
                    self._log.warning(
                        "could not parse iperf3 version output; "
                        f"rebuilding from source ({self._branch}) as a "
                        "precaution against the known UDP segfault"
                    )
                    install_from_src = True
                elif version >= self._buggy_version_min:
                    self._log.info(
                        f"iperf3 {version} has known UDP segfault, "
                        f"rebuilding from source ({self._branch})"
                    )
                    install_from_src = True
                else:
                    self._log.debug(
                        f"iperf3 {version} is below buggy threshold "
                        f"{self._buggy_version_min}, keeping distro version"
                    )
        else:
            install_from_src = True
        if install_from_src:
            self._install_from_src()
        return self._check_exists()

+    # iperf3 versions >= 3.14.0 have been observed to segfault in UDP mode
+    # with high parallelism (-P 64+), so versions in that range are replaced
+    # with a source build from _branch (the last known-good release for the
+    # workloads exercised by LISA).
+    #
+    # Upstream tracking: https://github.com/esnet/iperf/issues  (link the
+    # specific issue/commit here once filed).
+    # First fixed release: TBD - bump _buggy_version_min to that version, or
+    # remove this workaround entirely, once the upstream fix is confirmed.


github-actions · 2026-06-16T06:48:57Z

✅ AI Test Selection — PASSED

4 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

	Count
✅ Passed	4
❌ Failed	0
⏭️ Skipped	0
Total	4

Test case details

Test Case	Status	Time (s)
verify_sriov_interrupts_change (lisa_0_2)	✅ PASSED	215.371
verify_irqbalance (lisa_0_1)	✅ PASSED	319.887
verify_sriov_ethtool_offload_setting (lisa_0_0)	✅ PASSED	345.340
verify_device_statistics (lisa_0_3)	✅ PASSED	321.006

…review - Replace generic 'Upstream tracking' URL and 'TBD' placeholder with an explicit TODO that calls out the missing upstream issue link and first-fixed release, so the action item is harder to forget

github-actions · 2026-06-16T07:04:19Z

✅ AI Test Selection — PASSED

4 test case(s) selected (view run)

Marketplace image: canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest

	Count
✅ Passed	4
❌ Failed	0
⏭️ Skipped	0
Total	4

Test case details

Test Case	Status	Time (s)
verify_sriov_interrupts_change (lisa_0_2)	✅ PASSED	213.435
verify_irqbalance (lisa_0_1)	✅ PASSED	326.623
verify_sriov_ethtool_offload_setting (lisa_0_0)	✅ PASSED	383.282
verify_device_statistics (lisa_0_3)	✅ PASSED	324.598

Copilot AI review requested due to automatic review settings April 29, 2026 15:07

Gnandeep99 requested review from LiliDeng and johnsongeorge-w as code owners April 29, 2026 15:07

Gnandeep99 changed the title ~~Bugfix/iperf3 udp segfault handling 280426 155802~~ iperf3: force source build for versions >= 3.14 to avoid UDP segfault (Bug #51331262) Apr 29, 2026

Copilot started reviewing on behalf of Gnandeep99 April 29, 2026 15:15 View session

Copilot AI reviewed Apr 29, 2026

View reviewed changes

Gnandeep99 changed the title ~~iperf3: force source build for versions >= 3.14 to avoid UDP segfault (Bug #51331262)~~ iperf3: force source build for versions >= 3.14 to avoid UDP segfault Apr 29, 2026

Gnandeep99 force-pushed the bugfix/iperf3-udp-segfault-handling_280426_155802 branch from f690e60 to 319a91f Compare April 29, 2026 15:50

Copilot AI review requested due to automatic review settings April 29, 2026 16:05

Gnandeep99 force-pushed the bugfix/iperf3-udp-segfault-handling_280426_155802 branch from 319a91f to 33c3437 Compare April 29, 2026 16:05

Copilot started reviewing on behalf of Gnandeep99 April 29, 2026 16:07 View session

Copilot AI reviewed Apr 29, 2026

View reviewed changes

Gnandeep99 changed the title ~~iperf3: force source build for versions >= 3.14 to avoid UDP segfault~~ [AI Generated] BugFix: iperf3 force source build for versions >= 3.14 to avoid UDP segfault Apr 29, 2026

Gnandeep99 added the AI_GEN_PR label Apr 29, 2026

Gnandeep99 force-pushed the bugfix/iperf3-udp-segfault-handling_280426_155802 branch from 33c3437 to edabc9c Compare April 30, 2026 00:35

Gnandeep99 requested a review from Copilot May 11, 2026 01:19

Copilot started reviewing on behalf of Gnandeep99 May 11, 2026 01:20 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

[AI Generated] BugFix: iperf3 force source build for versions >= 3.14…

aa5f4e7

… to avoid UDP segfault

Gnandeep99 force-pushed the bugfix/iperf3-udp-segfault-handling_280426_155802 branch from edabc9c to aa5f4e7 Compare May 11, 2026 02:12

Copilot AI review requested due to automatic review settings June 16, 2026 06:39

Copilot started reviewing on behalf of Gnandeep99 June 16, 2026 06:39 View session

Copilot AI reviewed Jun 16, 2026

View reviewed changes

[AI Generated] BugFix: tighten iperf3 workaround comment per Copilot …

982d7ba

…review - Replace generic 'Upstream tracking' URL and 'TBD' placeholder with an explicit TODO that calls out the missing upstream issue link and first-fixed release, so the action item is harder to forget

		f"Failed to parse iperf3 JSON output. "
		f"Raw output (first 500 chars): {result[:500]}"

-                f"Failed to parse iperf3 JSON output. "
-                f"Raw output (first 500 chars): {result[:500]}"
+                "Failed to parse iperf3 JSON output. "
+                "This usually means iperf3 produced non-JSON output, such as "
+                "running without JSON mode enabled, a crash/segfault, or truncated "
+                f"output. Verify the iperf3 command used JSON output and inspect the "
+                f"full stdout/stderr or logfile for errors. Raw output (first 500 "
+                f"chars): {result[:500]}"

Uh oh!

Conversation

Gnandeep99 commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 29, 2026

✅ AI Test Selection — PASSED

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Apr 29, 2026

✅ AI Test Selection — PASSED

Uh oh!

github-actions Bot commented Apr 30, 2026

✅ AI Test Selection — PASSED

Uh oh!

LiliDeng commented May 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Gnandeep99 commented May 11, 2026

Uh oh!

github-actions Bot commented May 11, 2026

✅ AI Test Selection — PASSED

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jun 16, 2026

✅ AI Test Selection — PASSED

Uh oh!

github-actions Bot commented Jun 16, 2026

✅ AI Test Selection — PASSED

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Gnandeep99 commented Apr 29, 2026 •

edited

Loading