bsdkm x86_vecreg: nest counter, and other improvements.

[philljj]

Description

Add a nesting counter max (WOLFKMOD_FPU_MAX_NEST 128) for fpu state management in freebsdkm, and a few small improvements.

Fixes F-655.

note: typical nesting values when running benchmark / wolfcrypt test with intel-asm are 2-3 at most. Anything much larger would indicate runaway behavior, like infinite looping, recursion, etc.

[Copilot]

Pull request overview

This PR updates the FreeBSD kernel-module vector/FPU state management helpers to better guard against runaway save/restore nesting and to slightly optimize error-path branching.

Changes:

• Add WOLFKMOD_FPU_MAX_NEST and enforce an upper bound on fpu_states[].nest during nested wolfkmod_vecreg_save() calls.
• Add __predict_false(...) annotations to several error checks to bias branch prediction toward the success path.

Comments suppressed due to low confidence (1)

bsdkm/x86_vecreg.c:197

• On the error path after calling wolfkmod_fpu_kern_enter(), the function returns EINVAL without calling wolfkmod_fpu_kern_leave(). That can leave PCB_KERNFPU set and keep migration/preemption restrictions in effect for the current thread, potentially causing hard-to-debug scheduler/FPU state issues.

        if (__predict_false(fpu_states[PCPU_GET(cpuid)].nest != 0 ||
            td_tid != 0)) {
            printf("error: wolfkmod_fpu_kern_enter() with nest: %d, %d\n",
                   fpu_states[PCPU_GET(cpuid)].nest, td_tid);
            return (EINVAL);
        }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[philljj]

Retest this please.

java.io.IOException: Unexpected EOF.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10987

Scan targets checked: bsdkm-bugs, bsdkm-src

No new issues found in the changed files. ✅