Fix: WolfCrypt 7 fenrir fixes

[aidankeefe2022]

Description

Fixed a series of Fenrir issues that mostly pertained to alignment UB when reading byte arrays as various sizes of integers.

https://fenrir.wolfssl.com/finding/6438

FSPSM RSA private-decrypt never updates caller's *outLen (passes &outLen instead of outLen)

• Swapped the double reference to just pass the outLen ptr

https://fenrir.wolfssl.com/finding/5414

SLH-DSA AVX2 x4 SHAKE helpers cast hash byte streams to word64

• Changed from direct cast byte -> word64 / word64 -> byte to use Unaligned memory helpers

https://fenrir.wolfssl.com/finding/5415

ML-DSA little-endian pack/unpack paths cast encoded byte buffers to word pointers

• Fixed a variety of possible alignment issues.

https://fenrir.wolfssl.com/finding/6165

scrypt scryptROMix big-endian Integerify shifts bytes without width cast (signed overflow/oversized shift on 16-bit int)

• Explicitly cast all intermediate values to word32 to make sure that there is no chance of signed in overflow during bit shifting.

https://fenrir.wolfssl.com/finding/6166

wc_ecc_init_id (SE050) reinterprets byte id[] array as word32 via pointer cast

• Used unaligned helper function to read byte array as word32

https://fenrir.wolfssl.com/finding/6167

XMSS hash-address byte buffers cast to word32* in address-encoding helpers (unaligned access / strict aliasing)

• Removed cast to use endian independent operation that copies the byte value in to the last byte because the word is stored in network byte order

https://fenrir.wolfssl.com/finding/6177

PKCS#12 parse leaves decrypted SafeContents (incl. plaintext private KeyBag) in freed heap buffer

• Added new variable to track temp buffer size and ForceZero it when it is freed

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10808

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10808

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-crypto-only

• FLASH: .text +64 B (+0.0%, 173,738 B / 262,144 B, total: 66% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +64 B (+0.0%, 768,196 B / 1,048,576 B, total: 73% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +64 B (+0.0%, 278,000 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +64 B (+0.0%, 278,576 B / 1,048,576 B, total: 27% used)

stm32-sim-stm32h753

• FLASH: .text +64 B (+0.0%, 181,748 B / 2,097,152 B, total: 9% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10808

Scan targets checked: wolfcrypt-bugs, wolfcrypt-port-bugs, wolfcrypt-src

No new issues found in the changed files. ✅