Red Screen of Death on HPE ProLiant Gen9 servers with SN260 NVMe drives

README.md

Red Screen of Death on HPE ProLiant Gen9 servers with SN260 NVMe drives

Context

• Three HP ML350 Gen9 servers
• One HP DL360 Gen9 server
• One HP MicroServer Gen8 server
• Six HGST SN260 HHHL NVMe cards

Problem

When at least one HGST SN260 HHHL NVMe card is inserted in a Gen9 server, the server fails POST with a Red Screen of Death (RSOD). See YouTube Video.

Error message:

X64 Exception Type 0D - General Protection Exception

The call stack contains references to HpSmbiosDxe.

The server fails to boot entirely (RBSU or Boot menu is not even reachable).

But, the HP MicroServer Gen8 does not exibit the issue.

Technical details

• BIOS P92 v2.92 (11/23/2021), up-to-date with latest SPP
• iLO up-to-date with last version post-SPP
• NVMe card with up-to-date firmware 122

Attempts to solve the issue

Unsuccessful attempts to solve the issue:

• Only one NVMe card per server
• Card inserted in a slot connected to CPU1, then CPU2
• Switch to Legacy boot
• Switch back to UEFI boot
• Disable "UEFI Boot Optimization" in RBSU
• Remove other PCIe cards from server
• Downgrade the NVMe card firmware to version 111
• Reformat the NVMe namespace with LBA Format 0 (512B blocks) instead of 4K blocks
• Reset BIOS to manufacturing defaults (in RBSU)
• Clear NVRAM and CMOS using the Maintenance Switch 6
• Insert card into the sole PCIe 2.0 slot of the server
• Server Security > Embedded Diagnostics: Enabled -> Disabled
• Boot options > UEFI Optimized Boot: Enabled -> Disabled
• Embedded UEFI Shell > Embedded UEFI Shell: Enabled -> Disabled
• System Options > Boot Time Optimization > Dynamic Power Capping Functionnality: Enabled -> Disabled
• Advanced (Ctrl-A or Ctrl-Q) > Advanced Crash Dump Mode: Disabled -> Enabled
• iLO 4 Configuration > Settings Options > iLO 4 Functionality: Enabled -> Disabled
• Advanced > UEFI Assert: Disabled -> Enabled
• Advanced > PCI Express 64-bits BAR Support: Enabled -> Disabled
• Have a look in the hidden menus of RBSU (see above)
• Try to find an HP firmware for the NVMe drive (crossflash): HP NVMe Drives, best matching drive seems to be VO003840KWVMU / P10216-B21.
• Downgrade the BIOS using maintenance switches S1, S5 and S6
• Disable HPE Intelligent Provisioning
• Investigate Storage options in RBSU

Documentation

• Gen9 RBSU Guide
• HPE ProLiant DL325 Gen10 Plus - UEFI Embedded Diagnostics 6.9.5.0183 Fails With a Red Screen of Death (RSOD) When Two or More NVMe HDDs Are Installed
• DL380 GEN9 - NVME Enablement Express Bay kit question

Forums

• STH
• reddit.com/r/homelab

rsod1.png

rsod2.png

I am experiencing the same issue, did you ever get this fixed?

I get various versions of this issue, but this one is one of them.

I never found a way to fix it !

Little more info for future readers: somehow, this occurs when I install a 3rd PCIe (HP branded Intel i350 4x 1Gbit NIC, in this case) card, but when I remove it, everything is fine. I have two other NICs installed, so that will have to do for now. Bit of a bummer, but it is whatever.

I never found a way to fix it !

ah, that's too bad :(