Hi all, I think I’m running into some issues with Windows 11 as a VM on Proxmox on my Epyc hardware (HL15 2.0, RomeD8-2T board) that seems to be CPU related. It shows up in 1) General VM performance, and 2) VM-to-VM networking.
Based on my searching, it appears that there may be some fairly deep issues with Windows 11 VM’s on Proxmox 8 and 9 that seem to be difficult to get around. Something about memory/NUMA architecture, but it’s a bit beyond me. Has anyone else run into these issues?
More specific details regarding these issues:
-
General VM performance
I’m using Windows with GPU passthrough for Davinci Resolve work, mainly - renders and transcodes. I’ve tested various CPU types for the VM. Host generally provides the worst performance, but critically, allows WSL (VM nesting) to work. The ‘EPYC-Rome-v4’ CPU type provides the best performance, but WSL doesn’t work. The areas that performance differences show up the most are in Cinebench and random disk I/O benchmarks.
I’ve tried quite a few CPU options and flags and recorded benchmarks, but nothing seems to get close to ‘native’ performance with the host CPU type while having WSL work. The best solution I’ve found so far is a custom CPU type documented here (more discussion here). It has the best performance I’ve found yet while allowing WSL to work, but still isn’t what ‘host’ would allow. Some more discussion on the issue. -
VM to VM network performance.
This is odd, and is what makes me think that perhaps there is a issue specific to Epyc and Proxmox. I have a TrueNAS VM as well, to which I’ve passed through the HBA for all my disks, as well as NVME for L2ARC and Special VDEV.
I have a network bridge setup in Proxmox, not attached to any physical NIC, that allows the Windows 11 VM to access the SMB shares from TrueNAS. Jumbo frames enabled, RSS enabled in Windows and TrueNAS, multiqueue enabled in Proxmox, using VirtIO drivers. I only get ~22gbit/s between the VM’s on this platform according to iperf3, whereas it seems like other platforms can do double - even quadruple - that. I know it’s CPU-bound, but I have EPYC 7282 16 Cores / 32 Threads, which shouldn’t be a slouch. Testing throughput across the bridge with AJA Disk Speed Test / Blackmagic Disk Speed test gives me only about 1.3GB/s. In comparison, my MacBook Pro M4 Max, connected over a 25Gbps NIC to my TrueNAS VM, gets the expected ~2.6GB/s from my pools (though that’s probably just hitting the 25gbps link limit).
The rabbit hole on this one seems to go much deeper, and in my research, seems to have something to do with the CPU on Proxmox, which is what makes me think that it’s related to the general performance issue as well. Could be something to do with NUMA setup, but in my messing around I haven’t had this help performance at all. xpc-ng had an extremely long discussion that resulted in a patch. Some discussions on the issue around the internet -
In conclusion - anyone else running Windows 11 VM on Proxmox with Epyc? What’s your experience been? Run into anything similar?