GPU die straps for GPU#2 (power side)
There is a video made by YouTuber Craft Computing where he does the Tesla K10 to K2 mod
Craft Computing YouTube recommendation is where my entire Tesla K10 journey has started. I got Tesla K10 from eBay that was already converted to Grid K2 by someone.
It works in bare-metal windows machine in a PC tower, however my goal was to use it inside of the Proxmox potato server I have. vGPU does not work under KVM (unless it is RHEL) and I have never succeeded in passthrough of the Grid K2 to the VM, unresolvable code 43. I basically gave up on K2 and my next goal is to get full hardware mode of the K10 into K5000, however I don't think it is necessary anymore.
I have created another way to do it without hardware mode.
====SOLUTION BELOW====
Solution is still under development and tested on GPU #1 ("video port" side) only on Tesla K10. GPU #2 (power connector side) and new rom test is required, however my GPU#2 requires repairs, thus I will test it only later in April when I get my components.
Disclaimer: I am not an Assembly programmer nor GPU designer (yet? still studying to become one). Whatever is written below was just my final most dumb approach (what if it just works?).
0. GPU differences:
GPU: ------- CUDA? ------- 3D enabled ------- Video out? ------- QEMU KVM?
K5000 ------ yes ------------- yes -------------- yes -------------- yes
Tesla K10 --- yes ------------ no ---------------- no --------------- yes
Grid K2 ----- no ------------- yes --------------- no ---------------- no (RHEL KVM works)
So, the goal is to bring 3D-acceleration to the Tesla K10 in one form or another...
1. Project was initially inspired by krutav's PCIe ID spoofing method, because, in my opinion, it was not tested enough.
What do we know?
PCIe ID change using hardware mode enables some of the features, but not all of them... 660Ti or some other cards converted to Grid K2 does not enable all features of the Grid K2.
PCIe ID spoofing using purely Proxmox enables some of the features, but again, not all of them... Probably, KVM limitation, probably something else.
My PCIe ID spoofing results of softstraping Tesla K10 to K5000:
DirectX: NO
Vulkan: YES (partially)
NVENC: yes
CUDA: yes
CAD: yes
What is my goal?
Make it work as true Quadro K5000.
What prevents other GPUs (including K10) from being a true hacked target GPU (e.g. K5000)?
If you check GTX780ti schematic I have sent before and you check page 28 (ROM and straps) you will see that GPU die straps are responsible for the bunch of secondary GPU settings that are not PCIe ID related. VGA enable and many other ones. I believe, some of them may affect the final GPU performance.
Additionally, my assumption that GPU's vBIOS also heavily affects final performance, because it is obviously not just VRAM config and CLK config. My assumption. that vBIOS also controls GPU functionality and allows to enable or disable features.
This assumption comes from some old Plex Media Server NVENC hack from habr.com written by some Russian user, who hacked nvidia to run multiple NVENC sessions on GeForce.
While Grid K2 and Tesla K10 share 99.99% same PCB (except for 3rd nibble resistor), Tesla K10 and VGA-enabled GPUs (GTX680, K5000 etc) do not share same strap resistor config pattern (GTX780ti, I can also drop other Kepler schematics, I have GTX770 that uses GK104), thus configuration of the Tesla K10 to K5000 may not be that simple as I initially thought.
2. What is my idea?
What if I can flash K5000 vBIOS to K10 without doing a hardware mod?
Long time ago in this thread, some users mentioned rewriting vBIOS and replacing PCIe IDs and PCIe subsystem IDs using hex editor to change values and KeplerBiosTweak to verify the checksum. That's exactly what I decided to do!
I took K5000 vBIOS and replaced PCIe and subsystem IDs of the K5000 with Tesla K10 IDs. Why? Because nvflash tool cannot even force-flash a card if hardware IDs are incorrect.
So, my solution is to flash K5000 BIOS with K10 IDs and fixed checksum. ROM is attached below for anyone who is willing to play with this. Only Tesla K10 is supported and ONLY GPU #1 is tested right now.
Use nvflash tool with mismatched ID patch.
Command:
nvflash.exe --index=0 -6 k10-1-5000.rom
Install your card under KVM based VM using krutav's method.
args: -cpu 'host,hv_time,kvm=off' -device 'vfio-pci,host=06:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=quadrok5000.rom,x-pci-vendor-id=0x10de,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x0965'
Remember to change the host-0X:00.0 to your GPU IDs using lspci -v command in terminal.
Also, quadrok5000.rom is the real K5000 vBIOS from TechPowerUp, not a modified version below.
Install drivers for K5000 either using default .exe (Windows) / .run (Linux) or Quadro Experience. It just works.
3. Did it work?
Kinda. More yes than no.
DirectX: supported now both under games and benchmarks (DirectX < 12), older games written in DirectX9 tested and work
Vulkan: works (as much as it can)
OpenGL: works
3D rendering: works (Blender and PhotoView360 tested)
NVENC: fully enabled (including cloud streaming e.g. Parsec Remote Gaming)
CUDA: works
CADs: work (SolidWorks 2020)
4. Conclusion:
I plan to do same mod to the second vBIOS of the K10 and run two VMs under Proxmox with full GPU passthrough.
Whatever bug inside drivers/Proxmox/vBIOS compare to original krutav's method actually encloses your true GPU to the system and Quadro Experience, MSI Afterburner and games/benchmarks see it as Tesla K10, while Windows sees it as K5000.
vBIOS is available below:
https://github.com/amidg/teslak10-3d-enabled