I have not had any time to hack on this further. I can share some info on what it will take to get the MacPro1,1 & 1,2 working. The problem is with the bootloader on the install media and the one that is written to disk durring install. Replacing these files with ones that are compatible with the older MacPro will result in a working ESXi 5.0 install.
1. Setup a bootable USB stick formatted fat32 and clone the esxi .iso contents to the stick.
2. Replace the boot files on the stick with working ones
3. Boot from the USB stick and install ESXi 5.0
4. Boot back in to OS X and mount the efi boot partition
mount_msdos /dev/disk0s1 /Volumes/gpt
5. Replace the boot files on this partition with working ones
6. Reboot the system holding the option key down to select the ESXi volume
Now the hard part is getting and configuring the working boot files. I’m guessing that you could take a build of EFI GRUB2 and configure it to work as the needed bootloader. What we are missing is the details on how to configure it.
Here is a source for the files needed.
I have tested this process with some boot files I have on hand from the ESXi 5 beta. However I can’t share those files due to VMware’s licensing. From my poking around it looks like replacing the files with open source MacPro compatible ones will do the trick.
I have seen some issues with the networking on the MacPro2,1 but I don’t know if the issue is my box specifically or ESXi not working well with it’s nic.
Somebody with GRUB skills or knowledge of configuring it to boot ESXi should be able to get it working.