Full 3D gaming in virtual machine

Status
Not open for further replies.

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
Hi everyone,
This is not a question. I just wanted to share my experience in gaming in a virtual machine.
This is for gamers with advanced knowledge in computers and linux.
My current gaming rig (from start of Nov 2011) is an Asrock Z68 Extreme4 motherboard with an i5-2500 (not K) CPU.
I currently run debian GNU/Linux testing with Xen 4.1.2 hypervisor. It's output is i5's GPU, with a Creative Audigy PCI card. This is enough for what I need under Linux.

I also use an ATI 5850 GPU with a Creative X-Fi Titanium PCIe sound card which have no drivers loaded under linux. These 2, thanks to Intel's VT-d, are controlled exclusively by the Windows 7 virtual machine. Note: VT-d is different than VT-x. VT-x allows virtual machines to run native CPU code, while VT-d maps PCI addresses to guest memory space. VT-d is not present in ix-2xxxK models. At least according to Intel website.
Just as a note, the 5850 is a secondary GPU to the virtal machine. The 1st is the emulated one. Also, input is handled by passing through the 2nd USB2 onboard controller (thank you Intel for not sticking with the 4xUSB1 virtual controllers/1 USB2 controller).

The first game I ever played under this setup is Deus Ex: HR, just 1 day before Skyrim. The next day (11-11-11) I started playing Skyrim, and over 3 week-ends+working days in between I clocked around 80 hours. That is 80 hours in a virtual machine. With some sessions more than 5 hours.

I also used a Realtek 8169 PCI network card, until I changed to the virtualized NIC. The 8169 was the 1st ever device I managed to pass to the VM.

While this is an acceptable solution for me, there are some issues:
- Creative drivers do not seem to work. After installing them, no device is seen in sound control panel (Device manager sees it ok). However, it seems the X-Fi Titanium PCIe can emulate a HDA audio device.
- Have tried onboard audio (Realtek ALC892) and 5850 HDMI audio. All of them (including the X-Fi in HDA emulation) seem to exhibit some clicks&pops, like when the no sample is given for a few 10s of ms. The onboard is the worst, but still playable. The X-Fi is the best. With the onboard audio, sometimes I lost the device completely after 30mins, but restarting the machine would allow me to play for hours.
- Onboard USB3 sort of works (tested with USB2 devices), as in after a certain point, no removals or insertions are detected.
- Marvell onboard SATA controller is detected, but I could not make any HDD visible.
All the issues except for the Marvell are probably due to how IRQs are passed to the VM. I even saw different behaviors while moving the cards around.

My reason for doing all this is that I wanted to release myself from MS OS, but I'm still addicted to games. Please refrain to comment about dual-booting and consoles because:
- dual-booting means I always have to close all apps and restart to play, eventually remaining in Windows to do everything (tried it in the past years)
- consoles do not provide the same experience I got used to for over 10 years of FPS and racing games. They also do not have some of the best racing sims (iRacing, rFactor, GTR) and do not provide the same freedom (think PS3 OtherOS).
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
One more issue I forgot to write: streaming with flash with hardware acceleration is broken. Flash player show green picture, correct audio, 100% 1CPU load and crashes in 30s. Probably also related to IRQ handling. I have not tried other video SW. But unchecking "hardware acceleration" provides good image.
 
Would you mind giving a review of the experience of gaming under those conditions? Do you have frequent crashes or any drawbacks? You mentioned you played many hours with it, but never gave an description of that experience.
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
No crashes whatsoever....Even when the sound had broken (it started by slowing down the sound to maybe half speed) I could still exit the game and shutdown the VM. But I needed patience because the game and some OS apps had extreme timeouts (like 2 mins on exiting the game, and 2 more on shutdown). And even this happened only about 3 or 4 times.

Regarding performance, I played with the setting Skyrim detected (High if I remember correctly), with AF pushed to 16x instead of the default 8x.
I have not done any benchmarks, so I cannot tell you what penalties are there (like 1%, 5% 10%..i have no idea). Also once I got it running, the "native" win7 never got used again.
Windows performance index shows HDD transfer rate at 6.6, everything else above 7 (I don't recall the exact scores, but there were 7.9s).
For the HDD in VM, I actually gave it a complete partition, not a file. It's available though an emulated intel IDE controller, so that would explain the 6.6 score.
 

vitornob

Distinguished
Jun 15, 2008
988
1
19,060
@mathew7

Sincerely, all this trouble, problems, because you don't want to have a Windows 7 copy?

Don't get me wrong. I do love experiments, tried some VM of my own. But games are made to this OS, and they works great! Without clicks, pops, hiccups. Using native OS makes more free resources to app.. etc.. You know this better than anyone.
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
Who sais I don't have a copy? The VM runs Windows 7. Read again. I'm using the native Windows ATI drivers (catalyst 11.11). Just that linux is also running on the same PC, at the same time.
And what resources would you mean? CPU? Well I'm the only one, so I take a little care of linux cpu-hog apps. RAM? I have 8GB, with 6GB to the virtual machine. I have no 64-bit games (they are not made yet), so my games are limited by 4GB anyway, leaving also enough space for the OS. In linux 2GB is enough.

I don't spend all of my PC time in games, so the rest I don't want to spend it in Windows. And belive me, I planned this before I got the HW. It was bought with VT-d in mind. I spent a lot of time during the summer looking for any confirmed VT-d-working motherboards. But most of them were Q35, Q45 boards. I don't even recall seeing 1156 boards.

As a different application, you can think that this is 2 PCs in 1. So by sharing the CPU and MB, you could provide your sibling a gaming PC and for you a work/surf PC. Being in the same room, you could even control his language (yes, I'm talking about 2 monitors, 2 keyboards and 2 mice).

I even want to convice at least one of my friends to visit me with their GPU, so I can try the 2nd gaming VM (2-player LAN with only 1 PC anyone?).
 

vitornob

Distinguished
Jun 15, 2008
988
1
19,060
@mathew7

Don't need to get harsh on commentaries. My point was running games through native Windows 7, not from VM. It would save you a lot of problems.

Isn't that an easy possibility? Since you don't play a lot of games you could boot native Windows 7 (no problems, pops, clips, etc) , play whenever you want, and them go back for linux.

Is Windows for surf/net that bad?!?
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860




Need I say more?

As for the clicks and pops, they are not every second. That is why I said it's acceptable. I could have even 10 mins between succesive pops. And they are very "soft". I think it's just "turning off" because no samples have arrived followed by resuming when they do. It's not the "blow my speakers" kind. I had similar behavior during the Win95 games, in native mode.
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680
This has got to be the most insane coincident ever on this forum; I'm running an extremely similar setup myself (got here through a Google search a few minutes ago, details below)!

I'm in the middle of a setup of Windows Vista in my virtual machine, specs below:
CPU: i7-2600
Mobo: ASRock Z68 Professional Fatal1ty Gen3 (Extreme4 Gen3 was really hard to get where I live)
RAM: Corsair XMS3 8GB
GPU: Sapphire Radeon HD 5850 Toxic 1GB
Sound: Creative X-Fi Professional Fatal1ty PCI
(at this point you might think I just bought the Fatal1ty stuff because I wanted to "feel cool", but trust me, that's not it)
OS: Debian Sid x86_64 + Virualized Windows Vista Ultimate x86_64 (old reused license)
VM: Xen 4.1.2

Now, the problem is that when I try to boot Vista from the HDD I get this error:
(XEN) physdev.c:170: dom15: 16:-1 already mapped to 16
(XEN) physdev.c:170: dom16: 16:-1 already mapped to 16
(XEN) <vm_resume_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1047
(XEN) Domain 16 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-4.1.2 x86_64 debug=n Not tainted ]----
(XEN) CPU: 6
(XEN) RIP: 0000:[<0000000000007b3e>]
(XEN) RFLAGS: 0000000000010296 CONTEXT: hvm guest
(XEN) rax: ffff82c4801044a2 rbx: ffff83020b316000 rcx: 000000000000a100
(XEN) rdx: ffff82c4801bea90 rsi: 0000000000007be4 rdi: ffff82c4801bc34a
(XEN) rbp: ffff82c480178b29 rsp: 0000000000007c00 r8: 00000000000007bd
(XEN) r9: ffff830277a5ff18 r10: 0000000000000000 r11: ffff82c48010447b
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: ffff830277a5ff18 cr0: 0000000000000010 cr4: 0000000000000000
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 07c0 es: 09a0 fs: 0000 gs: 0000 ss: 0000 cs: 0000

I'm using this config:
# Xen configuration file
# Written for Xen 4.1
#
# Currently boots Windows Vista from /dev/sdb

# Name of machine
name = "WinVista"

# RAM (MB)
memory = 4096

# CPU
vcpus = 4
cpus = "4-7"

# Disk drive(s)
disk = [ 'phy:/dev/sdb,ioemu:hda,w', 'phy:/dev/sr1,ioemu:hdc:cdrom,r' ]
#cdrom='/dev/sr0'

# Boot device (c = HDD, d = CDROM)
boot = 'dc'

# Chainloader and emulation layers
kernel = '/usr/lib/xen-default/boot/hvmloader'
device_model = '/usr/lib/xen-default/bin/qemu-dm'
builder = 'hvm'
vnc = 1
sdl = 0
acpi = 1
apic = 1
stdvga=0
serial='pty'

# Networking
#vif = [ 'bridge=xenbr0' ]

# PCI (the fun part)
#gfx_passthru=1 # Enable when Xen 4.2 hits the repo
pci = [ '01:00.0', '01:00.1', '09:01.0']

# Temporary solution until I've figured out mouse forwarding
usbdevice='tablet'

Installation went fine, so I'm not exactly sure why it won't boot from HDD.
It's my first time using Xen, but I'm learning fast.

Anyway, congrats on your setup, I like it a lot ;)
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860


I'm not a Xen expert (I could say I had good luck in making it work), so I have no idea where to start.
I don't think this relates to your problem, but I would sugest to use "01:00.*" instead of "01:00.0"+"01:00.1". In latter case, it would place 2 devices on the virtual bus "0:x.0 and 0:y.0". In my sugestion, the PCI functions are kept. I also append "@x" to force a specific PCI bus assignment. My pci= part (hint:my disk= part is similar):
pci=[]
# Radeon 5850
pci+=['01:00.*@10']
# Realtek 8169 PCI
#pci+=['07:01.0@11']
# Intel EHCI USB controller #2
pci+=['00:1a.0@12']
# Creative XFI
pci+=['02:00.0@13']
# Intel HDA
#pci+=['00:1b.0@13']
# Marvell SATA controller
#pci+=['03:00.*']

My troubleshooting ideas are:
1. does it actually load something from HDD? Maybe a windows logo appears shortly followed by the "recovery" options on the 2nd try? I ask this because you have a "CDROM then HDD" boot order, also your CDROM is /dev/sr1 which is 2nd cdrom (ok..xen would complain if it's not there). So do you boot something from CD 1st? Try with boot = "c".
2. Does your debian GUI autoload windows partitions? I made an "unformatted" partition and that is exported as my VM drive (so it's /dev/sda2 for me). It seems my debian does not recognize a (virtual) MBR as a valid filesystem (huray!). But I wanted to avoid exposing a complete device to the VM just for this reason.
3. Make sure VT-d is working: try with only 1 device, since I said my X-Fi has issues. Also try with no passthough (comment the line).
4. I have no idea how hyperthreading can affect the virtual CPUs. My CPUID reports 4x the cache sizes (like having 4 CPUs, each with only one core active). I think it's a long shot, but you could try with 1 VCPU.
5. I don't recall when I added these, but I have msi=1 in GRUB_CMDLINE_LINUX_DEFAULT (for the dom0 kernel) and also a pci_msitranslate=1 in my VM config file. I don't know if I added them when troubleshooting for sound or troubleshooting for VM starting. But you may try these (5850 can use MSI instead of IRQs).

I can only say that the "16:-1 already mapped to 16" seems to be ok, since I also have it and it works, but it could be IRQ sharing related and the reason for my sound problems.

PS: I'm actually looking at USB sound solutions, but more because I would like an SPDIF-passthrough capable sound card with coaxial output (2nd part is where my X-Fi fails).
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680


Thanks, it boiled down to being a Vista specific issue, XP and 7 were unaffected.
The solution is to first install XP and then install Vista on top of that.
The MBR Vista writes is too buggy for Xen to handle, and there aren't any signs of interest in fixing this since 7 makes Vista mostly obsolete.

The /dev/sr1 part leaved me puzzled as well, but I do have 2 CD drives.
Apparently it switched the order so that the SATA one became sr1 and the IDE one sr0. No biggie.

As for the part about PCI, I think it's a great idea to use wildcards, just didn't know wildcards worked :)
Thanks!

I'm thinking of using a bridged network instead of a forwarded PCI device, looking into it now.

Thanks for your help!
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860

That's the kernel which loads IDE first and SATA after.

You're welcome. It took me some time on Xen wiki to find out.

I used this on Xen Wiki. But this means good-bye too NetworkManager.

Here is my /etcf/network/interfaces:
[cpp]# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface xenbr0 inet dhcp
bridge_ports eth0
[/cpp]

Initially it was "iface eth0 inet dhcp" (see the difference?).
Then you can use your commented "vif" line (maybe with added MAC?).
 

gaurdianaq

Distinguished
Dec 29, 2011
5
0
18,510
This is very intriguing, so you were able to run games in a VM with a decent framerate? I'm assuming you need to have the specific type of cpu and motherboard to do this then...

I'd be very interested to see this kind of technology in future virtual machines and cpu's/motherboards as I would very much like to switch over to linux...
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680


Thanks again, got it all working now!

The thing is, before I installed Xen the SATA drive was detected first.
Not really worth looking into I'd guess, as long as they're both found and working the order doesn't really matter.




Yes, you need both a CPU and motherboard which supports VT-d, it's a set of instructions that allows you to access physical devices through a VM.
The problem is, not too many are using the VT-d feature on their home PCs, which results is lots and lots of buggy implementations.

The motherboards we chose have been tested by some brave souls who bought them on chance and then tested the actual VT-d support.
There's no simple way to know from the specs and/or manual, you have to either do your research or take the risk of suffering from critical bugs.

As for the CPU, pretty much all of the mid to high end non-K Intel i5/i7 CPUs supports VT-d.
I think AMD Phenom II and Bulldozer has it too, but then it goes under the name IOMMU (the proper name in my opinion).

This isn't all too new, it's just that Xen has evolved enough to reach a state where it's not a pain to set up a forwarded GPU.
There are alternatives to Xen as well, I've read about similar attempts with QEMU + KVM or VMWare ESXi.

I think we need some big tech site to make an article on it, the power of VT-d is mostly unknown to the average geek.

By all means, if you have the interest and the hardware, go for it!
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860


Actually there are no instructions, as opposed to VT-x (which is pure CPU dependent). For VT-d, just the memory controller receives a MMU for PCI(e) devices (thus IOMMU; CPU already has a MMU for applications since 386).
For Core2Duo, I think any processor could do it if you had a VT-d chipset (only Qxx chipsets, Q35 or newer) and MB BIOS implementations, even though I see the 1st CPUs with VT-d are E8xxx and Q9xxx, launched in Q1/Q2 2008, while the Q35 was Q3 2007 (...speculations begin now.....).

But for the Core ix platforms, Intel moved the memory controller to the CPU package (with which the IOMMU is directly linked), so now the CPU is the main "supporter" (HW-wise), the BIOS being responsible just for some initial configurations (and being exposed to MB maker shortcuts, even if the "option" is present).

On AMD side, I have no idea, since not many developers jump that side, unless management forces them. (Hint: I said "not many developers", not "no developers")
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680
I got the impression that for AMD systems, IOMMU is in the motherboard but the CPU needs to be aware of it.
This got me to the conclusion that IOMMU required additional and/or modified instructions, either in the CPU and/or the chipset.

Of course, for Intel systems VT-d is in part in the CPU, as you stated, since the memory controller is in the CPU package.

AMD has a nice illustration on page 19 in this document on IOMMU.

From what I've read, AMD boards tend to have proper enough IOMMU support more often than Intel boards have proper VT-d.
These IOMMU capable boards usually have 880, 890 or 990 series chipsets, but I think there was one with 870.
To anyone with one of above chipsets; this might already be possible for you, check dmesg for entries about IOMMU.
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
Interesting. Maybe the older i5&i7s could use a similar architecture, since the IOMMU needs to be between memory controller and PCI/PCIe bus. But the 2nd gen core-i's definitely have it in the CPU package because of the integrated PCIe lanes.
As for the "awareness", it's done by programming the other HW's (memory controller and bus controller). So no new CPU instructions are used, just some additional initializations done in SW, by BIOS and/or OS (with info from BIOS).
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680
I have hit another issue that previously passed by unnoticed; I have internet connection from my dom0 and my domU, but not between them.

/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface eth0 inet manual

# The xenbr0 interface, replaces eth0 for dom0
auto xenbr0
iface xenbr0 inet dhcp
bridge_ports eth0

# NetworkManager doesn't like Xen very much anyway
#allow-hotplug eth0
#NetworkManager#iface eth0 inet dhcp

iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-out vif5.0 --physdev-is-bridged
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif5.0 --physdev-is-bridged

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Google didn't give me anything useful :(
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
If you want a simple bridge to external network (just like a separate device on the network), then loose the "iface eth0...". Did you make those 2 iptables entries? If yes, loose those also.
Check what "vif-script" is active in /etc/xen/xend-config.xsp. I have "(vif-script vif-bridge)" uncommented. I don't recall the default, but I know I played with it.
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680


Nah, I just posted the iptables output because it seemed to be useful in troubleshooting as in http://wiki.kartbuilding.net/index.php/Xen_Networking .
I'm trying to use what looks like the 1st case.

I commented the eth0 line, but I also figured out I was trying to ping the wrong IP from my domU, using the correct one works.
However, I still can't ping my domU from dom0, which has left me really puzzled this time.
Also notable is that my domU won't get an IP from DHCP, so far I've been using a static IP which gave me access to the entire network, or so it seemed.

Does Vista have ping disabled in its firewall?

P.S.
Turns out disabling Vistas firewall blocked ping.
I feel the pain of the worlds IT admins.
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860

What kind of DHCP do you use? You should be aware that if you don't provide a MAC on the "vif" parameter, Xen generates a different MAC for each session. So if you have a MAC-restricting DHCP server, that would be a problem.
Regarding Vista firewall, I never actually tried ping, but one of the 1st things I installed was tightvnc (both of my video outputs go to the same monitor), and I had to disable the FW to make it work (ok...I could have put an exception, but I wanted a fast solution).
But I still don't get your P.S.: when the firewall is disabled, ping is not allowed? By default all Win firewalls disable ping responses.
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680
Alright, I've got everything working now, except for that I can't use Synergy for UAC dialog boxes.

I'm looking into having a dedicated mouse hidden behind the keyboard for this purpose.
The question now is; forward a single USB device or a whole controller.

Since I also want to use my Logitech G25, forwarding a controller like you did might have the advantage.
Any experience on device vs controller?
 

mathew7

Distinguished
Jun 3, 2011
295
0
18,860
I played with device forwarding on virtualbox and it was a nightmare because it can only be done in SW, so a generic driver needs to be forced in the host. I would sugest using VT-d on one controller.
You can check which ports are under which controller by putting an usb device in every (2nd) port and see with lsusb on which bus they appear. Usually (have not seen any exception) 2 stacked ports will be on the same controller.
My MB has all USB2.0 on the back port and 1 (2.0) header on 1st USB controller and the other 2 (2.0) headers on the 2nd 2.0 controller.
I've had trouble with the USB3 ports (no linux driver and bad VT-d experience), so I'm not using them (I have 4).
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680


The P.S. was a typo, it was supposed to be "Turns out disabling Vistas firewall allowed ping.", guess that's what stressful days does to my brain ;)

The MAC address is locked, did that immediately when I found out it changed for every boot.
The router utilizes static DHCP to keep track of our IP addresses, however the virtual machine doesn't really need DHCP.
Keeping track of a single machine is easy enough.
Anyway, after adding a firewall rule for IMCPv4 (to allow ping), most issues seem to be sorted.

Now there's another issue; sometimes the domU crashes the dom0.
I have been unable to find any indirect cause, but it seems to be boot related.
It crashed on me once on a reboot while I was away, and once in the background while booting.

I would guess this is yet another Vista specific issue, in retrospect I probably should had gone with 7 instead...
 

Djhg2000

Distinguished
May 16, 2009
165
0
18,680
It seems I have successfully got rid of the crashing issue.
I simply disabled the sound card in device manager.
Later tests suggests that it's one of the Creative X-Fi enhancements poking around in weird memory areas.
I used the latest drivers for my card, version 2.18.0015 .

In case anyone else had thought of using this particular card, it doesn't look too good.
I will try to get a USB sound card and report the outcome.

On the topic of USB, I have now decided to use one of the controllers, but I can't find which PCI device to forward.
I have looked trough "lspci -v", "lsusb" and "lshw" and I'm either missing something obvious or looking at the wrong commands.

I could try poking out the controllers one by one over SSH, but to me that feels a bit like trying to drive an F1 car with shotgun rounds; it will eventually stop, but it might not do so in a mint condition.

Overall, I'm pretty satisfied with this setup apart from the sound and lack of USB connectivity.
Xen 4.2 will bring us the ability to get rid of that Cirrus graphics adapter as well, but for now disabling it in the device manager works well enough.
 
Status
Not open for further replies.