Socket 1366 MB with IOMUU compatibility and decent OC

[Furi0usGe0rge]

I have a ci7 920 on a Rampage II Gene, which I like, but unfortunately it does not support IOMMU.

I'm looking for a MB that does, but I'd rather not buy a new CPU too (unless it is cheaper than buying a compatible mb).

I'm pretty easily able to get a 40% CPU OC and 25% RAM OC (over non-oc ram speed), so I'd like to get a MB with decent OC features.

Beyond that, if I still get to have my druthers, I could use ATX or larger form factor for the extra PCIe bays.

This if for GPU passthrough to KVM VMs in Linux.

Any help is much appreciated.

 

[Furi0usGe0rge]

GA-X58A-UD3R looks like a possibility.

Thinking about it more, there are a couple of other things I would like, but are not deal breakers ny any means. Dual LAN would be great. So would a board that had an extra PCIe slot even when all the PCIe16s are taken.

The Gigabyte board above does neither of those.

Another problem might be that I must necessarily get a very expensive board for it to be dual pcie16 and have IOMMU and be LGA 1366.

I'm going to start looking into newer sockets and CPU combos.

I will probably buy a new CPU anyway, since I have enough parts to make two.

 

[Furi0usGe0rge]

Working backwards.... I'm buying a 990x on ebay. Now I need a socket 1366 MB to support it with SLI and IOMMU.

 

[GhislainG]

A 1366 motherboard is VT-d (IOMMU is AMD), but your i7-920 may not support it. https://www.manualowl.com/m/Asus/Rampage-II-GENE/Manual/271232?page=94 If VT-d is available, then you should enable it.

 

[Furi0usGe0rge]

A 1366 motherboard is VT-d (IOMMU is AMD), but your i7-920 may not support it. https://www.manualowl.com/m/Asus/Rampage-II-GENE/Manual/271232?page=94 If VT-d is available, then you should enable it.

It's enabled, but trying to enable it in the OS causes the CPU to hardlock almost before the boot even starts.

https://pve.proxmox.com/wiki/Pci_passthrough#Intel_CPU

(Vt-d can be called IOMMU on intel boards, at least according to the linux kernel)

The chip supports it, according to Intel.
http://ark.intel.com/products/37147/Intel-Core-i7-920-Processor-8M-Cache-2_66-GHz-4_80-GTs-Intel-QPI

I do currently run KVM (obviously) so to that extent the CPU support is shown to be working. However, for VFIO/Vt-d to work, the board must support it, as well as the BIOS. I believe what you are demonstrating is that the Rampage II Gene supports Vt-x, not full Vt-d support.

https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0

No reports of anyone getting it working with the Rampage II Gene, or any other Rampage prior to the IV Extreme

In addition, many ASUS boards that do try to implement VT-d, such as the Rampage III, get it wrong in the BIOS.

(can't find link right now)

... and ASUS doesn't appear interested in fixing it:

https://communities.intel.com/thread/28389?start=0&tstart=0

I purchased a GA-EX58-UD5 (not 3 as I mentioned above) based on that list and similar parts on hand to some of the setups therein.

Also, I get a second NIC and extra PCI ports even with 3 GPU! What luxury.

On another forum someone mentioned issues with the current linux kernel and vfio support is broken for x58 chipsets. I'm not using the current latest however.

 

[GhislainG]

I've seen that Intel page for the i7-920, but I didn't find VT-d in the specs. It also isn't on the list of processors that include the VT-d feature: http://ark.intel.com/search/advanced?VTD=true&MarketSegment=DT As an example, my old E8400 has it, but a much newer i3-4170 doesn't. I can use VT-d on my i7-3770 system because both the CPU and the motherboard support it.

According to the manual that I linked, you motherboard supports VT-x (Intel Virtualization Tech on page 3-22) and VT-d (Intel VT-d on page 3-24).

At the https://communities.intel.com/thread/28389?start=0&tstart=0 link, the poster is using an i7-920 which, unless proven otherwise, doesn't seem to support VT-d.

 

[GhislainG]

Will you buy a Xeon X5550 (or similar) with official VT-d support if the GA-EX58-UD5 also doesn't support VT-d with the i7-920?

 

[Furi0usGe0rge]

A 1366 motherboard is VT-d (IOMMU is AMD), but your i7-920 may not support it. https://www.manualowl.com/m/Asus/Rampage-II-GENE/Manual/271232?page=94 If VT-d is available, then you should enable it.

It is enabled on the MB but it is not full Vt-d.

While I can run KVM instances (proof that the CPU supports it), if I pass the intel IOMMU kernel option from the article in my distro's wiki, it hard locks.

Here is the CPU spec from intel showing support: http://ark.intel.com/products/37147/Intel-Core-i7-920-Processor-8M-Cache-2_66-GHz-4_80-GTs-Intel-QPI

All it takes is a few Google searches to see that no one has ever gotten or passthru working on this board. As I said before, only the Rampage IV series has it, and that's a full two generations later.

As much as I'd like it to work, it won't.

 

[GhislainG]

The Intel link shows VT-x (Intel Virtualization Technology); it doesn't show VT-d (Intel Virtualization Technology for Directed I/O).

An E8400 has VT-d: http://ark.intel.com/products/33910/Intel-Core2-Duo-Processor-E8400-6M-Cache-3_00-GHz-1333-MHz-FSB?q=e8400
So does an i5-4590: http://ark.intel.com/products/80815/Intel-Core-i5-4590-Processor-6M-Cache-up-to-3_70-GHz

If it doesn't say VT-d, then you shouldn't presume that Intel made a mistake on their web page.

The motherboard should work with a Xeon processor that supports VT-d.

 

[Furi0usGe0rge]

I very much stand corrected.

I'm gonna keep the GB board anyway. I can migrate vms back and forth and spread some of the load around.

Thanks for your persistence.

 

[Furi0usGe0rge]

The Intel link shows VT-x (Intel Virtualization Technology); it doesn't show VT-d (Intel Virtualization Technology for Directed I/O).

An E8400 has VT-d: http://ark.intel.com/products/33910/Intel-Core2-Duo-Processor-E8400-6M-Cache-3_00-GHz-1333-MHz-FSB?q=e8400
So does an i5-4590: http://ark.intel.com/products/80815/Intel-Core-i5-4590-Processor-6M-Cache-up-to-3_70-GHz

If it doesn't say VT-d, then you shouldn't presume that Intel made a mistake on their web page.

The motherboard should work with a Xeon processor that supports VT-d.

And on second look, it gores back to that link I could not find earlier on. Here it is: http://lime-technology.com/forum/index.php?topic=32978.10;wap2

At one point the ARK page did say yes, but they took it off because with 1366 boards it depended on the mb implementation. On other boards it did not. The best source I can find for that is above, but there are several reports of people getting passthrough working with 920s.

Anyway, it looks like I'm back to looking at the board as an issue. The existing 920 will come in handy for flashing BIOSs to latest version to support the x5670.

 

[GhislainG]

Did you have a chance to try the i7-920 with the Gigabyte motherboard?

 

[Furi0usGe0rge]

Did you have a chance to try the i7-920 with the Gigabyte motherboard?

It will arrive on Monday. I'll also update the BIOS on Gene to latest and see if there is any improvement. I don't have much faith that it will ever work, but I'm curious.

I'll keep you posted.

Thanks again.

 

[GhislainG]

Thanks to you for letting us know because it may help other posters in the future. Not that many people use passthrough.

 

[Furi0usGe0rge]

Here's the update:

I have a GA-EX58-UD5, with a beta BIOS that supports VT-d: http://ggts.gigabyte.com/FileList/1134809/ex58ud5.13s

I also have a Xeon E5570, just to be extra sure VT-d is supported.

I'm using Debian latest with Proxmox installed, and as such I'm following this doc: https://pve.proxmox.com/wiki/Pci_passthrough

If I DO NOT enable VT-d in the bios and boot with the intel_iommu=on parameter in grub.comf, I am able to boot. When I get to a console, I can see IOMMU and DMAR:

dmesg | grep -e DMAR -e IOMMU
[ 0.000000] DMAR: IOMMU enabled
[ 0.029681] DMAR-IR: This system BIOS has enabled interrupt remapping

But when I run the "dmesg | grep ecap" command, nothing gets returned, indicating something is wrong with IRQ remapping.

Similarly, I get nothing back form "find /sys/kernel/iommu_groups/ -type l", so this setup is definitely not right.

If I enable VT-d in the bios AND leave in the intel_iommu=on grub parameter -- as instructed in the official doc -- I get a hardlock very early on in the boot, just like with the Rampage II Gene / ci7-920.

As a third option, I leave VT-d on, and remove the intel_iommu=on parameter. In this case, not surprisingly, IOMMU is no longer enabled, but DMAR is much busier:

# dmesg |grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x00000000CFEE94C0 0000B0 (v01 IntelR AWRDACPI 322E3030 DRWA 00000002)
[ 0.029719] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 0.714911] DMAR: Host address width 36
[ 0.714913] DMAR: DRHD base: 0x000000fe711000 flags: 0x0
[ 0.714927] DMAR: dmar0: reg_base_addr fe711000 ver 1:0 cap c9008010e60262 ecap f0207a
[ 0.714928] DMAR: DRHD base: 0x000000fe710000 flags: 0x1
[ 0.714934] DMAR: dmar1: reg_base_addr fe710000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 0.714935] DMAR: RMRR base: 0x000000cfef0000 end: 0x000000cfefffff

Although I do now have remappable interrupts, as demonstrated by "ecap f020fe" and "ecap f0207a" above (specifically the trailing character a-f), just like with IOMMU enabled and VT-d not, I have no IOMMU groups in /sys/kernel/iommu_groups.

I'm out of time for the day, and I'm not sure where to begin troubleshooting tomorrow. It seems that I can have either iommu or VT-d enabled, but if I enable VT-d then I cannot have iommu.

I know people have gotten this working, with this MB, and in Linux as well. Tomorrow I'll start looking into bugs. I read in another thread that vfio on X58 can be a problem with some linux kernels. Since the kernel didn't change between servers, I'l leaning towards the problem being somewhere in there.

This probably deserves it's own thread again now, but I wanted to update this one too.

Thanks in advance.

 

[GhislainG]

Do you have similar issues if you test with ESXi or Hyper-V?

 

[Furi0usGe0rge]

Do you have similar issues if you test with ESXi or Hyper-V?

I'm going to avoid going that direction with the troubleshooting, for now, simply because I'm not very familiar with either.

There is a mailing list for vfio, and I'll report back after adding some alternate kernals.

 

[GhislainG]

You obviously enjoy that kind of challenge which is rather good for the IT community.

 

[Furi0usGe0rge]

You obviously enjoy that kind of challenge which is rather good for the IT community.

Troubleshooting is fun, but only to a very small and short extent, and quickly becomes frustrating. The getting it working part is much more rewarding.

 

[GhislainG]

You'll hopefully receive a solution and not have to acquire new hardware.

 

[Furi0usGe0rge]

According to the a very helpful member of the vfio-users mailing list:

What you're seeing is a hang in qi_submit_sync(), which is an impossible hang according the hardware specs, but comes up a fair bit more often than never. AIUI, it means that magic, undocumented bits in hardware aren't set the way they should be and you're going to need to hope for a BIOS update, which is all but impossible on a system as old as X58. Good luck :-\

I let it ride for a few days on the Gigabyte official support forum (keeping in mind that I only bought the board because of the beta BIOS they put out), but no response, and being as it is a linux-related problem, I did not expect one.

I'm back at square one, but very stoic about it.

 

[TheStevenator]

Hey Furi0usGe0rge,

Check out my struggles here: https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/

I can confirm that VT-d, even with an i7-950 (I'm now running a Xeon X5660) which does not have "official" VT-d support on ark DOES work in ESXi with no issues whatsoever for a variety of devices.

Linux is a different story. I'm at a state now where if I:

turn ON VT-d in BIOS, system freezes on boot @ initramfs load. This is with the following arguments: "intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=DEVV:IDDD"

turn OFF VT-d in BIOS, with the same arguments, system loads to shell (running Arch with... 4.5 kernel? 4.X anyways) and I see from dmesg that IOMMU is enabled but then I get no other dump about the particular device. However, the device is successfully attached to pci-stub. See my posts on that reddit thread, let me know what you think. I'm in touch with someone from kvm-devel mailing list but it's not going well so far.


Something is likely wrong with the linux kernel. If ESXi can do passthrough then that must be the case. In the BIOS I have the "PnP" setting (plug and play) enabled, which lets the os do device management, etc (as far as I understand). In this situation I'm not sure how much the ASUS DMAR tables, etc, come into play. I'm wondering if I should just start downgrading kernels until it works.

EDIT: I see that we are at exactly the same step. Some strange mismatch between VT-d support in the BIOS/OS.

EDIT 2: It's disappointing that the linux kernel is having so much trouble and that ESXi just has no issues. I'd rather not deal with all of vmware's crud/restrictions (and I'd like to have decent cli management), but at this rate......

EDIT 3: Why is this thread marked as solved? It's definitely not solved...

 

[Furi0usGe0rge]

I never saw this post, possibly because the thread is marked as solved.

The onboard sound was causing all the Asus boards to crash. The Gigabyte board needed a never-released beta bios. After that, both platforms would work, but only with one VM at a time or if all VMs are on the root disk.

Hey Furi0usGe0rge,

Check out my struggles here: https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/

I can confirm that VT-d, even with an i7-950 (I'm now running a Xeon X5660) which does not have "official" VT-d support on ark DOES work in ESXi with no issues whatsoever for a variety of devices.

Linux is a different story. I'm at a state now where if I:

turn ON VT-d in BIOS, system freezes on boot @ initramfs load. This is with the following arguments: "intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=DEVV:IDDD"

turn OFF VT-d in BIOS, with the same arguments, system loads to shell (running Arch with... 4.5 kernel? 4.X anyways) and I see from dmesg that IOMMU is enabled but then I get no other dump about the particular device. However, the device is successfully attached to pci-stub. See my posts on that reddit thread, let me know what you think. I'm in touch with someone from kvm-devel mailing list but it's not going well so far.


Something is likely wrong with the linux kernel. If ESXi can do passthrough then that must be the case. In the BIOS I have the "PnP" setting (plug and play) enabled, which lets the os do device management, etc (as far as I understand). In this situation I'm not sure how much the ASUS DMAR tables, etc, come into play. I'm wondering if I should just start downgrading kernels until it works.

EDIT: I see that we are at exactly the same step. Some strange mismatch between VT-d support in the BIOS/OS.

EDIT 2: It's disappointing that the linux kernel is having so much trouble and that ESXi just has no issues. I'd rather not deal with all of vmware's crud/restrictions (and I'd like to have decent cli management), but at this rate......

EDIT 3: Why is this thread marked as solved? It's definitely not solved...