Z170-Deluxe / M.2 PCIe Question

[Lude2Envy]

I just upgraded to a 6700K from my 3570K and went with the Asus Z170-Deluxe mobo to compliment it. I've been out of the loop for a while so I've been doing some reading on all of the new storage solutions and decided to go with a 256GB Samsung SM-951 NVMe stick to run Windows and my other core applications. I plan to also purchase a 512GB SM-951 once the NVMe model becomes available. The nice thing about the Z170-Deluxe is it has an onboard M.2 port and it comes with an expansion card to install a 2nd M.2 SSD. Now from what I've been reading, using these PCIe SSDs on older boards, like the Z97 (not sure about the X99), would steal some of the bandwidth from the GPU - cutting a x16 into x8/x8 as if running 2 GPU's in SLI. I believe the Z170-Deluxe has 4 dedicated PCIe lanes for the onboard M.2 connector, but if I put a second SSD into the expansion slot, will my GPU take a hit and lose some lanes - or does the board have dedicated lanes for 2 PCIe SSDs? Thanks in advance for any feedback.

 

[MarkW]

Anytime you have more than 1 device in an PCIe slot, you are using PCIe lanes. And there are still only 16 lanes available to the slots.

There are some new PCIe 3.0 type lanes available from the chipset, but those are restricted to x4 lanes per device. So your M.2 x4 just used one of those sets, and the USB ports use some, and SATA ports use some and so on...

Its still pretty much the same rules its been all along, but we have some new devices now, and one new slot type.

 

[hapkiman]

MarkW is correct. Your shiny new Z170 board has the nice additional PCI lanes made just for storage devices like your M.2. But now that you have that used up along with the other connections he mentioned, you will be robbing from the graphics card, and it will drop down to x8 speeds (which isn't the end of the world- just a tiny hit in performance). But - I would rather just run two SATA III 6Gb/s SSDs in Raid 0, or even just as single drives. They are still plenty fast and you would leave the graphics card at x16.


BTW, how do you like your Skylake rig? Your's is similar to mine and I'm loving it! Freakin' awesome.

 

[NerdIT]

Firstly, if it WAS to steal bandwidth, it would not make your card run like if SLI -that requires 2 cards- it would run it at x8.

The 6700k CPU only supports 16 PCIe lanes .. so if you run a card at full x16 speed -adding any other PCIe card to the bus would force card to run at x8 (although you probably would NOT notice a difference in performance...as the actually speed of the bus itself is nowhere near saturated even under load..)

PCI Express Revision 3.0
PCI Express Configurations ‡ Up to 1x16, 2x8, 1x8+2x4
Max # of PCI Express Lanes 16
http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz

Compared to the 28/40 lanes provided by the Haswell-E chips. Really this just makes SLI/XFIRE configurations different/possible or impossible at different speeds/configs.

 

[hapkiman]

NerdIT, I know you are correct about the i7 6700k only having 16 PCIe lanes, but are you sure about adding any PCIe card -automatically dropping the card to x8? I was under the impression the Z170 chipset has 20 PCIe lanes, 16 of which are used by SATA, USB, etc... So a single x4 is left over for a PCIe storage device.

http://www.computershopper.com/components/reviews/intel-core-i7-6700k/(page)/2#review-body

That review by Computer Shopper's Matt Safford states "For starters, Z170 boards will support up to 20 PCIe 3.0 lanes, rather than the eight lanes of PCIe 2.0 provided by standard Z97 boards. (That is in addition to the 16 PCIe 3.0 lanes provided by the CPU, which remains the same with Skylake.) The extra lanes come thanks to an upgrade to the DMI interface that connects the CPU to the chipset, and they are important, given the rising prevalence of PCIe-based storage (via M.2, SATA Express, or drives that plug directly into a PCI Express slot, like Intel’s SSD 750 Series)."

If I'm not understanding this correctly please enlighten me (I'm not being sarcastic, I am really asking)...

 

[MarkW]

I think we still need to have someone take an NVMe PCIe device, and add it into a Z170 based motherboard, and do the difficult work required to figure out which lanes are being used and which lanes are not being used. Because if Intel and Micron what to push these Optane (XPoint) cards into the market, I think they need to do it by NOT affecting other PCIe lane setups. Because the very people that would have enough cards to be affected by the Optane cards are the ones that are going to be the first adapters of Optane.

Optane is sexy so far. 1000x faster, 1000x duration, and 10x the density. Those are incredible jumps in technology. Now we need to see two more things. Price. And how PCIe is affected by adding an Optane NVMe card. And that will not happen until sometime in 2016.

 

[hapkiman]

I think we still need to have someone take an NVMe PCIe device, and add it into a Z170 based motherboard, and do the difficult work required to figure out which lanes are being used and which lanes are not being used. Because if Intel and Micron what to push these Optane (XPoint) cards into the market, I think they need to do it by NOT affecting other PCIe lane setups. Because the very people that would have enough cards to be affected by the Optane cards are the ones that are going to be the first adapters of Optane.

Optane is sexy so far. 1000x faster, 1000x duration, and 10x the density. Those are incredible jumps in technology. Now we need to see two more things. Price. And how PCIe is affected by adding an Optane NVMe card. And that will not happen until sometime in 2016.

+1.

Well said, and exactly what I was thinking.

 

[MarkW]

Skylake based motherboards have the standard 16 PCIe 3.0 lanes from the CPU, and whats new is that the chipset now has 26 HSIO (HIgh Speed I/O) PCIe 3.0 lanes that are allocated for I/O ports. You can see in the chart below how the PCIe lanes are designated. Motherboard manufacturers can reallocate some of those lanes to support other things. I am not yet certain what all those other things consists of yet. But we will find out over time.

 

[Lude2Envy]

Skylake based motherboards have the standard 16 PCIe 3.0 lanes from the CPU, and whats new is that the chipset now has 26 HSIO (HIgh Speed I/O) PCIe 3.0 lanes that are allocated for I/O ports. You can see in the chart below how the PCIe lanes are designated. Motherboard manufacturers can reallocate some of those lanes to support other things. I am not yet certain what all those other things consists of yet. But we will find out over time.

So if I use two m.2 NVMe drives and two SATA drives, I'll be stealing lanes from my GPU?

 

[hapkiman]

It would depend from what I am getting. First of all, not all M.2 NVMe drives are the same. If you have two x4 drives though, then yes I believe your card will drop to x8. But what exact NVMe drive are you talking about?

I would connect a single M.2 drive, and then check it with AIDA 64 or HWINFO64 and see what it says your drive is operating at. Those same utilities or GPUz will tell you if the graphics card is operating at a full x16.

At this point it is still about as clear as mud, because it looks like mobo manufacturers can allocate the lanes in different manners, and like I said some drives are x2 and others are x4.

Like this awesome Samsung is x4.
http://www.anandtech.com/show/9702/samsung-950-pro-ssd-review-256gb-512gb

From the article:
"While some vendors have put out M.2 PCIe devices that support two lanes of PCIe, Samsung went straight for four lanes with the XP941" drive"

Sorry to say, but if this many PCIe lanes are important to you, it sounds like LGA2011v3 might have been a better platform, for you.

 

[Lude2Envy]

It would depend from what I am getting. First of all, not all M.2 NVMe drives are the same. If you have two x4 drives though, then yes I believe your card will drop to x8. But what exact NVMe drive are you talking about?

I would connect a single M.2 drive, and then check it with AIDA 64 or HWINFO64 and see what it says your drive is operating at. Those same utilities or GPUz will tell you if the graphics card is operating at a full x16.

At this point it is still about as clear as mud, because it looks like mobo manufacturers can allocate the lanes in different manners, and like I said some drives are x2 and others are x4.

Like this awesome Samsung is x4.
http://www.anandtech.com/show/9702/samsung-950-pro-ssd-review-256gb-512gb

From the article:
"While some vendors have put out M.2 PCIe devices that support two lanes of PCIe, Samsung went straight for four lanes with the XP941" drive"

Sorry to say, but if this many PCIe lanes are important to you, it sounds like LGA2011v3 might have been a better platform, for you.

I purchased two 512 GB SM 950 Pro's. They ship on the 29th. I was doing some research and it looks like my GTX 980 won't even take advantage of the full 16 lanes. PCI-E 3.0 at x16 and x8 benchmark the same with a single GPU. Since I don't run two cards in SLI, I'm not worried about my GPU running with 8 lanes. I think the two 950 Pro's will be worth it.

 

[MarkW]

It all depends on how the motherboard is designed, and what lanes from the chipset are assigned to which devices.

The 16 PCIe lanes from the CPU should still be assigned to the PCIe slots though.
M.2 x4 slots should come only from the chipset. As should all I/O ports.

 

[Lude2Envy]

It all depends on how the motherboard is designed, and what lanes from the chipset are assigned to which devices.

The 16 PCIe lanes from the CPU should still be assigned to the PCIe slots though.
M.2 x4 slots should come only from the chipset. As should all I/O ports.

One of the M.2 ports is built straight into the motherboard. The other is via a PCI-E expansion card. I'll do some testing once I get the 950s set up.

 

[NerdIT]

Check this out:
http://www.enthusiastpc.net/articles/00003/2.aspx

You should read the whole thing as it is very informative, but here's a part of it explaining how Intel chips work with PCIe (note what i put in bold) I know this talks about LGA-1155- but the restriction is in Intel chips itself.

"On the previous page we touched on the AMD architecture concerning PCI Express where these 2 facts will be important in understanding the difference with the various Intel architectures:

The chipset determines the total number of available PCI Express lanes
How these lanes are routed on the mainboard (onboard devices and slots of various sizes) is mostly up to the mainboard manufacturer

As you might know, Intel 1155 mainboards do not have a Northbridge chip. In some ways this makes things easier for the mainboard manufacturers but it also limits things.

The primary PCI Express lanes (the ones used for graphics cards) on Intel 1155 mainboards do not originate from the Northbridge chip (as there isn't one) but instead originate from the CPU. While this makes mainboard design somewhat easier (less physical connections to route) it is also very restrictive. Remember, PCI Express lanes are physical connections on the mainboard that run from one chip to another chip or to a slot.

Intel Sandy Bridge processors using socket 1155 have only 16 physical lanes that run from the processor to the primary graphics slot or slots. These lanes are generally only used to connect graphics cards. This limits all socket 1155 processors to 16 lanes for graphics. On the simplest boards you will only find a single x16 slot that will hold a graphics card.

On SLI / CrossfireX capable boards you will find mostly 2 physical x16 sized slots for graphics cards. If only one graphics card is used in the primary slot, all 16 lanes from the processor socket will connect to this slot. However, if a videocard is inserted into the second x16 slot, the mainboard reroutes 8 of the 16 lanes from the primary slot to the secondary slot. The end result is that 8 physical lanes connect to each of the 2 slots.

It's important to take note of how this was written down: if slot 2 is populated, both slots operate with 8 lanes."



EDIT: So they fixed this with socket 2011 chips, I am looking for more info on LGA 1151 specifically

 

[MarkW]

Z170 motherboards have 16 PCIe 3.0 lanes running between the CPU and the PCIe slots.

Then there are 26 additional PCIe 3.0 lanes coming from the Z170 chipset which are intended for add ons... SATA III, SATA Express, USB, M.2 x4, and so on. That is what the colorful chart above shows. Some are predefined as to what Intel wants them to be used for, but most can be easily reassigned by a motherboard designer.