pci-e extension cable curiosity

[mmd123]

ok, so I am looking at buying a server motherboard that only has 2x pci-e x16 width slots, and that has 4x pci-e 8x slots, and I'm curious, on newegg you can buy pci-e 8x to 16x extenssion cables and I'm wondering just how practical these are.



IE if a video card that runs on the 16 width bus will successfully work and be fully usable on only an 8x bus/port. I'm not 100% sure if it will work or if it will not, because these cables are only really used for systems that are bitcoin mining, and I want to be able to use it either for an intel xeon phi card, or for 3 or more video cards, I know that if I use one of these types of cards for a device that either uses the 8x lane or less, ie 4x or 1x, that it SHOULD work perfectly because it was designed to use that or less number of lanes. however, I am completely unsure of whether it will work for a device that was only intended to be used on the full 16x lanes...



so thats why I decided to finally register here, to query the masses here. you guys would be more likely to know the answer to this than I would, on one hand it should work because if it did not then it would not be as prevelent in the bitcoin masses as it is, on the other hand, if your trying to run a device on something less than it was designed for, then there HAS to be something that would not work like it was designed for logically.



this is not actually a problem, so take your time to reply to this, this is for a future server build that I am planning... but I'm wracking my brain trying to decide on an answer and I simply do not have enough knowledge or expertise to make one.



the parts in question:

Motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813182939

video cards: http://www.newegg.com/Product/Product.aspx?Item=N82E16814487079

xeon phi card: http://www.newegg.com/Product/Product.aspx?Item=N82E16819116972&cm_re=xeon_phi-_-19-116-972-_-Product

pci-e extension cable: http://http://www.newegg.com/Product/Product.aspx?Item=9SIA0U01H79633&cm_re=pci-e_8x_riser_card-_-9SIA0U01H79633-_-Product

newegg parts list for the entire system: http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=32138748

originally posted this to bleeping computer, but thus far no responses over there, so I figured I'd double my post views by posting to a second website. hopefully that is not bad, I figure, even if I get replies to both, that I can get multiple opinions on the matter.

 

[buzznut]

So with a video card you can use a 1X to 16X for mining without issue. But in your case I imagine you'll want more bus bandwidth. [strike]With a video card on the other hand I would always recommend a powered riser. This may not be necessary for your needs.[/strike]

Scratch that, 245W you will definitely want a powered riser and a decent power supply to power it as well. So forget the ribbon cable and look for something more robust. Also May want to take a look at motherboards that have additional molex power for more PCI express power, such as the BTC pro boards.



I am a miner and have experience with powered risers. I have one that is a 1X -16X and it works perfectly, it uses a USB interface and it was around $12 on ebay.

 

[mmd123]

So with a video card you can use a 1X to 16X for mining without issue. But in your case I imagine you'll want more bus bandwidth. With a video card on the other hand I would always recommend a powered riser. This may not be necessary for your needs.

so it still works, and will still output video if I were to try to use it in a very multi-card solution? my main consurn is that it will not spit out video if I were to try to use it as if it were my only video card, or in the case of the xeon phi card, will it perform to its fullest potential as if it were natively plugged into a full x16 width slot?

thus far this is looking good based upon the information you have provided me with, so thank you for that information from your personal experience. that is MUCH appreciated!

 

[buzznut]

I run a PCI expressed SSD in another machine with 2 video cards. You will want to make sure whatever board you choose is able to functioan with the proicessor. Also the video cards will likely go in the X16 slots, in other words the Xeon would not go in the number 1 slot but the riser idea should work.

HOWEVER what you are proposing is not easy, be prepared to experince issues getting that going. Also I would suggest a robust chipset with plenty of PCI express lanes, which will mean a more expensive motherboard.

I recommend thinking about this very hard if this going to be worth what you propose.

 

[mmd123]

the video cards power consumption, for the base nvidia design, is 165W vs the 265 for the xeon phi card, so I can always run the video cards over that extension cable, and even get a powered extension cable if nesasary, I just want to know if it will work to its fullest potential and still spit out video...as long as it still works to its fullest potential and still spits out video on all the ports that it is supposed to do so for/supports 4 monitors per card, then I have no problems running the video cards off of the extension cables and running the xeon phi card off of the full x16 width busses. I just need to know if they will still work to their fullest potential even if running off of x8 slots. power is not an issue as I will be using, at minimum, my current 1.2Kw corsair gaming psu, which has a high amperage 12v rail, I think if I remember correctly, the 12v rail on this version is like 128A or 138A for a single 12v rail. that is, if my memory is serving me correctly. it was the last revision of the gs1200 series right before they did their latest revamp, IE the color on this one is the old silver color vs the new red color, thats how you'll know if you have the right model/revision.

 

[mmd123]

I run a PCI expressed SSD in another machine with 2 video cards. You will want to make sure whatever board you choose is able to functioan with the proicessor. Also the video cards will likely go in the X16 slots, in other words the Xeon would not go in the number 1 slot but the riser idea should work.

HOWEVER what you are proposing is not easy, be prepared to experince issues getting that going. Also I would suggest a robust chipset with plenty of PCI express lanes, which will mean a more expensive motherboard.

I recommend thinking about this very hard if this going to be worth what you propose.

in respects to pci-e lanes, yes, the system that I am looking at/configuring will have a meager 12 lanes to spare vs the total capacity that the motherboard supports, the dual processor system having the total capacity for 80 lanes and the system at full utilization, if every slot is taken with or without the extensions, of 68 lanes, that is if everything is operating at full speed full bandwidth. but yes, I am fully aware that what I am trying to do will be tough, I just want to know if it is even possible. I'll link you the parts list I have for this system on newegg...just keep in mind, this is my dream system, as in the system I would buy/build if I had enough money to do so. even if that meant saving up for 5 to 10 years. this is what I do for fun, is build dream systems that could rival the greatest single system setup.

parts list respectively: http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=32138748

 

[buzznut]

Wow. Ok, well I am not sure what you're planning but GL.

The system will likely NOT boot with the Xeon in the number 1 slot while cards are populating other slots. Again, the video cards will want/need the highest bandwidth lanes, or at least its likely that the board will see it that way. I had many struggles with these very issues, and I was just using a 4X SSD. There's really only one way it will work, and if I try to use more than 2 cards plus the SSD, it refuses to boot. Most likely due to PCI express lane limitations, but it could just as easily be the mobo.

I doubt that there are more than a dozen motherboards capable of utilizing that crazy Xeon thing.

As to full utilization, I can tell you that my Z77 board will only run the SSD in 2X mode with two viudeo cards, yet it performs exactly as expected. I have tried it in X4 and X8 slots with no difference. Hope that gives you some ideas.

 

[mmd123]

Wow. Ok, well I am not sure what you're planning but GL.

The system will likely NOT boot with the Xeon in the number 1 slot while cards are populating other slots. Again, the video cards will want/need the highest bandwidth lanes, or at least its likely that the board will see it that way. I had many struggles with these very issues, and I was just using a 4X SSD. There's really only one way it will work, and if I try to use more than 2 cards plus the SSD, it refuses to boot. Most likely due to PCI express lane limitations, but it could just as easily be the mobo.

I doubt that there are more than a dozen motherboards capable of utilizing that crazy Xeon thing.

As to full utilization, I can tell you that my Z77 board will only run the SSD in 2X mode with two viudeo cards, yet it performs exactly as expected. I have tried it in X4 and X8 slots with no difference. Hope that gives you some ideas.

if I'm remembering correctly, the main limitation is whether or not the cpu itself can churn through the raw data that the other devices are capable of outputting/inputting, so if you have enough sheer cpu power, then the other input/output devices should have enough to work with, I am not specialized in my knowledge on any one particular device, but I have more of an overall knowledge about computers, have used them since I was 5 in 96, and have been gaining more knowledge since and refuse to stop. you have a single processor system so your system will be much more limited in regards to what all it can simultaneously support via pci/pci-e, and yes, there will in all likelihood need to be a video card populated in the main slot, but at the same time I don't know if that is a requirement given that this motherboard also has built in video. that is something that I will have to find out if/when I actually get the funds for this beast. but in all likelihood I'll buy a powered pci-e adapter to run this beast from so that the video cards get full utilization, because that addon xeon phi card is more for web hosting/hosting of other os's than anything so I do not care all that much if it will be fully utilized, just so long as it works at all. mainly I am not worried because that card in and of its own right will require either 1x or 2x pci-e 6/8 pin power plugs to run the beast. so it does not require much power from the pci-e bus itself. that said, the xeon phi card(s) will likely be in one of the expansion cards to allow for use of all the pci-e ports on the motherboard. I have enough raw cpu power to support more than enough pci-e motherboard lanes, so I should be good that said, I just wanted to make sure that the devices powered by expansion ports would still operate as intended.

and to your comment about "I doubt there are many boards that can support that crazy xeon thing" there are actually quite a few because it is designed by intel to be mostly plug and play with their more powerful server systems, it is supposed to be a supplementary processor for server systems where it pays to have parallelization in mass but it is not affordable to purchase hundreds more servers. it operates as if it were part of the main intel processor and even allows for the same styles and same exact code to be executed on it as if it were being done by the main system processor itself. it has the same instruction sets and the same coding as their typical xeon processors. so just think of it as a video card sized xeon chip in and of its own right. that's whats so appealing about it and why I want one, it can be used as if it were the main/only processor in a system regardless of the actual processor in the system, hence it could even be used in a typical desktop computer assuming you have a spare 16x video card slot that is dual width and the power to spare for it.

 

[buzznut]

Well, my question would be to how effective is this in comparison to a board where you just drop two Xeons and go.
Let me put it to you this way. I got this PCI express SSD because it was a raid 0 on a card and because I thought it would be so much faster and this monster PC. Turns out I would have been much better off just getting 2 SSDs and raiding them myself. Believe me when I say I've had my regrets from the start. It required many days of research and going over every last forum entry on the Revo card before I finally found a solution that I was finally able to use to get windows on the damn thing. And if ya think my troubles ended there, you'd be sadly mistaken..

If you think its as simple as dropping the card in there and it working as advertised, I can almost certainly assure you that will not be the case. I firmly believe that the number of boards that will actually do this will be limited, especially comnpared to the number of boards that SAY they can.

It has to do with the PCI express being designed for things like video cards and sound. Not for processors and SSDs and things that have a ridiculous amount of I/O stuff. I am just trying to forewarn you, of course you will decide what you want, but my experience is that utilizing the PCI express for other things is hit and miss at best. I find it unlikely you'll see a significant boost in performance vs the headaches involved.

It was only after my purchase and struggles that I realized the limitations of what I was trying to do.

 

[mmd123]

although that was largely true for the first iteration of pci express, the bandwidths were SIGNIFICANTLY raised with the versions 2.0 and even more so with 3.0, so heavy I/O tasks are now infinitely more suited to pci-e, especially so with 3.0, the bandwidths with 1.0, 2.0 and 3.0 respectively are:

Base Clock Speed: PCIe 3.0 = 8.0GHz, PCIe 2.0 = 5.0GHz, PCIe 1.1 = 2.5GHz
Data Rate: PCIe 3.0 = 1000MB/s, PCIe 2.0 = 500MB/s, PCIe 1.1 = 250MB/s
Total Bandwidth: (x16 link): PCIe 3.0 = 32GB/s, PCIe 2.0 = 16GB/s, PCIe 1.1 = 8GB/s
Data Transfer Rate: PCIe 3.0 = 8.0GT/s, PCIe 2.0= 5.0GT/s, PCIe 1.1 = 2.5GT/s

thats the information for 3.0, 2.0 and 1.0 respectively...so there is more than enough capacity with the 3.0 revision so long as you have a heavy duty enough main cpu to handle it....having dual 14 core xeons in this build/configuration I should be fine...key words here being should be. in my personal experience I too have had much chagrin at getting things to work as advertised and as they theoretically should, but at the same time after a due amount of tinkering and experimenting I have almost always been able to get things to operate as I know they should based on facts that I have at my disposal. do I think that it will be up and running flawlessly the first time? heh...if only things were that easy and that simple, do I think that I CAN get them up and running over time? most likely, maybe not for a while, but I should be able to get things to work after some due tinkering and experimenting. will it be worth the cost? that I am unsure of, but hell, with something this fun, it should sure be worth the time and effort put into it at least in respects to the pleasure it will provide me once I get the thing functional even if only at partial capacity!

so long story short, this is what I do for fun, and it would provide me much fun to be able to get this sucker working even at partial capacity. hell, if I were to even succeed at amassing enough money for this beast in respects to just the raw parts cost, then thats further ahead than I have been in the past. so what have I got to lose?