Question is NVLink actually useful for anything and can it be used with two different model GPU's?

[Firestone]

I already have the RTX A4500 in my system ( https://www.nvidia.com/en-us/design-visualization/rtx-a4500/ )

I want to "upgrade" to a GPU that has 24GB+ of memory for some VRAM intensive AI / ML work, possibly something like the RTX A5000 ( https://www.nvidia.com/en-us/design-visualization/rtx-a5000/ )

However I am using the Asus Dark Hero motherboard, which has multiple PCIe x16 slots ( https://rog.asus.com/us/motherboards/rog-crosshair/rog-crosshair-viii-dark-hero-model/spec/ ) so I am also wondering about the possibility of just adding a second GPU, such as the RTX A5000, and keeping the old A4500 in the same system.

If I did this, I am wondering if it is possible, and useful, to use the NVLink feature to connect the cards. I am not actually sure if this give me any functional benefits as opposed to just running the GPU's discretely. Thoughts?

 

[hotaru.hino]

NVLink is basically an upgraded version of SLI, so all of the requirements to use it are still there: namely you still have to have the same video card model to use it.

 

[Firestone]

I have been searching online and keep getting conflicting reports of if it works with a configuration such as RTX A4500 + A5000

also not clear what this looks like from an OS and software level, like if I attach the NVLink bridge is the GPU going to automatically be detected as one device, or two devices still, and if I would have to do anything special in order for software that usually runs on a single GPU to be able to see and use the extra GPU's resources, etc..

this would be doing "work" under Linux (Ubuntu 22.04) but I would also want to switch back to Windows occasionally to play a game or two

 

[helper800]

The short answer is; No, and no.

 

[Firestone]

If I switched to something like 2x RTX 3090 with NV Link, assuming i found a use for it, would it prevent me from also using the GPUs for gaming? A lot of article snippets suggest "negative gaming experiences" with NV Link enabled but i haven't been able to figure out what that actually means and how gaming would be affected.

Mostly I want to be able to do work in Linux that could utilize NV Link, then just switch back to Windows and play a game now and then, without headaches.

 

[iTRiP]

In the case of my two AMD cards, it seems for such a setup to work right, all cards needs to run the same driver, I don't know If nvidia cards
combined with windows 11 require then same. This didn't use to be an issue in windows 10.

A good use for it is to mine on one gpu, and run your 3d app or game on the other gpu, but you have to also have a good enough PSU.


There isn't any real profit to be had mining, but if you are having winter you might make some extra heat, and have a more pleasant ambient temperature in your room where your pc is located.

 

[Firestone]

It seems that some of the judgements portrayed here might be inaccurate, there appears to be clear performance benefits from using NV Link for diverse use cases as shown by the benchmarks and reviews described here

Dual NVIDIA GeForce RTX 3090 NVLink Performance Review

In our dual NVIDIA GeForce RTX 3090 NVLink compute performance review, we see how scaling to multiple GPUs impacts performance

www.servethehome.com

There are effectively two large buckets of performance we are seeing here. First, there are applications that are not designed to use multiple GPUs. These tend not to be workloads like we see in the deep learning and scientific fields. We can understand why NVIDIA would start a SLI phase-out. In the rare instances, we saw slightly worse performance with two cards than one card. Our sense is that one will purchase two GPUs because it meets the needs of their most demanding use case. If you have a workload that will spend 80-95% of the time being 80-95% faster, and another workload that will be 5-20% of the time but be 5-20% slower, that is still a net win.

For those applications such as our deep learning training and inferencing benchmarks, we get great results. This is directly due to the domain areas focusing on multi-GPU support and NVIDIA creating NVLink to address that market. Since these are GeForce cards, double precision math is not great. We also cannot use scale-out features such as GPUDirect RDMA. Still, the impact of moving from a GeForce RTX 2080 Ti to dual GeForce RTX 3090’s can yield a 2-4x performance gain in many instances which can absolutely be career-changing and yield better end work product if limited by deadlines.

 

[Firestone]

In the case of my two AMD cards, it seems for such a setup to work right, all cards needs to run the same driver, I don't know If nvidia cards
combined with windows 11 require then same. This didn't use to be an issue in windows 10.

A good use for it is to mine on one gpu, and run your 3d app or game on the other gpu, but you have to also have a good enough PSU.


There isn't any real profit to be had mining, but if you are having winter you might make some extra heat, and have a more pleasant ambient temperature in your room where your pc is located.

at this point I am looking more towards a 2x RTX 3090 configuration for potential AI / ML work, since the price disparity for analogous RTX A5000+ configurations is a little prohibitive.

but as you describe, the power requirements for 2x 3090's is quite high, a stronger power supply is definitely needed (along with concurrent re-evaluation of the power capabilities of the household electrical setup that will be used); I am also hoping to benefit from imposing some power limits as described here;

View: https://www.reddit.com/r/Fedora/comments/11lh9nn/comment/jbp7a8g/

 

[iTRiP]

at this point I am looking more towards a 2x RTX 3090 configuration for potential AI / ML work, since the price disparity for analogous RTX A5000+ configurations is a little prohibitive.

but as you describe, the power requirements for 2x 3090's is quite high, a stronger power supply is definitely needed (along with concurrent re-evaluation of the power capabilities of the household electrical setup that will be used); I am also hoping to benefit from imposing some power limits as described here;

Yes also a good thing a you mention, to have the household or building wiring components compatible with said use age of the intended pc hardware.

Have you figured out if simultaneous cards from nvidia require the same driver to render them use full, Guese you'll really know once you bought and installed them.

If you don't mind me asking what AI/ML work, is it something you make money with, does it work on it's own or do you use it to create something with?

You know, you don't need an NVlink to use them independently in the same pc.

 

[Firestone]

Have you figured out if simultaneous cards from nvidia require the same driver to render them use full, Guese you'll really know once you bought and installed them.

If you don't mind me asking what AI/ML work, is it something you make money with, does it work on it's own or do you use it to create something with?

I have not figured it out yet, since my second 3090 has not arrived, and even then, I have yet to order the NV Link connector either (next step is figuring out how to determine the correct NV Link connector based on the card spacing in my PC case)

this is mostly just for potential use to mess with running local AI language models as per software such as these

GitHub - oobabooga/text-generation-webui: A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models. - oobabooga/text-generation-webui

github.com

GitHub - ggerganov/llama.cpp: LLM inference in C/C++

LLM inference in C/C++. Contribute to ggerganov/llama.cpp development by creating an account on GitHub.

github.com

(none of these actually require a dual 3090 setup, afaict 24GB GPU VRAM is enough for the larger 30B models, but I am interested in trying to max out my capabilities so as not to be restricted if I want to try more resource intensive tasks)

will also be playing with Stable Diffusion ( https://github.com/AUTOMATIC1111/stable-diffusion-webui ), but I was already able to run SD with much less resources so this configuration will be way overkill for that.

mostly just for hobby work, not for anything professional

 

[Firestone]

something interesting that I think is worth clarifying, it appears that having the exact same model of GPU is not technically a requirement for NV Link to work;

View: https://www.youtube.com/watch?v=y4fb_R5Ogw0


In this video, they are using an EVGA model RTX 3090 with what appears to be an Nvidia Founders Edition 3090, and it works. Except for the part where the two different model cards' NV Link sockets do not physically line up correctly, requiring crazy shenanigans to get the two cards physically connected

the vid also shows how game support for NV Link is not great

 

[Firestone]

regarding which Nvidia NVLink Bridge unit to use, it gets a little confusing since the RTX 3090 is so "old" at this point, it seems

The official Nvidia NVLink Bridge page does not seem to list the RTX 3090 model;

Design & Professional Visualization Solutions | NVIDIA | NVIDIA

Connect two NVIDIA Quadro graphics cards for scalable memory and performance.

www.nvidia.com

However according to this "old" article, it seems like the version of the 2-slot and 3-slot bridges for the RTX A6000 line should work with the RTX 3090

NVIDIA NVLink 2021 Update and Compatibility Chart

Using NVLink requires a physical bridge between two identical NVIDIA graphics cards, but with the new generation of Ampere-based GeForce RTX 30 Series and RTX A-series video cards which bridges work with which GPUs?

www.pugetsystems.com

one thing to note is that, in planning to try out the NVLink configuration, I made a point to select the EVGA XC3 ULTRA GAMING, 24G-P5-3975-KR mode cards for both of my 3090's, since they are a smaller size and only 2 slots wide

https://www.evga.com/products/product.aspx?pn=24g-p5-3975-kr

So, if I am reading this correctly, any of the 2-slot NVLink Bridge units should work for this, I think?

also worth noting that Nvidia appears to sell the bridges here

Buy Professional Graphics Cards & Workstations | NVIDIA RTX

Buy professional graphics cards, desktop and mobile workstations, and more in the NVIDIA RTX Store

store.nvidia.com

ofc the 2-slot variant is out of stock at this time

 

[iTRiP]

That does seem like a great gpu to have and use for nefarious purposes, why do they only sell two per household?

https://www.evga.com/products/product.aspx?pn=24g-p5-3975-kr

 

[Firestone]

Oh if you look closely you'll see use that are out of stock on EVGA's website. In case you didn't hear the news, EVGA had a falling out with Nvidia a few months back and has ceased all current and future production of video cards. So the only way to get these models at the moment is third party vendors and used (eBay)

 

[iTRiP]

Oh if you look closely you'll see use that are out of stock on EVGA's website. In case you didn't hear the news, EVGA had a falling out with Nvidia a few months back and has ceased all current and future production of video cards. So the only way to get these models at the moment is third party vendors and used (eBay)

Yeah read something about that, I'd suggest you go look at an ASUS card, they also state if the NVlink is possible on theirs, possibly state if and what the power requirements are, I'll see if I can find something suitable...

 

[iTRiP]

Are you looking at the Ti versions as well?

ROG Strix LC GeForce RTX 3090 Ti OC Edition 24GB GDDR6X | Graphics Card | ASUS South Africa

ROG Strix LC GeForce RTX™ 3090 Ti OC Edition 24GB GDDR6X reigns supreme by combining liquid cooling with the NVIDIA Ampere architecture.

rog.asus.com

ROG Strix GeForce RTX 3090 OC Edition 24GB GDDR6X | Graphics Cards

The ROG Strix GeForce RTX 3090 OC Edition 24GB GDDR6X unleash the maximum performance on NVIDIA Ampere Architecture, by featuring Axial-tech fan design, 0dB technology, 2.9-slot Design, Dual BIOS, Auto-Extreme Technology, SAP II, MaxContact Technology, and more.

rog.asus.com

https://www.amazon.com/ASUS-Graphics-DisplayPort-Axial-Tech-2-9-Slot/dp/B08J6GMWCQ

 

[iTRiP]

Here, pages like this is what you should be looking at:

These clearly state all the specs and requirements:

ROG Strix GeForce RTX 3090 OC Edition 24GB GDDR6X | Graphics Cards

The ROG Strix GeForce RTX 3090 OC Edition 24GB GDDR6X unleash the maximum performance on NVIDIA Ampere Architecture, by featuring Axial-tech fan design, 0dB technology, 2.9-slot Design, Dual BIOS, Auto-Extreme Technology, SAP II, MaxContact Technology, and more.

rog.asus.com

 

[Firestone]

Thanks. I am pretty much sold on the EVGA XC3 Ultra model though, I really do need to stick with the 2-slot form factor, also the fact that each card only uses two PSU cables (instead of 3 like on most other 3090's) is a huge benefit as well. This is because I am building into a Meshify C compact ATX case, so space is pretty tight. Also I went with 3090, instead of 3090ti, due to lower price and lower power draw as well.

 

[Firestone]

minor update: the EVGA RTX 3090 XC3 Ultra is not *actually* a "2 slot" graphics card, its more like "2.1" slots; there is just enough overhang on the card that it blocks the adjacent PCIe slot. So using the Asus ROG Dark Hero VIII motherboard, with two cards in the top two PCIe slots, it seems that the distance is actually 3-slots, and that the last PCIe slot on the board is unusable due to the overhang from the bottom GPU.

 

[Firestone]

a follow-up after working with the system for a few days (2x RTX 3090 EVGA XC3 Ultra + 3-slot NVLink Bridge)

the heat output by the GPU's is tremendous, for anyone reading, I would strongly consider instead trying to find blower-style RTX 3090 models if this is going inside a desktop PC case. There are a few floating around on eBay.

 

[ChipsetHS]

a follow-up after working with the system for a few days (2x RTX 3090 EVGA XC3 Ultra + 3-slot NVLink Bridge)

the heat output by the GPU's is tremendous, for anyone reading, I would strongly consider instead trying to find blower-style RTX 3090 models if this is going inside a desktop PC case. There are a few floating around on eBay.

Do you find the NVLink improves LLM/AI performance? I'm thinking of adding a second 3090 to my system and wondering whether to bother with the NVLink.

 

[Laen]

I just bought 2x RTX 3090 EVGA XC3 Ultra, it’s going in a Corsair 5000D. Doing the exact same thing for LLM purposes - any updates or tips for configuring? Was NVLink worth it?