News AMD fires back at Nvidia with instructions on running a local AI chatbot — recommends using a third-party app

[Admin]

AMD posted a tutorial detailing how users can get an AI chatbot running on Ryzen CPUs and Radeon GPUs. It's surprisingly straightforward.

AMD fires back at Nvidia with instructions on running a local AI chatbot — recommends using a third-party app : Read more

 

[shalako33]

AMD posted a tutorial detailing how users can get an AI chatbot running on Ryzen CPUs and Radeon GPUs. It's surprisingly straightforward.

AMD fires back at Nvidia with instructions on running a local AI chatbot — recommends using a third-party app : R

 

[shalako33]

I've had this installed and setup for months but haven't used it yet. I've been meaning to eventually mess around with it trying out different LLMs. I even downloaded the 7 billion and maybe a couple of smaller ones a while back for when i do. Its just i've got a bunch of games keeping me busy at the minute. Think i read somewhere there's a 12 billion or higher on the way.

 

[kealii123]

I wish someone could be clear on how much ram can be provisioned to the 780m iGPU (since the NPU doesn't seem to be used). I sold my 6800u device so I can't test myself. On my Intel laptop (i7 10th gen, 2080 super, 64 gig ram) I can run very large models, thanks to the 64 gigs of RAM, but is glacially slow since the CPU is doing the work all by itself.

On my M1 Macbook Pro, 32 gigs RAM, I can run the 13b param llama2 (Olamma) if I don't have too many chromium tabs open, and its faster than GPT4 since the M1 GPU is running the model, and it has access to as much of the 32gigs of shared system ram as it wants.

If the AMD chips with their decent iGPU can be set in the BIOS or somewhere to have access to an arbitrary share of the system RAM, it could be the cheapest/easiest way to get the larger, 70b models running since you can cheaply slap 64 gigs of RAM in a device and give 50 gigs of it to the iGPU.

 

[Alvar "Miles" Udell]

I downloaded it and am using the phi 2 3B LLM. On my 16 core 5950X it is using between 30-38% of the CPU, about 2.5GB RAM, and generates decently fast, about as fast as Copilot renders its answers, but when using GPU offload to my RTX 2070 it uses 90% and renders at a much faster rate, maybe 3-4x as fast.

Running slowly on anything less than an 8-core CPU, and likely even an 8-core CPU, would be quite slow indeed.

 

[Albert.Thomas]

"Chat with RTX is arguably the most fleshed-out of the solutions, as it can analyze documents, videos, and other files."

I've been toying with the idea of using a chatbot to assist in some of the work I do outside of cooling reviews, and Chat with RTX is the only one (that I know of) that is evenly remotely useful because of the ability to read documents.

But its lack of chat memory makes it limited in usefulness

 

[Alvar "Miles" Udell]

"Chat with RTX is arguably the most fleshed-out of the solutions, as it can analyze documents, videos, and other files."

I've been toying with the idea of using a chatbot to assist in some of the work I do outside of cooling reviews, and Chat with RTX is the only one (that I know of) that is evenly remotely useful because of the ability to read documents.

But its lack of chat memory makes it limited in usefulness

Sadly another feature us poor Turing users won't get.

 

[ezst036]

Sadly another feature us poor Turing users won't get.

That's the downside of choosing Nvidia. No longevity.

Arguably Nvidia has more programmers and pretty much more of everything, and faster hardware, and offers a full/complete package right out of the box for people. For that reason Nvidia is on its way to being the most valuable company on the planet.

The downside is you're completely dependent upon Nvidia. And once they kick you to the curb, your kicked. You have no recourse.

AMD isn't nearly as far along with so many things, and Nvidia users can (and do) list the ways and numbers of how those facts bear out. However, AMD's solutions work across a much further generational line of cards and heck, work on non-AMD systems or partial AMD-systems. So they are generally inclusive that way.

If you're someone who only keeps computers for a few short years - 2 or 3 years or some small number, no doubt you want an Nvidia solution and it surely isn't even a question.

But for anybody else, AMD is probably the better bet. Even if at the outset their new solution only covers 3 generations because its open source you can bet someone will easily adapt it to that fourth generation back. Or fifth. Or even sixth if reasonable performance exists. Generally speaking AMD does not put a stop to this and just the opposite, encourages it.

Much of what I said also applies to Intel as well, there's great longevity. Most of what they create now is open source as well. However, AMD does seem to lead the pack here.

But. For those 2-3 "brand new" years? Nvidia is simply the best of the best.