[SOLVED] Pushing Dell T7500 beyond 96GB RAM

May 28, 2020
8
0
10
Hi,
so I have a Dell T7500 with 96GB ram - 12x 8GB PC3 10600R (1333MHz).
I'd like to upgrade it to 192, but it gets a little confusing.

Pushing it requires 12x 16GB PC3 10600R (1333MHz) - they are quad rank.
OK, but there is a note in the manual:

"If more than one Quad rank DIMM is installed within a channel (DIMM1 & DIMM4, DIMM2 & DIMM5, DIMM3 & DIMM6) then the maximum DDR3 speed is reduced to 800 MHz. Spreading Quad Rank memory modules accross multiple channels is recommended."

What speed are they referring to?
Are we dropping from 1333MHz down to 800MHz? If so, that upgrade sucks, we get twice the ram but half the speed :/
Maybe it's the single rank speed being capped at 800MHz. If so, the 1333MHz memory runs at 667MHz anyway so no biggie. Wait, does the quad rank memory run at 333MHz???

They suggest a solution, spreading the Quad modules to different channels, and there is a configuration in the table with 128GB - 4 of the 6 channels get a single 16GB module, everything else is populated with 8GB - yay :)
Except after that the table jumps straight to 192GB configuration. What about 144GB???
6x16GB +6x8GB - the quads are spread to all the channels, we get 144GB at full speed.
But why did they not mention this? Is 144GB not supported for some reason???

Also what is the whole deal with 192GB needing Red Hat?
Does windows not support it??
 
Solution
Okay, this is basically server class hardware so there's some different rules on RDIMMs than all the usual UDIMM stuff.

One is that when you increase the number of ranks, your speeds drop. The amount speed dropping is usually down just one jedec speed level (1333 to 1066, 1066 to 800, etc), but it depends on how many times the speed drops. In your particular system, there's a single drop when you exceed for 4 ranks per bank.

And that's the other thing you have to deal with--the banking. You have 12 slots across 2x cpus, each cpu only getting 6 slots. And each of these 6 slots are divided into 3 channels, so a pair of slots becomes a bank.

And here's where it gets even more fun, each cpu's memory configuration needs to 'mirror'...
You are confusing 2 different things: DDR and the speed. Double Data Rate works by transferring data on both the rising and falling edges of a cycle. That happens regardless of the clock rate.

The quad channel, on the other hand, makes the data bus between the modules and the CPU appear to be 4 times as wide as the standard 64-bits for each module. So it transfers 256 bits each time it does a transfer.

"Is 144GB not supported for some reason???" I suspect that is correct since the modules would then not be the same size. Once you change sizes you probably get different density DRAM chips on the different modules.

I don't know about Linux. Sorry.

https://www.hardwaresecrets.com/eve...iple-and-quad-channel-memory-architectures/2/

https://www.tomshardware.com/reviews/ddr-dram-faq,4154.html

You may also find this helpful:
https://www.hardwaresecrets.com/memory-upgrade/
 
May 28, 2020
8
0
10
" I suspect that is correct since the modules would then not be the same size. Once you change sizes you probably get different density DRAM chips on the different modules."
That's what I thought but that already is happening in the 128Gig configuration that they suggest - 1st channel (16+8), 2nd channel (16+8), 3rd channel (8+8), 4th channel (16+8), 5th channel (16+8), 6th channel (8+8) - so I'm lost here :(
 
Okay, this is basically server class hardware so there's some different rules on RDIMMs than all the usual UDIMM stuff.

One is that when you increase the number of ranks, your speeds drop. The amount speed dropping is usually down just one jedec speed level (1333 to 1066, 1066 to 800, etc), but it depends on how many times the speed drops. In your particular system, there's a single drop when you exceed for 4 ranks per bank.

And that's the other thing you have to deal with--the banking. You have 12 slots across 2x cpus, each cpu only getting 6 slots. And each of these 6 slots are divided into 3 channels, so a pair of slots becomes a bank.

And here's where it gets even more fun, each cpu's memory configuration needs to 'mirror' the others AND the channels each need to be the same as each other to be considered 'optimal'.

Confused yet? :D

Basically in this type of setup, optimal memory configurations will be when all the banks are the same--16+8, 8+4, 4+4, 16+16, etc. You currently have this since you have all 8GB modules. If you have all 16GB modules it would be the same, etc.

Now, the ranking is what will determine a speed drop. If you exceed 4 ranks in a bank, you will get hit with a speed penalty. But considering the speed differences in ram are insignificant compared to the speed penalties of not having enough ram, it is actually inconsequential when talking about ram pools of this size. Still, you would want to keep speeds as fast as possible, and since ram technologies have evolved, you can. :)

16GB RDIMMs have been available in a 2Rx4 configuration for many years now. So you can actually maintain your current memory speeds and move to 12x16GB for a total of 192GB without any speed penalties.

And since 32GB modules now exist in 4Rx4, you actually can have a total of a whopping 384GB of ram using 12x32GB modules. But because you will need to have 8 ranks per bank to do this, there will be a speed penalty. Still 384GB vs 96GB might be heaven for your use case. :)

Hope this helps and let me know what you end up doing as I love supersized ram!
 
Solution
May 28, 2020
8
0
10
OK, so to solve the note: 16+8GB in a bank - there is a speed drop but from 1333 to 1066 so they probably didnt mention it. Having 16+16 is already 8 ranks in a bank so it triggers a second drop from 1066 to 800, but from its initial cinfiguration it's from 1333 to 800 which is a huge drop so they decided to mention it. That would actually explain why such a huge penalty all of a sudden. Because it's not a single speed penalty, it's trwo stacked on top of eachother.
As to the 32GB 4x4 there's also the compatibility issue. I'm guessing they don't mention them in the manual because they didn't exists back then, but still, I'll have to search around and check if anyone tried putting them into a t7500 and if they worked at all ;)
It's funny how cheap those modules are now: a 16gig module costs $30 and the 32Gig is around $90 :)
 
Gotcha. Yeah, that makes sense why they say 800Mhz then. Still, more ram is better even if it is slower ram.

While I haven't found any evidence of anyone putting 384GB in their T7500, I know that Dell's architecture in these systems definitely does allow for 32GB modules as my R710 is also able to take 384GB of ram via 12x32GB modules.
Again, the rank becomes the limiting factor as 2x 32gb 4Rx4 modules will equal 8 ranks in a bank, which is the maximum for a bank, even though my R710 has 3 slots in each bank (the 3rd stays empty in the 384GB configuration). Since your T7500 shares the same architecture as the R710, except that it has 12 slots vs 18, I could easily see 384GB working.

The manuals are always a snapshot in time, and as technology progresses there are usually some updates that allows these systems to use newer technology. One of the nice things about enterprise equipment where the initial investment for the first buyer is enough to warrant that the equipment lasts as long as possible with as many updates as possible.

One of the only RAM websites that I've found to be accurate when exceeding factory limits is compuram.biz, and it lists 384GB as the limit for the T7500:
https://www.compuram.biz/memory/del...tower-series/workstation-t7500/?st=tab_maxmem

They also accurately state the maximum ram for the R710, which a Dell addendum alluded to, but didn't spell out completely:
https://www.compuram.biz/memory/dell/server/poweredge/r-series/r710/?st=tab_maxmem

Yes, module prices have dropped from the $50,000 mark for 192GB back in 2009 that I saw in an article when researching all this. :eek: And while the prices have dropped significantly from those price levels, they are actually much higher than they were just a year ago when I was debating picking up used 32GB modules for $20/ea. I kick myself for not doing that as now you're lucky if 16GB modules are that cheap. :(

Generally if you can get ddr3 reg ecc for $1/gb that's a decent deal. I've purchased as low as 65 cents/gb and have seen ebay sales as low as 21 cents/gb, so 384GB of ram can be as cheap as $300 shipped if you can get the right deal. And if you do, I'd be interested in your 8GB modules. :D
 
May 28, 2020
8
0
10
OK then.
They're actually no that easy to find (I'm from Europe).
I found these:
https://allegro.pl/oferta/pamiec-ram-micron-32gb-ddr3-pc3l-10600r-ecc-1-35v-9214431335
They are low voltage but from what I understand all low voltage DDRs are actually dual voltage, so there shouldn't be a problem.
The good thing is I have 14 days to return them without even giving a reason so I'll order all 12 modules and test all possibe configs - then I'll pick one :D
 
Those are expensive compared to what you can get used on the US ebay--and those are expensive already too--double what they were just a year ago! :eek:

You are correct about the low voltage modules and ideally if they are supported at the lower voltage it will save power.

Sweet! Please post back with your results! It will be awesome to hear what you max it out to. :D
 
May 28, 2020
8
0
10
OK, after alot of testing, here's what I know about T7500 and RAM:
  • 32GB sticks are NOT supported, doesn't boot in any configuration :(
  • 16GB sticks work like a charm
  • basically all configurations with 16GB sticks work, I checked 128GB (4x(16+8)+2(8+8)), 144GB (6(16+8)), 176GB (4x(16+16)+2*(16+8)) and 192GB (6x(16+16))
  • what's even better, on my X5690 I did not notice a slowdown using more than one Quad module in a channel, they all still report as 1333MHz, CPU-Z reports frequency as 667MHz, and SiSoft Sandra tests the memory bandwidth at 22GB/s on all configs.
  • the only "slowdown" I noticed was in CPU-Z: Row Refresh Cycle Time jumped from 107 clocks up to 200 clocks. This change happened on 144GB, so 128GB is the last configuration that will give you full ram performance.
  • that said, like I mentioned, even at full 192GB I'm still getting 22GB/s memory bandwidth, exactly the same as with 96GB RAM previously, so I'm staying with this configuration.

By the way, Dell seems to be a little whiny about RAM sticks - while testing I had two of the sticks error out, thought I got faulty sticks, but after I rearranged them in the DIMM slots, they all worked. All of the modules are Samsung Micron, exactly the same model, 8 out of 12 are even from the same production batch :)
 

Karadjgne

Titan
Ambassador
Should read the blog, and read between the lines. The T7500 Is capable of running 384Gb of ram, but only if using specific ram. Otherwise you get stuck at the Dell limitations. 32Gb dimms will most likely be viable if using dual rank dimms vs quad rank dimms as the 32Gb dual rank didn't exist back when the T7500 was born. They talk about how newer tech, denser chips etc can and do change original spec limitations.

My guess would be that's not an original, factory tested 12 DIMM kit, it's more likely 2x kits 8/4 and thrown on a mobo and tested by the reseller, but it's going to be older ram, NOS most likely, and is going to be stuck at the same approximate tech level as the T7500, so runs into its limitations. Brand new ram direct from Micron is probably different. And likely more expensive.
 
May 28, 2020
8
0
10
That might be true, that said, finding new DDR3 modules is not that simple and definitely no that cheap, so I think I'm good right now, having spent $300 for an upgrade from 96 to 192GB - it's like christmas :) Also I had a thought, in 2-3 years I should be upgrading to a new machine, 256GB is the low bar now. 256GB of DDR4 costs around $1500! If I were to push my T75000 up to 384GB, my new machine would have to have 512GB RAM - that's 3k on RAM alone. That's more than Threadripper X3970 :)
 
OK, after alot of testing, here's what I know about T7500 and RAM:
  • 32GB sticks are NOT supported, doesn't boot in any configuration :(
  • 16GB sticks work like a charm
  • basically all configurations with 16GB sticks work, I checked 128GB (4x(16+8)+2(8+8)), 144GB (6(16+8)), 176GB (4x(16+16)+2*(16+8)) and 192GB (6x(16+16))
  • what's even better, on my X5690 I did not notice a slowdown using more than one Quad module in a channel, they all still report as 1333MHz, CPU-Z reports frequency as 667MHz, and SiSoft Sandra tests the memory bandwidth at 22GB/s on all configs.
  • the only "slowdown" I noticed was in CPU-Z: Row Refresh Cycle Time jumped from 107 clocks up to 200 clocks. This change happened on 144GB, so 128GB is the last configuration that will give you full ram performance.
  • that said, like I mentioned, even at full 192GB I'm still getting 22GB/s memory bandwidth, exactly the same as with 96GB RAM previously, so I'm staying with this configuration.
By the way, Dell seems to be a little whiny about RAM sticks - while testing I had two of the sticks error out, thought I got faulty sticks, but after I rearranged them in the DIMM slots, they all worked. All of the modules are Samsung Micron, exactly the same model, 8 out of 12 are even from the same production batch :)
Interesting about the 32GB modules as 4Rx4 modules will work in the R710 and R410 from what I've read (not tested yet myself, but there is one addendum addressing 4Rx4 modules. What is the ranking of the 16GB modules?

That doesn't surprise me about no slow down as you're not exceeding 4 ranks. I know on my R710, it only slows down once you exceed 4 ranks in a bank.

That's an awesome amount of ram at an awesome speed. :D
 
That might be true, that said, finding new DDR3 modules is not that simple and definitely no that cheap, so I think I'm good right now, having spent $300 for an upgrade from 96 to 192GB - it's like christmas :) Also I had a thought, in 2-3 years I should be upgrading to a new machine, 256GB is the low bar now. 256GB of DDR4 costs around $1500! If I were to push my T75000 up to 384GB, my new machine would have to have 512GB RAM - that's 3k on RAM alone. That's more than Threadripper X3970 :)
Yeah, and the new ones have a much higher chance to being fake, although that's usually not the case with ecc reg as the buyers of this stuff won't stand for it.

That's awesome that the upgrade worked, and you bring up some solid points on ram costs on a new machine--especially since ddr4 ecc reg is still priced much higher than ddr3 ecc reg. Of course at some point these levels of ram will come common, especially when having 128GB feels the same as having 128MB back in the day. :eek:
 
Or get really fancy and start aiming at 768Gb/1.5Tb with 64Gb dimms or 128Gb dimms at 2666MHz, the 64's run around $370ish each for a $4400 total or the 128's run @ $1400 each for a piddly little $16,800ish.

Feel cheap and small yet? 😅
Or pick up some older server that's already maxed out with 512GB of older FBDIMMs. :D Slower, but man all that ram!

Too bad the LRDIMMs aren't increasing in size as much as speed--the ddr3 days went from 4-8-16-32 pretty quick, but then the speeds stayed the same.