News 3D DRAM Proposal Paves the Road for a Density Increase

[Admin]

Lam Research has proposed 3D DRAM as a pathway to higher density memory solutions.

3D DRAM Proposal Paves the Road for a Density Increase : Read more

 

[InvalidError]

DRAM is quite similar to NAND apart from the writing/erasing method and I have said several months ago that we'll eventually see multi-layered DRAM made in a similar manner.

Not sure what the big deal is with "etching sideways" when conventional lithography lets you do fundamentally the same thing by adding layers of whatever material you need and then etching them to whatever shapes you need.

 

[Geef]

This isn't too much of a problem. PC memory hardly ever needs to be upgraded. For many years 8GB of memory was enough for everyone. Then it moved up to 16GB.

For bigger needs like servers there are a ton of slots for memory so its not really an issue, if it is just change motherboards for more slots.

Maybe in 20+ years it might become an issue for servers but PCs will be just fine running at - double the memory every 10 years? - A 64 GB PC as a base model and gamers running 128GB.

 

[InvalidError]

For bigger needs like servers there are a ton of slots for memory so its not really an issue, if it is just change motherboards for more slots.

The number of slots isn't the only problem. You also have the cost of platforms with more slots or slots that support higher density DIMMs. There are applications that can benefit from having more RAM but aren't necessarily worth having that much extra RAM at current prices. Then you have applications that already use all of the RAM that can be practically tossed at them and could use more if there was a cost-effective way of plugging it in. If you can cram 16X the amount of memory per chip, that opens up the possibility of stuffing all of the bandwidth mainstream CPUs and GPUs need in a single HBM-like chip stuffed directly on the CPU/GPU substrate or even 3D-stacked on a memory/cache/IO die/tile.

And besides all of the above reasons and any others I may have missed, nobody is going to protest being able to buy 2-4X as much RAM for a given amount of money either.

 

[Diogene7]

I would much, much prefer that they allocate ressources to develop manufacturing tools to scale up the manufacturing of Non-Volatile Memory (NVM) such as VCMA MRAM, or VG-SOT-MRAM (as per concept from European research center IMEC) to replace volatile DRAM as Non-Volatile Memory is a needed disruptive technology to enable so many improvements like « Normally-off Computing » : this is soooo much needed !!!

 

[InvalidError]

I would much, much prefer that they allocate ressources to develop manufacturing tools to scale up the manufacturing of Non-Volatile Memory (NVM) such as VCMA MRAM, or VG-SOT-MRAM (as per concept from European research center IMEC) to replace volatile DRAM as Non-Volatile Memory is a needed disruptive technology to enable so many improvements like « Normally-off Computing » : this is soooo much needed !!!

DRAM in self-refresh mode is only 20-30mW per chip, a negligible part of system power next to the 10-70W it takes just to display a static screen between key presses depending on how large and how bright the screen is. Most modern motherboards waste more power on on-board RGB than necessary for DRAM self-refresh.

If you want to reduce idle/standby power consumption, a better thing to focus on is reducing the number of background processes that needlessly wake up the CPU and its peripherals.

 

[hotaru251]

PC memory hardly ever needs to be upgraded. For many years 8GB of memory was enough for everyone. Then it moved up to 16GB.

for most ppl? sure.

However the more we get to "ai" type stuff more ram will be needed. (again this is for niche use cases but they will exist)

if it is just change motherboards for more slots.

has limitations as your memory traces get longer & your speed/latency suffer as result.

 

[usertests]

Controversial opinion: I would like higher memory capacity for less money. Let's see $0.10/GB memory.

This isn't too much of a problem. PC memory hardly ever needs to be upgraded. For many years 8GB of memory was enough for everyone. Then it moved up to 16GB.

For bigger needs like servers there are a ton of slots for memory so its not really an issue, if it is just change motherboards for more slots.

Maybe in 20+ years it might become an issue for servers but PCs will be just fine running at - double the memory every 10 years? - A 64 GB PC as a base model and gamers running 128GB.

Consumer memory requirements have mostly plateaued, suspiciously around the same time frame that cost per bit stopped reliably declining and memory prices zigzagged every few years.

Graph of Memory Prices Decreasing with Time

If everybody's memory suddenly quadrupled, we would have a good time. We still have 4 GB being sold in low-end laptops and tablets. Games could probably be made to use 64 GB soon, not necessarily as a requirement, but to brute force around slower storage. Stack a large amount of DRAM on a mega APU and that could be used for gaming or machine learning.

In servers/enterprise, many users could use more, and would also appreciate the cost per bit plummeting again.

If 3D DRAM octuples capacity/density at 8x the price, it doesn't matter that much. If it octuples at 0.5-2x the price, then things get interesting.

 

[Diogene7]

DRAM in self-refresh mode is only 20-30mW per chip, a negligible part of system power next to the 10-70W it takes just to display a static screen between key presses depending on how large and how bright the screen is. Most modern motherboards waste more power on on-board RGB than necessary for DRAM self-refresh.

If you want to reduce idle/standby power consumption, a better thing to focus on is reducing the number of background processes that needlessly wake up the CPU and its peripherals.

The « Normally-Off Computing » concept apply to the whole spectrum of computing devices and the benefits are not limited solely to lower power consumption, even though it is one of the important benefit, especially for battery operated devices (ex: IoT devices)

Yes one of the goal is to achieve overall lower system power consumption (through spintronics related technologies), but there is also the great advantage of much, much lower latency as you may not need (or at least less) need to load (shuffle) software/data from a slow non volatile storage device (HDD, SSD,…) to a fast volatile working memory (DRAM) : your Non-Volatile working Memory (NVM) (like MRAM) could be both used at the same time as fast, NVM DRAM and storage.

Your system only consume energy when it is doing a task, not when it is in idle, nor shut down, and is always immediately ready as soon as you turn on power (no boot time (or at least much less)) : I think that people don’t really realize how groundbreaking it would be because, as of 2023, such system doesn’t yet exist m, and therefore they never yet have the opportunity to experience the benefits of such a system (as Steve Jobs famoulsly said « A lot of times, people don't know what they want until you show it to them. »)

You could design an energy harvesting « bi-stable » systems (especially IoT devices, mobile devices,…) with gigabytes bi-stable, Non Volatile working memory/storage (VCMA MRAM, or VG-SOT-MRAM) and bi-stable color E-Ink display, with an embedded Artificial Neural Networks (ANN), for which the weights are stored in Non-Volatile SRAM/cache (VG-SOT-MRAM) : In Idle, your system wouldn’t consume any power, but as soon as a sensor is triggered, then your system wakes up, data go through the Non-Volatile ANN to decide what to do (compute), execute, and then go back to sleep.

For many infrequently triggered, battery operated IoT sensors it would be game changing : if the IoT device can harvest energy from the environment (small solar panel, vibration,…), and store it in a small battery, then it may provide enough energy to never have to manually recharge or change a non rechargeable coin cell battery in your devices.

 

[InvalidError]

Your system only consume energy when it is doing a task, not when it is in idle, nor shut down, and is always immediately ready as soon as you turn on power (no boot time (or at least much less)) : I think that people don’t really realize how groundbreaking it would be because, as of 2023, such system doesn’t yet exist

Standby mode has existed for over 20 years and gives you the benefit of a computer that wakes up faster than the monitor can sync up with the video signal. Having a computer that is "instant on" is nothing new to people who can be bothered to use it.

For many infrequently triggered, battery operated IoT sensors it would be game changing : if the IoT device can harvest energy from the environment (small solar panel, vibration,…), and store it in a small battery, then it may provide enough energy to never have to manually recharge or change a non rechargeable coin cell battery in your devices.

What sort of energy-harvesting IoT device does sufficiently complex tasks that it needs so much RAM that self-refreshing RAM becomes a meaningful concern? Typical sensors don't really do much besides measure, transmit measurement, go back to sleep. You can do that sort of stuff in less than 1KB of (S)RAM... unless you are running a few million lines of bloat in the form of an OS when all you really need from it is a barebones network stack.

The most popular IoT devices are things like voice assistants, smart plugs, doorbells, surveillance cameras, thermostats, etc., all of which wired/plugged-in, most of which always-on, continuously recording to catch trigger words or motion. Not much point in worrying about 30mW of self-refresh when the bus IO is drawing 300+mW.

 

[The Historical Fidelity]

I would much, much prefer that they allocate ressources to develop manufacturing tools to scale up the manufacturing of Non-Volatile Memory (NVM) such as VCMA MRAM, or VG-SOT-MRAM (as per concept from European research center IMEC) to replace volatile DRAM as Non-Volatile Memory is a needed disruptive technology to enable so many improvements like « Normally-off Computing » : this is soooo much needed !!!

Intel from 2016 is calling and asking where you were when they were losing money left and right producing non-volatile Optane memory.

 

[abufrejoval]

While density is a challenge, you may have left out why it's so important to overcome.
Because most would think it's just for capacity, while in fact it's mostly for bandwidth reasons.

Currently the most expensive parts in terms of production cost of the H100 cards is the HBM3 stacks.
And the main reason they have to resort to TSV and stacking is because they need the super wide busses to get better bandwidth than ordinary RAM.

Because LLMs are operating by sequentially predicting evey next token from the previous ones, they need to go through lots of the model all the time in search for the proper weights and you'll see how they are starved by memory much more than by compute resources, as they show very 20-30% compute utilization but 100% used memory bandwidth when you just run a single instance locally.

Of course, some of it is capacity, too, beause the model has to fit into GPU memory in its entiety.

A normal DDR5 RAM channels offers around 50GByte/s at 64 width, a high-end GDDR6X RTX 4090 at 6x width (384 bits) achieves 1000GByte/s and HBM3 delivers 3200GByte/s at 1024 width or 64x DRAM speed.

If the LLM had all day to answer a single question, current DRAM would do, but if you want to stay within reading or listening speed of humans, memory bandwidth needs to scale linear to model size, ...which it doesn't.

Other models and game graphics have much more locality or are less bandwdith dependent which is why they work well enough with bigger caches and "only" GDDRxx.

But if you can go really wide with DRAM that's stacked cheaply, Apple's Mx chips prove the value of that approach, unfortunately more to its bottom line than to consumers, who pay HBM prices for a DRAM effort, but that's another story.

 

[ekio]

This isn't too much of a problem. PC memory hardly ever needs to be upgraded. For many years 8GB of memory was enough for everyone. Then it moved up to 16GB.

For bigger needs like servers there are a ton of slots for memory so its not really an issue, if it is just change motherboards for more slots.

Maybe in 20+ years it might become an issue for servers but PCs will be just fine running at - double the memory every 10 years? - A 64 GB PC as a base model and gamers running 128GB.

I think that you should stop giving your opinion because you really sound like a clown here.
You think every use of ram is for checking their mail like you and that a few gb is enough for everyone.
That’s as stpuid to say that than saying why do industry need 50000hp engines ?? My car works just fine with a 100hp engine, they should stop bothering with bigger engine.
In many professional domains, having 10TB of ram would be the beginning of the solution…

 

[vehekos]

Those resistances are tiny. I wonder when they will stop being visible.

 

[Diogene7]

Standby mode has existed for over 20 years and gives you the benefit of a computer that wakes up faster than the monitor can sync up with the video signal. Having a computer that is "instant on" is nothing new to people who can be bothered to use it.

What sort of energy-harvesting IoT device does sufficiently complex tasks that it needs so much RAM that self-refreshing RAM becomes a meaningful concern? Typical sensors don't really do much besides measure, transmit measurement, go back to sleep. You can do that sort of stuff in less than 1KB of (S)RAM... unless you are running a few million lines of bloat in the form of an OS when all you really need from it is a barebones network stack.

The most popular IoT devices are things like voice assistants, smart plugs, doorbells, surveillance cameras, thermostats, etc., all of which wired/plugged-in, most of which always-on, continuously recording to catch trigger words or motion. Not much point in worrying about 30mW of self-refresh when the bus IO is drawing 300+mW.

You involuntarily have a « survivor bias » here (you could review Youtube video from Veritasium about it) in your assesment of IoT devices, because you are not considering all battery operated IoT applications that can’t yet exist (so you don’t take into account) because of the continuous energy draw that represent the 30mW self-refresh power consumption you are talking about.

There are many, many research papers about Non Volatile Memory (especially about MRAM) that point the issue that the Idle power consumption represent for battery operated IoT devices : many, many more (small) different kinds (temperature, humidity,…) wireless sensors (with a small battery + ideally energy harvesting part) would be able to populate our daily environment only if you would never have to bother about recharging them / changing the coin cell, which is not the case nowadays.

Nowadays, if you would have 100 or 1000 (or more) of them, it would be very time consuming to take the time to recharge them / replace the coin cell (and expensive if you have to pay for the human labor). (I do want my (home) environment to be populated by 1000’s sensors / small actuators to significantly improve life conditions, and it is more true for elderly and disabled people).

Regarding the « instant-on » you are talking about, it still requires the device being continuously powered-on, and what I want is a fully « powered down » device, that can be instantly ready to use without going through the lengthy booting process as soon as power is back…

Again, it doesn’t yet exist (still in R&D) so I believe most people don’t really grasp / apprehend how disruptive it would be in terms of new opportunities that it unleashes…

But with recent advancements in spintronics related technologies, especially Non-Volatile Memory MRAM (like VG-SOT-MRAM (there is an eetimes article dated February 2023 about it)), we are getting closer than ever before (still more likely to debut in expensive/professional devices first, and probably unfortunately still quite a long way for consumers).

 

[Diogene7]

Intel from 2016 is calling and asking where you were when they were losing money left and right producing non-volatile Optane memory.

Optane memory seems to be some kind of Phase Change Memory (PCM), which seems to gave very high read/write power consumption : this makes this technology a very bad fot for IoT devices to begin with…

The concept of Non-Volatile Memory (NVM) is by itself a good one, but you need to meet several key requirements to make sense in multiple different kind of applications.

As of 2023, so far, the NVM technology that seems to fit most of the requirements seems to be related to spintronics technologies, like VCMA-MRAM (that seems to offer very low read/write power consumption) or VG-SOT-MRAM (concept from European research center IMEC).

There is a February 2023 eetimes article talking about VG-SOT-MRAM, that seems to be currently investigated mainly for cache/SRAM replacement (but in theory, some of the technologies could be used differently to expand the application space)

 

[InvalidError]

You involuntarily have a « survivor bias » here (you could review Youtube video from Veritasium about it) in your assesment of IoT devices, because you are not considering all battery operated IoT applications that can’t yet exist (so you don’t take into account) because of the continuous energy draw that represent the 30mW self-refresh power consumption you are talking about.

If your IoT temperature/moisture/light/etc. sensor requires 2GB of DRAM to operate, start your power-optimizing journey with ditching Linux/BSD from your IoT device payload. A micro-controller running a bare-metal software stack could do that job using only 1KB of on-chip SRAM and 1/1000th the CPU-power.

 

[Diogene7]

My understanding/knowledge is limited in that domain, but yes, I think it is indeed one way of doing it more efficiently.

That said, there are likely many IoT devices for which more Non-Volatile Memory (NVM) could open new opportunities : hard to foresee which one exactly, but I have no doubt about that…

 

[InvalidError]

That said, there are likely many IoT devices for which more Non-Volatile Memory (NVM) could open new opportunities : hard to foresee which one exactly, but I have no doubt about that…

If your IoT device really requires GBs worth of working memory and storage, chances are it requires a lot more processing power than you can comfortably hook up to a coin/button-sized rechargeable battery.

 

[abufrejoval]

I would much, much prefer that they allocate ressources to develop manufacturing tools to scale up the manufacturing of Non-Volatile Memory (NVM) such as VCMA MRAM, or VG-SOT-MRAM (as per concept from European research center IMEC) to replace volatile DRAM as Non-Volatile Memory is a needed disruptive technology to enable so many improvements like « Normally-off Computing » : this is soooo much needed !!!

Well since Martin Fink could not deliver the memristor he promised, that looks further away than ever.

All the working NV-RAM technologies I've been seeing can't go near even current RAM density and the last I've read about was MRAM on a 45nm process for basically stop-clock logic: you can implement all register files and SoC caches as non-volatile and therefore don't need to spend time and effort on saving processor state when going to sleep.

It's interesting, even available, but in an embedded power domain that never got anywhere near the IoT hype predictions. So in fact the technology is there, I believe it's even used in some embedded storage controllers but the market is in fact so tiny it hardly makes the news.

Martin's memristor promised a single technology from register file storage to what's still stored on tape, SRAM speeds and tape economy and that would indeed be transformational.

The busy evolution of ever expanding storage technologies and densities at least makes for an interesting life as IT architect, Martin's vision would have been one big bang and then little else to do. So in a perverse way I'm happier about it now than I was ever before...

 

[InvalidError]

All the working NV-RAM technologies I've been seeing can't go near even current RAM density and the last I've read about was MRAM on a 45nm process for basically stop-clock logic

The latest numbers I can find is 0.041 um^2 for 28nm MRAM vs 0.0016 um^2 for Micron's D1a cells and 0.002 um^2 for previous-gen DRAM. So the smallest MRAM so far has ~20X worse density than contemporary DRAM.

 

[Diogene7]

Well since Martin Fink could not deliver the memristor he promised, that looks further away than ever.

All the working NV-RAM technologies I've been seeing can't go near even current RAM density and the last I've read about was MRAM on a 45nm process for basically stop-clock logic: you can implement all register files and SoC caches as non-volatile and therefore don't need to spend time and effort on saving processor state when going to sleep.

It's interesting, even available, but in an embedded power domain that never got anywhere near the IoT hype predictions. So in fact the technology is there, I believe it's even used in some embedded storage controllers but the market is in fact so tiny it hardly makes the news.

Martin's memristor promised a single technology from register file storage to what's still stored on tape, SRAM speeds and tape economy and that would indeed be transformational.

The busy evolution of ever expanding storage technologies and densities at least makes for an interesting life as IT architect, Martin's vision would have been one big bang and then little else to do. So in a perverse way I'm happier about it now than I was ever before...

It is still in R&D phase, but MRAM is progressing well, and gets closer to be an unified (both working memory and storage) Non-Volatile Memory (NVM).

The European research center IMEC, known to create many of tools used in semiconductor manufacturing fabs, is working on developing (a manufacturable) VG-SOT-MRAM concept, for embedded SRAM cache replacement, but with some modifications, it may also have some interesting properties for Artificial Neural Networks (ANN)

https://www.eeweb.com/mram-technologies-from-space-applications-to-unified-cache-memory/

https://www.eetimes.eu/sot-mram-architecture-opens-doors-to-high-density-last-level-cache-memory-applications/

I would expect that some form of VG-SOT-MRAM may start to appear in devices toward 2025 - 2030 timframe, especially if it can bring significant power reduction for AI applications, which could trigger Higher Volume Manufacturing, and from there help reduce the cost of this technology… But yes, it will likely take a lot of time before seeing economies of scale kicking in…

I am really, really wish that Non-Volatile spintronics / (VG-SOT) MRAM to be reasonably cost competitive with volatile DRAM as soon as possible, as I believe Non-Volatile Memory (NVM) is a key (required) technology to enable so many new opportunities !!!

 

[Diogene7]

The latest numbers I can find is 0.041 um^2 for 28nm MRAM vs 0.0016 um^2 for Micron's D1a cells and 0.002 um^2 for previous-gen DRAM. So the smallest MRAM so far has ~20X worse density than contemporary DRAM.

Thanks very much for this information.

I would think MRAM already is (or is close to) better density that the one you mention, because NXP has announced that it should be using it in 2025 in a 16nm TSMC automotive process.

Also there are multiple varitions of MRAM, as it could be tuned with different trade-off (speed, endurance, low power,… For exemple, VCMA-MRAM is very good in low power consumption), which could make Non-Volatile MRAM very versatile, and some variation may be easier to manufacture and to scale to smaller nodes than others.

But to compete with DRAM is really, really hard because even if you could make an MRAM have near DRAM density, at first, cost will likely be much, much more expensive (no idea how much more, but I would think it could easily be at least 100x more…).

That is the reason why I hope that European research center IMEC could find useful applications to drastically (10x / 100x / 1000x) improve AI / Artificial Neural Network (ANN) energy efficiency as it would provide a much greater financial incentive to speed-up the the manufacturing scale-up of this technology (which is needed to significantly reduce cost, and make it more cost competitive with DRAM…)

NXP and TSMC to Deliver Industry’s First Automotive 16 nm FinFET Embedded MRAM

NXP Semiconductors, today announced its collaboration with TSMC to deliver the industry’s first automotive embedded MRAM in 16 nm FinFET technology.

www.nxp.com

https://www.eeweb.com/mram-technologies-from-space-applications-to-unified-cache-memory/

https://www.eetimes.eu/sot-mram-architecture-opens-doors-to-high-density-last-level-cache-memory-applications/

 

[abufrejoval]

DRAM in self-refresh mode is only 20-30mW per chip, a negligible part of system power next to the 10-70W it takes just to display a static screen between key presses depending on how large and how bright the screen is. Most modern motherboards waste more power on on-board RGB than necessary for DRAM self-refresh.

If you want to reduce idle/standby power consumption, a better thing to focus on is reducing the number of background processes that needlessly wake up the CPU and its peripherals.

While that is true, it's also very much a personal computer perspective.

Most of the servers I operate, even my larger workstations, spend much more power (and money!) on RAM than they spend on the CPU on average (no, I'm no hyperscaler able to keep them at 90% utilization), some of the really big machines even at peak.

And no, not a single SAP HANA among them.

And yes, that's because even the most modern servers are designed with the base assumption that their owners will keep them busy and not tolerate chaotic latencies and response times any RAM energy savings heuristic would likely produce if they started trying.

As Google loves to say, broken SLAs cost far more than a bit of overdesign.

Personally whenever I have conflicts like that, I prefer to have a choice. I can imagine that unused bits in page table entries could be used by operating systems to mark pages of RAM as less latency sensitive, e.g. because they might hold data for batch jobs or their buffer cache. That might not fit the DRAM chip granularity at all without a bit of help, but if operating systems (and their applications) ever learned to handle Optane properly, they might also be taught to handle sleepy vs. busy RAM for a few Gigawatts of savings globally.

 

[abufrejoval]

Thanks very much for this information.

I would think MRAM already is (or is close to) better density that the one you mention, because NXP has announced that it should be using it in 2025 in a 16nm TSMC automotive process.

Also there are multiple varitions of MRAM, as it could be tuned with different trade-off (speed, endurance, low power,… For exemple, VCMA-MRAM is very good in low power consumption), which could make Non-Volatile MRAM very versatile, and some variation may be easier to manufacture and to scale to smaller nodes than others.

But to compete with DRAM is really, really hard because even if you could make an MRAM have near DRAM density, at first, cost will likely be much, much more expensive (no idea how much more, but I would think it could easily be at least 100x more…).

That is the reason why I hope that European research center IMEC could find useful applications to drastically (10x / 100x / 1000x) improve AI / Artificial Neural Network (ANN) energy efficiency as it would provide a much greater financial incentive to speed-up the the manufacturing scale-up of this technology (which is needed to significantly reduce cost, and make it more cost competitive with DRAM…)

I think the current Wikipedia article on MRAM has a rather balanced and fair assessment of the current state. And the net result is that in terms of endurance, density, speed, retention and power it's always a compromise, never able to push out the alternatives 1: 1. And it's not the fabrication scale that limits it, it's the underlying physics.

Stop-clock logic with its inherent resilience advantage against power loss is a creative escape route out of these compromises and sounds attractive for embedded use cases like automotive.

But there is so very little useful things an electric car can still do if it's got less power left than your smartphone, to have an entire industry flock to MRAM.

When smartphones use DRAM and Flash, MRAM may still have a niche in some ultra-high security or criticality enclave, even within phones, but still no mass market high volume appeal.

I work with CEA and IMEC people and I love their creativity and drive. But 250 layer Flash and current DRAM prices have a lot more mass market storage appeal than MRAM today.