News Benchmarking Blackwell and RTX 50-series GPUs with Multi Frame Generation will require some changes, according to Nvidia

[Pierce2623]

Benchmarking should only count real frames from the game engine. Just ignore frame generation for all benchmarking purposes. Those aren’t actually frames. Why count them as such?

 

[Pierce2623]

And thus the other shoe just dropped, nVidia is providing "guidance" for its affiliated partners, which I suspect they are required to follow under NDA or lose future access to products. This is why I'm going to wait until Steve on GN or Jay over at JTC do fully independent non-affiliated, unguided, and no-NDA testing. Steve was talking about how their team is looking to buy or borrow cards for the testing and report back to people about it.

Wait…you think Jayz2Cents isn’t a shill ?….ok then.

 

[JarredWaltonGPU]

Benchmarking should only count real frames from the game engine. Just ignore frame generation for all benchmarking purposes. Those aren’t actually frames. Why count them as such?

Because even though they're not the same as rendered frames, they can have an impact on how a game feels. There are definitely games where, even on lower tier hardware like an RTX 4060 Ti 16GB, turning on framegen can make a game feel better than non-FG. It's a very murky situation overall, though.

But it's the same problem as upscaling. I've reached the point where my benchmarks will generally be at native for comparison purposes, and I may use DLSS, FSR, XeSS, etc. for additional details, but the baseline will be native. Then I'll try the various upscaling and framegen solutions where applicable and tell how that goes.

 

[bolweval]

This!

Its sad that were approaching the POST TRUTH world, where everything is made up to fool people.
News, Video, Audio, Pictures, Ar, Capabilities, Power..etc..

Psssst, video games aren’t real..

 

[Pierce2623]

Because even though they're not the same as rendered frames, they can have an impact on how a game feels. There are definitely games where, even on lower tier hardware like an RTX 4060 Ti 16GB, turning on framegen can make a game feel better than non-FG. It's a very murky situation overall, though.

But it's the same problem as upscaling. I've reached the point where my benchmarks will generally be at native for comparison purposes, and I may use DLSS, FSR, XeSS, etc. for additional details, but the baseline will be native. Then I'll try the various upscaling and framegen solutions where applicable and tell how that goes.

Ok so let’s say I’ve got a 120Hz display but my GPU is only powerful enough for 100fps through standard DLSS. Would that 100fps not give me a better experience and less latency overall than using frame gen to get up to 120 but only having a “real” 60 fps?

 

[-Fran-]

I just had a question pop into my mind: the best justification for FrameGen would be to compensate for VSync "on" situations, so it would actually help there when Adaptive Sync is not available, making that experience actually better. Or when you go outside of the refresh range, I guess. So, the question was: does FrameGen require VSync to be turned on?

Also, how does the tearing come into play with FrameGen as well? I mean, unless you have a display with 240Hz, the chances of getting tearing when going over 120Hz is still there. So, I'll have to imagine there's still tearing happening?

Regards.

 

[Pete Mitchell]

Leave out the smoke and mirrors garbage entirely and test what matters, native resolution horsepower.
Testing faked frames will give faked results, which is exactly what nvidia wants you to do, sell the smoke and mirrors to the masses.

If you can't tell the difference between the two visually, does it matter? You people screaming "fake frames" make me laugh. I highly doubt you could even tell the difference.

 

[palladin9479]

Benchmarking should only count real frames from the game engine. Just ignore frame generation for all benchmarking purposes. Those aren’t actually frames. Why count them as such?

I believe the nVidia "guidelines" place special emphasis on those things, I think they have to be in the review. A LOT of nVidia's hype, and I mean enough to fill a Scrooge McDuck money bin, is centered of the letters A and I being put into everything and that includes the shiny new "AI enhanced gaming". They can have other things in the review but there is gonna be a graph with the 50xx framegen vs the 40xx framegen vs everything else without framegen.

From my POV it's no different then back when ATI was caught "padding" the numbers with the Radeon 8000 series.

ATI Radeon 8500 and Cheating in Benchmarks

As a follow up to Bates’ post the other day, ATI’s Radeon 8500 has been getting some very nasty PR lately. Kyle Bennett, of [H]ardOCP, has done some very interesting analysis in his review of the 8500…

arstechnica.com

The ATI driver would render frames at a lower quality if it thought you were benchmarking. Here nVidia is instead selling lower quality frames as a "feature" then convincing everyone that "bigger number better". "You won't notice the quality anyway, bigger number better, the more you buy the more you save".

With DLSS upscaling I can see a use if someone was trying to render to a higher resolution then they could otherwise do. Like a 4060 or 5060 trying to do playable frame rates on a 2160p screen. It's no where near the same as natively rendering to 2160 but might be better then rendering to native 1080p.

 

[palladin9479]

I just had a question pop into my mind: the best justification for FrameGen would be to compensate for VSync "on" situations, so it would actually help there when Adaptive Sync is not available, making that experience actually better. Or when you go outside of the refresh range, I guess. So, the question was: does FrameGen require VSync to be turned on?

Also, how does the tearing come into play with FrameGen as well? I mean, unless you have a display with 240Hz, the chances of getting tearing when going over 120Hz is still there. So, I'll have to imagine there's still tearing happening?

Regards.

FrameGen doesn't help anything, it's literally just interpolation. They have to use the term "AI" to pump more nVidia stock and sell more products though.

At a 2:1 ratio it's like this (interpolation)
Render frame 1
Render frame 2
Display frame 1
Interpolate frame 1.5
Display frame 1.5
Repeat


You can add as many interpolated frames as you want, it's anything between the two real frames they are based on. You need at least two unique frames to start the chain and afterwards you can just keep going by essentially doing triple buffering and just interpolating frames between the two back buffers before rotating them to the front. The only thing "unique" that nVidia is doing is the algorithm used to interpolate as trained by analyzing other games.

 

[freedomnow]

FPS has never been relevant. FPS was invented by folks who thought increasing this number meant you would have a better gaming experience thus pushing you to buy expensive cards. Watch a movie and you realize it's not important. Anything above 90 fps is not detectable, but you can detect a drop in frame rate even if it's 300 fps to 250 fps. Ever wonder what happens to the frames that are generated when you go past the refresh rate of the monitor, they get thrown away. You spend all that power to render the frame and you never see it.

Quality is not a measure of image quality (unless you take a screenshot) but quality is solely measured by latency and the delta between frames. Latency is key for providing fast interaction. Delta between frames is what you perceive as smoothness. The best gaming experience is to render frames at the exact same interval. If this interval just so happened to be synchronized with the monitors maximum refresh rate this gives you the maximum smoothness. In theory a game engine could use 4x frame generation to target the refresh rate of the monitor with the exact same interval . Then have the update / input loop run as fast as your CPU can process. The idea is to you reduce the load on the GPU so the CPU can run as fast as possible to process inputs, then you would have created the highest quality experience you can possibly achieve.

 

[aberkae]

If I wanted to see frame gen numbers I would just look at the Nvidia marketing slides! Also can you please benchmark using closed case to check for variances vs open bench for potential throttling?

 

[Elusive Ruse]

Thanks for sharing your plans and how you are gonna tweak your approach to benchmarking the new GPUs, Jarred, keeping your focus on native rendering and RT with a side of upscaling is I believe the ideal approach for the main review. You can do a deep-dive into MFG separately once you find the time.

 

[butidontwantausername]

Nothing changes. It doesn't matter if it's rasterized or rt, native resolution is the only true performance metric. And as for not testing everything, that's what already happens. We all know testing every anisotropic filtering level is a waste of time and nobody does it. It's the same with frame gen. Unless you're specifically testing upscaling, like with a new version, or frame gen vs frame gen it's just a waste of time.

 

[NinoPino]

Thank you Jarred for your communication. I totally agree with your choice.
A separate and specific analysis, for upscaling and frame generation quality and latency, is the better thing to do. Also the number of titles, chosen for quality analysis, can be much lower than those for performance.

 

[Kashmir35]

Contrary to what's stated in the article, MFG doesn't add much latency compared to FG, most of the added latency comes from buffering the two rendered frames.

 

[JarredWaltonGPU]

If I wanted to see frame gen numbers I would just look at the Nvidia marketing slides! Also can you please benchmark using closed case to check for variances vs open bench for potential throttling?

Unfortunately, the PCAT interposer card makes it hard/impossible to use a normal case. And I swap GPUs so often that the case just gets in the way. If you have a decent size ATX case with good airflow, that tends to run just as cool or in some cases cooler than an open test bench. If you're trying to stuff a power hungry GPU into a mini-ITX case? Well, have fun with that, and plan on higher speed fans that make more noise to keep things cool.

Contrary to what's stated in the article, MFG doesn't add much latency compared to FG, most of the added latency comes from buffering the two rendered frames.

I don't think I said it added more latency than regular framegen. It should basically be the same amount of latency, because it's just interpolating more frames in between the rendered frames. Well, except if the performance improvement from 2X framegen to 4X framegen (one vs. three interpolated frames) is only 65%, that means the base sampling rate will be lower for the same "framerate." And this is also why Reflex 2 exists, to try to compensate for even more latency. But Reflex 2 with no framegen will be even better than Reflex 2 plus MFG.

 

[Kashmir35]

Without framegen, user input would get sampled every ~10ms. With framegen, that drops to every ~20ms. With MFG, it could fall as far as sampling every ~40ms.

Here, maybe I read this wrong or don't understand something but, to me, it seems to imply that for a given base framerate (no FG), MFG would double the latency compared to FG. Which is what I expected when I first learned about the technology but it turns out my worries were misplaced. Still disappointed we're not getting FG by extrapolation because the use case for MFG will be relatively small I think. You'll need to hit a base 50-60 just like before, and it'll get you around 180-240 fps and not everyone has the kind of monitor required to display that.

 

[JohnyFin]

Thats correct. For 50xx we need new benchmark which considerate also latency... otherwise we will end up with 1000 fps and latency 20-40 ms. Which will misslead clients (they will not show latency in market)

 

[thestryker]

FrameView is fine, AFAICT. I've tried PresentMon and OCAT in the past, and FrameView basically gave the same results. It's critical for me because I want to get the power data (and temp and clocks as well), which OCAT and PresentMon don't provide. But FrameView uses the same core code, just with some additional data collection going on.

Forgot to look them up yesterday to make sure I had the right info, but wouldn't CapFrameX + ElmorLabs Benchlab with PMD PCIe slot adapter be the way to go to eliminate any potential vendor issues and pull power data directly?

Obviously this is basically the worst possible time to be making any changes since we've got new cards coming from all 3 vendors. Just something to potentially look into after the vacation you hopefully get after testing the 7 cards (and potentially the new DLSS/FSR/XeSS) coming out in the next couple of months.

 

[Elusive Ruse]

Unfortunately, the PCAT interposer card makes it hard/impossible to use a normal case. And I swap GPUs so often that the case just gets in the way. If you have a decent size ATX case with good airflow, that tends to run just as cool or in some cases cooler than an open test bench. If you're trying to stuff a power hungry GPU into a mini-ITX case? Well, have fun with that, and plan on higher speed fans that make more noise to keep things cool.

Have you seen Hardware Unbox’s new video about the B580 losing performance much more than the 4060 and 7600 with lower end CPUs? IIRC the 1080p results put the card on par or sometimes slower than the 4060.

 

[tamalero]

And thus the other shoe just dropped, nVidia is providing "guidance" for it's affiliated partners, which I suspect they are required to follow under NDA or lose future access to products. This is why I'm going to wait until Steve on GN or Jay over at JTC do fully independent non-affiliated, unguided, and no-NDA testing. Steve was talking about how their team is looking to buy or borrow cards for the testing and report back to people about it.

You know even Nvidia is worried about their BS when they do stuff like this.. yikes..

Psssst, video games aren’t real..

Prices are real.

 

[edzieba]

When everything renders the same way, we should have the same output — with only minor differences at most. But in the modern era of upscaling and frame generation, not to mention differences in how ray tracing and denoising are accomplished? It becomes very messy.

Sounds like going back to the days of GPU reviewing where ATI and Nvidia had different implementations for AA and AF (and other features) with different overall image quantity. As well as benchmarking, actual image quality had to be compared too, so you could end up with Card A being faster, but Card B providing higher image quality.

Funnily enough, in the days when radical changes to GPU capabilities (e.g. introduction of hardware T&L, introduction of pixel shaders, introduction of geometry shaders, introduction of unified shaders, etc) which left different GPUs being capable of different things and different games being supported by GPUs only after certain features were introduced, this was just the norm. Now, after a few years of the DX11 featureset being static, suddenly new featureset requirements are getting everyone in a panic.

 

[JarredWaltonGPU]

Here, maybe I read this wrong or don't understand something but, to me, it seems to imply that for a given base framerate (no FG), MFG would double the latency compared to FG. Which is what I expected when I first learned about the technology but it turns out my worries were misplaced. Still disappointed we're not getting FG by extrapolation because the use case for MFG will be relatively small I think. You'll need to hit a base 50-60 just like before, and it'll get you around 180-240 fps and not everyone has the kind of monitor required to display that.

If the framerate is constant, sampling latency changes. My point was to show that, if you had three different GPUs delivering 100 "FPS" to the monitor, one without framegen, one with framegen, and one with MFG, the base FPS is wildly different. Basically, it's to prove that framegen doesn't provide frames in the same way that normal rendering does. It feels different, in every game I've tested. It also looks different, usually smoother, so it's a personal decision on how much it helps (or not).

In general, it seems framegen gives about a 40~60 percent boost to "FPS" and MFG may give an additional 50~80 percent boost (or more!) over that. So what you really get in terms of user sampling rates will vary by the actual real-world performance. And that will vary on a game by game basis.

Let's say you have a game running at 100 FPS without any framegen. That's sampling input every 10ms, though actual latency depends on other factors as well. If framegen bumps the perceived FPS to 150, that would now have an 75 FPS base framerate before framegen, sampling input every 13.3ms. And then if MFG were to push that up to 225 FPS, the base framerate would be 56 FPS and input sampling every 17.8ms.

Again, that's just a hypothetical example and we'll need to do actual testing to see where things land. But that's probably close to what we'll see with actual games that support MFG.

 

[JarredWaltonGPU]

Forgot to look them up yesterday to make sure I had the right info, but wouldn't CapFrameX + ElmorLabs Benchlab with PMD PCIe slot adapter be the way to go to eliminate any potential vendor issues and pull power data directly?

Obviously this is basically the worst possible time to be making any changes since we've got new cards coming from all 3 vendors. Just something to potentially look into after the vacation you hopefully get after testing the 7 cards (and potentially the new DLSS/FSR/XeSS) coming out in the next couple of months.

I used a custom Powenetics setup previously to capture power use. It was totally independent of AMD, Intel, and Nvidia. It also looks janky AF, but it worked. The problem isn't just getting power, but getting it for every test. I had to run a separate workload with Powenetics to capture data, and basically it wasn't feasible to do that for every game. So when Nvidia made the PCAT, I tested some cards with both devices and found the results were basically the same.

The benefit is that FrameView (which uses PresentMon as a base and is free) talks to the PCAT and captures power for every single gaming benchmark I run. It's pretty awesome. People love to be suspicious about stuff like this, but really there's no indication I've seen that the FrameView or PCAT numbers are incorrect. Frankly, with Ada Nvidia pulled so far ahead on efficiency that it feels more like it just wanted to make sure everyone who wanted to work with them could get hard data showing just how efficient the 40-series was.

 

[TeamRed2024]

screw frame gen... only numbers I want to see between the generations is raw power.. ie turn off frame gen and DLSS.. I want to see RAW fps between the gen's.. I have a 4080.. I want to see raw 5080 and 5090 to compare to my raw 4080 data.

Agree.

I've never used DLSS. Ever.

I am interested in the raw improvement the 5090 has over the 4090. I've heard 33%.