Question What GPU Benchmarks Should I Add/Drop for Future Reviews/Hierarchy?

[JarredWaltonGPU]

Here's your chance to influence our future graphics card testing. Right now, our list of nine games that we use for GPU benchmarks consists of the following (parenthetical comments are API we use, and whether a game has AMD or Nvidia branding):

Borderlands 3 (DX12 AMD)
The Division 2 (DX12 AMD)
Far Cry 5 (DX11 AMD)
Final Fantasy XIV (DX11 Nvidia)
Forza Horizon 4 (DX12)
Metro Exodus (DX12 Nvidia)
Red Dead Redemption 2 (Vulkan)
Shadow of the Tomb Raider (DX12 Nvidia)
Strange Brigade (Vulkan AMD)

A few quick notes:

1. I really want to ditch Far Cry 5, FFXIV, and Strange Brigade -- I feel all are older and less meaningful representations of modern games. FFXIV in particular has got to go IMO.
2. I haven't been testing with any ray tracing or DLSS modes in the two games that support those features (Metro and SotTR), but when AMD's Big Navi arrives I'll very likely be looking to include at least two ray tracing games.
3. The final list of games should include a mix of AMD and Nvidia promoted titles, different APIs if possible (DX12 + Vulkan for sure, and a major DX11 game would be nice as well). It should also include a mix of genres if possible -- so Forza is currently my "car racing" benchmark, and I still feel it's better than the F1 20xx games ... but if you have strong feelings and want to argue the case for a different racing game, I'm open.
4. Ideally, I want games that are more demanding (on the GPU), better looking, and more popular -- so if a game is more demanding but no one plays it, I'd prefer not to use it.

My short list of games I may include when they become available:

Assassin's Creed Valhalla (AC games are usually very demanding and decently popular, but potentially buggy)
Cyberpunk 2077 (duh)
Dirt 5 (could be a reasonable swap out for FH4)
Doom Eternal (it's out, I've tested it already, but it takes more effort to benchmark)
Dying Light 2 (maybe -- the last one was a pain to benchmark)
Microsoft Flight Simulator (if there's a reliable way to test that's not a complete PITA)
Rainbow Six Quarantine (another big maybe)
Star Wars Squadrons (maybe -- Jedi Fallen Order was merely okay, didn't warrant inclusion IMO)
Vampire: The Masquerade -- Bloodlines 2 (if it doesn't suck)
Watchdogs: Legion (possibly)

Lots of other games I probably won't include, unless they end up being 'better' for testing GPUs:
Death Stranding (already out, not sure how 'relevant' it is -- weird game, pretty repetitive IMO, but maybe?)
Diablo 4 (probably not coming for a while, may not be very demanding graphically)
Gods and Monsters (maybe?)
Horizon Zero Dawn (belated PS4 port)
Marvel's Avengers (likely to be too 'light' IMO)
Project Cars 3 (I didn't do much with the previous 2, but maybe third time's the charm)
Serious Sam 4 (will it be any good, and will it push GPUs at all?)
Wasteland 3 (Unity engine, not super demanding most likely)
Werewolf: The Apocalypse - Earthblood (probably not going to be demanding or popular?

I'd like to keep the final list to no more than 10 games total. Games with built-in benchmarks are also lovely, because that can remove a lot of the difficulty of repeated testing and variability -- but I get why some prefer 'real' benchmarking. Anyway, that's it. Sound off and I'll be looking at responses to see if I need to seriously rethink anything I've listed above.

 

[Tom_nerd]

could you also include workstation tasks like blender using the GPU.

 

[Dantido]

Would recommend you to (re)include Warframe. Their latest open world (Cambion Drift) is quite demanding, much more than any other part of the game, and it's also quite beautiful.

 

[hotaru.hino]

For Microsoft Flight Simulator, supposedly you can have the AI fly a route. So at the very least you can have some repeatability as long as the other conditions are the same. The only thing you may have to turn off is cloud rendering since network hiccups could screw things up.

 

[animekenji]

Have you tried Unigine Superposition? That benchmark is brutal, even running on high end cards.

 

[Axon_Parker]

This might sound odd, but with the new 3080 FE thermal design, I'm very interested to see how case, CPU, and RAM temps will be impacted by the pass through FE fan that will blow air right up to the CPU.

Creating a standard platform for all air cooling and testing with open and closed case with both Intel and AMD and various CPU coolers would be a tremendous asset in deciding on how to proceed with a build using the new card. That might be a big ask, but that would be incredibly useful.

It would be even better if different ambient temps could be applied to the test PC to see thermal profiles.

 

[Axon_Parker]

Another test that would be interesting is system stability with different power and efficiency rated PSUs with these new power hog GPUs.

 

[1foxracing]

I second the Blender software test request but what I would really like to see id 4k Video rendering test with Vegas Pro17. Don't care about Adobe and all their buggy software period.
To me it's absolutely ludicrous to spend $500 let alone $1000 or $1500 on a graphics card to "play games" I use my computers to make money with photography and video work.

 

[sizzling]

Gears 5 has a fairly demanding built in benchmark.

 

[chfosmith]

Here is another vote for adding Microsoft Flight Simulator 2020 at 4K.
When the settings are at Ultra, the scenery is complex, and the clouds are fully defined, it will bring any GPU to its knees. (Less than 60fps)
This means much more powerful GPUs in the future can be properly evaluated.
It must be mentioned that Microsoft has already sold millions of copies of MFS 2020 world wide, so there will be a lot of interest in GPU ratings for this game.

 

[RagedAPE]

Your new list is solid, stick to it imo. Maybe add minecraft RT, control.

 

[-Fran-]

Having MMOs, like you had WoW in the past, would be nice.

Most MMOs still use DX9c and while they don't require expensive setups, they are still the popular game people plays. Having benchmarks on crowded areas would be a good thing and/or maps with a lot of things going on (like battlegrounds or big PvP areas).

Cheers!

 

[siddu patil]

does it run gta vice city properly. if right then thats the best computer in the world for me

 

[Juan_Bijero]

Well if you are going to test ray tracing, you should probably be using Control for your benchmarking.

 

[Umeed]

Here's your chance to influence our future graphics card testing. Right now, our list of nine games that we use for GPU benchmarks consists of the following (parenthetical comments are API we use, and whether a game has AMD or Nvidia branding):

Borderlands 3 (DX12 AMD)
The Division 2 (DX12 AMD)
Far Cry 5 (DX11 AMD)
Final Fantasy XIV (DX11 Nvidia)
Forza Horizon 4 (DX12)
Metro Exodus (DX12 Nvidia)
Red Dead Redemption 2 (Vulkan)
Shadow of the Tomb Raider (DX12 Nvidia)
Strange Brigade (Vulkan AMD)

A few quick notes:

1. I really want to ditch Far Cry 5, FFXIV, and Strange Brigade -- I feel all are older and less meaningful representations of modern games. FFXIV in particular has got to go IMO.
2. I

I'd say get rid of all of that AND shadow of the tomb raider. There's also a free bench mark tool that you can get from Steam it's called "Bright Memory Infinite Ray Tracing Benchmark". It's a fairly quick one that seems to focus on RTX and DLSS built in Unreal Engine, might be worth checking out? I used my 3080 ftw3 ultra on it and at 4k res got an avg 36fps with rtx = very high, and DLSS = quality lol

As for games themselves, I do think Anthem is worth a shot at least from a visual gpu driving perspective. Anthem 2.0 (possible no man's sky style redeemer?) will be a good option.

 

[Akira1364]

One general comment about the most recent version of the GPU Hierarchy... it seems like the positions of the 1660 Ti and 1660 Super were (accidentally?) swapped.

It makes no sense for the Super to be ahead... that would contradict this site's very own review of the 1660 Super, as well as basically all other available comparative data between the two cards from other reviewers. They're not hugely different, but the Ti definitely has the overall edge due to having more CUDA cores and more texture mapping units.

 

[JarredWaltonGPU]

One general comment about the most recent version of the GPU Hierarchy... it seems like the positions of the 1660 Ti and 1660 Super were (accidentally?) swapped.

It makes no sense for the Super to be ahead... that would contradict this site's very own review of the 1660 Super, as well as basically all other available comparative data between the two cards from other reviewers. They're not hugely different, but the Ti definitely has the overall edge due to having more CUDA cores and more texture mapping units.

The two cards were always very close. 1660 Ti has more cores, yes, but 1660 Super has faster GDDR6 memory and slightly higher GPU clocks. Some games favor the 1660 Ti by a few points, others favor the 1660 Super. I’m not actually sure why they swapped places but they’re effectively tied — like a 0.1 FPS difference once all the numbers are added together.

Actually, I think my weighting for the hierarchy used average of all FPS before, and in the last month I swapped to geometric mean in the spreadsheet. And that was enough for a 0.3 FPS change (because the 1660 Ti used to be slightly ahead). Of course, the GDDR6 memory on the Ti can probably hit similar overclocks and that would push it ahead, but I don’t think Nvidia is even making those anymore. It decided 1660 Super was preferable, possibly for yield and cost reasons.

 

[bohemond1099]

Here's your chance to influence our future graphics card testing. Right now, our list of nine games that we use for GPU benchmarks consists of the following (parenthetical comments are API we use, and whether a game has AMD or Nvidia branding):

Borderlands 3 (DX12 AMD)
The Division 2 (DX12 AMD)
Far Cry 5 (DX11 AMD)
Final Fantasy XIV (DX11 Nvidia)
Forza Horizon 4 (DX12)
Metro Exodus (DX12 Nvidia)
Red Dead Redemption 2 (Vulkan)
Shadow of the Tomb Raider (DX12 Nvidia)
Strange Brigade (Vulkan AMD)

A few quick notes:

1. I really want to ditch Far Cry 5, FFXIV, and Strange Brigade -- I feel all are older and less meaningful representations of modern games. FFXIV in particular has got to go IMO.
2. I haven't been testing with any ray tracing or DLSS modes in the two games that support those features (Metro and SotTR), but when AMD's Big Navi arrives I'll very likely be looking to include at least two ray tracing games.
3. The final list of games should include a mix of AMD and Nvidia promoted titles, different APIs if possible (DX12 + Vulkan for sure, and a major DX11 game would be nice as well). It should also include a mix of genres if possible -- so Forza is currently my "car racing" benchmark, and I still feel it's better than the F1 20xx games ... but if you have strong feelings and want to argue the case for a different racing game, I'm open.
4. Ideally, I want games that are more demanding (on the GPU), better looking, and more popular -- so if a game is more demanding but no one plays it, I'd prefer not to use it.

My short list of games I may include when they become available:

Assassin's Creed Valhalla (AC games are usually very demanding and decently popular, but potentially buggy)
Cyberpunk 2077 (duh)
Dirt 5 (could be a reasonable swap out for FH4)
Doom Eternal (it's out, I've tested it already, but it takes more effort to benchmark)
Dying Light 2 (maybe -- the last one was a pain to benchmark)
Microsoft Flight Simulator (if there's a reliable way to test that's not a complete PITA)
Rainbow Six Quarantine (another big maybe)
Star Wars Squadrons (maybe -- Jedi Fallen Order was merely okay, didn't warrant inclusion IMO)
Vampire: The Masquerade -- Bloodlines 2 (if it doesn't suck)
Watchdogs: Legion (possibly)

Lots of other games I probably won't include, unless they end up being 'better' for testing GPUs:
Death Stranding (already out, not sure how 'relevant' it is -- weird game, pretty repetitive IMO, but maybe?)
Diablo 4 (probably not coming for a while, may not be very demanding graphically)
Gods and Monsters (maybe?)
Horizon Zero Dawn (belated PS4 port)
Marvel's Avengers (likely to be too 'light' IMO)
Project Cars 3 (I didn't do much with the previous 2, but maybe third time's the charm)
Serious Sam 4 (will it be any good, and will it push GPUs at all?)
Wasteland 3 (Unity engine, not super demanding most likely)
Werewolf: The Apocalypse - Earthblood (probably not going to be demanding or popular?

I'd like to keep the final list to no more than 10 games total. Games with built-in benchmarks are also lovely, because that can remove a lot of the difficulty of repeated testing and variability -- but I get why some prefer 'real' benchmarking. Anyway, that's it. Sound off and I'll be looking at responses to see if I need to seriously rethink anything I've listed above.

I'd say what would be most useful in the current pit of ridiculous prices, would be a look at what can be done with older GPUs; i.e. 770 4GB/780/290/290X780Ti/970/980/1050Ti @1080P/1440P low-high depepending on the game; and inclusion of the most played games. Like CSGO and PUBG.

 

[aalkjsdflkj]

I'd love to see some comparison of VR performance. I've heard that NVIDIA is better with VR, but I don't know if that's true or not, and if so if AMD has addressed the difference with the 6xxx series. I don't have any particular game or test to suggest. Even if there's a generic test tool that can be used to generate scores for different cards that would be very helpful.

 

[AntonyLovric]

I like the CSGO and PUBG comment, but I realize the # of players is dropping and you might not get that much interest / views if the titles are not as popular. I do think you should include an RTX enhanced game that's GPU intensive.

 

[JoshTheDogBuildingNewPC]

Add Minecraft on the maximum possible settings, and Minecraft RTX
Minecraft RTX is the most demanding of the two, and could actually be surprisingly challenging for hardware if it is on maximum settings.

 

[rsouzadk]

I think that every game on that list that has some sort of RT on it should be added.

 

[alceryes]

@JarredWaltonGPU Could you write a quick line or two on your testing methodology? I believe I read it somewhere once. How many benches of each game, at each res and detail level? Also, are all tests done in a demo/benchmark mode where everything on screen is exactly the same with each run?
I believe a smattering of old games is actually good. Many, many gamers out there have a huge library of new AND old games. If anything, maybe remove some of the newest titles who's performance statistics would pretty much mirrors some other new title.

 

[JarredWaltonGPU]

@JarredWaltonGPU Could you write a quick line or two on your testing methodology? I believe I read it somewhere once. How many benches of each game, at each res and detail level? Also, are all tests done in a demo/benchmark mode where everything on screen is exactly the same with each run?
I believe a smattering of old games is actually good. Many, many gamers out there have a huge library of new AND old games. If anything, maybe remove some of the newest titles who's performance statistics would pretty much mirrors some other new title.

Games with a built-in benchmark are desirable, from a repeatability and time requirements perspective. Manually running through a sequence is possible, but it means I absolutely can't do anything else at the same time, which is a serious time sink if you want to test a bunch of GPUs. My standard approach is to run one pass at 1080p ultra and discard that result (it's the warm-up pass). Then run two tests each at 1080p, 1440p, and 4K ultra and use the best result from each. If there's more than a 1% difference between the two runs, I'll run another pass or two to figure out what the 'typical' performance looks like. Sometimes you get one fast or slow run, but after three or four passes either you discover a game simply has more variance between runs, or occasionally a run has a hiccup that inflates or reduces performance.

 

[alceryes]

Thanks.
I feel that any performance tests that have to be done manually should come with a huge, bold label stating that, "X game does not offer a built-in benchmark. These benchmarks were performed manually." Imagine, in a manual benchmark, a particular NPC actor comes into view 4 out of 10 runs. That could really skew results. Especially, when comparing multiple GPUs whose performance capabilities are all within 5% of one another.

Heck, maybe designers and publishers will take notice and start putting built-in benchmarks in all their AAA titles. Wouldn't that be nice?