Question What GPU Benchmarks Should I Add/Drop for Future Reviews/Hierarchy?

Page 2 - Seeking answers? Join the Tom's Hardware community: where nearly two million members share solutions and discuss the latest tech.
Here's your chance to influence our future graphics card testing. Right now, our list of nine games that we use for GPU benchmarks consists of the following (parenthetical comments are API we use, and whether a game has AMD or Nvidia branding):

Borderlands 3 (DX12 AMD)
The Division 2 (DX12 AMD)
Far Cry 5 (DX11 AMD)
Final Fantasy XIV (DX11 Nvidia)
Forza Horizon 4 (DX12)
Metro Exodus (DX12 Nvidia)
Red Dead Redemption 2 (Vulkan)
Shadow of the Tomb Raider (DX12 Nvidia)
Strange Brigade (Vulkan AMD)

A few quick notes:
  1. I really want to ditch Far Cry 5, FFXIV, and Strange Brigade -- I feel all are older and less meaningful representations of modern games. FFXIV in particular has got to go IMO.
  2. I haven't been testing with any ray tracing or DLSS modes in the two games that support those features (Metro and SotTR), but when AMD's Big Navi arrives I'll very likely be looking to include at least two ray tracing games.
  3. The final list of games should include a mix of AMD and Nvidia promoted titles, different APIs if possible (DX12 + Vulkan for sure, and a major DX11 game would be nice as well). It should also include a mix of genres if possible -- so Forza is currently my "car racing" benchmark, and I still feel it's better than the F1 20xx games ... but if you have strong feelings and want to argue the case for a different racing game, I'm open.
  4. Ideally, I want games that are more demanding (on the GPU), better looking, and more popular -- so if a game is more demanding but no one plays it, I'd prefer not to use it.

My short list of games I may include when they become available:

Assassin's Creed Valhalla (AC games are usually very demanding and decently popular, but potentially buggy)
Cyberpunk 2077 (duh)
Dirt 5 (could be a reasonable swap out for FH4)
Doom Eternal (it's out, I've tested it already, but it takes more effort to benchmark)
Dying Light 2 (maybe -- the last one was a pain to benchmark)
Microsoft Flight Simulator (if there's a reliable way to test that's not a complete PITA)
Rainbow Six Quarantine (another big maybe)
Star Wars Squadrons (maybe -- Jedi Fallen Order was merely okay, didn't warrant inclusion IMO)
Vampire: The Masquerade -- Bloodlines 2 (if it doesn't suck)
Watchdogs: Legion (possibly)

Lots of other games I probably won't include, unless they end up being 'better' for testing GPUs:
Death Stranding (already out, not sure how 'relevant' it is -- weird game, pretty repetitive IMO, but maybe?)
Diablo 4 (probably not coming for a while, may not be very demanding graphically)
Gods and Monsters (maybe?)
Horizon Zero Dawn (belated PS4 port)
Marvel's Avengers (likely to be too 'light' IMO)
Project Cars 3 (I didn't do much with the previous 2, but maybe third time's the charm)
Serious Sam 4 (will it be any good, and will it push GPUs at all?)
Wasteland 3 (Unity engine, not super demanding most likely)
Werewolf: The Apocalypse - Earthblood (probably not going to be demanding or popular?

I'd like to keep the final list to no more than 10 games total. Games with built-in benchmarks are also lovely, because that can remove a lot of the difficulty of repeated testing and variability -- but I get why some prefer 'real' benchmarking. Anyway, that's it. Sound off and I'll be looking at responses to see if I need to seriously rethink anything I've listed above.
 
  • Like
Reactions: AnnaMichel
Thanks.
I feel that any performance tests that have to be done manually should come with a huge, bold label stating that, "X game does not offer a built-in benchmark. These benchmarks were performed manually." Imagine, in a manual benchmark, a particular NPC actor comes into view 4 out of 10 runs. That could really skew results. Especially, when comparing multiple GPUs whose performance capabilities are all within 5% of one another.

Heck, maybe designers and publishers will take notice and start putting built-in benchmarks in all their AAA titles. Wouldn't that be nice?
You'd be surprised how little difference there is between running the same sequence manually vs automated built-in tests. Cyberpunk 2077 for example, I walk along a 60 second path in Night City, and the number and variety of NPCs and cars seems relatively random. However, I ran the same setup on the same sequence 10 times (to check for variability between runs) and there was only a 2% spread. I've seen 'bad' built-in benchmarks (eg, Assassin's Creed Odyssey) where the difference between runs can be almost 10%, just because clouds and weather were randomized.

Fundamentally, anyone really getting hung up on a 5% difference has already lost sight of the purpose of benchmarks. If one card scores 100 fps and another scores 105 fps, I would say something like, "Nvidia is a bit faster than the AMD card here, but it's not a huge difference -- one or two settings tweaks would close the gap." It's really only 10% and larger differences that become truly meaningful in terms of the user experience.
 
Mar 9, 2021
4
0
10
Warframe on open world nodes can become very demanding for CPU and GPU. Cambrian drift has become a place of horror for players with lower end computer builds.

And is there a reason you are not using GTA5? Has a built in benchmark and with graphics turned up can give a 5 year old mid tier computer serious problems even at 1080p
 
Warframe on open world nodes can become very demanding for CPU and GPU. Cambrian drift has become a place of horror for players with lower end computer builds.

And is there a reason you are not using GTA5? Has a built in benchmark and with graphics turned up can give a 5 year old mid tier computer serious problems even at 1080p
GTA5 is now six years old, so I think that's long enough in the past that it's worth not including. In fact, I dropped it from my testing at PC Gamer probably around 2018. It's been superseded by Rockstar's Red Dead Redemption 2 as well, which is newer and more demanding. What it really comes down to is time, though. Testing every popular game on every new GPU, and then retesting every few months with new drivers and updates, means anything more than about a dozen games ends up as a massive slog, and fewer than 10 games is better.
 
And is there a reason you are not using GTA5? Has a built in benchmark and with graphics turned up can give a 5 year old mid tier computer serious problems even at 1080p
That sounds like it's more of a problem with the software, which can mask the potential of the hardware. It's like saying the original Crysis should still be tested because it "crushes" high-end PCs (as in, no PC has yet to reach a sustained triple digits FPS in that game). Crysis however, was designed in a time when it was thought that 10GHz single core CPUs were on the horizon and so it doesn't work well with our modern day multi-core ones, among other issues.
 
  • Like
Reactions: JarredWaltonGPU
@JarredWaltonGPU
How about picking a couple benchmarks (3D Mark, Unigine Superposition, etc) or a couple super-popular games and adding a column or two to the GPU hierarchy to show the performance of that card at a set quality level (Ultra, for instance) in those benchmarks/games.
I understand the reasoning behind just using a relative score to rank the cards, but it's a score in a vacuum. It doesn't give the reader any way of knowing how their GPU will perform in the games they are looking to use it for.

I also think the relative score unnecessarily pushes people into the mindset that their card isn't a good performer. We were raised to think that getting a 60% or lower, in anything, is a failure, but the RTX 2070 Super is actually a pretty good card that will work fine for 99.99% of the games out there at reasonable settings. Of course, IT people and heavy gamers/tweakers know the truth but adding a couple columns to explain the actual performance of said card would go a long way towards making the hierarchy chart more understandable to the masses.
 
@JarredWaltonGPU
How about picking a couple benchmarks (3D Mark, Unigine Superposition, etc) or a couple super-popular games and adding a column or two to the GPU hierarchy to show the performance of that card at a set quality level (Ultra, for instance) in those benchmarks/games.
I understand the reasoning behind just using a relative score to rank the cards, but it's a score in a vacuum. It doesn't give the reader any way of knowing how their GPU will perform in the games they are looking to use it for.

I also think the relative score unnecessarily pushes people into the mindset that their card isn't a good performer. We were raised to think that getting a 60% or lower, in anything, is a failure, but the RTX 2070 Super is actually a pretty good card that will work fine for 99.99% of the games out there at reasonable settings. Of course, IT people and heavy gamers/tweakers know the truth but adding a couple columns to explain the actual performance of said card would go a long way towards making the hierarchy chart more understandable to the masses.
The hierarchy includes charts for all six tested resolutions at the bottom, and the Best GPUs article has the full set of 60 charts, for anyone that wants to look at the micro-view rather than the macro-view.
 
  • Like
Reactions: alceryes