Radeon HD 7990 Vs. GeForce GTX 690: The Crowd Picks A Winner

[cangelini]

The premise of the testing methodology was good but the execution was flawed. You began the test by using the first example that was NOT a true test and you admitted it. By using the multi-monitor test first you gave the test participants a much higher likely hood of detecting which system used which card. The testing to be accurate should have done the single monitor first to begin opinions on closer to a level playing field.

Remember that the participants who tested Eyefinity vs. Surround were different than the ones that tested single 30" vs. single 30".

 

[guggi4]

The premise of the testing methodology was good but the execution was flawed. You began the test by using the first example that was NOT a true test and you admitted it. By using the multi-monitor test first you gave the test participants a much higher likely hood of detecting which system used which card. The testing to be accurate should have done the single monitor first to begin opinions on closer to a level playing field.

Remember that the participants who tested Eyefinity vs. Surround were different than the ones that tested single 30" vs. single 30".

sure, but there is some talking between people. "Nah, the black is the AMD one"

 

[flexxar]

Why do sites even use Metro as a benchmark game? It's always that oddball game that seems to just get thrown in there. They always include popular games that everyone heard of like Skyrim, Crysis, Batman, Battlefield 3... and then there's Metro 2033. The only reason I know what Metro 2033 is is because it always gets benchmarked. I get the feeling that nvidia pays people to use it because it doesn't play well on amd. If it was a game that everyone played, amd would have fixed any issues. But it's not... and they don't care enough because of that.

 

[ddpruitt]

Where'd you see a room of six people? I'd estimate we had between 20 and 30 throughout the day, but the number of respondents we had (12--6 with frame pacing and 6 without) represented a full day of testing. I would have loved to have had 100, but that would have been a full work-week of gaming.

Basically it comes down to this--we could have run one game with A-B comparison testing and accommodated five times as many people (but they had issues with game sample size) or five games with the number of people we did (but had issues with participants). As it was, many people had to go home without getting to do the experiment because it simply consumes too much time to have two people test two systems in five games.

I suggest you brush up on statistical analysis. You have 6 people on a single game, even if it's a different 6 people on each game it makes the results statistically flawed (and I doubt it was 12 as chance would have said that at least one percentage out of 15 wouldn't have been divisible by 17). Your only getting 6 (or 12) samples each time, you can't average them together and declare a result that's bad math. To get something that's valid you'd need closer to 30 people per game. I agree that it's a start but to be relevant you would have to do large scale testing, sorry but that's the short end of it.

 

[cangelini]

The premise of the testing methodology was good but the execution was flawed. You began the test by using the first example that was NOT a true test and you admitted it. By using the multi-monitor test first you gave the test participants a much higher likely hood of detecting which system used which card. The testing to be accurate should have done the single monitor first to begin opinions on closer to a level playing field.

Remember that the participants who tested Eyefinity vs. Surround were different than the ones that tested single 30" vs. single 30".

sure, but there is some talking between people. "Nah, the black is the AMD one"

Isolating the participants certainly would have made the event more scientific. This is the first event of this nature we've done, and much was learned about how many people it takes to *run* (more than one), and we pulled away some very valuable advise from the folks in attendance. I don't claim that this is perfect, but we did come from it with meaningful information, presented here.

 

[Marcopolo123]

well

 

[guggi4]

The premise of the testing methodology was good but the execution was flawed. You began the test by using the first example that was NOT a true test and you admitted it. By using the multi-monitor test first you gave the test participants a much higher likely hood of detecting which system used which card. The testing to be accurate should have done the single monitor first to begin opinions on closer to a level playing field.

Remember that the participants who tested Eyefinity vs. Surround were different than the ones that tested single 30" vs. single 30".

sure, but there is some talking between people. "Nah, the black is the AMD one"

Isolating the participants certainly would have made the event more scientific. This is the first event of this nature we've done, and much was learned about how many people it takes to *run* (more than one), and we pulled away some very valuable advise from the folks in attendance. I don't claim that this is perfect, but we did come from it with meaningful information, presented here.

Sure, that wasnt really critic, i just wanted to say that it was most likely quite easy to identify the AMD maschine due to to eyefinity/surround testing first, maybe it would be better to test first with framepacing and then collect opinions with multimonitors

 

[Marcopolo123]

well

 

[JMcEntegart]

Looking good, Chris, and an excellent event! Keep up the good work!

 

[geok1ng]

cangelini, like i said, i respect the efforts of the site, but Metro LL has issues outside of VGA performance comparisons that do not allow it to be used in an article like this one. A VGA maker sponsoring one game title is one thing. Another is that game title showing odd and artificial behaviors everytime one uses the competing brand, no matter how powerful the tested card is. Dx 10.1 and assassin's creed was another precedent of this behavior, but at the time at least top AMD cards could reach more than 20fps min, something that Metro LL does not allow even if you OC the cards to 1300Mhz.. NVIDIA went the extra mile to create FCAT so we could measure how much better than CF SLI is. But now that CF is starting to behave better in FCAT we are going back to the old days of " it feels smoother somehow" ? Do not get me wrong, the whole article just restates what every other review showed before: AMD architeture is dominant at 1600p but multicards and multimonitor setups are better served by NVIDIA hardware this generation.

 

[jackbling]

Forming opinions based on brand loyalty and selective acceptance of empirical data should not be present in these comments; sure the test isn't a hermetical, double-blind, infallible study, but it is close enough that, when combined with benchmark data, the results are corroborated.

 

[Bill Reinhardt]

One thing is clear: Beer always wins.

 

[Squall321]

Still an AMD-boy, sorry folks.

 

[Ashley Henderson]

You were supposed to be testing the difference now that frame pacing has been introduced.

You knew that there were issues with the amd card in metro...

why would you include metro in the testing as that issue is not related to frame pacing.

Your results are skewed... really poor thought

 

[JPForums]

As some other said, there seems to be a slight bias here. Though I would suggest that the bias doesn't really apply to the multi-monitor setup. The way you describe the testing, you started them off with the multi-monitor setup and then moved them to the single monitor setup. Put another way, you started them off with a legitimately flawed setup (no frame pacing) and then asked them to judge the same setup without the faulty condition, but also without anyway of knowing the condition was fixed. Human nature is such that when you find fault with something, you tend to expect the fault, even when it is no longer there. The participants that identified fault with the system in the multi-monitor setup went in to the single monitor comparison preconditioned to expect the system to be faulty.

That all said, the bias couldn't have been terribly significant as the Crysis results show a move from no contest to a win in the other direction. So I wouldn't expect the results to change enough to displace the winner. Still, if you do this again (I hope you do), such a bias could be avoided by waiting to do the flawed (no frame pacing) comparison until after the apples to apples comparison is completed. This way there are no preconceptions skewing the apples to apples comparison. On the up side, my GTX670FTW SLI setup performs near identically to the GTX690 that won in the article.

Update: I just saw in another comment that the participants judging the multi-monitor setups were different than those judging the single monitor setups. Guess I was right when I said the bias couldn't have been terribly significant. I suppose there could be a slight bias from those watching the others game before testing themselves, but that's really not all that likely. Good article. Hope to see another maybe a year from now when AMD should have everything squared away.

 

[JPForums]

IMO, a double blind study with at least 10-15 participants would be more definitive, but testing again with a larger group is only going to confirm what don has already found. time and effort would be better suited to find what is going on rather than carrying out subsequent tests or bashing a legit qualitative study.

I'm fairly confident that what Don saw was related to the games simulation time. I believe the technical results are valid. Even in the games where motion appears disjointed, when the FCAT results suggest smooth frame rates, that is what the monitor is getting. Monitors don't vary update time and the FCAT method will show dropped, runt, and partial frames.

What FCAT won't show is the fact that the game simulation is completely unaware of frame pacing. Take this scenario: A frame is rendered at time T1 ms and gets delayed by D ms for frame pacing. The next frame is rendered at time and isn't delayed as the previous frame wouldn't have been delayed if this one needed to be. The placement of objects in the second frame assumes that T2 - T1 time has passed and calculates movements accordingly. However, by the time they reach the screen T2 - (T1+D) time has passed. In this frame, the simulation has moved the objects D ms longer than it should have at the monitor. The very next (or previous) frame can give the opposite situation where the simulation moved the objects D ms shorter than it should have. If D is small compared to T2 - T1, then the inaccuracy will go unnoticed. If it is not small, however, it will show up as disjointed motion. This effect should be more noticeable in faster paced games with lower frame rates. Most likely, the method that nVidia is using for frame pacing causes less deviation from the game simulation time than ATi's method.

 

[flexxar]

It's not a bad article. It's just that the whole timing of it didn't sit right with me. The 690 had the performance crown for many months. Amd releases new drivers that take back the performance crown. The article yesterday basically declares the 7990 the new performance winner. The article today basically sounds like a rebuttal saying, "Forget that the 7990 is the new performance winner, amd still sucks." It's exactly what I would expect an Nvidia fan to say the day after the 7990 is declared the winner. If you can't win with numbers, just win by "feel", right? That's why Metro was thrown into the benchmarks? The whole ordeal rubbed me as biased through and through.

 

[paul_durham]

In one of my favorite movies, Happy Accidents, the director makes constant fun of contemporary Westerners by depicting them as spoiled brats.
- How was the exhibition ?
- Oh, disgusting, absolutely awful ! Disgusting !

That's what comes to my mind when reading most of these comments

For my part, I'm very grateful to Chris for organizing such original events, writing content that I surely don't find anywhere else, and for being constantly eager to share his enthusiasm with his/her readers.

 

[cmi86]

Good article and i"m not bitching about the results but why did you do away with Tomb Raider were the 7990 crushes the 690.. and then keep metro where an obvious ISSUE is keeping the 7990 from being playable at all ? That's just gimme points for the 690. It's actions like these that make me read your reviews with a grain of salt.

 

[cmi86]

Dons article yesterday showed that with the new drivers the 7990 consistently put up higher max/average FPS than the 690 and also effectively eliminated the majority of issues experienced with crossfire. By the numbers it was the winner,. Now but a only day later this event appears and what do you know "The nvidia feels smoother..." While you have people playing on a jagged eyfinity setup with no frame pacing and then switching over to titles that knowingly don't work on AMD hardware to formulate an opinion about which they prefer... I miss the old toms, the one where I didn't have to read between the lines.

 

[cangelini]

Where'd you see a room of six people? I'd estimate we had between 20 and 30 throughout the day, but the number of respondents we had (12--6 with frame pacing and 6 without) represented a full day of testing. I would have loved to have had 100, but that would have been a full work-week of gaming.

Basically it comes down to this--we could have run one game with A-B comparison testing and accommodated five times as many people (but they had issues with game sample size) or five games with the number of people we did (but had issues with participants). As it was, many people had to go home without getting to do the experiment because it simply consumes too much time to have two people test two systems in five games.

I suggest you brush up on statistical analysis. You have 6 people on a single game, even if it's a different 6 people on each game it makes the results statistically flawed (and I doubt it was 12 as chance would have said that at least one percentage out of 15 wouldn't have been divisible by 17). Your only getting 6 (or 12) samples each time, you can't average them together and declare a result that's bad math. To get something that's valid you'd need closer to 30 people per game. I agree that it's a start but to be relevant you would have to do large scale testing, sorry but that's the short end of it.

Good article and i"m not bitching about the results but why did you do away with Tomb Raider were the 7990 crushes the 690.. and then keep metro where an obvious ISSUE is keeping the 7990 from being playable at all ? That's just gimme points for the 690. It's actions like these that make me read your reviews with a grain of salt.

We started using Tomb Raider and *not* Battlefield, due to its age. Unfortunately, Tomb Raider has a bad habit of overwriting save points too often, so it was impossible to set participants up at the same spot. We went through three of the exact same save points, overwriting them all, before we had to abandon it because we couldn't replicate the same sequence for everyone.

 

[littleleo]

So in this blind test most of the testers choose Nvidia. It seems the consensus is Nvidia provides a better gaming experience out of the box. After months AMD may offer a better driver to improve performance but NIvidia drivers seem pretty complete from the the time you purchase the card. This mirrors my experience and those of the majority of my customers. AMD has to keep dropping its prices to compete otherwise it's just not in the game.

 

[cangelini]

Dons article yesterday showed that with the new drivers the 7990 consistently put up higher max/average FPS than the 690 and also effectively eliminated the majority of issues experienced with crossfire. By the numbers it was the winner,. Now but a only day later this event appears and what do you know "The nvidia feels smoother..." While you have people playing on a jagged eyfinity setup with no frame pacing and then switching over to titles that knowingly don't work on AMD hardware to formulate an opinion about which they prefer... I miss the old toms, the one where I didn't have to read between the lines.

Sounds like you just want a story that concludes AMD won
In this piece, we found that of the admittedly limited audience we could have game testing in one full day, AMD's position in blind testing improved almost to the point of matching Nvidia's 690 if you throw out Metro completely. This is stated clearly.
If you read Don's story yesterday, then you know that Metro's benchmark runs *fine* on both cards. It's only when we started playing the actual game did we learn that the 7990 has an issue. We presented it to AMD before either story went live, and received no feedback.

Here's the deal: between Don's story and mine, you have FCAT data, video data, and blind testing data from people who have nothing to do with AMD or Nvidia. You can ignore our analysis altogether if you want and still have more information in front of you than if these stories didn't go up. You can twist the intention of the game choice however you see fit, but the original plan was Tomb Raider, BioShock, Crysis 3, Grid 2, and Metro. That's three AMD Evolved games, one Intel game, and one Nvidia game. AMD has an issue in Metro--fact. There is no reading between the lines. Everything we meant to say was said.

If this story were my own personal opinion based on the hours that I spend testing and playing with graphics cards, I'd recommend buying two GTX 770s and be done with it. If a friend says, "But Chris, I really want a dual-GPU card, then I tell them that the 690 is a more elegantly-built product and that AMD's solution to its issues is still a beta driver." At $1000, it's 690 all the way. I feel so strongly about the workmanship that went into the actual construction of 690 and Titan that I petitioned Nvidia for access to the guys who came up with the ID for a story that I hope will be coming soon. Given the recent price drops, I imagine there are some folks willing to snag AMD's eight-game bundle and a fast dual-GPU card for $700. Right now, the way things stand, I would not be one. Happy to debate the ins and outs, but I see this as a difference of opinion.

 

[flexxar]

Seriously... Lets all just be real for a second. Most of these tests were won or lost by a single vote. Yes, 1 person's opinion gets to decide the winner of which card provides the better experience. If you ever studied statistics, that is well within the margin of error even if this test was done perfectly. This article is flame war bait. Shouldn't have been published. At the very least, it should come with a huge disclaimer at the beginning stating that it is not a proper test.

 

[cmi86]

Dons article yesterday showed that with the new drivers the 7990 consistently put up higher max/average FPS than the 690 and also effectively eliminated the majority of issues experienced with crossfire. By the numbers it was the winner,. Now but a only day later this event appears and what do you know "The nvidia feels smoother..." While you have people playing on a jagged eyfinity setup with no frame pacing and then switching over to titles that knowingly don't work on AMD hardware to formulate an opinion about which they prefer... I miss the old toms, the one where I didn't have to read between the lines.

Sounds like you just want a story that concludes AMD won
In this piece, we found that of the admittedly limited audience we could have game testing in one full day, AMD's position in blind testing improved almost to the point of matching Nvidia's 690 if you throw out Metro completely. This is stated clearly.
If you read Don's story yesterday, then you know that Metro's benchmark runs *fine* on both cards. It's only when we started playing the actual game did we learn that the 7990 has an issue. We presented it to AMD before either story went live, and received no feedback.

Here's the deal: between Don's story and mine, you have FCAT data, video data, and blind testing data from people who have nothing to do with AMD or Nvidia. You can ignore our analysis altogether if you want and still have more information in front of you than if these stories didn't go up. You can twist the intention of the game choice however you see fit, but the original plan was Tomb Raider, BioShock, Crysis 3, Grid 2, and Metro. That's three AMD Evolved games, one Intel game, and one Nvidia game. AMD has an issue in Metro--fact. There is no reading between the lines. Everything we meant to say was said.

If this story were my own personal opinion based on the hours that I spend testing and playing with graphics cards, I'd recommend buying two GTX 770s and be done with it. If a friend says, "But Chris, I really want a dual-GPU card, then I tell them that the 690 is a more elegantly-built product and that AMD's solution to its issues is still a beta driver." At $1000, it's 690 all the way. I feel so strongly about the workmanship that went into the actual construction of 690 and Titan that I petitioned Nvidia for access to the guys who came up with the ID for a story that I hope will be coming soon. Given the recent price drops, I imagine there are some folks willing to snag AMD's eight-game bundle and a fast dual-GPU card for $700. Right now, the way things stand, I would not be one. Happy to debate the ins and outs, but I see this as a difference of opinion.

To be completely honest I don't really care which on wins. I'm a fan of both sides of the fence, the 760 is my favorite card right now and I really wish I knew it was going to drop a month after I bought my then $250 King 7870 LE.. That said there is no twisting of the game selection, it is what it is. And what it is is that you went from originally planing to use a game where the 7990 worked to one where you full well knew it didn't. If that wasn't enough you actually factored these technical forfeit points into the conclusion of the review ?? If you honestly didn't know that LL would perform that way the results of that test should have simply been omitted and that would have made a lot of people happy because it's the right thing to do instead of appear to favor nvidia by giving the card that lost in dons review free gimmie points. It's all about formulating a public opinion, people will keep a bad taste in their mouth because of the choppy eyfinity set-up they sat down at which just so happens to be the same machine one of the games didn't even work on.. You see where I am going with this ? When it all boils down I am still happy to see positive progress being made with crossfire.