Editor's Corner: Getting Benchmarks Right

Status
Not open for further replies.

Hamsterabed

Distinguished
Jan 29, 2009
17
0
18,510
How very odd, when i saw the benches i immediately thought there was a problem. Glad you guys made an article to explain and backup you numbers and i hope we get some answers. don't have another driver fail Nvidia...
 

rdawise

Distinguished
Apr 18, 2008
225
0
18,680
Thank you Chris for this follow-up article..now where is kknd to argue....

I am sorry but we all know that at lower resolutions the Core i7 will beat the P2, but as the article states, but real world the PII is hitting the high notes. Could this be a driver screw up from Nvidia...probably since you're elimnating everything else. Are there any other x-factors out there...oh yes plenty more. However I think people will get the wrong impression if they read this and think the PII is "more powerful" than the Core i7. Some one who reads this should come away thinking that the PII will give you almost as great gaming as some of the Core i7s can for less money. (Time for a price cut intel).

I do a question what if you tried using memory with different timings. I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons. Either way it gives us something to look forward to in the CPU world. Good follow-up.
 

rdawise

Distinguished
Apr 18, 2008
225
0
18,680
Thank you Chris for this follow-up article..now where is kknd to argue....

I am sorry but we all know that at lower resolutions the Core i7 will beat the P2, but as the article states, but real world the PII is hitting the high notes. Could this be a driver screw up from Nvidia...probably since you're elimnating everything else. Are there any other x-factors out there...oh yes plenty more. However I think people will get the wrong impression if they read this and think the PII is "more powerful" than the Core i7. Some one who reads this should come away thinking that the PII will give you almost as great gaming as some of the Core i7s can for less money. (Time for a price cut intel).

I do a question what if you tried using memory with different timings. I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons. Either way it gives us something to look forward to in the CPU world. Good follow-up.
 

sohei

Distinguished
Jan 13, 2009
93
0
18,630
"I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons"

wow 7-7-7-20? this is the performance...indeed
P2 works with ddr2 great and you wary about timings
 
G

Guest

Guest
great article!!
just a thought: what about previous generation of nvidia cards? could be this is a GTX 260/285/280/... problem. maybe you could try with one of 9xxx series.
 

StupidRabbit

Distinguished
Jan 27, 2009
60
0
18,630
awesome article.. only two pages long but it changes the way i look at the previous benchmarks. good to see you focus not only on the hardware itself but also on the benchmarks with a real sense of objectivity.. its what makes this site great.
 

cobra420

Distinguished
Jul 8, 2008
27
0
18,530
so it looks like a gpu issue . why not try a gtx 295 ? or is that why you set the video so low ? now you found the issue theirs no need to try a different card ? ati sure did a good job on there 4870 series . nice job toms
 
G

Guest

Guest
Maybe Farcry optimized it more on ATI, maybe Intel is throwing sticks at the wheels of nVidia at the hardware level, maybe, maybe ... :S
Why is Intel supporting multi-ATI config, but not multi-nVidia? Why doesn't Intel let nVidia use its Atom freely? Why, oh why?
There are so many factors. I think if you replace Farcry with a synthetic test, there will be less unknowns. Just maybe :)
 

jcknouse

Distinguished
Oct 23, 2008
447
0
18,780
Fantastic work, Chris. Simply awesome. I really, really, really enjoyed that analysis.

Something I need to ask tho:

Is the GTX280 a dual processor VDA, like the Radeon 4870X2? If not, wouldn't you expect significant gains of 2 GPUs over 1?

Also...

Wouldn't you expect to see even a small decline in performance (possibly miniscule/negligible...but still present) simply because running 2 individual VDAs in 2 PCI-E x16 2.0 slots requires work to be sorted between two physical devices (handling going over the SLi interface), whereas using the Radeon 4870x2 work is sent to and split on-card?

I'd really be interested in seeing a performance difference between 2 4870s and 1 4870x2 with the same catalyst version, and latest firmware.

Also, I am quite shocked at the nVidia suffering. I've been an nVidia customer for years. I am going to have to look at going ATI Crossfire if I build a new AM3 gaming platform later this year.

This is one of the reasons I value Tom's so much...articles like this.

Thank you again, Chris. This was invaluable to me both technically as well as a consumer looking for the best bang for my buck, even tho my budget isn't limited.
 

jeffunit

Distinguished
May 19, 2008
117
0
18,680
Why are you running games which stress graphics for a cpu review?
Wny not run *cpu benchmarks*, which measure cpu performance for a cpu
review?
 
G

Guest

Guest
I would love to see a full testing article of AMD cpu's with ATI and NVidia cards VS Intel CPU's with ATI and NVidia cards. Maybe the architecture of the ATI cards works better with Intel chips and Nvidia works better with AMD chips. Wouldn't that be ironic since ATI is AMD.
 

squatchman

Distinguished
Oct 31, 2008
211
0
18,680
Much better, it was probably a ton of work, but you see that people complain less with comprehensive results.

To jcknouse: Yea, the 4870x2 is two GPUs on one board and it should be beating the GTX280. To jeffunit: Check the original article for the non-synthetic CPU benchmarks, or any other article for that matter.
 

bf2gameplaya

Distinguished
Mar 19, 2008
262
0
18,780
Finally some quantitative analysis from Tom's, with reasonable methodology and an interesting subject. Well done, you have educated me!

Yet, I see that Far Cry 2 was the only title tested and I have no way of knowing if any of these conclusions carry over to any other title.

But now I have the right questions to ask, Thanks Tom's!
 

jameskangster

Distinguished
Aug 29, 2008
17
0
18,510
Chris, thank you for the editorial. I really appreciate the fact that you guys try your best to respond and answer our nit-picking comments and questions. Now based on what I have observed from your articles, I have some suggestions that might help (or might not at all). What if the bottleneck is related to the motherboard and its memory timing setup?

I did a quick comparison of your past setups for benchmarks (From this article, 2009-02-09 AMD AM3 article, 2009-01-07 Phenom II review article, 2008-11-03 i7 review article).

Basically, you used the same hardware for i7 920 in articles 2009-01-07 and 2008-11-03
Motherboard: Intel DX58SO Revision 403
Memory: A-DATA DDR3-1600 2X2GB set to DDR3-1333 CL 7-7-7-20
Video: MSI N280GTX-T2D1G-OC

Also just as an external reference I used Anandtech's setup from this article
http://www.anandtech.com/cpuchipsets/showdoc.aspx?i=3512&p=3. For those of you who hate Anandtech I apologize, but this article had the most comparable hardware setup:

Motherboard: Intel DX58SO
Memory: Qimonda DDR3-1066 4x1GB (7-7-7-20)
Video card: eVGA GeForce GTX 280

In these articles Intel's i7 920 2.66 GHz performance seemed to be dare I say better than Phenom II (although in one article Phenom II did not exist, but just looking at raw numbers) specifically related to gaming benchmarks. The interesting point here is that they all used Intel DX58SO motherboard, using 7-7-7-20 timing for the memory. The number of modules varied; however, it didn't seem to make a huge difference from what I have read so far.

In the 2009-02-09 article and its subsequent editorial article Tom's Hardware used the following setup for i7.
Motherboard: Asus Rampage II Extreme (X58/ICH10) LGA 1366
Memory: Corsair Dominator DDR3-1600 8-8-8-24 @1.65V 3x2GB (caveat here Tom's hardware did try overcloking in the editorial article so that does vary)
Video card: Nvidia GeForce GTX 280 1 GB

Overall the video card chip model remained the same, the driver revisions were different but not siginifically; however Tom's used a different board and its memory setup was drastically different from its previous setups.

It would be interesting to see the differences in performance comparing these setups (some of these setups might not be possible due to hardware/BIOS limitations, I didn't have time to look into that part):
1. Rampage mobo with 8-8-8-24 memory timing vs Intel DX58SO with 8-8-8-24 memory timing
2. Rampage mobo with 7-7-7-20 memory timing vs Intel DX58S0 with 7-7-7-20 memory timing
3. Intel DX58SO with 8-8-8-24 memory timing vs Intel DX58SO with 7-7-7-20 memory timing
4. Rampage mobo with 8-8-8-24 memory timing vs Rampage mobo with 7-7-7-20 memory timing

Obviously I would use i7 920, the same video card and driver for above setups.

I highly doubt that the memory timing would cause such a performance difference. My bet is on the motherboard.

Also it would answer another question, or maybe obsfucate even more. Was it truly Nvidia's video card's fault that i7's potential was not translated into raw performance? or was it due to the motherboard, or due to memory setup? or both? or all three?

I'm not expecting another editorial article for this, but it would be good to see get this straightened out. I'm really hesitant to place the blame on Nvidia for this yet.
 

Ho0d1um

Distinguished
Jan 9, 2009
34
0
18,530
[citation][nom]Madis Kalme[/nom]There are so many factors. I think if you replace Farcry with a synthetic test, there will be less unknowns. Just maybe[/citation]
This article is testing real world results and not just number crunching
 

kknd1967

Distinguished
Mar 15, 2006
11
0
18,510
This is simply a great analysis. thumb up for Chris' good work. I too originally suspected NV card + i7 behavior or X58 chipset in gaming. 2 other websites have i7+NV260 leading in CPU bounded low res test, but falling in GPU bounded high rest test significantly, which just does not make sense for a typical system behavior (usually one would expect leader still leads but by smaller margin in high res).

What is interesting is now it seems best to use Intel i7 with AMD video card for gaming. So who is the loser? Nvidia...
 

jonyb222

Distinguished
Jan 20, 2008
334
0
18,780
The data suggests that, using an AMD Radeon-based graphics card, you'll likely see the scaling that many other sites have presented, with Intel's Core i7 besting the Phenom II right up to 2560x1600 (refer to the first chart on this page for proof there).

Nvidia's card cannot translate the Core i7's microarchitecture into the same performance advantage, giving AMD's Phenom II-series chips the advantage seen in the AM3 story and in the two pages you've just read.

So, if I understood correctly, currently Nvidia cards/drivers are slowing down intel processors while AMD one can go full speed (which is why Phenom beats I7 in that case). And ATI cards let's both processors go fulll speed (which is why I7 beats Phenom in that case)

Bummer for Intel/Nvidia, Horray for AMD/ATI?
 

brianbed

Distinguished
Nov 1, 2008
6
0
18,510
I have heard this story somewhere before... ahh yes, the K6-2/Pentium 2. It turned out that 3DFX products were the way to go for the K6-2, but NVidia worked better with the P2. So, if you have an i7 buy an AMD GPU, and oddly enough if you have and AMD Phenom II don't but an AMD GPU =).

Needless to say, processor (and GPU) architectures will always yield differences in benchmark results (sometimes slight, some times major) from one piece of software to the next. It is all in how the code is written and compiled.

Code by better design is designed to maximize branch prediction and cache hits. This is how the i7 changed Intel's playing field. Changes and cache sizes and latencies greatly effect how poorly written code performs (Yes, most games are poorly written, dev houses would rather spend money on CG). Compare Oracle (well-written), to Half Life 2 (decently written), to Crysis (UGH!). Suddenly architectures are playing ping-pong.

The point is simple. Tom's was not at completely at fault here. None of the benchmarks were flawed. With rare exception, no reviewer site provided a "complete" comparison. This follow-up, however, is exemplary. Good job, guys, and I hope to see more comprehensive comparisons like this one in the future.
 

jp182

Distinguished
Apr 11, 2003
105
0
18,680
Thanks for taking the time to run all of those tests.

They didn't use another game as the original purpose was to prove that they conducted their tests properly. Now, if they want to see what the problem is with nvidia's cards then that would be a whole other set of tests and it'll probably take a week for them to get all of their data together.
 

kknd1967

Distinguished
Mar 15, 2006
11
0
18,510
i am here and thanks for posting two comments to find me :D
but I wonder why you read the article conclusion as "we all know that at lower resolutions the Core i7 will beat the P2, but real world the PII is hitting the high notes". My reading says NV GTX280 screw up somewhere, likely in driver.
But I fully agree with you on "the PII will give you almost as great gaming as some of the Core i7s can for less money." as we know gaming is still mostly a GPU show at this point. If all the system are behaving normally, a 3DMark GPU score will be close for all systems. That is what I am saying from the begining.

Much appreciate Chris' effort. My apologies for my initial comments to Chris on the previous article, after seeing a not so ususal 3DMark. This article carries a lot of good information although the final words are not conclusive yet. Would be nice to do a sanity check on 3DMark or another game.


[citation][nom]rdawise[/nom]Thank you Chris for this follow-up article..now where is kknd to argue....I am sorry but we all know that at lower resolutions the Core i7 will beat the P2, but as the article states, but real world the PII is hitting the high notes. Could this be a driver screw up from Nvidia...probably since you're elimnating everything else. Are there any other x-factors out there...oh yes plenty more. However I think people will get the wrong impression if they read this and think the PII is "more powerful" than the Core i7. Some one who reads this should come away thinking that the PII will give you almost as great gaming as some of the Core i7s can for less money. (Time for a price cut intel).I do a question what if you tried using memory with different timings. I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons. Either way it gives us something to look forward to in the CPU world. Good follow-up.[/citation]
 
Status
Not open for further replies.