Editor's Corner: Getting Benchmarks Right

Page 2 - Seeking answers? Join the Tom's Hardware community: where nearly two million members share solutions and discuss the latest tech.
Status
Not open for further replies.

jameskangster

Distinguished
Aug 29, 2008
17
0
18,510
I agree with you on the Far Cry 2 test result; however, their Crysis performance did show that i7 920 did perform significantly better than AMD counterparts (http://www.bit-tech.net/hardware/cpus/2009/02/11/amd-phenom-820-810-720-710-am3-cpus/5). Even though on Tom's Hardware Core i7 920 trailed Phenom II X4 940 BE. So on that one I would say that was more related to a driver optimization issue. It's interesting to note that they used Intel X58 SO2624 motherboard for their Core i7's where as Tom's Hardware used Asus Rampage II Extreme motherboard.

[citation][nom]Ryun[/nom]Great followup thanks. If there's any more doubt Bit-Tech just posted their review of the AM3 CPUs which also experienced the same anomalies as Tom's: http://www.bit-tech.net/hardware/c [...] am3-cpus/6For the record they also used a GTX 280.[/citation]
 

beeyang78

Distinguished
Oct 10, 2008
23
0
18,510
I love this site! The people here are just great and very entertaining as well. If people were reading Intel's notes at the launch of Core I7, they did say for you not to expect too much from gaming so I'm not in one bit surprised at the outcome of everything. But I guess we have to justify why we purchased such an expensive piece of equipment when it doesn't outperform something cheaper. So at least there is a glimmer of hope from AMD for Core I7.
 

curnel_D

Distinguished
Jun 5, 2007
741
0
18,990
Chris, it's about time to balls up and start writing decent articles full time in place of the morons like kevin parish. If we can gauge the potential of quality of work from this article, then Toms Hardware has the potential to be just as good as the day I added it to my list of home page tabs. Get to it.
 

maxwellsmart_80

Distinguished
Feb 10, 2009
4
0
18,510
Is is *that* hard for someone to believe the AMD part could just simply be faster than the Intel part at something?

C'mon - I know AMD had a bad year last yet, but please - let them redeem themselves - don't we all remember Netburst? Is anyone still holding that over Intel's head? Who knows - Core i7 could be the *new* netburst- it may *never* make a good Gaming CPU, giving AMD it's edge back on the Gaming front.

I know AMD is currently handing it to Intel on the server front - especially when it comes to HW Virtualization.
 

marraco

Distinguished
Jan 6, 2007
671
0
18,990
[Of course gaming on a Radeon is going to give you a different result—especially in a game. But we didn’t expect variance to this extreme]

A possibility is that ATI profiles disable hyperthreading, giving the i7 a big boost.

The i7 hyperthreading is a big hindrance on games not heavy threaded. if you disable it, would get much better FPS on most games.

Anyway, your article is very good.

--Congratulations.--

You show interest in making good data, and search the true.

Also this article is very useful to tweakers.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]rdawise[/nom]Thank you Chris for this follow-up article..now where is kknd to argue....I am sorry but we all know that at lower resolutions the Core i7 will beat the P2, but as the article states, but real world the PII is hitting the high notes. Could this be a driver screw up from Nvidia...probably since you're elimnating everything else. Are there any other x-factors out there...oh yes plenty more. However I think people will get the wrong impression if they read this and think the PII is "more powerful" than the Core i7. Some one who reads this should come away thinking that the PII will give you almost as great gaming as some of the Core i7s can for less money. (Time for a price cut intel).I do a question what if you tried using memory with different timings. I believe 8-8-8-24 was used last test, but how about 7-7-7-20? Just trying to help think of reasons. Either way it gives us something to look forward to in the CPU world. Good follow-up.[/citation]

Thanks rda. Yes, we suspect that there is a driver issue at play here. Should Nvidia discover something wrong with the way its card is interfacing with Core i7 and fix it, Intel will likely take the lead again. That doesn't change the reality of today, though, where a Phenom II/GeForce will outperform an i7/GeForce in some games.

Changing the timings could have a minimal impact on performance, but as you saw from the numbers, even clocking up to 3.8 GHz/DDR3-1523 didn't even contribute that much performance, surprisingly.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]tempestor[/nom]great article!!just a thought: what about previous generation of nvidia cards? could be this is a GTX 260/285/280/... problem. maybe you could try with one of 9xxx series.[/citation]

Temp,
Thanks, yes, this could be limited to the 280. My 295 is on loan to Thomas for an upcoming GTX 295 water-cooling review, but as soon as I get it back (or as soon as Nvidia tries a different board and sends some feedback), I'll be sure to update and clarify. Great suggestion.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]Madis Kalme[/nom]Maybe Farcry optimized it more on ATI, maybe Intel is throwing sticks at the wheels of nVidia at the hardware level, maybe, maybe ... :SWhy is Intel supporting multi-ATI config, but not multi-nVidia? Why doesn't Intel let nVidia use its Atom freely? Why, oh why?There are so many factors. I think if you replace Farcry with a synthetic test, there will be less unknowns. Just maybe[/citation]

Madis,
Of course, the results of this particular story can only be tied to Far Cry, but the piece was motivated by strange results in all of our gaming scenarios, not just this one game. There were a ton of scenarios to test for in this piece, so I stuck to one game with a great benchmarking tool to make it move a little quicker, but it isn't a stretch to think that these results will carry over to the other tests as well. More info once Nvidia has had a chance to dig into this a bit more.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]jcknouse[/nom]Fantastic work, Chris. Simply awesome. I really, really, really enjoyed that analysis.Something I need to ask tho:Is the GTX280 a dual processor VDA, like the Radeon 4870X2? If not, wouldn't you expect significant gains of 2 GPUs over 1?Also...Wouldn't you expect to see even a small decline in performance (possibly miniscule/negligible...but still present) simply because running 2 individual VDAs in 2 PCI-E x16 2.0 slots requires work to be sorted between two physical devices (handling going over the SLi interface), whereas using the Radeon 4870x2 work is sent to and split on-card?I'd really be interested in seeing a performance difference between 2 4870s and 1 4870x2 with the same catalyst version, and latest firmware.Also, I am quite shocked at the nVidia suffering. I've been an nVidia customer for years. I am going to have to look at going ATI Crossfire if I build a new AM3 gaming platform later this year.This is one of the reasons I value Tom's so much...articles like this.Thank you again, Chris. This was invaluable to me both technically as well as a consumer looking for the best bang for my buck, even tho my budget isn't limited.[/citation]

jck,
The GTX 280 is a single-GPU chip--the 295 would be the next logical step up for testing, if I had one in-house. As a result, I tested the two Radeon HD 4870 512s, which are single-GPU cards as well, and found a common graphics bottleneck on those cards, indicating that the GTX 280's bottleneck on Core i7 is something other than pure GPU muscle. I hope that answers your question, and thanks for your commentary!
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]jeffunit[/nom]Why are you running games which stress graphics for a cpu review?Wny not run *cpu benchmarks*, which measure cpu performance for a cpureview?[/citation]

Jeff, because running games at unrealistically low resolutions doesn't tell you anything about the game. I can agree that there is a purely scientific gain to be had by running at 640x480, but that wouldn't uncover the issues we've stumbled on here, right? Beyond that, we might as well include a 640x480/no detail test right next to 3DMark Vantage CPU in the synthetic section.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]bf2gameplaya[/nom]Finally some quantitative analysis from Tom's, with reasonable methodology and an interesting subject. Well done, you have educated me!Yet, I see that Far Cry 2 was the only title tested and I have no way of knowing if any of these conclusions carry over to any other title.But now I have the right questions to ask, Thanks Tom's![/citation]

No problem, bf2. The good (or bad) news is that we saw the same phenomenon in all of our game testing, so while we've gone deep in just this one game, it isn't a stretch to imagine that the connection is there. It's a good story idea, though.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]jameskangster[/nom]Chris, thank you for the editorial. I really appreciate the fact that you guys try your best to respond and answer our nit-picking comments and questions. Now based on what I have observed from your articles, I have some suggestions that might help (or might not at all). What if the bottleneck is related to the motherboard and its memory timing setup?I did a quick comparison of your past setups for benchmarks (From this article, 2009-02-09 AMD AM3 article, 2009-01-07 Phenom II review article, 2008-11-03 i7 review article).Basically, you used the same hardware for i7 920 in articles 2009-01-07 and 2008-11-03Motherboard: Intel DX58SO Revision 403Memory: A-DATA DDR3-1600 2X2GB set to DDR3-1333 CL 7-7-7-20Video: MSI N280GTX-T2D1G-OCAlso just as an external reference I used Anandtech's setup from this articlehttp://www.anandtech.com/cpuchipse [...] i=3512&p=3. For those of you who hate Anandtech I apologize, but this article had the most comparable hardware setup:Motherboard: Intel DX58SO Memory: Qimonda DDR3-1066 4x1GB (7-7-7-20)Video card: eVGA GeForce GTX 280In these articles Intel's i7 920 2.66 GHz performance seemed to be dare I say better than Phenom II (although in one article Phenom II did not exist, but just looking at raw numbers) specifically related to gaming benchmarks. The interesting point here is that they all used Intel DX58SO motherboard, using 7-7-7-20 timing for the memory. The number of modules varied; however, it didn't seem to make a huge difference from what I have read so far.In the 2009-02-09 article and its subsequent editorial article Tom's Hardware used the following setup for i7.Motherboard: Asus Rampage II Extreme (X58/ICH10) LGA 1366Memory: Corsair Dominator DDR3-1600 8-8-8-24 @1.65V 3x2GB (caveat here Tom's hardware did try overcloking in the editorial article so that does vary)Video card: Nvidia GeForce GTX 280 1 GBOverall the video card chip model remained the same, the driver revisions were different but not siginifically; however Tom's used a different board and its memory setup was drastically different from its previous setups.It would be interesting to see the differences in performance comparing these setups (some of these setups might not be possible due to hardware/BIOS limitations, I didn't have time to look into that part):1. Rampage mobo with 8-8-8-24 memory timing vs Intel DX58SO with 8-8-8-24 memory timing2. Rampage mobo with 7-7-7-20 memory timing vs Intel DX58S0 with 7-7-7-20 memory timing 3. Intel DX58SO with 8-8-8-24 memory timing vs Intel DX58SO with 7-7-7-20 memory timing4. Rampage mobo with 8-8-8-24 memory timing vs Rampage mobo with 7-7-7-20 memory timingObviously I would use i7 920, the same video card and driver for above setups.I highly doubt that the memory timing would cause such a performance difference. My bet is on the motherboard.Also it would answer another question, or maybe obsfucate even more. Was it truly Nvidia's video card's fault that i7's potential was not translated into raw performance? or was it due to the motherboard, or due to memory setup? or both? or all three?I'm not expecting another editorial article for this, but it would be good to see get this straightened out. I'm really hesitant to place the blame on Nvidia for this yet.[/citation]

Great analysis Jake--no worries on linking elsewhere. Just because I write here doesn't mean I don't respect the work done elsewhere around the community :). At the very least, I'll pass this along to Nvidia. Next month we're going to be taking a longer look at running in 64-bit environments and the effects of more/less memory--this is something that I will try to fit into that, as you could very well be onto something as the motherboards are concerned.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795
[citation][nom]jp182[/nom]Thanks for taking the time to run all of those tests. They didn't use another game as the original purpose was to prove that they conducted their tests properly. Now, if they want to see what the problem is with nvidia's cards then that would be a whole other set of tests and it'll probably take a week for them to get all of their data together.[/citation]
Exactly, I suspect that'd be a good story.
 

cah027

Distinguished
Oct 8, 2007
456
0
18,780
I think Nvidia Hates Intel so much that they wrote their drivers this way to give AMD a boost in sales. Plus it will boost their own sales with AMD P2 fans. If P2 users want the best setup they would go with 2x 280's over the 4780x2
 

squatchman

Distinguished
Oct 31, 2008
211
0
18,680



That sounds like an interesting article. I've also heard that some setups get unstable if you run them at full memory capacity.

Now if only that Jane McEntegart hack over at Tom's guide would stop getting articles from Google.
 

jameskangster

Distinguished
Aug 29, 2008
17
0
18,510
To Chris (cangelini),
Thanks for the response and your attention to detail. I just wish I had my own setups to test them on my own, but they are kind of expensive just to satisfy my curiosity... BTW, you called me Jake... OUCH. My name is James (jameskangster, "james" "kangster"), but you can call me whatever you want as long as Tom's Hardware keeps writing quality articles/editorials.
 

Pei-chen

Distinguished
Jul 3, 2007
1,279
5
19,285
[citation][nom]maxwellsmart_80[/nom]Is is *that* hard for someone to believe the AMD part could just simply be faster than the Intel part at something?....[/citation]
Fact: i7 is the fastest desktop CPU available. Enough said.

[citation][nom]squatchman[/nom]That sounds like an interesting article. I've also heard that some setups get unstable if you run them at full memory capacity. Now if only that Jane McEntegart hack over at Tom's guide would stop getting articles from Google.[/citation]
Jane is a free freelancer. She isn’t being paid to write, she just like writing on her Macbook / iPhone in a Starbuck.

I wish Tom’s had kept Sara, Tamara, Rob and Ben.

Good job Chris. Finding the problem is halfway of fixing it.
 

cangelini

Contributing Editor
Editor
Jul 4, 2008
1,878
9
19,795


LOL I'm sorry James--it's been a long two days of little sleep and my mind put your first and last names together into Jake for some reason. Won't happen again =)
 

maxwellsmart_80

Distinguished
Feb 10, 2009
4
0
18,510
Pei-chen
Fact - can you see a difference w/o a synthetic benchmark? No!
Enough Said.

People put way too much stock in synthetics that mean nothing in real world scenarios. I'd be willing to bet if I sat you down in front of an AMD machine and an Intel machine, you would *not* be able to tell me which was which. Fact!

In reality, the CPU doesn't make up as much "real-world" performance on a PC as a lot of folks think, which is why I think arguments like this are so dang silly.

Besides, even talking gaming - who really cares as long as it's over 60FPS anyway?

For example, I have a 22" display that runs at 1680x1050. I can play any game I want at a steady 60fps why would I need a faster card and/or CPU - just to waste money so my e-wang could be bigger?
 

sublifer

Distinguished
Apr 25, 2008
519
0
18,980
Chris, like many others have said, Great job! this is the kind of stuff that brought a lot of readers like me to Tom's back in the 90's and it is great to be seeing it again.

I did notice one thing that it seems no one else picked up on though:

Phenom II X4 940 and Radeon HD 4870 X2(Numbers Added) 68.95 65.30

Core i7 920 and Radeon HD 4870 X2 (Numbers Added) 85.87 74.85

with these results, yes the i7 does win and show it can be more powerful than the Ph2 but if you consider the scaling it reveals that the Ph2 should also be stretching its legs more than it is. Increasing the resolution on the i7 drops the performance by over 10% whereas increasing the resolution on the Ph2 drops performance by less than 5%. This suggests that the Ph2 is much less affected by the increased demand which is oddly indicative of being GPU bound.
With all this sideways thinking I'm starting to confuse myself... anyway, keep up the good work!
 

squatchman

Distinguished
Oct 31, 2008
211
0
18,680
[citation][nom]maxwellsmart_80[/nom]Pei-chenFact - can you see a difference w/o a synthetic benchmark? No!Enough Said.People put way too much stock in synthetics that mean nothing in real world scenarios.[/citation]

Real world scenarios like A/V encoding and productivity benchmarks that i7 is 20-30% faster in?

If I wanted a computer just for games, I would get a console.
 

scrumhalf

Distinguished
Jun 22, 2004
173
0
18,680
Great article, really good investigative work here.[citation][nom]maxwellsmart_80[/nom]Pei-chenFact - can you see a difference w/o a synthetic benchmark? No!Enough Said.People put way too much stock in synthetics that mean nothing in real world scenarios. I'd be willing to bet if I sat you down in front of an AMD machine and an Intel machine, you would *not* be able to tell me which was which. Fact!In reality, the CPU doesn't make up as much "real-world" performance on a PC as a lot of folks think, which is why I think arguments like this are so dang silly.Besides, even talking gaming - who really cares as long as it's over 60FPS anyway?For example, I have a 22" display that runs at 1680x1050. I can play any game I want at a steady 60fps why would I need a faster card and/or CPU - just to waste money so my e-wang could be bigger?[/citation]
I edit HD video, the difference is quite significant. I can apply effects to 1080p and preview them in realtime. The amount of time that will save me, in addition to rendering/encoding tasks will be well worth my dough.
 

scrumhalf

Distinguished
Jun 22, 2004
173
0
18,680
Woops, hit submit before I finished typing. @maxwellsmart_80: The reason many people want a machine that hits higher than 60FPS is to be future proof for next year's game. Or it could be that people get enjoyment out of perfectly tuning their PC to get the maximum performance, much like people that like to (re)build their cars. Some do it for the reasons you stated, just remember there's always other reasons.
 

VTOLfreak

Distinguished
Jan 23, 2006
77
0
18,630
Try turning on AA and AF. I run thesame resolution on a 20" with AA set at 16x. It makes a huge difference in image quality. It however also slows modern games to a crawl. So you need +400$ cards just to run 30fps in 1680x1050.

To cangelini:

Great work on the follow-up article.

Looks like Nvidia needs to work on their drivers. I really do hope the fault lies with their drivers and not the GPU itself. Can't patch sillicon.

The fact that the 4870X2 can produce such great results kinda proves the fact that the CPU is not the bottleneck. If the i7 would be slow then you'd have bad results with both the GTX280 and the 4870X2.
 
Status
Not open for further replies.