Editor's Corner: Getting Benchmarks Right

[mapesdhs]

A friend in CA told me top-spec PC sales are quite an earner there.


Btw, you're wrong about 30fps. The perceptive cutoff rate
varies with each person. You may not need more than 30, but I do.
I've spent years researching and working with visual simulation
technologies, so as such for me anything less than 60 is very
visible. For others it may be that 50 is enough, or 40, or like
you 30, or some may have even sharper vision where 60 is still
flickery and they'de prefer 85 or more. It's all just a bell
curve.

To say there is any specific cutoff refresh rate that matters
is factually wrong. Remember human vision is comparatively slow,
eg. a bird would prefer more like 200Hz to see things even
remotely smoothly (human vision is lame by comparison). All one
can do is go by a general guide, for which most gaming/review
sites choose 30, but reviews of top-spec hw do tend to focus
on 60 instead. SGI had a saying, "60Hz, 30 hurts", thus their
DVR technology in IR gfx to guarantee a 60Hz update rate no
matter what the gfx (by varying the frame buffer width/height
in real-time frame by frame if necessary).

It all depends on what you're used to.

Ian.

 

[mapesdhs]

(I was of course referring to rlevitov's post)

 

[mapesdhs]

neiroatopelcc,

How does the AI in Crysis and Far Cry 2 compare to that in
Stalker Clear Sky? Have you played the newer Stalker game?
Just wondered.

Ian.

 

[neiroatopelcc]

I did try clear sky, but that ran very poorly on my system. So I never actually made it out of the wooden something you started in. I couldn't get anything more than 5fps no matter what I did, so I don't have that game anymore.
The ai in crysis is rather good btw. It's surprising you from time to time. Crysis is letting people down by being monotonous and by introducing a silly alien I don't like very much. The korean ai and the visuals are fine in that game. But I did decide not to buy warhead because I was afraid of more stupid aliens. Farcry's ai is letting you down bigtime though. It's incredibly predictable and not very smart. You could hide between some trees and snipe someone, and the rest will jump into a bunch of jeeps and charge you - ignoring that they can't actually run me over in a forest. And they'll park so close to you that the minigun on the roof can't aim at you, and they'll have to resort to unmounting and going at you with a pistol - if they're not dead at that point in time. Also everything is scripted, and it's done so obvious that you don't have any difficulty using that to your advantage. It's a silly way to ruin a potentially good game really. It's a bit like the single player campaigns of supreme commander - forged alliance. It's a good multiplayer game and all, but the singleplayer missions fail to impress because they're scripted rather obviously. So you could just sit at the first part of the objective and build up a huge army of experimentals before completing the first part. At which point you've won no matter what. It's annoying that sometimes even potentially good games let you down by poor finish. But it probably beats stuff like rockstar or 3do/nwc where gameplay would've been great if just the game hadn't been so buggy.

 

[mapesdhs]

Ach, sorry to hear it didn't run well, but I'm surprised that's
the case - I'd have thought your E6600/4870 system would run it
ok, if not at high detail or high res (atm I'm more used to the
lesser demands of the orig Stalker game, which runs just fine
max-detail 2048x1536 on an 8800GT).

So far I've not played FC2 or Crysis. Odd, as time has passed
and I've seen more reviews/clips on TV, FC2 seems to look better,
though as you say the best visuals can be spoiled if the AI sucks.

Did you play the original Stalker? I quite liked that (still
playing it) but I gather the major mod that was done also
improves the AI. Just wondered how it compared to the AI in the
newer Stalker game, and vs. how it works in Crysis/FC2/etc.

Ian.

 

[cangelini]

Quick update--ran with HyperThreading disabled and got some new scores that didn't change things much. Will update the story itself shortly.

 

[bydesign]

Question what happens if Hyper threading is disabled on the i7? Kind of curious if that would have any impact of the results. On the P4 it resulted in some negative effect in some apps.

The other thing that comes to mind is that it would be nice to be using the same type of memory, perhaps that will be possible in the future. I would also throw in a core 2 for a baseline. Good effort though.

 

[mapesdhs]

i7 does not use DDR2, but perhaps you're referring to the newer
AM3 Phenom2. I don't think it would make much difference though.

I see no value in comparing to a Core2 as there are already
plenty of articles comparing i7 to Core2.

Ian.

 

[cangelini]

[citation][nom]ByDesign[/nom]Question what happens if Hyper threading is disabled on the i7? Kind of curious if that would have any impact of the results. On the P4 it resulted in some negative effect in some apps.The other thing that comes to mind is that it would be nice to be using the same type of memory, perhaps that will be possible in the future. I would also throw in a core 2 for a baseline. Good effort though.[/citation]

Update posted to the first page of this editorial. There's a slight performance increase, but this might very well be a result of the fresh Windows installation (running on the same partition as that test), meaning turning on/off Hyper-Threading isn't making any difference at all here under the latest GeForce drivers.

 

[john_densmore]

Could you please try with the radeon AND hyper-threading off.

 

[neiroatopelcc]

@ mapesdhs : I didn't try the original Stalker, so I have no idea about it's ai really. Anyway, fc2 might look better on a commercial, but imo crysis has the better actual graphics. FC2's foilage is much more visibly a flat plane than it is in crysis, rocks and stuff seem a bit more 'randomly placed' than would be expected in real nature. In general the textures look a lot more like just that in fc2. Like flat 2d textures applied to angular objects. Hell, there were even trees on rail tracks some place (1.0). I assume those were fixed with a patch or something. I haven't updated mine as I'm using a downloaded iso (so the collectors ed dvd still has the foil cover on the dvd). But overall, the ai and the wierd parts of the game kinda overshadows everything else. Stuff like being able to fix a vehicle by tightning a bolt on the radiator no matter how it got broken (bullets, rockets, hitting trees) and the ability to flip a turned over car with your bare hands takes away a bit of the realism leaving you with a dumbed down shooter game set in a freeroaming world with scripted 'border events'. It's a bit like playing cs in an oblivion world really - good looking at first glance but requiring absolutely no skill and providing no thrills.

 

[mapesdhs]

cangelini writes:
> Update posted to the first page of this editorial. ...

I'm curious, this performance hit for using NVIDIA gfx wth i7, does it
only apply to gamer cards? ie. would professional users see the same
effect if, for example, using a Quadro FX vs. a FireGL?


neiroatopelcc writes:
> I didn't try the original Stalker, so I have no idea about it's ai
> really. ...

Generally it's ok, though the mod is said to improve it, and the newer
Stalker is apparently much better again; trying to circle a camp of
bad guys will not work so well if you're spotted - they'll scatter,
outflank, etc. So I've read anyway.


> ... Anyway, fc2 might look better on a commercial, but imo crysis
> has the better actual graphics. FC2's foilage is much more visibly a
> flat plane than it is in crysis, rocks and stuff seem a bit more
> 'randomly placed' than would be expected in real nature. ...

So far I've only seen FC2 in TV adverts and reviews.


> ... In general
> the textures look a lot more like just that in fc2. Like flat 2d
> textures applied to angular objects. ...

I know it's hard to compare on the fly without being able to wander
around aswell, but here's some screenshots I took of Stalker (1st game)
on my system (I couldn't capture at 2048x1536, so these are just at
1920x1200).

I know people vary in their tastes of such things, but I rather like
the murkey appearance of Stalker's visuals, a bit like the original
PS2 Mercenaries game. I'm not keen on some of the newer 1st-person
shooters that always seem to be overbright, glossy and colourful,
like the contrast is a bit whacko.


> ... Hell, there were even trees on
> rail tracks some place (1.0). ...




> ... But overall, the ai and the wierd parts of the game kinda overshadows
> everything else. ...

That's a pity.


> ... Stuff like being able to fix a vehicle by tightning
> a bolt on the radiator no matter how it got broken (bullets, rockets,

That does sound a bit dumb.


> hitting trees) and the ability to flip a turned over car with your
> bare hands takes away a bit of the realism ...

8D

Ian.

 

[neiroatopelcc]

Stalker seems to suffer from some of the things I don't like too.
At first glance it looks fine, but there's an absense of shadow details, and some objects are a bit odd. Like on page 3 there are two bricks half sticking out the wall on the right lower side of the hole - and they're a completely different colors than the bricks that are in the 'solid part'. Probbly because all standalone bricks are using the same texture. Also on the first image - where the big tree slightly to the left of the middle touches the ground ... it looks like sawn off and sat there. But then again, that's normal for a game that isn't completely new. Some foilage to cover it up would've been nice though.
The 4th picture is a good example of shadow btw. There appears only to be static shadows 'painted onto' stuff like brick walls and metal fences (which looks fine actually) and the type of shadow that just draws a dark uniform cover over the earth representing an object. I don't remember what it's called. But it's certainly not softshadows.
I think the only reason stalker looks to your liking, is because the textures they've made are high quality and match up together well. And ofcourse because they haven't overdone the hdr effects - possibly hdr isn't even on?

anyway I'm with you on the non shiny thing - ea hasn't noticed my dislike for their shiny roads yet though, so every need for speed game since and including underground has had this kind of wet shine with reflections on the road. Annoying, and taxing on the graphics considering how non-spectacular the graphics actually are.
Anyway anyone can do the mistake and overdo a feature. If you'd enable hdr on oblivion some stuff would unnaturally shine a lot too - but that was first gen of the hdr games, so that's to be expected. Returning to bloom effect made the game much better looking, and the game's great anyway.

 

[mapesdhs]

neiroatopelcc writes:
> Stalker seems to suffer from some of the things I don't like too.

Worth pointing out of course that it's been much improved with the
newer Stalker game.


> details, and some objects are a bit odd. Like on page 3 there are two
> bricks half sticking out the wall on the right lower side of the hole
> - and they're a completely different colors than the bricks that are
> in the 'solid part'. ...

Yes, you're right about that. I guess I don't notice these subtle
details when I'm running around trying not to get wasted.


> like sawn off and sat there. But then again, that's normal for a game
> that isn't completely new. Some foilage to cover it up would've been
> nice though.

Indeed, something to cover it might have been wise. Ironc, Champions
of Norrath (Return to Arms) actually did this better.


> The 4th picture is a good example of shadow btw. There appears only
> to be static shadows 'painted onto' stuff like brick walls and metal
> fences (which looks fine actually) ...

I'm not sure which shadows you're referring to there, but all foliage
shadows are fully dynamic.


> I think the only reason stalker looks to your liking, is because the
> textures they've made are high quality and match up together well.

Probably!


> And ofcourse because they haven't overdone the hdr effects - possibly
> hdr isn't even on?

Not sure it even uses it, given the rather different rendering
technique being employed. Then again, for the 'time of day' in the
game I took the pics, the sunlight is pretty weak anyway, so shadows
were not that pronounced at the time.


> dislike for their shiny roads yet though, so every need for speed
> game since and including underground has had this kind of wet shine
> with reflections on the road. Annoying, and taxing on the graphics
> considering how non-spectacular the graphics actually are.

Yes, it's that shineyness on everything that I find annoying.
Everything looks like it's been made from obyidian.


> Anyway anyone can do the mistake and overdo a feature. If you'd
> enable hdr on oblivion some stuff would unnaturally shine a lot too -
> but that was first gen of the hdr games, so that's to be expected.
> Returning to bloom effect made the game much better looking, and the
> game's great anyway.

Indeed, I kinda vary as to whether I like it or not in Oblivion. Here
are some pics I took a year ago, before I bought my 8800GT (thus the
pics are only 1600x1200; I normally run it at 2K, no AA, 16X AF).

Ian.

 

[neiroatopelcc]

My internet's way too unstable right now to download those pics, but I can tell you which shadows I was talking about in picture 4 at least.
Look at the ground on the right side of the image. There's painted on shadows of the building which lay flat on the ground. Appearently those shadows are cast by a strong light (sun) comming from the bottom right corner. Be amazed by the pronounced siluette of the building, and the ability of the window or whatever it is that makes sure no shadow is cast 'under' the left most pillar of the actual roof you can see - wierd ehh? And now look to the left where there's a tree casting a shadow on the building behind it. Notice something? now the shadow falls to the right of the tree. So there must be two suns or otherwise two strong sources of light, as the tree's siluette is very sharpedged as well. That's the effect of not having softshadows. Now why there are two 'sun' light sources I don't know. But that does mean that either when you turn around you should see a strong light both on your 2:00 and 10:00 or thereabout. Either that, or shadows aren't as dynamic as you think. But ye, upping the pace of the gameplay makes you not notice those things. In stuff like farcry you have time to actually look around and stuff though - like in oblivion. Only in oblivion graphics were a lot better than anything seen before. With farcry that isn't the case, as even the older crysis has similar if not better graphics.
Anyway a good graphics engine alone doesn't make a good game. Take unreal 3 as example of that. Fast graphics, good graphics, but exceptionally boring gameplay (compared to their predecessors).

 

[Guest]

I would like to see some day a a true benchmark comparison where you compare apples to apples, and not ferraris to hummers, then you say that the ferrari of $200K is faster than a Hummer of $40K. What I mean is that you must try to compare systems based on some equal specification. By now the more equal specification for a system, processor, or graphics card is PRICE, so you can compare processors of the same price and then do the benchmark, I think that would make sense, or else you are comparing some $$$$ processor with a less pricey $$ processor is almost obvious that the first one will pwerform better.

Silvio from Valera, Venezuela

 

[Guest]

i just think it's because of the architecture.

other than the dx10.1 enabled on farcry, simply, the nvidia gpu reached it's limit on how much a code can be optimized and so it stops, while the Ati's, depending a lot more on driver compiling optimized code, that made me think that, in the case of i7, the faster is the driver compiling the code for the gpu, the fastes it reaches(sry about my english i'm italian ). I think simply the amd processor have at least 2 years of memory controller optimizations and experience - while intel just obtained the tech - that can lead on a faster game's performance since, in disrespect of the bandwith of the ram, can still have a edge on latency, and maybe the 790FX chipset have lower pci express latency and faster pciE/cpu communication also, due to hypertransport(we always forgot how amazing and innovative that feature was- far far ahead the intel technology, and it continue to evolve)and permicts a strictly horse-power architecture like NVIDIA's to get a little more efficency when it comes down to a high-optimized and at the same time complex engine like Far Cry 2's one.

 

[Mike995]

OK, so the Phenom II still blows... its still MUCH slower clock for clock than the i7. People seem to neglect the fact that a 2.66 GHZ 920 i7 is compared to the 3.0 GHZ not so phenomenal II, and it still gets beat out. The i7 is and will always be the better overclocker, and since the history of the amd vs intel battles. Overclocking has ALWAYS been Intel favored, 920 i7 ='s much much faster clock for clock and has a higher clock speed ceiling. Flame on amd fans, you still don't have a competitive chip.

 

[mapesdhs]

AMD competes on price and price/performance, but they clearly
don't have a top-end chip atm. I tend to lean slightly to the
performance side of the equation, but everyone is different.

Given where the real bottleneck is in most games (GPU) I'd say
an oc'd Phenom2 system makes a lot of sense right now, especially
if paired with the surprisingly good 4850x2 2GB. For gaming, an
i7 system can be faster, but whether it's worth the extra cost
is something about which every person would have a different
opinion. It's other application areas where i7 really shines,
especially video encoding (what I want it for), rendering,
scientific apps, databases, etc.

Early results suggest a 2.5X speedup for i7 XEON render clusters
over the old XEON. And SGI's just landed 6 of 7 US DoD
supercomputer contracts which will be using large i7 XEON based
systems ($40M worth).

All the arguing about this is pointless IMO. AMD has a cheap
product, Intel has a fast product, both can be overclocked,
there's a degree of overlap, so there's something for everyone.
It would be rather boring & pointless if both company's CPUs did
the same thing at the same speed & clock.

If budget is not an issue, an i7 gaming system will be faster,
but not by a huge margin. Those touting AMD are merely pointing
out the obvious, namely that for *gaming*, one would see a much
larger fps performance increase by getting a Phenom2 and using
the cost difference to obtain a much more powerful GPU solution,
at least for those games where the bottleneck is indeed the GPU,
which let's face it is most games really.

If I was building a new gaming system right now, I'd get a
Phenom2 setup for the reasons given above. But I'm getting an i7
system instead because my current system is ok atm for gaming
(6000+/8800GT) and I want the i7 system for video encoding, not
gaming.

If you don't mind paying the extra for the i7 setup, that's ok,
lots of others will too, but not everyone can afford to, or wants
to even if they can afford it.

Ian.

 

[Inktfish]

Thanks for pointing this out

 

[Marcus52]

Excellent work thanks Chris! I am very much interested in Nvidia getting this right, since I run at 1920x1200 and have the i7 920 paired with a GTX 280. It could well make a difference when I SLI this rig - Crossfire instead? I like the Cuda and PhysX support on the Nvidia GPU, but am not willing to give up a 30%+ difference in performance to do so.

Now if we could just get more benchmarks to compare vid cards with instead of just frame rates. . . maybe that 30% is worth it after all?

 

[touchdowntexas13]

The only title that showed amd with the upper hand was Far Cry 2 right? I would still call the i7 a winner if that is the situation. It may not have the best price, but it's fast...