objective benchmarks

[SPAM]

Premise:
Since there is a real disparity in tests results between different hardware reviews,
Since this leads to confusion and mistrust.
Since optimizations of video drivers can as history shows in fact be cheats.

We need objective, cheat proof benchmarks and VGP reviews do you have any suggestions???!!!

Here are mine:
-Use custom time demo's of popular games NEVER use included benchmarking utilities except to reveal cheats.
-look for anomolies in card performance that indicate "optimizations" or cheats.
-Use a larger pool of games to determince performance consistancy.
-Evaluate video cards with more than one generation of drivers
to test both Image Quality and speed.

What do you think?

 

[TheRod]

Well!

Most of your points are what most hardware review site try to do.

USE CUSTOM GAME DEMOS : The problem with that is the choice of the "scene" can affect performance... So different review can still have diffreent results depending on the type of sequence that were choosen in the demos.

LOOK FOR ANOMALIES : Any anomalies are eventually discovered. And I'm pretty sure that ATI and nVidia learned in the past few months that "cheating" is not the way to have good press.

LARGER POOL OF GAME : This would be great, but this would multiply the job for the reviewers a lot! Why not having game company to add banchmark of different GPU for there specific game. For example : HL2 performance in high quality on this GPU is 100-120FPS, on the other is 70-90FPS... And so on!

GENERATION OF DRIVERS : Like I said before, this would add a lot of work to reviewers.

This was some arguments... Of course, more is always better! But I don't want to read reviews 3 months after the pruducts are out!!!

--
Would you buy a GPS enabled soap bar?

 

[SPAM]

A lot of the review sites use standard benchmarks and this makes them likely targets of "optimizations". I know that comoaring results between sites may be more difficult if home brewed time demo's are used. If however the tests compare the competitors cards fairly, the strengths and weaknesses should be clearly seen AND it would help to produce real world results on the frames per second test numbers.

 

[SPAM]

Generations of drivers:

I see your point Perhaps (as on some sites) video drivers are tested to look for performance improvements, image quality, and the scaling of results when different DX9 features are enabled.
This would be the place to discover anomolies and optimzations that would artificially inflate performance numbers.