-Fran-
Glorious
anbello262 :
Actually, I also agree that it is not such a good idea to benchmark mmorpgs. My reason for that is that when you REALLY are cpu bound is when there are lots of players fighting lots of things with effects, and that is the hardest thing to produce consistently. In order to produce it consistently, you would require huge amounts of time and effort, that could be better spent benchmarking several extra games instead.
BUT there are some ways of doing it repeatably (or semi-repeatably) that I can think of:
1- Empty place, no people, some or many mobs, only 1 person fighting: Very repeatable, but useless as a benchmark since there is no taxing workload. Useless
2- Main commercial city filled with people on a busy day at the highest time: Repeatable enough, as long as it's done in the same day at the same approximate time. Somewhat taxing, but no effects and no fighting. The main disadvantage is that a result from a certain date is useless in any future, as game population varies too much on the medium term, so no comparability with future hardware. The benchmark results are only good for one review. Basically useless
3- Set up your own private server and configure many bots to fight repeatedly against certain mobs: Perfect repeatability and realistic taxing, but it would cost too much (money and time) to benchmark only one game. Even worse if you want several mmorpgs. But an advantage: Setting up just one server is good enough for several years, since mmorpgs don't change much over time, and only a few are popular at any given period of 3-5 years.
Last option would be the one that is feasible (although quite costly for only 1 game).
I don't know almost anything about crysis, so I wont comment on that.
BUT there are some ways of doing it repeatably (or semi-repeatably) that I can think of:
1- Empty place, no people, some or many mobs, only 1 person fighting: Very repeatable, but useless as a benchmark since there is no taxing workload. Useless
2- Main commercial city filled with people on a busy day at the highest time: Repeatable enough, as long as it's done in the same day at the same approximate time. Somewhat taxing, but no effects and no fighting. The main disadvantage is that a result from a certain date is useless in any future, as game population varies too much on the medium term, so no comparability with future hardware. The benchmark results are only good for one review. Basically useless
3- Set up your own private server and configure many bots to fight repeatedly against certain mobs: Perfect repeatability and realistic taxing, but it would cost too much (money and time) to benchmark only one game. Even worse if you want several mmorpgs. But an advantage: Setting up just one server is good enough for several years, since mmorpgs don't change much over time, and only a few are popular at any given period of 3-5 years.
Last option would be the one that is feasible (although quite costly for only 1 game).
I don't know almost anything about crysis, so I wont comment on that.
Let's drop Crysis for the time being, lol. I don't want to go back to 2007 discussions XD
So, about MMORPGs... Yes, they are hard to test and most probably a PITA as well since to reduce the error margins in your measurements, you'd have to repeat the tests at least a dozen times.
Like you say, there are ways. Toms is a review power house, they have done a lot to discover issues and in these almost 20 years that I've been reading the site (since 2001), I've found a lot of interesting articles that actually go against "common understanding". The bet example I have is the 2D rendering investigation they did "just because" and prompted ATI to get their act together. Also the Pentium 3 errata.
I mean, I am raising the bar here, because I know Toms has capable people that can do it. I mean, they were already doing it with WoW. I don't see why they can't do it again. They could even have a dedicated review series just dedicated to investigate how CPU scaling works with MMORPGs and MP games as a separate review/article. Options are out there for them to get the information to us.
But yeah, it is a lot to ask and I do get that. We can contribute if they ask (referencing your next post) or they could contact the Devs behind some of the MMORPGs and see if they could accomodate a testing server for them or something? We could even go there and participate or something? A logistic nightmare for sure, but I'm 1000% sure it would be fun as hell.
Cheers!
EDIT: Typos.