Samsung Accused of Inflating Galaxy S4 Benchmarks

[fulle]

People only seem to point to benchmarks when they support their argument.

These benchmarks ARE misleading, since generally we're trying to get an idea of how the device would perform in prolonged gaming when we run GPU benchmarks, and in actual gaming situations the GPU would run at 480MHz, not 533MHz. I'm glad Samsung got called out on it.

That said, the "problem" with the S4's graphics performance (which isn't bad at all, actually), is with the drivers. Apple does a great job optimizing PowerVR drivers for their devices, so that they get more performance from the GPU. Android drivers for PowerVR GPUs are inferior, so beefier hardware is needed to get similar results.

 

[wemakeourfuture]

eiskrystal

Well, it depends. If the point of your benchmarking is to find out the phone's top speed then forcing it into its higher speed mode IS what you want to do.

It may be that the phone won't automatically push itself because it doesn't recognise the benchmarking tool as needing full power and they put in code to force it into that mode.

....

You still don't get it...

Yes benchmarks are made to push the device, that's not the issue. The issue is Samsung introduced specific instructions to detect a benchmark versus non-benchmark software and increase its clock speed to produce results that the benchmark will get but other apps won't.

Do you get it?

If not, please read the article again.

http://www.anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4

 

[Bloob]

I wonder what, if anything, would happen to the device if you ran the benchmark for a few hours.

 

[house70]

You are missing something. There's a difference between having architecture that boosts based on resource intensive cases compared to having explicit instructions to only produce higher results when benchmarking software is being used.

There's CPU's that boost the clock speed when needed, this is an architecture design to improve performance. Then you have Samsung that says lets boost clock speed when a benchmark is being performed so results are inflated and not a true representation of the hardware performance.

The manufacturer says: "Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance." IMO, that means higher GPU clocks are used not only by benchmarks, but also other apps that requires it. This effectively negates your argument.
Before you jump at my throat, just letting you know I have no dogs in this. I couldn't care less what happens to Samsung or any other manufacturer, for that matter. If the hardware runs the way it's supposed to run, it's OK in my book. Like I said in my first post, I never research benchmarks, all I care about is real life performance. I think this whole thing is blown out of proportions.

 

[Vladislaus]

1. Way to go Samsung, you make great hardware and you don't need to inflate your benchmarks.

2. Apple still spanks your GPU when it comes to same generation hardware - make better GPUs, it's not rocket science...

3. Using Franco Kernel, I can dynamically push my Nexus 7's GPU for specific games - It would be great if Samsung gave the users that ability but doing it behind your back and then only for specific apps (benchmarks) speaks volumes.

4. I would love to see the same people make the same excuses if Apple had done the same... Inflating benchmark results beyond what normal apps use deceitful.

First The GPU isn't overclocked to 532MHz when running the benchmarks. Benchmarks are run at the rated speed of the GPU. Even before the Galaxy S4 was released it was known that the rated clock of the GPU was 532MHZ. All other applications are run with the GPU underclocked.

The iPhone 5S will most likely have the performance of about the current iPad 4. So I have my doubts it will have any change against the Snapdragon 800 equipped Galaxy S 4.

But I do agree that Samsung should be ashamed of doing this.

 

[wemakeourfuture]

house70
The manufacturer says: "Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance." IMO, that means higher GPU clocks are used not only by benchmarks, but also other apps that requires it. This effectively negates your argument.
Before you jump at my throat, just letting you know I have no dogs in this. I couldn't care less what happens to Samsung or any other manufacturer, for that matter. If the hardware runs the way it's supposed to run, it's OK in my book. Like I said in my first post, I never research benchmarks, all I care about is real life performance. I think this whole thing is blown out of proportions.

I think you're not understanding why AnandTech did the test to confirm the claim made by someone else. And AnandTech's conclusion and why everyone is not content with what Samsung has done.

AnandTech's partial conclusion:

We’ve said for years now that the mobile revolution has/will mirror the PC industry, and thus it’s no surprise to see optimizations like this employed. Just because we’ve seen things like this happen in the past however doesn’t mean they should happen now.

What Samsung needs to do going forward is either open up these settings for all users/applications (e.g. offer a configurable setting that fixes the CPU governor in a high performance mode, and unlocks the 532MHz GPU frequency) or remove the optimization altogether. The risk of doing nothing is that we end up in an arms race between all of the SoC and device makers where non-insignificant amounts of time and engineering effort is spent on gaming the benchmarks rather than improving user experience. Optimizing for user experience is all that’s necessary, good benchmarks benefit indirectly - those that don’t will eventually become irrelevant.



----


The issue is they have very specific instructions built into the device to only fire up when a benchmark is occurring to inflate results. This is done for the sole purpose of misleading/marketing. This is the issue that other have seen, that prompted AnandTech to investigate and why many aren't thrilled about what Samsung did.

Now Samsung claims certain apps get access to 533MHz except very power intensive apps, but long and behold the exception is benchmarks which have special instructions to give the impression that the device will run faster on heavy demand but non-proprietary apps won't enjoy it, even though you're using a non-proprietary benchmark.

It's equivalent to Intel saying they have a 4.0Ghz chip (no boost mentioned in the marketing), but the only time you'll ever see it is with a benchmark other than that you get 3.6Ghz (unless you use a handful of Intel's proprietary software)

After phones are released there are many sites out there using benchmarks to compare various models. If a manufacturer is inflating results just for benchmarks without disclosing this, they are being disingenuous in the minds of many (including myself).

 

[house70]

...
Now Samsung claims certain apps get access to 533MHz except very power intensive apps, but long and behold the exception is benchmarks which have special instructions to give the impression that the device will run faster on heavy demand but non-proprietary apps won't enjoy it, even though you're using a non-proprietary benchmark.

It's equivalent to Intel saying they have a 4.0Ghz chip (no boost mentioned in the marketing), but the only time you'll ever see it is with a benchmark other than that you get 3.6Ghz (unless you use a handful of Intel's proprietary software)
...

Sorry, but what you just said doesn't make much sense. Also, the analogy you're trying to draw is incorrect, because Samsung has specifically stated that other apps (besides benchmarks) have access to the highest clock settings.
I can see why you'd have a beef with the benchmarks being added to that list of apps that trigger the highest clock, but that's what it is, if they weren't, then the benchmarks would not be able to benchmark properly the phone at the highest settings.
I regard this as a voluntary underclock (and possibly undervolt) for the vast majority of apps that don't need the full power of the GPU, rather than a deceitful overclock (which is not, anyways).
Then there is the issue of believing Samsung or not when they say other apps are designed to trigger the high GPU settings, besides the benchmarks. Obviously you choose to not believe them, I choose to believe them. If the apps are proprietary or not, that's a different issue altogether. Samsung makes the hardware, it is natural they'll design software to take full advantage of that. If a certain game can't push the GPU to do that, maybe you should contact the game's developer (as an example).