Samsung Accused of Manipulating Benchmarks Again

Page 2 - Seeking answers? Join the Tom's Hardware community: where nearly two million members share solutions and discuss the latest tech.
Toggle sidebar Toggle sidebar
Status
Not open for further replies.


Usually true. If you add in that the benchmark program artificially overclocks the processor in a way that no other program does and it's extra true.
 
@haswell5271 short answers: 1. yes. 2. no. 3. yes. long answers: 1. yes, of course people care, why wouldn't they want to know the performance of whatever product they're buying? 2. no, that's the whole point of this article. there is code written into the OS that ONLY when benchmarking software is detected will the CPU/GPU run at full speed. every other bit of software will run at a throttled down speed, that's why it's CHEATING THE BENCHMARK. 3. yes, you are wrong, for reasons why you're wrong and why the rest of your post misses the point completely, please see above answers and reread the article.
 
@bombebomb "If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though"

that's EXACTLY the case! what article did you just read? they explicitly state that running a benchmark activates all cores and max's CPU speed, then RENAMING said benchmark and running it again, three cores were disabled with the 4th throttled down.
 
@bombebomb "If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though"

that's EXACTLY the case! what article did you just read? they explicitly state that running a benchmark activates all cores and max's CPU speed, then RENAMING said benchmark and running it again, three cores were disabled with the 4th throttled down.
 
@bombebomb "If the tests were done differently say, all cores hit 2.6ghz during benchmarks and nothing else that would throw lots of red flags. This is not the case though"

that's EXACTLY the case! what article did you just read? they explicitly state that running a benchmark activates all cores and max's CPU speed, then RENAMING said benchmark and running it again, three cores were disabled with the 4th throttled down.
 
I have no tide with Samsung and not owing any of their latest hi-end phones, but honestly, how many of smartphones/notes buyers really care about benchmarking. For me, I am more interested in how good the screen is, how long the battery can last and how high the resolution of the camera can go.
 
justhandguns
I have no tide with Samsung and not owing any of their latest hi-end phones, but honestly, how many of smartphones/notes buyers really care about benchmarking. For me, I am more interested in how good the screen is, how long the battery can last and how high the resolution of the camera can go.
Click to expand...

Agreed, so why would Samsung try to cheat the results?

To me, these companies know most customers don't look at benchmarks but they do know if they can advertise, "*Fastest phone!"

* based on highest score of cpu on tests X,Y,Z

there's a marketing advantage, so I feel the cheating is not for the people looking at benchmarks because most of them will be aware of Samsung's shameless tactics. Its more for some promotion / marketing at the highest.
 


And you think benchmarks show the true performance of hardware in the real world anyways?

So when a 8 core CPU gets 2x the score of a quad core, the majority of real world situations would call for that same 2x scaling, correct?

The problem is that benchmarks are just benchmarks. To truly test it you want to run real apps and see the performance for yourself. I have a S4 and I can tell you that it is truly a powerful phone that's smooth compared to my Droid Razr Maxx work phone.

Most of the Android smartphones will use the same CPU in the same market segment. The top end is on the Snapdragon 800 for the most a part and apart from difference in clock speed they are the same at the core level so overall performance will be the exact same in the real world.

I don't like the idea of inflated benchmarks but then again if you are using synthetic benchmarks as a gauge for real world performance and not for what it is, which is performance in a perfect world then you need to rethink benchmarks.
 
@calam1ty - there are massive gas guzzling engines that shut off cylinders to *save fuel*, I bet they state the max horse power not the hp while fuel saving is active.

I don't really see the problem with this to be honest. To me a benchmark should be the maximum attainable performance for the system being tested, under normal circumstances. So for instance, if an app CAN use all four cores at 2.3ghz then to me thats what I want benchmarked.

That being said I don't see why the benchmark doesn't stress the phone enough to automatically enable the other cores?
Also there's only a 20% gain by forcing full power mode? Thats 4x 2.3ghz versus 1x 300Mhz?
 
@Iatemyself you missed the point completely, the note 3 WILL NOT run at 2.6Ghz, stock is 2.4ghz. Hence why I used that specific frequency if it ran at +200mhz that would be a pure overclock for one reason.

Sorry if it was confusing.
 
Sorry but This is not cheating what it is doing is showing what the phone can do. There is nothing stopping other companies from doing the same. I myslef when benchmark put my Xperia ZL into full throttle to see what my max scores will be. From what I am seeing here from people is they would rather see lower scores from a given device which does not relate to what the hardware can actually do. The whole idea of a benchmark is to get the most out of the hardware so no samsumg has done nothing wrong here all it means is no one else had the idea first.

If you are going to give samsung backlash for this well then maybe you should do the same for pretty much every hardware site on the internet because they all overclock the hardware to get the most out of it. Some will even put a overclocked video card against a factory clocked card & say the factory card is slower ummm will no duh.

It is not like samsung actually overclocked their hardware it is all running within spec but set to performance mode which most of us do anyways when we bench our own device. So to expect anything less from the hardware when I bench a device is just insane. It would be like getting a car hyrid that has a v6 in it but only runs on 3 while everyday driving. Let me tell you I would be upset if I stepped on the pedal & all six pistons did not produce power. Same idea so what people are saying is samsung is wrong for turning it up a notch when benchmarking but these same people would be crying how slow the device was if they left it in battery mode or if samsung left it in performance mode all the time they would complain how their battery life sucked. I guess you can not please everyone. By the way I am not a samsung fanboi I dont own any phone that is samsung because I don't like the look of a galaxy device.
 
@hardrock40 "Sorry but This is not cheating what it is doing is showing what the phone can do."

this is showing what the phone can do only under specific circumstances as defined by samsung, which is, when it is detecting a certain benchmark. the article stated that taking the same benchmark and renaming it, the performance enhancement did not engage, simply because the coded file samsung put onto the phone did not recognize the name of the benchmark. this is not "performance mode" vs "battery mode". to fit into your analogy with the v6 in a hybrid... the equivalent of what samsung did is all 6 cylinders are running when flooring it on the daytona speedway, but then when trying to do the same on an unknown track it automatically shuts off 3 cylinders.
 
hardrock40
Sorry but This is not cheating what it is doing is showing what the phone can do. There is nothing stopping other companies from doing the same. I myslef when benchmark put my Xperia ZL into full throttle to see what my max scores will be. From what I am seeing here from people is they would rather see lower scores from a given device which does not relate to what the hardware can actually do. The whole idea of a benchmark is to get the most out of the hardware so no samsumg has done nothing wrong here all it means is no one else had the idea first.

If you are going to give samsung backlash for this well then maybe you should do the same for pretty much every hardware site on the internet because they all overclock the hardware to get the most out of it. Some will even put a overclocked video card against a factory clocked card & say the factory card is slower ummm will no duh.

It is not like samsung actually overclocked their hardware it is all running within spec but set to performance mode which most of us do anyways when we bench our own device. So to expect anything less from the hardware when I bench a device is just insane. It would be like getting a car hyrid that has a v6 in it but only runs on 3 while everyday driving. Let me tell you I would be upset if I stepped on the pedal & all six pistons did not produce power. Same idea so what people are saying is samsung is wrong for turning it up a notch when benchmarking but these same people would be crying how slow the device was if they left it in battery mode or if samsung left it in performance mode all the time they would complain how their battery life sucked. I guess you can not please everyone. By the way I am not a samsung fanboi I dont own any phone that is samsung because I don't like the look of a galaxy device.
Click to expand...

No, its like a car having a special key for auto reviewers/testers. When they insert the key the car gets an extra 20% Horse Power unlocked and augments the test results.

Whereas another car actually using the same/similar engine, didn't have a special key for reviewers/testers and their results are actually what everyday consumers would get.

You cannot replicate benchmark results with other apps since Samsung treats benchmark apps different than other apps even though they are as resource intensive.

 
So Samsung made benchmarks run good on its phone. I ask the simple question of "How does this impact you?" Is the phone slower than you anticipated? Does this impact your decision to buy one or not? Samsung is forcing their hardware to run full throttle when benchmarking.

That doesn't sound like cheating to me. What it sounds like is everyone else complaining because their phone doesn't have code to specifically run at max speed when its running a benchmark. Weather or not this is indicative or "real world" app performance, it is indicative of "maximum" performance.

P.S. - cheating on a benchmark suite is historically all they are good for imo.
 
it is kind of silly that they did this but in the end the phone really did as well as the benchmark said it did. inflating numbers makes me think they lied about performance which they did not.

Either way you look at it the phone did as well as the bench said it did.
 


You know, since a lot of people game on phones, there are actually a lot who place stock on how powerful the hardware is in a phone. When someone is going in to buy a phone saying "I want the fastest phone for my buck" and Samsung has skewed benchmark results compared to the competitions, this is a potentially decision making factor. It amazes me how fast so many people are to excuse this type of BS.
 
Well the point is would you want your device to be running full bore at all times & draining the battery faster no you would not. Did samsung do anything wrong not really all they did was make sure the device was able to perform at full speed when the user is testing their device to see how fast it is. This is called performance profiling and is not meant to be used for everyday use but just when benchmarking.

Did they make it a bit hinking maybe but I prefer knowing that my device is set to max when testing for speed. By the way why wouldn't they use a profile made for the benchmarks how else is it supposed to know when a user is running a set of tests. AMD & Nvidia have performance profiles for their Video cards that tweak setting for different games to make them run faster or tweaks for the benchmarks themselves but you rarely hear about that on these sites. Why because it is just common sense that a device maker or card maker will want to make sure their hardware is giving the best result so they give a better user experience to the end user.

like I said samsung & most likely others to follow will be doing this type of thing for years to come & I for one want my device to perform at it's best when testing it so I don't feel cheated by a purchase I just made. It is called using performance profiling for benchmarks it has been going on in the PC world for years & now a mobile device maker is doing it & everyone thinks it is a bad thing. just my 2 cents on this & just my opinion on it. Everyone has the right to have their own opinion in this world.
 
Just amazing how far Scamsung will go to beat fellow Android companies and Apple. Samsung does it fact make quality products but its business ethics are extremely poor. I liked Samsung until I read all about their business practices in Korea and abroad.
 
If benchmarks were irrelevant than why would Samsung do this?

Just because YOU don't use them doesn't meant everyone doesn't.

Just because YOU don't buy a car based on performance results doesn't mean everyone else does the same as you.

 
@hardrock40 i'm not sure how you still don't understand what happened here, but based on your response you really have no idea what the article is saying and missed the point completely. perhaps you should consider reading the article over again.
 
And here I thought Overclocking was all about finding the maximum your PC would run stable 24/7. Atleast that's what I do. My 980x purrs along at 4.6Ghz happily all day long. I do have Watercooling, but still. It's my stable max 24/7 speed.
 
"Shocked at the sheer stupidity of people who don't see what's wrong with this or support Samsung. And I thought Apple fanboys where bad!"
I own an S3 and don`t really care about benchmarks,since most of the current generation phones are pretty close in performance, in fact never installed one in my phone, i bought it because i like it better than iphone 4/5, so pretty much i don`t give a s*** if apple/htc/samsung/lg/sony does anything similar.
 
Status
Not open for further replies.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom