• Sign in to Neowin Faster!

    Create an account on Neowin to contribute and support the site.

Sign in to follow this  

OnePlus and Meizu just got busted for benchmark cheating

Recommended Posts

Yorak    416

OnePlus-3T-Review-1-840x560.jpg

 

The OnePlus 3, OnePlus 3T, and Meizu Pro 6 are rigged to cheat in benchmarks, a report from XDA-Developers revealed.

 

XDA-Developers, with help from Primate Labs (makers of the popular benchmark suite Geekbench), discovered that the three phones are set to artificially increase their CPU performance when running certain benchmarks. This is more than just optimization – the software on the devices specifically looks for a number of widely-used benchmarks and tells the CPU to kick into overdrive whenever they are detected.

 

The rigging can be observed directly, by monitoring the CPU activity when regular apps and benchmark apps are run. In the case of the OnePlus 3 and OnePlus 3T, evidence of the cheating can also be found in the devices’ firmware, in the form of a hard-coded list of benchmarks: Geekbench, AnTuTu, Androbench, Quadrant, Vellamo, and GFXBench.

 

It’s telling that a “disguised” version of Geekbench 4 called “Bob’s Mini Golf Putt” did not trigger the high-performance mode on the OnePlus 3T. In other words, OnePlus can’t claim that the CPU kicked in due to the load generated by the benchmark – if that were the case, the phone would have behaved identically when running Geekbench 4 and “Bob’s Mini Golf Putt,” which is the same app with a different name.

 

CPU shenanigans

 

So how exactly does the rigging work? In the case of the OnePlus 3 and 3T, the CPU is rigged to idle at 1.29 GHz for the big cores and 0.98 GHz for the small cores, even when there’s no load on the processor. For non-benchmark apps, both the small cores and big cores idle at 0.31 GHz.

 

Meizu has a different – and curious – approach: on the Pro 6, the big, high-performance cores of the CPU are set to activate whenever benchmarks are detected. The weird thing is these big cores should be kicking in anyway, but they normally don’t do so when running regular apps. So instead of tweaking the processor to work as it’s supposed to (big cores kick in when needed), Meizu programmed the CPU to simulate normal functioning when benchmarks are detected.

 

The XDA-Developers report has all the technical details, and it’s definitely worth a read. We’ll just note that the results of the cheating are quite modest. As you can see in the chart below, the performance increase generated by the benchmark rigging is small, just a few percentage points.

 

OP3T-Multi-Core-Throttling-1.jpg

 

OnePlus came clean

 

When confronted by XDA, OnePlus admitted the existence of the benchmark rigging behavior:

 

“In order to give users a better user experience in resource intensive apps and games, especially graphically intensive ones, we implemented certain mechanisms in the community and Nougat builds to trigger the processor to run more aggressively. The trigger process for benchmarking apps will not be present in upcoming Oxygen OS builds on the OnePlus 3 and OnePlus 3T.”

 

The company said the OnePlus 3 and 3T CPUs are also hard-coded to go into high-performance mode when running games and that’s not going to change. But while that’s understandable (you want graphic-intensive games to benefit from the extra CPU oomph), there’s no good reason to have the CPU rigged for benchmarks.

 

To its credit, OnePlus acknowledged its mistake and promised to fix it quickly. And, according to XDA-Developers, the behavior was not present in the firmware when the OnePlus 3 launched, as it was added when the Oxygen OS (global) and Hydrogen OS (China) development teams were merged.

 

Nevertheless, there is no justification for deceptive behavior that ultimately hurts consumers.

 

Those who forget history are doomed to repeat it

 

It’s not the first time phone makers are caught red-handed when it comes to benchmarks. In 2013, AnandTech found that the Galaxy S4 gamed benchmarks and a following survey from the same website showed that multiple devices from HTC, Asus, LG, Samsung were rigged to fool one or multiple benchmark applications. Following the public outcry, some companies vowed to change their practices, while others, including Samsung, refused to admit any wrongdoing.

 

Following the 2013 revelations, some benchmark makers tried to proof their apps against manipulation, though it’s only so much they can do against a determined manufacturer.

 

It’s not just OnePlus and Meizu

 

One last twist in the XDA report: the publication tested other devices from multiple manufacturers and found some that failed the hidden benchmark test that exposed the OnePlus 3 and Meizu Pro 6. The report did not reveal which devices are suspect, pending more analysis, but it did reveal the manufacturers that were not caught cheating: HTC, Xiaomi, Huawei, Honor, Google, and Sony. That leaves some very big names out.

 

 

 

Article from Android Authority.

  • Like 1

Share this post


Link to post
Share on other sites
Yogurth    2,197

I wouldn't call this cheating on benchmarks as the devices run on full power  when benchmarked as it normally should be, what is curious is that are they slowing down their phones during the regular usage. Could this indicate that Snapdragon heat issues haven't been resolved after all?

Share this post


Link to post
Share on other sites
HawkMan    5,232
1 hour ago, Yogurth said:

I wouldn't call this cheating on benchmarks as the devices run on full power  when benchmarked as it normally should be, what is curious is that are they slowing down their phones during the regular usage. Could this indicate that Snapdragon heat issues haven't been resolved after all?

There's a reason I won't e buying a snapdragon phone again, no matter what.

 

even without the heat issues it has other issues, looking at samsung phones that are sold with both SD and Exynos in different markets. Not only is the Exysnos models in general faster, they are also fast for the phones life, while the SD models gradually slow down over the months to become noticeably sluggy. 

  • Like 1

Share this post


Link to post
Share on other sites
Draconian Guppy    13,037
11 minutes ago, HawkMan said:

There's a reason I won't e buying a snapdragon phone again, no matter what.

 

even without the heat issues it has other issues, looking at samsung phones that are sold with both SD and Exynos in different markets. Not only is the Exysnos models in general faster, they are also fast for the phones life, while the SD models gradually slow down over the months to become noticeably sluggy. 

I dunno,

I've come to think, since samsung doesn't release source, that the exynos android OS is somehow optimized with samsung witchcraft (being they design and make the exynos), because Meizu (who use exynos) never has the same performance as their samsung variants. I mean on a real life basis, not benchmarks. Eg. OS responsiveness and fluidity.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.