Late last year Ars Technica noticed that some Samsung phones artificially boost performance when running Geekbench 3. This boost inflated Geekbench 3 scores by up to 20%. Since benchmarks are only meaningful when they're treated the same as any other application, we have been working on determining which devices "benchmark boost", and what we should do with results from these boosted devices. I'd like to share what we've discovered.
In order to determine which devices artificially boost performance when running Geekbench we added a "boost detector" to Geekbench 3. The detector embeds a report in each Geekbench 3 result uploaded to the Geekbench Browser. After analyzing thousands of reports we determined that the following Android devices artificially boost performance when running Geekbench 3:
- Samsung Galaxy Note 10.1 (2014)
- Samsung Galaxy Note 2
- Samsung Galaxy Note 3
- Samsung Galaxy S 3
- Samsung Galaxy S 4
- Sony Xperia Z
- Sony Xperia Z Tablet
- Sony Xperia Z Ultra
- Sony Xperia Z1
- Sony Xperia ZL
On both Samsung and Sony devices the boost appeared in Android 4.3. Earlier versions of Android (up to and including Android 4.2.2) did not boost. Anthony Schmieder and Daniel Malea, two Geekbench developers, worked with Ars Technica to find the code responsible for the boost on Samsung devices.
In order to combat benchmark boosting we have decided to exclude results from these devices running Android 4.3 from the Android benchmark chart. This way the results on the chart reflect the true performance, not the boosted performance, of each device. We have also added a list of excluded devices to the chart. We will continue to monitor the detector reports, and we will update this list if we discover other devices or Android versions that apply a benchmark boost.
There is one bit of good news that our detector uncovered — Samsung removed the benchmark boost from their Android 4.4 update. We hope that Sony follows Samsung's lead and also removes their benchmark boost from their Android 4.4 update as well.