main

Apple accused of cheating over G5 benchmarks?

Daniel Fleshbourne   on 24 June 2003 - 16:28 · 54 comments & 1750 views

Advertisement (Why?)
Benchmark results cited by Apple at the launch of its Power Mac G5 desktops yesterday have already come under fire for seeming to not only tweak the Mac test system to improve its performance beyond anything an ordinary user might experience, but crippling rival systems to deliver below-par average user performance. The tests described by Apple CEO Steve Jobs were conducted on the company's behalf ("under contract") by VeriTest. The benchmarks used are SPEC CPU 2000 integer and floating-point tests. Apple asked VeriTest to compare a pre-release a dual 2GHz Power Mac G5 with a Dell Precision 650 workstation based on twin 3.06GHz Intel Xeon CPUs and a Dell Dimension 8300 based on a 3GHz Pentium 4.

The Dell's were running Red Hat Linux 9.0, the G5 Mac OS X 10.2.7. The test software was compiled using GCC 3.3 and NAGware Fortran 95. VeriTest recorded SPECint base score of 800, 889 and 836 for the G5, 8300 and 650, respectively. The equivalent SPECfp base scores were 840, 693 and 646. So the G5 out-performs the other machines, yes?

Well, so says Apple, but a closer look at VeriTest's documentation, freely available from its web site, suggests otherwise. Certainly SPEC figures published on the SPEC web site do, as Register readers noted, along with readers at a number of web sites today. The corresponding SPECint and SPECfp base Dell-provided results for the 650 are 1089 and 1053. Equivalent figures for the Dimension 8300 are not available.

View: The full story
News source: The Reg


AIMutation features:

  • Plug-ins that add new and exciting features to AIM.
  • Transparency on Windows 2000/XP.
  • Make transparent windows opaque when they're active.
  • Modify the Login Logo with an image of your choice.
  • Use Login Logos from skins.
  • Log IM conversations
  • Add start and finish timestamps to logs.
  • Group IM windows into a single tabbed interface.
  • Flash IM windows when you receive messages.
  • Fast and convenient tab switching through Ctrl+Tab keyboard shortcut.
  • Set IM windows to be 'Always On Top'
  • Set 'Always On Top' option automatically for all IM windows.
  • Set transparency level automatically for all IM windows.
  • Skin AIM with your own graphics!
  • Hide the ad window on the buddy list.
  • Hide the various buttons on the buddy list.
  • Cloning.
  • And more to come soon!

Post a comment · Send to friend Comments · There are 54 additional comments
#1 username on 24 Jun 2003 - 16:30
AMDZone had a good article about the benchmarks
http://www.amdzone.com/articleview.cfm?art...eid=1296&page=2
i don't know where Apple got some of those numbers
(3 replies) #2 nullie on 24 Jun 2003 - 16:38
Uh, it's not cheating, the configuration info used for the benchmarks has been openly available from the beginning. Apple never tried to hide anything.

Last edited by 30000 on 24 Jun 2003 - 17:21
#2.1 threedaysdwn on 25 Jun 2003 - 02:56
I hope you're kidding. Please tell me you're kidding.
#2.2 nullie on 25 Jun 2003 - 14:35
Apple: These are the results, here's the configuration used to get the results.

How hard is it to understand?
#2.3 threedaysdwn on 25 Jun 2003 - 22:19
On their page they list the comparison as being between a "Dual 2Ghz G5" , a "Intel Xeon with HyperThreading" and a "Intel Pentium 4 Xeon 3.06Ghz with Hyperthreading."

But yet, they disabled HT in at least in 2 of the 3 benchmarks that the G5 wins (it was enabled in the benchmark that the G5 lost to both competitors in).

There's a word for that: "misleading"
(1 reply) #3 aStRaLgOd on 24 Jun 2003 - 16:38
*Takes a big breath* *Laughs endlessly*

Ok, who's gonna cheat next? Compaq? Dell? HP? lol
#3.1 mcguirexn on 24 Jun 2003 - 21:04
compaq is no more its hp
(2 replies) #4 JaggedFlame on 24 Jun 2003 - 16:43
I think they should stop worrying about who makes the absolute fastest computer and just focus on user experience. Who cares which one is fastest? It's not as if you're going to notice it all of a sudden. I can barely tell the difference between an Athlon XP 1700+ system and the latest one out now.
#4.1 kairon on 24 Jun 2003 - 17:13
This may make sense in the PC world, but with PC users constantly whining about how "slow" Apple is, it only makes sense for Apple to raise the clock speeds.Of course, now they will have some other excuse to use. Doesn't it just suck how people push their type of computer onto other people?
#4.2 psw on 24 Jun 2003 - 20:41
because it sells and apple needs to sell !
(7 replies) #5 Neobond on 24 Jun 2003 - 16:49
It doesn't surprise me that Apple or any other company for that matter markets its own product using the more favorable results while ignoring other aspects of the test that didn't shine too well on them.

Its a fact of corporate marketing nothing more!

People in the real world (like independant benchmarking sites) convince me what is good and what isn't

Its also funny to note that while Apple were marketing stability last time round (ie: the Megahertz myth etc etc) now the seem to be focusing on speed themselves, ah ho hum.
#5.1 kairon on 24 Jun 2003 - 17:10
How do you propose they speed up their chips without making them clock faster them?You mention the megahertz myth, yet you can't seem to comprehend that megahertz are a big performance booster in CPU. And even though the chips are at 2 ghz, they still top a higher clocked Intel (and definitely AMD!). Seems like the megahertz myth is still in effect to me.
#5.2 username on 24 Jun 2003 - 17:24
you are missing the point kairon, Apple posted SPEC P4 and Xeon benchmarks that were A LOT lower then what SPEC themselves had when they tested the same setups



Last edited by 11321 on 24 Jun 2003 - 17:35
#5.3 aeons on 24 Jun 2003 - 17:27
I think what Neobond's point was, Apple was bitching about the megahertz myth, now they are using the flip side to that, saying because the mhz is higher, thier chips out perform Intel. I'm saying they need to pick a side of that argument, they can't flop back and forth when it suits them.
#5.4 macrosslover on 24 Jun 2003 - 18:39
well in Apple's defense (and i'm not particularly an apple fan) they don't have faster chips than Intel or AMD clock for clock, however there chips seem (ACCORDING TO APPLE sponsored tests let's remember that) to perform better than anything in the x86 world in most of the tests at a lower clock speed. so it kind of proves there mhz myth, because they are saying looking we don't have a fast clock for clock chip compared to them, but our chip is so great it performs the best, so you really don't need a 3ghz chip to do everything the fastest.
#5.5 Jstphish on 24 Jun 2003 - 19:07
Apple has more efficent chip architecture and slower clock speed while the PC has a less efficent chip architecture but compensates by beefing up it's clock speed. It is as simple as that.
#5.6 threedaysdwn on 25 Jun 2003 - 02:28
That's what apple would have you believe, Jstphish.

But it's BS.

The G5 is far slower at 2Ghz than a P4 or Athlon at ~3Ghz or 3000+ rating. All benchmarks have shown that.

Apple's benchmarks didn't include AMD chips, and the Intel chips they used were outdated (533mhz bus variant), on an outdated platform, with SSE/SSE2 and HyperThreading DISABLED!!! But did they disable AltiVec to make it "fair"... they sure as hell didn't. In fact, they ran the P4/Xeon tests with a crippled compiler, and ran the G5 tests with a perfectly tuned AltiVec-designed compiler setup. They even tweaked prefetcher and memory access settings in ways that *couldn't work in the real world*.

What's worse, they ran their tests using a hacked, single-threaded library specifically to inflate their test scores. A library that defeats the purpose of the benchmark because it can't work in a real-world environment.

Their benchmarks are intentionally misleading and by a very, very large margin. It's one thing to leave out the Athlon and Opteron, knowing that they both make the G5's performance look anemic. But to include "fixed" scores and use that as the basis for claiming the "Fastest desktop PC in the world" is ludicrous. I'm surprised no one has taken legal action against Apple for this yet. But I guess you can't sue someone for assuming their consumers are dumb.
#5.7 floyd_thedroid@yahoo.com on 25 Jun 2003 - 17:34
What would you have us believe, threedaysdwn?

SPEC benchmarks are for testing specific parts of chips. When you're testing for integer performance you don't usually use SSE (or AltiVec for that matter) and HyperThreading shouldn't help all that much for a single process which should already be optimized to fully use the chip. Yes, optimized. Code such as this isn't usually just compiled from C, it's tweaked at the machine code level to fully use whichever chip it's run on. This tweaking is why you should never bother with benchmarks, they really don't mean anything, unless all you do is run benchmarks.

Forums about benchmarks/speed always remind me of when I was about five, listening to some of my friends talk about how great their fathers were. They went back and forth, making more and more ridiculous claims until one person said that his father could flap his arms and fly. I guess things never change.

Anyway, threedaysdwn, all your posts are quite venomous (legal action?), why?
#6 midsummerstorm on 24 Jun 2003 - 17:21
Well, I think it's fair to use the best configuration possible to get the best test results for your machine, but what's really funny is how they downgraded the Dells (no SSE2, half-disabled HyperThreading, sub-par compiler) just to dis them, and gain public acclaim. Man, the apple is getting rotten...
(2 replies) #7 nullie on 24 Jun 2003 - 17:26
QUOTE
sub-par compiler
GCC is an excellent compiler, it's also one of the only crossplatform compilers. It wouldn't have been very logical to use something like the Intel compiler, because it's well known as a cheat compiler (it lowers fp precision, does very shoddy math, etc, to gain speed), and it's not used by real world developers (the code it generates is very buggy and unstable.. it's only useful for testing, and maybe the enthusiast that doesn't give a f**k about anything but speed).
#7.1 threedaysdwn on 25 Jun 2003 - 02:35
Actually the GCC compiler wasn't the problem. The Fortran compiles are the majority of SPEC's Rate tests. The compiler they used (NagWare) was fine-tuned for the G5, and known to be complete crap on Intel platforms.

Calling Intel's compiler "shoddy" is ridiculous. Many, and I mean many, developers rely on that compiler. It sees far more use on programs for the Windows platform than GCC does. GCC is known for its lack of x86 optimizations.

They used a hacked, single-threaded Malloc library that is completely unusable in the real-world. Apple altered the G5 platform by enabling "Memory Read Bypass" which allows the G5 to run the benchmark without really completing all the requested work.

The fact that they disabled SSE2 and HyperThreading, yet clearly identify the chip as a "Pentium 4 3.06Ghz with HT" is appauling.

Mac users above all should be furious, they're the ones being played!
#7.2 nullie on 25 Jun 2003 - 14:29
QUOTE
The compiler they used (NagWare) was fine-tuned for the G5, and known to be complete crap on Intel platforms.
Lets see some proof/numbers to back this up please.. hopefully from an independant third party that has conducted their own comparison/tests.

QUOTE
Calling Intel's compiler "shoddy" is ridiculous.
Ridiculous to a n00b like yourself that doesn't know what they're talking about.

QUOTE
Many, and I mean many, developers rely on that compiler.
Very few real world developers use the Intel compiler for anything but benchmarking and testing.

QUOTE
It sees far more use on programs for the Windows platform than GCC does.
Really? I'd like to see your proof to back this up.

QUOTE
GCC is known for its lack of x86 optimizations.
GCC is actually well known for having great x86 optimizations.

QUOTE
They used a hacked, single-threaded Malloc library that is completely unusable in the real-world.
This is the only thing that's really questionable about the comparison, although it's quite possible the G5 will ship with some kind of enhanced malloc library.

QUOTE
Apple altered the G5 platform by enabling "Memory Read Bypass" which allows the G5 to run the benchmark without really completing all the requested work.
Apple didn't modify anything, the configuration was set up by Veritest. Also, Apple has already stated that all the modifications made by Veritest (except for possibly the malloc) will ship enabled in the final version of the G5.

QUOTE
The fact that they disabled SSE2
SSE2 was enabled per the GCC documentation.

QUOTE
and HyperThreading
HyperThreading was disabled because it degrades performance (in benchmarks, anyway). Apple has offered to have the tests redone with HT, but stressed that it'd only make themselves look even better.

QUOTE
yet clearly identify the chip as a "Pentium 4 3.06Ghz with HT" is appauling.
HyperThreading/HT isn't mentioned anywhere on Apple's G5/benchmarks page. The Xeon and P4 are identified as "3.06GHz Xeon" and "3GHz Pentium 4".
(4 replies) #8 vetDazzla on 24 Jun 2003 - 17:36
Apple cheated? So can we say Intel and AMD cheated when they submitted their spec scores using amazingly optimised compilers?

GCC is a cross platform compiler and if it doesn't perform as well in the SPEC tests as the highly customized compilers then tough.

Ultimately the SPEC scores aren't nowhere near as important as real world scores (of which there are none at all), because each vendor optimises it to their own accord. The x86 platform being as experienced as it is has a higher level of optimisation.

Point is, it seems to be almost level pegging across the board in regards to power, with no-one "owning" the other and all performing amazingly well. Price/performance ratio is still squarely in the x86 court, especially with the 3.2GHz P4 outperforming the 1.8GHz G5 and being cheaper, but for me OS X offsets this price difference; for others it doesn't. Each to their own.

Roll on the Athlon 64 so we can bring 64bit computing really into the mainstream.
#8.1 macrosslover on 24 Jun 2003 - 18:44
even with the Athlon 64, 64 bit computing is still a ways off from mainstream. if you look at AMDs roadmap they don't expect a huge amount of 64 bit software until like 18 months to 2 years from now. the only thing it can do for is help the Mac side, because Apple can force them to make all their new software 64bit and "force" people to upgrade to 64bit chips to use this new "exclusive" software.


but honestly with their marketshare it probably won't be such a big hit to the MS/Intel side of things. it's not like many professionals let alone consumers are saying where the hell is my 64bit software.
#8.2 Jstphish on 24 Jun 2003 - 19:13
QUOTE
it's not like many professionals let alone consumers are saying where the hell is my 64bit software.

Good point macroslover.
#8.3 vetDazzla on 24 Jun 2003 - 19:23
Perhaps I should have clarified what I meant. I meant it's going to bring it more into the mainstream. And by mainstream I don't mean granny and grandpa browsing the net and checking their email because 64bit means jack **** to them.I'm talking about high end 3d and 2d work where the 4gb barrier really is a barrier. The 1.6GHz model ruins this because it only has 4 dimm slots, but for the other 2 the prospect of being ables to adress 8GB is very valuable.
#8.4 threedaysdwn on 25 Jun 2003 - 02:37
I still can't believe it's limited to 8GB...

Most 64-bit platforms are "limited" to addressing 4 TeraBytes of memory.
(1 reply) #9 Colonel Sanders on 24 Jun 2003 - 17:55
isn't the G5 (PPC 970) built off the Power4 processors? Then why is the power4 so high. This just seems all speculation. Not to mention these tests come from AMDZone (bias?).
#9.1 threedaysdwn on 25 Jun 2003 - 02:39
The tests don't come from AMDzone. They come from SPEC themselves.


The Power4 is so much higher because it is actually 4 chips in one. The PPC970 is a crippled single-core Power4 with AltiVec (Apple's crappy SIMD engine) thrown in.

It also has far smaller caches than the Power4. Considering the Power4 is by far the most expensive chip on there (costing at least more than a 4-way Opteron or 2-way Itanium 2), it damned well better hold its own.
(3 replies) #10 notbychoice on 24 Jun 2003 - 18:14
hahaha of course they cheated

do they honestly want users to believe that now with the revolutionary G5 (just like the 4 others before it), the mac now ACTUALLY outperforms a pc at 1/3 the price? of course not. pc outperforms mac anyday, the pc has tens of thousands more software available, hundreds more hardware companies competing and producing better cheaper products. oh.. and you can upgrade them as neccessary.
#10.1 Scorched on 24 Jun 2003 - 20:24
But they're so pretty.
#10.2 mcguirexn on 24 Jun 2003 - 21:06
so well said! agreed
#10.3 Mgz on 25 Jun 2003 - 10:39
DID you MARRY A MAC,Scorched ?


GAY MARRIAGE IS LEGAL HERE IN CANADA,but I am not sure about a Mac
#11 GaMMa on 24 Jun 2003 - 18:41
I don't think Apple is using the right marketing campaign, I mean just saw a post about this at slashdot.org and this is going to give the G5 a ton of negative attention, something this company doesn't really need. They are basically telling the public "look we have the best computer in the world, we finally beat the x86 architecture, join us while we take on Microsoft with and beat the x86 architecture." I read somewhere Sun tried this approach with their CPUs and look where they are today .
#12 Jstphish on 24 Jun 2003 - 19:09
In the end you can't trust anyone. Point taken?
#13 Gweedo on 24 Jun 2003 - 19:13
Is it just me, or does everyone seem to be cheating lately?
(2 replies) #14 OrangeSoul on 24 Jun 2003 - 20:23
Apple G5 is the fastest computer when it comes to getting things like photo editing, image processing, movie making and gaming. didnt you see the G5 video. they were boasting the speed of products in the OS because of G5. not overall
#14.1 g33kb0y on 24 Jun 2003 - 21:11
Doesn't it defeat the purpose to say "The world's fastest super computer" if you're not the overall fastest computer?

Hey, look! We're the fastest at loading Photoshop, and Quarx, therefore we've got the fastest computer in the WOOOOORRRRRRLLLLLD.
#14.2 threedaysdwn on 25 Jun 2003 - 02:44
I find this amusing. "didnt you see the G5 video. " he says.

Sigh... when will they learn.

As a video editing student I can promise you that Apple has long lost the video editing market to much bigger and better players.

Sure, iMovie might be great for home videos, but for *real* editing, you use a PC. You buy any of Discreet's editing packages (Fire, Inferno, etc.), they come on a PC.

Everything from SGI is basically a PC. Avid's best software is now on PCs only, and when they do offer cross-platform support, performance ALWAYS favors the PC.

The G5 isn't going to change that. Sure, it's a nice step up for Mac users. But no self-respecting video editor is going to switch to a Mac "because they saw the G5 video."
(2 replies) #15 divertom15 on 24 Jun 2003 - 20:26
it doesnt really matter if you have invented the fastest ever desktop pc you need more software backing it than the mac currently has...
#15.1 Martog on 24 Jun 2003 - 20:45
Not to mention tons and tons of useless software.
#15.2 mcguirexn on 24 Jun 2003 - 21:08
I think this is the point:
Who needs Dual-PowerPC 2GHz processors and this much ram to run IMOVIE???
#16 g33kb0y on 24 Jun 2003 - 21:20
I'm just glad I'm not the only one that can see through Apple's ploys. Don't misconstrue that, I never said that other companies don't lie, steal, cheat to increase market share. But, Apple, the company that is considered wholesome and qualitative, has been doing more lying, stealing, and cheating that meets the eye.

I get offensive with Apple really quick when I see zealots come out to protect it so blindly. I like Windows, but I bash Microsoft left and right - especially when it comes to the IT side of things. I do the same to Linux, Unix, Sun, Cisco, Intel, AMD, Cyrix, VIA, Nvidia, ATi, etc..etc..etc....

We have one Apple server where I work. We only have it because we have a graphics teacher who wants nothing but macs in his department. So, for remote storage, we purchased a Mac G4 Server with an external drive. It's cost was enough to purchase two PC servers of equivalent speed and storage. I doubt we'll be running out to purchase a new G5 computer - server OR personal computer. Not when we could buy one he** of a nice 1.8ghz Opteron server system for less.
#17 Mgz on 24 Jun 2003 - 21:23
If G5 is THAT GOOD, why Apple cheat ?

--> pirce of crap BTW
(3 replies) #18 TheDeputy on 24 Jun 2003 - 22:25
All I can say is if these are true and apple does have this fast machine. THEN the PC Market lets take INTEL for example better get off their asses and produce something better. It's about time someone came out with a 64 BIT processor that will be available to the public at a more resonable price. WAKE UP INTEL and get on it. It's sad when Apple comes out with something before the PC.

Sad day it is.
DOWN WITH APPLE
#18.1 macrosslover on 24 Jun 2003 - 22:46
technically Intel has had a 64bit chip for a while now, tell me why exactly do you need a 64bit chip?? why does the average consumer need a 64bit chip?? my current 32bit chip handles everything i need it to do just peachy...why do i need a 64bit chip???
#18.2 mealbundy on 24 Jun 2003 - 23:34
My friends said the same thing to me when I decided to buy my 2nd PC, a Pentium 90 system over a 486DX200. And that was a 16bit vs. 32bit fight! They asked me why i needed so much power. I got another good 3 years from it while their 486 died the year i bought it.
#18.3 threedaysdwn on 25 Jun 2003 - 02:54
16-bit vs. 32-bit and 32-bit vs. 64-bit is NOT that same thing. Not by any stretch of the imagination.

And the 486 was not a 16-bit chip. The 386 was the first 32-bit chip.

The switch to 32-bit that everyone remembers was really the switch to 32-bit Protected Mode operation. It was the ability to address more than 640k of "conventional memory." It was the ability to run the kernel, the rest of the OS, and individual programs in their own protected memory spaces. DOS 3.5 was the first OS to start doing this.

Moving to 64-bit offers a few advantages, but they mostly, if not completely, apply to high-end workstation and server use. A desktop user going from a 32-bit Athlon to a 64-bit Athlon with an otherwise identical design will notice no difference. Even with a 64-bit OS, the difference will be completely negligible. It won't be more stable, and it won't be noticeably faster.

The reason Opteron (and other future Hammer chips) perform so well is not because they can execute x86-64 code. It's because they have on-chip memory controllers, an incredibly balanced pipeline that is always fed and always makes use of its 800mhz (and soon higher) system bus. They have big, smart, fast caches. They have great branch prediction and comparatively light penatlies for mispredicts. It's simply a better chip than its predecessors.

It works wonders in SMP configurations because it is the first solution to offer each chip its own completely independent bus, using HyperTransport. Sound familiar? The G5 works in a similar way.

The move to 32-bit registers "back in the day" was also improtant. But nowadays most chips (P4 and Athlon included) already have 128-bit SIMD registers!

It's certainly true that there's no disadvantage to moving to 64-bit platforms. There will be *some*, if unnoticeable improvement. It's the natural evolution of processor design, but it's not *needed* on desktops to any degree. That's why Intel is sticking to its 32-bit guns for a couple more years. They figure, and rightly so, that desktop users won't be needing several GBs of RAM anytime soon.

Until then, they'll scale the Prescott P4 (or P5 as it's apparently being called now) to 5Ghz and beyond, offering competitive performance for sure.


#19 razar on 24 Jun 2003 - 22:30
You people are sick... What, are these companies(all companies) paying you??? No. So let them defend there own damn overpriced computers, and wares.
#20 blackice912 on 24 Jun 2003 - 23:23
Right when I saw that news title I started laughing my ass off
(1 reply) #21 aleni on 25 Jun 2003 - 10:41
what can u expect from apple? just a nice gui, nothing else.
#21.1 TheDeputy on 25 Jun 2003 - 14:43
Yeah their 64Bit processor can process there nice GUI faster. WOOPIDYDOO!
#22 macrosslover on 25 Jun 2003 - 15:25
i still think this is bogus until some non Apple sponsored 3rd party verifies/ or does their own results. but of course the Mac faithful will believe anything Apple says. they do their tests in god knows what kind of optimal "unrealstic" conditions.


Apple fans tell me when's the last time you got 5 or 6 hours of battery life on your ibook/powerbook?? Apple claims it can do it, but i've never seen it done. i had an ibook with the battery fully charged and nothing running (like IE, dvd etc...) just the operating system and i let it sit there and it only got 2 hours. now if it does that just sitting there, i HIGHLY doubt it can get 6 hours if you actually do some work on it.

Apple can claim whatever they want, i'll believe somebody not paid by Apple anyday.
#23 Colonel Sanders on 25 Jun 2003 - 15:39
I got 4 and a half hours out of my ibook, and I was writing a word document.

Commenting has either been disabled on this article or you are not logged in. Click here to login or register, its free!

Note: Anonymous commenting is disabled in order to keep the quality of responses to a high standard.

Advertisement (Why?)