Microsoft is not a fan of RoboHornet; still takes first place

Google recently ousted a new benchmarking tool called RoboHornet and Microsoft has wasted no time in degrading the tools value by saying that it does not measure real world performance.

Microsoft loves to remind us that IE10 excels in real world performance and is not designed to win benchmarking awards. The RoboHornet tool, as Microsoft points out in a post on their blog, is targeted at specific aspects of browser performance, rather than the entire picture.

Microsoft also built a website too, that extrapolates the idea that real world performance is not the same thing as benchmark performance. In the video below, Microsoft built a test site and then compared it to Chrome and IE10, and as you would expect coming from Microsoft, IE10 trumped that of Chrome.

Interestingly, even though RoboHornet is a Google built product, IE10 actually takes home the gold for performance rankings, but, for Microsoft, it's not about benchmarks, its about real world perfofmance. What it comes down to is that Microsoft is not a fan of RobotHornet because in their minds, it is a limited test of performance.

This benchmarking tools is only one of many on the market and it will be interesting to see if it gains any weight with developers or if it will be another tool that is mostly ignored. 

Source: Microsoft

Report a problem with article
Previous Story

Barnes and Noble announces Nook Video service

Next Story

Far Cry 3 PC gets official hardware specs

27 Comments

Commenting is disabled on this article.

Considering the results in the video Microsoft displayed I believe they were running chrome with hardware rendering not enabled.

Phalanger said,
Considering the results in the video Microsoft displayed I believe they were running chrome with hardware rendering not enabled.

Chrome has proper hardware rendering? Since when? Dont they do most through WebGL, which is horrible in performance compared to DX10/11?

Microsoft's advantage on hardware acceleration is the utilisation of the Windows Desktop Composition Manager found in Windows 6.x. In other words the operating system decides how much of the hardware is used while the browser only cares for the rendering.

Riva said,
Microsoft's advantage on hardware acceleration is the utilisation of the Windows Desktop Composition Manager found in Windows 6.x. In other words the operating system decides how much of the hardware is used while the browser only cares for the rendering.

Chrome and Firefox are allowed to use DirectX ya know, just like IE.
But Chrome uses WebGL which is absolutely HORRIBLE. Their Google maps WebGL thin is so shakey and laggy, not fun. They need WebGL to make Google maps as good graphically as Bing maps.. thats just sad IMHO.

Shadowzz said,

Chrome and Firefox are allowed to use DirectX ya know, just like IE.
But Chrome uses WebGL which is absolutely HORRIBLE. Their Google maps WebGL thin is so shakey and laggy, not fun. They need WebGL to make Google maps as good graphically as Bing maps.. thats just sad IMHO.

Windows Desktop Composition Manager is a lot more than DirectX
I agree that WebGL is **** though
Only thing Google ever developed is their search engine. Everything is else borrowed.

PROTIP: IE10 prides itself at being fully hardware accelerated browser.

Firefox turns off hardware acceleration for lots of reasons. It is also not as efficient in rendering anything in general. Developers are realizing this now as their new animated tabs fail to run smoothly.

Chrome is mostly CPU. Its hardware acceleration is a joke and doesn't make any difference on or off.

Opera has something resembling HWA except it crashes, leaks memory, and fails at being better than its software rendering.

This should not be surprising to anyone that IE10 does well with this type of benchmark, especially involving graphics.

Chrome is far more CPU bound.
Chrome uses less threading for obtaining, retrieving, composing, rendering, displaying a site.

There is also the curve of performance degradation. Chrome may perform well on a higher end system, but as you move to average or lower end systems, the performance that is CPU bound drops out from under it. IE10 in contrast has fairly consistent performance from a low end system to a high end system, due to the way it utilizes every drop of performance available, even if has to dump work to the GPU or dump GPU work to additional cores.

This is important for end user experience, and for developers wanting to produce high quality HTML5/CSS3/RIA sites. With IE, they do not have to worry about the end user's system performance making the site fail to work. So what you are seeing today is site development that designers are able to provide content to all IE10 visitors, and for Firefox, Safari, and Chrome, they are having to gauge system performance and are delivering various incarnations. So the Netbook users running Chrome gets a different site than the i7 user using Chrome, yet the IE10 user gets the same, and richer quality site on both systems. (Sadly this scale of performance is beginning to create more work and fragmentation for web developers that want to use more advanced HTML5 features.)


Anyone that is curious as to how and why IE9/10 works differently, go find the building IE9 blogs and videos at channel9.

IE9 was not only a complete revamp of the trident engine, it also was a major shift in how web content is treated by an engine. Starting with IE9, the concept of what the engine did went from being a document rendering model to a runtime model.

Starting with IE9, all web content is treated like 'code' instead of a 'document'. So instead of 'displaying' the web content, IE9 essentially compiles and runs the web site. This is what they mean when they say it runs on 'metal/hardware', as web sites are kind of going through a runtime compiler.

This paradigm shift has a lot of advantages, from handling dynamic content to threading out portions of the site easier to less redundant code running and a direct line of access to the rendering mechanisms of the GPU and GP-GPU operations as available.

Chrome and other browsers are still stuck on the older model, where their engine is a document display technology, and an application trying to display interpreted content will never match the performance of a runtime compiler letting the content 'run'.

Anyone that has looked at WinRT/Modern development, should see the correlation of the IE engine embedded in the Modern development framework. This is how and why HTML5 Apps on Windows 8 can run a near native code speeds, as they are not being hosted by a document viewing engine, and instead are being hosted by a runtime compiler.

+1 for the walltext.

Anyways, most (+99%) of the javascript codes is about visual control (accessing and modifying the DOM) because it is the real objective of javascript.

For example, jquery (a javascript library) is all about accessing the DOM.

The animation is smooth on Chrome 23; 16.408 seconds. On Firefox 16 it runs in 4.368 seconds but the animation is crappy. Doesn't work on IE9.

Sly_Ripper said,
The animation is smooth on Chrome 23; 16.408 seconds. On Firefox 16 it runs in 4.368 seconds but the animation is crappy. Doesn't work on IE9.

Since it requires IE10, I bet it doesn't work on IE9.

Sly_Ripper said,
The animation is smooth on Chrome 23; 16.408 seconds. On Firefox 16 it runs in 4.368 seconds but the animation is crappy. Doesn't work on IE9.

Well hello, Captain Obvious!

The point of RoboHornet is that it does reflect real world performance not arbitrary marks like SunSpider, Octane and the like. Not sure what Microsoft is grumbling about.

mrbester said,
The point of RoboHornet is that it does reflect real world performance not arbitrary marks like SunSpider, Octane and the like. Not sure what Microsoft is grumbling about.

No you got th at wrong, the point of a google created benchmark is "proving" that Chrome is faster then other browsers. MS kicked their asses for this benchmark and used their #1 position to let people know that benchmarks are USELESS.
Who gives a f*cking sh*t about 0.1second faster loads? its all a bunch of bull that works under the "If you repeat that bullsh*t often enough, people will believe it" idea.
Chrome, IE and FF are quite on par when it comes to loading pages in real world times. They're counting the differences in goddamn milliseconds.

That's cool. Now get some decent addons for your browser then I'll consider switching. Speed is nice, but I would like replacements for my current addons before I ever switched from Chrome... and fix your UI. I'd like to have my favorites bar visible, but not if it's going to look like a completely different theme than the glass.

KSib said,
That's cool. Now get some decent addons for your browser then I'll consider switching...

I've always found it interesting that MS, a company with substantially more resources and arguably "talent" than say Mozilla, comes up with such horrible designs for browser extensibility.

There's basically no plugin model for IE that makes it easy to create them. It's better now, than in previous versions but it is comparatively difficult (http://www.add-in-express.com/...12/07/09/create-addons-ie10).
In IE10 RT, they're actually going in the opposite direction by making it completely plugin free.

Edited by ahinson, Sep 25 2012, 5:28pm :

ahinson said,

I've always found it interesting that MS, a company with substantially more resources and arguably "talent" than say Mozilla, comes up with such horrible designs for browser extensibility.

There's basically no plugin model for IE that makes it easy to create them. It's better now, than in previous versions but it is comparatively difficult (http://www.add-in-express.com/...12/07/09/create-addons-ie10).
In IE10 RT, they're actually going in the opposite direction by making it completely plugin free.

Which is done for security since plug-ins by the way they work open up the browser and system to attack. Let's not forget all the problems we have/have had thanks to flash and it's bugs/exploits etc. In the RT world, at this point, they've got things pretty locked down and I'd say that's the best way to do it seeing as it's the platform for the non-techie consumer going forward who likes their shinny mobile devices/tablets etc.

KSib said,
That's cool. Now get some decent addons for your browser then I'll consider switching. Speed is nice, but I would like replacements for my current addons before I ever switched from Chrome... and fix your UI. I'd like to have my favorites bar visible, but not if it's going to look like a completely different theme than the glass.

You can turn on favorite bar to be always visible on IE10

GP007 said,

Which is done for security since plug-ins by the way they work open up the browser and system to attack. Let's not forget all the problems we have/have had thanks to flash and it's bugs/exploits etc. In the RT world, at this point, they've got things pretty locked down and I'd say that's the best way to do it seeing as it's the platform for the non-techie consumer going forward who likes their shinny mobile devices/tablets etc.


This is why an add-on/plugin should run in a sandbox, which is scoped to browser functionality only. I know bugs will always exist, so there may always be the potential for exploits, but that aside. I firmly believe both Mozilla and Google understand the need for a solid plugin/add-on system.

For years, MS continued to push the insecure ActiveX extension framework for IE, meanwhile, Firefox had moved to a Javascript based plugin model and exposed large portions of the DOM, CSS and other browser functions to add-ons. This and this alone was a huge factor in many switching browsers.

GP007 said,

Which is done for security since plug-ins by the way they work open up the browser and system to attack. Let's not forget all the problems we have/have had thanks to flash and it's bugs/exploits etc. In the RT world, at this point, they've got things pretty locked down and I'd say that's the best way to do it seeing as it's the platform for the non-techie consumer going forward who likes their shinny mobile devices/tablets etc.

Chrome is insecure?

There is a difference between a plug-in like Flash, and an extension like those used in Firefox and Chrome.

thomastmc said,

Chrome is insecure?

There is a difference between a plug-in like Flash, and an extension like those used in Firefox and Chrome.


Well considering Google's own supported research in browser vulnerabilities earlier this year showed that IE9 beats Chrome when it comes to exploits and serious security flaws.
So if IE is insecure, what makes that Chrome?

ahinson said,

This is why an add-on/plugin should run in a sandbox, which is scoped to browser functionality only. I know bugs will always exist, so there may always be the potential for exploits, but that aside. I firmly believe both Mozilla and Google understand the need for a solid plugin/add-on system.

For years, MS continued to push the insecure ActiveX extension framework for IE, meanwhile, Firefox had moved to a Javascript based plugin model and exposed large portions of the DOM, CSS and other browser functions to add-ons. This and this alone was a huge factor in many switching browsers.


What need? Your personal need? the hell does the general population use for plugins? NONE.
And most who actually use plugins, use noscript or ABP. Both have been default functions for IE since IE7. (well before Chrome even existed)

So again tell me, what need for plugins? Most is utter junk barely anyone ever uses. Plugins are heavily overrated. Find an average computer bobo's FF or Chrome and tell me how many plugins he got.

Oh offcourse, people switch browsers because of the underlaying plugin architecture. Really?
How come people switched to chrome in the beginning? Its plugin system was FAR worse then IE's untill version 10-15 or something.

Shadowzz said,

What need? Your personal need? the hell does the general population use for plugins? NONE.
And most who actually use plugins, use noscript or ABP. Both have been default functions for IE since IE7. (well before Chrome even existed)

So again tell me, what need for plugins? Most is utter junk barely anyone ever uses. Plugins are heavily overrated. Find an average computer bobo's FF or Chrome and tell me how many plugins he got.

Oh offcourse, people switch browsers because of the underlaying plugin architecture. Really?
How come people switched to chrome in the beginning? Its plugin system was FAR worse then IE's untill version 10-15 or something.

You might want to take a look at the number of add-ons that exist for both FF and Chrome and then compare them to those available to IE (http://www.iegallery.com/Addons). IE has some but the add-ons are very limited in number and function.

Content is what drives acceptance, creating add-ons is a way of creating content by substantively changing the experience. Regarding the average "bobo" (wtf is a bobo) not using add-ons and that add-on content is bad, and not due the effort, well that's your opinion and you're entitled to it. Also, remember, early adopters are generally technically savvy. They command an influence of their less savvy family and friends. If they switch who may follow?

You make it seem like this is a new thing, and users have only just begun switching from IE to something else. It in fact has been going on for years, and the reasons are quite varied (http://www.scottberkun.com/blo.../why-i-switched-to-firefox/).

Shadowzz said,

Well considering Google's own supported research in browser vulnerabilities earlier this year showed that IE9 beats Chrome when it comes to exploits and serious security flaws.
So if IE is insecure, what makes that Chrome?

Sounds like Google has quite a proactive approach to finding and patching that many threats. They put a huge incentive for people to find threats. The more bugs found the more hardened the software becomes. Maybe Microsoft should put out a bounty because it seems to be working for Google.

"Google recently outed a new benchmarking tool called RoboHornet and Microsoft has wasted no time in degrading the tools value by saying that it does not measure real world performance."

Is Google and Microsoft muddled up here? The video favours IE and it's running on microsoft.com.