Freakonomics and Microsoft trade words over 'Bing It On' challenge accuracy

Just over a year ago, Microsoft launched its "Bing it On" challenge, where it asked users to view search results from Bing and Google and pick which offered the best answer to their search queries. At the time, Microsoft said it commissioned an independent study where it claimed Bing search results were preferred by a group of 1,000 people over Google's by "nearly 2-to-1 in blind comparison tests."

This week, Ian Ayres of the Freakonomics website has put Microsoft's claims to the test in his own study. The site questions the number of participants in Microsoft's original study, which suggests that 1,000 people is simply not big enough of a pool to come up with an accurate final result. Ayres also questions why Microsoft does not release the results of its online "Bing It On" challenge, which millions have taken since the site launched.

Freakonomics conducted its own survey, using the online "Bing It On" website. The Ayers said:

We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were “ties”).  This is not even close to the advertised claim that people prefer Bing “nearly two-to-one.”  It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge, if, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate.

This caused Microsoft to post its own response on the official Bing blog. Matt Wallaert, Bing's behavioral psychologist, defended the number of people used for its original study, saying, "A 1,000 person, truly representative sample is actually fairly large. As a comparison, the Gallup poll on presidential approval is just 1,500 people." As far as releasing the data on the Bing It On website, Wallaert says it's not being posted because of privacy concerns. He stated:

It isn't conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it. So we simply don't track their results, because the tracking itself would be incredibly unethical. And we aren't basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific).

Wallaert ended the blog with an invitation for anyone to send him an email for questions about the Bing-Google study and the Bing It On challenge.

Source: FreakonomicsMicrosoft | Image via Microsoft

Report a problem with article
Previous Story

Microsoft Research teams up with Xbox One team to help create Kinect 2.0 camera

Next Story

Microsoft downplays its CES 2014 participation

30 Comments

Commenting is disabled on this article.

Wallaert says it's not being posted because of privacy concerns. He stated:

ERRRRRMMMMM WTF privacy concern really how do you identify someone from a number that contains nothing identifiable about the person or is that it might give away the truth of the mater which MS don't want you to know

I've had multiple customers take the test. Only one picked Google -- most pick Bing and sometimes it even surprises me.

I have also noticed that Google has changed their results since the test first released. The sites are FAR more similar now where in the past Bing generally included more visual results (graphics, additional links, etc.). Now that Google is doing the same thing, it's a lot harder to pick a winner.

It has only been very recently that Bing has started returning accurate results for technical queries.
I am less reliant on Google for these sorts of things now.

I use Bing and most of their apps. I visited the US recently, and Bing was phenominal! Miles better than Google. In Canada it's great too, but not as good as in the U.S. Obviously, all their apps are top notch too.

I hope they work on streamlining their international results and offerings if they truly want to dominate as most people have side, Bing is not up to par in other countries.

I usually use Bing too, but sometimes it really misses the ball and then I'll try google again.
For images Bing is def the winner.

But some simple queries about MS itself sometimes get handled better by google, go figure

I'm using Bing, and I'm happy. When Bing can't find it, I go to Google, only to see that Google can't find it either (or even worse).

Bing just needs to improve their algorithms for gathering information and understanding the search like Google does with comparing things, as well as add a timer and a graphing calculator.

Like many others I use both Bing and Google. I took the Bing-it-On challenge earlier this year. i did not use general entries like the Bing commercial does. I entered local business and library names to see how both did locally. In two cases Bing's top reply was for someplace in a different state. Google got these correct. The others were fine. When using Bing it works perfectly when entering the name of the town a business is in, along with the business name.

I understand why Microsoft won't release the actual web results as everyone has their own style of query entry. The results may be less than the 2:1 claimed.

The system would be really easy to game in either direction though, because it's obvious which result is from Google simply because it has the "cache" link in its responses. Even with a controlled study, some folks could game their answers based on that unless Microsoft makes the look and feel of the responses 100% identical between both Bing and Google.

As such, even though Microsoft made an ethical study with a small pool of people, the results are questionable unless the study took the look and feel of results into account so it was truly a "blind" study.

i guess Ian Ayres of the Freakonomics would read these comments and see that MS was not lying, i use bing also ...

Been using Bing for years now. Bing does not get local results as good as Google, but I still always find what I'm looking for. Good enough is not good enough for most people, they need to improve.

I use both Bing and Google all the time. Sometimes Bing gets it horribly wrong though, which makes me sad because I hate using Google because I disagree with how they track and monetize us as users.

Totally agree with you. That's my feelings exactly, which is even more disheartening since they want us using Bing at work...

(which is horrible, considering even one of the OneNote dev team members claimed to using Google over Bing)