Microsoft gives more info on Hotmail's SmartScreen

In December, Microsoft launched a number of new features for its Hotmail web-based email service. One of those new features has to do with SmartScreen, which previously has dealt with the massive amount of email spam that Hotmail users get every single day. Now in a new entry on Microsoft's Windows Live blog site, the company's Dick Craddock writes about how SmartScreen is used for newsletters as well.

Microsoft discovered that when a person checks their Hotmail inbox, 14 percent of those emails are person-to-person, while two percent are true spam and another two percent are named as "other". The remaining 82 percent of email in a typical Hotmail email box is called "graymail" by Microsoft.

The company decided to go over what kinds of emails are under the graymail category. The final result saw that 50 percent of a Hotmail inbox is with newsletters and deal emails. Craddock says, "Every day the average person’s inbox is flooded with messages from thousands of different retailers, clubs, societies, and schools, or with coupons, deals, and notifications from deal aggregators talking about all the exciting things that people need to be buying, doing, or seeing."

So how does SmartScreen filter out all those newsletter and deal emails? Craddock states:

To get Hotmail to identify newsletters for us, we began by making a list of newsletter characteristics and built a piece of software to extract them from incoming emails. This list forms the model of what makes newsletters different from all other mail and includes three aspects: presence of the List-Unsubscribe header, the sending email address, and what gets shown to the user.

With a clear definition of what we considered a newsletter, we created a reference set of about 10,000 messages that we classified as “newsletter” or “not a newsletter.” Think of the reference set as a test for our newsletter filter: the rate at which it correctly identifies newsletters defines its accuracy.

Microsoft used what it calls machine learning to have Hotmail adjust how it detects all those 10,000 newsletter messages. It then used its own employees to test the system in what it calls "dogfooding": Craddock says:

We provided the dogfood users with a way to report missed and incorrectly identified newsletters just as we do for the occasional spam message that gets through our filters. We spent several weeks analyzing the failures and adjusting the model until we’d worked out the known kinks.

For example, a major problem we identified early on was that financial services businesses tend to send all their mail from the same domain, and often have a lot of boilerplate language that closely resembles newsletters—even though they may not be. Rather than take the risk of filing away your bank statements, we decided it was better to leave these messages alone and trained the newsletter filter to ignore them.

Microsoft's new system now allows the user to sweep or filter all those newsletters that are of interest to them. Craddock says that Microsoft is adding new newsletter categories to better help Hotmail users handle all of that traffic.

Report a problem with article
Previous Story

Walmart offers Lumia 710 for free on contract

Next Story

Rupert Murdoch calls Google "piracy leader"; supports SOPA

28 Comments

Commenting is disabled on this article.

Hotmail/Gmail doesn't matter it's all down to personal taste.

I just wish they would let me have a little control of SmartScreen or whitelist/blacklist email addresses. I have an email from my credit card that smartscreen insists is malicious even though I have added it to contacts and tried all options to say it's safe.

Yes, I still use hotmail. It is linked to my Xbox and GFWL account and has always been for the past 5 years and never have any problems security wise. I always fetch my mail from my hotmail through my heavily modified thunderbird program so I am not worried about the interface though I do check in webmail interface from time to time.

Your are all wrong whining who has the best and prettiest webmail, if your hardcore like me use SMTP on the command line on a Commodore Amiga... you foggots!

Redz0ne said,
Your are all wrong whining who has the best and prettiest webmail, if your hardcore like me use SMTP on the command line on a Commodore Amiga... you foggots!

I think it is you that are the foggot or you would obviously be using squirrelmail.

I dropped Hotmail as my main email years ago, finally stopped logging in a couple of years back..
It was bloated, took for ever to load, options and feature were limited, the spam was horrendous and the security lacking. Unless you got an account very early, all the decent names were taken. Could have been worse though, a lot of my friends still have AOL-mail arghghg

Mike Chipshop said,
It was bloated, took for ever to load, options and feature were limited, the spam was horrendous and the security lacking.

Old hotmail != new hotmail

Do people still use Hotmail?

I created a test account recently and I was horrified at the poor quality user experience.

The original eMail from the early 90's had a far more usable and intuitive user interface.

Brian Miller said,
Do people still use Hotmail?

I created a test account recently and I was horrified at the poor quality user experience.

The original eMail from the early 90's had a far more usable and intuitive user interface.

Yes, it has the most active users than any other email service.

Brian Miller said,
Do people still use Hotmail?

Yup! In fact I've had (and used) my Hotmail account since around 2000. To date the only spam that I get is one of Microsoft's newsletters and even the majority of those are sent straight to the 'Junk Mail' folder.

If hotmail had last logged in IPs like gmail then I would go back to hotmail. Security wise I believe that hotmail is weaker than gmail, and security is most important.

eomas said,
If hotmail had last logged in IPs like gmail then I would go back to hotmail. Security wise I believe that hotmail is weaker than gmail, and security is most important.

You think Gmail is more secure than Hotmail? I can assure you it isn't. Microsoft takes security way more seriously than Google.

Because the only way to 'unsubscribe' from many spam lists is for them to start getting bounces. Its amazing what disabling an account for a couple weeks can do. I don't care how well they filter it if I still have to weed though a junk folder.

I really like the new hotmail. It is far better than gmail imo. I just wish it could be better in keeping out spam. I get about 5 a week but would nice if it was 0

Deihmos said,
I really like the new hotmail. It is far better than gmail imo. I just wish it could be better in keeping out spam. I get about 5 a week but would nice if it was 0

I don't think it is better than GMail. For one you only get an email account when you sign up for a hotmail account. You don't get Docs, Calendar, Photos, G+, ect. While Microsoft may have some of these services, they're not easily accessible like they are when using GMail. Second, unless they completely reworked Hotmail in the last few months the options are horrible. Google has clean and easy to use options for GMail. In addition to that you get all the addons and gadgets which are fun to use and pretty useful. Last of all, GMail's interface is modern and easier to use than hotmail in my opinion.

Deihmos said,
I really like the new hotmail. It is far better than gmail imo. I just wish it could be better in keeping out spam. I get about 5 a week but would nice if it was 0

Gmail has way more features, like mail filtering, than hotmail. A gmail account gets you alot more than a hotmail account to. But to each is own. I have no complaints about hotmail i just prefer gmail.

Beyond Godlike said,

Gmail has way more features, like mail filtering, than hotmail. A gmail account gets you alot more than a hotmail account to. But to each is own. I have no complaints about hotmail i just prefer gmail.

Gmail has not more featuers than Hotmail and Live Services. Hotmail/Live are much more comprehensive. You get Skydrive cloud service that handles all your Office documents and photos. Hotmail offers you contact, calendar and Messenger with Facebook/LinkedIn/MySpace integration. Live again integrate with your Xbox. And If you have a WP7 it integrates with everything above and you get an additional device service to configure and manage your phone. In addition to all that you get Live Mesh which syncs your files between computers and handles remote access to your computers.

Edited by Gungel, Jan 16 2012, 2:14am :

I have not looked into all the features gmail offers. I have a gmail account but I prefer to have Live as my main email. One reason is my last name is 4 letters and gmail does not allow email address with 5 letters. With a Live.com address my email is so nice and short.

Windows Live does offer lots of services.

I use Hotmail as my main email account and to be honest im getting worried with the downright lacking ui that microsoft has lost the plot. Im too lazy to switch and actually worry about Googles bigger plans, but seriously Gmail look at least 5 years ahead. I mean honestly how hard is it to write some css

Iridium said,
I use Hotmail as my main email account and to be honest im getting worried with the downright lacking ui that microsoft has lost the plot. Im too lazy to switch and actually worry about Googles bigger plans, but seriously Gmail look at least 5 years ahead. I mean honestly how hard is it to write some css

there is no need to worry about the UI, it will be updated. It makes perfect sense to improve the foundation then apply the UI. Microsoft is doing a lot right now and trying to make sure all teams are putting quality first.

Deihmos said,
I really like the new hotmail. It is far better than gmail imo. I just wish it could be better in keeping out spam. I get about 5 a week but would nice if it was 0

I like how people are telling you you're wrong even after you said it was YOUR opinion. Anyway, do you maklrk those email as spam?

De.Bug said,

I don't think it is better than GMail. For one you only get an email account when you sign up for a hotmail account. You don't get Docs, Calendar, Photos, G+, ect. While Microsoft may have some of these services, they're not easily accessible like they are when using GMail. Second, unless they completely reworked Hotmail in the last few months the options are horrible. Google has clean and easy to use options for GMail. In addition to that you get all the addons and gadgets which are fun to use and pretty useful. Last of all, GMail's interface is modern and easier to use than hotmail in my opinion.

What????? you get email, contacts, calendar, SkyDrive, Office web apps, Familly safety, Xbox Live, Live mesh, facebook/linkedin integration. Along with the Live suite of apps (live photo gallery, movie maker, messenger, mail, familly safety) - what you smokin dude?

De.Bug said,

I don't think it is better than GMail. For one you only get an email account when you sign up for a hotmail account. You don't get Docs, Calendar, Photos, G+, ect. While Microsoft may have some of these services, they're not easily accessible like they are when using GMail. Second, unless they completely reworked Hotmail in the last few months the options are horrible. Google has clean and easy to use options for GMail. In addition to that you get all the addons and gadgets which are fun to use and pretty useful. Last of all, GMail's interface is modern and easier to use than hotmail in my opinion.

Hotmail gives you those same services and way more than GMail... I guess you haven't been on Hotmail in the last 10 years or so...

De.Bug said,

I don't think it is better than GMail. For one you only get an email account when you sign up for a hotmail account. You don't get Docs, Calendar, Photos, G+, ect. While Microsoft may have some of these services, they're not easily accessible like they are when using GMail. Second, unless they completely reworked Hotmail in the last few months the options are horrible. Google has clean and easy to use options for GMail. In addition to that you get all the addons and gadgets which are fun to use and pretty useful. Last of all, GMail's interface is modern and easier to use than hotmail in my opinion.

1) You get Office web apps and Skydrive, it's right up there at the top of the screen.
2) They completely reworked Hotmail recently.
3) The interface is much cleaner and modern now. Not sure about addons though.