What Bill Gates Really Could Do About Spam, Part 2

I started writing this series quite some time ago. I published the first part last month, and I'm finally getting to part two now. Hopefully part three and four will not take as long.

Note: if you're reading this on my blog's home page, the article may appear to be truncated. The article is quite a bit longer than anything else I've ever posted here, and the text must be exceeding the summary data limit. Click here for the full story.

In part one, I wrote about how, just a few days before I was to give a presentation about spam at Lotusphere this past January, Bill Gates had publicly declared that the spam problem would be solved within two years. I then went on to review the three approaches to spam that various news reports indicated that Gates had spoken about: Challenge/Response, Computational Tax, and Postage. (You can find my brief explanations of each of these approaches back in part one.) After some further reading of other reports based on the same Gates speech, I have subsequently found that although they all reported that he had spoken about three approaches, some of the reports included Filtering as one approach, then lumped both Challenge/Response and Computational Tax into a second approach, and Postage as the third. Filtering is a broad topic, and it's pretty well understood, so I won't delve into it an explanation here. I should, however, at least say that Microsoft isn't new to filtering, but they have definitely stepped up their efforts, and I'll have some more to say about that a little later.

It's interesting to note, by the way, what Bill Gates didn't mention as one of the strategies earlier this year: legal action. At least, none of the reports about Gate's speech that I read mentioned it. It is worth noting, however, that Microsoft was pursuing legal action in various countries, in some cases before Bill Gates' January speech, sometimes on their own as a civil matter, and sometimes in cooperation with law enforcement authorities and that they have continued to do so. They just won a $4 million judgement in one spam case. BTW: even though I fall in witht he majority who don't believe that legal action is likely to be of much use in the battle against spam, I am glad that Microsoft is continuing to pursue it in the most egregious cases.

One of the reasons that this series of articles is taking so long to complete is that the landscape in the arena of the anti-spam battle is constantly changing. For instance, much of the industry feels that Challenge/Response (also known as Sender Address Verification or SAV) is a poor approach, and there's an additional complication due to the fact that a company called MailBlocks claims patent rights over the method, but AOL just bought MailBlocks a few days ago, and although many people might really want to disagree with me on this, I think there's substantial evidence that where AOL goes, the Internet follows. (I don't mean to say that AOL is on the leading edge of technology. Not by any means. But they are one of the leading technology popularizers, and due to the size fo their installed base they have a huge influence on the widespread acceptance of new tools. E.g., AIM did far more to bring chat into the mainstream of daily home and business life than any of the chat systems that preceded it, and one could also make the argument that it's no coincidence that both Internet email and the World Wide Web were began their trend toward wide adoption as business tools shortly after AOL started offering them to consumers.) Back to the point: each time the landscape changes, I've had to re-think some details of what I want to say here.

The Radicati flap also had a bit to do with my not getting around to finishing these essays. It was just too much fun to ignore ;-), and there's only so much time in the day for blogging.

I've allowed myself to digress a bit above because in a roundabout way it actually reinforces the major point I want to make in this installment of the series: Bill Gates and Microsoft are adapting to the changes in the anti-spam landscape, too. What they've been saying lately is significantly different from what they were saying at the beginning of the year.

And by the way, lest I create the wrong impression, Gates' statement in January, though by far the most dramatic, was not by any means the first time he had talked about spam. For example, he wrote a letter about spam to the FCC last sprint, outlined Microsoft's plan to combat spam last summer, and he spoke about spam at Comdex last November. Clearly, Bill Gates and Microsoft have been watching the anti-spam landscape and adapting to the changes since well before this January.

Now, let's get to the real point of this installment in the series: what Bill Gates is saying about spam now. He recently delivered an anti-spam progress report. Let's have a look at what he included in his major points... and what he didn't include.

• He did talk about SmartScreen, Microsoft's filtering technology, which is now depoloyed in HotMail, MSN, Outlook 2003, and as a free download add-on for Exchange 2003. He claims "Since Hotmail deployed it six months ago, SmartScreen has been blocking more than 95 percent of all incoming spam — an average of nearly 3 billion messages every day." He didn't mention the false positive rate, but I've heard no rumblings on the grapevine that Hotmail's filters are generating unacceptable false positives.

• He did talk about Exchange Edge Services, which will "incorporate our latest filtering and security technologies, and will enhance our platform for third-party anti-spam solutions". In the same section of his update, he also mentioned protection against directory harvest attacks, which is interesting; and he wrote about the problem of blocking messages from "email servers are misconfigured in ways that allow email to be relayed from outside", but unless he's talking about something more sophisticated than DNS blacklists this is nothing new at all.

• He did mention the Anti-Spam Technical Alliance, the recently created industry group of major ISPs that "endorsed a set of anti-spam best practices for email service providers and large senders ". I tried following the links within the Microsoft site to get to the ASTA policy recommendations, but the links took me in circles. I did, however, find this document on the Yahoo site. Amongst the most interesting parts are those that recommend setting rate limits for outbound email and detecting and quarantining compromised computers. (We'll talk a lot about the latter issue in part 3 of this series.)

• He did mention the Sender ID Framework, which is the result of the merger of Microsoft's previous Caller ID proposal (which was widely disparaged due to the fact that Microsoft was claiming patent rights over Caller ID) and Meng Wong's Sender Policy Frameworkd (a.k.a., "SPF" or "Sender Permitted From"). Sender ID is a DNS-based email forgery detection tool. Enforcement of Sender ID will cut spammers off from one of their most elementary tools: forging the "From" information in their messages. Microsoft has announced that Hotmail, MSN, and microsoft.com will start enforcing Sender ID by October 1st.

• He did mention the Computational Tax approach: "We're also developing ways by which senders unfamiliar to recipients could choose to "qualify" their email in order to guarantee its delivery, such as by demonstrating that their PC performed a special set of computations in the process of sending the email. This would involve an expenditure of computing time that would be trivial for most senders, but would cause a dramatic slowdown in spammers' operations, given the massive volumes of email they send."

• He did mention, briefly, the Challenge/Response approach as well, lumping it in with the Computational Tax approach: "... servers receiving suspect email could reply to the sender with a challenge, perhaps a computational puzzle or one solvable only by a human sender. If the sender responds appropriately, with human interaction or by expending a small amount of computing time, only then would the email gain access to the recipient's mailbox."

• He also did mention legal strategies. Not in detail, as of course that would not be expected of a CEO of a company involved in multiple legal actions: "In March, we joined with other leading email service providers in filing the first major lawsuits under the new U.S. law against hundreds of individuals allegedly responsible for some of the world's biggest spamming operations. And we assisted the federal agencies who, in April, filed the first joint criminal and civil actions against a group of alleged spammers. With another 17 lawsuits that we filed in June, Microsoft's anti-spam enforcement activity has resulted in more than 90 legal actions worldwide. "

• Notably, Gates completely backed off from the Postage concept: "Some have suggested that such systems might open the door for service providers to charge senders a fee for email delivery. We firmly believe that monetary charges would be inappropriate and contrary to the fundamental purpose of the Internet as an extremely efficient and inexpensive medium for communications. The goal instead is to thwart spammers' misuse of the Internet, so that everyone else can continue to enjoy its enormous benefits." Now, contrast that to what CBS News reported that Gates said in January:

  But the most promising, Gates said, was a method that would hit the sender of an e-mail in the pocketbook.

  People would set a level of monetary risk - low or high, depending on their choice - for receiving e-mail from strangers. If the e-mail turns out to be from a long-lost relative, for example, the recipient would charge nothing. But if it is unwanted spam, the sender would have to fork over the cash.

  “In the long run, the monetary (method) will be dominant,” Gates predicted.

  It went from Gates saying that postage will be dominant, to "some have suggested..." That's a nice little attempt to re-write some history. but the point is that postage is off his radar screen, and that's a good thing in my opinion. Postage is a wildly unpopular concept and it's also dependent on development of an secure but efficient micropayment system that people would actually be willing to use -- something that has been one of the holy grail's of the Internet for the past ten years, but so far nobody has managed to come up with.

• Gates did not mention a two year timetable for elimination of spam. He didn't mention any timetable at all. I'm guessing that he heard the guffaws... perhaps not from the audience at Lotusphere when I told them of what Gates had said, but elsewhere guffaws.

In addition to Gates' progress report letter, there's also a recent update to a Microsoft anti-spam web page initially published back in November 2003. I'm not going to review that page point-by-point here. Most of it duplicates and expands upon things covered in the progress report. There are several interesting links on the page. One is a link to a 1998 paper about Bayesian spam filtering from Microsoft Research (PDF here), which was obviously well before the 2002 Paul Graham 'A Plan For Spam' essay that is widely credited as being the catalyst for all the work that has been done with this technique recently. Of course, statistical text analysis had been around since at least 1964, when Mosteller and Wallace used Bayesian analysis to verify James Madison's authorship of the Federalist. That paper is not on line, but there are numerous references to it. The point I'm making here, by the way, is that contrary to what a lot of people might like to think, Microsoft isn't a latecomer jumping on the Bayesian bandwagon, and the same is true of quite a few other companies that have been incorporating statistical text analysis into their anti-spam engines for quite some time.

Well, that pretty much covers what Bill Gates and Microsoft are saying about spam today. Part three, when I finally get to it, will go over what I believe is right and wrong, or good and bad about Microsoft's approach to spam. Part four will cover my suggested plans for Microsoft, and I really hope the writing on this doesn't drag on for too long because the whole point of my going through this exercise is to get these ideas out there. It's not that I think that I've got the perfect plan for defeating spam. First of all, I'm not that crazy. Secondly, I do have some plans... but that doesn't mean I have all the details. The plans I'm thinking about might be of interest to some people who are a whole lot smarter than I am, and who just might be able to see something useful in them. Hey... you never know ;-)

Back To Part One