page207
the future of iNterNet immuNe systems
185
The Future of Internet Immune Systems
(Originally published on
InformationWeek
’s
Internet Evolution (www.
informationevoution.com), November 19, 2007.)
Bunhill Cemetery is just down the road from my flat in London.
It’s a handsome old boneyard, a former plague pit (“Bone hill”
— as in, there are so many bones under there that the ground
is actually kind of humped up into a hill). There are plenty of
luminaries buried there — John “Pilgrim’s Progress” Bunyan,
William Blake, Daniel Defoe, and assorted Cromwells. But my
favorite tomb is that of Thomas Bayes, the eighteenth-century
statistician for whom Bayesian filtering is named.
Bayesian filtering is plenty useful. Here’s a simple example
of how you might use a Bayesian filter. First, get a giant load of
non-spam emails and feed them into a Bayesian program that
counts how many times each word in their vocabulary appears,
producing a statistical breakdown of the word-frequency in good
emails.
Then, point the filter at a giant load of spam (if you’re having
a hard time getting a hold of one, I have plenty to spare), and
count the words in it. Now, for each new message that arrives
in your inbox, have the filter count the relative word-frequencies
and make a statistical prediction about whether the new message
is spam or not (there are plenty of wrinkles in this formula, but
this is the general idea).
The beauty of this approach is that you needn’t dream up “The
Big Exhaustive List of Words and Phrases That Indicate a Mes-
185
Content 2nd gal final.indd 185
7/4/08 9:57:35 AM