Independent Individuals and Wise Crowds, or Is It Possible to Be Too Connected? James Surowiecki http://conferences.oreillynet.com/cs/et2005/view/e_sess/7022 At the O'Reilly Emerging Technology Conference San Diego, California, 16 March 2005 Impressionistic transcript by Cory Doctorow doctorow@craphound.com -- I have no coherent answer for these questions, but I want to raise questions on collaboration, cooperation and collective intelligence. One of the major themes of the last decade has been collective action/collaboration: Slashdot, Google, prediction markets, FlashMobs, Wiki, Linux, social software, etc. All projects that involve large groups of people coming together to work explicitly or implicitly (e.g. Google) on complicated problems and to solve them. There's an emotional affinity among these movements, a common thing in how we experience them. There's been a lot of writing about network effects, smart mobs, emergent behavior, etc. I want to talk about the differences between them, to explore how these things are not alike. Ontology is overrated, but classification is useful, because different kinds of problems need different kinds of solutions. Use the wrong kind of solution for your problem and you'll end up with a worse problem. There's a lot of fuzzy thinking about what we mean when we talk about collective action. I wrote a book called "Wisdom of Crowds" -- an argument about the power of groups under the right circumstances to be smarter than the smartest person within it. Here's the model for collective intelligence: Here's a large group of people acting independently, making judgments -- add a mechanism that collects the judgments and aggregates them. Example: A contest at a country fair: guess the weight of an ox after it had been slaughtered and dressed. It was predictive and needed to account for the butcher's skill. 800 guessers, including expert butchers and farmers and also merchants and family members. After the contest, taking the average you got 1197 lbs -- and the actual weight was 1198 lbs. It wasn't a coincidence, nor limited to ox-weight-guessing. At racetracks, the odds on horses predict almost perfectly how likely a horse is to win. Odds are determined collectively by correlating all bets and establishing statistical judgments. Eli Lilly's think-tank has an internal stock market for predicting which drug candidates will make it to Phase 3 clinical trials: vital to narrowing down which pharma product to invest in. They open the market to 100 "semi-experts" who have some info but aren't on the inside, and collectively they identify which candidates will win. Wisdom of crowds works on problems where there's a true answer, or when some choices are better than other in some Platonic sense. The reason this works is that people are operating on private info, which may be bad or fragmented. The opinions are diverse -- not consensus but disagreements. People don't know much about what others are betting on or guessing -- not a lot of interpersonal interaction. Compare with Linux: large group of people working on problems, but ultimately one person writes the code that gets committed. The decision is centralized: one or a small number of people get to commit code to the kernel. Compare with ant-hill: Often a metaphor for human behavior. How to use a bunch of dumb agents (ants don't know much) whose interaction produce stunningly intelligent results, e.g. finding food with least amount of energy. E.g. ant graveyards and food supplies are equidistant. Ants follow simple rules and pay a lot of attention to those around them: interaction is the essence of intelligence. The only way to get where they want to go is by paying close attention to one another. Here's my message: HUMAN BEINGS ARE NOT ANTS. We do not have the biological programming or tools to allow this kind of interaction to produce intelligence. We don't have the ability to sense or secrete formic acid. Interaction for humans is problematic, especially with groups. If there's too much interaction or the wrong kind, groups end up being less intelligent. The more we talk to each other the dumber it's possible for us to become. Here's why: * Human being herd: we stick with what others are doing. "Better to fail conventionally than to succeed unconventionally." Portfolio managers herd, even though their business is predicated on outperforming each other, they attend closely to one another in order to appear reasonable. * Human beings imitate: It's a quick-and-dirty heuristic for getting things done. If you stand on a corner and gaze at the sky, the passersby will also do so, and the more people there are looking up, the more people will stop and look at the sky. If a lot of people think that something is valuable, then it must be valuable -- a generally reasonable assumption. But slavish, unreflective imitation leads to staring up at the sky on a street-corner. Economists have a term for this: Information Cascade (viz Tipping Point and Linked). Once an info-cascade gets going it's very hard for people late in the process not to do what others have done. If you've got two restaurants and no way to know which is better and you're choose one at random, the next selector will choose yours on the grounds that someone is there, and it cascades, so the next person won't brave the other. Everyone assumes there's some value to people choosing one restaurant. After a certain point, it actually becomes rational to follow the crowd, even if you know something that the others don't, provided that everyone else is rational. That's the tipping point: making the same decision as those in front of you. If you have a group of networked people, their collective decision isn't necessarily tied to quality any more than it is tied to the inertia of a ball rolling. A site with a lot of inbound links accumulates more links: no guarantee that the group is intelligent. What are the consequences? Pascal: "All problems arise from the fact that man cannot sit in his room quietly by himself." Exchanges of information don't have to lead to homogeneity, if we can share insights. Some exchange is useful -- why is the horse I favor running at 40-1? What do they know that I don't? Some projects like software coding require collective action. The question for me is: how do you have interaction without information cascades, without losing independence? The best thing is to keep ties loose. Don't be tightly networked with strong ties -- be loosely tied. Minimizes the influence that others have on you. I don't think information cascades are as big a problem as others believe. I don't think people are as subject to the influence as others around us, but we are shaped by them. We do need to limit their power. Also: be exposed to many diverse sources of information. Inject randomness into the system, see evolution's law of necessary variation. There's a corollary in group decisions -- diversity is in and of itself a good thing. Think about the blogosphere and the way it works, or the way that orgs make decisions: it's why we want cross-disciplinary, trans-hierarchical teams. You *can* be too connected if the connections are the wrong kind and reinforce you prejudices rather than altering them. You can pay too much attention to those around you even if they're really smart and know a lot. The inverse of isolation is the net's cacophony: the bombardment of information and voices. Isolation and cacophony allow you arrive at the same place: independence from the ideas of those around you.