One reason that innovation software, and the idea of crowd sourcing, has become popular in the last few years has been the influence of a single book, James Surowiecki’s The Wisdom of Crowds. While in the market of business publications there have been plenty of sensations – Malcolm Gladwell’s The Tipping Point would be an obvious example – it is hard to think of any in the last decade whose influence is unchallenged and still growing. I should make it clear, however, that this blog post is not a book review, but rather a discussion of some of the main ideas of “The Wisdom of Crowds”.
The other reason for the attention paid to Surowiecki’s ideas is that they are radically different to received wisdom and could be very important to many companies at a time when the margin for error is becoming very small. The technical and innovation lead that Europe and the US had over China and other Asia Pac economies has been vanishing gradually over decades, as quality of manufacturing and speed of innovation have improved steadily. Companies have to be strong in every area of their operation, from market analysis through to delivering the products that fit and even redefine the market. Intelligence, innovation and decision making have to be done well, and fast!
How businesses usually solve problems
The standard model of business decision making has been that the more important the decision, the more senior and the fewer people are involved in it. Where the expertise at the top is insufficient, outside experts are called in to assist. The rationale of course is that the best brains in the business have got to the top, and there is no point using the company’s second-best. Plenty of tools have emerged to stimulate analytical and idea creation abilities within these groups, but the point remains that the make-up of these groups is very restricted.
The problems with this approach:
1. Life changes: It is a truism that in each new war, the generals at the top, i.e. the commanders who fought most successfully in the previous war, are soon replaced because they are fighting the wrong war. Likewise in companies, a history of promotion shows at best success in dealing with past challenges, at worst a series of lucky breaks or a talent for getting spotted.
2. Experts are people who seem to know something you don’t. It is very hard to tell whether they really know it. Experts are also famous for disagreeing with each other: the top expert is often simply someone who scored the most recent victory in an continuous argument.
3. Similar to point 1, experts have a history of solving past problems, but this guarantees nothing in respect of new ones, especially where these need a new way of thinking. For experts, too, awareness of past success can create a block to envisaging new solutions.
4. Experts often share a common academic background, which can bias their take on a problem, and often respond to each other either for validation or differentiation, both of which remove the critical element of independence.
5. The smartness of an organization has a relationship to the total smartness of all the people in it – the sheer weight of numbers of people thinking about a problem makes a real difference to the likelihood of successful analysis.
6. Restricting analysis of a problem to a small number of people leads to the danger of consensus for political reasons – either fair of disagreeing with the boss, or trying to pre-agree, i.e. to predict what the boss will want to hear, or maintaining consensus in the group. These are powerful factors and make it a dangerous business to try to be heterodox or introduce information that does not fit.
That all seems pretty damning, but to date, getting wider feedback in a practical way has been very difficult both technically and politically, and this is perhaps the main reason that we have stuck with the status quo for so long. However, collaboration and innovation technologies make solving these problems a practical proposition, and one that it makes no sense to ignore. But before looking at how such solutions can work, we should briefly look at why they work.
The organization is smarter than any of its subsets
That is the main single point that Surowiecki has to make, and he has compiled a lot of convincing research in social psychology to make the case. In experiment after experiment, a diverse group of people who are not expert has more predictive ability than any single expert or of a smaller group with more expertise. The phenomenon has long been known. Over a century ago, Francis Galton compared ordinary people’s guesses of the weight of an ox (in a competition in a country fair) to those of experts (e.g. cattle breeders). Galton was a snob, to put it bluntly, and wanted to prove how defective the judgment of the ordinary person was. The averaged guess of the crowd was astonishingly close, however, to the real weight, and much better than that of the experts. This phenomenon has been seen repeatedly since in experiments pitching large diverse groups of people against small groups of experts to predict election results, share prices and first-week box office takings.
Target practice: amateurs can miss by more, but their combined score hits the spot
In many cases, the accuracy of predictions made by a large, assorted group of people is almost eery, as though there is some paranormal power at work. More prosaically, though, it is easier to think of it as like throwing a large number darts at a target. An unskilled group will generally miss by quite a bit, but not all in the same way, whereas star players may well share strategies and training, hence make similar kinds of errors. Applying the idea to more information-related themes, one could say that a large, diverse group of people have all kinds of different ignorance, and make all kinds of errors, compared with a more homogenous group.
What is essential for groups to make accurate judgments:
1. Diversity. Groups work well when the types of people and the information sources they use are diverse. Larger numbers of participants offer more diversity and less risk of “groupthink”. Small groups, especially where members share common interests, are likely to polarize towards a consensus, rather than risk a split.
2. Independence. Crowd sourcing works best when participants are not responding to each others’ opinions – or trying to predict them. This is perhaps the biggest single source of problems in decision making. One effect is to predict what the most important or prestigious group member might think, meaning that the focus is on second-guessing, rather than on the real problem. A second effect is cascading, when one person’s judgment sways that of others, independent of their own ideas. Nearly all forms of open discussion cause positive feedback effects or distortion
3. Decentralization. When questions of loyalty, prestige or career advancement come in, the effects of distortion and feedback are serious. These distortions are particularly strong when information flow is strongly influenced by, and reinforces, hierarchies.
3. Selection. There has to be a way to actually get the information from the people who have it, to the people who want it, that does not create bias or feedback. It is easy enough to talk about open businesses, but it is a model that is extremely rare in practice, because it is part of a manager’s job to stay in control, and without the sense of control it is hard to manage! This is perhaps the place where collaboration and innovation software play their greatest part, enabling decision makers to retain control, but gain a wide enough variety of input.
The power of social technologies to improve analysis and idea selection
One thing that sets The Wisdom of Crowds apart from titles that have caused a storm and then disappeared, is that the thesis does not depend on the internet for its validity. The research cited spans many decades. However, social technologies are the great enabler. Companies need a strong sense of leadership, and people who have demonstrated good judgment, effort and responsibility are rightly chosen to lead. However, that does not mean they have to have all the ideas. What they need is a way of selecting and aggregating the best of a diverse input of ideas from all the brains of the company – and of course that includes their own ideas, too. What employees need is a simple, enjoyable way to contribute those ideas, and one that establishes their worth as idea generators for the company. This is not a utopian vision, but a simply reality, and one that is increasingly being espoused by companies that dare to think positively.