What Does Bad Research Look Like and Is There Still Value?

I heard an astounding conclusion on NPR today. Researchers, Ernest O'Boyle Jr. and Herman Aguinis are claiming that the venerable bell curve, the normal distribution, is all wrong when applied to human performance. In their Personal Psychology article, The Best and the Rest: revisiting the norm of normality of individual performance, they make the astounding claim that the normal distribution is a poor tool for predicting human performance. Instead, they claim, the Paretian Distribution more accurately describes human performance as shown in the following figure borrowed from the article.

The black line is the familiar normal distribution; the grey area is the paretian distribution. It shows that the large majority of human performers are subpar, and that a small minority account for the bulk of accomplishment. Superstars are much more valuable than previously thought and account for the majority of success in a group.

The researchers conducted 5 studies, using 198 samples, that included over 633,000 participants in the areas of researchers, entertainers, politicians, and top amateur and professional athletes. The research showed for example, that a small number of researchers wrote far more papers than the rest; that a small number of athletes contributed to the most runs, and that the vast majority of entertainers only received one Emmy nomination while a small number received many nominations.

The data seems to support the conclusions, but there is a fundamental, and rather ironic problem. The authors put it best when they argue that previous research made fundamental mistakes:

When performance data do not conform to the normal distribution, then the conclusion is that the error “must” lie within the sample not the population. 

The irony is that the authors make this same mistake: Their sample is not random. They are examining the cream of the cream, the far, far tail of the normal distribution, and are totally ignoring the "rest of us."

Take a classroom, any classroom, any grade, look into your crystal ball, and divide up the children into 5 groups plus one. Five groups represent those children that would become researchers, politicians, entertainers, and high-performing athletes. The sixth group would be the leftovers. The first five groups are mostly empty, while that sixth group is nearly the entire class. Any child in one of the first five groups can be considered extraordinary.

The authors are focusing on groups drawn from that population that are, by definition, extraordinary. These are people that are living on the far, far right tail of the normal distribution.

Clearly the conclusion is flawed, but is there something we can learn? I think so. The authors state that the normal distribution was invented to help manage assembly lines. The goal of the assembly line is to maximize overall productivity of the system. How would a superstar fair in such a system? Since the optimal assembly line operates at the maximum capacity of the slowest station, this model would force a superstar to become average.

Most of our socio-educational systems are designed like an assembly line: they embrace the average at the expense of the extraordinary. The lesson seems to be that we should seek out social systems that continue to embrace the majority, but provide room for the extraordinary minority - on both ends of the bell curve.

What else can we learn? I've had the opportunity to work in a few organizations where the population was full of "Emmy nominees." They were extraordinary teams within an otherwise normal company.  When it came time to evaluate these folk, the 'normal' evaluation tools were poorly matched to the task. The performance review process assumed a normal distribution, but these groups weren't normal, they were definitely paretian. On a scale of 1 - 5, there were no 1s or 2s, a few 3s, a lot of 4s, and several 5s. Those 5s were amazing.

The lesson here is, when dealing with an extraordinary group of people, normal thinking, normal management, and normal evaluation, are inadequate. Most companies want extraordinary people, but often manage them as if they were ordinary. Those people will leave, and the company will continue to be 'normal.' A company that aspires to greatness needs to learn how to identify, attract, manage, and retain extraordinary people.

The final lesson comes to hiring. Extraordinary people are rare, and true superstars are even harder to find. When hiring, not only do you need to know how to identify, attract, manage, and retain extraordinary people, but you must also be very patient. You will not see many top performers, much less, superstars. When you find one, your 'normal' hiring processes may not be adequate for getting that individual on-board.

The research really helped me think about what is normal and what is extraordinary. It reinforced my conviction that the world of the extraordinary is different and requires a different way of thinking. The research I discussed seems fundamentally flawed to me, yet it is quite valuable if you want to understand life on the edges of 'normal.'