Are your evidence appraisals a victim of the Dunning-Kruger effect… or are you just better than the rest?
Thomas Frost & David Nunan
Almost twenty years ago, two researchers from Cornell University published an Ig Nobel Prize-winning paper. If you haven’t read it, you should really check it out.[i] In the interests of time and the $11.99 fee, it begins with the endearing story of a would-be bank robber named McArthur Wheeler.
In 1995, Mr Wheeler made the critical error of confusing the ‘invisible ink’ properties of lemon juice with the visual properties of everything else and walked into a bank with lemon juice smeared all over his face. Robbing two banks over the course of a single day, he was reported to be incredulous when the police caught him later that same day, using CCTV footage of his face. “But I wore the juice!”, he exclaimed.[1]
Fast forward 20 years, and reflecting on 100 days in the job, President Trump retorts “I thought it would be easier. This is actually more work.” Or as he might have put it “But I wore the juice!”.
What Wheeler and Trump demonstrate in action is a cognitive bias first described in 1999. Over several in-house experiments (none of which resulted in jail for the participants), David Dunning and Justin Kruger observed a finding that has gained cult status: participants tested on a task and asked to assess their own performance behaved in a seemingly paradoxical way.
In every instance, those who scored highest would under-score their performance as just above average, while the worst performers were ‘over-optimistic’ in their self-perceptions. Whether the test was humour, logic, or grammar, the findings were the same – as cognitive talent worsens, so too does ‘meta-cognition’ (or the ability to assess ourselves accurately in that area). Thus, the Dunning-Kruger effect was born.
Sometimes though, the Dunning-Kruger effect is mischaracterised as ‘stupid people don’t know they’re stupid’, which is an unfortunate and ironic misunderstanding. The bias has little to do with ‘intelligence’ per se. There are plenty of smart, confident, but bad drivers out there. These drivers don’t think they are F1-level drivers, they just rate themselves as ‘pretty good’ at the skills they think define good driving.
The point is that we judge our own performance based only on the markers of skill that we already know and think about. Most people know what F1-driving ability looks like, and can confidently say they aren’t at that level. But in terms of general driving ability, if we don’t know the small details that might make a good driver, we’ll never include these qualities when assessing our own ability.
What Dunning and Kruger showed was that as you become increasingly skilful at a task, and begin to appreciate how little you really know, you start to rate your ability less favourably than those who are less skilful. Until you reach that level, you’re destined to hype yourself up based on the limited knowledge you have.[2]
A cartoon depiction of the Dunning-Kruger effect:
Which brings us, in a roundabout way, to the point of this post. If the Dunning-Kruger effect can be demonstrated for a range of skills, whether manual or intellectual, sporting or vocational, it isn’t much of a leap to say that this effect probably exists for anyone who uses and appraises research. We’d wager there’s a Dunning-Kruger relation between confidence in the ability to appraise research and how many research biases the reader is aware they know.
Why not try it out yourself? Or are you just better than the rest?
Thomas Frost: University of Oxford final Year Medical Student
David Nunan: Departmental Lecturer and Senior Researcher at the Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford. He is also the lead tutor of the Practice of Evidence-based Health Care module on the MSc in Evidence-based Health Care. You can follow him on Twitter @dnunan79
Conflicts of interest: none reported
References
[1] Tragically, Mr Wheeler tested out his theory first with a polaroid camera. Sure enough, the ‘selfie’ he took printed as a blank image, almost certainly due to defective film.
[2] Meanwhile, at the other end of the scale, the truly expert individuals have a mistaken perception of how ‘basic’ certain pieces of their knowledge are (relative to the sheer unknowns still out there), and over-optimistically attribute that knowledge to others.
[i] Kruger, J and Dunning, D. “Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments.”. Journal of Personality and Social Psychology, 77(6), 1999, pp. 1121-1134.
