Blink not lest ye be judged a fool

This NYTimes article by Daniel Kahneman, Don't Blink! The Hazards of Confidence, is interesting throughout. Kahneman starts by telling a story about his experience in the Israeli military. He was assigned to a job evaluating candidates for officer training. He and a colleague would have candidates perform certain challenges—he describes one in some detail, "the leaderless group challenge"—and evaluate the men on the basis of their performance on the challenges. They were quite confident in their evaluations.

But, alas, their evaluations were wrong, more often than not.

Every few months we had a feedback session in which we could compare our evaluations of future cadets with the judgments of their commanders at the officer-training school. The story was always the same: our ability to predict performance at the school was negligible. Our forecasts were better than blind guesses, but not by much.

We were downcast for a while after receiving the discouraging news. But this was the army. Useful or not, there was a routine to be followed, and there were orders to be obeyed. Another batch of candidates would arrive the next day. We took them to the obstacle field, we faced them with the wall, they lifted the log and within a few minutes we saw their true natures revealed, as clearly as ever. The dismal truth about the quality of our predictions had no effect whatsoever on how we evaluated new candidates and very little effect on the confidence we had in our judgments and predictions.

Kahneman elaborates on this in various ways, including some discussion of self-confidence in the world of investment and, you guessed it, there seems to0be plenty of unwarranted confidence in the ability to pick smart investments. The moral of the story (boldface mine):

As I first learned on the obstacle field, people come up with coherent stories and confident predictions even when they know little or nothing. Overconfidence arises because people are often blind to their own blindness.

True intuitive expertise is learned from prolonged experience with good feedback on mistakes. You are probably an expert in guessing your spouse’s mood from one word on the telephone; chess players find a strong move in a single glance at a complex position; and true legends of instant diagnoses are common among physicians. To know whether you can trust a particular intuitive judgment, there are two questions you should ask: Is the environment in which the judgment is made sufficiently regular to enable predictions from the available evidence? The answer is yes for diagnosticians, no for stock pickers. Do the professionals have an adequate opportunity to learn the cues and the regularities? The answer here depends on the professionals’ experience and on the quality and speed with which they discover their mistakes.

H/t Rich Fritzson.