Why a simple statistical method is more successful than a complex scouting program
By Cam Charron
9 years ago
If you haven’t read Rhys’ dirty ditty exposing the Vancouver Canucks scouting office for being, among other things, complete frauds, then go ahead and do that. It was interesting to me, because having read Thinking Fast and Slow by Daniel Kahneman, the Nobel Prize-winning psychologist actually has a full two chapters dedicated to the benefits of simple statistical measures when attempting to assess performance.
The thing that Rhys’ piece exposes is that for the longest time, people in the game of hockey are determined to believe that hockey is a more complex game than it actually is. Winning teams, though, score more goals than the opposition. It is impossible to determine how many goals a player prevented, so counting how many goals the player contributed to the cause is basically half of our equation. However scouts try to add to the equation usually bogs down the process.
Kahneman, like another of my favourite thinkers, Nassim Nicholas Taleb, is very skeptical of experts. He won the Nobel Prize in economics, which is notable, because he’s not an economist. His research done on biases and the way people think is not only fascinating, but has a lot of useful market applications.
In his book, Kahneman quotes the study of a character named Paul Meehl, who performed several studies to determine whether “experts” in a certain field were better off at predicting future success than a simple formula.
In a typical study, trained counselors predicted the grades of freshmen at the end of the school year. The counselors interviewed each student for forty-five minutes. They also had access to high school grades, several aptitude tests, and a four-page personal statement. The statistical algorithm used only a fraction of this information: high school grades and one aptitude test. Nevertheless, the formula was more accurate than 11 of the 14 counselors.[snip]The range of predicted outcomes has expanded to cover medical variables such as the longevity of cancer patients, the length of hospital stays, the diagnosis of cardiac disease, and the susceptibility of babies to sudden infant death syndrome; economic measures such as the prospects of success for new businesses, questions of interest to government agencies, including assessments of the suitability of foster parents, the odds of recidivism among juvenile offenders, and the likelihood of other forms of violent behaviour; and miscellaneous outcomes such as the evaluation of scientific presentations, the winners of football games, and the future prices of Bordeaux wine. Each of these domains entails a significant degree of uncertainty and unpredictability. We describe them as “low-validity environments.” In every case, the accuracy of experts was matched or exceeded by a simple algorithm.
It isn’t just the Vancouver Canucks who are susceptible to this. The world is beginning to realize the importance of data, but it’s important not to be paralyzed by analysis. When evaluating a young junior hockey player, scouts like to weight the player’s proficiency in both the offensive and defensive zone, how big he is, where he was born, how hard he works (or appears to work), the quality of his skating stride, the quality of his shot, how much his teammates respect him on the bench, and even submit the poor kid to an interview. In an effort to prove their usefulness by identifying the parts of the game that the common fan couldn’t see, scouts get out-performed by statistics, year, after year, after year, after year.
There’s an issue with resistance to the obvious choice. As Kahneman notes: “Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula!” Hidden in plain sight.
Kahneman also writes:
Facts that challenge basic assumptions—and thereby threaten people’s livelihood and self-esteem—are simply not absorbed. The mind does not digest them. This is particularly true of statistical studeies of performance, which provide base-rate information that people generally ignore when it clashes with their personal impressions from experience.”
In other words, “have any of you nerds ever even PLAYED the game?”
Hockey is fun to watch. It’s why we do it. We wouldn’t be doing any of this if we didn’t like watching hockey and the speed and the skill and the personalities that go along with it. It’s entertaining television. I have to admit I feel for anybody who likes to tell me that they rely on “watching the game” for analysis. I picture them judiciously compiling mental notes while their friends around them are drinking beer and having a good time. It’s as if there’s a higher purpose to this whole experiment, which is sort of silly. We spend dozens of hours a week caring about a game played by people we don’t know and will never know, for no reason other than it’s fun. The best hockey writers aren’t the ones who provide the best analysis, but the self-aware writers who can still contextualize the game in the realm of “fun”.
But that doesn’t mean we can’t glean lessons from it. People inside the Vancouver Canucks have not fared as well as my diabolical evil twin Sham Sharron in predicting the future success of NHL players, and I would hardly doubt it stops there. I would also caution current scouts and managers that your cognitive abilities are only as good as their inputs, and you aren’t as objective as you think you are.
Recent articles from Cam Charron
