# Why a simple statistical method is more successful than a complex scouting program

If you haven’t read Rhys’ dirty ditty exposing the Vancouver Canucks scouting office for being, among other things, complete frauds, then go ahead and do that. It was interesting to me, because having read Thinking Fast and Slow by Daniel Kahneman, the Nobel Prize-winning psychologist actually has a full two chapters dedicated to the benefits of simple statistical measures when attempting to assess performance.

The thing that Rhys’ piece exposes is that for the longest time, people in the game of hockey are determined to believe that hockey is a more complex game than it actually is. Winning teams, though, score more goals than the opposition. It is impossible to determine how many goals a player prevented, so counting how many goals the player contributed to the cause is basically half of our equation. However scouts try to add to the equation usually bogs down the process.

Kahneman, like another of my favourite thinkers, Nassim Nicholas Taleb, is very skeptical of experts. He won the Nobel Prize in economics, which is notable, because he’s not an economist. His research done on biases and the way people think is not only fascinating, but has a lot of useful market applications.

In his book, Kahneman quotes the study of a character named Paul Meehl, who performed several studies to determine whether “experts” in a certain field were better off at predicting future success than a simple formula.

In a typical study, trained counselors predicted the grades of freshmen at the end of the school year. The counselors interviewed each student for forty-five minutes. They also had access to high school grades, several aptitude tests, and a four-page personal statement. The statistical algorithm used only a fraction of this information: high school grades and one aptitude test. Nevertheless, the formula was more accurate than 11 of the 14 counselors.

[snip]

The range of predicted outcomes has expanded to cover medical variables such as the longevity of cancer patients, the length of hospital stays, the diagnosis of cardiac disease, and the susceptibility of babies to sudden infant death syndrome; economic measures such as the prospects of success for new businesses, questions of interest to government agencies, including assessments of the suitability of foster parents, the odds of recidivism among juvenile offenders, and the likelihood of other forms of violent behaviour; and miscellaneous outcomes such as the evaluation of scientific presentations, the winners of football games, and the future prices of Bordeaux wine. Each of these domains entails a significant degree of uncertainty and unpredictability. We describe them as “low-validity environments.” In every case, the accuracy of experts was matched or exceeded by a simple algorithm.

It isn’t just the Vancouver Canucks who are susceptible to this. The world is beginning to realize the importance of data, but it’s important not to be paralyzed by analysis. When evaluating a young junior hockey player, scouts like to weight the player’s proficiency in both the offensive and defensive zone, how big he is, where he was born, how hard he works (or appears to work), the quality of his skating stride, the quality of his shot, how much his teammates respect him on the bench, and even submit the poor kid to an interview. In an effort to prove their usefulness by identifying the parts of the game that the common fan couldn’t see, scouts get out-performed by statistics, year, after year, after year, after year.

There’s an issue with resistance to the obvious choice. As Kahneman notes: “Several studies have shown that human decision makers are inferior to a prediction formula even when they are given the score suggested by the formula!” Hidden in plain sight.

Kahneman also writes:

Facts that challenge basic assumptions—and thereby threaten people’s livelihood and self-esteem—are simply not absorbed. The mind does not digest them. This is particularly true of statistical studeies of performance, which provide base-rate information that people generally ignore when it clashes with their personal impressions from experience.”

In other words, “have any of you nerds ever even PLAYED the game?”

Hockey is fun to watch. It’s why we do it. We wouldn’t be doing any of this if we didn’t like watching hockey and the speed and the skill and the personalities that go along with it. It’s entertaining television. I have to admit I feel for anybody who likes to tell me that they rely on “watching the game” for analysis. I picture them judiciously compiling mental notes while their friends around them are drinking beer and having a good time. It’s as if there’s a higher purpose to this whole experiment, which is sort of silly. We spend dozens of hours a week caring about a game played by people we don’t know and will never know, for no reason other than it’s fun. The best hockey writers aren’t the ones who provide the best analysis, but the self-aware writers who can still contextualize the game in the realm of “fun”.

But that doesn’t mean we can’t glean lessons from it. People inside the Vancouver Canucks have not fared as well as my diabolical evil twin Sham Sharron in predicting the future success of NHL players, and I would hardly doubt it stops there. I would also caution current scouts and managers that your cognitive abilities are only as good as their inputs, and you aren’t as objective as you think you are.

• argoleas

Reminds me a bit of the last US election, when one side had a gut feeling it will pull out a win, and the statistics, which said otherwise. Guess who was right.

But isn’t the ‘gut feeling’ or ‘experience’ just the absorption of previous data? The quotation above about people rejecting data that contradicts their own views is a telling one.

• Cam Charron

“People inside the Vancouver Canucks have not fared as well as my diabolical evil twin Sham Sharron in predicting the future success of NHL players, and I would hardly doubt it stops there. I would also caution current scouts and managers that your cognitive abilities are only as good as their inputs, and you aren’t as objective as you think you are.”

False.

Run the simulator for the Canucks in the 1998 draft if you would like to see the limitations of Sham Sharron…

Stop living in a bubble.

If one is truly confident in the Sham model, one should run the simulator for the other 29 teams and have a look at the results.

Because 29 other teams also have “expert” scouts and you should have the confidence in running this statistical method against all 30 NHL teams instead of using highly flawed anecdotal evidence.

Collectively, the 30 NHL teams using their “expert scouts” as part of their decision making process are going to destroy the Sham model.

For starters, virtually every single CHL scoring forward selected using the Sham model was selected by the 30 NHL teams.

In addition to this bounty that 30 NHL teams enjoy by mocking the Sham model, let’s see what else they would have:

1. Virtually every single goaltender in the NHL

2. Virtually every single defenseman in the NHL

3. Virtually every single forward in the NHL that did not play in the CHL during his draft eligible season.

Even from a pure forward standpoint, it’s going to be the Detroit Red Wings vs the (largely) Canadian pluggers that Don Cherry wants in the NHL versus extremly skilled European players.

Hilariously myopic…

• Steve Dangle

But, at the least, you should run it to include goalies & defenseman as well as for all 29 teams before bringing Delorme’s livelihood into this…

• pheenster

Wow. NM00 is clearly Ron Delorme. Or, more likely, his wife.

• JCDavies

Is this where all the Gillis hate comes from?

• Spiel

The “Sham” model is amusing, but I don’t think it proves the point that the Canucks drafting sucks in as compelling of a way as you guys make it out.

The “for realsies” knowledge that Sham uses to narrow down the player pool relies on NHL team scouting methods and evaluation that goes beyond Sham’s purported mindless criteria.

I would say that Sham is much more than an intern. Sham is an NHL spy. Sham does some dirty work (who knows what?) to get other team’s rankings, and then applies his own arbitrary criteria because he doesn’t have the hockey “knowledge” of the scouting departments to use as a basis for a final choice.

It would be a more compelling case, if Sham were to stop using the “for realsies” knowledge and instead base his choice on public knowledge available at the time of the draft (such as NHL central scouting) before applying his arbitrary criteria. I’m sure the Canucks drafts would still come out looking like turds.

• Goon

Isn’t that why they did?

Maybe I misread, but I thought Sham’s drafting was based only on publicly-available point totals and points-per-game, not on scouting reports or pre-draft rankings or any of that other information.

• Spiel

No it wasn’t. Here is rule 4 from Sham:

4) The Canucks’ selection will be the player still on the draft board that scored the most points in their 17 year old CHL season that was for-realsies taken between Vancouver’s selection and Vancouver’s subsequent selection.

So what they did was look at the CHL forwards actually selected at and after the Canucks pick and then take the player with the most points.
To get the list of forwards to select from, they used future knowledge (players that were going to be taken) that would not have been available at the time of the pick.

I get the point, but the method used to prove the point uses the scouting knowledge they are trying to dismiss as the filter for possible players.

Look at the 2000 NHL draft for example. Sham takes Justin Williams in the first round who has 83 points instead of several QMJHL players (including Brandon Reid) who had 100+ points. Clearly Sham has some inside knowledge to know he doesn’t need to take Reid in the first round…but how does he know this important info? Sham has been talking to scouts.

That’s why I suggest using all public info like the NHL central scouting list instead of “for realsies” as the filter to make it more convincing.

I think the problem with just points and no “for realsies”, and the author probably recognized this, is it doesn’t fit the desired conclusion. At least it doesn’t for the 2000 draft, not sure about other years, but the case is likely the same.

• Spiel

This is important if we’re going to conclusively throw the Canucks’ scouting team under the bus.

I expect that if only public info was used the comparison would be much less stark. Having said that, I don’t have the time to do that and I suspect the CA bloggers don’t either…I enjoyed the article as is.

@NM00 Read the article with your brain turned on and realize it isn’t meant to be taken as fact. It’s essentially a satire piece that dramatizes something we all know to be true (the Canucks’ scouting has been terrible).

• Spiel

Yes, I suspect you are right that it takes time to come up with a filter based on public data. But the main problem seems to be the ultra high scoring QMJHL that skews results. I think Sham should use the same method, but remove rule 4, and change rule 1 to be all players are selected from the OHL or WHL.

• orcasfan

“It’s essentially a satire piece that dramatizes something we all know to be true (the Canucks’ scouting has been terrible).”

I actually don’t know that, relative to 29 other teams and with respect to draft position, the Canucks’ “scouting” was bad.

• Cam Charron

Is what Sham did not essentially like saying that a list like the ISS ranking was used? It only has to identify guys who were viewed as consensus BPA within that 29 pick range. Basically, yes, things like size were taken into account by draft rankings but that doesn’t imply Sham talked to scouts or that Vancouver even had a scouting department. He just used listed information available before the draft.

• Spiel

I would say it is close, but not quite.
Teams diverge pretty drastically from the “consensus” lists like iss or central scouting after the first round and even within the first round, and Sham getting one decision right makes a big difference to the outcome.

Use the 2000 entry draft as an example.
Central scouting has Justin Williams ranked #19 and he has 83 points. But Central scouting also has Yanick Lehoux (92 pts) and Carl Mallette (125 pts) at #35 and #37. Williams was taken in the first round. Mallette and Lehoux were taken in the 3rd and 4th rounds. Neither had an NHL career. Clearly teams knew something about these players that stats and central scouting didn’t catch. Sham’s method gives him access to the rankings of all teams that he would not have otherwise had.

Replacing Williams with one of these other guys drops 800 games played and 500 points from Sham’s total. I wouldn’t doubt that Sham still ends up on top, but the difference is likely less. Still not a glowing endorsement of the Canucks drafting acumen.

• Cam Charron

I don’t disagree. Sham is hardly a genius after all, but I believe his method was a proxy for using central scouting lists. His approach just happened to save time. However, I wouldn’t be surprised if picking the top point producer from central scouting’s list BUT only to a certain point (say from the top 15 possible picks) would yield similar results.

I would love to see if the simulation was run again considering only those on central scouting lists within a certain range.

I also wonder how robust these rules are, to see if the model is on the right track or purely lucky (as your Justin Williams example attempts to point out). By varying the parameters slightly (ie picking among a slightly larger or smaller amount of players each selection) and seeing if the results change wildly. If they do, then Sham is just the beneficiary of beginners luck.

We’re just giving CanucksArmy suggestions for new features now. I suspect royalties shall ensue haha

• Steve Dangle

‘If one is truly confident in the Sham model, one should run the simulator for the other 29 teams and have a look at the results.”

RELEASE THE SHARRON.

• Goon

You clearly don’t understand either basic math or the point of this exercise.

• orcasfan

Just when I thought you couldn’t get any more full of it. This is so clueless that it’s delusional.

Sham so completely knocked selections out of the park compared to ANY team, as obvious by the depth that resulted under the simulation. The Canucks could simply have traded for blue chip defensemen, goalies and non-CHL forwards given their incredible depth and supplemented their efforts with free agent signings. Defensemen and goalies tend to come cheaper than forwards on the trade market anyways.

Sham was deliberately a simple simulation, but it revealed how easy it is to come up with an edge in drafting because the scouts are collectively poor performers. This isn’t news. The majority of portfolio managers under-perform the market, etc, etc.

• Cam Charron

Completely and utterly false.

The 30 NHL teams using scouting, proprietary analytics or even names out of a hat would not overlook goalies, defenseman and non-CHL forwards.

The 30 NHL teams collectively would destroy Sham’s model.

“Sham so completely knocked selections out of the park compared to ANY team”

How can you possibly know this based on one, extremely flawed simulation that doesn’t even look at the 29 other teams in the NHL?

• Cam Charron

You clearly still don’t understand what the simulation actually was. It used no future knowledge. It just picked CHL forwards based on points totals.

As mentioned in my last comment (which you totally glazed over), the forward depth from picking CHL forwards only would have been so significant that defense and goalies could frankly be acquired through trades and free agency. Why not just pick CHL forwards if that’s the case? Saying scouts would never do it is moronic. Scouts will do what’s best for the club and if drafting only CHL forwards is a solid approach, they’ll do it.

As for the rest of the teams’ actions, again you really don’t get what was done. Who Sham picks might change a few picks after the Canucks because the team who drafted Sham’s pick in real life would pick someone else, but by the time 29 other teams have picked, and it’s Sham’s pick again, it’s essentially the same list of players. No one picks 1st rounders in the 2nd round (or 2nd rounders in the 1st) because of the shift in BPA. It just means a few picks shuffle about unless others teams go REALLY far off the board.

• Cam Charron

Okay, I’m going to say the brutally obvious, NM00, and hope that you think about it for a second: neither Rhys nor Cam are advocating that a real NHL team use Sham Sharron’s method for drafting. Neither is even making a case that they have found a model which predicts future success for junior players. Rather, this is a tongue-in-cheek way to make a bigger point about all the so-called “intangibles,” and the fascination with size. The point is that an NHL team should be able to draft forwards better than Sham Sharron. It’s only a baseline measure, nothing more. And commenter Vic in the last article took out the extra forwards, and found that Sham still outdid the Canucks drafting. (Btw, I’m sure that Rhys and Cam would be thrilled if you tried this on the other 29 teams. Go for it!)

Of course, it would be fun to see the model made more complex, and to use other measures (like TOI) so that at least Dmen can be included. But there’s only so much time and energy that bloggers have, and this was a good first start. Don’t overthink it.

• Cam Charron

Okay, I’m going to say the brutally obvious, antro, and hope that you think about it for a second:

It makes zero sense to begin this process without looking at what the 29 other teams have done during the same time period and what they WOULD HAVE done using this model or a variation of this model.

Hiding behind “tongue and cheek” is a copout, though I did enjoy some of the quips along the way.

As an example, consider that, at minimum, 8 teams should have selected Giroux before Philly did at #22 in 2006.

And quite possibly more than 8 because, if every team used the simulator, he may not have been available at #14.

The simulator is going to make MANY teams look silly when you look at each team INDIVIDUALLY.

However, when you look at all 30 teams COLLECTIVELY, the “experts” within NHL front offices that Cam is clumsily attempting to bring down a notch are going to destroy the model since they are not limiting themselves to less than half of the available talent pool.

And why even bring Delorme into this if one is not privy to his individual draft board?

And why take baseless shots at Burke when, as a commenter pointed out in a previous thread, the Canucks are bloody lucky he selected Henrik & Daniel over Brendl & Saprikyn?

Misdirection.

Nothing more…

• Goon

In the stock market guys get paid to predict the future. but they also influence the future as it relates to the markets. I banks pump what they cover, hedge funds front run their own holdings.

same concept for pro scouts and their player development colleagues. Once a player is drafted, his future is materially influenced by the guys predicting he’ll have a good future.

My point – you can’t plug in an algorithm to predict what happens next or down the road. Too many variables influence outcomes. Great article though.

• orcasfan

I was also reminded of Kahneman’s excellent story about simple emergency-room algorithms outperforming expert diagnoses, and also thought of Barry Schwartz’s Paradox of Choice (too much noise in the data leading to worse decision-making). This is an even better rebuttal to Steve Simmons’ ridiculous anti-stat diatribe, not that he would understand this stuff either.

I’m sure NHL teams have tried to come up with formulas and models based on past draft results, but it makes you wonder whether they were looking at the right variables and using the right data. Sham’s “rules” were intentionally ridiculous, riffing off the hilarious GM vs potato bit, but it’s certainly possible that you could design a type of algorithm that would crunch specific pre-draft data over decades, and see which factors bubble up to indicate later success. Maybe the data for junior leagues hasn’t really been good enough in the past, but that must be coming.

Also, I put this link at the bottom of the Sham post as a bit of sobering non-Canucks draft analysis that sadly shows just how bad the team’s recent drafting has been compared to other teams since 2006 (scroll down for the league-wide stuff): http://thepensblog.com/2014-archives/shero-draft-piece.html

• orcasfan

@JeremyOK

Thanks for the link. I don’t know why but the charts at the end didn’t show up. But the bloggers did single out Vancouver as particularly bad, so that’s worrisome.

@NM00:

Are you related to Ron Delorme? It seems like all you want to do is blame Gillis. But 2007, maybe the worst draft year, is pre-Gillis. (Seriously, Patrick White?)

You do bring up a good point which would be fun to know: how much does the GM run the draft? My memory is fuzzy, but I remember that the GM sets guidelines, but isn’t it the chief scout that does most of the draft after the first round? And obviously the scouts have influence on the first round pick as well. All in all, my guess is (and I’m happy to be corrected), the draft isn’t just the product of the GM at the top.

• orcasfan

Hmm…weird, the graphs show up for me. And it’s kind of useless without them. Basically they’re comparing the “depth drafting” (ignoring the first round picks) of forwards from every team since 2006, and the Canucks are at the bottom of every measure (points, games played). And of course you could slice and dice different ways to measure draft success, but looking at the contributions the better teams get from their depth forwards is pretty depressing (as a ‘nucks fan).

I think this was linked here recently too, and although it doesn’t focus specifically on the Canucks, it makes another attempt to quantify draft success across all teams since 2003: http://nhlnumbers.com/2014/5/9/drafting-for-success

I don’t know how you could look at those numbers and defend the team’s drafting record. The fact that they’ve had the fewest draft picks during that span is not a defense — the percentage of those picks who have played at least 50 games is just putrid, and worst in the league. Let’s just hope they figure it out eventually.

• JCDavies

Sigh.

It has nothing to do with Delorme specifically.

As you admit in the last part of the comment, you have no idea to what degree a team’s selections is based on the scouting director.

And neither do I.

Do you actually advocate excessively criticizing a scout without looking at his board?

In any case, one way or another, this would fall under the responsibility of the GM since he can veto any and every scout in the org and also sets the organizational philosophy at the draft.

What if Delorme and co wanted to take Karlsson over Hodgson?

What if Delorme and co absolutely despised the overager fixation the Canucks had for a while but was overruled by the GM?

What if Delorme and co absolutely despised the shift from skill to size beginning with the 2011 draft?

As for 2007, that was an unmitigated disaster no doubt.

But let’s look at 2003-2007 a little differently.

Hypothetically, let’s say that White & Ellington were selected in the 3rd and 9th round of the 2004 draft.

And let’s say that Edler & Hansen were selected in the 1st and 2nd round of 2007.

The talent in the org would be the same.

Point being, the homerun of 2004 has offset the strikeout of 2007.

The same goes for Burke picking Henrik & Daniel in 1998 and missing on other picks.

If Gillis’ later round picks can offset some of the underwhelming selections/trades that have depleted the system, I’d gladly acknowledge that he has balanced the books…

• orcasfan

On this one, I agree with NMOO and Spiel. Where the original article was “amusing”, the intention was clearly to ridicule and demonize the Canucks scouting staff, especially Delorme. This is nothing new. This topic has been beaten flat over the years. And, on further (real) analysis, it has been repeatedly shown that the success of the Canucks drafting has not been as awful as suspected, and, has been, in fact, about average (compared to all the other teams).

So, if this was yet another sly attempt at demonstrating the failure of Delorme & Co, it was poorly done.

However, like some others here, I would love to see a real attempt made at just using stats in drafting, comparing the results to the actual performance of draftees in later years.

Of course, like everyone else, I want to see more success from Canucks’ drafting. I don’t know if the obstacle to greater success is really Delorme, or has it been the GM in place at that time. None of us really knows what happens in those draft strategy meetings, after all…

• orcasfan

It was poor at demonstrating the failure of Delorme because it is a critique of NHL teams’ scouting approach in general. I don’t think you fully grasped the content of the article if you think this was aimed at Delorme at all.

If anything it was a qualifying aside on the previous article which claimed Vancouver has a scouting problem. In reality, the NHL has a scouting problem.

• JCDavies

“And, on further (real) analysis, it has been repeatedly shown that the success of the Canucks drafting has not been as awful as suspected, and, has been, in fact, about average (compared to all the other teams).”

If you can produce these analyses, I would very much like to read them…

• Cam Charron

nateb123 has got it.

It isn’t a Canuck problem. It’s an NHL problem.

• Cam Charron

And you wonder why the stats crowd gets a bad rap? It’s idiotic statements like this. So the entire NHL has it completely backwards when it comes to drafting? Pity all of us poor fools who have had to watch a subpar product for all these years — who are these hidden gems who’ve been missed by a simple statistical approach to drafting?

Come on. The original article was amusing but not exactly, well, original. Do you really need this “method” to show that the Canucks drafting has been poor? Was this a revelation that we somehow didn’t know already? Wouldn’t we be able to tell simply by looking at the prospect pool we have or by the number of games played by our draftees, which is currently what’s usually used to judge the quality of scouting?

I don’t get why this has to be an all or nothing proposition, stats vs. guts and nothing in between. Statistical analysis makes a lot of sense in some, perhaps many cases. Possession numbers do correlate to what happens on the ice, in a very real sense. The Carlyle’s of the world who ignore or dismiss them are betraying a fundamental lack of knowledge of the game as it’s played on the ice and represented through this statistical method. But what this (intentionally lame) approach to scouting does is completely ignore all the other variables that go into predicting whether or not a draft pick will develop into a decent player, not to mention ignores significant data in the process. This isn’t a simple or flawed method, it’s not a really much of a method at all.

I would really hope you have a little more critical take on the value and meaning of numbers than this post indicates. The automated underwriting software algorithms that told us that the subprime mortgage instruments were a viable and sustainable market also had the sterling confidence of the financial world at one point too…

• Cam Charron

Yes, in fact.

I think that idiotic management decisions by 30 teams make a more inferior product. We could be watching a league of teams built like Chicago, or we could sit through more of the Leafs and Flames and other teams that play the way the game is “meant” to be played.

It isn’t only drafting. Watch the Portland Winterhawks. They never dump and chase. It just isn’t something they do. And yet they’re the exception in a sport determined to play conservatively.

My 100% opinion is that the NHL is a terrible product, and I’m not going to pay anymore on my cable to watch it next year. I’m interested in management decisions and I’ll always comment on those but I really have little interest compared to years ago when I never missed a game.

If it changes, who knows? At this rate? I’m convinced that 25 teams deliberately attempt to make the game less interesting.

• Cam Charron

Really? This is honestly your response? That the NHL is a terrible product and that every single team is managed poorly? Based on what? That’s a pretty sweeping statement. If you’re as interested in evidence-based analysis as you claim to be, how about a little more evidence than what appears to be your gut reaction to what you don’t like seeing.

There are many franchises that are well-run (the Canucks aren’t one of them), that bring new talent into the system and provide an entertaining product on a regular basis. There are many that do not. But it’s a rather odd (sad?) comment for someone who contributes to a hockey blog to make such a sweeping generalization on the basis of I have no idea what.

• Cam Charron

hell yeah! if you’ve outgrown the product, the product must grow. We the ones throwing cash around.

• Cam Charron

yes. and yes, again. and also, it’s a world problem; and it will be until all the thinkers decide that they would rather use formulas to make decisions instead of turning that over to human “experts” with allegiances to the corporates and plain old self-preservation. Hockey is the best. Never let it die!

• pheenster

@JeremyOK: Thanks for the explanation, and the second link. The second link puts the Canucks at below average, but not last place. So the metric is important.

I’d love to know what the better teams are doing differently.

@NM00: There’s something I don’t get. You have tons of time to criticize and argue with everyone on these boards, but if you are so dissatisfied, please, go ahead and run the simulation across all 30 teams, or come up with a different simulation and do it. I’d love to read your findings. Why all the negativity, and misplaced arrogance?

I for one think that the Canucks should be able to outdo Sham, no matter who’s in charge.

On other matters, is anyone besides me worried that Benning favored trading Seguin?

• pheenster

I don’t really care whether NM00 is right or wrong, but are you really defending this crappy “method”? Whether tongue-in-cheek or not, the blog writers are suggesting that this proves that all NHL drafting has it all wrong. It’s long been said of all sports drafting that it’s as much of a crapshoot as careful science — you are making educated guesses at how a 17-year-old skill set will transfer to competition with men 5-10 years older and in the due course of time. Why should we run a completely useless method on all 30 teams to prove that you’ve got as much luck drafting entirely on the basis of points as on anything else? The purpose of the articles seems to be to call into question the way scouting and drafting is done now. I’d love to see how analytics can actually make better predictions. None of what’s been written comes close to doing that. If the CA folks have something concrete to suggest I’d love to hear it.

• pheenster

On the bright side, the dialogue Rhys’ post has created would be useful if one of the CA bloggers wants to tweak the criteria and try another simulation that would hopefully look at all 30 teams.

However, the flaws in this should be pretty obvious.

Specifically the positional stuff as well as being cut off entirely from Europe.

Opposing NHL teams would love if Sham was an actual NHL GM since he’s ignoring more than half of the talent pool…

• Cam Charron

You are pretty dense. The point isn’t that analytics can provide a better method. It’s that the Canucks could save a tonne of money if they didn’t have a scouting department, and it wouldn’t affect their results by much.

The scouts talk about a careful science less as a data objective way and more of a “is he leadershippy enough” kind of a way. Few scouts do it right. I’d put Sham Sharron or a variation thereof against 30 scouting departments and every prospect pundit on the planet.

• Cam Charron

Yes, I am dense. You are completely correct. Why have a scouting department at all? Or coaches? Or trainers? Or really any staff at all?

As I said before, it shouldn’t be all or nothing. I’ve never said that we shouldn’t have any analysis — by all means include metrics to make data-driven decisions. And you are an idiot if you think every single scout out there are going on a hunch. If you want to prove that current scouting methods don’t work, don’t use a stupid straw man (“they are just doing it by gut instinct”), provide some actual evidence of what you see as a flaw in their method. If you think that scouting staffs don’t have their own metrics you are an idiot. Are they the right methods? Clearly not all strategies work (Gillis’ overager strategy for example). But do some actual research if you’re going to denigrate the current scouting methods rather than pulling a critique out of your ass. You’re doing exactly the same thing you accuse the scouts of doing, being obtuse and sticking to your guns over your gut feelings.

Your comments are so over the top as to be absurd. You’d really put a dumb-ass method that drafts no d-men or goalies or European players against 30 scouting departments and prospects on the planet?

Now I know you’re just trolling me…

• Cam Charron

Cam, any idea why Rhys’ article used a method that was only applicable in hindsight – the potato, sham sharron, was only able to select based on top CHL points-getter *out of the players who were between the canuck pick being selected and the next cancuks pick*

Such a thing is ony do-able if the draft order is known beforehand. I get why it was done, you want to avoid it drafting guys way, way above of where they were picked. By why try to make a point that a simple method – potato – would be better than a scouting department when that simple method is utterly inapplicable except in hindsight? Why not use simply points without the specified selection range or use ISS lists or somehting?

• Cam Charron

You could. I mean, Rhys’ method was imperfect due to the time he had and his computer capabilities. I’m sure if you ranked 17-year-old forwards by CHL points, the record would be even better.

But that’s time-consuming, and we’ve already proved our point here.

• Spiel

Wow.

You actually believe you have made a logical point here about the flaws within NHL front offices.

To be clear, I’m sure there are a number of flaws within the NHL.

But nothing in your piece is useful in regards to sheding light on that issue.

I’m not going to run a simulator for every draft.

But hopefully looking at one draft will show you the swiss cheese hole in your thinking and Rhys’ thinking if you were to run the simulator WITHOUT the benefit of hindsight and the work of NHL front offices.

I’ll use the 2004 NHL draft since it is 10 years old.

291 players were selected.

http://www.hockeydb.com/ihdb/draft/nhl2004e.html

If NHL front offices were to ignore all talent aside from draft eligible CHL forwards, here is who would be missed among 1st rounders alone:

Ovechkin, Malkin, Wheeler, Smid, Stafford, Radulov, Zajac, Meszaros, Schneider & Green.

Along with Booth, Grossmann, Goligoski, Krejci, Sekera, Emelin, Regin, Edler, Franzen, Porter, GRABOVSKI, Santorelli, Polak, Hunwick, Campoli, Rinne, Streit, Winnik, Clitsome & Hansen.

For argument’s sake (you are free to count the CHL forwards for an accurate number), let’s say that 146 of the 291 players drafted were CHL forwards.

Virtually every single one of these 146 forwards would have been amongst the top 146 draft eligble scoring forwards in the CHL.

Implicit in your “I’m sure if you ranked 17-year-old forwards by CHL points, the record would be even better” absurdity is that you would (pretty much) put up CHL draft eligible scoring forwards from #146 – 291 against ALL goalies, ALL defenseman and non-CHL forwards in your competition against NHL front offices.

The talent pool on Sham’s team would get destroyed by NHL front offices and it wouldn’t even be particularly close…

• Cam Charron

If I facepalm any harder, I’ll have a concussion.

This is an “all else being equal” approach. That’s the empirical method. Running the sim using Sham’s rules for all 30 teams would be like changing EVERY variable and then trying to discern what the underlying equation is. That’s not how it’s done. I don’t know what else to say to you except “get an education and stop wasting our time”.

• Cam Charron

Sigh.

You should understand that ANY method, such as your previous “proxy for using central scouting lists” has a chance to beat a FEW of the 30 teams in the NHL.

In no way does that validate “why a simple statistical method is more successful than a complex scouting program”.

Even though the statistical method is highly flawed and dependent on the same people that Cam is belittling…

Even if Rhys adjusted the flaw in his design so that it was not dependent on hindsight, it cannot be taken seriously without looking at ALL 30 teams.

Consider the example that Spiel provides as well as what would have happened in the Sedin draft using the “highest scoring forward in the CHL” method.

“get an education and stop wasting our time”.

Thank you for including the quotes for me…

• Cam Charron

If only your mind worked as quickly as your fingers. I agree that other teams should be examined (it would certainly be interesting) but given the current analysis, the title is still perfectly accurate.

As for your fascination with picking out specific examples (namely successful European players) as proof against this model, that is a mind-blowingly stupid approach. You could make the same argument with literally every draft program ever. This is what every person who complains about draft selections does: “Oh you picked X instead of Y, therefore your drafting is crap”. For a guy who keeps making claims about the overuse of hindsight in Sham’s model, it’s hilarious that you’re entire counter argument is based on NOTHING BUT hindsight.

• Cam Charron

“As for your fascination with picking out specific examples (namely successful European players) as proof against this model, that is a mind-blowingly stupid approach.”

Not only European players.

Also defenseman, goalies & imports playing in the CHL.

I happened to choose 2004.

You can use any recent draft and you will find the exact same thing.

“You could make the same argument with literally every draft program ever.”

Really…

It’s surprising that you can’t even comprehend that we agree on this point.

If Cam and co want to show that “a simple statistical method is more successful than a complex scouting program”, they actually have to, you know, show us that it is more successful.

They haven’t even come close to doing so.

All they are doing is limiting the talent pool from which they are making selections which opposing general managers would quite enjoy…

• Cam Charron

I don’t know if you have though?

I absolutely agree with you that it is likely that CHL points (maybe even league adjusted PPG or something like that) will be better than “expert” scouting. The thrust of Rhys’ piece and your article is fine. But you seem to both be using the specific ‘simple statistical method’ as a strong piece of evidence to support that, when it is inherently flawed. Why not be a bit more rigorous in your proof?

• Cam Charron

So instead of trying to actually make meaningful conclusions from peer-tested statistical analysis, you decide to half-ass it and call it “good enough”? Any moron from Hfboards knows that Vancouver has bad scouting. How about actually trying to prove it instead of whatever that mess of an article was?

• Cam Charron

i tried to give you 2 thumbs up.

• Cam Charron

I didn’t read this carefully enough. Oops! Concrete prediction were not the purpose of this exercise (that’s obvious!). I think I just got excited that you were going after NM00 and that appealed to me because he annoys me. Most likely because I’m like him; making opinions without reading things thoughtfully.

• Cam Charron

It takes a few minutes multitasking on work stuff to argue with delusional Canuck fans…

Surely you have a better solution than “go start your own blog” if you disagree with my criticisms or the best criticism made by another commenter about rule #4 of this simulation.

“I for one think that the Canucks should be able to outdo Sham, no matter who’s in charge.”

I suspect you would think differently if rule #4 were adjusted so that, you know, this simulation actually made sense AND all 29 teams were evaluated by the same method.

Surely you can understand the limitations of focusing the gaze on 1 team, ignoring how 29 other teams would perform based on Rhys’ criteria as well as the mind-numbingly obvious flaw that if all 30 teams used this method, defenseman, goaltenders and non-CHL forwards would fail to be drafted…

• andyg

“Hockey is fun to watch. It is why we do it.”

I think we have a few people on this site who have forgotten this! 🙂

• argoleas

Never has so much vapor been expended by so many over something so trivial.

For those that have criticized this statistical exercise/joke, you know you are welcome to run your own analyses for the other 29 teams then write about it, and even tweak the methods to add defensemen and goalies, or more if you so wish. We would look forward to reading them.

• argoleas

Testy lot, you all.

NM00, you must be exhausted.

On a different note, I’d like to see Sham’s take on the upcoming draft. Could set up a good piece for after the draft too…

Now go to bed NM00.

• JCDavies

@Charron

“The point isn’t that analytics can provide a better method. It’s that the Canucks could save a tonne of money if they didn’t have a scouting department, and it wouldn’t affect their results by much.”

After seeing the reaction of ‘hockey twitter’ and reading no less than three responses to a recent Steve Simmons article, I can’t help but wonder how comments such as this responsibly further the debate on the role of analytics in hockey.

• Cam Charron

and ya I’m drunk and have no life. so? what are you doing? reading this?

• Cam Charron

@Spiel: great and very reasonable discussion.

@NM00:

Several of us have already said that Sham is not a model for drafting. It’s a baseline, and a deliberately bad one! If the Canucks scouts using all leagues, etc., had been able to find better forwards, using metrics like GP and Pts, then you would have a point. But they didn’t. And again, they didn’t even if you include the Dmen and goalies that the Canucks did choose into Sham’s selections.

There’s only certain kinds of data available for prospects. I agree with you that the results aren’t conclusive, but they are suggestive. An interesting first step, to repeat myself. I’d love to see better stuff, so the invitation to put up still stands.

• Cam Charron

“If the Canucks scouts using all leagues, etc., had been able to find better forwards, using metrics like GP and Pts, then you would have a point. But they didn’t. And again, they didn’t even if you include the Dmen and goalies that the Canucks did choose into Sham’s selections.”

For starters, they did in 1999 as another commenter pointed out (Sedin, Sedin vs Brendl & Sparikyn IIRC).

Just because this arbitrarily begins in 2000 (the Delorme stuff can’t be taken seriously without access to his board) in no means validates this simulation.

Also, you absolutely cannot take the results of Rhys’ simulation seriously based on criterion #4.

IF he adjusted this so that Sham simply took the best scoring CHL forward left on the board every time the Canucks made a draft pick, then that would be a start.

Not nearly good enough, mind you, as more than 1 of the 30 teams in the NHL would need to be examined to take this seriously.

For example, if a slightly altered, hindsight free version of Sham beat 7 of 30 NHL teams, what would be your opinion of this model?

Using hindsight, I’m sure a few teams every single year (Canucks in 2007 being one example) would have been better off simply taking the highest player remaining on a publicly available ranking system.

That has nothing to do with “a simple statistical method is more successful than a complex scouting program”.

Now if Rhys or anyone else has found a system that can beat more than 15 of the 30 NHL teams at the draft table, THAT would be something…