Draft Theory: On Risk and Reward

money puck
May 18 2015 12:34PM

Hockey is full of time honored cliches: "We're going to take it one game at a time" "A two-goal lead is the hardest one to hold on to in hockey" "When you put the puck on net, good things happen" "We need to play a full 60 minutes" "We need to give 110%" "Your goalie has to be your best penalty killer". I'm not sure if it's that we love cliches, or the play-by-play guys and players love to repeat them.

In any case, one of the most overused cliche's as it relates to prospects is "He's a high risk, high reward player", but in many cases this term is pretty inaccurate. 

I'll explore the relationship between risk and reward when evaluating prospects after the jump.

On a recent post by Garret Hohl, of Jets Nation and Hockey Graphs, he reminded me of an excellent graph from Michael Schuckers in his study of the value of an NHL draft pick

schuckers

Here is his summary of the probability of a prospect playing over 200 NHL games by position and by draft round: 

Position Round 1 Round 2 Round 3 Round 4 Round 5 Round 6 Round 7 Total Overall Probability
F 35% 18% 17% 8% 10% 9% 3% 306 25%
D 43% 12% 14% 8% 6% 9% 8% 145 21%
G 33% 25% 8% 11% 9% 6% 8% 36 19%

As we can see, after the early first round picks, the probability of drafting an actual NHLer plummets.  

What Does this Mean and Why Do We Care?

By definition, other than the Connor McDavids and Jack Eichels of the world, virtually every prospect is a high risk pick, so unless you're talking about an elite sure thing, of course the a prospect carries a high amount of risk. 

So when we're looking at a particular prospect, it's not their inherent risk itself that is important, it is their risk as compared to the inherent, or average risk, of that draft position.  

Last month, I presented on the topic of prospect evaluation at the DC Hockey Analytics conference (DCHAC). The basic premise of the approach I've developed along with Josh Weissbock is pretty simple: if we can find a list of players (ie: a cohort) who most closely match the attributes of a particular prospect, we can use this information to help predict whether or not the prospect will be successful or not. It's a long read, but to summarize the results, in simulated re-drafts, teams that utilized cohort success rates along with NHL equivalency adjusted points/game scored 23% more points than the players actually drafted by the NHL teams. This was despite the test method selecting players actually drafted by NHL teams only 58% of the time, indicating that 42% of the players tested by our model never really had a chance at NHL success without teams invested in their development. 

Since this data was presented last month, we've refined our model considerably, and now our database returns results based on the closest comparables for a prospect using the following criteria: 

- League 

- Age (matched to actual birth date, not just birth year)

- Point production 

- Height

The output from our database is a list of comparable players, Prospect Cohort Success %, or PCS%, which is the number is the percentage of comparable players who played over 200 NHL games (Jonathan Willis explains why 200 games is important here), and PCS Pts/GM, which is the NHL points per game of the players within the cohort of comparable players. 

To give it a test drive, I queried the top undrafted players from the 2014 draft in terms of PCS%. Here were the results: 

Player Team League Pos Birthdate GP G A Pts P/GM Height PCS% PCS Pts/GM
VLADIMIR TKACHYOV Moncton Wildcats QMJHL LW 1995-10-05 20 10 20 30 1.50 179 38% 0.51
SEBASTIAN AHO Skelleftea SHL D 1996-02-17 21 1 4 5 0.24 178 30% 0.42
LAWRENCE PILUT HV71 SHL D 1995-12-30 22 0 4 4 0.18 179 28% 0.35
LUKE PHILP Kootenay Ice WHL C 1995-11-06 71 31 46 77 1.08 178 16% 0.45
JOE HICKETTS Victoria Royals WHL D 1996-05-04 36 6 18 24 0.67 173 15% 0.41
TYSON BAILLIE Kelowna Rockets WHL C/RW 1995-11-16 56 22 33 55 0.98 178 15% 0.39
TYLER HORE Sarnia Sting OHL D 1995-10-25 63 7 13 20 0.32 191 15% 0.20
IVAN NIKOLISHIN Everett Silvertips WHL C/LW 1996-04-12 72 18 41 59 0.82 175 13% 0.40
RILEY STADEL Kelowna Rockets WHL D 1996-01-16 63 10 25 35 0.56 178 12% 0.26
RIHARDS BUKARTS Brandon Wheat Kings WHL LW 1995-12-31 65 28 26 54 0.83 177 11% 0.35













The list above, is pretty interesting: 

- Tkachyov (or Vladimir Tkachev as you probably know him), of course, is the undersized forward who wowed the Oilers brass in a camp invite, and eventually signed a contract with the Oilers, only to have it later ruled as ineligible

- Sebastian Aho (the Swedish one) seems to be a consensus 2nd/3rd round pick in his second year of draft eligibility. Corey Pronman has him ranked as the 42nd best prospect in this draft class, and CSS has him ranked as the 13th highest international player.

- Joe Hicketts was signed by Detroit after the draft, and went on to beat out many highly touted defensemen to play for Team Canada at last year's World Junior tournament, winning a gold medal along the way.

- Rihards Bukarts is a key component of the Brandon team which just lost to the Kelowna Rockets in the WHL finals. More importantly, he's a Latvian with a pluralized first and last name, and I just can't get enough of them right now. 

All of these players had higher than average probabilities of success than the 7th round averages per the Schuckers study. In essence, these players had lower than average risk profiles, relative to players typically selected in later rounds.

In addition, each prospect had PCS Pts/GM rates that should have piqued the interest of NHL scouts. Some of these players, like Sebastian Aho, would have been an excellent pick in earlier rounds as well. Given his size he would most definitely been mis-characterized as a "high risk, high reward" pick. In fact, given his PCS% and PCS Pts/GM rates, Aho could have been accurately described described as a "low risk, high reward" pick relative to the other prospects on the board in the third round. 

This brings us to the case of the two most recently signed Canucks prospects: Jordan Subban and Mackenze Stewart. Here is the data from their draft years: 

Player Team Draft Year League Overall Pick Position GP G A P/GM Height PCS% PCS Pts/GM PCS N
Jordan Subban Belleville Bulls 2013 OHL 115 D 68 15 36 51 175 14% 0.24 22
Mackenze Stewart Prince Albert Raiders 2014 WHL 186 D 55 5 4 9 191 10% 0.135 120

What may come as a surprise to many, both Subban and Stewart's PCS% actually exceeded the probability of success of their draft position, although Subban much more so than Stewart. The following graph show's Stewart and Subban's PCS% relative to the probability of success of their draft position: 

PCS% subban stewart

As we can see, neither pick was particularly risky, relative to the draft position, although both players would correctly be considered to be long shots to make the NHL. Subban's PCS% is lower than one may expect, but the reality is that there really hasn't been many players his size who have made it. More importantly, it was significantly LESS risky than the defense picks in that round, historically. 

In terms of Stewart, he unquestionably has NHL size, but players like Stewart who haven't been able to demonstrate offensive aptitude haven't fared particularly well historically, so we see his PCS% only slightly higher than the historical rate of 7th round defense draft picks. 

The real difference between these two is the reward. The graph below shows the Pts/GM of 2014-15 NHL Defensemen as well as the PCS Pts/GM of Subban and Stewart: 

PCS pts gm

Subban's PCS Pts/GM was 0.24, while last year only 60 NHL defensemen produced at at that rate or better. If Subban can beat the odds and becomes an NHLer who produces consistent with the rate of his peer group, he'll be a very valuable player to the Canucks in the future. On the other hand, there were 170 NHL defensemen who produced at a rate of 0.135 pts/gm or better, meaning that if Stewart bucks the odds and finds his way to the NHL and produces consistent with his peer group, he'll roughly have the value of a replacement level defensemen, at least offensively.  

Conclusion

Through the use of PCS% and PCS Pts/GM we can estimate both the risk, relative to draft position, and reward, relative to the player's closest peers who have come before them. By definition, replacement level players are easy to find through free agency or the waiver wire, so it really makes little sense to use valuable assets on players with replacement level upside. 

The data presented at DCHAC indicates that when teams take these quantitative factors into consideration, there is the potential to improve their drafting significantly. Inherently, every draft prospect carries a high level of risk - that's a given. However, as we've seen from the players passed over in 2014, teams consistently pass over players who carry lower risk, and higher reward profiles. 

And with that in mind, the focus of future posts will be players worth targeting in the 2015 draft. Stay tuned.

E71500c6cbed1a488b0c1837d3e24bc3
Just another grammatically challenged blogger who loves spreadsheets. Follow me on twitter @MoneyPuck_
Avatar
#1 PB
May 18 2015, 01:31PM
Trash it!
1
trashes
Props
13
props

This is a really interesting and innovative approach. I get why you're using league, age and point production as proxies for quality of competition, development trajectory and output respectively but I still don't know why height is included as part of the criteria -- the complaints about size that we always hear notwithstanding it seems like a much more arbitrary criteria (why not use weight?) and as you've written about on this site before might simply be self-reinforcing.

Also, given the number of times we've heard that it's a waste to spend a first or second rounder on a goalie I'd be curious to hear a response from your colleagues about the success rate of keepers from the first graph.

Avatar
#2 Chris Withers
May 18 2015, 01:46PM
Trash it!
1
trashes
Props
9
props

Probably because you can change your weight either way with the right training regimen. Can’t change your height.

Avatar
#3 money puck
May 18 2015, 02:50PM
Trash it!
0
trashes
Props
20
props

@PB

Good question. The inclusion of height was based on earlier work that showed that height is a statistically significant variable in predicting future success. Weight wasn't included as we think this can be addressed by diet, training, development, etc.

Context is important though. Basically, if all else is equal, pick the bigger player. However, size can't make up for a lack of skill, so if you're looking at a small guy who produces a lot and a big guy that doesn't, you're almost always better off betting on the former.

http://canucksarmy.com/2015/4/11/the-draft-files-the-historical-cohort-based-approach-gets-its-sham-on

Avatar
#4 Fred-65
May 18 2015, 07:28PM
Trash it!
1
trashes
Props
5
props

I'v alway thought that trying to consider all 1st round picks as being equal and taking an average for heir likelyhood of success is skewed. I'd have thought that the top 5 wre removed from the averaging process would show even first round picks are pretty high risk of success

Avatar
#5 Sebguru
May 18 2015, 11:38PM
Trash it!
25
trashes
Props
5
props

Look i like the fact that there is a growing sophistication amongst fans and the hockey community.....but people must put everything into perspective...

One of life's great risks is putting information or knowledge to people who do not have the depth to understand it....disastrous results can happen....look at what can happen putting high powered advanced military equipment in the hands of uneducated dystopian religious fanatics in the middle east....

Firstly. It is a fundamental fact that nothing mankind has developed over thousands of years can predict future events. Science has not evolved to that. The fundamental premise of science is to look at the world and approximate its behaviour, then create a thesis, and a thesis only exists until it is broken, ie a single observable deviation to the thesis, and then it is essentially scraped.

Secondly. Past performance is NOT A PREDICTOR, of future performance. Anyone with formal statistical, or economic formal education will know this fact. I repeat, PAST PERFORMANCE IS NOT A PREDICTOR OF FUTURE PERFORMANCE. Be my guest and discuss this notion with any member of academia and business.

However, there are certainly CORRELATIONS between past and future events. But these should not be confused with PREDICTIONS!!!!! I accept that height has a corelation, that points scored in their draft year has a corelation. But these are NOT predictors.

Unless the ananlyitics community wants itself to be compared to Nostrademus,. they need to understand their body of knowlegde in much more greater detail.

Statistics is a way to simplify the complexities of life to the lowest common denominator, which is our brain. We cannot comprehend the world around us. Statistics removes these complexities in life so our prehistoric brain can try to understand what is going on. But removing these details, distances ourselves from reality. The more we remove, the less accurate our approximation becomes.

Statistics are indeed a useful tool, otherwise why does the world use them in the furst place. But you dont see statisticians running multi billion dollar companies, or countries, or thank god doctors are not statisticians, otherwise everyone is a walking dead man. Because if statitics could predict the future, the stats people would be runing this world. But an astute business man will refer to the statistical analysis, among many other inputs!!!!!!

Winston Churchill once said "Plans are of no importance, but planning is everything".....

Avatar
#6 Thefreshpots
May 19 2015, 12:36AM
Trash it!
2
trashes
Props
12
props

@Sebguru

I think you've completely missed the point of analytics...

Avatar
#7 Dirty30
May 19 2015, 12:57AM
Trash it!
0
trashes
Props
10
props

@Sebguru

You make excellent points except when you say that 'we cannot comprehend the world around us' when in fact we do make sense of the world by looking for patterns -- and that is what statistics tries to formalize.

However, the problem with stats is the belief that we can make predictions when in fact what they do is "reduce error". So the case for reducing error in making a draft pick, particularly when it's a non-renewable action in each situation.

Here's my query of the OP: given that prospects are ranked and that the selections each draft are not set -- which introduces variance that has little bearing on the prospects' value/success, might it even more fruitful to simply randomize all the picks in a draft and then test the equation on a random sample?

The reason I ask is that I wonder about things like team success -- and team capabilities in coaching and scouting and player development.

Would Sid have been successful on any team or more successful if he had gone to a different team? Can that information help to predict the success of other players?

The greats hing about stats is that they can make sense of aggregate data ... The bad thing is they are useless at predicting individual behaviour.

The history of top picks who have flopped and bottom picks who have flourished is a cautionary tale for anyone who is betting on the sure thing and hoping stats will reduce that error. In the aggregate it may, but individually, it may prove futile.

But it's a fascinating exercise!

Avatar
#8 seburu
May 19 2015, 09:06AM
Trash it!
3
trashes
Props
3
props

@Dirty30

look i agree, analytics/statistics has a role to play. but its role is reducing the randomness, NOT eliminating the randomness, as the latter is impossible.

Its when i hear people say "predict", that is just plain hogwash, we cannot "predict" what the future holds.

There are millions of factors that contribute to an individual's future, let alone a team's future.

Statistics and analytics always talk about sample size, sample size. But an individual is only one person.

Just because we calculate the mean of every billionaire on this plant their IQ, height, and weight, or anything else you can come up with, does not mean the opposite to be true. Ie if we look for someone with the same attributes can we predict they are either a billionaire or they will become one? Analytics and statistics don't work like that!

There is a difference between Cause and Effect, and Correlation. Correlation is something always found after the fact, and that is what advanced stats in hockey, baseball, etc is really all about. Advanced stats is not Cause & Effect. And the advanced stats community seem to lose sight of this in much of their language.

but just because their language irks me, doesn't mean i don't like it! Its fun! and its debate is what generates more advancement on the subject!!

Avatar
#9 benner49
May 19 2015, 10:58AM
Trash it!
0
trashes
Props
2
props

So this method argues that it could select slightly better players than the teams actually did 42% of the time...

Would be interesting to see what teams do well and what teams do poorly based on where they actually get to pick and put a rating on the scouting staffs. Should be possible... would be very interested to see how GMMG did.

Avatar
#10 PB
May 19 2015, 12:44PM
Trash it!
0
trashes
Props
2
props

@money puck

Thanks for this response. It makes sense -- though I also recall a discussion earlier posted by someone (maybe you but maybe another of the writers) that the success of taller players seemed a partly self-fulfilling prophecy; that is, taller players were more likely to see time in the NHL (if the 200-game-threshold is the measuring stick) in part because teams would tend to invest more in the bigger players under the assumption that they 'look the part'.

I suppose that doesn't take away from your point that height has shown to be a significant variable but since you're trying to innovate a better and more rationale way to select future players then perhaps a different fourth (or more) criteria might be useful to think through -- the other three are all measures of performance, development or competition but this one is just luck of the draw.

I'm loathe to get into the stupid WHY CAN'T ANALYTICS PREDICT THE FUTURE debates but suffice it to say that we've long used some kind of measure to predict the future performance of athletes (as long as there've been drafts) and I have zero idea why trying to make that less haphazard and arbitrary is so threatening to some. There was just a piece on ESPN about the overall success of point guards versus centers taken in the lottery in basketball and that you have a 65% chance of drafting a total bust if you take a Big Country Reeves or Michael Olowakandi rather than an 80% chance of success with a Kyrie Irving or Jason Kidd. And yet teams still drool over "big men" in the NBA. Different sport different stats, different predictions but the same truth holds -- drafting prospects based on intuition and hunches intensifies the already tricky art of evaluating what a 17/18 year old will become in a decade's time.

Avatar
#11 RandomScrub
May 20 2015, 02:30AM
Trash it!
0
trashes
Props
6
props

@seburu

Hi seburu,

I like that you're trying to bring another perspective, and I think you articulate some of your thoughts well, but I don't know who your points are aimed at, as I don't believe anyone here fails to understand the notion that an individual can buck any trend: there are outliers in any statistical analysis.

You're of course correct that correlation is not causation, just as inductive reasoning is not deductive reasoning. We cannot know for sure that the sun will rise tomorrow, just because it has every single day before... but that being said, I'd feel pretty comfortable betting money that it will.

Now, you mention as much yourself, but let me restate as well, that your suggestion that we should not use the word "predict" is nitpicking semantics.

There's never been any implication, I feel, or certainly I doubt any reader would feel there has been one, that when projecting success for hockey prospects, "predictions" are intended to be stated as unfailing and absolute - a sure indication of future results. A prediction in this context inherently implies "based on past trends, it's much more likely that A than B".

I don't think any analytics advocates in the hockey blogger community deny that traditional scouting and analytics work best in harmony when attempting to make the best possible drafting decisions, and no one in their right mind would anticipate a 100% record of "success" (which one would have to quantify anyhow) in drafting, regardless of the methods employed. . So... yeah. I'd suggest we just let money puck do his awesome work and enjoy it, and focus our discussion on the content, since these sort of semantics arguments are really ultimately unnecessary, when you think about it.

Cheers.

Comments are closed for this article.