28
Photo Credit: Hickling Images / Dan Hickling

Which Service Has the Most Outlandish 2017 NHL Draft Rankings?

The 2017 NHL Draft has had it’s fair share of shade thrown on it: it’s top prospects aren’t generational enough; it’s second tier of prospects are apparently basically interchangeable; it’s a generally “weak” draft (though, come on, enough with that). Earlier in the season, ESPN’s Corey Pronman called it potentially the weakest draft since 2002. Ouch.

In concert with these criticisms is the fact that the rankings by the various services are all over the place. Being one of those spreadsheet nerds, I decided to measure the volatility of the draft rankings statistically, and see whose rankings are in line with the crowd, and whose are the most outlandish – the hottest of takes, if you will.

While this began as kind of a “fun” exercise, it does have some applicability. It’s hard for any of us to familiarize ourselves with prospects that might go in the first round, let alone the hundreds of others that could potential be picked on day two. Even though we like to formulate our own opinions on players, we rely on ranking services to keep us informed on who belongs in what general vicinity, since they have experience (and the budget) to watch and keep track of a lot more hockey players. Even the most celebrated statistical models perform substantially better when they incorporate an ordered ranking of prospects to choose from.

That’s why it’s so important to know which services are out to lunch. It’s important to have variety of course, and no two lists should look exactly alike, but when some lists are so far from the norm, you naturally begin to wonder two things: one, do they really know what they’re doing? And two, are they as concerned with getting attention as they are with supplying an accurate list?

In terms of distance from the norm, my natural assumption was the McKeen’s was going to be right up near the top of the list. For the last couple of years, they have insisted on placing the consensus second ranked player over the consensus first ranked player (though Hischier over Patrick is admittedly considerably less controversial than Laine over Matthews), and jamming a centre that belongs in the middle of the first round into their top three (Michael McLeod/Michael Rasmussen). Coincidentally, the aforementioned Corey Pronman has made a habit of doing the former as well, and the two are currently the only mainstream lists that have Hischier on top.

Before we look at the rankings themselves, it’s important to make a distinction here. The purpose of this article is to find out which service’s list is the most outlandish with regards to the average between them. It doesn’t measure which rankings have had the most success in the past, and as such it’s possible that some lists, in deviating from the norm, are actually a better bet to be correct in the future. Though from a logical perspective, it would seem that the consolidated version of a dozen mainstream rankings would have a fair amount of accuracy of player potential, utilizing a large amount of scouting experience while also filtering out bias. Saying for sure which rankings have a better history of projecting prospects is certainly a topic for future research, and one that I hope to tackle before the draft – but it isn’t covered here.

Let’s get into it then.

This colourful chart tells us a whole lot about the various services just by looking at the colours themselves. Before we even delve into the numbers, you can get a pretty good indication of what the results are going to be by knowing these two guidelines: lighter colours mean more run-of-the-mill rankings, while darker colours indicate outlandishness.

Here’s how the above chart was formulated. Each player was given an average rank based on twelve different lists (shown in the third column). The next column shows the range between the high and low rank for that player, and the next shows the standard deviation. The colour formatting in each of the subsequent columns is based on how many standard deviations each rank is from the player’s average, with green representing higher than average and red indicating lower than average.

This is important because not all players should be treated equally with respect to where their expected position is. The fact that Gabriel Vilardi is ranked 10th by The Draft Analyst, roughly seven spots away from his average, is then considered more of a stretch than Craig Button ranking Urho Vaakanainen 57th, nearly 30 spots away from his average rank, because of their respective standard deviations.

Scanning across the rows left-to-right, you can see the variation in individual players. Many of the players in the presumptive top five show a fair amount of stability (and thus lighter colours) across their rows, given that they’re all ranked pretty close to where they’re expected to go. After that though, all bets are off, and left-to-right scans take on a feel of sheer randomness at points. This volatility within individual players is also represented by the standard deviation, so we can see numerically how severe some case are. The worst in this sample is Nikita Popugayev, who is ranked as high as 17th (McKeen’s) and at low as 67th (Button) for a ridiculous range of 50 spots and a standard deviation of 14.6.

Player volatility is interesting, but analyzing service volatility was our intended objective, so let’s take a look at that instead. To measure this, I assigned a value to each player-rank (the number of standard deviations between each rank and the player’s average), and then averaged those values for each service among the top 31 prospects (according to the consolidated rankings). Here’s what I got:

The average of the projected first round draft prospects is 0.74, meaning that each ranking diverges from the average by approximately three quarters of a standard deviation. Services with averages below that number are more in line with the average, while the higher the number, the hotter the takes of the respective service.

Before we get to the hot takes, note that one of the most conservative services is TSN’s Bob McKenzie. This is probably not a coincidence, given the fact that McKenzie’s rankings themselves are in fact consolidations of surveys he puts out to scouts and executives on various NHL teams. His list already has the same benefits of my consolidated lists: varied expertise with at least some built-in bias protection. The only list closer to average than McKenzie’s was that of Peter Harling for Dobber Prospects, whose takes are so room temperature that they themselves could have been consolidated, though they weren’t noted as such.

Many of the other ones fall within the range of the overall average, including mainstream services like Future Consideration, McKeen’s Hockey, HockeyProspect.com, and ISS Hockey, and media rankings like Jeff Marek (Sportsnet) and Ryan Kennedy (The Hockey News). All six of these fall within 0.1 standard deviation of the average.

Now we get to the fun stuff: there were four lists with an average standard deviation of 1.00 or higher. The Draft Analyst was a very interesting case, because it started out straight up kooky, ranking Casey Mittelstadt as the number one prospect, followed by Hischier at two and Patrick at three. There are a couple of scorching hot takes right there, and there were a few other odd choices early on, like having Eeli Tolvanen at four or Cale Makar at 18 – but other than other, the list isn’t much more outlandish than the closer-to-average ones.

Corey Pronman (ESPN) is next, and he gets off to a hot start by having Nico Hischier in the one-hole, though he is followed by Patrick thankfully. Tolvanen appears at four on his list as well, which, while not popular, isn’t surprising given that he had Tolvanen at 2 on his previous rankings and apparently had to convince himself not to put the Finnish sniper first overall. Some of his placements, like Tolvanen at 4, Kostin at 5, or Heiskanen at 16 seem more outdated than anything else, and might be adjusted closer to average by his final rankings, while having Mittelstadt as far back as 11th really wouldn’t have been normal at any point in the season. He gets dinged a little bit for a pair of choices that I actually really like – draftalytics approved prospects Kailer Yamamoto at 10 and Nick Suzuki at 12, while keeping draftalytics maligned Michael Rasmussen in the middle of the first round. Sometimes it’s good to be outlandish.

Next is Craig Button (TSN), whose already popular lists have probably gained a little bit more credibility around these parts recently because of his 2014 assessment of Jake Virtanen (he had him ranked 41st prior to the draft) which is aging pretty well. However, a pattern seems to have formed here. Like Virtanen, there are several other prospects that Button is not as high on as others. When he doesn’t see a player as a potential first round pick though, he doesn’t simply bump them out and into the 30’s – he punts them into the 40’s, 50’s, or even 60’s. There are already several examples of Button purging popular players in his most recent rankings.

Button’s average of 1.13 doesn’t take the cake though, as there is one more service that simply won’t be beat when it comes to hot takes.

The Most Outlandish Rankings Belong To…

Draftbuzz Hockey, congratulations!

Draftbuzz’s average standard deviation of 1.20 is an impressive feat to be sure, especially since they still had Hischier and Patrick as their top two. Surely if they really wanted to get some attention, they could have dropped Patrick down to 5th, or something crazy like that. The rest of their first round is is just an absolute yard sale, with Gabriel Vilardi at 10, Mittelstadt at 13 and Timothy Liljegren at 18, bumping very good 10-15 picks like Lias Andersson and Elias Pettersson over their heads into the top five. After the top two, only two of the remaining 29 players are even within 3 spots of their respective averages (Cody Glass – 10.9, 8; and Shane Bowers – 27.7, 30).

While Draftbuzz’s rankings are kooky enough on their own, the really problematic part is this: their rankings were relatively normal a month earlier. Between releases on rankings on March 12th and April 30th, just a month and a half apart, the Draftbuzz staff completely changed their minds on almost every prospect on their enormous list. Players routinely moved 20, 30, even 40 spots or more in the span of seven weeks. Hardly anyone is even in the same group of 10 as they were before.

(Note: Draftbuzz’s draft board has been removed at the request of its site owner.)

The troubling thing about this is that there are only seven more weeks to go until the NHL Entry Draft, which makes you wonder how much more these rankings could change in that time. One would think that scouting services, especially ones that are charging money, should probably have a lot firmer grasp on where they see this year’s draft prospects this close to the big day. At this point, you’d be wise to take this particular service’s rankings with a very large grain of salt.

Bonus

If you’ve read this far, I assume you reeeeally like draft rankings, so here’s a little bit of bonus information for you: Canucks Army’s Prospect Profiles will start rolling in the coming weeks, and we’ve all but nailed down our top 100 prospects for the upcoming draft. Given that I’ve got our list virtually ready to go, I thought I’d see how outlandish we are. In terms of how close Canucks Army’s list is to the consolidated top round, we have an average standard deviation of 0.92 – hotter than your average draft ranking takes, but not too hot that you’re wondering if we’ve lost our marbles.

A good portion of our variability comes from two players in this list of 31 who we’ve ranked considerably lower than most other rankings, and two players that we a fair bit higher on. Take another glance at the ranked chart above and hit up the comments to guess who those four players might be, but you’ll have to wait for our profiles to hit the blogosphere before you can start seeing our list in all its glory.

  • Killer Marmot

    Rather than using average and standard deviation, it might be better to use robust measures such median and MAD (median absolute deviation).

    That way, a particularly outlandish ranking doesn’t skew the result much.

  • apr

    I see Phoenix as a good trade partner, as they probable one of the top D. You can still get Glass, Mittlesdat, or Villardi at 7 and perhaps get another 2nd rounder – or flip the Nucks 2nd rounder with Phoenix’s late first.

  • Dan-gles

    Kind of a tangent but somewhat related. I read Canucks army everyday. They keep talking about the depth of our prospect pool. I can agree it’s vastly improved from when benning took over. But it’s not very good if you compare it to arizona, buffalo, Toronto or Winnipeg for instance. I find it concerning we don’t have any elite talent (game breakers) on the horizon. Until then I feel we may be in troubled waters.

    • kablebike

      It would appear that the Canucks are probably 4-6 years of draft development behind those teams you named. In 4-6 years, the Canucks will be in a similar situation and those teams should be a step above them with players coming into their own (or not).

  • HockeyTruther69

    Up – Robert Thomas because JD loves Matchbox 20
    – Tie between Jason Robertson, Nick Suzuki, Kailer Yamamoto, and Tyler Inamoto because you are outside the systematic pressures of the hockey industry
    Down – Sean Dhooghe because biometrically he has the least amount of heart in the draft
    – Cameron Crotty because he’s bad

  • truthseeker

    The problem with all of this is that fans seem to think these analysts rankings are almost gospel. Especially Button and McKenzie. And if a team goes off those analysts pick than it’s an excuse to chastise their team’s management. Canuck fans especially; the kings of second guessing and hindsight bias.

    But the teams are going far beyond the information the analysts use. They will put as much focus into the interviews of these players to gain insights on their character and ability to adapt to a professional mindset, as they do their goals and assists and fancy stats. I believe the “psych profiles” (for lack of a better phrase) will be even more important to a teams choice, than the numbers. (for each chunk of talent at whatever position obviously….nobody’s saying take a guy ranked by the analysts at number 175, at number one).

    So people will say they should take this guy or that guy at this position and if they miss that player then they’ve screwed up…..but the teams don’t see it like that. They will have a bunch of different options at whatever pick they have that will make them happy.

    • Preaching to the choir here. I was a Psych major in University. The Combine interviews are one of the most interesting features of the draft for me. I’d love to be involved in that sort of thing.
      Surely numbers should play a very important role, but I’m reminded of the scene early in the Moneyball book where the A’s staff is going through potential draft choices and just chucking out bad apples without even looking at the stats. Just not worth the risk to try to develop guys like that when there are so many other options available.

      • truthseeker

        Yeah no doubt. I bet it turns into a bit of a game as well. Cause you know the players are being briefed on the interviews and how to “give good answers”, so then the teams need to find away around that and on and on…lol.

        Of course…then we need to see how all the teams rank in terms of their psych evaluations. Haha….I’m sure some teams are much better at it than others.

  • LTFan

    First of all thanks to Jeremy for putting this together. A lot of information to digest and go through. This is one blog that I will refer back to as draft day gets closer. Teams use their own set of standards rate players which is why some players are drafted higher than expected and others lower. The proof of how well each teams system or standards of ranking players shows up several years later if the player is playing in the NHL.

    Still with the draft and rankings, it is my opinion the draft age should be one year later (19). A lot can happen in that year as players develop and mature. Except for the top 4 or 5 each year – it is a roll of the dice, so to speak, if a player will develop into an NHL player or not.

  • Adamemnon

    Thanks for the article Jeremy, great job, thoroughly researched and reasoned as always. I’m determined to give the handful of good writers at Canucks Army credit where it’s due. I’ve been reading you guys for a couple of years now, and kudos to all the great talent that has passed through the site. I think you by and large do an awesome job.

    I have been a bit disheartened by some of the articles by the newbies not being as up to the high standards you set yourselves, but I always scan the comments section, at least as much as my stomach can take, and am appalled by how ignorant and a majority of the commenters’ opinions are. At least an article like this one seems to attract a more sophisticated readership, and I’m heartened to note the comments are all intelligent and intelligible.

    Anyhow just wanted to pipe up and say great job, I really appreciate reading articles like this, and don’t let the (at times overwhelming number of) trolls get you down! There are folks out there that really dig what you’re doing!

  • justmyopinion

    As already commented by a few other posters – interesting article. I’m especially looking forward to your next article comparing which rankings have a better history of projecting prospects as actual NHL’ers. Now that would be fascinating because at the end of the day, isn’t that the reason why we have all these draft surveys? Much thanks in advance!

  • kagee

    Canucks would be wise to choose Kristian Vesalainen, his value went down when he went to a different age bracket, but he is a teenager among men, imo he’ll be this year’s Rantanen.

  • rakish

    If you study the history of the draft, you will find that the run-of-the mill picks are terrible. For instance, had the Sabres chosen the top ranked player on the NA CSS board for 6 years (2005-2010), they would have 2 or 3 NHL players from those 6 years.

    I started following the draft more closely in 2014. Button had an excellent draft, mine of good, Hockey News’ has fallen off, partly because Bennett’s career has gone south, and partly because Holmstrom got hurt this year. In last, most years, has been Tim Murray, because from my view, the interview process wasn’t helping him much.

    The 2015 draft was a learning experience for me because I believed in something that was untrue. I wouldn’t have learned this if I hadn’t been looking at my results very closely. My draft contest used Winnipeg in 2015 (because everyone would have taken Eichel) and Winnipeg has killed the rest of us, so far.

    For 2016, the results are early, but I’m happy with where I am, Button, McKenzie, Pronman, Tim Murray, not so much

    My point is draftbuzz’ methodology could be good (my guess is not, I would take Liljegren at 2), but it doesn’t have to do with outlandishness. Whether a methodology works depends on results. So I invite you to put together a draft board and enter my draft contest, and we’ll see how you did over the next few years. There are no stakes in Mom’s Basement Stakes, so you have nothing to lose. http://www.45b.us

  • on2ndthought

    Thanks for this great work. This is a very interesting chart. What I’d really like is to compare it to a ‘re-draft’ done in 5 or 6 years.

    AAMOF, I wish I had the internet chops to put together such a comparison for the past dozen years, just to see which prognosticators are; a) more accurate (to what the GMs actually do), and b) more insightful (as to actual long term value of players).