Which Service Has the Most Outlandish 2017 NHL Draft Rankings?

The 2017 NHL Draft has had it’s fair share of shade thrown on it: it’s top prospects aren’t generational enough; it’s second tier of prospects are apparently basically interchangeable; it’s a generally “weak” draft (though, come on, enough with that). Earlier in the season, ESPN’s Corey Pronman called it potentially the weakest draft since 2002. Ouch.

In concert with these criticisms is the fact that the rankings by the various services are all over the place. Being one of those spreadsheet nerds, I decided to measure the volatility of the draft rankings statistically, and see whose rankings are in line with the crowd, and whose are the most outlandish – the hottest of takes, if you will.

While this began as kind of a “fun” exercise, it does have some applicability. It’s hard for any of us to familiarize ourselves with prospects that might go in the first round, let alone the hundreds of others that could potential be picked on day two. Even though we like to formulate our own opinions on players, we rely on ranking services to keep us informed on who belongs in what general vicinity, since they have experience (and the budget) to watch and keep track of a lot more hockey players. Even the most celebrated statistical models perform substantially better when they incorporate an ordered ranking of prospects to choose from.

That’s why it’s so important to know which services are out to lunch. It’s important to have variety of course, and no two lists should look exactly alike, but when some lists are so far from the norm, you naturally begin to wonder two things: one, do they really know what they’re doing? And two, are they as concerned with getting attention as they are with supplying an accurate list?

In terms of distance from the norm, my natural assumption was the McKeen’s was going to be right up near the top of the list. For the last couple of years, they have insisted on placing the consensus second ranked player over the consensus first ranked player (though Hischier over Patrick is admittedly considerably less controversial than Laine over Matthews), and jamming a centre that belongs in the middle of the first round into their top three (Michael McLeod/Michael Rasmussen). Coincidentally, the aforementioned Corey Pronman has made a habit of doing the former as well, and the two are currently the only mainstream lists that have Hischier on top.

Before we look at the rankings themselves, it’s important to make a distinction here. The purpose of this article is to find out which service’s list is the most outlandish with regards to the average between them. It doesn’t measure which rankings have had the most success in the past, and as such it’s possible that some lists, in deviating from the norm, are actually a better bet to be correct in the future. Though from a logical perspective, it would seem that the consolidated version of a dozen mainstream rankings would have a fair amount of accuracy of player potential, utilizing a large amount of scouting experience while also filtering out bias. Saying for sure which rankings have a better history of projecting prospects is certainly a topic for future research, and one that I hope to tackle before the draft – but it isn’t covered here.

Let’s get into it then.

This colourful chart tells us a whole lot about the various services just by looking at the colours themselves. Before we even delve into the numbers, you can get a pretty good indication of what the results are going to be by knowing these two guidelines: lighter colours mean more run-of-the-mill rankings, while darker colours indicate outlandishness.

Here’s how the above chart was formulated. Each player was given an average rank based on twelve different lists (shown in the third column). The next column shows the range between the high and low rank for that player, and the next shows the standard deviation. The colour formatting in each of the subsequent columns is based on how many standard deviations each rank is from the player’s average, with green representing higher than average and red indicating lower than average.

This is important because not all players should be treated equally with respect to where their expected position is. The fact that Gabriel Vilardi is ranked 10th by The Draft Analyst, roughly seven spots away from his average, is then considered more of a stretch than Craig Button ranking Urho Vaakanainen 57th, nearly 30 spots away from his average rank, because of their respective standard deviations.

Scanning across the rows left-to-right, you can see the variation in individual players. Many of the players in the presumptive top five show a fair amount of stability (and thus lighter colours) across their rows, given that they’re all ranked pretty close to where they’re expected to go. After that though, all bets are off, and left-to-right scans take on a feel of sheer randomness at points. This volatility within individual players is also represented by the standard deviation, so we can see numerically how severe some case are. The worst in this sample is Nikita Popugayev, who is ranked as high as 17th (McKeen’s) and at low as 67th (Button) for a ridiculous range of 50 spots and a standard deviation of 14.6.

Player volatility is interesting, but analyzing service volatility was our intended objective, so let’s take a look at that instead. To measure this, I assigned a value to each player-rank (the number of standard deviations between each rank and the player’s average), and then averaged those values for each service among the top 31 prospects (according to the consolidated rankings). Here’s what I got:

The average of the projected first round draft prospects is 0.74, meaning that each ranking diverges from the average by approximately three quarters of a standard deviation. Services with averages below that number are more in line with the average, while the higher the number, the hotter the takes of the respective service.

Before we get to the hot takes, note that one of the most conservative services is TSN’s Bob McKenzie. This is probably not a coincidence, given the fact that McKenzie’s rankings themselves are in fact consolidations of surveys he puts out to scouts and executives on various NHL teams. His list already has the same benefits of my consolidated lists: varied expertise with at least some built-in bias protection. The only list closer to average than McKenzie’s was that of Peter Harling for Dobber Prospects, whose takes are so room temperature that they themselves could have been consolidated, though they weren’t noted as such.

Many of the other ones fall within the range of the overall average, including mainstream services like Future Consideration, McKeen’s Hockey, HockeyProspect.com, and ISS Hockey, and media rankings like Jeff Marek (Sportsnet) and Ryan Kennedy (The Hockey News). All six of these fall within 0.1 standard deviation of the average.

Now we get to the fun stuff: there were four lists with an average standard deviation of 1.00 or higher. The Draft Analyst was a very interesting case, because it started out straight up kooky, ranking Casey Mittelstadt as the number one prospect, followed by Hischier at two and Patrick at three. There are a couple of scorching hot takes right there, and there were a few other odd choices early on, like having Eeli Tolvanen at four or Cale Makar at 18 – but other than other, the list isn’t much more outlandish than the closer-to-average ones.

Corey Pronman (ESPN) is next, and he gets off to a hot start by having Nico Hischier in the one-hole, though he is followed by Patrick thankfully. Tolvanen appears at four on his list as well, which, while not popular, isn’t surprising given that he had Tolvanen at 2 on his previous rankings and apparently had to convince himself not to put the Finnish sniper first overall. Some of his placements, like Tolvanen at 4, Kostin at 5, or Heiskanen at 16 seem more outdated than anything else, and might be adjusted closer to average by his final rankings, while having Mittelstadt as far back as 11th really wouldn’t have been normal at any point in the season. He gets dinged a little bit for a pair of choices that I actually really like – draftalytics approved prospects Kailer Yamamoto at 10 and Nick Suzuki at 12, while keeping draftalytics maligned Michael Rasmussen in the middle of the first round. Sometimes it’s good to be outlandish.

Next is Craig Button (TSN), whose already popular lists have probably gained a little bit more credibility around these parts recently because of his 2014 assessment of Jake Virtanen (he had him ranked 41st prior to the draft) which is aging pretty well. However, a pattern seems to have formed here. Like Virtanen, there are several other prospects that Button is not as high on as others. When he doesn’t see a player as a potential first round pick though, he doesn’t simply bump them out and into the 30’s – he punts them into the 40’s, 50’s, or even 60’s. There are already several examples of Button purging popular players in his most recent rankings.

Button’s average of 1.13 doesn’t take the cake though, as there is one more service that simply won’t be beat when it comes to hot takes.

Draftbuzz Hockey, congratulations!

Draftbuzz’s average standard deviation of 1.20 is an impressive feat to be sure, especially since they still had Hischier and Patrick as their top two. Surely if they really wanted to get some attention, they could have dropped Patrick down to 5th, or something crazy like that. The rest of their first round is is just an absolute yard sale, with Gabriel Vilardi at 10, Mittelstadt at 13 and Timothy Liljegren at 18, bumping very good 10-15 picks like Lias Andersson and Elias Pettersson over their heads into the top five. After the top two, only two of the remaining 29 players are even within 3 spots of their respective averages (Cody Glass – 10.9, 8; and Shane Bowers – 27.7, 30).

While Draftbuzz’s rankings are kooky enough on their own, the really problematic part is this: their rankings were relatively normal a month earlier. Between releases on rankings on March 12th and April 30th, just a month and a half apart, the Draftbuzz staff completely changed their minds on almost every prospect on their enormous list. Players routinely moved 20, 30, even 40 spots or more in the span of seven weeks. Hardly anyone is even in the same group of 10 as they were before.

(Note: Draftbuzz’s draft board has been removed at the request of its site owner.)

The troubling thing about this is that there are only seven more weeks to go until the NHL Entry Draft, which makes you wonder how much more these rankings could change in that time. One would think that scouting services, especially ones that are charging money, should probably have a lot firmer grasp on where they see this year’s draft prospects this close to the big day. At this point, you’d be wise to take this particular service’s rankings with a very large grain of salt.

If you’ve read this far, I assume you reeeeally like draft rankings, so here’s a little bit of bonus information for you: Canucks Army’s Prospect Profiles will start rolling in the coming weeks, and we’ve all but nailed down our top 100 prospects for the upcoming draft. Given that I’ve got our list virtually ready to go, I thought I’d see how outlandish we are. In terms of how close Canucks Army’s list is to the consolidated top round, we have an average standard deviation of 0.92 – hotter than your average draft ranking takes, but not too hot that you’re wondering if we’ve lost our marbles.

A good portion of our variability comes from two players in this list of 31 who we’ve ranked considerably lower than most other rankings, and two players that we a fair bit higher on. Take another glance at the ranked chart above and hit up the comments to guess who those four players might be, but you’ll have to wait for our profiles to hit the blogosphere before you can start seeing our list in all its glory.