29

In Defence of Consolidated Rankings

I’ve been awfully busy these past few months, between prepping for the NHL Draft, developing and assessing metrics, and watching a ton of hockey, as well as some personal matters here and there. So I haven’t been writing as much as I usually do, and when I do, it’s typically something that I think is important to share with the hockey world.

So perhaps you can imagine my surprise yesterday when I was accused of trying to garner views and “sell followers”. Granted, it was pretty clear that I had pissed off the source of the comments with my Friday article on the variability of mainstream draft rankings. In analyzing which services deviated the most from average rankings, I was admittedly a little harsh on one service in particular – Draftbuzz – and site founder Anthony Mauro let me know about it on Twitter.

Advertisement - Continue Reading Below

Now, I’m not active on Twitter, so I wasn’t exactly in a position to defend myself – big up’s to my co-writers JD Burke and Ryan Biech for sticking up for me in that regard – but there are a couple of points I’d like to touch on, and this is the space in which I’m able to do that.

What follows is a few of those thoughts, centred around some of the feedback and accusations.

Mea Culpa: I Published Subscriber Content

I posted an image of Draftbuzz’s top 100 ranked prospects, which is a no-no, and something that I should have considered. Draftbuzz only posts a top 31 on their website, and users pay extra for longer lists.

Given that I had a longer list, you can probably guess that I myself am a subscriber. I’ve given some of my hard earned money to Draftbuzz, and as a consumer I am within my rights to raise concerns about the product – such as massive month-to-month swings. While I’ve had conversations with others about where the line is when criticizing the work of others in the business, I’ve always thought of this as one of the advantages of writing for a blog instead of a newspaper or mainstream outlet: there’s quite a bit more leeway for honesty.

Advertisement - Continue Reading Below

As for this accusation:

I don’t believe I’m being unfair to any other service. Of the 12 lists that I included in Friday’s article, four of them (HockeyProspect.com, McKeen’s, ESPN, and of course Draftbuzz) require paid subscriptions to see full lists – though each of them outside of ESPN provides at least a top 31 for free. In some cases, such as noting that a subscription-based service has a possible first round player ranked in the 40’s, I may be flirting with the line a little bit. I may have to be slightly more careful of this in the future, though if you’ve seen the other Consolidated Rankings I’ve published, I don’t typically provide where the players are ranked by individual services anyway.

I do understand of course that free giving away something that Draftbuzz is trying to sell is, to be blunt, a dick move. Which is why I had zero problems with removing the list upon request. At this point, Mr. Mauro thanked us for removing the draft board.

I was quite pleased at this point with Draftbuzz taking the high road in spite of the fact that I’d criticized their work; I figured this whole thing was put to rest.

Advertisement - Continue Reading Below

Then a couple of hours later I was being called trash and a hack. Not sure what happened there.

The Aggregates Are For You, Not For Us

This one had me going a little bit. The implication here is that we, purporting to be knowledgeable about prospects, have to aggregate the opinions of others in order to rank our players.

The insinuation that we can’t come up with our own opinions is a bit insulting, for starters. It’s also entirely untrue. Exhibit A might be the fact that we watch a LOT of f#$&ing hockey (though we don’t get to watch many games in person, mainly for budgetary reasons – if someone wants to pay us to travel and watch games, we’d be good with that). We also have access to a bevy of statistical metrics (some publicly available, and some that were created in-house) that have some scientific merit but often go ignored by other rankers.

Advertisement - Continue Reading Below

I noted at the bottom of the article that we are going to be rolling out a top 100 starting in a couple of weeks. I went on to explain how our list deviates from the consolidated average of the mainstream rankings more than the majority of the ones used to make the aggregation. This should be a clear indicator that we don’t use the consolidated rankings to generate our list.

The consolidated rankings are in fact not for us, they are for you: the readers. Another thing that I mentioned in Friday’s article was that I think there’s value in aggregating mainstream rankings, because you get the benefit of a whole lot of scouting experience, while also accounting for outliers and bias that may creep in if you focus solely on the rankings of a single service.

We also understand that the majority of hockey fans don’t have the time to familiarize themselves with more than a handful of prospects, and also probably don’t have the time to analyze which services have the best track records, or when a service is making a particularly bold statement on a player.

The consolidated rankings provide a service to those fans that want to know how the scouting world as a whole orders the available players. That’s probably why there has been a generally positive response to the previously published articles regarding the consolidated rankings. And it’s probably why six other editors on the Nation Network deemed that those articles had enough value that they okayed me publishing them on their individual sites.

Consolidation versus Consensus

This criticism didn’t come from Anthony Mauro, but I thought it was worthy of discussion regardless:

There’s a reason that I refer to the aggregated rankings as Consolidated, and not Consensus: but I’m not an idiot and obviously there is no consensus, that’s why I’m aggregating in the first place.

Perhaps the most comical notion about this criticism is that fact that it’s exactly what Bob McKenzie does. The Godfather himself sure isn’t out there scouting kids. He’s using his incredible reach and network of contacts to find out what NHL teams think of players before consolidating them into his draft rankings – and I’m pretty sure nobody is telling him that no one cares about aggregating opinions or is giving him a hard time for it. That’s an awfully broad brushstroke to paint with.

Deviation is Not an Insult

A couple of remarks were made that I was using the high volatility of Draftbuzz’s list to demean the service.

I did note off the top in my article though that all I was measuring was how each service deviated from the average, specifically noting that in some cases deviating from the norm can be a good thing, and that this particular article wouldn’t be covering each service’s history of success.

Before we look at the rankings themselves, it’s important to make a distinction here. The purpose of this article is to find out which service’s list is the most outlandish with regards to the average between them. It doesn’t measure which rankings have had the most success in the past, and as such it’s possible that some lists, in deviating from the norm, are actually a better bet to be correct in the future.

I did, however, table that as a topic for future research, indicating my interest in getting that type of analysis published before the upcoming draft; which makes it particularly odd that Mr. Mauro would suggest that I “wouldn’t dare” doing that.

This is something that I was hoping to get done, but you can be damn sure I’ll be making every effort to get it done now. If Draftbuzz has a history of picking sleeper picks as they claim, you can also be sure that I’ll give Mr. Mauro his due – I’m fair like that.

Based on a quick once over from Ryan though, it doesn’t seem like that’s going to be a slam dunk.

Further to this point, it’s noteworthy that Draftbuzz has only been creating lists for a few years, and Mr. Mauro is basing his success not off of the actual accomplishments of the registered players, but where they ended up going in the draft. The numbers below were reported on the Draftbuzz website, measuring the accuracy of their 2015 rankings. I will have to assume that these are pretty solid numbers, though I have compared them against any other services myself (there is a rudimentary comparison here).

Entire Draft

Selected: 150 / 211 = 71.0%

Top 30:  24 / 30 = 80.0%

Top 60:  45 / 60 = 75.0%

Top 120: 93 / 120 = 78.0%

This is better than no self-assessment at all, certainly, but it only validates one’s ability to predict how an NHL team thinks, thus using an appeal-to-authority argument as a measuring stick, rather than measuring success directly. It will be some time before Draftbuzz is able to conclusively say that they have years of sleeper picks that others “couldn’t sniff at”.

Our website, by contrast, is not interested in predicting where players go in the draft, but rather picking players who are eventually going to have greater success in the league, which I thought was the actual point, to be perfectly honest.

Anyway, while I did use the term “kooky” in reference to the list itself (not meant to be an incendiary term, but I can see where that would have been irksome), the variance relative to the aggregate was not really what I took issue with – it was the volatility from one month to the next. I’m not alone in those concerns either.

We Are Aware That There Was a Tournament Last Month

Yesss, I do follow the draft and am in fact aware that April is a big month, given that the U18’s took place in Slovakia. I know this because they were televised and I watched them. There is absolutely no doubt that the U18’s are an important event in a player’s draft year, and they’re going to have some influence on the rankings. The problem lies in just how much influence they have.

Every ranking changes from one iteration to the next – that’s an innate feature of ranking something that is continuing to showcase itself as time passes. However, they don’t all change to the same extent.

The top 31 players in Draftbuzz’s most recent ranking moved an average of 7.4 spots since the previous iteration, published seven weeks prior. By contrast, the top 31 player’s on ISS Hockey’s most recent list moved an average of 4.0 spots since its previous version, which was published four weeks earlier (a huge chunk of that is owed to Miro Heiskanen, who was slotted 5th after being unranked earlier). HockeyProspect, which hasn’t published a ranking since February, had their top 31 prospects move an average of 4.4 spots since January. Sportsnet’s moved an average of 2.3 spots from January to March, while McKeen’s moved an average of 3.8 spots from December to February. Future Considerations’ two most recent lists were published three whole months apart, but their top 31 moved an average of just 4.7 spots in that time. No one comes close to 7.4.

I’m not sure if that even does it justice, given that a lot of the real big movers (we’re talking 40, 50, 60, even 70+ spots) are in the second and third rounds. Despite the fact that the season is over and the U18 tournament has taken place, no other lists are changing this much. But I point this out, and I’m the one getting called a hack.

Closing Thoughts

What I’m trying to do here is do right by our readers and expand upon currently available hockey knowledge. What I’m not trying to do is sensationalize content in the name of garnering views and followers. While the Network itself appreciates pageviews, and I am their employee, I don’t get compensated any extra for articles that generate high numbers of clicks or comments. As for followers, my twitter account isn’t even active, and my website’s twitter account already has four times as many followers as the Draftbuzz account. Like I mentioned off the top, I publish articles because I think the content is interesting and will be of value to people that read it.

I also don’t have any intention of getting into skirmishes with strangers in the hockey community – though I’m clearly not going to shy away from defending the integrity of this site if someone is going to take shots at us.

While I felt it necessary to set to record straight on a number of fronts (and I got support from the site to do so), I do hope that this can be put to bed. As I mentioned, I have paid for the insider content at Anthony Mauro’s site, and I intend to keep on using. I’ll dig into whatever success time has been able to support at this point, and I do hope that he turns out to be right on the advantages of his site – I’d like to know that I’m getting bang for my buck.

In the meantime, I’ll continue trying to keep you informed on updates in the prospect world. Not for click and followers and views – but because I legitimately feel that there’s a thirst for the information.



  • Double U Tee Eff

    I thought Jeremy’s article on the draft rankings was an interesting read and quite informative. The title of the article is probably what got draftbuzz’s ginch in such a twist “Which Service Has the Most Outlandish 2017 NHL Draft Rankings?” But really theres nothin to see here, move along.

  • I do get the idea about your article. But this year’s draft has been the most volatile I’ve seen since 2003. Players in the end of the first could become better than those in the Top 10. From my point of view, I’ve had Elias Pettersson #1 overall since a couple months. And Patrick is 8 on my list, subject to changes in june. Nothing wrong about deviating from consensus, it just takes balls to do so, and assume it. By the way, I follow a lot of European leagues, and this year’s Swedish/Finish crop is out of his world. DLB may Top 10: 1- Pettersson 2- Liljegren 3- Vesalainen 4- Brännström 5- Kostin 6- Hischier 7- Tolvanen 8- Patrick 9- Glass 10- Vilardi. http://danslabande.com. (excuse my french…)

  • defenceman factory

    Quite enjoyed the consolidated rankings article. There are a lot of lists, nice to have them all side by side, for which players there is little consensus and which services deviate from the norm the most.

    I don’t understand the sensitivity to draftbuzz being upset. CA insults players, coaches and executive every day often with less empirical evidence used to show the outlandishness of draftbuzz rankings. You didn’t expect them to like your article did you?

    • I didn’t entirely expect them to read it, to be completely honest. I sometimes forgot about the site’s reach and assume that I’m just talking at you folks, not the people actually *in* the article.
      Certainly Draftbuzz had a right to be mad. I was openly critical of his work. He threw stones back, and in response, I simply tried to set the record straight with measured arguments and hard facts. I’ll assume that this matter is now put to bed, and we’ll all move on with our lives.

      • Marvin Duey

        Jeremy – you are a breath of fresh air in the world of hockey commentary with your insightful, original, and interesting content. This article – an honest and fairly written reply – is demonstrative of that. Those comments from Draftbuzz strike me as petulant.

        If there was one thing about the hockey world I wish I could change, it would be the passive-aggressive, moral high ground schtick that most mainstream hockey commentators seek to cultivate wherein their opinions are carefully crafted to fit within the Mike Milbury/Don Cherry-esque mean in which the “put your head down and work hard” (AND DON’T HAVE A PERSONALITY OR BE ORIGINAL) traits we value so much in players are imposed upon the hockey commentary world and fans generally. The result is that original and interesting opinions are ignored or overtly suppressed in favour of xeroxed “good guy” takes. We don’t need more of that – please keep writing.

  • Rodeobill

    Maybe draftbuzz is selling something different than the other sites that are more conservative by trying to be a little “braver” with their list, once in a blue moon you will hit one and then everyone will be… “whoa, only that guy saw that coming.” Smart actually. If you can say one example of success most people will tend to forget about the misses. If it is a business sometimes you need to sell something a little different than the rest.

  • BlazerFan

    So buddy gives you props for taking down his list, checks Canucks Army Twitter followers and starts a Twitter beef.
    Pretty obvious who’s trying to increase page views and subscribers here.

    • Locomotion

      Not sure if that’s joke. Look it up.
      You got a price in there, a kopitar, letang and even a tuuka/quick. All draft are actually not the greatest or even “great” like last year. Besides That the last great draft I’d say is the Kesler year draft I think 03

  • Dirty30

    But if a weak argument from Draftbutt — first, if he is constantly updating his rankings (putatively to reveal hidden gems that no one else can find) then any list published here would be out of date the moment he published his new list.

    Second, all the draft eligible players are in the public domain — only his ranking of them is proprietary and even that seems a bit far-fetched as it’s not like he has exclusive contracts with NHL teams to do their scouting and provide a specific draft list.

    Yes, he has subscriptions to his site, but is he really going to lose them all because some other site used his list for comparative purposes? He might even pick up a few subscribers who are interested in what he’s doing.

    If there is any new culpa, it is in not simply asking for a list to publish here prior to writing the article. Refusal to allow it would have been as telling as the list itself. There might have even been some interesting discussion about what goes into creating a unique draft list.

  • Locust

    Don’t see the big deal.
    Canucks Army (unlike other Nation sites) purposefully trolls its own team and anyone and everyone attached to it.
    Douchebaggery has no borders……

  • Ragnarok Ouroboros

    So how would one measure the success of a draft list? My best guess is that the players at the top of the list would make the NHL, and all the chaff would fall to the bottom of the list. Perhaps the top 100 list can be broken down into 10 player segments. The First segment representing the top 10 players, you would measure whether they played a minimum 200 NHL games, and played on 1st of 2nd line. The next segment would be players drafted 11 to 20, and you would measure whether they played a minimum 200 NHL games and played on 3rd or 2nd line. IE for each segment of players you would reduce the expectations. You would then score each segment by the number of players that met those expectations to give an overall score for the draft list.

  • Jeremy, your article was great and, as a guy who doesn’t watch juniors at all but is still interested in how things are shaking up prior to the draft, I found it most useful.

    One of the most interesting points was seeing how high most of those services are still ranking Liljegren, since I’m convinced Benning is taking him no matter who else is on the board.

  • RobG

    Here’s the million dollar question. Which of these 12 ranking services have the “most correct” over the last 10 years? How have their rankings stacked up to the actual draft order and who of these 12 has been the most accurate in their pre-draft rankings?

  • ShockTop

    Anthony Mauro must be a sensitive little flower. What a cry baby. Keep up the awesome work Jeremy! If Mauro thinks his work is infallible, then he needs to grow up. Mauro will probably lose more subs from this petty twitter war than the brief time the additional rankings were exposed.