In Defence of Consolidated Rankings

I’ve been awfully busy these past few months, between prepping for the NHL Draft, developing and assessing metrics, and watching a ton of hockey, as well as some personal matters here and there. So I haven’t been writing as much as I usually do, and when I do, it’s typically something that I think is important to share with the hockey world.

So perhaps you can imagine my surprise yesterday when I was accused of trying to garner views and “sell followers”. Granted, it was pretty clear that I had pissed off the source of the comments with my Friday article on the variability of mainstream draft rankings. In analyzing which services deviated the most from average rankings, I was admittedly a little harsh on one service in particular – Draftbuzz – and site founder Anthony Mauro let me know about it on Twitter.

Now, I’m not active on Twitter, so I wasn’t exactly in a position to defend myself – big up’s to my co-writers JD Burke and Ryan Biech for sticking up for me in that regard – but there are a couple of points I’d like to touch on, and this is the space in which I’m able to do that.

What follows is a few of those thoughts, centred around some of the feedback and accusations.

I posted an image of Draftbuzz’s top 100 ranked prospects, which is a no-no, and something that I should have considered. Draftbuzz only posts a top 31 on their website, and users pay extra for longer lists.

Given that I had a longer list, you can probably guess that I myself am a subscriber. I’ve given some of my hard earned money to Draftbuzz, and as a consumer I am within my rights to raise concerns about the product – such as massive month-to-month swings. While I’ve had conversations with others about where the line is when criticizing the work of others in the business, I’ve always thought of this as one of the advantages of writing for a blog instead of a newspaper or mainstream outlet: there’s quite a bit more leeway for honesty.

As for this accusation:

I don’t believe I’m being unfair to any other service. Of the 12 lists that I included in Friday’s article, four of them (HockeyProspect.com, McKeen’s, ESPN, and of course Draftbuzz) require paid subscriptions to see full lists – though each of them outside of ESPN provides at least a top 31 for free. In some cases, such as noting that a subscription-based service has a possible first round player ranked in the 40’s, I may be flirting with the line a little bit. I may have to be slightly more careful of this in the future, though if you’ve seen the other Consolidated Rankings I’ve published, I don’t typically provide where the players are ranked by individual services anyway.

I do understand of course that free giving away something that Draftbuzz is trying to sell is, to be blunt, a dick move. Which is why I had zero problems with removing the list upon request. At this point, Mr. Mauro thanked us for removing the draft board.

I was quite pleased at this point with Draftbuzz taking the high road in spite of the fact that I’d criticized their work; I figured this whole thing was put to rest.

Then a couple of hours later I was being called trash and a hack. Not sure what happened there.

This one had me going a little bit. The implication here is that we, purporting to be knowledgeable about prospects, have to aggregate the opinions of others in order to rank our players.

The insinuation that we can’t come up with our own opinions is a bit insulting, for starters. It’s also entirely untrue. Exhibit A might be the fact that we watch a LOT of f#$&ing hockey (though we don’t get to watch many games in person, mainly for budgetary reasons – if someone wants to pay us to travel and watch games, we’d be good with that). We also have access to a bevy of statistical metrics (some publicly available, and some that were created in-house) that have some scientific merit but often go ignored by other rankers.

I noted at the bottom of the article that we are going to be rolling out a top 100 starting in a couple of weeks. I went on to explain how our list deviates from the consolidated average of the mainstream rankings more than the majority of the ones used to make the aggregation. This should be a clear indicator that we don’t use the consolidated rankings to generate our list.

The consolidated rankings are in fact not for us, they are for you: the readers. Another thing that I mentioned in Friday’s article was that I think there’s value in aggregating mainstream rankings, because you get the benefit of a whole lot of scouting experience, while also accounting for outliers and bias that may creep in if you focus solely on the rankings of a single service.

We also understand that the majority of hockey fans don’t have the time to familiarize themselves with more than a handful of prospects, and also probably don’t have the time to analyze which services have the best track records, or when a service is making a particularly bold statement on a player.

The consolidated rankings provide a service to those fans that want to know how the scouting world as a whole orders the available players. That’s probably why there has been a generally positive response to the previously published articles regarding the consolidated rankings. And it’s probably why six other editors on the Nation Network deemed that those articles had enough value that they okayed me publishing them on their individual sites.

This criticism didn’t come from Anthony Mauro, but I thought it was worthy of discussion regardless:

There’s a reason that I refer to the aggregated rankings as Consolidated, and not Consensus: but I’m not an idiot and obviously there is no consensus, that’s why I’m aggregating in the first place.

Perhaps the most comical notion about this criticism is that fact that it’s exactly what Bob McKenzie does. The Godfather himself sure isn’t out there scouting kids. He’s using his incredible reach and network of contacts to find out what NHL teams think of players before consolidating them into his draft rankings – and I’m pretty sure nobody is telling him that no one cares about aggregating opinions or is giving him a hard time for it. That’s an awfully broad brushstroke to paint with.

A couple of remarks were made that I was using the high volatility of Draftbuzz’s list to demean the service.

I did note off the top in my article though that all I was measuring was how each service deviated from the average, specifically noting that in some cases deviating from the norm can be a good thing, and that this particular article wouldn’t be covering each service’s history of success.

I did, however, table that as a topic for future research, indicating my interest in getting that type of analysis published before the upcoming draft; which makes it particularly odd that Mr. Mauro would suggest that I “wouldn’t dare” doing that.

This is something that I was hoping to get done, but you can be damn sure I’ll be making every effort to get it done now. If Draftbuzz has a history of picking sleeper picks as they claim, you can also be sure that I’ll give Mr. Mauro his due – I’m fair like that.

Based on a quick once over from Ryan though, it doesn’t seem like that’s going to be a slam dunk.

Further to this point, it’s noteworthy that Draftbuzz has only been creating lists for a few years, and Mr. Mauro is basing his success not off of the actual accomplishments of the registered players, but where they ended up going in the draft. The numbers below were reported on the Draftbuzz website, measuring the accuracy of their 2015 rankings. I will have to assume that these are pretty solid numbers, though I have compared them against any other services myself (there is a rudimentary comparison here).

This is better than no self-assessment at all, certainly, but it only validates one’s ability to predict how an NHL team thinks, thus using an appeal-to-authority argument as a measuring stick, rather than measuring success directly. It will be some time before Draftbuzz is able to conclusively say that they have years of sleeper picks that others “couldn’t sniff at”.

Our website, by contrast, is not interested in predicting where players go in the draft, but rather picking players who are eventually going to have greater success in the league, which I thought was the actual point, to be perfectly honest.

Anyway, while I did use the term “kooky” in reference to the list itself (not meant to be an incendiary term, but I can see where that would have been irksome), the variance relative to the aggregate was not really what I took issue with – it was the volatility from one month to the next. I’m not alone in those concerns either.

Yesss, I do follow the draft and am in fact aware that April is a big month, given that the U18’s took place in Slovakia. I know this because they were televised and I watched them. There is absolutely no doubt that the U18’s are an important event in a player’s draft year, and they’re going to have some influence on the rankings. The problem lies in just how much influence they have.

Every ranking changes from one iteration to the next – that’s an innate feature of ranking something that is continuing to showcase itself as time passes. However, they don’t all change to the same extent.

The top 31 players in Draftbuzz’s most recent ranking moved an average of 7.4 spots since the previous iteration, published seven weeks prior. By contrast, the top 31 player’s on ISS Hockey’s most recent list moved an average of 4.0 spots since its previous version, which was published four weeks earlier (a huge chunk of that is owed to Miro Heiskanen, who was slotted 5th after being unranked earlier). HockeyProspect, which hasn’t published a ranking since February, had their top 31 prospects move an average of 4.4 spots since January. Sportsnet’s moved an average of 2.3 spots from January to March, while McKeen’s moved an average of 3.8 spots from December to February. Future Considerations’ two most recent lists were published three whole months apart, but their top 31 moved an average of just 4.7 spots in that time. No one comes close to 7.4.

I’m not sure if that even does it justice, given that a lot of the real big movers (we’re talking 40, 50, 60, even 70+ spots) are in the second and third rounds. Despite the fact that the season is over and the U18 tournament has taken place, no other lists are changing this much. But I point this out, and I’m the one getting called a hack.

What I’m trying to do here is do right by our readers and expand upon currently available hockey knowledge. What I’m not trying to do is sensationalize content in the name of garnering views and followers. While the Network itself appreciates pageviews, and I am their employee, I don’t get compensated any extra for articles that generate high numbers of clicks or comments. As for followers, my twitter account isn’t even active, and my website’s twitter account already has four times as many followers as the Draftbuzz account. Like I mentioned off the top, I publish articles because I think the content is interesting and will be of value to people that read it.

I also don’t have any intention of getting into skirmishes with strangers in the hockey community – though I’m clearly not going to shy away from defending the integrity of this site if someone is going to take shots at us.

While I felt it necessary to set to record straight on a number of fronts (and I got support from the site to do so), I do hope that this can be put to bed. As I mentioned, I have paid for the insider content at Anthony Mauro’s site, and I intend to keep on using. I’ll dig into whatever success time has been able to support at this point, and I do hope that he turns out to be right on the advantages of his site – I’d like to know that I’m getting bang for my buck.

In the meantime, I’ll continue trying to keep you informed on updates in the prospect world. Not for click and followers and views – but because I legitimately feel that there’s a thirst for the information.