E is for Edgar

When it comes to models, you typically want one that replicates reality as closely as possible. Unless you’re the editor for the Sports Illustrated swimsuit issue, I suppose.

But hey, we’re not here to discuss the fantasies of teenage boys. This is about the on-ice reality of NHL hockey and how best to analyze it. And the methods and models used to do that are getting more and more complex.

It’s been over four years since I wrote a really basic guide to advanced hockey stats right here at Canucks Army, and boy have things come a long way since then. In that time we’ve moved from shot attempts (Corsi/Fenwick) to scoring chances (SC) to expected goals (xG), and now can combine a variety of factors to estimate overall value of players in terms of Goals (GAR) or Wins Above Replacement (WAR).

But we’re not here to talk about those either. We’re here to talk about models used to predict the outcome of hockey games. And in particular, we’re here to talk about Edgar.

Edgar is the model Micah Blake McCurdy (@IneffectiveMath) built to simulate the 2017-18 NHL season. “Simulate” is the key word here, and it’s the reason I thought it might be worth delving into this topic.

There are now quite a few models out there that combine a variety of statistical regression techniques to generate predictions. Actually, probabilities is probably a better term for what they generate. Prediction implies a binary, win/loss result. But in, er, simulated reality, there are no sure things. A model can only ever produce the probability of a win or loss. So while the input might be binary, in terms of 1s and 0s, the output will be a probability between 0 and 1:

Please tip your waitress, and try the veal.

Anyway, bad attempts at jokes aside, the difference between Edgar and most models produced to date is that Edgar is a bottom up simulation of events during a hockey game, rather than a combination of statistical regression techniques that was “trained” on a previous data set.

Advertisement - Continue Reading Below

As Micah himself puts it:

It works by estimating the patterns of shots that will be taken in a game and then simulating the result of those shots. This simulation approach makes it completely different from my past models, which are all essentially statistical. Edgar contains no parameters that are obtained from any statistical algorithm; that is, it is not “trained” by optimization of any function. You should think of Edgar as a “model” of hockey like you would think of a wind-tunnel carving as a model of a race car or of the theory of gravitation as a model of planetary movement: that is, as a simplified version of something which interests us.

That last sentence is the key. With a simulation, we now have a model whose outputs are based on directly on the interaction between empirical observations and the constraints and parameters of a hockey itself. This type of model then allows you to better test different hypotheses because you are applying them to a representation of a hockey game, rather than relying on a statistical representation of similar situations that occurred in the past and trusting them to predict the result. As with the scale model in a wind tunnel example above, simulation models will allow us to change certain input parameters in isolation, like, say, shot locations or shot rates, to test which has a bigger impact on results. Statistical models will always have limitations because of confounding factors affecting both inputs and outputs. It would be like trying to determine the best design for the spoiler on a race car by looking at lap times in real races for various cars.

All that being said, it is important to remember this much quoted aphorism:

All models are wrong, but some are useful.

— George Box

And yes, if there’s one thing we can be sure of, it’s that Edgar, like all models, will be wrong a lot of the time. In fact, Corsica has a running tally of game-by-game model predictions going for this season, and Edgar is currently in 15th in terms of overall accuracy, a couple of spots below the Average prediction of all the models for each game:

Advertisement - Continue Commenting Below

But that’s not to say that it is not useful. Keep in mind, this is Micah’s first iteration of a simulation model and we are only 15 games into the season, which means much of the input is still based on previous years’ observations. As this season gets past the 20-25 game mark, I would expect that . I’m sure the floggings will continue until performance improves.

Despite the less than stellar predictive performance, the neat thing about simulation models is that the error is often consistent, i.e. the same error will apply to all permutations of the simulation. So if you want to compare two things and are looking at the relative change in results, the error will often simply cancel out. There’s a very good chance the building you are sitting in right now was probably designed and built using an energy model that was similarly “wrong” in terms of predicting overall energy use. But when the designers use the model to compare different types of window systems, they can get a pretty good sense of the relative importance of one window vs another. It is exactly in this way that Edgar, and models like it, will eventually be very useful in analyzing hockey.

So, what does this all mean for the casual fan? Probably not a heck of a lot right now, but I expect that in the future this approach will provide us with some additional insights into how hockey works and which on-ice events are actually important.

Until then we are left with Edgar as the state of the art hockey simulation model, and this is what it told us about the Canucks coming into this season, together with actual progress through last night’s game (if you want to read more about CUSUM and this type of chart, I wrote about it last year on Hockey Graphs):

So far the Canucks are basically staying within a standard deviation of the predicted points coming into the season, and at this rate would finish near the predicted 86 point mark. Definitely not good enough for the playoffs, but maybe bad enough to sneak into the top 10 at the draft.

But where they actually wind up, well I guess we’ll just have to wait and see.




  • Chris the Curmudgeon

    So, I think your story here begs the question a little bit, in that you’ve already decided that the model works even though it’s not that predictive, and then finds justification for why it fails without explaining it to us. However, I can see what you’re getting at. My real question is: how did this model manage to predict the Canucks to earn 86 points, after they got 69 this year and most prognosticators considered them unlikely to improve much? Obviously we don’t know where they’ll end up, but it’s interesting that someone trying to model the season had any cause for optimism about the team, at least as compared with the talking heads.

    • Some quick thoughts….I don’t think suggesting the model “works” is 100% the idea. For now, it’s being tested in predictive power, which we can tentatively call “working”. However, model failure is equally useful, especially for determining if Edgar is picking up on a variable other models are missing. The point about the model using older data is important, as Edgar lacks a sizable sample to draw from. So, based on what we know of the variables Edgar accounts for (and what we might deem to be important), we might expect it to improve in predictive power as the season goes on.

      As for how Edgar determined 86 points, it might be important to remember that 86 points still isn’t good. That’s a somewhat below-average team (LA and Winnipeg both missed the post-season with 86 and 87 points last year, for reference). The team would have to be pretty abysmal to sink below 70 points again. The recent improvement in goaltending and scoring production has also probably influenced the current model projections to closer to a .500 record. Right now, Vancouver is likely outpacing projections from experts because the Travis Green Effect is elevating the play of this roster. I have a feeling Edgar’s projection of 86 is probably within the ballpark of where Vancouver will finish.

  • Killer Marmot

    It’s interesting that the boundaries of the confidence interval (or whatever you’re calling it) appear linear. I would not expect that. Rather I would expect it to be more hyperbolic, which means wider than what you show when the number of games are small, but approaching linear as the number of games gets large.

  • Off topic: In case you missed it, Francesco Aquilini gave a rare interview on Sporsnet 650 yesterday. It’s nice to hear the man speak. I like the part where he said everyone is always telling him what to do.
    I think you should re-sign Jim Benning.

  • Ragnarok Ouroboros

    The problem with statistical models is that they do not anticipate player injuries. The models are based on players having perfect health for the whole season. When a team loses a key player like Leon Draisaital, the effect on the teams predicted success can be huge. Conversely, losing a plug to injury does not really affect team success.

    I think a successful model needs to constantly re-evaluate predictions over time, and take into account the expected absence of injured players. IE if Draisaital is gone for 2 weeks, then you could expect the Oilers to lose a few more games during that time period, but you would expect the team to revert back to normal when Draisaitl comes back from injury.

  • Doodly Doot

    I modelled the Canucks season back in August. I used meditation, tea leaves, dice, fish entrails and an old t-shirt (to clean up the fish mess). It showed that the Canucks may or may not make the playoffs. Currently they are within the standard deviation. That may or may not continue.

    Kudos to Micah for noting Edgar’s weaknesses including insufficient explicitness, omissions and simplifications. My model does not have those weaknesses.

  • Locust

    When I read this from an analytics guy – “I expect that in the future this approach will provide us with some additional insights into how hockey works and which on-ice events are actually important.” – It just confirms what I already knew, most ‘analytic guys’ don’t “know how hockey works” or “which on-ice events are actually important”. Thanks for clearing that up.

    I’ve come up with a few NEW analytics acronyms for your reading pleasure….

    WIMP = Whats IMPortant. Strangely enough, CA needs more WIMP.
    RACK = Resisting All things CanucK. Just like a Japenese anime cartoon, you guys are all over the RACK. But you need to grow up and enter the real world one day….
    GRIT = Guaranteed Return quotIenT. Definitely lacking, need to actually watch the games to see more WIMP.

    And finally, it appears to a few regular readers here that CA writers see themselves “in another position” and that’s why you routinely dis everything Canuck. You are LAZY = Longing for AriZona Y’all. Good luck with that…. I mean it …..

  • defenceman factory

    I have never said this before (and may not ever again). I enjoyed this article from Petbugs. The analytics of hockey is interesting and continues to evolve. Analytics, used appropriately, are useful. I’m glad Canucks Army provides updates on developments. I really appreciate reading articles which lack the usual stupid sarcasm about how bad some player or team management is.

    • Doodly Doot

      Well said df. I also liked the tone and candour of the article. Links to source data respect the reader and are helpful for the curious. I like the ‘idea’ of modelling and simulations but am highly skeptical of designer methodology and authentic relevance of output. Though I reserve the right to make light of it, I am open to being captivated someday by the work of a savant. Until then, I’ll continue to perceive the relevance of analytic fragments as asymmetric information and at best a hypnotic shadow of reality.