# Adjusted Save Percentage: Measuring the Impact Defense has on Goaltending Statistics

One of the reasons we love sports as much as we do is
because of the great debates it can spawn. I came across a classic hockey
debate the other day in relation to Ryan Miller. “Of course Miller didn’t post
elite numbers last year. He played for Buffalo!”

progress in terms of quantifying the impact of defense and separating that
contribution to statistics for the goalie themselves. We can tell the
percentage of shots a goalie save (SV%), but we have little to no insight into
the difficulty level of the shots themselves.

From a quantitative standpoint, we understand the game
better now than ever before. We understand shot attempt differential, which we
know we can be used to approximate possession. We call this corsi, and we understand this to be a
key component of overall team success. We can also begin to isolate the impact a player has
on their team’s shot attempt differential, which is a solid representation of
strong two way play.

However, there is one factor that detractors of advanced
stats have always struggled with. Detractors have long protested that not all
shots have an equal chance of going in, and they’re right. Is it possible to adjust save-percentage to reflect easier or more difficult workload? Find out after the jump.

In the 2013-14 season, goalies league wide faced 73,614
shots. Using Greg Sinclair’s immensely useful website we
can break shot attempts down by the distance the shot was taken from the net.
As you would expect, shots taken from within 10 feet, such as rebounds,
redirections, cross crease feeds, and other scoring chances, have the highest shooting percentage at 19%.
This shooting percentage is over 4.5 times higher than shots taken outside 30
feet, but shots from here are significantly more rare (only 7% of total shots).

Table 1

This confirms something we’ve always known: not only is the
goal of defenses to limit shot attempts (which we can measure with corsi), but
perhaps just as importantly, limit the chances the other team gets in high
percentage areas. Based on the information above, a shot from inside 10 feet
from the goal is almost five times as likely of finding the twine than a shot
from 30 feet out. Naturally, some defensemen do a better job of clearing the
puck from high percentage areas than others, and we can use this data to
calculate how much of a goalies save percentage is the result of having a better
or worse defense in front of them.

Here is the distribution of shots faced for each goalie who
played over 1,000 minutes in 2013-14:

Table 2

My next step was to calculate each goalie’s save percentage
for each of the four distances:

Table 3

Then in order to isolate the impact of team defense, I
applied the redistributed the total shots faced by each goalie by the average
shot distribution at each of the four distances, then used the goalies actual
save percentage at each distance to calculate an adjusted save percentage
(excluding empty nets and shootouts):

Table 4

Now before I get into discussing the result, there are obviously limitations of this approach which are worth noting. Using the four distance buckets obviously runs the risk of overly aggregating shots that are dissimilar. For example, a goal mouth rebound from two feet out likely has a much higher chance of going in than a shot from nine feet out where the goalie has position. In the future, as we track more variables such as whether the shot is coming of a rebound or a one-timer, or even shot velocity, I could envision a much more complex version of this approach. There are also issues with inconsistencies in today’s data gathering when it comes to shot distance data from one arena to the next. In short, this isn’t perfect by any means.

Detractions aside, I think the data above provides an interesting glimpse into the varying
levels of difficulty each goalie faced. Lack, Miller, and Luongo, all faced a
lower proportion of shots in the 0-10 foot and 10-20 foot categories than the
league average, meaning that the defenses in front of them did a better than
average job of limiting the amount of high percentage shots they faced from in
close.

A number of other interesting data points jump off the page.
For instance, Steve Mason enjoyed a renaissance season in Philadelphia last
year, but when you adjust for shot distribution you find that his adj SV% at
.909 was actually right in line with his career SV% of .907. Sorry Philly fans,
you still may not have solved your goalie problems.

On the other end of the spectrum, Jonas Hiller faced an
incredibly difficult shot distribution. When adjusting for shot distribution he
went from a SV% of .911 to an adjusted SV% of .918, which was slightly above
his career SV% of .916. He signed with Calgary for \$4.5M per year, which makes
me wonder how much more he could have gotten if teams realized just how much
the Duck’s defense hung him out to dry last season.

In the summer of analytics, goaltending has been noticeably absent from the discussions of major areas of advancement in our understanding of the sport we love. While the above approach is somewhat rudimentary based on the limitations of the data we currently have to work with, it very well may turn out to be an area of interest going forward.

• BrandonC

The core problem with this analysis is we know shot distance is not the key component of average shot quality differences on the offensive side of the game for individual players.

We know that the average shot when Sidney Crosby is on the ice is significantly more difficult than the average shot when Daniel Winnik is on the ice (just look up their long-term on-ice shooting percentages to see – the difference is significant).

Numerous studies have shown that shot distance/location distributions is not significantly different across players and does not play a significant role in average on-ice shot quality.

On-ice shooting percentages vary significantly across players, shot distance/location does not. The conclusion must be that disparity in distance/location is not the significant driver or on-ice shot quality across players (it exists but is relatively minor).

What is? Probably things like taking shots on the rush, one-timers from cross ice passes, shooting talent (speed/accuracy), etc. The circumstances leading up to the shot is what mostly drives shot quality, not the distance or location.

Unfortunately, until player and puck tracking technology comes into play these other factors can’t be directly quantified. Until then we are left somewhat in the dark the extent that team defense impacts goalie save percentage.

• TheNitsguy

It seems to me like the data used here includes deflections/rebounds/etc as shots within 10 feet, and not necessarily where the shot originated from. I believe this was a major point of issue from the numerous studies you mention, but that’s just my assumption – please correct me if I am mistaken!

With that in mind, even with the limitations of the data (as mentioned within the article), there is still useful information to be gleaned. I’m even more glad I didn’t pick Steve Mason in my pool, for example!

• TheNitsguy

This is amazing stuff money puck! Keep up the high quality work!

Won’t be long ’til the new guy gets scooped up too, that’s for sure…

• TheNitsguy

Stop trying to make sports articles into financial reports. Your stats focus is crap and needs to stop. Be a man.

• TheNitsguy

Do the right thing!

Heheh, trolls still be trollin’… I mean, you know what you’re getting into if you read any of the nation sites…

• TheNitsguy

Are you able to split Miller’s numbers into Buffalo and St. Louis separately? Just curious.

• Squibbles

Yes. I aggregated the data, but you can could use http://somekindofninja.com/ to split Miller’s number by team.

• BrandonC

Boyle had a number of great goalie articles on Sportsnet last year.

http://www.sportsnet.ca/hockey/nhl/nhl-goalies-better-with-high-shot-volumes/

I wonder how much of these adjusted numbers would relate at all to goalie workload, from where all the extra shots came in 30+, 40+ shot games etc.

Great article Moneypuck…

• Squibbles

It would be interesting to see this somehow combined with the shooting percentage of the players on the ice. In theory making a save when there’s high percentage shooters on the ice should be more valuable? Or maybe some kind of stat where you compare shooting percentage of the players on the ice with how far away the shots are taken from to see the effectiveness of the defenders at keeping the skilled players to the perimeter?

• BrandonC

Hi David – I agree that the factors you mention such as taking shots on the rush, one-timers from cross ice passes, shot velocity/accuracy, etc, will be more instructive than simply using distance data as I’ve done here. My hope is that we’ll one day be able to utilize that type of data within a framework like this, but as you know we’re a ways away from being able to do that.

While you’re right that many studies have shown limited differences in shot distribution across shooters at a macro level, when you look at individual shooters there can be pretty significant differences. For example, I wrote a post earlier looking at Vrbata’s shooting style and wondered if he’d be a fit with the Sedin’s as he historically had been more of a perimiter shooter, than a net front presense/rebound machine:

http://canucksarmy.com/2014/9/24/offseason-transactions-deep-dive-free-agent-frenzy-signings

The question there is whether the difference in shot distribution across players is a player/skill issue or differences in coaching systems, but on a player by player level there definitely does appear to be differences in shot distribution.

• Squibbles

This analysis might overrate goaltenders with poor rebound control.

• BrandonC

This is a very good point. I’m hopeful that when shot data matures, we’ll be able to apply a more sophisticated version of this type of framework to assess goalie performance. Admittedly, at this stage there are limitations in the data that makes this analysis somewhat more theoretical, than conclusive.

• BrandonC

Great analysis, money puck.

It’ll be interesting to learn how repeatable these save percentage buckets are.

Maybe after work I’ll run the same numbers for the other years.

• BrandonC

Ondrej Pavelec is good….?

• BrandonC

@David Johnson
Whether or not there are differences between players is not relevant to this particular analysis.

Do individuals differ in their shooting ability? Yes, they do. But EVERY player has a lower shooting percentage from long distances than from in close – that’s just hockey. So the writer is correct in that controlling for the distances from which the shots originated will provide a clearer picture of the goalies’ skills. A goalie that only has to face shots from 50 feet away should have a high save percentage, whereas a goalie that only faces ten foot shots would be expected to let in more goals, even if they are highly skilled. You are correct that you could get an even better metric if you included the quality of the shooter, but that doesn’t mean distance isn’t worth looking at.

@money puck
The only issue I have here is where you compared the adjusted save percentage to the goalie’s career save percentage. Hiller’s career sv% is lower than his adjusted, but to make this a realistic comparison you need to compare his career ADJUSTED sv% to last year’s adjusted sv%. Otherwise you are comparing apples and oranges.

• BrandonC

The comparison is actually 2013-14 SV% vs 2013-14 adjusted SV%, not career SV% in Table 4. I should have made that clearer.

• BrandonC

One stat that I am very curious about is big saves in pressure situations. Velocity and rebounds are great, but we all know there are different players that show up in big situations and others that shrink.

To me, the Oilers game was a positive for Miller because he didn’t sulk or show frustration. Opening home game, big contract Ect.. He seemed to stay calm and get better and better. In the third and SO he was big and allowed the team to come back.

Sagas will always be tough, because time and situation always play a role and those can’t always be quantified. You have to see it, to understand it.

• BrandonC

@money puck
You said:
“On the other end of the spectrum, Jonas Hiller faced an incredibly difficult shot distribution. When adjusting for shot distribution he went from a SV% of .911 to an adjusted SV% of .918, which was slightly above his career SV% of .916.”

This was not in the table, but in the text. Your point was that the adjusted SV% looks more in line with his career SV%, which you used as justification that he was a decent goalie last year and could have commanded more on the market.
My point is that his career Adjusted SV% could very well differ from 0.916.

Anyway, a great article, and nice concept – definitely a better measure of goalie skill than the raw stats.

• BrandonC

“Then in order to isolate the impact of team defense, I redistributed the total shots faced by each goalie by the average shot distribution at each of the four distances, then used the goalies actual save percentage at each distance to calculate an adjusted save percentage”

If I’m reading that right, this analysis is saying that the only thing that team defence has control over is the distribution of shot locations. When you arrive at your adjusted sv%, you’ve assumed everything else is the same; the only thing that is adjusted is the distribution of shots by location. Same amount of shots faced, same save percentages at the different distances, only the distribution of shots faced changes.

That seems like a wildly simplistic view of the effect team defence has on save percentage.

Without getting too deep into esoteric discussions about blocking/challenging shots and that sort of nuanced tactical elements, defence has the ability to impact the volume of shots a goalie faces as well, and that’s not accounted for. Compare the amount of shots St. Louis gives up per game to a team like Toronto or Buffalo.

In 2013, a writer at PPP looked into the myth that Randy Carlyle’s defence was “pushing shots to the outside”, since this was the explanation given for why the Leafs suddenly had success in the lockout season. He found it was bunk.

http://www.pensionplanpuppets.com/2013/6/17/4436736/randy-carlyle-coaching-myths-part-five-shot-distance

Essentially, and this matches the shot ratios that Reimer and Bernier have in your tables above, they found that the Leafs weren’t minimizing shots in dangerous areas and pushing those shots to the outside; the Leafs were giving up just as many shots from the areas in close to the ice, they just added a number of shots from father out.

I don’t think you’ve isolated for team defence here. I think you’ve controlled for goalies that are outliers in terms of the distribution of the shot locations compared to league average.

• BrandonC

How about including some type of weighting metric related to the shooter’s career SH% (or on-ice SH%),

Shots by Crosby would be weighted more heavily than shots by Colton Orr.

• BrandonC

Further, You could determine a dSv% = Sv% – (100-[shooter-weighted Sh%]) All at 5v5 or ES would probably be more representative.

A +ve dSv% would mean that a goalie’s SV% is higher than expected from the shots faced, weighted by shooter quality.

• BrandonC

Nice work. Nabakov looks good, how bad were the Isles dmen then?

Only quibble is it would be nice to see the data you built it on (total shots) Not really needed in the article but a link to it would be nice

• Greg

Great article! The limitations are obvious but it’s doing the best you can do with the data quality and tools available at this point, and it’s exposing some valuable insights already (don’t pick mason in your pool!).

As data quality and analysis tools improve, this approach will seem primitive, but so will Corsi I suspect. We’ll start seeing things not just like shot differentials and complex shot quality metrics that factor in shot type and player and shot speed, but pre-shot metrics that impact shot quality. A slapshot from the hashmarks isn’t the same as a cross-ice one timer from the hashmarks, even if it’s taken by the same player with the same speed. And that one timer is a different quality in a 2-on-1 break than a 5-on-4 power play. And that 5-on-4 power play cross-ice one timer is a different quality if there’s traffic in front of the net vs clear shooting lane.

The ironic thing is that as the data and tools improve in cost and complexity, the returns will actually diminish. The most significant, impact full, actionable insights like Mason’s false Goodyear and Hiller’s false bad year can already be seen with this rudimentary data set and approach. Better data and tools will might make that more obvious, or measure it to a greater accuracy, and will expose some previously undetectable insights, but those insights will likely be less impactful given they were too small to be measured earlier.

I’m sure we’ll continue to get surprising new insights that change the game for the better, and a lot of those will be in the area of goaltending given how little is currently known. But they’ll be more and more like squeezing water from a rock as the biggest ones get exposed early – I wouldn’t be at all surprised if the biggest ones have already been found, like the importance of puck possession or counter-productivity of face-punchers.

So, in short, despite it’s limitations, I think this is very worthwhile analysis. This alone could keep a GM from making a career ending or franchise killing rash decision on a goaltender.