Adjusted Save Percentage: Measuring the Impact Defense has on Goaltending Statistics

money puck
October 16 2014 01:30PM

One of the reasons we love sports as much as we do is because of the great debates it can spawn. I came across a classic hockey debate the other day in relation to Ryan Miller. “Of course Miller didn’t post elite numbers last year. He played for Buffalo!”   

As I was thinking about this, it struck me that while we have made huge strides in advanced statistics overall, we haven’t made much progress in terms of quantifying the impact of defense and separating that contribution to statistics for the goalie themselves. We can tell the percentage of shots a goalie save (SV%), but we have little to no insight into the difficulty level of the shots themselves.  

From a quantitative standpoint, we understand the game better now than ever before. We understand shot attempt differential, which we know we can be used to approximate possession. We call this corsi, and we understand this to be a key component of overall team success. We can also begin to isolate the impact a player has on their team’s shot attempt differential, which is a solid representation of strong two way play.

However, there is one factor that detractors of advanced stats have always struggled with. Detractors have long protested that not all shots have an equal chance of going in, and they’re right. Is it possible to adjust save-percentage to reflect easier or more difficult workload? Find out after the jump. 

In the 2013-14 season, goalies league wide faced 73,614 shots. Using Greg Sinclair’s immensely useful website we can break shot attempts down by the distance the shot was taken from the net. As you would expect, shots taken from within 10 feet, such as rebounds, redirections, cross crease feeds, and other scoring chances, have the highest shooting percentage at 19%. This shooting percentage is over 4.5 times higher than shots taken outside 30 feet, but shots from here are significantly more rare (only 7% of total shots).

Table 1

adj sv% table 1

This confirms something we’ve always known: not only is the goal of defenses to limit shot attempts (which we can measure with corsi), but perhaps just as importantly, limit the chances the other team gets in high percentage areas. Based on the information above, a shot from inside 10 feet from the goal is almost five times as likely of finding the twine than a shot from 30 feet out. Naturally, some defensemen do a better job of clearing the puck from high percentage areas than others, and we can use this data to calculate how much of a goalies save percentage is the result of having a better or worse defense in front of them.

Here is the distribution of shots faced for each goalie who played over 1,000 minutes in 2013-14:

Table 2

adj sv% table 2

My next step was to calculate each goalie’s save percentage for each of the four distances:

Table 3

adj sv% table 3

Then in order to isolate the impact of team defense, I applied the redistributed the total shots faced by each goalie by the average shot distribution at each of the four distances, then used the goalies actual save percentage at each distance to calculate an adjusted save percentage (excluding empty nets and shootouts):

Table 4

adj sv% table 4

Now before I get into discussing the result, there are obviously limitations of this approach which are worth noting. Using the four distance buckets obviously runs the risk of overly aggregating shots that are dissimilar. For example, a goal mouth rebound from two feet out likely has a much higher chance of going in than a shot from nine feet out where the goalie has position. In the future, as we track more variables such as whether the shot is coming of a rebound or a one-timer, or even shot velocity, I could envision a much more complex version of this approach. There are also issues with inconsistencies in today's data gathering when it comes to shot distance data from one arena to the next. In short, this isn't perfect by any means. 

Detractions aside, I think the data above provides an interesting glimpse into the varying levels of difficulty each goalie faced. Lack, Miller, and Luongo, all faced a lower proportion of shots in the 0-10 foot and 10-20 foot categories than the league average, meaning that the defenses in front of them did a better than average job of limiting the amount of high percentage shots they faced from in close.

A number of other interesting data points jump off the page. For instance, Steve Mason enjoyed a renaissance season in Philadelphia last year, but when you adjust for shot distribution you find that his adj SV% at .909 was actually right in line with his career SV% of .907. Sorry Philly fans, you still may not have solved your goalie problems.

On the other end of the spectrum, Jonas Hiller faced an incredibly difficult shot distribution. When adjusting for shot distribution he went from a SV% of .911 to an adjusted SV% of .918, which was slightly above his career SV% of .916. He signed with Calgary for $4.5M per year, which makes me wonder how much more he could have gotten if teams realized just how much the Duck’s defense hung him out to dry last season.

In the summer of analytics, goaltending has been noticeably absent from the discussions of major areas of advancement in our understanding of the sport we love. While the above approach is somewhat rudimentary based on the limitations of the data we currently have to work with, it very well may turn out to be an area of interest going forward. 

E71500c6cbed1a488b0c1837d3e24bc3
Just another grammatically challenged blogger who loves spreadsheets. Follow me on twitter @MoneyPuck_
Avatar
#1 David Johnson
October 16 2014, 01:59PM
Trash it!
7
trashes
Props
11
props

The core problem with this analysis is we know shot distance is not the key component of average shot quality differences on the offensive side of the game for individual players.

We know that the average shot when Sidney Crosby is on the ice is significantly more difficult than the average shot when Daniel Winnik is on the ice (just look up their long-term on-ice shooting percentages to see - the difference is significant).

Numerous studies have shown that shot distance/location distributions is not significantly different across players and does not play a significant role in average on-ice shot quality.

On-ice shooting percentages vary significantly across players, shot distance/location does not. The conclusion must be that disparity in distance/location is not the significant driver or on-ice shot quality across players (it exists but is relatively minor).

What is? Probably things like taking shots on the rush, one-timers from cross ice passes, shooting talent (speed/accuracy), etc. The circumstances leading up to the shot is what mostly drives shot quality, not the distance or location.

Unfortunately, until player and puck tracking technology comes into play these other factors can't be directly quantified. Until then we are left somewhat in the dark the extent that team defense impacts goalie save percentage.

Avatar
#2 TheNitsguy
October 16 2014, 01:59PM
Trash it!
2
trashes
Props
6
props

This is amazing stuff money puck! Keep up the high quality work!

Won't be long 'til the new guy gets scooped up too, that's for sure...

Avatar
#3 Richard
October 16 2014, 02:13PM
Trash it!
54
trashes
Props
2
props

Stop trying to make sports articles into financial reports. Your stats focus is crap and needs to stop. Be a man.

Avatar
#4 TheNitsguy
October 16 2014, 02:26PM
Trash it!
2
trashes
Props
1
props

@David Johnson

It seems to me like the data used here includes deflections/rebounds/etc as shots within 10 feet, and not necessarily where the shot originated from. I believe this was a major point of issue from the numerous studies you mention, but that's just my assumption - please correct me if I am mistaken!

With that in mind, even with the limitations of the data (as mentioned within the article), there is still useful information to be gleaned. I'm even more glad I didn't pick Steve Mason in my pool, for example!

Avatar
#5 TheNitsguy
October 16 2014, 02:30PM
Trash it!
1
trashes
Props
6
props
Richard wrote:

Stop trying to make sports articles into financial reports. Your stats focus is crap and needs to stop. Be a man.

Do the right thing!

Heheh, trolls still be trollin'... I mean, you know what you're getting into if you read any of the nation sites...

Avatar
#6 94eleven
October 16 2014, 02:58PM
Trash it!
1
trashes
Props
4
props

Are you able to split Miller's numbers into Buffalo and St. Louis separately? Just curious.

Avatar
#7 hockey4life
October 16 2014, 03:02PM
Trash it!
1
trashes
Props
1
props

there was a really good article awhile back that did a indepth statistical analysis between rask and price. was really informative. thought people would enjoy

http://www.sportsnet.ca/hockey/shot-quality-reveals-rasks-true-value/

Avatar
#8 Squibbles
October 16 2014, 03:08PM
Trash it!
1
trashes
Props
1
props

It would be interesting to see this somehow combined with the shooting percentage of the players on the ice. In theory making a save when there's high percentage shooters on the ice should be more valuable? Or maybe some kind of stat where you compare shooting percentage of the players on the ice with how far away the shots are taken from to see the effectiveness of the defenders at keeping the skilled players to the perimeter?

Avatar
#9 money puck
October 16 2014, 03:12PM
Trash it!
1
trashes
Props
1
props

@94eleven

Yes. I aggregated the data, but you can could use http://somekindofninja.com/ to split Miller's number by team.

Avatar
#10 money puck
October 16 2014, 03:21PM
Trash it!
1
trashes
Props
2
props

Hi David - I agree that the factors you mention such as taking shots on the rush, one-timers from cross ice passes, shot velocity/accuracy, etc, will be more instructive than simply using distance data as I've done here. My hope is that we'll one day be able to utilize that type of data within a framework like this, but as you know we're a ways away from being able to do that.

While you're right that many studies have shown limited differences in shot distribution across shooters at a macro level, when you look at individual shooters there can be pretty significant differences. For example, I wrote a post earlier looking at Vrbata's shooting style and wondered if he'd be a fit with the Sedin's as he historically had been more of a perimiter shooter, than a net front presense/rebound machine:

http://canucksarmy.com/2014/9/24/offseason-transactions-deep-dive-free-agent-frenzy-signings

The question there is whether the difference in shot distribution across players is a player/skill issue or differences in coaching systems, but on a player by player level there definitely does appear to be differences in shot distribution.

Avatar
#11 van
October 16 2014, 03:58PM
Trash it!
1
trashes
Props
6
props

This analysis might overrate goaltenders with poor rebound control.

Avatar
#12 BrandonC
October 16 2014, 04:22PM
Trash it!
1
trashes
Props
1
props

Great analysis, money puck.

It'll be interesting to learn how repeatable these save percentage buckets are.

Maybe after work I'll run the same numbers for the other years.

Avatar
#13 ben
October 16 2014, 05:42PM
Trash it!
1
trashes
Props
1
props

Ondrej Pavelec is good....?

Avatar
#14 TC
October 16 2014, 06:28PM
Trash it!
2
trashes
Props
3
props

@David Johnson Whether or not there are differences between players is not relevant to this particular analysis.

Do individuals differ in their shooting ability? Yes, they do. But EVERY player has a lower shooting percentage from long distances than from in close - that's just hockey. So the writer is correct in that controlling for the distances from which the shots originated will provide a clearer picture of the goalies' skills. A goalie that only has to face shots from 50 feet away should have a high save percentage, whereas a goalie that only faces ten foot shots would be expected to let in more goals, even if they are highly skilled. You are correct that you could get an even better metric if you included the quality of the shooter, but that doesn't mean distance isn't worth looking at.

@money puck The only issue I have here is where you compared the adjusted save percentage to the goalie's career save percentage. Hiller's career sv% is lower than his adjusted, but to make this a realistic comparison you need to compare his career ADJUSTED sv% to last year's adjusted sv%. Otherwise you are comparing apples and oranges.

Avatar
#15 JFR
October 16 2014, 06:56PM
Trash it!
2
trashes
Props
0
props

One stat that I am very curious about is big saves in pressure situations. Velocity and rebounds are great, but we all know there are different players that show up in big situations and others that shrink.

To me, the Oilers game was a positive for Miller because he didn't sulk or show frustration. Opening home game, big contract Ect.. He seemed to stay calm and get better and better. In the third and SO he was big and allowed the team to come back.

Sagas will always be tough, because time and situation always play a role and those can't always be quantified. You have to see it, to understand it.

Avatar
#16 NM00
October 16 2014, 07:56PM
Trash it!
1
trashes
Props
1
props
hockey4life wrote:

there was a really good article awhile back that did a indepth statistical analysis between rask and price. was really informative. thought people would enjoy

http://www.sportsnet.ca/hockey/shot-quality-reveals-rasks-true-value/

Boyle had a number of great goalie articles on Sportsnet last year.

For me, one of the more interesting was this

http://www.sportsnet.ca/hockey/nhl/nhl-goalies-better-with-high-shot-volumes/

I wonder how much of these adjusted numbers would relate at all to goalie workload, from where all the extra shots came in 30+, 40+ shot games etc.

Great article Moneypuck...

Avatar
#17 money puck
October 16 2014, 07:59PM
Trash it!
1
trashes
Props
0
props

@TC

The comparison is actually 2013-14 SV% vs 2013-14 adjusted SV%, not career SV% in Table 4. I should have made that clearer.

Avatar
#18 money puck
October 16 2014, 08:07PM
Trash it!
1
trashes
Props
2
props
van wrote:

This analysis might overrate goaltenders with poor rebound control.

This is a very good point. I'm hopeful that when shot data matures, we'll be able to apply a more sophisticated version of this type of framework to assess goalie performance. Admittedly, at this stage there are limitations in the data that makes this analysis somewhat more theoretical, than conclusive.

Avatar
#19 TC
October 16 2014, 09:44PM
Trash it!
1
trashes
Props
2
props

@money puck You said: "On the other end of the spectrum, Jonas Hiller faced an incredibly difficult shot distribution. When adjusting for shot distribution he went from a SV% of .911 to an adjusted SV% of .918, which was slightly above his career SV% of .916."

This was not in the table, but in the text. Your point was that the adjusted SV% looks more in line with his career SV%, which you used as justification that he was a decent goalie last year and could have commanded more on the market. My point is that his career Adjusted SV% could very well differ from 0.916.

Anyway, a great article, and nice concept - definitely a better measure of goalie skill than the raw stats.

Avatar
#20 clrkaitken
October 17 2014, 08:06AM
Trash it!
0
trashes
Props
0
props

"Then in order to isolate the impact of team defense, I redistributed the total shots faced by each goalie by the average shot distribution at each of the four distances, then used the goalies actual save percentage at each distance to calculate an adjusted save percentage"

If I'm reading that right, this analysis is saying that the only thing that team defence has control over is the distribution of shot locations. When you arrive at your adjusted sv%, you've assumed everything else is the same; the only thing that is adjusted is the distribution of shots by location. Same amount of shots faced, same save percentages at the different distances, only the distribution of shots faced changes.

That seems like a wildly simplistic view of the effect team defence has on save percentage.

Without getting too deep into esoteric discussions about blocking/challenging shots and that sort of nuanced tactical elements, defence has the ability to impact the volume of shots a goalie faces as well, and that's not accounted for. Compare the amount of shots St. Louis gives up per game to a team like Toronto or Buffalo.

In 2013, a writer at PPP looked into the myth that Randy Carlyle's defence was "pushing shots to the outside", since this was the explanation given for why the Leafs suddenly had success in the lockout season. He found it was bunk.

http://www.pensionplanpuppets.com/2013/6/17/4436736/randy-carlyle-coaching-myths-part-five-shot-distance

Essentially, and this matches the shot ratios that Reimer and Bernier have in your tables above, they found that the Leafs weren't minimizing shots in dangerous areas and pushing those shots to the outside; the Leafs were giving up just as many shots from the areas in close to the ice, they just added a number of shots from father out.

I don't think you've isolated for team defence here. I think you've controlled for goalies that are outliers in terms of the distribution of the shot locations compared to league average.

Avatar
#21 Rob
October 17 2014, 08:27AM
Trash it!
0
trashes
Props
1
props

How about including some type of weighting metric related to the shooter's career SH% (or on-ice SH%),

Shots by Crosby would be weighted more heavily than shots by Colton Orr.

Avatar
#22 Rob
October 17 2014, 08:36AM
Trash it!
0
trashes
Props
1
props

Further, You could determine a dSv% = Sv% - (100-[shooter-weighted Sh%]) All at 5v5 or ES would probably be more representative.

A +ve dSv% would mean that a goalie's SV% is higher than expected from the shots faced, weighted by shooter quality.

Avatar
#23 Fogghorn
October 17 2014, 09:58AM
Trash it!
0
trashes
Props
1
props

Nice work. Nabakov looks good, how bad were the Isles dmen then?

Only quibble is it would be nice to see the data you built it on (total shots) Not really needed in the article but a link to it would be nice

Avatar
#24 Greg
October 17 2014, 01:51PM
Trash it!
0
trashes
Props
2
props

Great article! The limitations are obvious but it's doing the best you can do with the data quality and tools available at this point, and it's exposing some valuable insights already (don't pick mason in your pool!).

As data quality and analysis tools improve, this approach will seem primitive, but so will Corsi I suspect. We'll start seeing things not just like shot differentials and complex shot quality metrics that factor in shot type and player and shot speed, but pre-shot metrics that impact shot quality. A slapshot from the hashmarks isn't the same as a cross-ice one timer from the hashmarks, even if it's taken by the same player with the same speed. And that one timer is a different quality in a 2-on-1 break than a 5-on-4 power play. And that 5-on-4 power play cross-ice one timer is a different quality if there's traffic in front of the net vs clear shooting lane.

The ironic thing is that as the data and tools improve in cost and complexity, the returns will actually diminish. The most significant, impact full, actionable insights like Mason's false Goodyear and Hiller's false bad year can already be seen with this rudimentary data set and approach. Better data and tools will might make that more obvious, or measure it to a greater accuracy, and will expose some previously undetectable insights, but those insights will likely be less impactful given they were too small to be measured earlier.

I'm sure we'll continue to get surprising new insights that change the game for the better, and a lot of those will be in the area of goaltending given how little is currently known. But they'll be more and more like squeezing water from a rock as the biggest ones get exposed early - I wouldn't be at all surprised if the biggest ones have already been found, like the importance of puck possession or counter-productivity of face-punchers.

So, in short, despite it's limitations, I think this is very worthwhile analysis. This alone could keep a GM from making a career ending or franchise killing rash decision on a goaltender.

Comments are closed for this article.