Coaching Effect Continued

A few days ago, tentative as a newborn horse, I took my first steps towards bashing together some statistical analysis of my own. I was trying to help address one specific issue that might be relevant to the upcoming Leafs’ season: against which statistical measuring geegaws, come the end of the season, should we assess the performance of the team’s coaching staff?  In other words, what numbers should I look at to try and figure out whether Ron Wilson and his staff are doing a good job?

The first conclusion I came to was that it is incredibly difficult,  possibly related to some sort of an international conspiracy, to embed any kind of a chart in WordPress.  The second, perhaps more illuminating conclusion was that there appear to be wide year-to-year, essentially random, variations in teams’ goals for and goals against ledgers.  Absent an enormous – on the order of 20% or more – change, therefore, it is probably not possible to confidently ascribe any meaning to differences in the year to year totals.  In other words, if you’re trying to divine something about the efficacy of an NHL team’s coaching staff, you might as well dig through goat entrails as comb through the Goals For and Goals Against numbers.  They’re likely equally informative on the subject.

After I posted the raw data, it occurred to me that the changes to last year’s Leafs roster following January 31st (hereinafter known throughout the land of Blue and White as “Emancipation from Vesa Toskala Day”) might illustrate the point too.  It occurred to me that, given the large turnover of the roster on that day (White, Hagman, Mayers, Blake, Stajan and Toskala all out, Phaneuf, Giguere and Sjostrom in, plus help summoned from the minors), you might look at the first 57 games as one season, and the final 25 games as another mini season.  I thought this would be interesting because, given the Olympic break and the compressed schedule (25 games in 68 days, including the three week break, so really 25 games in about 47 days), it was not very likely that any substantial on-ice instruction could occur in the post-trade timeframe.  In other words, examining the pre- and post-trade data separately comes very close to affording an opportunity to examine two data sets in isolation from the coaching effect – because no significant coaching of a substantially changed team could have occurred following the trade.

Based on the subtraction of the alleged goaltending of Vesa Toskala alone, I felt confident that the Leafs’ goals against numbers would be vastly improved in the post-trade period.  A quick look at Figure 1 (oooh, how text book-y of me) shows that the data bear out that assumption:

Vesa Toskala's Goaltending Ought to be Tried for Crimes Against Humanity
Fig. 1: Things Got Better When Dion Came and Toskala Left

As you can see, I’ve taken the actual data observed in both the pre- and post-trade period and prorated them over 82 games to try and get to a place where we can compare apples to apples.  Interestingly, the Leafs scoring prowess *cough* remained essentially undisturbed, as Dion Phaneuf’s Leafs continued to put biscuits in the basket at almost exactly the same rate as Team Stajan (213 GF vs. 214).   As suspected, goals against took a nose-dive when Toskala was deported, changing the Leafs from a 283 goals against squad to a team that was on pace to have given up 216 over an 82 game schedule.

First things first: to go back to the original point of the exercise, the Leafs changed from a team that would surrender 283 goals over an 82 game schedule to a team that would cough up 216, and that had to have happened in an atmosphere when the coaches were unable to get the team together for any substantial on-ice drills or systems instruction.  In other words, the goals against dropped by approx 67 total goals or .8 GA per game without any significant contributing coaching effect

Some other things caught my eye about the data, though:  to put the apparent change in the Leafs’ defensive prowess following the Phaneuf and Giguere trades, remember that the Leafs’ went from a 283 GA pace to a 216 GA pace.   The Edmonton Oilers, who finished in 30th place in the league last year, and who finished the season with the league-worst goals against total, gave up 284 opposition tallys.  By contrast, 216 goals against, would have put the Leafs tied with Detroit for the 8th best GA number in the league, behind only San Jose, New Jersey, Phoenix, Chicago, Calgary, Buffalo and Boston.  Among that group, only Calgary failed to reach the playoffs.  That would be the same Calgary team that finished the year with Matt Stajan and Vesa Toskala on its roster.  See how these things come full circle?

Lastly, I hadn’t realized that the Leafs’ GF rate went virtually unchanged in the post-trade period.  It’s almost difficult to believe that a team could trade their then  top point-getter (Stajan), their then leading goal scorer (Hagman) and their 2nd leading point scorer among defencemen (White), yet suffer virtually no drop off in their offensive success rate  (source: NHL.com story).  They also traded Jason Blake who, when not skating eighteen laps around the offensive zone in the course of a seven-minute shift, then firing a 45 foot shoot into the precise geographic middle of the goaltender’s chest protector, occasionally seemed to rack up some points.

This last point screams out to me that the players the Leafs shipped out on the 31st were nothing truly special, just a bunch of guys who got points because they were there, the skating embodiment of the “replacement player” involved in GVT calculations.  I can’t support that hypothesis at this point with any hard data, but it sure looks like the points that those stiffs collected were the points that would be collected by whichever collection of stiffs the Leafs chose to throw over the boards (with apologies to Nicklas Hagman and Ian White for the “stiff” thing.  I liked you both).

Obviously, there are dangers involved in extrapolating full season numbers out of smaller data sets;  it’s exactly that process that every year has some idiot projecting that [insert random scrub here] “is on pace for a 164 goal season this year” as the highlights of his two-goal first game roll on TSN.   I understand the dangers of paying too much attention to data from small sample sizes.

In fact, that last point – small sample size – got me thinking about another possible explanation for the Leafs’ apparent improvement in the post-trade period, one that I hope to take a look at in the next post in this series.   Stay tuned.

Setting and Measuring Expectations: The Leafs Coaching Staff

No Strategy yet HPIM0785
In search of a clean slate for the X's and O's

For Leafs fans, the upcoming season will be an important one. Though it is (once again) extremely unlikely that the Leafs could win the big silver beer stein on offer at the end of the postseason tournament, fans of the team will be watching very closely for signs that any of the existing questions about the team might be answered. We’ll dig through the statistics like the oracles of old pawed through goat entrails, looking for evidence that augers well for a brighter future ahead. It is pretty safe to assume that Brian Burke and his staff will be engaging in a similar process.

Many of those questions concern individual players: what, for example, can we realistically expect from players like Jonas Gustavsson, Luke Schenn, Tyler Bozak and Nikolai Kulemin, all of whom are approaching their likely peak athletic potential in the next few years.   Other questions concern more collective issues:  what improvement can we expect from the Leafs’ power-play and penalty killing units?

All of those questions merit discussion, but they all relate to issues about the players; with Ron Wilson entering his third season as Maple Leafs head coach, and keeping in mind that last season in particular represented a disappointing step backwards, it’s safe to say that questions must also remain about the suitability of the current staff for the task ahead.

One of the things I like most about the hockey blogosphere is the very strong tendency to attempt to quantify, measure and make concrete and expressible these sorts of issues.  When we speak of “issues” and “questions” about the coaching staff, the reality is that there must be some set of performance metrics against which it is reasonable measure the observed outcome of this season, in an effort to dispassionately judge whether the coaches are making a discernible difference in the team’s play (and whether that difference represents an improvement).

Statistical analysis isn’t my strong suit, and I don’t pretend to have the facility with numbers that many other hockey bloggers have ably demonstrated, but I thought I’d try my hand at attempting to cobble together an answer to this last question.  What types of numbers should we look for when attempting to grade Messrs. Wilson, Hunter and Acton at the end of this season.  Please accept this analysis for what I hope it is:  a starting point for the discussion, and a jumping off point for others with the statistical chops that are absent from my toolkit.  Criticisms, comments and refinements are welcome – put ’em in the comments below!

I wish I could figure out a way to embed the tables I compiled directly into this post, but two hours of futzing about with Google, Google docs, WordPress, Excel and Numbers have failed to surrender any such secrets, assuming they exist.  Unfortunately, therefore, I have to just insert a link to the table I compiled.  All data are sourced from hockey-reference. com.

I thought the most logical place to start in assessing the performance of the coaches would be year over year changes in goals for and goals against.  I compiled the goals for and goals against data for all 30 teams in each season since the lockout, calculated the percentage change in each from the previous year.  I then tried to normalize the percentage change data by calculating the average change each year and the standard deviation of the data.  I then selected out those results that lie between one and two standard deviations away from the mean (classified as “moderately exceptional”), and those results that lie two standard deviations or more away from the mean (classified as “significant”).

Link to Google docs spreadsheet re: YOY data: change in GF and GA

Assuming that the year-to-year changes are normally distributed, if I remember my statistics class correctly, the results that are interesting are those that fall more than one or two standard deviations from the mean. Those are the results I mentioned above, with the moderate desirable increments marked in light green, the significant desirable increments marked in dark green, the moderate undesirable increments marked in pink, and the significant undesirable increments marked in red.

If I’m reading all of the data correctly, it would appear that the standard deviation of the Goals Against data is typically between about 9 and 12 per cent.  Thus, an increase or decrease of anything less than 9 to 12 per cent, statistically speaking, represents the mushy random middle, results in the 68% of data that cluster around the mean in a normal distribution.  If I am applying the theory correctly, it would be unwise to come to any conclusion that the team’s performance had either improved or deteriorated based on data of this nature.  To make that sort of judgement, I would suggest that to even make a weak judgment about significant differences in performance, we would need to observe an increment (or reduction) of between 9-12% and 18 to 24% (these would be the results between one and two standard deviations from the mean).  Variances of more than 18 to 24% from last year’s data could confidently be said to represent a clear indication of differential performance.

Two thoughts come to my mind: first, it’s important to keep in mind the (perhaps obvious) but important point that increases or decreases in a team’s goals for or goals against are not solely attributable to coaching.  In fact, it’s probably a live question whether coaching can be said to have a demonstrable effect upon the results at all.  Certainly, the old saw is that “you can’t teach scoring,” though it is generally believed that coaches and their systems can and do have a more pronounced effect upon the defensive side of the game (and, by extension, the goals against ledger).    If anyone has any thoughts on how to examine the evidence in that regard, I’d love to hear about it.

Second, the numbers involved are fairly large. I think the data seem to be telling us that wide variances in the numbers may be expected from year to year for purely random (or at least statistically uninformative) reasons.

If that last conclusion is correct, unless there is an enormous change in the Leafs goals against totals this year (more than +/- 20%, which in practice would translate into about a 54 goal change either way), it seems that we ought not to make any judgements about the performance of the coaching staff based upon these numbers.

Thoughts?

Jason Blake: When More is Less

Tonight, over at Pension Plan Puppets, in the Leafs/Bruins live game thread, mf37 posted some numbers about Jason Blake’s shooting percentage.  Essentially, the stats indicated that, between 2003 and 2007 (i.e. during his final seasons with the Islanders), Blake’s shooting percentage doubled (and in ’07 nearly tripled) over his then current career average.  Since joining the Leafs (and signing a hefty five-year contract – curse you, JFJ!) his shooting percentage has dropped to a number so low, you’d think it was expressing a person’s chance of getting hit by lightning while winning the lottery, being abducted by our alien overlords and riding a three-legged dog backwards to a nineteen length Triple Crown victory.

It got me thinking…

Observed phenomenon: Jason Blake takes a lot of shots from “the perimeter”, which is a polite way of saying that he was standing in the parking lot and unable to directly observe his intended target at the time of launching.

Known facts, and important (but blindingly obvious) inference to be drawn from them: NHL goalies (with the lamentable and all-too obvious exception of Andrew Raycroft) are not blind.  Most of them (again, except for Raycroft) have the ability to exert some amount of control over the movement of their extremities.  As a result, NHL goaltenders only infrequently whiff on shots taken from different continents.  Shooting from long distances is not, therefore, an effective strategy of scoring goals.

Statistical evidence:

This is what hockey-reference.com has to say about what Jason Blake has done in his career, in terms of offensive performance:

Season Team GP SOG/G S Pct. AVG TOI
1998-99 Los Angeles Kings 1 5 20 17:13
1999-00 Los Angeles Kings 64 2.05 3.8 11:17
2000-01 L.A./NY Islanders. 47 2.13 5 13:40
2001-02 New York Islanders 82 1.66 5.9 12:54
2002-03 New York Islanders 81 3.12 9.9 17:38
2003-04 New York Islanders 75 3.24 9.1 18:49
2005-06 New York Islanders 76 4 9.2 18:47
2006-07 New York Islanders 82 3.72 13.1 18:09
2007-08 Toronto Maple Leafs 82 4.05 4.5 17:49
2008-09 Toronto Maple Leafs 13 4.07 3.8 16:35
Career 3.08 8.1 16:22

Legend: GP = Games Played, SOG/G=Shots on Goal per game, S Pct.=Shooting Percentage, AVG TOI=Average time on ice.

Throw out the ’98-99 season (small sample size) and this year (small sample, incomplete data), and you’re left with 8 (more or less) complete seasons to consider.  Here’s what I noticed:  beginning in 2002, Blake’s average time on the ice per game goes way up – increasing by almost 40% from about 13 minutes a game in ’01/’02 to almost 18 minutes a game in ’02-’03.  Also in 2002, the goals start to bang home for Jason, as his shooting percentage jumps to 9.9%;  prior to that, Blake was lighting the lamp at something like a 4 to 6% clip.  Blake’s average ice time stays pretty steady, from then on, in the 18-19 minutes per game region.

Now look at what happens to his shots on goal per game.  That figure jumps from a career low 1.66 per game in  ’01-’02 to 3.12 per game the very next season – the same season his average icetime increases by 40% per game.

What the numbers show is that beginning in 2002/03, Blake was spending about 40% more time on the ice than he had before, but he began shooting the puck about twice as often as in the past.  That’s a lot of extra shots, proportionally, to fit into the extra ice time.  For a few years with the Islanders, it seems to have worked out because he was scoring goals roughly twice as often too, at the 9 – 13% clip.

Consider these numbers in the context of a hypothesis:  a tired skater is more likely to shoot the puck from a long distance. Rather than skate and drive towards the net, a fatigued player will – more often than not – elect to shoot from where he is when he receives the puck.

Blake is playing about just a bit less now than he did in his final years with the Islanders (the years of 9-13% shooting success).  He is shooting the puck about the same number of times, on average, if not a little bit more.  We have observed that – since he’s been in Toronto, anyway – he frequently shoots the puck from long distances.  It is unlikely that Blake was shooting from these distances while in Long Island;  it simply beggars belief that they’d be going in as frequently as they did for him;  it would take a major league marathon of sustained and repeated whiffage by a series of goaltenders, over a period of four years, for that to be true.   The key point is this – the long shots we’ve seen Blake take as a Leaf aren’t being taken in addition to the blasts he habitually took when he was an Islander.

Rather, I think the data suggest that Blake is currently replacing shots from closer in to the net  (i.e the quality shots he took as an Islander) with long distance bombs that have little or no chance of success.  One obvious explanation for that phenomenon is related to the “tired players take long shots” theory.  In short, the numbers suggest that  – now in his mid 30s, and with well-documented health concerns – Blake may well not be up to the challenge physically, and that fatigue or lack of conditioning is preventing him from scoring at the rate he previously did.

Discuss.  Am I missing something?