Coaching Effect Continued

A few days ago, tentative as a newborn horse, I took my first steps towards bashing together some statistical analysis of my own. I was trying to help address one specific issue that might be relevant to the upcoming Leafs’ season: against which statistical measuring geegaws, come the end of the season, should we assess the performance of the team’s coaching staff?  In other words, what numbers should I look at to try and figure out whether Ron Wilson and his staff are doing a good job?

The first conclusion I came to was that it is incredibly difficult,  possibly related to some sort of an international conspiracy, to embed any kind of a chart in WordPress.  The second, perhaps more illuminating conclusion was that there appear to be wide year-to-year, essentially random, variations in teams’ goals for and goals against ledgers.  Absent an enormous – on the order of 20% or more – change, therefore, it is probably not possible to confidently ascribe any meaning to differences in the year to year totals.  In other words, if you’re trying to divine something about the efficacy of an NHL team’s coaching staff, you might as well dig through goat entrails as comb through the Goals For and Goals Against numbers.  They’re likely equally informative on the subject.

After I posted the raw data, it occurred to me that the changes to last year’s Leafs roster following January 31st (hereinafter known throughout the land of Blue and White as “Emancipation from Vesa Toskala Day”) might illustrate the point too.  It occurred to me that, given the large turnover of the roster on that day (White, Hagman, Mayers, Blake, Stajan and Toskala all out, Phaneuf, Giguere and Sjostrom in, plus help summoned from the minors), you might look at the first 57 games as one season, and the final 25 games as another mini season.  I thought this would be interesting because, given the Olympic break and the compressed schedule (25 games in 68 days, including the three week break, so really 25 games in about 47 days), it was not very likely that any substantial on-ice instruction could occur in the post-trade timeframe.  In other words, examining the pre- and post-trade data separately comes very close to affording an opportunity to examine two data sets in isolation from the coaching effect – because no significant coaching of a substantially changed team could have occurred following the trade.

Based on the subtraction of the alleged goaltending of Vesa Toskala alone, I felt confident that the Leafs’ goals against numbers would be vastly improved in the post-trade period.  A quick look at Figure 1 (oooh, how text book-y of me) shows that the data bear out that assumption:

Fig. 1: Things Got Better When Dion Came and Toskala Left

As you can see, I’ve taken the actual data observed in both the pre- and post-trade period and prorated them over 82 games to try and get to a place where we can compare apples to apples.  Interestingly, the Leafs scoring prowess *cough* remained essentially undisturbed, as Dion Phaneuf’s Leafs continued to put biscuits in the basket at almost exactly the same rate as Team Stajan (213 GF vs. 214).   As suspected, goals against took a nose-dive when Toskala was deported, changing the Leafs from a 283 goals against squad to a team that was on pace to have given up 216 over an 82 game schedule.

First things first: to go back to the original point of the exercise, the Leafs changed from a team that would surrender 283 goals over an 82 game schedule to a team that would cough up 216, and that had to have happened in an atmosphere when the coaches were unable to get the team together for any substantial on-ice drills or systems instruction.  In other words, the goals against dropped by approx 67 total goals or .8 GA per game without any significant contributing coaching effect

Some other things caught my eye about the data, though:  to put the apparent change in the Leafs’ defensive prowess following the Phaneuf and Giguere trades, remember that the Leafs’ went from a 283 GA pace to a 216 GA pace.   The Edmonton Oilers, who finished in 30th place in the league last year, and who finished the season with the league-worst goals against total, gave up 284 opposition tallys.  By contrast, 216 goals against, would have put the Leafs tied with Detroit for the 8th best GA number in the league, behind only San Jose, New Jersey, Phoenix, Chicago, Calgary, Buffalo and Boston.  Among that group, only Calgary failed to reach the playoffs.  That would be the same Calgary team that finished the year with Matt Stajan and Vesa Toskala on its roster.  See how these things come full circle?

Lastly, I hadn’t realized that the Leafs’ GF rate went virtually unchanged in the post-trade period.  It’s almost difficult to believe that a team could trade their then  top point-getter (Stajan), their then leading goal scorer (Hagman) and their 2nd leading point scorer among defencemen (White), yet suffer virtually no drop off in their offensive success rate  (source: NHL.com story).  They also traded Jason Blake who, when not skating eighteen laps around the offensive zone in the course of a seven-minute shift, then firing a 45 foot shoot into the precise geographic middle of the goaltender’s chest protector, occasionally seemed to rack up some points.

This last point screams out to me that the players the Leafs shipped out on the 31st were nothing truly special, just a bunch of guys who got points because they were there, the skating embodiment of the “replacement player” involved in GVT calculations.  I can’t support that hypothesis at this point with any hard data, but it sure looks like the points that those stiffs collected were the points that would be collected by whichever collection of stiffs the Leafs chose to throw over the boards (with apologies to Nicklas Hagman and Ian White for the “stiff” thing.  I liked you both).

Obviously, there are dangers involved in extrapolating full season numbers out of smaller data sets;  it’s exactly that process that every year has some idiot projecting that [insert random scrub here] “is on pace for a 164 goal season this year” as the highlights of his two-goal first game roll on TSN.   I understand the dangers of paying too much attention to data from small sample sizes.

In fact, that last point – small sample size – got me thinking about another possible explanation for the Leafs’ apparent improvement in the post-trade period, one that I hope to take a look at in the next post in this series.   Stay tuned.