Monday, February 26, 2007

The Coaching Class of 2004 | by Mike

With Year 2 of the Charlie Weis tenure officially in the books, the search is on for metrics by which to judge his performance. What numerical data can we use to check our biases and provide context? Over the long term, every Notre Dame coach will be judged by whether or not he delivered a national championship. However, while five different coaches have won national championships at Notre Dame, none of these championships was won in the coach's first two years. Thus it's too early to apply this standard to Weis.

One of the obvious measures by which to judge a coach in the first few years of his tenure is his performance relative to his predecessor. In his first two years, Charlie Weis guided Notre Dame to 19 wins. In the two preceding years, Tyrone Willingham guided Notre Dame to 11 wins. While Irish fans certainly appreciate those eight extra wins (and may appreciate the eight fewer soul-crushing losses even more), it's not clear how much this tells us about Weis at Notre Dame.

To provide context, we decided to look at the change in win totals for all the Division I coaches hired following the 2004 season. Twenty-two new head coaches were hired that offseason. Because there was such a large amount of turnover (changes at almost one-fifth of the D-I schools), we should have a decent-sized sample to examine. A Dennis Dodd article from early 2005 provided a comprehensive list of the new hires at that time, and that list is reproduced below. (Dodd's ranking of the hires is also included, for entertainment purposes only.)

Dodd's Rank Coach School
Dodd's Rank Coach School
1 Urban Meyer Florida
12 Frank Solich Ohio
2 Steve Spurrier South Carolina
13 Dave Wannstedt Pitt
3 Ed Orgeron Mississippi
14 Walt Harris Stanford
4 Mike Gundy Oklahoma State
15 Dick Tomey San Jose State
5 Les Miles LSU
16 Hal Mumme New Mexico State
6 Kyle Whittingham Utah
17 Terry Hoeppner Indiana
7 Tyrone Willingham Washington
18 Bronco Mendenhall BYU
8 Charlie Weis Notre Dame
19 Shane Montgomery Miami (OH)
9 Ron Zook Illinois
20 Skip Holtz ECU
10 Greg Robinson Syracuse
21 Brent Guy Utah State
11 Mike Sanford UNLV
22 Bill Cubit Western Michigan

Now let's compare the win totals at these programs in the two years preceding the new coach's arrival and in the two years following his arrival. The following list ranks the coaches by this change in win total:

Coach School Wins (03-04) Wins (05-06) Delta Dodd's Rank
Bill Cubit Western Michigan 6 15 +9 22
Skip Holtz ECU 3 12 +9 20
Bronco Mendenhall BYU 9 17 +8 18
Charlie Weis Notre Dame 11 19 +8 8
Dick Tomey San Jose State 5 12 +7 15
Frank Solich Ohio 6 13 +7 12
Urban Meyer Florida 15 22 +7 1
Terry Hoeppner Indiana 5 9 +4 17
Steve Spurrier South Carolina 11 15 +4 2
Ron Zook Illinois 4 4 0 9
Tyrone Willingham Washington 7 7 0 7
Les Miles LSU 22 22 0 5
Brent Guy Utah State 6 4 -2 21
Walt Harris Stanford 8 6 -2 14
Hal Mumme New Mexico State 8 4 -4 16
Mike Sanford UNLV 8 4 -4 11
Dave Wannstedt Pitt 16 11 -5 13
Mike Gundy Oklahoma State 16 11 -5 4
Greg Robinson Syracuse 12 5 -7 10
Kyle Whittingham Utah 22 15 -7 6
Ed Orgeron Mississippi 14 7 -7 3
Shane Montgomery Miami (OH) 21 9 -12 19

What do these numbers tell us, and how much significance should we attach to these deltas? Let's dispense with the low-hanging fruit. The first conclusion we can draw is that Dodd's prognostication abilities are lacking, and in this respect I think the numbers are fairly reliable. Dodd ranked just three of the nine coaches that improved on the prior two years' win total in the top half of new hires. Even worse, three of the six worst totals were produced by coaches Dodd ranked among the six best hires. While (for reasons that will be discussed below) one cannot look solely at the delta in evaluating coaching performance, the numbers are not flattering to Dodd.

Moving on to the more substantive analysis, the numbers suggest that one cannot look at the delta in isolation. The programs that had the highest win totals in 2003-2004 generally garnered fewer wins in 2005-2006 following the coaching change. Conversely, the teams with the fewest win totals in 2003-2004 generally improved their win total in 2005-2006 following the coaching change. I believe there are two phenomena at work here. First, the three programs in this group that had 20+ wins in '03-04 were left searching for new coaches in the winter of 2004 because their coach had left for greener pastures. Thus the new coaches at LSU, Utah, and Miami (OH) were all following coaches who had exceeded the historical performance standards at their program. One would expect matching such standards would be difficult.

Second, I believe it is harder to improve on a mediocre win total than an abysmal win total for three reasons: (1) when the win total is higher in the previous years, there are fewer opportunities to garner additional wins; (2) as the previous win total increases, the quality of teams against whom additional wins can be garnered will tend to increase; and (3) increases on truly bad performances may simply represent regression to the mean. This suggests that an individual coach's delta will be "inflated" relative to his peers where his absolute total of wins in 2005-2006 is middle-of-the-pack, and an individual coach's delta will be "undervalued" relative to his peers when his absolute total of wins in 2005-2006 is high.

A few examples illustrate this point. Les Miles, Tyrone Willingham, and Ron Zook all failed to improve on the preceding two years' win total. However, even if Miles had won every game the past two years, he would only have improved on Saban's record by four wins. Saban's record in his final two years at LSU was 22-4 and Miles matched this record in his first years. Yet if Miles had notched these four wins, he would have won back-to-back national titles, and would be regarded more highly than any other coach on this list. Conversely, Willingham matched Keith Gilbertson's 7-16 record and Ron Zook matched Ron Turner's 4-19 record. Willingham and Zook inherited bad situations, but there are many losses to bad teams in Willingham and Zook's tenures at Washington and Illinois. Would it have been too much to ask Willingham to beat, say, Air Force? The truth is that matching Nick Saban's performance is simply a greater achievement than matching Gilbertson or Turner's, even if the result is the same delta.

Urban Meyer's numbers tell a similar story. Four coaches improved their school's record by more wins than Meyer, but it is Meyer that has reached the mountaintop. That Florida's players were able to win 15 games in 2003-2004 despite Zook's mismanagement speaks volumes about just how much talent Meyer inherited. Yet to generate a +7 improvement in wins, Meyer had to defeat teams like Ohio State and LSU, teams that finished in the top five. Again we see that the difficulty of achieving marginal improvement increases with the previous success of a given program.

(I recognize that many attribute much of Miles' and Meyer's recent success to Bo Pelini, Jimbo Fisher, and Charlie Strong. While I can agree with this to an extent, a head coach deserves credit for hiring strong coordinators and must share the blame for hiring weak coordinators.)

In conclusion, while I think the change in a team's win total tells us something about coaching quality, such numbers must be viewed in light of the starting point. The raw delta is not as informative as it might initially appear. In this respect, I do see a positive sign for the Notre Dame faithful. As down as Irish fans are following the Sugar Bowl loss and the last-minute defection of a handful of recruits, the long-term trend for the program remains positive. Only three coaches in this sample were able to deliver improvement on a double-digit win total: former national champion coach Steve Spurrier (+4 on 11 wins), current national champion coach Urban Meyer (+7 on 15 wins), and our own Charlie Weis (+8 on 11 wins). Time will tell whether Weis can build the defense necessary to meet the real measuring stick of great coaches - a national championship.