Robust Rankings for College Football
Contents
1. rankings that intelligently takes into account the possibility of I switched games but without knowing anything else about the switched games This is characteristic of robust optimization approaches which differentiates them from stochastic ones This paper is organized as follows Section 2 reviews the CM method and discusses the data we use in the paper We also describe our focus on FBS rankings even though our data contains non FBS data as well Section 3 then empirically investigates the sensitivity of the top rankings in the CM method to modest changes in the win loss outcomes of games between teams with losing records In Section 4 we propose and study the MINLP which we solve to make the CM method robust to modest changes in the data In Section 5 we provide several examples and repeat the experiments of Section 3 except with our own robust rankings We conclude that our rankings are significantly less sensitive than the CM rankings Finally we conclude the paper with a few final thoughts in Section 6 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings 2 The Colley Matrix Method and Our Data Colley proposed the following method for ranking teams called the Colley Matrix CM method 12 The CM method uses only win loss information as required by the BCS system and autom2. Journal of Quantitative Analysis in Sports be the index of the top t 25 teams under z Also let 7 be the robust rankings determined by W W A where A D L for some L As in Section 3 we investigate the distributions of the top team switch measures H L y faw wW W W A AeD L J for each football year y 2006 2011 and each L 1 2 As in Section 3 we then actually combine H L 2006 H L 2011 into a single histogram for each L the results of which are shown in Figure 4 50 40 30 Frequency 20 10 0 50 40 30 Frequency 20 10 0 One Game Switched L 1 for P 5 mean 1 8 mediam 0 min 0 max 10 stdev 2 4 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Switch Measure Two Games Switched L 2 forl 5 mean 2 5 media 2 min 0 max 14 stdev 2 8 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Switch Measure Figure 4 Histograms illustrating the sensitivity of the CM rankings of the top teams to modest changes in the win loss outcomes of inconsequential games We can compare Figure 4 directly with Figure of Section 3 Note in partic 16 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings ular that all histograms are plotted on the same scale
3. Looking at both the plots and summary statistics we see very clearly that the distributions in Figure 4 are con siderably lower than those in Figure 1 This demonstrates that indeed our CM rankings are less sensitive than the CM rankings under the same number of switches L 1 or L 2 and higher values of I will further stabilize the robust rankings We conduct one last set of experiments to compare directly the sensitivity of the CM and CM rankings Again we fix 5 and take L 1 2 For each L the histogram corresponding to L in Figure 4 is based on all possible switches of L games For each of these same switches we also calculate the switch measure for the regular CM rankings just as in Section 3 In Figure 5 we then plot the point x y where x is the CM switch measure for that instance and y is the CM switch measure for the same instance Then over all switches to show the frequency for various x y pairs we use a bubble chart where the area of a bubble is proportional to the frequency of its x y center Please also note that the line x y is plotted for reference L 1 forl 5 L 2 for 5 ye ye o oO o N N N o Oo o is ak oO N oO o N o Robust Switch Measure a Robust Switch Measure 0 10 20 0 10 20 Colley Switch Measure Colley Switch Measure Figure 5 Bubble plots of CM versus CM switch measures CM has 5 in both plots Also L 1 2 and
4. champion especially if the bowl match ups are chosen well However there has always been considerable debate over how to choose the bowl match ups Prior to the year 1998 the bowl match ups were made in a less formal man ner than today One of the most important factors for determining the match ups were the human poll rankings such as the AP Poll provided by the Associated Press As a result the poll rankings have long had considerable influence in college football Although computer generated rankings existed at the time they were not used with any consequence In fact prior to 1998 the national champion was gen erally considered to be the team ranked highest in the polls after completion of the bowl games However even this simple rule was problematic because the final polls could disagree on the top ranked team This occurred for example after the 1990 season Since 1998 the Bowl Championship Series BCS has been implemented to alleviate the ambiguity of determining the national champion in college football 1 The BCS procedure is essentially as follows At the end of the regular season multiple human poll and computer rankings are combined using a simple mathe matical formula to determine a BCS score for each FBS team The FBS teams are then sorted according to BCS score which determines the BCS rankings Then fol lowing a set of pre determined rules and policies ten of the top teams according to the BCS rankings are matche
5. introduced here KEYWORDS rankings college football robust Author Notes Department of Management Sciences University of Iowa Iowa City IA 52242 1994 USA Email samuel burer uiowa edu The research of this author was supported in part by NSF Grant CCF 0545514 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings 1 Introduction College football has been played in the United States since the 1860s and enjoys enormous popularity today Colleges and universities of all sizes across the country sponsor teams that play each year or season within numerous conferences and leagues We focus our attention on teams in the Football Bowl Subdivision FBS Roughly speaking the FBS includes the largest and most competitive collegiate football programs in the country In 2011 there were 120 teams in the FBS each of which typically played 12 games per season not including post season games For historical reasons the FBS teams do not organize themselves into an elimination tournament at the end of a season to determine the best team or national champion Instead the most successful teams from the regular season are paired for a group of extra games called bowl games In particular every FBS team plays in at most one bowl game Ostensibly the bowl games serve to determine the national
6. is a user specified vector p norm It is not immediately clear that 4 can be solved in a tractable manner either practically or theoretically We focus on the case p 2 and argue next that even though 4 is a mixed integer nonlinear In the first version of this paper we focused on the case p oo for which 4 can be solved as a 10 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings program that appears to be NP hard we can devise an exact solution procedure that works well in practice at least for relatively small numbers of inconsequential games and relatively small values of I So fix p 2 We first transform 4 by minimizing the maximum squared norm and separating the objective function via 3 _ pyre TER AT IIA AR THe min Cr b eee Cr A A e 5I A Al 5 By introducing an auxiliary variable t we can rewrite the inner maximization using a set of explicit linear constraints min Cr bl t 6 s t b6 Cr A A e 4 A A el lt t YA EDT It is important to note that A is no longer a variable Rather there is one linear constraint in r t for each specific A D T As such 6 is a strictly convex quadratic program with a unique optimal solution that can in principle be solved by CPLEX for example There is still one challen
7. optimization 7 and ro bust systems of equations 13 Ultimately this leads to a mixed integer nonlinear programming MINLP model which serves as the robust version of the system Cr b Solving this MINLP provides a robust ratings vector r which is then sorted to obtain the final robust rankings just as in the CM method We remark that there exist other ranking methods that utilize optimization see for example 10 11 15 Our method depends on a user supplied integer I gt 0 which is the number of switched inconsequential games to protect against In this way the parameter I signifies the conservatism of the user mimicking the robust approach of 8 For example if the user is not worried about inconsequential games affecting the top rankings at all then he she can simply set I 0 protect against no games chang ing and then the ratings vector r is simply the usual CM ratings On the other hand choosing I 10 means the user wants robust rankings that take into account the possibility that up to 10 inconsequential games happen to switch It should be pointed out that there is no best a priori choice of I rather it will usually depend on the user s experience and conservatism It is important to point out that our approach is not stochastic For example we do not make any assumptions about the distributions of switched inconsequential games and we do not study average rankings Rather we calculate a single set of
8. r but prior to computing the FBS rankings we will delete the non FBS teams from r before sorting and ranking In this way the FBS rank ings are computed using all available FBS data but we focus our rankings on just the FBS teams Colley handles non FBS teams in a more involved pre processing step but he likewise maintains a focus on FBS teams 2 3 Sensitivity of the Colley Matrix Method In this section we empirically investigate the sensitivity of the Colley Matrix CM rankings to modest changes in the win loss outcomes of inconsequential games We specifically focus on the sensitivity of the rankings of the top teams Given the win matrix W let 7 be the permutation vector encoding the CM rankings for W Given an integer t n define T i n m lt t to be the index set of the top t teams ranks between 1 and t In contrast let w 0 1 be given and define Za Wa Bis f n S Wy Wa lt s to be the bottom teams winning percentages less than w As long as t is relatively small and w is relatively close to 0 it is highly likely that T and B are disjoint For example in all experiments we take t 25 and w 0 3 and find that for the years 2006 2011 T and B never intersect Note that B does not depend on the rankings m whereas T does We call a game inconsequential if it has occurred between two bottom teams 7 7 B and we define T i j EBxB i lt j Wi Wj gt 0 t
9. rankings 7 based on W Because 7 is also calculated allowing that all games might switch it must hold that 7 a More precisely the set of scenarios b optimized over in problem 4 is the same for both W and W because T is so large and so 7 7 Between the two extremes I 0 as sensitive as CM and I gt N com pletely insensitive it is reasonable to expect the rankings will become less sen sitive as I increases and we now exhibit this at the intermediate fixed value of I 5 So let 7 be the I robust rankings determined by the original W and let T 14 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM 10 15 20 25 20 25 Figure 3 Colley Matrix Plus rankings versus I for all years except 2008 Burer Robust Rankings 0 p SO e OE E eee e SSS es TST_ _ T a poco co OO O l EEEN o a cats e SS peano A ae 3 5 7 9 1 3 5 7 9 a 2006 b 2007 m 0 o SoS o p Eaa Eei TE S a O ee 20 Eaa 25 3 5 7 9 1 3 5 7 9 c 2009 d 2010 Op OO se O m p 20 MMMM coe 25 1 3 5 7 9 e 2011 15 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to
10. would say that l 8 is more robust than I 3 by construction and we investigate this empirically in the next subsection but in the absence of further analysis we believe it can be challenging to compare any two rankings objectively So here we would simply like to point out some observations that we believe are relevant concerning the robust rankings as I changes First as I increases the rankings are sensible compared to T 0 For example we do not see teams making huge jumps in the rankings In fact the ranking of each team moves by at most two positions over all I lt 10 Second the changes in the rankings appear to involve several separate groups of closely ranked teams and each group switches ranks among itself only For ex ample Utah and Texas Tech switch places while Brigham Young Missouri and North Carolina adjust to accommodate a decline in the rank of Brigham Young Two additional groups are Oklahoma St Florida St Virginia Tech and Michigan St Ball St Boston College Third the rank trends are not necessarily monotonic i e a team s rank can increase and then decrease or decrease and then increase as I increases However 12 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings Rank 0 p r 2 T 3 4 5 T 6 T 7 8 9 10
11. 1 Oklahoma Oklahoma Oklahoma Oklahoma Oklahoma Oklahoma 2 Florida Florida Florida Florida Florida Florida 3 Texas Texas Texas Texas Texas Texas 4 Utah Texas Tech Texas Tech Texas Tech Texas Tech Texas Tech 5 Texas Tech Utah Utah Utah Utah Utah 6 Alabama Alabama Alabama Alabama Alabama Alabama 7 Penn State Penn State Penn State Penn State Penn State Penn State 8 Boise St Boise St Boise St Boise St Boise St Boise St 9 Southern Cal Southern Cal Southern Cal Southern Cal Southern Cal Southern Cal 10 Ohio State Ohio State Ohio State Ohio State Ohio State Ohio State 11 Cincinnati Cincinnati Cincinnati Cincinnati Cincinnati Cincinnati 12 Georgia Tech Georgia Tech Georgia Tech Georgia Tech Georgia Tech Georgia Tech 13 Georgia Georgia Georgia Georgia Georgia Georgia 14 TCU TCU TCU TCU TCU TCU 15 Pittsburgh Pittsburgh Pittsburgh Pittsburgh Pittsburgh Pittsburgh 16 Oklahoma St Oklahoma St Oklahoma St Oklahoma St Oklahoma St Virginia Tech 17 Florida St Florida St Florida St Florida St Florida St Florida St 18 Virginia Tech Virginia Tech Virginia Tech Virginia Tech Virginia Tech Oklahoma St 19 Michigan St Michigan St Ball St Michigan St Ball St Ball St 20 Ball St Ball St Michigan St Ball St Michigan St Boston College 21 Boston College Boston College Boston College Boston College Boston College Michigan St 22 Brigham Young Brigham Young Brigham Young Missouri Missouri Missouri 23 Missouri Missouri M
12. 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings are quite sensitive to changes in just a few inconsequential games For L 1 the mean of 6 W W is 5 1 and the maximum or worst case is 20 and both of these statistics increase noticeably for L 2 The standard deviation is also relatively large and increases between L 1 and L 2 In our opinion such sensitivity is an undesirable property of the CM rankings especially since rankings are relied upon so heavily in college football In Table 1 we provide an illustrative albeit worst case example of how the CM rankings can change when the outcome of a single inconsequential game is switched The first column of teams contains the top 25 CM rankings for 2007 These teams comprise T in our experiments The second column shows the per turbed rankings when the result of the inconsequential game between Marshall and Rice is switched Note that in 2007 both Marshall and Rice had winning per centages below 30 and Marshall beat Rice in real life In both columns a bold typeface indicates a ranking that changes For this example 6 W W 20 and one can see quite plainly that there is a significant amount of shuffling in the rank ings Some of the shuffling is logical For example West Virginia beat Marshall in real life and when Marshall loses to Rice hypothetically West Virginia then be comes a weaker team with a lower ranking What is unclear however is why West Vir
13. 4 41 PM Burer Robust Rankings 5 Glitch in the Colley Matrix puts Boise State at 10 LSU 11 in BCS standings http www sports ratings com college_football 2010 12 glitch in the colley matrix puts boise state at 10 lsu 11 in bcs standings html December 2010 Accessed October 4 2011 6 sd Wes Colley Alabama Huntsville researcher talks about his BCS error http www al com sports index ssf 2010 12 wes_colley_alabama huntsville html December 2010 Accessed October 4 2011 7 A Ben Tal L El Ghaoui and A Nemirovski Robust Optimization Prince ton Series in Applied Mathematics Princeton University Press Princeton NJ 2009 8 D Bertsimas and M Sim The price of robustness Operations Research 52 1 35 53 2004 9 T P Chartier E Kreutzer A N Langville and K E Pedings Sensitivity and stability of ranking vectors SIAM J Sci Comput 33 3 1077 1102 2011 10 B J Coleman Minimizing game score violations in college football rankings Interfaces 35 6 483 497 2005 11 B J Coleman Ranking sports teams A customizable quadratic assignment approach Interfaces 35 6 497 510 2005 12 W N Colley Colley s bias free college football ranking method The Colley Matrix explained Manuscript 2002 Available at http www colleyrankings com 13 L El Ghaoui and H Lebret Robust solutions to least squares problems with uncertain data SIAM J Matrix Anal Appl 18 4 103
14. 5 1064 1997 14 A Y Govan A N Langville and C D Meyer Offense defense approach to ranking team sports J Quant Anal Sports 5 1 Art 4 19 2009 15 D S Hochbaum Ranking sports teams and the inverse equal paths problem In Proceedings of the Second International Workshop on Internet and Network Economics WINE 2006 Lecture Notes in Computer Sciences volume 4286 pages 307 318 2006 16 ILOG Inc ILOG CPLEX 12 2 User Manual 2011 17 J P Keener The Perron Frobenius theorem and the ranking of football teams SIAM Rev 35 1 80 93 1993 19 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports 18 K Massey Statistical models applied to the rating of sports teams Master s thesis Bluefield College 19 MATLAB version 7 11 0 R2010b The MathWorks Inc Natick Mas sachusetts 2010 20 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM
15. Journal of Quantitative Analysis in Sports Manuscript 1405 Robust Rankings for College Football Samuel Burer University of Iowa 2012 American Statistical Association All rights reserved Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Robust Rankings for College Football Samuel Burer Abstract We investigate the sensitivity of the Colley Matrix CM rankings one of six computer rankings used by the Bowl Championship Series to hypothetical changes in the outcomes of actual games Specifically we measure the shift in the rankings of the top 25 teams when the win loss outcome of say a single game between two teams each with winning percentages below 30 is hypothetically switched Using data from 2006 2011 we discover that the CM rankings are quite sensitive to such changes To alleviate this sensitivity we propose a robust variant of the rankings based on solving a mixed integer nonlinear program which requires about a minute of computation time We then confirm empirically that our rankings are considerably more robust than the basic CM rankings As far as we are aware our study is the first explicit attempt to make football rankings robust Furthermore our methodology can be applied in other sports settings and can accommodate different concepts of robustness besides the specific one
16. atically adjusts for the quality of a team s opponent also called the team s strength of schedule We refer the reader to Colley s paper for a full description we only summarize it here Let n 1 n be a set of teams which have played a collection of games in pairs such that each game has resulted in a winner and a loser i e no ties Define the matrix W R via W number of times team 7 has beaten team j In particular W W 0 if 2 has not played j and W 0 for all 7 Note that ij th entry of W WT encodes the number of times that i and j have played each other and letting e be the all ones vector the i th entries of W W e and W WT e give the total number of games played by i and its win loss spread respectively With J the identity matrix also define C 21 Diag W W e W W7 1 b e i W W e 2 where Diag places its vector argument into a diagonal matrix Colley shows that C is diagonally dominant and hence positive definite which implies in particular that C exists He then defines the ratings vector r to be the unique solution of the linear system Cr b or equivalently r C7 1b Then Colley sorts r in descending order i e determines a permutation 7 of n such that the vector r Tx is sorted in descending order Then the rankings vector is precisely 7 that is the ranking of team 7 is 7 If any of r s entries are equal one can easily adjus
17. d into 5 bowl games In particular the top 2 teams are matched head to head in a single game so that the winner of that game can reliably be called the national champion Even with the BCS now in place there is still considerable reliance on rank ings human and now also computer It is clear that quality rankings are necessary for the BCS to function properly i e to reliably setup the game that will determine the national champion and to setup other quality games Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports However it can be quite challenging to determine accurate rankings espe cially in college football Intuitively good sports rankings are easy to determine when one has data on many head to head matchups which allow the direct com parison of many pairs of teams This happens for example in Major League Base ball where many pairs of teams play each other often and thus a team s winning percentage is a good proxy for its ranking In college football the large number of teams and relatively short playing season makes such head to head information scarce For example for 120 FBS teams each playing 12 games against other FBS teams only 720 games are played out of 7140 5 possible pairings In reality even less information is ava
18. er week in practice Our approach can be extended in a number of ways First just as the Colley Matrix method can be applied to many sports beyond football so can ours This opens the way to robust basketball rankings chess rankings etc In addition our notion of robustness can be modified to the user s liking For example simple changes could be to alter the parameter w 0 1 the winning percentage defining the bottom teams or to include FCS games in the inconsequential set Z One could also manually choose a completely different inconsequential set Z the analysis and the methodology of the paper will go through unchanged For example one may wish to protect the rankings against games that were very close e g where the winner was determined by less than 3 points Z could then be constructed to contain just pairs of close games References 1 Bowl Championship Series Official Website http www bcsfootball org Accessed October 4 2011 2 FCS Grouping System http colleyrankings com iaagroups html Accessed October 4 2011 3 Peter wolfe s college football website http prwolfe bol ucla edu cfootball Accessed October 4 2011 4 Wilson performance ratings http homepages cae wisc edu dwilson Ac cessed October 4 2011 18 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12
19. ge however Since D T contains et ele ments where N is the total number of inconsequential games for most combina tions of N and we cannot simply list and solve over all linear constraints the number of such constraints is simply too large So instead we adopt the following strategy First we solve 6 over a limited subset of constraints to generate an ap proximate solution 7 t of 6 Then we solve the following subproblem over the variable A AT T 1 A AT 2 F max b CF F A AT Je H A AT el 3 Let A be an optimal solution If the optimal value of the subproblem is positive then we have determined a violated constraint of 6 and this constraint is added to the approximate model and the process is repeated On the other hand if the optimal value is nonnegative then we have proved that the current 7 t is optimal for 6 and hence 7 is the robust ratings vector Solving the subproblem for A is actually a difficult problem in theory but CPLEX is able to solve it quickly as long as N and I are not too large In all instances of this paper solving 4 via 6 and the procedure just outlined requires polynomial time LP However in this case there were many alternative optima r which introduced considerable ambiguity in the resultant rankings We thank the anonymous referees for suggesting and encouraging a switch to a different p norm 11 Brought to you by University of lowa Libraries Seria
20. ginia drops four spots which in our opinion seems excessive 1 Virginia Tech LSU 2 LSU Virginia Tech 3 Missouri Missouri 4 Ohio State Georgia 5 Georgia Ohio State 6 Oklahoma Oklahoma 7 West Virginia Florida 8 Florida Hawaii 9 Hawaii Kansas 10 Kansas Arizona St 11 Arizona St West Virginia 12 Boston College Boston College 13 Southern Cal Southern Cal 14 South Florida South Florida 15 Clemson Clemson 16 Brigham Young Brigham Young 17 Illinois Illinois 18 Tennessee Tennessee 19 Cincinnati Virginia 20 Virginia Cincinnati 21 Connecticut Auburn 22 Auburn Connecticut 23 Wisconsin Texas 24 Oregon Wisconsin 25 Texas Oregon Table 1 Comparison of the 2007 Colley Matrix rankings before left and af ter right the result of the inconsequential game between Marshall and Rice is switched Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports 4 The Robust Method The basic idea of our robust method is to calculate a ratings vector r that works well even if the win matrix is modestly perturbed from its real life value W Robust rankings will then be determined by sorting r in descending order just as with the regular Colley Matrix CM method We call our method the Colley Matrix Plus CM method Recall the CM system of equations Cr b for the wi
21. ilable because FBS teams often play non FBS teams and because conference teams mainly play teams in the same conference making it hard to compare across conferences Nevertheless there are many ranking systems for college football that per form well in practice One such method which is one of six computer rankings used by the BCS and which we will investigate in this paper is the Colley Matrix CM method 12 For a given schedule of games involving n teams this method sets up a system Cr b of n equations in n unknowns where the n x n matrix C depends only on the schedule of games and the n vector b depends only on the win loss outcomes of those games In particular b does not depend in any way on the points scored in the games The solution r is called the ratings vector and it can be shown mathematically to be the unique solution of Cr b To determine the rankings of the n teams the entries of r are sorted with more positive entries of r indicating better ranks The CM method shares similarities with other ranking systems see for example 17 18 14 In this paper we investigate whether computer ranking systems can be im proved and we focus in particular on improving the CM method We do not mean to presume or imply that the CM method needs improvement while the other five BCS computer rankings do not but the CM method is the only one of the six that is published fully in the open literature and hence can be systematically inve
22. ine D L fa ED X gt Ayl lt r i lt j to be those perturbations that switch no more than L inconsequential games For example D 0 0 and D 1 consists of all perturbations changing exactly 1 or 0 games Letting N D jez Wij Wji one can see that the number of perturbations in D L equals 54o For any A D define W W A and consider the CM rankings 7 based on W We investigate the differences between the rankings m and 7 of the teams in T via the measure 5 W W X mi 7i icT Alternatively 6 W W is the 1 norm of the sub vector indexed by T of the differ ence 7 7 We call 6 W W the switch measure For example if the top 2 teams switch places but no other ranks change then 6 W W 2 if the first and third teams switch places but no other ranks change then the switch measure is 4 and if the top team drops to fourth place but otherwise the orderings remain the same then 6 W W 6 If all teams in T remain in the top t of 7 then 6 W W is an even number but it can be odd if some team drops out of the top t For each football year y 2006 2011 and each L 1 2 we examine the distribution of switch measures H L y fav w mn i A E D L Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Jour
23. issouri Brigham Young Brigham Young North Carolina 24 North Carolina North Carolina North Carolina North Carolina North Carolina Brigham Young 25 Nebraska Nebraska Nebraska Nebraska Nebraska Nebraska Or Rank 10 15 25 aeee 5 T Figure 2 Colley Matrix Plus rankings with 0 lt I lt 10 for the 2008 football season Note that I 0 yields the regular CM rankings though these do not nec essarily match the rankings on Colley s website 12 due to the different handling of non FBS data as described at the end of Section 2 13 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports the ranks appear to stabilize for larger I For Figure 2 in particular all ranks are stable for 7 lt I lt 10 Finally the rankings confirm the robustness of the top 3 teams since they each retain their rank as I increases This could be interpreted as an affirmation of the CM rankings I 0 for these top teams in 2008 In a similar manner the top 15 CM rankings are confirmed to be mostly robust with the exception of Utah and Texas Tech In Figure 3 we show similar charts for the remaining years 2006 07 and 2009 2011 These depict very similar trends as 2008 5 2 Sensitivity of the robust rankings In Section 3 we inves
24. ls Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports less than a minute using CPLEX 12 2 16 within Matlab R2010b 19 on an Intel Core 2 Quad CPU running at 2 4 GHz with 4 GB RAM under the Linux operating system However larger values of N and I may lead to solve times that take a few minutes or even a few hours 5 Behavior of the Robust Method In this section we examine the behavior of our Colley Matrix Plus CM method in practice on the football data from 2006 2011 5 1 Variation as I increases Figure 2 presents the top 25 CM rankings for the 2008 football season for eleven choices of I IT 0 1 10 Note that l 0 yields the regular CM rankings though keep in mind that these do not necessarily match the rankings on Colley s website 12 due to our different handling of non FBS data as mentioned at the end of Section 2 The figure includes both a text table and a graphical chart Each line in the chart depicts the rank trend of a particular team For example Oklahoma is ranked 1 for all I and this corresponds to the top most flat line In contrast the rank line for Virginia Tech starts at 18 and ends at 16 When examining Figure 2 on its own it is difficult to make and support claims such as The rankings for 8 are better than the rankings for T 3 Of course we
25. mes against the FBS teams We are also motivated by a recent work of Chartier et al 9 that investigates the sensitivity of the Colley Matrix rankings and other types of rankings under perturbations to a hypothetical perfect season in which all teams play one another and the correct rankings are clear 1 e the top team wins all its games the second team beats all other teams except the top team etc In this specialized setting the authors conclude that the Colley Matrix rankings are stable but also present a real world example where the rankings are unstable We propose that the top rankings provided by computer systems should be more robust against the outcomes of inconsequential games that is games between teams that should clearly not be top ranked Of course the top rankings should still be sensitive to important games played between top contenders or even to games played between one top contender and one non contender To this end we develop a modification of the CM method that protects against modest hypothetical changes in the win loss outcomes of actual incon sequential games We do not handle the case of omitted games as exemplified in the quote above since in principle accidental omission can be prevented by more careful data handling Rather our goal is to devise a ranking system whose top rankings are stable even if a far away inconsequential game happens to have a different outcome This is our choice of
26. n matrix W For a user specified integer T gt 0 we consider perturbations W W A for A D T as introduced in the preceding section and we analyze the perturbed system C r b for W where C and b are given by 1 2 except that W takes the place of W Note that T plays essentially the same role as L in Section 3 but we will actually use the two parameters and L in slightly different ways for our experiments in Section 5 To facilitate the discussion therein we introduce and use T in this section Using properties of A it holds that A AT 0 which implies C 21 Diag W W Je W W 21 Diag W A W A e W A W A 7 C Diag A A7 e A A C i e the perturbation A does not alter the matrix C This makes sense because C depends only on the schedule of games which is not changed by A On the other hand it holds that b b h A A e 3 and so b changes linearly with A In total we are faced with perturbed systems Cr b where A ranges over D T Because C is invertible there is clearly no single r that solves Cr b for all A D T except in the special case when equals 0 A standard idea from robust optimization and the study of robust systems of equations is to search for an r that minimizes the worst possible violation of Cr b over all A D T i e to solve the optimization problem j Cr b 4 rain ta oe 4 where p
27. nal of Quantitative Analysis in Sports where t 25 and w 0 3 This involves enumerating all A D L and cal culating 6 W W for each Computationally calculating 6 W W is quick and enumeration of each A D L is reasonable for L lt 2 It turns out that with L fixed the distributions H L y of the switch mea sure behave similarly irrespective of the year y and so to save space we merge H L 2006 H L 2011 into a single histogram for each L 1 2 The result ing two histograms are shown in Figure 1 with basic summary statistics One Game Switched L 1 50 40 mean 5 1 a median 4 30 t min 0 7 max 20 i 20 stdev 4 9 10 m 0 i 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Switch Measure Two Games Switched L 2 50 t 4 40 t mean 6 5 a median 4 30 min 0 g max 28 iL 20 stdev 6 3 10 0 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Switch Measure Figure 1 Histograms for the switch measure 6 W W for L 1 2 switched games each over the years 2006 2011 This illustrates the sensitivity of the Colley Matrix rankings of the top teams to modest changes in the win loss outcomes of inconsequential games One can see from Figure 1 that the CM rankings of the top t 25 teams Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1
28. o be the set of all pairs playing inconsequential games Note that to remove redun dancy i j Z implies i lt j by definition We wish to examine the sensitivity of the CM rankings of teams in T to modest changes in the win loss outcomes of games between pairs 7 7 T For 6 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings this we define perturbations W W A of the win matrix W that switch the outcomes of a few inconsequential games Formally define Agee 0 V ij ZT Die Ie 7 T Nea 0 V 4 7 ET Wi lt Aij lt Wji y i j ET The condition A A 0 for all i j Z guarantees that only inconsequential games are switched and the equations A Aj 0 for all i j Z ensure that any switch is mathematically consistent between i j and j i For example if we wish to switch a game having W 0 and W 1 then we need to perturb W by 1 and W by 1 Finally the inequalities W lt A lt Wj limit the number of switched games for i j Z For example in case W 1 and W 2 it is clear that we logically need W 1 lt Ay lt 2 Wj We also define a convenient restriction of D Given A D the quantity J i lt Aij equals the number of games switched by A For any integer limit L gt 0 we def
29. stigated 6 We are especially interested in the robustness of the CM rankings and our research was in part motivated by a situation that arose at the end of the 2010 regular season i e immediately before the bowl match ups were to be determined The final BCS ratings show LSU ranked 10th and Boise State 11th But Wes Colley s final rankings as submitted to the BCS were incorrect The Appalachian State Western Illinois FCS playoff game was missing from his data set the net result of that omission in Colley s rankings is that LSU which he ranked ninth and his No 10 Boise State should Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Burer Robust Rankings be switched Alabama and Nebraska which he had 17th and 18th would also be swapped LSU and Boise State are so close in the overall BCS rankings 0063 that this one error switches the order Boise State should be 10th in the overall BCS rankings and LSU should be No 11 5 In other words the CM rankings and hence the BCS rankings proved quite sen sitive to the outcome or rather the omission of a single game Moreover this game was played between two FCS Football Championship Subdivision teams and FCS teams are generally considered to be much less competitive than the top ranked FBS teams and play relatively few ga
30. t the rankings to exhibit ties but this is unlikely to occur in practice In the following section we will provide a specific example of the CM rankings We now discuss the data used throughout the paper We downloaded foot ball data from the website 4 for the 2006 2011 regular seasons This website appears to be an archive of Wolfe s website 3 In particular no post season data is included For each regular season the data contains the outcomes of all col lege football games played in the United States but we limit our focus to just FBS teams For example consider the 2010 college football season which included 3 960 games played between 730 teams around the country Of the 730 teams 120 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports were FBS teams and of the 3 960 games 772 involved at least one FBS team We focus our attention on these 772 games since they contain all data directly related to FBS teams In the case of 2010 these 772 games yield n 195 because the FBS teams played 75 outside teams Throughout this paper ratings will be done for all n teams in a given season but only ratings and rankings for FBS teams will be discussed since our interest is in ranking these teams Specifically we will rate all n teams using the vector
31. the instances are all switches of L inconsequential games The line x y is plotted for reference Figure 5 shows clearly that the CM rankings are much less sensitive than the CM rankings on the same instances Specifically the fact that most bubbles are below the x y line illustrates that on any given instance the switch measure for CM is less than that of CM We also note that the CM rankings are particularly successful at lessening the sensitivity of the worst case switch measures for CM 17 Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports approximately 15 20 for L 1 and 20 25 for L 2 6 Conclusion In recent years the desire to develop robust analytical models has emerged in many fields including finance medicine and transportation and we believe that com puter sports rankings can also benefit from increased robustness This paper has introduced a particular concept of robustness for college football rankings via the Colley Matrix method Through experimentation we have shown that our concept of robustness is consistent and more robust to modest changes in the data In addi tion the time needed to compute the robust rankings typically less than a minute is not an obstacle since rankings would be recalculated about once p
32. tigated the sensitivity of the CM method when the rankings are recalculated after L games are manually switched We now conduct the same experiment except this time with our robust rankings Our goal is to verify that our rankings are indeed less sensitive than the CM rankings at least for certain values of In this section it is important to keep in mind the different roles played by L and I The parameter L determines the number of games that switch before recalculating the robust rankings whereas I is the user supplied parameter that controls the conservatism of the rankings In particular the two parameters are set independently We first describe two important properties of the robust rankings that typify extreme cases First when I 0 the robust rankings are clearly as sensitive as the CM rankings since they are exactly the CM rankings Second we claim that when T is sufficiently large the robust rankings are completely insensitive to L switches Said differently for very large I the robust rankings cannot change upon recalculation after any number of manual switches To see this let the win matrix W be given and suppose gt N where N 97 y lt 7 Wij Wj is the total number of inconsequential games Then the I robust rankings 7 based on W take into account the possibility that all inconsequential games might switch Next let W W A be any perturbed win matrix with A D L and calculate the robust
33. what it means for rankings to be robust While there certainly may be other valid definitions of robustness we believe our approach addresses a limitation of computer rankings and could also be easily mod ified for other definitions Our approach also depends on the definition of incon sequential but this can be adjusted easily to the preferences of the user too We also remark that since our approach considers only win loss outcomes it naturally incorporates other notions of robustness that strive to produce similar rankings even when a game s point margin of victory is hypothetically perturbed We stress our point of view that one should protect against modest changes to the inconsequential games As an entire collection the inconsequential games are probably of great consequence to the top ranked teams and so we do not propose Brought to you by University of lowa Libraries Serials Acquisitions University of lowa Libraries Serials Acquisitions Authenticated 172 16 1 226 Download Date 6 15 12 4 41 PM Submission to Journal of Quantitative Analysis in Sports say simply deleting the inconsequential games from consideration before calculat ing the rankings Rather our approach asks Suppose the outcomes of just a few of the inconsequential games switched but we do not necessarily know which ones Can we devise a ranking that is robust to these hypothesized switches Our approach is derived from the fields of robust
Download Pdf Manuals
Related Search