The Enduring Debate: Data vs. Human Judgment in College Sports Selection

The quest to create a fair and unbiased selection process for college sports postseason tournaments has been a long and winding road, littered with discarded systems and ongoing debates. Bill Hancock, a figure deeply entrenched in this history, having overseen both the Bowl Championship Series (BCS) and the College Football Playoff (CFP), offers a unique perspective on the challenges and compromises inherent in balancing objective data with subjective human judgment.

Hancock readily admits the BCS, the precursor to the CFP, was “a brilliant mathematical system.” Designed to pit the top two teams in a national championship game, the BCS formula incorporated computer rankings, national polls, and strength of schedule calculations. While seemingly objective, the system was plagued by criticisms of being opaque, inflexible, and lacking the crucial “human element.” Hancock, reflecting on its shortcomings, emphasizes that he never wants to see college football revert to a purely data-driven selection mechanism.

The BCS’s operational simplicity, as Hancock describes it, highlights its stark contrast to the current CFP selection committee process. He received weekly emails containing computer rankings, forwarded them to a neutral entity for compilation with other factors, and the formula spat out the championship game participants. There were no extensive meetings, no nuanced discussions, no agonizing over the merits of one team versus another – a far cry from the intense deliberations and scrutiny that accompany the CFP selection process today.

The CFP, established ahead of the 2014 season, was intended to address the perceived deficiencies of the BCS. It introduced a selection committee comprised of individuals with diverse backgrounds in college football, tasked with evaluating teams holistically and making subjective judgments based on their performance, record, and other intangible factors. This human element, while intended to improve fairness, has also opened the door to controversy and accusations of bias.

The inherent tension between objective data and subjective human judgment is particularly evident as Selection Sunday approaches for March Madness. College basketball, in contrast to college football, relies heavily on a wealth of statistical metrics to inform the NCAA Tournament selection process. Selection committee members have access to a plethora of predictive and results-based ratings, offering a granular view of each team’s strengths and weaknesses.

These metrics are far more sophisticated and comprehensive than anything available during the BCS era or even in the early years of the CFP. The availability of this data, coupled with the larger sample size of games in college basketball, allows for a more statistically robust evaluation of teams. However, even with this abundance of data, human judgment still plays a significant role in the final selection and seeding decisions.

The debate over objectivity versus subjectivity in college sports selection is fueled by the inherent differences between football and basketball. Ken Pomeroy, the creator of kenpom.com and a pioneer in college basketball analytics, finds the disparity between the two sports puzzling. He questions why college football, with its extensive rating systems, doesn’t rely more heavily on objective criteria. He suggests that the perceived failures of the BCS might have left a lasting scar, hindering the adoption of more data-driven approaches.

Matt Morris, ESPN’s director of analytics, acknowledges that the CFP selection committee does have access to metrics like ESPN’s Football Power Index and Strength of Record. However, he argues that the smaller sample size in college football makes data analysis more challenging and prone to anomalies. Each game carries a significantly higher weight in football, making it more difficult to accurately assess a team’s true strength and potential.

Kevin Pauga, an athletic administrator at Michigan State and creator of the Kevin Pauga Index, echoes this sentiment. He points out that each college basketball game accounts for only 3% of a team’s resume, while in college football, that figure jumps to 8.5%. This difference significantly amplifies the impact of individual games and can lead to skewed perceptions of a team’s overall performance. The relative abundance of data in basketball allows for more sophisticated adjusted efficiency numbers, making it a more data-friendly sport.

Despite the challenges, Morris and his team at ESPN developed a “playoff predictor” for college football, but they found it significantly more difficult to model the human element of the selection committee. Predicting the committee’s decisions, particularly with a newly expanded format, proved to be a complex undertaking. The "nebulous eye test," as Morris calls it, seemed to play a more prominent role in football than in basketball. Narratives, such as the desire to avoid multiple teams from the same conference, can influence the selection process, even if the underlying data suggests a different outcome.

The 16-team NCAA men’s hockey tournament offers a contrasting example of a more metrics-based selection process. With a smaller pool of teams, the tournament relies on a computer system that considers record against common opponents, head-to-head record, and the Ratings Percentage Index (RPI) to determine at-large berths. However, the smaller sample size and the limited number of Division I hockey programs make this approach less directly applicable to the larger and more complex landscape of college football and basketball.

Ultimately, Hancock believes that no matter the system in place, some teams will inevitably feel unfairly left out. He advocates for a human committee, informed by data, that can exercise human judgment. He emphasizes the importance of consulting and using data but firmly believes that the current approach, which balances data with human discretion, is the best way forward for both college basketball and college football.

However, Hancock acknowledges the ever-changing nature of college sports and refrains from making definitive pronouncements about the future. The ongoing discussions about expanding the CFP and granting more automatic bids to the Big Ten and SEC suggest that the debate over selection criteria is far from settled. As college sports continues to evolve, the delicate balance between data and human judgment will undoubtedly remain a central and contentious issue.