Pages

RPI: The Power of Misinformation

3/18/2006

Now that the NCAA tournament has started, most of the backlash and criticism surrounding the brackets this year has been drowned out by the excitement of what is still the greatest thing going in sports. How can anyone complain when there's basketball on from 11:00 am to 11:00 pm? Still, the NCAA needs to take a hard look at some of the issues w/ the job of the Selection Committee, such as the lack of consistency in assessing teams and more importantly the role of committee member influence in the selection process.

The other thing the NCAA needs to look at is the quality of the data that it is providing to the Committee members to make these key decisions. It doesn't take a rocket scientist to figure out that if you give the Committee poor information, it's going to lead to poor decision making. You can really see this if you look at the last 3 years, which I did in a recent posting. To summarize:

2004: Old RPI (Bad Information) --> Bad Seedings
2005: Old RPI (Bad Information), New RPI (Worse Information) and Sagarin Ratings (Good Information) --> Good Seedings
2006: New RPI (Worse Information) --> Disastrous Seedings

I predicted this level of disaster would actually happen last year when the NCAA made the home/road adjustments for the first time. The crude adjustments make the ratings even more misleading and inaccurate than the old ratings, which is really saying something. However, Bowlsby & Co. were bailed out by relying on the Sagarin ratings, which is a very good body-of-work index. After using just the New RPI this year, Littlepage made Bowlsby look like a genious by comparison.

But Littlepage can't really be blamed for relying on the information that is put in front of him. He can't do anything other than take the numbers at face value and assume that they provide useful and accurate information on the thing they were designed to measure.

Of course, the Committee says every year that the RPI plays a very small role in the selection process, however it's clear that this year that the 'numbers' did play a large role in most of the decisions where the committee deviated from the general consensus. The issue isn't the fact that the Committee relied on the numbers. The issues is the numbers that the Committee relied on were misleading. If the numbers were better, the Committee would've made far better decisions.

To demonstrate what I'm talking about, take a look at the various examples below. The RPI numbers are shown in comparison with their JCI counterparts. The JCI information is a far more accurate assessment of what the RPI factors are trying to represent. In each case, ask yourself: 'Do you think the Selection Committee would've reached the same conclusion if they had the second line of information instead of the first?' If the answer is yes, then you know the numbers played a large role.

THE OVERSEEDS

Tennessee
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
2 RPI Based 21 - 7 6 6 7 10 6-4 3-2 1-2 6-3
5 JCI Based 21 - 7 16 36 31 105 -0.103 3-2 1-3 9-2

Note: My last 10 figure looks at how the team performed relative to the rankings over the last 10 games, so captures who you are playing (and where it's at). A +.1 or -.1 would mean that the team won one more/one less game than would be expected for a team with that ranking.

Comment: Get used to the wide differences in NC RPI and NC SOS -- you'll start to notice a lot of it.

UCLA
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
2 RPI Based 27 - 6 10 29 9 14 8-2 4-1 1-3 6-1
5 JCI Based 27 - 6 21 85 59 79 0.004 0-2 2-2 9-2

Comment: UCLA probably gets most of its lift simply by playing Arizona, the most overinflated team on the list, 3 times. UCLA gets 3 Top 25 wins when Arizona shouldn't even be considered Top 50. You can see how this can have a major impact on a team's resume. Even if the RPI is used simply as an organizational tool, as the NCAA hs suggested, it can create misleading results due to misclassifying teams.


Oklahoma
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
6 RPI Based 20 - 8 16 27 12 24 6-4 1-3 2-1 4-3
9 JCI Based 20 - 8 28 57 46 160 -0.042 1-4 3-0 3-3

Comment: Oklahoma is simply not a very good basketball team. UW-Milwaukee had to be licking their chops when they saw Oklahoma as a #6 seed. Their RPI numbers are inflated across the board.

Nevada
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
5 RPI Based 26 - 5 19 98 11 44 10-0 1-1 2-1 9-1
8 JCI Based 26 - 5 31 147 75 114 0.108 1-1 0-0 7-1


Wichita St
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
7 RPI Based 23 - 8 27 58 67 83 7-3 2-4 4-4 2-0
9 JCI Based 23 - 8 32 75 77 128 -0.063 0-2 6-6 3-0


Washington
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
5 RPI Based 24 - 6 35 92 14 158 8-2 4-1 1-0 2-2
8 JCI Based 24 - 6 34 109 18 301 0.037 3-0 0-0 6-3

Comment: Hard to rip on Washington's seed after they knocked off my Illini, but it's amazing just how far off that NC SOS can be.

UAB
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
9 RPI Based 23 - 6 32 115 75 150 8-2 1-2 0-0 6-4
11 JCI Based 23 - 6 39 108 86 181 0.037 1-2 0-0 5-4


UNC-Wilmington
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
9 RPI Based 25 - 7 28 103 94 141 9-1 0-1 3-2 5-2
11 JCI Based 25 - 7 44 124 118 165 0.138 0-0 3-3 1-2


Seton Hall
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
10 RPI Based 18 - 11 58 36 34 36 5-5 2-3 2-1 4-4
12 JCI Based 18 - 11 46 40 50 138 -0.034 4-3 1-1 3-5


George Mason
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
11 RPI Based 23 - 7 26 89 30 32 8-2 0-0 2-4 6-2
Out JCI Based 23 - 7 45 111 99 115 0.089 0-0 2-4 1-3

Comment: Having the George Mason AD on the Selection Committee probably did more for their cause then the numbers, but the numbers are inflated across the board. Look at the difference in NC SOS (a reoccuring theme)...32 vs. 115! Every year you have some controversial pick end up going deep in the tourney, making the Committee look like they knew what they were doing. This year it's George Mason. Kudos for knocking off Michigan State and North Carolina, but George Mason didn't earn the opportunity to play those teams.

Alabama
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
10 RPI Based 17 - 12 57 12 103 18 6-4 3-3 2-2 4-4
Out JCI Based 17 - 12 50 38 87 85 0.116 3-3 2-3 5-5


Arizona
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
8 RPI Based 19 - 12 23 7 8 1 6-4 1-6 1-1 6-2
Out JCI Based 19 - 12 51 50 80 18 -0.017 1-6 1-1 7-3

Comment: Congrats to Lute Olsen and the Arizona staff for beating the system. I guess this how you guarantee that you'll keep that 25-year tourney streak going.

California
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
7 RPI Based 20 - 10 52 61 126 153 6-4 2-4 1-1 4-2
Out JCI Based 20 - 10 54 78 89 238 -0.006 1-3 1-1 7-3

Comment: Cal barely fits the profile of a tournament-worthy team, much less a 7 seed.

Air Force
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
13 RPI Based 22 - 6 50 158 57 273 7-3 0-0 0-1 5-2
Out JCI Based 22 - 6 58 215 66 281 -0.060 0-0 0-1 3-2

Comment: Littlepage called Air Force a 'tough out', but does being a tough out mean losing to #161 Wyoming in the first round of the conference tournament (they also lost to Wyoming in the regular season)? Air Force simply was not a tournament-caliber team.

Utah State
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
12 RPI Based 22 - 8 46 102 16 75 7-3 1-2 0-0 6-3
Out JCI Based 22 - 8 73 177 78 315 -0.014 0-0 1-2 5-0

Comment: There's not much more I can say about Utah State, other than directing you to click here and here. This statement just about sums it up: Utah State was slightly more deserving of a tournament bid than Northwestern (13-15) and DePaul (12-15).

Average for 15 'Overseeds':
Average RPI Difference: -9.1
Average SOS Difference: -30.5
Average NC RPI Difference: -26.4
Average NC SOS Difference: -80.7


THE UNDERSEEDS

Illinois
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
4 RPI Based 25 - 6 14 54 15 192 6-4 5-3 4-2 4-0
2 JCI Based 25 - 6 7 26 3 154 -0.099 6-4 3-1 6-1

Comment: It sure would've been nice to see Illinois get a better draw than they did. It's clear that their deflated RPI and SOS played a role, because there's no other way to justify them as a '4' seed. I just hope they learn to schedule a little better in the future.

George Washington
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
8 RPI Based 26 - 2 37 207 107 323 9-1 0-0 3-0 2-2
4 JCI Based 26 - 2 9 159 45 282 -0.034 0-1 1-0 8-1

Comment: GW definitely got dinged for their weak schedule, especially in non-conference play. I'm not going to defend their schedule, but the numbers aren't as bad as the RPI would've led the Committee to believe. Even with the injury to Pops, GW deserved a better draw.

Pittsburgh
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
5 RPI Based 24 - 7 11 30 20 227 6-4 3-2 4-3 6-1
3 JCI Based 24 - 7 10 28 2 285 -0.072 4-4 5-2 7-0

Comment: The Anti-UCLA. Misclassification of teams hurt Pitt, where instead of having 4 Top 25 wins and 9 top 50 wins, the RPI shows them with 3 and 7, respectively.

Bucknell
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
9 RPI Based 25 - 4 42 181 29 37 9-1 1-3 1-0 1-0
7 JCI Based 25 - 4 22 153 111 21 -0.020 1-2 0-1 2-0


North Carolina St
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
10 RPI Based 21 - 9 51 64 74 206 5-5 1-5 2-0 8-3
8 JCI Based 21 - 9 25 43 25 133 -0.175 2-5 3-1 6-2

Comment: NC St's number are deflated across the board. I guess taking down Cal in the first round can't be considered too much of an upset. Maybe their seeds should've been flipped.

Arkansas
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
8 RPI Based 22 - 9 45 68 82 215 7-3 4-4 0-2 3-1
6 JCI Based 22 - 9 26 49 29 144 0.097 3-4 2-3 5-1


Texas A&M
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
12 RPI Based 21 - 8 44 67 48 240 8-2 1-5 0-0 6-2
8 JCI Based 21 - 8 38 74 22 323 0.213 1-3 0-2 7-2

Comment: If the Committee values performance over the last 10 games, then Texas A&M is one team that deserved even more credit than they received. A 8-2 record given their schedule is over 2 games better than expected for a typical 38th-ranked team.

Cincinnati
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
Out RPI Based 18 - 12 40 5 28 22 5-5 2-6 2-2 7-3
7 JCI Based 18 - 12 30 6 21 96 0.018 3-8 2-2 7-1

Comment: Cincinnati should've been a tournament team either way you look at it. Maybe they got dinged for their 5-5 finish, but my Last 10 measure shows that going 5-5 given their last 10 games is roughly consistent with expectations for a 30th-ranked team.

Michigan
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
Out RPI Based 18 - 10 47 37 36 147 3-7 3-6 0-2 6-1
10 JCI Based 18 - 10 33 33 9 254 -0.167 2-8 1-0 7-1

Comment: Michigan definitely got dinged for their lackluster finish, but you'd have to overweight their last 10 games to a unfair degree before they would look inferior to Air Force or Utah State. Is it body-of-work or not? Body-of-work says Michigan is in.

Florida St
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
Out RPI Based 19 - 9 63 94 97 316 6-4 1-4 1-0 6-4
9 JCI Based 19 - 9 37 53 15 278 0.036 1-5 1-0 6-3

Comment: If the Committee had the JCI numbers, there's no question that Florida St is in. If Utah State gets credit for coming close to beating #31 Nevada, shouldn't Florida St get credit for beating one-seeded Duke?

Missouri St
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
Out RPI Based 20 - 8 21 46 5 41 8-2 1-2 3-6 1-0
10 JCI Based 20 - 8 40 73 28 188 0.131 0-0 4-8 1-0

Comment: Missouri St is the scapegoat so the Committee can say they don't rely on the RPI. It's clear from Littlepage's comments that they had a number in mind for MVC bids: 4. Missouri St lost out when compared to the other MVC teams.

The Missouri St debacle might be an example of where the Committee had some sense that the RPI numbers were inaccurate in the case of the MVC teams, since it did get a lot of press, but they didn't have any sense on just how far they were off. This forced them down the alternate path of comparing the MVC teams head-to-head and setting the cut-off at four. While it's true that Missouri State's RPI numbers were inflated, their true numbers would've still been enough to get them in.

Hofstra
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
Out RPI Based 24 - 6 30 129 62 281 8-2 0-0 3-2 4-3
10 JCI Based 24 - 6 41 128 56 244 0.056 0-0 3-2 3-1


Maryland
Seed Team W-L RPI SOS NC RPI NC SOS Last 10 1-25 26-50 51-100
Out RPI Based 18 - 12 49 14 45 64 5-5 1-6 1-1 5-5
11 JCI Based 18 - 12 42 16 39 82 -0.011 1-8 1-1 5-3

Average for 13 'Underseeds':
Average RPI Difference: +10.3 (+12.8 excluding Missouri St)
Average SOS Difference: +15.2 (+12.8 excluding Missouri St)
Average NC RPI Difference: +18.7 (+22.2 excluding Missouri St)
Average NC SOS Difference: -13.3 (-2.2 excluding Missouri St)

0 comments:

 
Wegoblogger #31 © 2011 | Designed by Bingo Cash, in collaboration with Modern Warfare 3, VPS Hosting and Compare Web Hosting