The best way to measure the accuracy of different rating systems is to look at its R-Squared, or how well the inputs (the ratings along with home-court info) capture the volatility in the data (the wins and losses over the course of the season). Consider the crudest ranking system one could think of...just ranking teams by their win-loss records. We can use this as a baseline and compare other ranking systems by the amount of lift they provide over this baseline. Here are the results over the last few years:
A couple of observations:
1. The JCI provided 47% - 49% greater lift than the new RPI in 2003-04 and 2004-05 and is showing 70% lift so far in 2005-06.
2. The new RPI (with 40% boost for road wins and 40% penalty for home losses) is about 5% to 10% worse than the old RPI. The new RPI is showing greater lift so far this season, but expect that gain to deteriorate over the rest of the season. The new RPI inaccurately helps teams that win road games against very poor teams, and those types of games are more prevalent during the conference season.
3. The R-Squared goes down as the season progresses (tougher for one rating to explain 30 games instead of 20), so the R-Squared for 2005-06 will eventually go down closer to 2003-04 and 2004-05 levels.