Our customizable consensus rankings, which combine multiple expert rankings into single cheat sheets, have been one of our most utilized features. Since so many of our members use this tool, we thought it would be a good idea to keep track of how accurate some of the more popular combinations have been.
If you’re new to the site, check out our Cheat Sheet Wizard and try a few custom combinations of experts to see what I’m talking about when I say “customizable consensus rankings” – admittedly a pretty confusing term.
So are two heads (or 40) better than one? Well, before I jump to the accuracy results, here’s a little background on how we combine rankings from multiple experts.
We purposely do not call them “Average Rankings” because that’s not how we determine each player’s rank. There are several issues with using averages that I won’t get into here, but suffice it to say that we explored that option thoroughly and found it to be a suboptimal method. Instead, we use a system of “Rank Points” similar to how the AP Top 25 is calculated. Each expert gets equal voting power and each ranked position is worth a different number of points. For example, for WRs, we give the top WR 50 points and go down one point per rank until the fiftieth WR, who gets 1 point. We add up all the points for each ranked player across all the experts to get to a consensus ranking for each player.
Our default list of approximately 40 fantasy experts are combined to produce what we call our “Expert Consensus Rankings” or “ECR” for short. This set of player rankings is what you get if you go to the Rankings Tab at the top of our site. Note that there may be less than 40 experts at the time you view the rankings because we add them during the week as they become available.
ECR is just one option though. You can actually consolidate any combination of the available experts to create your own custom version. If you don’t trust certain experts or want to avoid ones that haven’t achieved high scores in our accuracy study, you can easily exclude them from your custom cheat sheet. So far, it seems that the most popular combinations are those that package together the top experts in one form or another. Seems pretty logical to me!
Here are some consensus combinations that we’ve tracked for accuracy, and how well they have performed over the first 6 weeks of this season. The Rank # shown is the rank that each consensus option would have achieved vs. our individual experts. The % shown is the actual PAY™.
[table “51” not found /]Expert Consensus Ranking (ECR) – Our default rankings that incorporate all the experts into one consensus. ECR has preformed surprisingly well. It’s only been six weeks, but I think this early data points to the benefit of reducing risk from outliers or, in our world, “bold predictions.” The experts that go out on a limb frequently sometimes nail their predictions, but they also often miss completely. These incorrect predictions can have a dramatic impact on their relative accuracy scores since all the other experts are getting those predictions right. Consensus rankings in general help to weed out those outlier picks to create a relatively “safer” cheat sheet.
2009 Top 5 Overall Experts – Some of our members have been going with a consensus of last year’s top 5 experts. Perhaps they feel more comfortable using data from a full season (16 weeks) of accuracy assessments. This consensus ranking achieved the highest overall score out of the four versions we analyzed. It probably helped that Andy Behrens and Pat Fitzmaurice (our #1 and #2 guys from last year) have done really well this year too. One thing to note is that we only had 22 experts in our study last year compared to the almost 40 that we have this year.
2009 Top 5 Experts by Position – Another popular consensus combination is to take the top experts by position from last year and use their rankings for each position. This is likely a cut favored by those that believe “position specialists” exist. In 2010, we’ve definitely seen some of the experts do really well in particular positions. This version using the top 5 2009 experts for each position did well overall, but the position-specific ranks were not as high as I would have expected. Only time will tell if this is due to insufficient data, no “position expertise” correlation, or both.
2010 Rolling Top 5 Overall Experts – If you took the Overall Top 5 Experts YTD each week and used their consensus rankings for the following week, this is how accurate the rankings were. Some of our users are more inclined to trust recent data over 2009 data, so they’ve been actively combining the most accurate experts YTD each week. Over the course of the season, the leaderboard should become more and more stable, so it will be interesting to see how these consensus rankings perform in the future. I have to imagine that they’ll get better over time.
Bottom line is that there’s been a very small difference in performance between these options. More importantly, though, all of these options have performed really well – top 5 overall in each case. It’s still early in the year, but so far there seems to be a real benefit to creating consensus rankings from the most accurate experts.
We’ll try out some other combinations in the future to see if there’s a specific cocktail mix that goes down easy and provides a nice kick for accuracy. Is there a specific combination that you’ve been using that you like? Please share it with us…we may run a couple more versions through our assessments in the future.
Photo Credit: Link
