Accuracy Study Results

by dave
Aug 19, 2010

We’ve had an overwhelming response to last week’s Accuracy Awards announcement. After being knee deep in data for the past several months, it was awesome to see that other people really care about this topic as much as we do. In light of this, we wanted to share some more of the detail and insights from the study.
 
 
Key Findings
 

  1. The most accurate expert was between 19% and 31% better than the least accurate expert, depending on the position. RB had the tightest spread while QB had the largest.
  2. There were drastically larger spreads when looking at individual weeks. On average, the difference between the best and worst expert was 58% for RB and 182% for TE.
  3. When looking purely at whether the expert’s predictions were right or wrong (i.e. with no weighting based on the value of the predictions), the spreads were considerably tighter.
  4. The data we’ve pulled together offers a ton of insight beyond just the summary accuracy scores that we’ve published so far.


 
More Thoughts
 
1. A lot of experts were bunched together with very little spread in their accuracy scores. This makes sense to us. Similar to sports betting, it’s really difficult for experts to distance themselves from the pack over the long haul. But also similar to sports betting, something as small as a 5% edge can make a huge difference. How many times have you lost by a small margin in your heads up match each week? Just a few points can mean the difference between a win and a loss! A few other takeaways:
 

  • Be wary of sites that claim their advice is 40% more accurate than other sites – especially if they’re making you pay for this advice. It’s especially concerning that some of these sites don’t publish their methodology. If you have a site that claims this and can back it up, we’d love to hear from you and enter you in our 2010 accuracy study.
  • Player list size and average fantasy points per position naturally influenced the size of the spreads. We assessed 40 RB and 50 WR spots compared to 20 QB and 15 TE spots. From a “fantasy points per prediction” perspective, QB was the highest and TE was the lowest. These two factors – fewer predictions and more points per prediction – contributed to the relatively larger accuracy differences for the QB position.

 
2. Similar to sports gambling again (disclaimer: we are not sports gambling site and do not promote this activity in any way, unless you’re with us in Vegas), these “fantasy cappers” can get hot and cold with their picks from week to week. The amount of variance is again related to the number of predictions that we analyze – the more predictions, the smaller the spread. This is evidence that:
 

  • It may not be a good idea to draw hard conclusions from studies that only examine one ranking per year. This is one reason why we chose to focus on weekly rankings (16 weeks of data) vs. draft rankings (1 list per year). In the future, we’re going to track both draft and weekly rankings and keep year-to-year records.
  • Despite the fact that the average edge may get smaller and smaller with the more data that we gather, I’m a strong believer that the best experts will rise up over time. Similar to poker, anyone can drag a monster pot or even win a tournament or two, but the guys that can show positive results over millions of hands are the true experts.

 
3. On a Win % basis, where we just calculate how often the expert was right, the spread between the best and worst expert was only 14% to 19% depending on position. We don’t use Win % as our final accuracy rating because it doesn’t incorporate the value of each prediction. When you pick a guy that scores 15 points more than the other guy, that prediction should be worth more than when you correctly predict something that only nets 1 fantasy point. Also, when looking at these numbers, please keep in mind that:
 

  • We’re not including every possible prediction. We only score the predictions that involve at least some disagreement between the experts. There’s no reason to score the Chris Johnson vs. Kevin Faulk match-up if every expert is picking Chris Johnson. It’s also not a match-up a typical advice seeker would actually seek. So, when you look at the Win %, think of it as the expert’s ability to make the right call on decisions that are actually contemplated by fantasy players. Including all predictions would naturally improve every expert’s Win %.

 
4. Running this analysis just got us more curious about other insights that our data can provide. We’ll do our best to share more of this cool information as we dig through it. Here are just a few of the questions we’re hoping to answer:
 

  • For each expert, are there certain players that they have pegged much better than other experts? Are there certain players that they just completely missed on?
  • Similarly, are there certain players that the expert tended to overvalue or undervalue relative to the other experts?
  • Are certain experts more likely to go against the consensus opinion than others? When they do back the underdog, are they correct more often than not?
  • Which experts tend to agree or disagree with each other the most?
  • When an expert misses badly on a player, does he tend to over correct the next week, or does he stick to his original opinion?
  • The list is endless…we’re just getting started!

 
Please note that these summary numbers refer to the RB, WR, QB, and TE positions only. DST and K were evaluated and the ratings are published, but given the wide differences in league settings for these two positions, I would take the results with a grain of salt. In fact, we did not include these two positions for our Overall category.

What's your take? Leave a comment

Fantasy Games
DRAFT photo
Get a FREE $3 Best Ball entry with your 1st deposit
FanDuel photo
Play for your share of $1m+
for FREE with first deposit
CBS Sports photo
Serious Fantasy Football
Get our Mobile App!

Enter your phone number below, and we'll text you a link to download the app.

1Le'Veon Bell (PIT)RB
2Todd Gurley (LAR)RB
3David Johnson (ARI)RB
4Antonio Brown (PIT)WR
5Ezekiel Elliott (DAL)RB
6DeAndre Hopkins (HOU)WR
7Odell Beckham Jr. (NYG)WR
8Michael Thomas (NO)WR
9Kareem Hunt (KC)RB
10Leonard Fournette (JAC)RB
 View All Rankings 
11Alvin Kamara (NO)RB
12Julio Jones (ATL)WR
13Melvin Gordon (LAC)RB
14Mike Evans (TB)WR
15A.J. Green (CIN)WR
16Dalvin Cook (MIN)RB
17LeSean McCoy (BUF)RB
18Devonta Freeman (ATL)RB
19Keenan Allen (LAC)WR
20Davante Adams (GB)WR
21Mark Ingram (NO)RB
22Jordan Howard (CHI)RB
23Brandin Cooks (NE)WR
24Tyreek Hill (KC)WR
25Derrick Henry (TEN)RB
26Adam Thielen (MIN)WR
27Rob Gronkowski (NE)TE
28Doug Baldwin (SEA)WR
29Christian McCaffrey (CAR)RB
30Carlos Hyde (SF)RB
1Mike Trout (LAA)CF
2Jose Altuve (HOU)2B
3Nolan Arenado (COL)3B
4Paul Goldschmidt (ARI)1B
5Mookie Betts (BOS)RF
6Bryce Harper (WSH)RF
7Trea Turner (WSH)SS
8Charlie Blackmon (COL)CF
9Giancarlo Stanton (NYY)RF
10Clayton Kershaw (LAD)SP
 View All Rankings 
11Kris Bryant (CHC)3B,RF
12Max Scherzer (WSH)SP
13Carlos Correa (HOU)SS
14Joey Votto (CIN)1B
15Manny Machado (BAL)3B
16Corey Kluber (CLE)SP
17Chris Sale (BOS)SP
18Freddie Freeman (ATL)1B,3B
19Anthony Rizzo (CHC)1B,2B
20Aaron Judge (NYY)RF
21Francisco Lindor (CLE)SS
22J.D. Martinez (FA)RF
23George Springer (HOU)CF,RF
24Cody Bellinger (LAD)1B,LF
25Jose Ramirez (CLE)2B,3B
26Stephen Strasburg (WSH)SP
27Josh Donaldson (TOR)3B
28Corey Seager (LAD)SS
29Madison Bumgarner (SF)SP
30Gary Sanchez (NYY)C
1Kevin Durant (GSW)SF,PF
2Giannis Antetokounmpo (MIL)SF,PF
3James Harden (HOU)PG,SG
4Stephen Curry (GSW)PG,SG
5Russell Westbrook (OKC)PG
6Karl-Anthony Towns (MIN)C
7Anthony Davis (NOR)PF,C
8Kawhi Leonard (SAS)SG,SF
9LeBron James (CLE)SF,PF
10Nikola Jokic (DEN)PF,C
 View All Rankings 
11John Wall (WAS)PG
12Chris Paul (HOU)PG
13Damian Lillard (POR)PG
14Jimmy Butler (MIN)SG,SF
15Rudy Gobert (UTH)C
16Kyrie Irving (BOS)PG,SG
17Hassan Whiteside (MIA)C,PF
18Myles Turner (IND)PF,C
19Paul George (OKC)SG,SF
20Kyle Lowry (TOR)PG
21Draymond Green (GSW)SF,PF
22Kemba Walker (CHA)PG
23CJ McCollum (POR)PG,SG
24Bradley Beal (WAS)SG
25Klay Thompson (GSW)SG,SF
26Marc Gasol (MEM)C
27Kevin Love (CLE)PF,C
28Paul Millsap (DEN)PF,C
29Joel Embiid (PHI)PF,C
30Eric Bledsoe (MIL)PG,SG