Accuracy Study Results

by dave
Aug 19, 2010

We’ve had an overwhelming response to last week’s Accuracy Awards announcement. After being knee deep in data for the past several months, it was awesome to see that other people really care about this topic as much as we do. In light of this, we wanted to share some more of the detail and insights from the study.
 
 
Key Findings
 

  1. The most accurate expert was between 19% and 31% better than the least accurate expert, depending on the position. RB had the tightest spread while QB had the largest.
  2. There were drastically larger spreads when looking at individual weeks. On average, the difference between the best and worst expert was 58% for RB and 182% for TE.
  3. When looking purely at whether the expert’s predictions were right or wrong (i.e. with no weighting based on the value of the predictions), the spreads were considerably tighter.
  4. The data we’ve pulled together offers a ton of insight beyond just the summary accuracy scores that we’ve published so far.


 
More Thoughts
 
1. A lot of experts were bunched together with very little spread in their accuracy scores. This makes sense to us. Similar to sports betting, it’s really difficult for experts to distance themselves from the pack over the long haul. But also similar to sports betting, something as small as a 5% edge can make a huge difference. How many times have you lost by a small margin in your heads up match each week? Just a few points can mean the difference between a win and a loss! A few other takeaways:
 

  • Be wary of sites that claim their advice is 40% more accurate than other sites – especially if they’re making you pay for this advice. It’s especially concerning that some of these sites don’t publish their methodology. If you have a site that claims this and can back it up, we’d love to hear from you and enter you in our 2010 accuracy study.
  • Player list size and average fantasy points per position naturally influenced the size of the spreads. We assessed 40 RB and 50 WR spots compared to 20 QB and 15 TE spots. From a “fantasy points per prediction” perspective, QB was the highest and TE was the lowest. These two factors – fewer predictions and more points per prediction – contributed to the relatively larger accuracy differences for the QB position.

 
2. Similar to sports gambling again (disclaimer: we are not sports gambling site and do not promote this activity in any way, unless you’re with us in Vegas), these “fantasy cappers” can get hot and cold with their picks from week to week. The amount of variance is again related to the number of predictions that we analyze – the more predictions, the smaller the spread. This is evidence that:
 

  • It may not be a good idea to draw hard conclusions from studies that only examine one ranking per year. This is one reason why we chose to focus on weekly rankings (16 weeks of data) vs. draft rankings (1 list per year). In the future, we’re going to track both draft and weekly rankings and keep year-to-year records.
  • Despite the fact that the average edge may get smaller and smaller with the more data that we gather, I’m a strong believer that the best experts will rise up over time. Similar to poker, anyone can drag a monster pot or even win a tournament or two, but the guys that can show positive results over millions of hands are the true experts.

 
3. On a Win % basis, where we just calculate how often the expert was right, the spread between the best and worst expert was only 14% to 19% depending on position. We don’t use Win % as our final accuracy rating because it doesn’t incorporate the value of each prediction. When you pick a guy that scores 15 points more than the other guy, that prediction should be worth more than when you correctly predict something that only nets 1 fantasy point. Also, when looking at these numbers, please keep in mind that:
 

  • We’re not including every possible prediction. We only score the predictions that involve at least some disagreement between the experts. There’s no reason to score the Chris Johnson vs. Kevin Faulk match-up if every expert is picking Chris Johnson. It’s also not a match-up a typical advice seeker would actually seek. So, when you look at the Win %, think of it as the expert’s ability to make the right call on decisions that are actually contemplated by fantasy players. Including all predictions would naturally improve every expert’s Win %.

 
4. Running this analysis just got us more curious about other insights that our data can provide. We’ll do our best to share more of this cool information as we dig through it. Here are just a few of the questions we’re hoping to answer:
 

  • For each expert, are there certain players that they have pegged much better than other experts? Are there certain players that they just completely missed on?
  • Similarly, are there certain players that the expert tended to overvalue or undervalue relative to the other experts?
  • Are certain experts more likely to go against the consensus opinion than others? When they do back the underdog, are they correct more often than not?
  • Which experts tend to agree or disagree with each other the most?
  • When an expert misses badly on a player, does he tend to over correct the next week, or does he stick to his original opinion?
  • The list is endless…we’re just getting started!

 
Please note that these summary numbers refer to the RB, WR, QB, and TE positions only. DST and K were evaluated and the ratings are published, but given the wide differences in league settings for these two positions, I would take the results with a grain of salt. In fact, we did not include these two positions for our Overall category.


What's your take? Leave a comment

Fantasy Games
FanDuel photo
Win your first contest
Or we'll credit your entry fee
DRAFT photo
Get a free best ball entry with your 1st deposit!
CBS Sports photo
Serious Fantasy Football
1David Johnson (ARI)RB
2Le'Veon Bell (PIT)RB
3Antonio Brown (PIT)WR
4Odell Beckham Jr. (NYG)WR
5LeSean McCoy (BUF)RB
6Julio Jones (ATL)WR
7Melvin Gordon (LAC)RB
8A.J. Green (CIN)WR
9Mike Evans (TB)WR
10Devonta Freeman (ATL)RB
 View All Rankings 
11Jordy Nelson (GB)WR
12Jay Ajayi (MIA)RB
13Jordan Howard (CHI)RB
14DeMarco Murray (TEN)RB
15Michael Thomas (NO)WR
16Dez Bryant (DAL)WR
17T.Y. Hilton (IND)WR
18Todd Gurley (LAR)RB
19Leonard Fournette (JAC)RB
20Amari Cooper (OAK)WR
21Doug Baldwin (SEA)WR
22Ezekiel Elliott (DAL)RB
23Rob Gronkowski (NE)TE
24Lamar Miller (HOU)RB
25Isaiah Crowell (CLE)RB
26DeAndre Hopkins (HOU)WR
27Brandin Cooks (NE)WR
28Demaryius Thomas (DEN)WR
29Aaron Rodgers (GB)QB
30Allen Robinson (JAC)WR
1Mike Trout (LAA)CF,DH
2Paul Goldschmidt (ARI)1B
3Jose Altuve (HOU)2B
4Nolan Arenado (COL)3B
5Charlie Blackmon (COL)CF
6Max Scherzer (WSH)SP
7Chris Sale (BOS)SP
8Joey Votto (CIN)1B
9Giancarlo Stanton (MIA)RF
10Mookie Betts (BOS)RF
 View All Rankings 
11Kris Bryant (CHC)1B,3B
12Corey Kluber (CLE)SP
13Freddie Freeman (ATL)1B,3B
14Anthony Rizzo (CHC)1B,2B
15Corey Seager (LAD)SS
16Manny Machado (BAL)3B,SS
17George Springer (HOU)CF,RF
18Daniel Murphy (WSH)1B,2B
19Nelson Cruz (SEA)RF,DH
20Francisco Lindor (CLE)SS
21Aaron Judge (NYY)RF,DH
22Clayton Kershaw (LAD)SP
23Edwin Encarnacion (CLE)1B,DH
24J.D. Martinez (ARI)RF
25Robinson Cano (SEA)2B
26Yu Darvish (LAD)SP
27Cody Bellinger (LAD)1B,LF
28Jose Ramirez (CLE)2B,3B
29Miguel Cabrera (DET)1B,DH
30Josh Donaldson (TOR)3B,DH
1Russell Westbrook (OKC)PG
2James Harden (HOU)PG,SG
3Stephen Curry (GSW)PG,SG
4Kevin Durant (GSW)SF,PF
5Karl-Anthony Towns (MIN)C
6Anthony Davis (NOR)PF,C
7LeBron James (CLE)SF,PF
8Chris Paul (HOU)PG
9Kawhi Leonard (SAS)SG,SF
10DeMarcus Cousins (NOR)PF,C
 View All Rankings 
11Giannis Antetokounmpo (MIL)PG,SG
12Paul George (OKC)SF,PF
13Hassan Whiteside (MIA)C,PF
14Damian Lillard (POR)PG
15John Wall (WAS)PG
16Kyle Lowry (TOR)PG
17Paul Millsap (DEN)PF,C
18Jimmy Butler (MIN)SG,SF
19Draymond Green (GSW)SF,PF
20Kemba Walker (CHA)PG
21Kyrie Irving (CLE)PG,SG
22Al Horford (BOS)PF,C
23Victor Oladipo (IND)PG,SG
24Isaiah Thomas (BOS)PG
25C.J. McCollum (POR)PG,SG
26LaMarcus Aldridge (SAS)PF,C
27Brook Lopez (LAL)C
28Klay Thompson (GSW)SG,SF
29Kristaps Porzingis (NYK)PF,C
30Carmelo Anthony (NYK)SF,PF