Accuracy Study Results

by dave | Accuracy | Aug 19th 2010

We’ve had an overwhelming response to last week’s Accuracy Awards announcement. After being knee deep in data for the past several months, it was awesome to see that other people really care about this topic as much as we do. In light of this, we wanted to share some more of the detail and insights from the study.
 
 
Key Findings
 

  1. The most accurate expert was between 19% and 31% better than the least accurate expert, depending on the position. RB had the tightest spread while QB had the largest.
  2. There were drastically larger spreads when looking at individual weeks. On average, the difference between the best and worst expert was 58% for RB and 182% for TE.
  3. When looking purely at whether the expert’s predictions were right or wrong (i.e. with no weighting based on the value of the predictions), the spreads were considerably tighter.
  4. The data we’ve pulled together offers a ton of insight beyond just the summary accuracy scores that we’ve published so far.


 
More Thoughts
 
1. A lot of experts were bunched together with very little spread in their accuracy scores. This makes sense to us. Similar to sports betting, it’s really difficult for experts to distance themselves from the pack over the long haul. But also similar to sports betting, something as small as a 5% edge can make a huge difference. How many times have you lost by a small margin in your heads up match each week? Just a few points can mean the difference between a win and a loss! A few other takeaways:
 

  • Be wary of sites that claim their advice is 40% more accurate than other sites – especially if they’re making you pay for this advice. It’s especially concerning that some of these sites don’t publish their methodology. If you have a site that claims this and can back it up, we’d love to hear from you and enter you in our 2010 accuracy study.
  • Player list size and average fantasy points per position naturally influenced the size of the spreads. We assessed 40 RB and 50 WR spots compared to 20 QB and 15 TE spots. From a “fantasy points per prediction” perspective, QB was the highest and TE was the lowest. These two factors – fewer predictions and more points per prediction – contributed to the relatively larger accuracy differences for the QB position.

 
2. Similar to sports gambling again (disclaimer: we are not sports gambling site and do not promote this activity in any way, unless you’re with us in Vegas), these “fantasy cappers” can get hot and cold with their picks from week to week. The amount of variance is again related to the number of predictions that we analyze – the more predictions, the smaller the spread. This is evidence that:
 

  • It may not be a good idea to draw hard conclusions from studies that only examine one ranking per year. This is one reason why we chose to focus on weekly rankings (16 weeks of data) vs. draft rankings (1 list per year). In the future, we’re going to track both draft and weekly rankings and keep year-to-year records.
  • Despite the fact that the average edge may get smaller and smaller with the more data that we gather, I’m a strong believer that the best experts will rise up over time. Similar to poker, anyone can drag a monster pot or even win a tournament or two, but the guys that can show positive results over millions of hands are the true experts.

 
3. On a Win % basis, where we just calculate how often the expert was right, the spread between the best and worst expert was only 14% to 19% depending on position. We don’t use Win % as our final accuracy rating because it doesn’t incorporate the value of each prediction. When you pick a guy that scores 15 points more than the other guy, that prediction should be worth more than when you correctly predict something that only nets 1 fantasy point. Also, when looking at these numbers, please keep in mind that:
 

  • We’re not including every possible prediction. We only score the predictions that involve at least some disagreement between the experts. There’s no reason to score the Chris Johnson vs. Kevin Faulk match-up if every expert is picking Chris Johnson. It’s also not a match-up a typical advice seeker would actually seek. So, when you look at the Win %, think of it as the expert’s ability to make the right call on decisions that are actually contemplated by fantasy players. Including all predictions would naturally improve every expert’s Win %.

 
4. Running this analysis just got us more curious about other insights that our data can provide. We’ll do our best to share more of this cool information as we dig through it. Here are just a few of the questions we’re hoping to answer:
 

  • For each expert, are there certain players that they have pegged much better than other experts? Are there certain players that they just completely missed on?
  • Similarly, are there certain players that the expert tended to overvalue or undervalue relative to the other experts?
  • Are certain experts more likely to go against the consensus opinion than others? When they do back the underdog, are they correct more often than not?
  • Which experts tend to agree or disagree with each other the most?
  • When an expert misses badly on a player, does he tend to over correct the next week, or does he stick to his original opinion?
  • The list is endless…we’re just getting started!

 
Please note that these summary numbers refer to the RB, WR, QB, and TE positions only. DST and K were evaluated and the ratings are published, but given the wide differences in league settings for these two positions, I would take the results with a grain of salt. In fact, we did not include these two positions for our Overall category.


What's your take? Leave a comment

Fantasy Games
Get 5 free NFL entries
with your first deposit
Join a League for Free
Serious Fantasy Football
1Odell Beckham Jr. (NYG)WR
2David Johnson (ARI)RB
3C.J. Anderson (DEN)RB
4Antonio Brown (PIT)WR
5Julio Jones (ATL)WR
6A.J. Green (CIN)WR
7DeAngelo Williams (PIT)RB
8Lamar Miller (HOU)RB
9Ezekiel Elliott (DAL)RB
10DeAndre Hopkins (HOU)WR
 View all Flex Rankings 
11Alshon Jeffery (CHI)WR
12Allen Robinson (JAC)WR
13Adrian Peterson (MIN)RB
14Brandin Cooks (NO)WR
15Amari Cooper (OAK)WR
16Brandon Marshall (NYJ)WR
17Mike Evans (TB)WR
18Mark Ingram (NO)RB
19Spencer Ware (KC)RB
20Jordy Nelson (GB)WR
21Matt Forte (NYJ)RB
22Kelvin Benjamin (CAR)WR
23Todd Gurley (LA)RB
24Latavius Murray (OAK)RB
25Jordan Reed (WAS)TE
26LeSean McCoy (BUF)RB
27Doug Baldwin (SEA)WR
28Rashad Jennings (NYG)RB
29Jordan Matthews (PHI)WR
30Dez Bryant (DAL)WR
1Mike Trout (LAA)CF,DH
2Paul Goldschmidt (ARI)1B
3Nolan Arenado (COL)3B
4Josh Donaldson (TOR)3B,DH
5Jose Altuve (HOU)2B,DH
6Bryce Harper (WSH)CF,RF
7Manny Machado (BAL)3B,SS
8Mookie Betts (BOS)CF,RF
9Anthony Rizzo (CHC)1B
10Kris Bryant (CHC)1B,3B
 View All Rankings 
11Miguel Cabrera (DET)1B,DH
12Max Scherzer (WSH)SP
13Clayton Kershaw (LAD)SP
14Edwin Encarnacion (TOR)1B,DH
15Madison Bumgarner (SF)SP
16Chris Sale (CWS)SP
17Jake Arrieta (CHC)SP
18Carlos Correa (HOU)SS
19George Springer (HOU)CF,RF
20Carlos Gonzalez (COL)RF
21Joey Votto (CIN)1B
22Starling Marte (PIT)LF,CF
23Charlie Blackmon (COL)LF,CF
24Nelson Cruz (SEA)RF,DH
25Andrew McCutchen (PIT)CF
26Corey Kluber (CLE)SP
27Xander Bogaerts (BOS)SS
28J.D. Martinez (DET)RF
29David Price (BOS)SP
30David Ortiz (BOS)1B,DH
1Anthony Davis (NOR)F
2Stephen Curry (GSW)G
3James Harden (HOU)G
4Russell Westbrook (OKC)G
5Chris Paul (LAC)G
6LeBron James (CLE)F
7Kevin Durant (GSW)F
8DeMarcus Cousins (SAC)F
9Damian Lillard (POR)G
10John Wall (WAS)G
 View 2016 Rankings 
11Kawhi Leonard (SAS)G
12Klay Thompson (GSW)G
13Jimmy Butler (CHI)G
14Paul Millsap (ATL)F
15Marc Gasol (MEM)C
16Blake Griffin (LAC)F
17Carmelo Anthony (NYK)F
18LaMarcus Aldridge (SAS)F
19Serge Ibaka (ORL)F
20Rudy Gobert (UTH)C
21Eric Bledsoe (PHO)G
22Al Horford (BOS)C
23Kyle Lowry (TOR)G
24Paul George (IND)F
25Nikola Vucevic (ORL)F
26Draymond Green (GSW)F
27Kevin Love (CLE)F
28Pau Gasol (SAS)F
29Gordon Hayward (UTH)G
30DeAndre Jordan (LAC)C