Frequently Asked Questions – Accuracy


Accuracy FAQ
In a nutshell, how do you measure an expert’s accuracy?
Do you rate Preseason (Draft) or In-Season (Weekly) accuracy?
What is PAY™ (Prediction Accuracy Yield)?
These accuracy scores seem low, does that mean all the experts are bad?
Can you provide more details on your accuracy methodology?
How and why is your methodology different from others?

Have a different question? Check out our Main FAQ Page.

In a nutshell, how do you measure an expert’s accuracy?
We rate the accuracy of each expert’s cheat sheets (Preseason and In-Season) by comparing their player predictions to the actual outcomes. Experts are awarded for correct predictions based on the incremental fantasy points achieved by making the correct lineup decision. The most accurate experts are those that provide cheat sheets that help you optimize your draft and weekly start/sit decisions. Read more about our Accuracy Methodology.

Do you rate Preseason (Draft) or In-Season (Weekly) accuracy?
Both. We have In-Season accuracy scores going back to the 2009 season. Our Preseason accuracy scores go back to 2010. However, we keep these accuracy scores separate from each other since many experts that do draft rankings do not offer weekly rankings. Also, we believe In-Season accuracy scores provide a more significant analysis of advice quality since there’s simply much more data to work with (16 cheat sheets vs. 1). We suggest looking at both sets of scores, but caution you against putting too much weight against Preseason accuracy.

What is PAY™ (Prediction Accuracy Yield)?
PAY™ is our yardstick for measuring accuracy. It shows how much benefit the expert’s predictions provided on meaningful lineup decisions. Our system turns each expert’s cheat sheets into thousands of individual “who should I start/draft” recommendations. We then score whether the expert’s picks were correct and how beneficial the advice was, based on the net fantasy points obtained by choosing the right player. A 100% PAY™ means the expert selected the right player for every meaningful predicition. A “meaningful” prediction is one that is not unanimous among our experts. We do this so that we’re scoring advice that you realistically might seek out (not obvious advice like starting Chris Johnson over Donald Brown). This is why PAY™ percentages tend to be lower than what some people would expect.

These accuracy scores seem low, does that mean all the experts are bad?
No. While the PAY %’s may look only slightly better than a coin flip, please keep in mind that:

  • We scrub out all the really “easy” predictions. No one is looking for advice on whether they should start Arian Foster (HOU) vs. John Kuhn (GB). To make sure we’re not padding the numbers with these slam dunk decisions, we take out all the unanimous expert predictions. Their PAY represents how they’re doing on the tough decisions that many fantasy players struggle with.
  • PAY is not Win %. We weigh each prediction based on the value of the decision (in fantasy points). If an expert makes a correct call to start Hakeem Nicks over DeSean Jackson, he’ll earn the net points that you would have gained from listening to his advice. If Hakeem barely beats DeSean, the prediction is worth less than if he had blown him away. PAY measures how perfect an expert is at capturing all of the available points for these lineup decisions. A high PAY is tough to achieve because of those surprise performances where an underdog scores a lot of points.
  • Accurate football prognostication is hard. Just ask the billion dollar sports gambling industry if it’s easy to predict winners. In that industry, a 60% win rate is God-like. The experts we track spend a considerable amount of time researching, analyzing and keeping up with the latest news (a lot more time than I can spend!), so we believe they deserve a little more credit than they’re normally given.
  • A small edge is all you need. You’ll notice that the PAY spreads between experts is not large. In fact, we suspect these numbers will grow closer over time as we collect more data. Using Poker as an analogy, the top players in the world have a small statistical advantage over average players but that small advantage yields them great returns. How many times have you lost your matchups over just a couple fantasy points? One more correct decision can be the difference between making or not making the playoffs!


Can you provide more details on your accuracy methodology?
Absolutely. Please review an overview of our Accuracy Methodology. While other sites consider their methodology to be a “secret sauce” of sorts, we prefer to be completely transparent. We believe that if we’re going to grade the experts, it’s only fair that we tell the experts how we’re coming up with the grades.

How and why is your methodology different from others?
If you’ve seen other accuracy ratings, you may notice that our conclusions can be different. This is because we approach accuracy assessments in a way that we haven’t seen before. More specifically, here are some of the ways we do things differently:

  • We rate both Preseason Draft Rankings and In-season Weekly Rankings. However, we prefer weekly rankings because they provide more data points to analyze (more data means more accurate), and are impacted less by outside variables that are almost impossible to predict (injuries, trades, suspensions, etc.).
  • We analyze the expert’s ability to predict outcomes, not spreads in rank. A flaw in other accuracy assessments is that they consider each rank spot to be of equal value. If you predict a player will be the 5th best RB, and he ends up 10th, you’re given the same score as predicting that a player will come in 35th when he comes in 40th. The reality is, the latter prediction probably cost you very little since there’s usually not much performance difference between the 35th and 40th RB.
  • We calculate the impact (in fantasy points) that these predictions generate. We account for the magnitude of the outcome by awarding correct predictions with exactly the difference in fantasy points between the two players. If you correctly predict that DeSean Jackson will beat out Anquan Boldin but he only beats him by 1 fantasy point, your prediction should be worth exactly 1 fantasy point. In other words, the better the expert’s advice, the more credit he/she gets.
  • We assess predictions that actually matter to fantasy players. We only score the player predictions that are relevant. When all the experts agree, their predictions are not scored because we assume this represents advice that hardly anyone is seeking (e.g. Should I start Adrian Peterson or Kevin Faulk?). This approach ensures that experts are judged based upon their advice for decisions that fantasy players actually contemplate.
  • We weigh the positional results to get to an overall accuracy rating. In judging who gives the best overall advice, we don’t just add up or average the scores from each position. We believe fantasy players derive different levels of benefit from each of the positions. To set our weights, we first look at the average spread in points for match-ups in each position. This gives us what a good decision is ‘worth’ at each position. We then multiply that by the typical number of decisions a player needs to make at each position, based on # of starting slots by position in a standard league. The resulting numbers give us our weights.


Back to the Top | Main FAQ Page