Frequently Asked Questions – Accuracy

Accuracy FAQ
In a nutshell, how do you measure an expert’s accuracy?
Do you rate Preseason (Draft) or In-Season (Weekly) accuracy?
What is Accuracy Gap?
How do you determine the projected points values for each expert?
How are unranked players treated for the experts?
Can you provide more details on your accuracy methodology?
Why did you switch away from the PAY % tracking?

Have a different question? Check out our Main FAQ Page.

In a nutshell, how do you measure an expert’s accuracy?
We rate the accuracy of each expert’s cheat sheets (Preseason and In-Season) by comparing their player predictions to the actual outcomes. Each prediction receives a score (Accuracy Gap) based on how close it is to the player’s actual output. We then add up the experts’ player scores at each position to given them an overall rating. The most accurate experts are those that provide cheat sheets that help you optimize your draft and weekly start/sit decisions. Read more about In-Season (weekly) methodology and Preaseason (Draft) methodology.

Do you rate Preseason (Draft) or In-Season (Weekly) accuracy?
Both. We have In-Season accuracy scores going back to the 2009 season. Our Preseason accuracy scores go back to 2010. However, we keep these accuracy scores separate from each other since many experts that do draft rankings do not offer weekly rankings. Also, we believe In-Season accuracy scores provide a more significant analysis of advice quality since there’s simply much more data to work with (16 cheat sheets vs. 1). We suggest looking at both sets of scores, but caution you against putting too much weight against Preseason accuracy.

What is Accuracy Gap?
Accuracy Gap represents the difference (absolute value) between an expert’s projected points for a player and the actual points scored for the player. For each expert, these values are added up across the players evaluated to generate an overall Accuracy Gap. The closer this value is to zero, the better it is for the expert because it indicates their predictions were closer to each player’s actual point production. Another way to think of Accuracy Gap is as the expert’s “Error” for their predictions. A perfect gap would be 0, indicating that there was no error between the expert’s predicted rank and the actual rank for a player.

How do you determine the projected point values for each expert?
We evaluate the experts by assigning a projected point value to each fantasy-relevant player in their rankings (learn more about our player pool). This value is based on the actual production of the rank # the expert slotted the player at. As an example, if an expert ranks Eric Decker at WR #28 in his/her draft rankings, we’d assign a projected point value for this prediction based on the production of the player that actually finished as WR #28 for the season. In 2015, that point value was 125.9 points. As noted above, the Accuracy Gap in this scenario is the difference between the expert’s prediction for Decker (125.9 points) and the fantasy points that Decker actually scored during the season (161.7 points). We believe this is a fair way to determine projected points because by ranking Decker at WR #28, the expert is effectively predicting that Decker will be the 28th best fantasy scorer among WRs. The point value associated with that rank slot is precisely what we use when evaluating the expert.

How are unranked players treated for the experts?
We aim to include all fantasy-relevant players in our analysis. If an expert fails to rank someone that is part of the player pool, we assign a rank that is 1 spot lower than the lowest ranking the player received across experts. For example, if an expert ranked 60 RBs in the 2015 preseason, yet failed to rank Charcandrick West, we cannot simply assume that they valued West as their RB #61. This would be problematic because an expert who took the time to create a deep set of rankings, and slotted West anywhere below RB #61 would be negatively affected by this assumption. We instead look at the lowest ranking West was given across experts (i.e. RB #110) and slot the expert’s rank 1 spot below it (RB #111). This ensures that we are not overstating the projection for any expert that failed to rank a player. It ensures that experts who create a deep set of rankings are not unfairly penalized.

Why did you switch away from the PAY % tracking?
The decision to change our methodology came with much consideration. PAY (Percentage Accuracy Yield) was accepted by the fantasy community and validated by numerous analysts. With that said, we decided to transition away from PAY for several reasons:

1) Our new methodology offers increased transparency. We’ve always been transparent as far as our methodology is concerned. Our new system lends itself very well to offering reports and analysis that show exactly why each expert rated the way they did. The experts have access to these reports and we plan to offer them on the site soon as well.

2) The new system is more flexible. Simply put, we’ll be able to do more with the new Accuracy Gap metric. This includes rating experts for additional scoring settings (PPR, Half PPR) as well as determining who the most accurate experts are for individual players. The additional insight should benefit fantasy owners and experts alike.

3) The PAY %s were often misinterpreted. The complexity of the PAY calculation made the output difficult to interpret. We frequently found that the %s were misconstrued as the % of time an expert made a correct pick. In actuality, the PAY values were more nuanced than that. We believe our new system represents an equally fair grading system and will be easier to interpret the results in meaningful ways.

The bottom line is our aim is to create an accuracy system that levels the playing field in evaluating experts, is fully transparent, offers results that are actionable for both fantasy experts and fantasy owners, and (ideally) does not require a PhD to interpret.

Can you provide more details on your accuracy methodology?
Absolutely. Please check out of our Accuracy Methodology for In-Season (weekly) rankings and Preaseason (Draft) rankings. While other sites consider their methodology to be a “secret sauce” of sorts, we prefer to be completely transparent. We believe that if we’re going to grade the experts, it’s only fair that we tell the experts how we’re coming up with the grades.

Back to the Top | Main FAQ Page