Fantasy Football Accuracy | Fantasy Baseball Accuracy
We invested a significant amount of time to make sure we had an objective and accurate way of assessing fantasy expertise. Our innovative approach has been validated by Professor Jeff Ohlmann, Ph.D. – an expert in sports analytics with a background in mathematics.
Please note that this overview covers our process for scoring In-Season (Weekly) Rankings. Our accuracy methodology for Preseason (Draft) Rankings is similar, but uses each expert’s preseason cheat sheet.
Step 1: Collect the right data.
For us, this means weekly player rankings and projections. If an expert produced both projections and rankings, we did our best to determine which the expert considered to be their “final word” on player advice. For simplicity, we’ll use the word “rankings” in this document. Rankings embody the final output of the expert’s insights and opinions, so they provide a natural summary of predictions for us to analyze. Since most serious fantasy experts produce rankings each week, they also provide a consistent and frequent source of data to assess (much better than relying on just one draft ranking each year). To make sure we collect the right data, we:
- Verify that we’re grabbing “weekly” rankings and not “rest-of-season” rankings.
- Ensure that we have the expert’s final rankings before the week’s first game kicks off.
- Go through a rigorous QA process to ensure that our data matches the expert’s public rankings perfectly.
- Use weekly rankings from Week 1 to Week 16. We throw out Week 17 since most serious leagues end their seasons prior to that week.
- Ensure that the expert rankings meet our minimum requirements for the number of players ranked at each position. For 2012, these minimum requirements were: QB-20, RB-40, WR-50, TE-15, DST-15, and K-15.
Step 2: Calculate each player’s actual fantasy points.
We chose to use “Standard” league scoring since there were many more expert rankings for this setting, as opposed to “PPR” or “TD Only.” To determine our specific settings, we analyzed each of the expert’s assumed scoring (when available) and set our settings to what was most typical. Surprisingly, the settings were extremely close to each other. DST/DEF and K were the only exceptions to this, which is one of the reasons why we chose not to include them in our overall assessment. QB Points/TD was the only other area that had a noticeable difference (4-6 points per TD), but the majority of the sites used 4 points per passing TD. Here are the settings that we used for each of the positions in our overall accuracy rankings (RB, WR, QB, and TE):
- Rushing/Receiving TDs – 6 points
- Rushing/Receiving Yards – 1 point for every 10 yards
- Passing TDs – 4 points
- Passing Interceptions Thrown – Negative 2 points
- Fumbles Lost to Opponent – Negative 2 points
- Passing Yards – 1 point for every 25 yards
- Passing/Rushing/Receiving 2 Point Conversions – 2 points
- We used fractional scoring to three decimal places
Step 3: Determine the initial set of data to analyze.
Since each expert’s ranked list can contain different players, we need a way to ensure an apples-to-apples comparison. We do this by creating a consensus ranking of the players based on all our expert rankings. We assign ‘rank points’ for each slotted player within the minimum cutoff range for each position, and add these points together for each player across the experts. For example, in the RB category, we assess 40 rank spots so we award the #1 ranked RB 40 points and go down a point per rank until the #40 RB. This approach is better than using an ‘average rank’ because an average requires you to assign ranks for unranked players. In our eyes, that method skews results. We do not want to assume anything for the experts! Utilizing a rank point system is the best way to determine the initial set of players that we want to compare. This consensus list of players helps us to:
- Generate every possible heads up match-up between the consensus players. Basically a round-robin where each player goes up against all the other players in the consensus list.
- Use the expert’s relative ranking of the players in each match-up to determine his prediction. If an expert ranks Anquan Boldin at #9 and DeSean Jackson at #14, we assume that the expert is predicting that Anquan Boldin will score more fantasy points than DeSean Jackson that week.
- Pull in “substitute players” as necessary. Fantasy expertise is not just about making the right predictions between two players. It’s also about finding the right sleepers – guys that no one else ranks high but you do. Since these sleepers can easily fall off of a consensus list, we simply find out how many of the consensus players the expert ranked within our defined range of spots for the position. The gap in players is then filled with the expert’s substitute players – the guys he has ranked over the consensus players that he left off of his list.
Step 4: Scrub out the data that we do not want.
The thousands of match-ups generated in Step 3 are then refined to ensure that the assessment is meaningful for advice seekers. We scrub out player match-ups when the match-up has “full and unanimous voting” by the experts. If all the experts agree that Adrian Peterson is going to outscore Alex Green this week, we believe this match-up is meaningless because (a) it doesn’t impact the relative accuracy score of the experts, and more importantly (b) it’s not a match-up that fantasy players are likely considering. Again, we want to assess the expert’s fantasy advice on predictions that actually matter to fantasy advice seekers.
Step 5: Score the predictions.
Finally, the fun part – we take the final list of player match-ups and determine the % of times the expert made the right call. However, we don’t stop here because each correct prediction is not worth the same. A great call where a player destroys another player is worth much more than a correct prediction that only yields one net fantasy point. So, we take the net points in the match-up and award it to the expert when he makes a correct prediction. We then sum up these points and compare it to the total possible points available (i.e. if he predicted 100% of the match-ups correctly). The % of the possible points he achieved is what we call his PAY™ or Prediction Accuracy Yield. Simply, if you followed all his relevant advice, what % of the total available fantasy points would he generate for you? This is somewhat similar to looking at your team’s actual score vs. the “optimal score” had you made all the right start/sit decisions.
Step 6: Rank the experts.
For each position, we simply rank the experts from top to bottom based on their PAY™. For the Overall assessment, we:
- Only use the PAY™ from the QB, RB, WR, and TE positions. DST and K are excluded because (a) many experts do not produce rankings for these positions, (b) they represent the widest spectrum of fantasy scoring which can impact the results, and (c) many players believe that predicting performance for these two positions involve much more luck relative to the other positions.
- Weigh each position’s PAY™ based on the average opportunity (in fantasy points) that each position provides.
- We determine this by first getting the average net fantasy points per match-up for each position. This represents, on average, the gain in fantasy points that you get for making one correct start/sit decision for that position. Not surprisingly, a correct TE prediction was worth less than a correct QB or RB prediction.
- We then multiply this number per position against how many decisions you likely need to make for each position each week. This is admittedly more subjective. We believe the typical league (if there is such a thing) starts 1 QB, 2 RB, 2 WR, 1 Flex RB/WR, and 1 TE. So, in any given week, the typical fantasy player needs to make a decision for each of these starting spots on his roster.
- The number of decisions multiplied by the average net fantasy points per decision gives us the total opportunity by position. We use this number to then determine the % weight that each position should have.
- Finally, combining the Weighted PAY™ for each position together gives us the overall accuracy score for each expert.
We hope this detailed overview was helpful. Thanks for being interested enough to read through it!