Accuracy Methodology

 
Fantasy Baseball Accuracy | Fantasy Football Accuracy
 
Given the highly statistical nature of fantasy baseball, it’s no surprise that there have been several studies that have measured the accuracy of player projections. We’d like to augment this existing research by providing our own accuracy assessment. What follows is a general overview of what’s been published by others, some of the challenges of determining accuracy within a roto scoring format, and a summary of our unique methodology. If you want to skip the background, feel free to jump to our PAY™ Methodology for assessing the accuracy of fantasy baseball rankings.
 
Background: What’s been done by others?
 
Here’s a quick summary of existing studies that we were able to find online. We recommend taking a look to see what others have done.
 

  • FSTA organizes a baseball projections challenge each year.
  • TangoTiger simulates leagues based on forecasts from Pros and Joes in 2011.
  • TangoTiger reviews several projection systems from 2007 to 2010.
  • Razzball analyzes projections through various testing methods.
  • BaseballHQ discusses the challenges of assessing projection accuracy.
  • VegasWatch uses RMSE (Root Mean Squared Error) to rate accuracy through 2009.
  • RotoDaddy evaluates projection systems from 2004 to 2008.

 
While all the resources above provide valuable data on accuracy, our system is unique in two important ways:
 

  • We analyze the accuracy of player rankings, not projections.
  • Our methodology does not rely upon assigning player values.

 
Player rankings are by far the most predominant form of draft advice available, so it makes sense to us to rate this advice. Projections are absolutely valuable (and are often the starting point to building accurate rankings), but the average fantasy player cares most about relative player value. In other words, all other things being equal, which guy should I take? Rankings are valuable because they simplify the answer to this question.
 
For those of you that are familiar with our methodology for assessing fantasy football accuracy, you know that we base everything on the relative value of making correct predictions. When you pick Player A and he performs better than Player B, we know exactly how much benefit you received from making the right decision (the spread in actual fantasy points between the two players). This is a lot tougher to measure in fantasy baseball rotisserie leagues because roto points are based on how your team scores relative to other teams. This is why it’s rare to see a list of the top fantasy baseball players from the previous year (something fantasy football players can measure and rank easily).
 
A few sites do create roto values for players in an attempt to rank them at the end of the season, or to help them turn their projections into rankings. Here are some sources that we are aware of:
 

  • ESPN provides its Player Rater along with rankings from Elias Rating, Inside Edge, and Baseball Encyclopedia rankings.
  • FantasyInfoCentral has a methodology for creating 5×5 FIC scores for players.
  • Razzball has a Point Share system to estimate the impact a player has on an average team’s roto points.
  • MrCheatsheet provides WERTH values to estimate roto points gained over an average fantasy starter.
  • TheHardballTimes helps summarize a few valuation methods including SGP (Standings Gain Points), Replacement Level Theory, and Marginal SGPs.
  • LastPlayerPicked is one of many sites that produce player valuations and dollar values.

 
There’s an immediate question that comes to mind when seeing all these different valuation systems. Which system is right? In order for us to judge predicted results against actual results for fantasy baseball, we’d have to use a system that assigns player values (either someone else’s system or our own). This fundamentally creates bias in the scoring system since there’s no consensus opinion on a perfect player valuation formula. Our methodology avoids the need to rely on a subjective system because it scores results solely based on how the expert’s rankings performed relative to the other experts in the competition – using a traditional rotisserie format for scoring. The most accurate experts are those that can score the most points across the 10 categories found in standard 5×5 roto leagues.
 
Our Methodology: PAY™ for Fantasy Baseball
 
Our PAY™ (Prediction Accuracy Yield) grading system for fantasy baseball rankings naturally awards experts for being more accurate than their peers in two ways:
 

  • The players the expert ranked in their top xx at each position performed better than their peers. In other words, they included better players in their top rankings.
  • The relative rankings of the players within the top xx at each position were more accurate than their peers. In other words, they ranked players more accurately.

 
Our accuracy methodology focuses on the expert’s ability to provide player rankings that perform better in a rotisserie scoring format, relative to the other experts in our assessment. We do this by first understanding which players the experts would recommend drafting within each position (based on their positional player rankings), and then by having their recommended players compete against other experts in a 5×5 roto format. Experts gain points the same way you would in a rotisserie league – by performing better than their peers within each scoring category. PAY™ measures the % of total possible roto points that the expert scores in the competition. For example, if we have a field of 50 experts, there are a total of 500 points available to each expert (10 scoring categories multiplied by 50 total available points for each category leader). An expert that generates 300 points would have a PAY™ of 60%.
 
Here’s more detail on our three step process for measuring the accuracy of fantasy baseball rankings:
 
Step 1: Collect the correct rankings for each expert.
 

  • Use rankings that are meant for Standard 5×5 Roto Leagues.
  • Set a player’s position based on where the majority of experts rank the player (since there is no standard for position eligibility).
  • Scrub out players from secondary positional assignments (to avoid duplicating players across multiple positions).
  • Ensure the expert meets the minimum depth requirement at each position. For 2013 we analyzed each expert’s top:
    • 20 players each for 1B, 2B, 3B, SS, and C
    • 60 players for OF
    • 75 starting pitchers
    • 30 relief pitchers

 
Step 2: Determine who the expert would recommend for each possible draft decision within each position.
 

  • Run a round robin of ‘Who would you draft?’ decisions for each player in the expert’s positional rankings.
  • For each decision, the player with the higher rank is selected onto the expert’s team.
  • The net outcome is that each expert’s team consists of the players he prefers, weighted naturally toward who he prefers more (an expert’s #1 player will weigh more heavily than his #10 player).
  • Calculate each expert’s average player stats across the 5×5 scoring categories, weighted by the number of starters in a typical league (3 OF, 1 SS, etc.).

 
Step 3: Rank the experts at each 5×5 category and assign points based on their rank.
 

  • If the field consists of 50 experts, the top ranked expert in a category receives 50 points and the lowest ranked expert receives 1 point.
  • Points are added up within the hitting and pitching categories to find the top ranked hitters and pitchers experts. Hitting and pitching categories are added together to find the top Overall expert.
  • PAY™ is calculated for each expert by taking his total points and dividing by the total available points.
    • In this example, an expert could earn a maximum of 500 points (10 categories multiplied by 50 max points per category).
    • His PAY™ shows what percentage of the total he was able to achieve. A 100% PAY™ would mean he beat all the other experts in every roto category.

 
To test our methodology, we ran various post season rankings, such as ESPN’s Player Rater, through our assessment as if they were one of the experts in the competition. These rankings logically dominated the field. We also ran our ECR™ (Expert Consensus Ranking) through the assessment and it performed extremely well (consistent with what we’ve documented with our fantasy football accuracy studies, where ECR™ was #1 Overall). We’ll be sharing more specifics about how ECR™ performed in our fantasy baseball accuracy study in the near future.