Have you ever wondered how good the player projections are at the big league management sites? A few weeks ago I wrote a guest post at the NY Times Fifth Down about the accuracy of these “computer” projections.
Since then, we’ve received many requests to add them into our weekly accuracy rankings, so we went ahead and made the update over Thanksgiving break (it’s ok; we needed a break from the stuffing and the hyper nephews anyway).
We’ve also received a surprising number of requests to track a newcomer to the space called NumberFire. It’s a slick startup that uses a unique algorithm to produce – according to them – significantly more accurate results than Yahoo and ESPN’s projections. Care to see how accurate they all are? Read on to find out!
Let’s start with NumberFire. Here’s what they say on their homepage:
“You can go by Yahoo! or ESPN’s projections, but they’re wildly inaccurate. There’s no science, no reasoning behind them – just some fat guy and his opinion…Our crew of stat geeks and math nerds have a unique, custom algorithm that matches current situations to historical games, gives them a rating based on the similarities, and makes a smart, hyper-accurate prediction based on the historical results.”
In Week 11, NumberFire was the #39 expert out of the 44 that we scored. In Week 10, they were slightly better at #34. We’re fans of the site, so we were definitely surprised by these results. Given how aggressively they claim to be more accurate than who they refer to as “the twin behemoths of inaccuracy,” it was equally surprising to see them rank lower than Yahoo and ESPN in both weeks. You can view the weekly rankings by going to the Week Tab of our Accuracy page.
We need to take these results with a huge grain of salt though. First, we’ve only been tracking them since Week 10. We’re a firm believer that you need a lot of data over a significant period of time before you can call your accuracy assessments accurate. We definitely don’t think you should draw too many conclusions from two weeks of data! Second, and probably more important, how we assess accuracy is very different from how NumberFire does it (from what we can tell from published information).
They compare their projected fantasy points for each player to those provided by Yahoo and ESPN. The site with projected points that are closer to the player’s actual points is deemed to be more accurate. Here’s NumberFire’s Week 11 Test Case (Note that I’m not sure if their accuracy claims are based on just the handful of players shown here or something more comprehensive). Here’s an example using two players from their spreadsheet:
They projected Chad Ochocinco to have 10.39 points while Yahoo predicted that he’d have 6.43. Chad actually scored 8.8 points, so NumberFire was deemed to be more accurate because their projection was closer to the actual result. The same logic was used for Brandon Lloyd. NumberFire’s projection of 10.07 was closer to Lloyd’s actual score of 13.8 than Yahoo’s projection of 9.69. This is all pretty straightforward and logical.
The problem is, if you have Ochocinco and Lloyd on your team, you probably care more about which guy to start. NumberFire’s projections would suggest that you start Ochocinco (they predicted that he would score more points), while Yahoo’s projection would suggest that you start Lloyd. From this perspective, NumberFire got it wrong and your team would have scored 5 less points for following their advice (Lloyd’s 13.8 points minus Ochocinco’s 8.8 points).
Our accuracy methodology is about predicting “relative” fantasy performance between players. We believe most of us use projections and rankings to make lineup and roster decisions, so it’s the accuracy of these predictions that matter the most. An expert gets credit for correctly advising you on which player to start, and the credit we award is directly related to the difference in fantasy points (since some start/sit decisions are more costly than others). In the previous example, our system would say Yahoo made the right call, not NumberFire.
Moving on to the Big Three, here’s how each site’s projections rank in overall accuracy through 11 weeks of the season, out of the 41 experts that we have season long data for:
Yahoo! Sports Projections – #16
CBS Sports Projections – #18
ESPN Projections – #37
It’s probably a good idea at this point to clarify what we mean by “site projections.” Using Yahoo as an example, we track six different “experts” that they have. This includes four human experts: Behrens, Evans, Pianowski and Funston. The “Staff Composite” rankings that we track are the average player rankings of these human experts that get published on Yahoo’s rankings page. The “Site Projections” that we’re talking about in this post are the player projections that you see in the Yahoo league management system (for example, on the Matchup page that shows whether you or your opponent is projected to win).
So where do these site projections come from? A company called AccuScore actually powers the projections that you find on several big league management sites. Here’s what they say on their site:
“For Fantasy Players, AccuScore’s Fantasy Scout data is the #1 Service in the industry and we are the only company that is and has ever been simultaneously paid by ESPN.com, Yahoo! and CBS Sports for fantasy data…AccuScore uses past performance statistics to describe how players perform under different environmental, matchup, and game-situation conditions. Using projected starting lineups AccuScore simulates each game of the season one play at a time.”
Yahoo’s projections come directly from AccuScore. On CBS, users have the option to use projections from AccuScore, one of their resident fantasy experts (Jamey Eisenberg and Dave Richard), or an Average of all three projections. Since the Average is the default setting, this is the version that we use for “CBS Projections.” It’s not as clear how ESPN’s projections incorporate AccuScore, if at all. So, even though all three big sites may be using AccuScore in some way, their accuracy rankings are different because each site’s player projections are different. Had we used the “Accuscore” option on CBS, their scores would have been identical to Yahoo’s.
We hope you found this background information helpful. We’ll continue to track these sites as we work to gather a much longer history of the accuracy of their fantasy advice. Also note that these are not the only sites that use some sort of computer-based algorithm to project and rank players. We already track a few others, such as WhatIfSports.
Our goal is to discover and promote the best fantasy experts – human or computer – so contact us if you know of a site that provides great weekly rankings or projections. If there’s a critical mass of folks that want the same info, we’ll gladly pull it together.
Thanks and good luck this weekend!
