Accuracy of Projections from Yahoo, ESPN, CBS & numberFire

by dave
Nov 27, 2010

Have you ever wondered how good the player projections are at the big league management sites? A few weeks ago I wrote a guest post at the NY Times Fifth Down about the accuracy of these “computer” projections.

Since then, we’ve received many requests to add them into our weekly accuracy rankings, so we went ahead and made the update over Thanksgiving break (it’s ok; we needed a break from the stuffing and the hyper nephews anyway).

We’ve also received a surprising number of requests to track a newcomer to the space called NumberFire.  It’s a slick startup that uses a unique algorithm to produce – according to them – significantly more accurate results than Yahoo and ESPN’s projections. Care to see how accurate they all are? Read on to find out!

Let’s start with NumberFire. Here’s what they say on their homepage:

“You can go by Yahoo! or ESPN’s projections, but they’re wildly inaccurate. There’s no science, no reasoning behind them – just some fat guy and his opinion…Our crew of stat geeks and math nerds have a unique, custom algorithm that matches current situations to historical games, gives them a rating based on the similarities, and makes a smart, hyper-accurate prediction based on the historical results.”

In Week 11, NumberFire was the #39 expert out of the 44 that we scored. In Week 10, they were slightly better at #34. We’re fans of the site, so we were definitely surprised by these results. Given how aggressively they claim to be more accurate than who they refer to as “the twin behemoths of inaccuracy,” it was equally surprising to see them rank lower than Yahoo and ESPN in both weeks. You can view the weekly rankings by going to the Week Tab of our Accuracy page.

We need to take these results with a huge grain of salt though. First, we’ve only been tracking them since Week 10. We’re a firm believer that you need a lot of data over a significant period of time before you can call your accuracy assessments accurate. We definitely don’t think you should draw too many conclusions from two weeks of data! Second, and probably more important, how we assess accuracy is very different from how NumberFire does it (from what we can tell from published information).

They compare their projected fantasy points for each player to those provided by Yahoo and ESPN. The site with projected points that are closer to the player’s actual points is deemed to be more accurate. Here’s NumberFire’s Week 11 Test Case (Note that I’m not sure if their accuracy claims are based on just the handful of players shown here or something more comprehensive). Here’s an example using two players from their spreadsheet:

They projected Chad Ochocinco to have 10.39 points while Yahoo predicted that he’d have 6.43. Chad actually scored 8.8 points, so NumberFire was deemed to be more accurate because their projection was closer to the actual result. The same logic was used for Brandon Lloyd. NumberFire’s projection of 10.07 was closer to Lloyd’s actual score of 13.8 than Yahoo’s projection of 9.69. This is all pretty straightforward and logical.

The problem is, if you have Ochocinco and Lloyd on your team, you probably care more about which guy to start. NumberFire’s projections would suggest that you start Ochocinco (they predicted that he would score more points), while Yahoo’s projection would suggest that you start Lloyd. From this perspective, NumberFire got it wrong and your team would have scored 5 less points for following their advice (Lloyd’s 13.8 points minus Ochocinco’s 8.8 points).

Our accuracy methodology is about predicting “relative” fantasy performance between players. We believe most of us use projections and rankings to make lineup and roster decisions, so it’s the accuracy of these predictions that matter the most. An expert gets credit for correctly advising you on which player to start, and the credit we award is directly related to the difference in fantasy points (since some start/sit decisions are more costly than others).  In the previous example, our system would say Yahoo made the right call, not NumberFire.

Moving on to the Big Three, here’s how each site’s projections rank in overall accuracy through 11 weeks of the season, out of the 41 experts that we have season long data for:

Yahoo! Sports Projections – #16
CBS Sports Projections – #18
ESPN Projections – #37

It’s probably a good idea at this point to clarify what we mean by “site projections.” Using Yahoo as an example, we track six different “experts” that they have. This includes four human experts: Behrens, Evans, Pianowski and Funston. The “Staff Composite” rankings that we track are the average player rankings of these human experts that get published on Yahoo’s rankings page. The “Site Projections” that we’re talking about in this post are the player projections that you see in the Yahoo league management system (for example, on the Matchup page that shows whether you or your opponent is projected to win).

So where do these site projections come from? A company called AccuScore actually powers the projections that you find on several big league management sites. Here’s what they say on their site:

“For Fantasy Players, AccuScore’s Fantasy Scout data is the #1 Service in the industry and we are the only company that is and has ever been simultaneously paid by, Yahoo! and CBS Sports for fantasy data…AccuScore uses past performance statistics to describe how players perform under different environmental, matchup, and game-situation conditions. Using projected starting lineups AccuScore simulates each game of the season one play at a time.”

Yahoo’s projections come directly from AccuScore. On CBS, users have the option to use projections from AccuScore, one of their resident fantasy experts (Jamey Eisenberg and Dave Richard), or an Average of all three projections. Since the Average is the default setting, this is the version that we use for “CBS Projections.” It’s not as clear how ESPN’s projections incorporate AccuScore, if at all. So, even though all three big sites may be using AccuScore in some way, their accuracy rankings are different because each site’s player projections are different. Had we used the “Accuscore” option on CBS, their scores would have been identical to Yahoo’s.

We hope you found this background information helpful. We’ll continue to track these sites as we work to gather a much longer history of the accuracy of their fantasy advice. Also note that these are not the only sites that use some sort of computer-based algorithm to project and rank players. We already track a few others, such as WhatIfSports.

Our goal is to discover and promote the best fantasy experts – human or computer – so contact us if you know of a site that provides great weekly rankings or projections. If there’s a critical mass of folks that want the same info, we’ll gladly pull it together.

Thanks and good luck this weekend!

What's your take? Leave a comment

Fantasy Games
FanDuel photo
Play for your share of $1m+
for FREE with first deposit
DRAFT photo
Play fantasy drafts for cash with a money back guarantee
CBS Sports photo
Serious Fantasy Football
Get our Mobile App!

Enter your phone number below, and we'll text you a link to download the app.

1Aaron Rodgers (GB)QB
2Tom Brady (NE)QB
3Matt Ryan (ATL)QB
4Derek Carr (OAK)QB
5Drew Brees (NO)QB
6Cam Newton (CAR)QB
7Matthew Stafford (DET)QB
8Ben Roethlisberger (PIT)QB
9Antonio Brown (PIT)WR
10Kirk Cousins (WAS)QB
 View all Flex Rankings 
11Le'Veon Bell (PIT)RB
12Russell Wilson (SEA)QB
13Philip Rivers (LAC)QB
14Jameis Winston (TB)QB
15Dak Prescott (DAL)QB
16Carson Wentz (PHI)QB
17Marcus Mariota (TEN)QB
18Jay Cutler (MIA)QB
19Julio Jones (ATL)WR
20Carson Palmer (ARI)QB
21Alex Smith (KC)QB
22Jay Ajayi (MIA)RB
23Trevor Siemian (DEN)QB
24A.J. Green (CIN)WR
25DeShone Kizer (CLE)QB
26Mike Evans (TB)WR
27Kareem Hunt (KC)RB
28Andy Dalton (CIN)QB
29Eli Manning (NYG)QB
30Ezekiel Elliott (DAL)RB
1Mike Trout (LAA)CF,DH
2Paul Goldschmidt (ARI)1B
3Jose Altuve (HOU)2B
4Nolan Arenado (COL)3B
5Charlie Blackmon (COL)CF
6Max Scherzer (WSH)SP
7Mookie Betts (BOS)RF
8Joey Votto (CIN)1B
9Chris Sale (BOS)SP
10Kris Bryant (CHC)1B,3B
 View All Rankings 
11Giancarlo Stanton (MIA)RF
12Anthony Rizzo (CHC)1B,2B
13Clayton Kershaw (LAD)SP
14Corey Kluber (CLE)SP
15Freddie Freeman (ATL)1B,3B
16Corey Seager (LAD)SS
17George Springer (HOU)CF,RF
18Manny Machado (BAL)3B,SS
19Daniel Murphy (WSH)1B,2B
20Nelson Cruz (SEA)RF,DH
21J.D. Martinez (ARI)RF
22Francisco Lindor (CLE)SS
23Jose Abreu (CWS)1B,DH
24Edwin Encarnacion (CLE)1B,DH
25Robinson Cano (SEA)2B
26Ryan Braun (MIL)LF
27Yu Darvish (LAD)SP
28Cody Bellinger (LAD)1B,LF
29Jose Ramirez (CLE)2B,3B
30Anthony Rendon (WSH)3B
1Russell Westbrook (OKC)PG,G
2James Harden (HOU)PG,SG
3Stephen Curry (GSW)PG,SG
4Kevin Durant (GSW)SF,PF
5Karl-Anthony Towns (MIN)C
6Anthony Davis (NOR)PF,C
7LeBron James (CLE)SF,PF
8Chris Paul (HOU)PG,G
9Kawhi Leonard (SAS)SG,SF
10DeMarcus Cousins (NOR)PF,C
 View All Rankings 
11Giannis Antetokounmpo (MIL)SF,PF
12Paul George (OKC)SF,PF
13Hassan Whiteside (MIA)C,PF
14Damian Lillard (POR)PG,G
15John Wall (WAS)PG,G
16Kyle Lowry (TOR)PG,G
17Paul Millsap (DEN)PF,C
18Jimmy Butler (MIN)SG,SF
19Draymond Green (GSW)SF,PF
20Kemba Walker (CHA)PG,G
21Kyrie Irving (BOS)PG,SG
22Al Horford (BOS)PF,C
23Victor Oladipo (IND)PG,SG
24Isaiah Thomas (CLE)PG,G
25C.J. McCollum (POR)PG,SG
26LaMarcus Aldridge (SAS)PF,C
27Brook Lopez (LAL)C
28Klay Thompson (GSW)SG,SF
29Kristaps Porzingis (NYK)PF,C
30Carmelo Anthony (NYK)SF,PF