The NFL Draft fascinates us, and has become a major topic of study at Zone Reads, in large part because of the gap of knowledge between what we know and what we want to know; that is to say, the NFL Draft is far from an exact science, and while some of the causes for this are random (bad luck happens, injuries are unpredictable, growth projections made from a solid process still don’t quite pan out), we think teams make a lot of mistakes, too, particularly in that they often overrate measurables over football skills, as though they were drafting for “best physical specimen” rather than “best football player.”

All these factors matter, though, which is why I wanted to talk a bit about what we learned from last year’s film-heavy approach, and what we need to refine, not only in film-watching itself, but in our ability to incorporate other information as well.

1. Film, then measurables, then off-field

Our priority when evaluating the draft last year was to go by the film, and that stood us well for the most part. We weren’t perfect, but when trying to determine whether or not a football player is good enough to become a professional football player, how well they play football goes a long way, usually better than how well they work out in a specific, controlled environment that is not a football game.

Of course, one can be a good, skilled, technically sound football player who makes an impact in college, but is not athletic enough to make the same impact in the NFL. (Bjoern Werner is finding this out right now.) In this regard, athleticism is important, both in terms of the necessary baseline to compete on an NFL playing field, and to measure a player’s potential ceiling (players who are athletic enough to dominate on an NFL field will always go early even when their technique is bad, because such a thing is so rare). Still, though, film comes first, because functional athleticism shows up on film: There’s a difference between a raw, technically unsound athlete who makes plays, and one who runs around like a chicken with its head cut off. Many coaches believe you can “coach up” lack of technique and discipline, and in some cases you can, but we find it more accurate to take the “you are who you are” approach to players, and not project too much growth without reason.

Similarly, we’ll have higher grades on players who seem deficient in certain measurable areas but who play well on film; these players tend to have their ceilings capped due to their physical limitations, but they can often make a positive impact in a specific role if given the chance. We’ll move a guy up if we think he has a strong chance of being a positive contributor to a winning team, even if his ceiling is limited and he’ll never be an All-Pro. (Michael Campanaro is a good example; his measurables led the Wake Forest wide receiver to drop all the way to the seventh round, but we think he displays the kind of traits to make him a strong slot receiver in the Wes Welker mode of “get open short and catch everything.” Now, that’s a very specific skill set, and at 5’11”, 190, he’s never going to be the all-around terror you want when you draft a receiver early. However, the fact that we think he could be a quality starter in that role moved him up on our boards, because we think that’s a valuable thing.)

In many cases, the difference between our evaluations and where a player got drafted came down to off-field stuff. That sort of thing is tricky for us to evaluate: All we have is news reports that are publicly available to everyone. We don’t have a chance to meet with players, we don’t get to have our team doctors evaluate them, and we don’t get the information we need to determine whether off-field incidents or injuries were youthful indiscretions and/or flukes, or if they were part of a recurring pattern. So we do the best we can. We try to only let off-field factors affect our grades to the degree we can be confident in our knowledge of those factors, and that confidence is not common given our limited information.

Of course, even these rules have exceptions. We rated Martavis Bryant as a second-round pick last year, which seemed more of a measurement of potential and athleticism than the film would suggest. However, we thought that Bryant’s athleticism translated to the game itself; in addition, the stories we heard about Bryant off the field is that, although he was still raw, he had recently begun taking his preparation much more seriously. We believed these stories, so we gave more weight to the idea that he would be able to develop his talents. So, sometimes, you do indeed have to go with the athleticism and your opinion of a player’s work ethic, when it seems like the most accurate evaluation of the prospect’s potential to make it as an NFL player.

2. We’re still better at evaluating skill positions than the trenches.

Without doing a comprehensive study of our 2014 evaluations and where those prospects ended up, my initial impression is that we missed more than anywhere else on linemen– offensive line more than defensive, and interior linemen more than outside players.

Brandon Linder is probably the best example of this from last year; we rated the Miami (FL) guard as a fringe-draftable prospect, and yet the Jaguars not only drafted him in the third round, but he played well enough to be considered a part of their future plans. It’s possible we overemphasized his struggles in pass protection. It’s possible we underrated his athleticism and ability to adjust to the next level. It’s possible he was poorly coached at Miami and played much better with a better staff and better supporting teammates. (The fact that two of his Miami teammates, Seantrel Henderson and Allen Hurns, well outperformed their expectations suggests the coaching in Miami isn’t getting the most out of its players.)

We were mostly wrong on centers, too, having Bryan Stork lower than most, Weston Richburg much lower than where he was drafted, and Marcus Martin higher than his performance warranted. (We were right that Corey Linsley was a sleeper with a good chance to start, though.) I’m not sure where exactly our offensive line scouting is deficient, but it’s something we’ll continue to refine.

3. Consider the whole body of work.

We underrated some players because they had poor tape as a result of nagging injuries, when their tape from when they were healthy shows a much better player. On the other hand, trying to correct for this also led up to overrate some players because their highlights were impressive, but their consistency was lacking. Probably the best examples I can remember of this are Louis Nix and Adrian Hubbard, players whose 2013 film was stellar, but for whom injuries (Nix) and loss of playing time (Hubbard) made for much more questionable prospects than our grades reflected.

4. In film watching, consider the context.

Consider the competition.
This is one reason small-school prospects are always met with scrutiny: It’s a lot easier to look like the fastest guy on the field against a bunch of non-scholarship student-athletes than it is to look like the fastest guy on the field against a team full of four- and five-star recruits who have NFL futures of their own. The idea with the Combine is that it provides a baseline for athletic comparison, but again, what the Combine measures doesn’t always manifest itself in gameday performance. (Running a 40-yard dash or a 3-cone drill in workout gear is a different beast than getting off the line of scrimmage first and beating the tackle to the corner. Just ask Trent Murphy.)

I’ve decided to start paying more attention to whom exactly a prospect is going up against, and weighing performances against other highly rated prospects more heavily.

Consider the system.
It’s a lot easier to look like a good player if you are in a system that consistently sets you up to do so. If an offensive system makes decisions easy for its quarterback, the quarterback deserves to have additional scrutiny placed on his decision-making abilities. If a receiver puts up big numbers in a system that consistently helps him get open, we need to determine how much of that is the system vs. his own abilities. (Similarly, we may have concerns about said receiver’s ability to make tough catches in traffic, if he never goes into traffic to make a catch.)

Consider the teammates.
On the one hand, considering the Tennessee Titans may go into 2015 with him as the starter, our evaluation of Zach Mettenberger as a second-round talent feels pretty good. On the other hand, considering the success of Odell Beckham Jr., Jarvis Landry, Jeremy Hill, and even Alfred Blue this season, we might have paused to ask ourselves why LSU’s offense didn’t perform better despite all that talent, and we might have settled on Mettenberger as the reason. I don’t even think our evaluation was bad– Mettenberger does have a number of positive traits, and I feel our evaluation accurately reflected those– but I don’t think we sufficiently considered that context, and we would have had him lower on our boards if we had.

Trying to separate what a player does from the context he does it in is tricky, but it’s necessary.

All in all, what this post is really about is looking for flaws in our process and refining them.  We’re never going to be perfect, or even the best, but now that the NFL season is over and the “real” season for draftniks has begun, I’ve begun to also think about our approach in the lead-up to watching film on prospects. Hopefully we can get better at it and contribute to the growth of knowledge.