Is The NFL's Concussion Protocol Working?

Cleveland Browns cornerback Joe Haden suffered his first concussion of the NFL season in an October 11 loss to the Ravens. He returned three weeks later against the Arizona Cardinals and suffered a second concussion, which would eventually land him on the injured reserve list.

On October 25, Browns wide receiver Artrell Hawkins suffered a concussion against the St. Louis Rams. Three weeks later, he returned against the Pittsburgh Steelers, experienced another concussion, was taken to the hospital, and eventually put on injured reserve as well.

Bryan Stork, center for the New England Patriots, suffered a concussion during the preseason, but played in the season opener. After the game, he was placed on injured reserve with a concussion.

Ladarius Green of the San Diego Chargers suffered a concussion in practice before the team's regular season opener. However, he still played in the first two games of the season before experiencing another concussion. He missed one game before returning Week 4 against the Browns.

On November 19, D.J. Fluker, also of the San Diego Chargers, suffered a concussion in practice, his third reported concussion in three seasons, all during practice. However, Fluker was cleared to play for that weekend's game against the Chiefs, and suffered another reported concussion. Fluker was cleared to play again on December 10.

All of these players were cleared through the NFL's concussion protocol before their second concussions. Medical research suggests that suffering a second concussion while you're still healing from a previous one can result in worse and prolonged symptoms and additional brain damage; in rare cases, it can result in a life-threatening condition known as second impact syndrome, which includes rapid and potentially fatal brain swelling. This begs the question of whether the above players truly were ready to return to play, and raises concerns about the efficacy of the NFL's concussion protocol.

The protocol requires daily examination and a slow ramping up of physical activity and neurocognitive abilities until the player is able to return to full practice. As the NFL's website states:

"A player feeling normal one day after the game might pass cognitive testing Tuesday and begin a light exercise program, intensify their exercise routine Wednesday, participate in non-contact aspects of practice on Thursday and return to full practice Friday. But if a player has a history of concussions or isn't progressing as quickly as planned, the process moves accordingly."

After the team doctor signs off, the player is sent to an "unaffiliated concussion expert physician" approved by both the NFL and NFL Players Association. (Dr. Stan Herring, one of the members of the NFL's Head, Neck and Spine Committee told NFL.com in 2013 that the unaffiliated experts and team doctors have "never had a disagreement." Earlier this season, the Washington Redskins' independent neurologist resigned as a result of bungling Robert Griffin's protocol.) This, in a nutshell, is how the protocol works. But a closer look at this year's injury and media reports begs the question: even as the number of diagnosed concussions rise, are players being allowed to return before it is safe to do so?

For the last several years, the NFL's self-reported concussion rates declined. League senior vice president of health and safety policy Jeff Miller explained this apparent trend by stating, "Players are changing the way they're tackling." Likewise, NFL Executive V.P. of Football Operations Troy Vincent told NBC San Diego, "I started the off-season visiting maybe, eight to 10 players, and we just talked about their of play, eliminating the impermissible use of the helmet, and we saw players adjust. We saw coaches teaching a different of technique, and we saw those numbers decline throughout the season."

But the league reversed course last week, when it announced 190 total concussions were diagnosed during the 2015 regular season, including practices, 32 percent more than the previous year. Dr. Richard Ellenbogen, co-chair of the NFL's Head, Neck and Spine Committee, told anyone who would listen that the concussion numbers skyrocketed because the NFL has "lowered the threshold" on diagnosing concussions. Although this isn't mutually exclusive from players tackling in a supposedly safer manner, it implies the NFL has a happy explanation for any movement on concussion statistics.

To get a better sense of whether the protocol for dealing with diagnosed concussions is working, I examined the official NFL injury list and various media reports to determine the number of days each player diagnosed with a concussion spent in the protocol, from the time of the injury to the first full practice participation and/or game. (Although a full practice is technically the fifth and final step of the protocol, allowing a player to take part in a full-contact practice is tantamount to subjecting them to the risks of a game.)

In 2015, some players appeared on the injury report multiple times, but the vast majority of players in the NFL never officially reported a concussion. By my count, 134 players appeared on the league's injury report with a concussion through 14 weeks of play. Eight players were eventually placed on injured reserve due to concussions. Through Week 14 of this NFL season, approximately 141 cases of concussed players appeared on the league's injury report.

(A few quick notes on methodology: I used Week 14 as a cutoff because players needed time to progress through the protocol for the analysis below. Also, I say "approximately" because injury reports are neither perfect nor completely transparent. For example, they do not distinguish between a full-contact practice and a non-contact full practice).

It's also important to state some caveats. First and foremost, Dr. Herring is right when he told NFL.com that every player is different, particularly with head injuries, and requires individual care. Without being in the room for such examinations, it's impossible to make any definitive statements about the protocol's efficacy.

Second, experts believe that concussions are under-reported, particularly in the NFL, but also in contact sports at all levels. A 2014 Harvard study found 26 out of 27 concussions in college football are not reported. In the NFL, players such as Malcolm Jenkins of the Eagles hide their concussions so they can still play, or otherwise do their best to mask symptoms so they can be cleared to play again. In addition, the maligned and problematic ImPACT concussion testing technology the NFL uses as part of its protocol leads to under-reporting of post-concussive symptoms, making it harder to accurately assess a player's recovery.

Further, relying on the NFL's injury data—primarily used for gambling purposes—is hardly the most scientific method for medical information.

As such, please regard the following as an analysis of reported concussion rates in the NFL as represented to the public, not actual concussion rates in the NFL. That being said: 48 percent of players diagnosed with a concussion through Week 14 cleared the protocol in five days or less, meaning they didn't miss a single game. It took the median case five days to clear the protocol, which seems to follow the exact five-day pattern the NFL.com article outlined, which—maybe coincidentially, and maybe convieniently for the purposes of ongoing professional football employment—allows a player who was concussed in a game to play the following week.

I asked Dr. Matthew Matava, the St. Louis Rams team doctor and former president of the NFL Physicians Society, about these findings. He wrote to me via email, "I would agree that about half the players go back to play within one week, but that can occur only if the player's symptoms resolve by Monday (or Tuesday at the latest) given the step-by-step nature of the return-to-play protocol. Practically speaking, if a player's symptoms don't resolve by then, there just isn't enough time left in the week for them to 1) progressively increase their exertion level in a safe fashion, 2) be cleared by the Independent Neurosurgical Consultant, and 3) get in enough practice time to be prepared to play that week from a football perspective."

Looking at team-by-team concussion reporting yields some interesting results. The Cardinals and Packers each placed only one player on the injury report for a concussion through the first 14 weeks of the season, while the Chargers and Browns has 12 and 10 cases of concussed players on the injury report, respectively. The median team reported four cases of concussed players during the first 14 weeks, or one case every 3.5 games.

What does the above mean? First and foremost, there is no evidence to suggest players on some teams suffer more concussions than others. We know players under-report head injuries, so any team with more reported concussions than others could be a sign of a healthier locker room culture where players feel comfortable telling their coaches that something isn't right. On the other hand, the Chargers and Browns had so many cases of reported concussions in part because, as outlined above, players on both teams got cleared to play and then suffered a concussion their first game back.

Most alarming are not the teams with so many reported head injuries, but the ones with so few. Seven teams (20 percent of the league) have reported two or fewer concussions through 14 weeks of football, or a concussion every two months. Perhaps it's just a sampling error; the Packers, for example, put two players through the protocol for Week 17 and had three players working through the protocol during training camp.

Indeed, these findings may have more to do with statistical variation in small sample sizes rather than any solid epidemiological finding. Matava said as much: "I would not read too much into this variation since we are only looking at one season in time," he wrote to me via email. "Any epidemiologic assessment of injury trends must be considered over several (at least more than one) seasons to negate any possibility that the numbers are skewed one way or another due to any number of variables."

Matava also defended the work done by NFL teams on the head trauma front. "The 32 medical staffs have been extremely diligent to diagnose concussions. If anything, I think we may be overly cautious in over-diagnosis rather than under-diagnosis. I would not conclude that the team(s) with one concussion have not diagnosed them. Rather, I would consider those team(s) fortunate that they only had one."

Of course, one concussion is enough to land in the NFL protocol, which brings us back to our initial question: is the system keeping players as safe as possible? On one hand, the league's handling of brain injuries has improved since an NFL concussion committee that ignored the American Academy of Neurology's warnings while declaring in published research that "return to play does not involve a significant risk of a second injury either in the same game or during the season." Any protocol is better than no protocol; sideline neurotrauma consultants are better than smelling salts.

On the other hand, cases like D.J. Fluker and Joe Haden suggest that players still may be returning from concussions too early. And that, in turn, reflects the biggest issue with the NFL protocol: because scientific understanding of concussions and other forms of brain trauma remains incomplete, nobody really knows how long a concussed player should sit out. Even in ideal circumstances—doctor, player and team all placing health above winning games—the correct amount of time is an educated guess. To wit: boxers who are knocked out generally aren't allowed to fight for 60-to-90 days; recent research by brain injury experts Chris Giza and Mike McCrea (another member of the Head, Neck and Spine Committee) has shown that brain function doesn't return to pre-injury levels even after symptoms subside, which means a week or less may not be enough time for full recovery. In the end, the most conclusive thing we can say is that the protocol is a work in progress, just like virtually everything else about concussion management. There is still so much we don't know—and just how much the NFL wants to know remains an open question.