At a Cybersecurity round table last week in Greenville, SC, there were discussions surrounding some recent high-profile data breaches relating to unpatched systems. The typical knee-jerk reaction to such stories is to slap your forehead and exclaim, “They didn’t patch? UNBELIEVABLE!” But one of the network engineer-types in attendance highlighted the problem with patching in a very succinct way. “Patches break things,” he said. “We know that—historically—automatic operating system patches break your system every so often, so if we turn those on automatically we are accepting that we will break our system at some point.“ This was a very good point and highlights the complexity and challenges with managing cyber risk. For a non-technical person, “just patch it” seems incredibly simple. The idea that a system wouldn’t be patched instantly given how many exploits rely on unpatched vulnerabilities seems unreasonable, if not like reckless disregard.

But things are rarely so simple. Let’s take the recent, highly-publicized Equifax hack, which has been reported to involve an Apache Struts 2 vulnerability. At a high level, Apache Struts is a framework that provides an architecture for implementing web applications. The particular vulnerability implicated was CVE-2017-5638, which was initially reported in March 2017. This vulnerability presents one of the most confounding situations for an organization because it had several features that make it particularly problematic. As soon as it was reported, there were exploits published and distributed widely, which could run executable code on web servers. This code could be run without any authentication from anywhere. Further, websites could be easily probed to see if they were vulnerable. It was scored as completely threatening to the confidentiality, integrity, and availability of data. It had low complexity, allowing even unsophisticated hackers to carry out an exploit. In other words, it was as critical of an exploit as it gets. But even more problematic was that the fix was not a matter of merely running a patch and forgetting about it. Although a patch was immediately made available for the Apache Struts framework, the framework is used for web applications. At a minimum, the web applications need to be recompiled and may even require changes or testing to insure they do not break. In other words, this type of patching can implicate development teams, not simply IT personnel. It is not as simple as “install the patch.”

So this situation presented a perfect storm of sorts. It was the highest threat that was instantly being exploited, coupled with some of the more difficult circumstances for patching. This is not a commentary on whether a particular company was reasonable in their approach, but rather a demonstration on the complexities of this fight the world is immersed in. The bad guys have it easy; in this instance they were able to take a widely-shared exploit and modify it for their own purpose. Companies have to take systems offline; work through change tracking procedures; and coordinate business, development, and IT teams all for this one “patch” for this one system. In retrospect, for at least one company this was the big one, but many of these could be the big one (there was another highest rated critical Apache Struts vulnerability disclosed in April 2016).

Unfortunately, those that undertook all these steps in time and managed to avert disaster for their own company were likely rewarded with not much more than another helping of critical vulnerabilities. Those of us with the technical understanding appreciate the difficulty of this effort. Perhaps this will help those that are less technical appreciate it as well by at least recognizing that not all patches are created equal.