Validating terrorism risk assessment models – Lessons learned from 11 models

Introduction A defining risk of our time is the possibly growing capability of terrorist groups to fabricate and deliver weapons of mass destruction (WMDs). That risk is characterized by extreme possible consequences, including tens of thousands of fatalities and initiation of global conflict. Yet by some definitions of WMDs, we have not, as of this writing, observed even a single full-scenario event. There are three other interrelated aspects of that risk: (1) the essential terrorist-defender game aspect of the risk, where the terrorist may be intelligent and adaptive to defensive actions, and may make decisions based on poorly understood processes of radicalization and poorly understood foreign and domestic subcultures; (2) terrorist incentives to develop and launch WMD attacks may be changing due to “The Great Unraveling” of international processes (Cohen, 2014; Haas, 2014); (3) terrorist capabilities can include step function increases due to Internet information, random meetings of individuals, and random opportunities. These considerations combine to create an almost overwhelming risk management challenge and an almost overwhelming risk assessment challenge for risk analysts. We pose that latter challenge as: How, in this context, do analysts apply all available data and analysis tools to generate the most effective risk management advice? That challenge has many facets. In this chapter, we address one particular facet: How should we validate WMD terrorism risk assessment models (here abbreviated as TRAMs, always as applied to WMD terrorism risk) in the absence of observed events? We define WMD as not including IEDs and IEDs with effects multiplied through, e.g., attacks on infrastructure. We began to address the challenge in “Validation in the Absence of Observed Events,” recently published in Risk Analysis (Lathrop & Ezell, 2015). There we create a special definition of validation of models concerning not-yet-happened events. The basis of that definition rests on considering why we validate models. Typical validation involves some form of testing how well the model results correlate with observed events. Those tests are not based on a desire simply to correlate with observed events, but more fundamentally are based on a desire to test how well the model can advise decision makers in their decisions.