Predicting proficiency in cyber defense team exercises

Performance assessment of human teaming in complex, real-world contexts is a fundamental challenge for research and training communities alike. We highlight a unique partnership between the cybersecurity training and research communities with the common goal of capturing human team performance. Whether in the context of a training assessment or a research endeavor; both are two sides of the same coin. As cybersecurity training events and competitions have evolved, there is increasing interest in analyzing the performance of the defenders as well as identifying the best methods of training, assessing, and educating the cybersecurity workforce. This includes analysis of the types of training, pedagogy, and the relationship between education, training, and commercial Information Technology (IT) certifications. In this work, we present a high-level look along with some initial findings from the premiere cyber training event for the Army National Guard Bureau focused on enhancing the capabilities of cyber defense teams to mitigate threats to critical national infrastructure. Our first year data collection effort was conducted in the spring of 2015 and involved multiple threads of data collection. The main realization to come from this study is that improved systems and procedures should be put in place to effectively monitor and quantify team behavior for valid analyses. As the research and training partnership matures, we hope to significantly improve the data collection and assessment of human team performance in follow-on exercises.