As the use of AI in DoD systems continues to grow, it becomes increasingly important to establish sound methodologies for testing and evaluating AI. To address this need, the Chief Digital and Artificial Intelligence Office (CDAO) has developed a set of T&E frameworks aimed at building trust in AI-enabled systems within the DoD.
These frameworks cover various dimensions of AI evaluation, including Model T&E, Human Systems Integration (HSI) T&E, Systems Integration T&E, and Operational T&E. Each framework equips testers with important insights into AI system assessment.
Model T&E focuses on the evaluation of the AI model itself, including its underlying data and algorithms. This approach ensures that critical elements like data integrity, bias reduction, system resilience, and drift monitoring are adequately addressed.
HSI T&E prioritizes the interaction between humans and AI systems. It emphasizes the necessity of understanding user trust in AI technologies, managing cognitive load, promoting AI transparency, and considering other human-centric factors needed for effective human-AI collaboration.
The Systems Integration T&E framework addresses the challenges of assessing AI within the broader system-of-systems context, focusing on functionality, reliability, interoperability, compatibility, and security for DoD systems.
Operational T&E, meanwhile, guides the evaluation of AI systems’ performance in real-world scenarios, assessing their effectiveness, suitability, and survivability within the operational environments of the DoD.
These frameworks represent the first step in the CDAO's comprehensive initiative to build a robust community focused on AI Test and Evaluation. They lay the groundwork for more detailed guidance for AI T&E within the DoD. Upcoming efforts will broaden the scope and delve into the technical depth of AI T&E, offering extensive resources such as guidebooks and codebooks. Additionally, in-depth analyses on specialized subjects are planned, which will refine and expand the T&E frameworks.
The CDAO’s commitment to responsible and effective AI is embodied in the release of these T&E frameworks. Our goal is to empower T&E professionals with the knowledge needed to navigate AI evaluation processes to foster the deployment of AI solutions that are safe, reliable, and ethically sound across the DoD.
If you are interested in future collaborations as we work to build out a community to create guidance and best practices of AI T&E, please fill out this form!