Just weeks prior to the CDAO reaching full operating capability and hosting its first Department of Defense (DoD) Digital and AI Symposium in June, MIT newcomer Diane Staheli was thrust into a very visible and critical leadership role, the CDAO’s Chief of Responsible Artificial Intelligence (RAI). On the precipice of the Department’s adoption of its overarching guidance for instilling and operationalizing the DoD’s AI Ethical Principles―the RAI Strategy & Implementation Pathway (RAI S&I Pathway)―Staheli spearheaded this important document to publication. All the while, Staheli was planning out the implementation activities and RAI team expansion, connecting with the DoD RAI community, and fielding related media inquiries.
In her own words, Staheli has taken a moment to reflect on the months since the RAI S&I Pathway adoption and to share her thoughts on what the future of RAI might look like at the Department.
1. What sparked your interest in the RAI Chief role and work in AI Assurance?
I came to the CDAO from Massachusetts Institute of Technology (MIT) Lincoln Laboratory, where I led multiple research groups that served mission areas such as cyber security, intelligence, joint operations, and homeland security. In all of these groups, the application of data analytics, data science, and artificial intelligence (AI) to mission problems has been a common theme. The group I led most recently had a strong focus on research in the RAI and social good ecosystem, including robustness, resilience, explainability, and ethics.
My educational background is in software engineering and human factors, and I’ve always been interested in human interaction with technology; the rise of AI introduces new challenges in this area. To a certain extent, coming to the CDAO AI Assurance Directorate was a natural evolution of my prior work, and all facets of my background can be utilized in service of Government problems.
I was also familiar with the organization, having worked with the former Joint Artificial Intelligence Center (JAIC) at its inception. I worked with the JAIC’s leadership to conduct a study that helped define its initial set of mission initiatives, and I am excited to be part of the next phase of its evolution, with the formation of the CDAO.
2. Please tell us about the implementation approach put forward in the Pathway.
The signing of the RAI S&I Pathway was an extremely important step forward for us. This document formalizes our approach to RAI and specifically defines the tasks, as well as the roles and responsibilities for execution, which will enable us to make good progress. We felt it was important to be extremely specific in defining the actions that need to be taken to operationalize our principles and achieve meaningful results. The CDAO’s role is to synchronize and coordinate efforts related to RAI S&I Pathway implementation, and our team will play a large part in bringing together the DoD community and beyond in this area.
3. What major projects are underway as a result of the Pathway?
The RAI team is heavily emphasizing the establishment of tools, guidance, and best practices that can be used to better equip the Department. Our first step is to figure out the needs, landscape, and roadmap for the kinds of tools we need, and which audiences need them. In addition to technical tools for developers, we intend to provide tools to support RAI practitioners, program managers, and senior leaders. We are preparing to kick off an analysis effort to produce a framework to depict tool categories specific to RAI (such as bias mitigation or explainability), versus general MLOps (machine learning operations) tools or best practices (such as continuous integration/delivery). In doing so, we also hope to identify candidate tools for a series of pilot programs to spur tool submissions to the DoD, so that we can conduct experiments and assess their utility for our capability development efforts.
We are also looking to expand our educational efforts, to develop a curriculum for RAI that can be rolled out across the Department. The effort will help define the baseline for what we teach in AI courses for the various educational roles, provide awareness of topic areas and concerns, as well as allow for deeper dives into various topics for personnel interested in immersing themselves, or potentially in choosing a career field.
Since this is still a relatively new career field, we are also investing in activities that will nurture the ecosystem beyond the DoD. We are starting an academic consortium to encourage more basic research on our topics, expose students to DoD problems, jumpstart innovation, and hopefully build a strong base of RAI practitioners across the nation.
The RAI team is collaborating with partners on many other Pathway-related efforts already underway across the CDAO. For example, the ADVANA GameChanger capability (https://www.ai.mil/blog_10_19_21_gamechanger.html) is being leveraged to collect data in support of the AI Inventory task in RAI and lines of effort (LOEs) 1.2.4. We are also working closely with the Tradewind team (https://tradewindai.com/) on the acquisition-related LOEs, and there are many more.
4. What are your goals for the year with respect to Responsible AI implementation?
Over the next year, we will significantly expand and mature our offerings and guidance to the Department, culminating in a minimally viable product of our suite of RAI tools. We have recently formalized and galvanized our community of RAI practitioners, and we are making good progress towards being data-driven in our RAI assessments of programs.
One of the core elements of the Pathway process has been the RAI Working Council. With the Pathway’s adoption, we have formalized the governance construct, as well as formed the core team of leadership and subject matter experts that will be moving the DoD ahead. This group was instrumental in shaping our current direction and will be a key governance element for the Department to identify bright spots, pain points, and shape our direction for the future.
One of the LOEs specifies the definition of key performance metrics. In addition to tracking completion of Pathway tasks, we also want to know if our work is having an impact, as well as driving accountability across the Department. After a year, we will have collected many data points and lessons learned that we can analyze. I also plan to continue the current tempo of an annual responsible AI milestone; next year, we will report out to the community of our progress and increase our transparency.