Opportunities for Measurement Science Research at NIST to Advance the Trustworthiness of AI Systems: A Workshop

Description

A National Academies workshop will consider opportunities for measurement science research at NIST that would advance the trustworthiness of AI systems. The workshop will engage experts and stakeholders from academia, the developers and users of AI in industry, and standards bodies. They will consider priority areas in AI, relevant dimensions of AI trustworthiness, potential approaches and frameworks, and research opportunities for NIST, other research performers and sponsors, and standards bodies.
Questions the workshop will explore include:

• Which AI techniques and applications are priorities for measurement science research today, and what are likely future opportunities?
  
• What attributes--including but not limited to security, robustness, accuracy, predictability, fairness, and explainability--characterize trustworthy AI systems? How do their definition, interpretation, and significance vary according to context and application?
  
• What is presently known about how to measure or evaluate these attributes and what research would advance our ability to measure them?
  
• How could these attributes and approaches to measuring them be applied in risk management frameworks for AI systems?
  
• Given large investments in AI by companies and other federal research sponsors, how can NIST best leverage its mission focus and capabilities in measurement science to advance our understanding of AI trustworthiness--and ultimately the trustworthiness of deployed systems?
  
• How could NIST measurement science work contribute to the advancement of AI capabilities by filling standards and evaluation gaps, such by developing standard test sets and challenge problems?
  

A workshop proceedings-in-brief summarizing workshop discussions will be prepared by an Academies rapporteur and published by National Academies Press following Academies peer review.