Psychological Foundations of Explainability and Interpretability in Artificial Intelligence

This paper defines and differentiates between the concepts of explainability and interpretability for AI/ML systems. The author uses explainability to refer to the ability to describe the process that leads to an AI/ML algorithm’s output, and argues that it is of greater use to model developers and data scientists than interpretability. Interpretability refers to the ability to contextualize the model’s output based on its use case(s), value to the user, and other real-world factors, and is important to the users and regulators of AI/ML systems. The author argues that the recent proliferation of explainability technologies has resulted in comparatively little attention being paid to interpretability, which will be critical for emerging debates on how to regulate AI/ML systems.