论文标题
使用置换测试评估公平性
Evaluating Fairness Using Permutation Tests
论文作者
论文摘要
机器学习模型对人们的生活至关重要，并以确定人们如何访问信息的基本方式影响社会。这些模型的重力赋予了对开发人员建模的责任，以确保他们以公平，公平的方式对待用户。在将模型部署到生产中之前，至关重要的是要检查其预测表现出偏见的程度。本文介绍了通过统计假设检验来检测机器学习模型所表现出的偏差。我们提出了一种置换测试方法，该方法进行了假设检验，即相对于任何给定指标，模型在两组之间都是公平的。越来越多的公平概念可以说明模型公平的不同方面。我们的目的是提供一个灵活的框架，使从业人员能够确定他们希望学习的任何指标的重要偏见。我们提供了一种正式的测试机制以及广泛的实验，以显示该方法在实践中的工作方式。
Machine learning models are central to people's lives and impact society in ways as fundamental as determining how people access information. The gravity of these models imparts a responsibility to model developers to ensure that they are treating users in a fair and equitable manner. Before deploying a model into production, it is crucial to examine the extent to which its predictions demonstrate biases. This paper deals with the detection of bias exhibited by a machine learning model through statistical hypothesis testing. We propose a permutation testing methodology that performs a hypothesis test that a model is fair across two groups with respect to any given metric. There are increasingly many notions of fairness that can speak to different aspects of model fairness. Our aim is to provide a flexible framework that empowers practitioners to identify significant biases in any metric they wish to study. We provide a formal testing mechanism as well as extensive experiments to show how this method works in practice.
