Facebook’s Civil Rights Audit Facebook reports that it has been engaging with external experts on AI fairness issues in a number of ways, including: • Facebook co-founded and is deeply involved in the Partnership on AI (PAI), a multistakeholder organization that seeks to develop and share AI best practices. Facebook states that it is active in PAI working groups around fair, transparent, and accountable AI and initiatives including developing documentation guidelines to enable greater transparency of AI systems, exploring the role of gathering sensitive user data to enable testing for algorithmic bias and discrimination, and engaging in dialogue with civil society groups about facial recognition technologies. • Facebook reports that in January 2020 that it sent a large delegation including engineers, product managers, researchers, and policy staff to the Fairness, Transparency, and Accountability Conference, the leading conference on fairness in machine learning, in order to connect with multidisciplinary academic researchers, civil society advocates, and industry peers and discuss challenges and best practices in the field. • Facebook is part of the expert group that helped formulate the Organization for Economic Cooperation & Development’s (OECD) AI principles which include a statement that “AI systems should be designed in a way that respects the rule of law, human rights, democratic values and diversity.” Facebook states that it is now working with the OECD Network of Experts on AI to help define what it means to implement these principles in practice. • Trust Transparency and Control (TTC) Labs is an industry collaborative created to promote design innovation that helps give users more control of their privacy. TTC Labs includes discussion of topics like algorithmic transparency, but Facebook states that it is exploring whether and how to expand these conversations to include topics of fairness and algorithmic bias. Through these external engagements, Facebook reports that it has begun exploring and debating a number of important topics relating to AI bias and fairness. For example, Facebook has worked with, and intends to continue to seek input from, experts to ensure that its approaches to algorithmic fairness and transparency are in line with industry best practices and guidance from the civil rights community. Even where laws are robust, and even among legal and technical experts, there is sometimes disagreement on what measures of algorithmic bias should be adopted—and approaches can sometimes conflict with one another. Experts are proposing ways to apply concepts like disparate treatment and disparate impact discrimination, fairness, and bias to evaluate machine learning models at scale, but consensus has not yet been reached on best practices that can be applied across all types of algorithms and machine-learning models. Similarly, Facebook has been considering questions about whether and how to collect or estimate sensitive data. Methods to measure and mitigate bias or discrimination issues in algorithms that expert researchers have developed generally require collecting or estimating data about people’s sensitive group membership. In this way, the imperative to test and address bias and discrimination in machine learning models along protected or sensitive group lines can trigger the need to have access to, or estimate, sensitive or demographic data in order to perform those measurements. Indeed, this raises privacy, ethical, and representational questions like: 79