Facebook’s Civil Rights Audit • During the 2019 India general elections, in order to assist human reviewers in identifying and removing political interference content, Facebook built a model to identify high risk content (for example, content that discussed civic or political issues). Facebook used the Fairness Flow tool to ensure that the model’s predictions as to whether content was civil/political were accurate across languages and regions in India. (This is important because systematically underestimating risk for content in a particular region or language, would result in fewer human review resources being allocated to that region or language than necessary.) 2. Piloting a fairness consultation process. Facebook has also begun to explore ways to connect the teams building Facebook’s AI tools and products to those on Facebook’s Responsible AI team with more expertise in fairness in machine learning, privacy, and civil rights. Beginning in December 2019, Facebook began piloting a fairness consultation process, by which product teams who have identified potential fairness, bias, or privacy-related concerns associated with a product they are developing can reach out to a core group of employees with more expertise in these areas for guidance, feedback, or a referral to other employees with additional subject matter expertise in areas such as law, policy, ethics, and machine learning. As part of this pilot effort, a set of issue-spotting questions was developed to help product teams and their crossfunctional partners identify potential issues with AI fairness or areas where bias could seep in, and flag them for additional input and discussion by the consultative group. Once those issues are discussed with the core group, product teams either proceed with development on their own or continue to engage with the core group or others on the Responsible AI team for additional support and guidance. This emerging fairness consultation process is currently only a limited pilot administered by a small group of employees, but is one way Facebook has begun to connect internal subject matter experts with product teams to help issue spot fairness concerns and subsequently direct them to further resources and support. (Part of the purpose of the pilot is to also identify those areas where teams need support but where internal guidance and expertise is lacking or underdeveloped so that the company can look to bring-in or build such expertise.) As a pilot, this is a new and voluntary process, rather than something that product teams are required to complete. But, Facebook asserts that its goal is to take lessons from these initial consultations and use them to inform the development of longer-term company processes and provide more robust guidance for product teams. In other words, part of the purpose of the pilot is to better understand the kinds of questions product teams have, and the kind of support that would be most effective in assisting teams to identify and resolve potential sources of bias or discrimination during the algorithm development process. 3. Participating in external engagement. Because AI and machine learning is an evolving field, questions are constantly being raised about how to ensure fairness, non-discrimination, transparency, and accountability in AI systems and tools. Facebook recognizes that it is essential to engage with multiple external stakeholders and the broader research communities on questions of responsible AI. 78