Facebook’s Civil Rights Audit issues, the Auditors would like to see more than one training take place, whether through periodic refresher training, or training updates, or some other training format. Fifth, while Facebook did not make any specific commitments in the last report regarding its algorithmic system for delivering ads, it did agree to engage academics, researchers, civil rights and privacy advocates, and other experts to study the use of algorithms by social media platforms. Part of that commitment included studying the potential for bias in such systems. While concepts of discrimination and bias have long been applied to models, advancements in the complexity of algorithms or machine learning models, along with their increasingly widespread use, have led to new and unsettled questions about how best to identify and remedy potential bias in such complicated systems. Facebook reports that since the last report it has participated in several ongoing engagements, including: • Creating a series of “Design Jams” workshops through Facebook’s Trust Transparency and Control (TTC) Labs initiative, in which stakeholders from industry, civil society and academia focused on topics like algorithmic transparency and fairness both in the advertising context and more broadly. Facebook states that more such workshops are planned over the coming months. • Conducting roundtable discussions and consultations with stakeholders (e.g., The Center for Democracy and Technology, The Future of Privacy Forum) on ways of advancing both algorithmic fairness and privacy—many approaches to measuring fairness in algorithms require collecting or estimating additional sensitive data about people, such as their race, which can raise privacy and other concerns. Facebook reports that it is working to better understand expectations and recommendations in this area. Facebook also agreed to meet regularly with the Plaintiffs in the lawsuits and permit them to engage in testing of Facebook’s ad platform to ensure reforms promised under the settlements are implemented effectively. Both of these commitments are underway. While Facebook deserves credit for implementing these prior advertising commitments, it is important to note that these improvements have not fully resolved the civil rights community’s discrimination concerns. Most of the changes Facebook made in 2019 focused on the targeting of ads and the choices advertisers were making on the front end of the advertising process; civil rights advocates remain concerned about the back end of Facebook’s advertising process: ad delivery. In March 2019, the Department of Housing and Urban Development (HUD) filed charges against Facebook alleging not only that Facebook’s ad targeting tools allow for discrimination, but that Facebook also discriminated in delivering ads (choosing which of the users within an ad’s target audience should be shown a given ad) in violation of fair housing laws. That charge remains pending. Furthermore, in December 2019, Northeastern University and the non-profit Upturn released a new study of Facebook’s advertising system that was carried out after the 2019 targeting restrictions were put into place. The study suggested that Facebook’s Special Ad Audiences algorithms may lead to biased results despite the removal of protected class information. 74