Facebook’s Civil Rights Audit 1. Hate Speech Pilots The June 2019 Audit Report described two ongoing pilot studies that Facebook was conducting in an effort to help reduce errors in enforcement of its hate speech policies: (1) hate speech reviewer specialization; and (2) information-first guided review. With the hate speech reviewer specialization pilot, Facebook was examining whether allowing content reviewers to focus on only a few types of violations (rather than reviewing each piece of content against all of Facebook’s Community Standards) would yield more accurate results, without negatively impacting reviewer well-being and resilience. Facebook completed its initial six-month long pilot, with pilot participants demonstrating increased accuracy in their decisions, and fewer false positives (erroneous decisions finding a violation when the content does not actually go against Community Standards). While more needs to be done to study the long-term impacts of reviewer specialization on the emotional well-being of moderators, Facebook reported that participants in the pilot indicated they preferred specialization to the regular approach, and that attrition among pilot participants was generally lower than average for content reviewers at the same review site. Given those results, Facebook will explore a semi-specialization approach in the future where reviewers will specialize on a subset of related policy areas (e.g., hate speech, bullying, and harassment) in order to significantly reduce pressure on reviewers to know and enforce on all policy areas. Facebook is choosing to pursue semispecialization instead of specialization in any given Community Standard area to limit the amount of time that any reviewer spends on a single violation type to reduce risks of reviewer fatigue and over-exposure to the same kind of graphic or troubling content. At the same time, the company continues to build out its tools and resources supporting reviewer well-being and resiliency. Facebook reports that it is also working on establishing a set of well-being and resiliency metrics to better evaluate which efforts are most effective so that the company’s future efforts can be adjusted to be made more effective, if necessary. The other pilot, Facebook’s information-first guided review pilot, was designed to evaluate whether modifying the tool content reviewers use to evaluate content would improve accuracy. The standard review tool requires reviewers to decide whether the content is violating first, and then note the basis for the violation. Under the pilot, the order was reversed: reviewers are asked a series of questions that help them more objectively arrive at a conclusion as to whether the content is violating. Facebook completed a successful pilot of the information-first guided approach to content review, with positive results. Facebook states that content reviewers have found the approach more intuitive and easier to apply. Because switching to information-first guided review required creating new review tools, training, and workflows, Facebook felt the need to fully validate the approach before operationalizing it on a broader scale. Having now sufficiently tested the approach, Facebook plans to switch to information-first review for all content flagged as hate speech in North America, and then continue to expand to more countries and regions, and more categories of content. While COVID-19 has impacted the availability of content reviewers and capacity to train reviewers on the new approach, Facebook indicates it is working through those issues and looking to continue its progress toward more widespread adoption of the information-first approach. 43