“the problem is that we do not and possibly never will have a model that captures even a majority of integrity harms [i.e., the spread of harmful content on the platform], particularly in sensitive areas”. 200 Underlining the extent to which these limitations in addressing “hate speech” are a result of Facebook’s core features, including news feed ranking, the author of the document states: “This leaves us in a difficult position. We do have reasonable metrics that can tell us when a given ranking change is likely to cause integrity harms – even with low precision and recall, we can get a decent sense of whether a launch is increasing hate speech, or misinformation, or other harms. However, we don’t have a way of effectively demoting this content in a targeted way (i.e. without significant collateral damage); and even if we did, we often won’t be able to launch them based on policy concerns”.201 Here, the author of this document suggests that Meta staff have the means to predict when tweaks to their content-shaping algorithms (such as those aimed at maximizing engagement) are likely to result in the amplification of “hate speech” and other harmful content. The author also admits, however, that Meta does not have the capacity to effectively reduce the amplification of harmful content in a narrow and targeted manner. They add that in any case, “policy concerns” often rule out any technical interventions which are aimed at reducing the spread of harmful content. It is not clear precisely which “policy concerns” are referred to in this document; however, this term in the Facebook Papers often refers to political considerations which influence company policy, such as whether a specific government or political actor will perceive an action by the company as biased. This logic of content-based solutions to algorithmically amplified harms rests on the premise that Meta is a neutral arbiter of content. In reality, Meta’s business model, along with its core products and services, fuel the spread of harmful content. Meta’s surveillance-based business model incentivizes the spread of maximally engaging, inflammatory content, including advocacy of hatred. These content-shaping algorithms which underpin Meta’s business model are analysed in further detail in Section 6.2, below. 6.2 ENGAGEMENT AT ALL COSTS: HOW META MAKES HATE GO VIRAL “We [Facebook] have evidence from a variety of sources that hate speech, divisive political speech, and misinformation on Facebook and the family of apps are affecting societies around the world. We also have compelling evidence that our core product mechanics, such as virality, recommendations, and optimizing for engagement, are a significant part of why these types of speech flourish on the platform” A Meta employee commenting in a leaked internal company document, August 2019 202 The Facebook Papers reveal in greater detail than ever before how human rights risks and harms are driven by Meta’s business model and its core content-shaping algorithms. This section provides an in-depth analysis of how these algorithms operate in practice, and how they fuel human rights harms. 6.2.1 VIRALITY One of Facebook’s most well-known features is algorithmic virality – whereby certain content is algorithmically boosted in such a way that enables it to reach a far wider audience than it could otherwise reach ‘organically’ (without being boosted). Another document contained in the Facebook Papers with potentially significant human rights implications is a post, entitled, “We Are Responsible for Viral Content”. 200 The Facebook Papers, “Demoting on Integrity Signals is not enough”, undated. 201 The Facebook Papers, “Demoting on Integrity Signals is not enough”, undated. 202 The Facebook Papers, “What is Collateral Damage?”, 12 August 2019, p. 34. THE SOCIAL ATROCITY META AND THE RIGHT TO REMEDY FOR THE ROHINGYA Amnesty International 41