5/27/22, 8:18 PM Integrating autoconversion: Facebook’s path from Zawgyi to Unicode - Engineering at Meta Content encoding detection To perform autoconversion, we first need to know the content encoding, that is, the encoding used when the text was to represent characters in Burmese and other languages. Because of this, we can’t tell whether a list of code points re string of code points makes sense in both encodings. With a model trained on text created in Zawgyi and Unicode, we Unicode keyboard. Our detection is based on the myanmar-tools library’s approach. We train a machine learning (ML) model on public F model keeps track of how likely a series of code points is to occur in Unicode versus in Zawgyi for each sample. Later, model’s prediction for whether that sequence of code points was more likely to have been entered in Unicode or in Za Device encoding detection Next, we need to know which encoding was used by a person’s phone (i.e., the device encoding) to understand wheth advantage of the fact that in one encoding, combining several code points will combine text fragments to create a sin separate characters. If we create a string on-device and check the width of that string, we can tell which font encodi tell the server in future web requests that the device is using Zawgyi or Unicode and make sure any content that is fe in question is Zawgyi or Unicode and sends that encoding as part of the locale field in the web request (e.g., my_Qaag Conversion https://engineering.fb.com/2019/09/26/android/unicode-font-converter/ 3/7