persona management; pro-government; propaganda; psychological operations; psyops; social
media; sock puppet*; troll*; Twitter.
In our 2017 report, there were two major limitations to our qualitative content analyses: media
bias and language. Media bias is a common limitation to content analysis that uses purposive
sampling (Earl, Martin, McCarthy, & Soule, 2004; Joyce et al., 2013). To help mitigate bias, we
used LexisNexis and the top three search engine providers—Google, Yahoo! and Bing—which
provided hits to a variety of professional, local and amateur news sources. To ensure that only
high-quality news sources were used to build our dataset, each article was given a credibility
score using a three-point scale. Articles ranked at one came from major, professionally branded
news organizations (see Appendix 1). Articles ranked at two came from smaller professional
news organizations, local news organizations, or expert commentary and professional blogs (see
Appendix 2). Articles ranked at 3 came from content farms, social media posts, or nonprofessional or hyper-partisan blogs. These articles were removed from the sample.
Language was a second limitation to conducting our qualitative content analysis. For this year’s
global inventory, we were able to draw on news articles and secondary resources written in
English, Spanish, German, Italian, Polish, Portuguese, Russian, and Arabic. We also worked with
BBC monitoring who provided an additional portal for collecting and aggregating high-quality
news and information on cyber troop activity, as well as translation services for news articles
from Malaysia, Kyrgyzstan and Taiwan. Thus, for this year’s report, we were able to examine
news coverage across 10 different languages. We relied on English-language-only reporting for
Armenia, Azerbaijan, Cambodia, Czech Republic, Hungary, Israel, Myanmar, Netherlands,
North Korea, Pakistan, Philippines, Thailand, Turkey, and Vietnam.
While we were unable to analyze local news sources for every country in this report, the third
phase of our research methodology—consultation with experts—allowed us to peer review the
English language reporting and secondary literature we found and discuss additional resources
and citations in alternative languages with native speakers. Experts were asked to review the
country profiles drafted by research assistants, and (1) fact-check the information and data for
accuracy; (2) provide additional citations to openly available material; and (3) provide general
feedback on the reliability of the data. We consulted experts for 34 of the 48 countries in our
sample, namely: Argentina, Austria, Azerbaijan, Bahrain, Brazil, Colombia, Ecuador, Egypt,
Germany, Hungary, India, Iran, Kenya, Malaysia, Mexico, Myanmar, Netherlands, Philippines,
Poland, Russia, Saudi Arabia, South Africa, South Korea, Syria, Taiwan, Thailand, Turkey,
Ukraine, United Arab Emirates, United Kingdom, United States, Venezuela, Vietnam, and
Zimbabwe.
8