Samuel R. Mendez

Performance of reading grade level estimators for public health communication on Twitter

The importance of clarity in public health communication is clearer than ever amid the ongoing impact of the COVID-19 pandemic. Estimated reading grade level is a prominent metric used to assess clarity of health communication. However, widely used estimators were not designed for health communications or social media specifically. Some estimators like the SMOG are adaptable to short texts, and others like the Flesch-Kincaid formula are usable on short texts without adaptation. However, using these estimators for social media posts in the content area of public requires nontrivial decisions around how to treat acronyms (e.g. COVID-19), treatment names (e.g. Paxlovid), URLs, usernames, sentence fragments, and other medium/domain-specific features. There is a need for empirical data on a relevant use case to illustrate how prominent estimators perform relative to each other, as well as under different methodological decisions within public health (e.g. inclusion “COVID-19” as a single polysyllabic word, compared to excluding it from estimators completely). This project applied 3 prominent reading grade level estimators (SMOG, ARI, and Flesh-Kincaid) to a corpus of Tweets about COVID-19 in 2020 from US state public health agencies prior to the emergency authorization of a vaccine, consisting of text alone or with photo media attachments (n=39825). This project examined to what extent the distributions of each estimator differed from each other under a set of different decisions around how to treat domain- and medium-specific text features. Finally, this project qualitatively described Tweets that had relatively stable reading grade level estimates across estimators, as well as those with extreme variation. This project presents preliminary findings on the amount of variation in the distributions of scores between the 3 estimators under different methodological decisions, as well as implications for the limitations of reading grade level estimators as a means of assessing clarity of public health communication on social media. This project’s results were interpreted in light of utility amid resource and time constraints during a public health emergency. A distribution of reading grade level estimates may prove more useful for public health communicators than an estimate for a single Tweet. Larger health agencies with more technical infrastructure may find it useful to use reading grade level estimators on a comprehensive data set of their Tweets as part of an organizational health literacy assessment. Smaller agencies with less technical infrastructure may benefit more from reviewing a purposive sample of Tweets with a checklist of best practices/guidelines. This project contributes empirical data on the performance of reading grade level estimators for public health agency Tweets. This case of COVID-19 communication provides useful insight into the limitations of reading grade level estimators, and offers practical suggestions for when use of these estimators may be most appropriate.

Bio: Sam (they/them) is a PhD student at the Harvard T.H. Chan School of Public Health. Sam's research focuses on bridging natural language processing, media studies, and health literacy to better understand and improve public health communication in today's complex media environment.