Motives and Metrics: Steps to Clarify Public Health Information
Is this real? Can I believe this? How do I know if this public health information is true? I honestly can’t even count how many times I have been asked those questions about public health news articles, preliminary research studies, memes, videos, and social media posts. The pandemic ignited a desire in most people to learn about what was going on, but theories, ideas, and crazy information started to flood the internet faster than public health education.
More true and false information than we could ever need exists online, so you can always find a fellow voice to join your narrative or see what you want to see, but does that make what you want to see true? Regardless of our personal thoughts and biases, which we all have to some degree, we should have some form of filtering process before we believe an article or pass along information.
Below is a list of basic steps I take when evaluating articles. The majority of steps are things we can all do. Towards the end are more technical research screening steps for my students reading this.
Pause. Whenever I come across an article to evaluate, before I jump into reading, I pause. I make sure that my mindset is right and that I am open to learning from new perspectives. Headlines first of all try to trigger emotional responses (see two points below). I know that if I don’t take a moment to pause before I read, and I just skim an article as fast as I can, that I will miss key pieces of information and jump into the narrative being projected. In a way, I think humility plays as much of a role as a clear mind. I have an area of expertise that I have been trained in, and I know my limitations. Others have been trained in different areas, so I try to have an open mind about learning something new when reading, regardless of which political party the author supports. As you pause, don’t rush through an article. Allow yourself to only read when you have time to clear your mind, walk through the steps below, and adopt the posture of humility and learning instead of looking for intentional bias.
Author info. Before I read an article, I look at not only who the author is, but what their background is and if they are qualified to educate on the topic. This was actually a frustrating aspect of the pandemic for me because everyone claimed to be an expert, wanted a voice, and wanted to be in the spotlight (some for money). Is the author a journalist with a degree in writing? If so, they can probably write compelling pieces, but may not have the appropriate training to interpret data to guide a nation. Does the author have “Dr.” in front of his or her name? If so, what does that mean? I remember reading a health article and the content was not aligned with many textbooks. I looked at Dr. ____’s background to see why she had a different take on the topic. It turns out, she had a doctorate in art history and no background in any health-related field whatsoever. That is tricky, because when we see Dr. ____ on a health article, we can easily make the assumption that she must be in that field. We could even look at that through a deeper lens, and I don’t want to go too far down a rabbit trail with this one (post for another day), but the area of expertise for someone with a PhD, MD, and DO is so important. In my field of public health, I am trained in data collection, analysis, and interpretation for populations. A physician (MD) who is trained in ophthalmology is trained to evaluate a person’s eye health one patient at a time (not populations) and likely conducts procedures. I would never write an article and tell an eye doctor how to perform eye surgery or what I think is best, because that is not my area of expertise (even though we both have “Dr.” before our names). I would also not tell a third-grade teacher how to design curriculum and run a classroom because, again, that is not my area of expertise. When you see people stepping outside of their respective areas to have a voice, especially when it comes to public health, you should pay attention to that and be cautious.
Selling and motives. Remember that doctor of art history in my previous point? I scrolled through her page, and she was actually trying to sell a product! Her goal was to get people’s attention, drive them to her page, and sell a product to make money. Public health is a sensitive topic because, I believe at the end of the day, people really do want to be healthier and are looking for fast ways to do that. But how did she get people to go to her website? Her article headline was fear-based, and then she tied hope to that fear in her products. News channels have a similar method, maybe not by selling products, but by getting clicks. Headlines are strategically crafted, especially by news channels, to trigger emotion. Any kind of emotion such as excitement or even the opposite extreme of fear leave you wanting or needing to know more. When our emotions trigger curiosity that makes us click to read! At the end of the day, news sources survive and actually make quite a profit on those clicks. Look through the source of the health article you’re reading and see if they are trying to sell you something or make money off of your clicks.
Agenda and affiliation. This is a challenging step for some to process, particularly without that pause and clear mind in step one. Similar to the profit aspect of news channels and some authors outside of their scope of practice, there can be agendas or affiliations of which to be aware. The obvious ones like Fox News, a far right channel and CNN, a far left channel, are easy to see. At the end of the day, true public health policy and regulations (from trained public health professionals) are not made for a political agenda. The messy part is that they get adopted or enforced (or not) by political parties in office. If the political party or politician has other agendas, which let’s be real, they usually do (in all parties), then that connection from public health to enforcement to the public can get messy. There will be an entire blog post about this topic because the process of analyzing public health data and designing policies is so multifaceted, and then there is a political step in enforcement, which adds bias and complexities. Whenever you read an article on public health, particularly from a news source with political affiliations or agendas, you will need to filter through what is really being said. I actually stopped watching the news early on in the pandemic, and I just never went back. The only time I filter through a news article is when I am asked to help someone interpret public health information that has usually been twisted.
Social media, memes, and bots. Let’s face it. We probably all have some form of social media whether it is Facebook, Instagram, TikTok, Twitter, Snapchat, or something too new and popular for me to even know about. As much as we love social media for connection, keeping in touch with friends and documenting memories (I’m honestly grateful for those in the context of social media), at the end of the day, they are not about that; social media platforms are fascinating and brilliant ways to collect information on you, to market to you, and ultimately make money. Every click, scroll, second, and photo are tracked with enhanced technology to know you and market to you. Have you ever seen an ad on a social media platform and wondered how they knew exactly what you needed or what you had been searching for earlier that week? Sadly, they don’t care about your social life, they care about knowing you and where you spend your money.
Now that we have that foundation, we could place social media in our previous category of selling and motives, right? But I think there is a little more to it. Anyone can make a meme. It is a simple process to take a photo, add a catchy saying, post it, and have others share. Just because a person writes some words on a photo doesn’t mean the words are true. This was something that shocked me through the pandemic. I could not believe how many memes were being shared with false data, and people just believed them! Most students are taught in high school and college that Wikipedia is not a reliable source to gather data for a paper. The reason is because anyone can edit and write anything they want. That means they do not need to show their credentials in the field or have any sort of training, and they can actually just be completely anonymous in the process. If my pre-teen had access to the internet and could go on Wikipedia, he would probably edit and make silly definitions for crazy topics. I can’t even imagine. Social media memes are the same, and maybe even worse. Not only can anyone just make them based on feelings (and make them look real), there are bots that generate them. I do not have expertise in measuring bot activity or knowing what percentage is accurate, so I can’t give you a precise percentage of how many bots make the posts you read; but know that all memes and posts are not always even created by people. Some of us in the field of public health created social media accounts to help increase the number of credible sources on social media, but we are far and few between. If you can’t follow step two, the author check, then you shouldn’t just believe information because it is on social media.
Go directly to the source. When an article is written on public health information, you should be able to find the original source (research article) linked. Look at the original research or health information, identify who published it, find the stage of research it is in, and read the results, implications, and conclusions with an eye for precision on interpretation. If you can’t find the original source, contact the author. Be cautious if you can’t see where the original information came from.
Access to data. What an incredible time we live in for knowledge and access to information. I think accessibility to information is growing faster than we can keep up with it. Something I vividly remember being lectured on in my master's and PhD program was how our jobs as researchers have shifted with learning what has been done in our respective fields (literature reviews). When many of my professors obtained their PhDs, they had to check out peer-reviewed journal articles from the library, read hard copies, and create their reference lists. Our roles evolved as graduate students, because we could get peer-reviewed research articles online instead of hard copies in libraries. The problem with that is that many organizations launched research journals without the rigor or peer-review process needed to advance a field. Just because a research article is published online, it does not necessarily mean that it went through the peer-review process it should. This is actually a topic I cover in several classes, so if this is new to you, the best approach is to ask someone in that field if the journal is credible.
A challenge we saw throughout the pandemic was access to preliminary data. The general public and even news channels and reporters were accessing preliminary research and sharing it as if it was fact and proven. Preliminary studies are meant to be launching pads for researchers to design new studies to replicate what has been done. They usually have smaller sample sizes and are meant to add to the knowledge base to build on in the near future. They are not usually the gold standard or to be used to make generalized interpretations. Our pandemic of misinformation was flooded with preliminary research that the world had access to, not just the researchers trained in interpretation and research design. It is important to recognize what stage of research an article is published in, the credibility of the journal, and the implications associated with the next steps (share with the public vs. conduct more studies).
Additional steps specifically for public health students:
Research design and psychometrics. When you’re reading through a research article (and I am already making the assumption that it is in a credible, peer-reviewed journal), you have to clearly identify the research design. Ask the question, is it experimental or nonexperimental? Observational? Can you identify cohorts? Is it a clinical trial? Was it blinded or double-blinded? Write down the research design, because that will help you understand the variables collected. Next, find the sample size. Is it adequate? Are there enough participants for accurate results? Is the sample size large enough for the given statistical power needed? Is the study at risk for a Type I or Type II error? Did the researchers transparently list their limitations? Perfect studies do not exist, especially when studying people and health behavior, but the process should be transparent and have sound methodology.
Next, look through the psychometric properties of the instruments used in the study. Are the instruments measuring what they are supposed to measure? Is it concrete or more abstract (constructs)? Is the instrument measuring what it should each time it is used? Is there evidence of validity and reliability? If a researcher does not have an instrument with measurements for reliability and evidence of validity, it could be in the beginning stages of development. Again, the transparency of this should be evident. At the end of the day, if a study is conducted and the instrument used for collecting data does not have adequate psychometric measurements, it shouldn’t be used for generalized information.
Interpretation. After you can identify the methodology of the study (instruments, data collection, and analysis), look at the results and discussion. These sections will include the exact interpretation of the research. Most researchers also add sections for future plans, implications, and limitations. What is the exact interpretation of the study based on the biostatistical test used for analysis? Is the news channel or article reporting the research findings using the appropriate interpretation or was it twisted? Here is a simple example to help understand the importance of interpretation. In one of my studies, I identified that women ages 18-49 with little to no education were 1.5 times more likely to not get tested for HIV compared to women with education secondary or higher in the Democratic Republic of Congo. When these findings are reported, they need to be explained exactly like that. A headline cannot change the wording and say, “Women who are less educated are less likely to be tested for HIV.” That study is not generalizable to the world (even with an instrument with sound psychometric properties). It can only be generalized to women of that age group in that country. Depending on the motives of the news reporter, that could be twisted to trigger emotion to get more followers. This is why we always go to the original source of the research or study being discussed and specifically look through the methodology and interpretations to see how we can use the findings to learn.