Book Review: The Case for Making Data-Driven Decisions

By Esther Landhuis

What guides an important decision? It might start with a core human emotion — fear, anger, a stubborn sense of hope. Some decisions draw from personal experience or from a gut feeling that a particular course of action is good and right. In such situations, decision makers typically trust the opinions of authorities and thought leaders and act according to their example. That is how societies and governments have operated for much of human history.

But what happens when expert opinion and practices conflict with real-life observations and experiences?

In “Beyond Belief: How Evidence Shows What Really Works,” science journalist and editor Helen Pearson profiles a handful of people working in medicine and other fields whose keen instinct for these tensions led them to take action. Eventually these individuals became leaders of the worldwide evidence movement, which encourages the use of data from rigorous research to direct treatments, policies, and plans. Pearson’s vivid descriptions make otherwise stodgy academics jump off the page as three-dimensional human beings with quirks, faults, and wonderings.

As an example, a young physician working overseas at a refugee camp in the late 1960s felt uneasy about withholding antibiotics to children with measles who didn’t have a confirmed bacterial infection. Although this is what he had been taught in medical school, “it was difficult to do nothing when faced with a child who seemed to be getting seriously ill,” Pearson writes.

In another case, a statistician came across a prominent academic review that suggested psychotherapy offered little benefit, and pondered it with dismay. The statistician’s own experience — fewer anxiety and neurosis flare-ups with eight years of weekly psychotherapy — ran counter to the review’s conclusion. In a third instance, an analyst who joined the New York Police Department in the early 1970s noticed a sharp rise in crime rates. It gave him pause and shifted his attention to the long-held three R’s of policing — random patrol, rapid response, and reactive investigation — raising questions about their effectiveness.

These are just a few scenarios described in the book in which an astute thinker felt the pangs of a nagging quandary in their field. Yet rather than submitting to conventional wisdom, they took initiative to address the matter by taking a closer look at existing data or conducting analyses to gather more.

The doctor, Iain Chalmers, upon returning home from the refugee camp, dug into data on measles treatment. He found six controlled clinical studies, already published by that point, which showed that giving antibiotics early in the course of disease to children with measles was effective at preventing serious bacterial infections. Sadly, he realized that some of the children who had been under his care worsened and died after he followed advice he’d learned in medical school that disagreed with the research evidence.

The statistician, Gene Glass, re-examined results and procedures in the published psychotherapy review that troubled him. Upon a closer read, he learned more about the author’s approach for evaluating research studies. The method, called vote-counting, involves sorting individual studies on a given treatment according to what they found — those showing the treatment was helpful, those showing it was harmful, and those finding no clear effects. The number of studies in each category was tallied, and the group with the most “votes” determined the review’s conclusion. This vote-counting method seemed sensible at the time but could produce misleading results — for example, by assuming all studies carry the same weight regardless of their number of participants.

Soon after starting his job at the New York Police Department, criminologist Lawrence Sherman learned about research that supported his suspicions that fundamental tenets of policing wouldn’t hold weight when put to the test. As a case in point, a Kansas City experiment compared police beats with no patrols, standard patrols, and double- or triple-level patrols — and found that burglary and vandalism rates were unaffected by the extent of patrolling.

Chalmers’ efforts led him to like-minded pioneers, such as Archie Cochrane, whose work inspired him to establish what would eventually become the Cochrane Reviews, now broadly used to inform decisions about treatments and care. Sherman’s scrutiny of policing policies paved the way to greater appreciation for randomized experiments in social policies.

Nowadays, evidence-based thinking guides decisions not only in health and policing but also a range of other fields including business, government, conservation, and education. Especially among educated circles in developed countries, it’s hard to imagine a world that isn’t awash with data and navigating information overload.

It’s curious that the evidence movement took root in the 1990s in medicine, a historically slow-to-change discipline, before transforming other fields. Pearson makes a critical observation about individuals who became leaders in evidence-based medicine: A key reason they were willing to heed their instinct that reality didn’t jibe with tradition was that they were not entrenched in the establishment that could potentially be threatened by the evidence.

Already accustomed to operating outside of established norms, these individuals “didn’t care if they put the noses of bigshot doctors and academics out of joint,” she writes. “They were rebellious, intent on using data and evidence to challenge the conventional way things were done.”

A successful movement also needed a catchy name. Early proponents had initially described the new approach as “critical appraisal” or “scientific medicine.” But the former didn’t sound very compelling, and the latter infuriated others in the field who felt they had been practicing “scientific” medicine all along. Before long, the term “evidence-based medicine” stuck — especially after a 1992 article in The Journal of the American Medical Association, a high-profile medical journal, popularized and legitimized the concept for a broader audience.

Still, the idea took a while to gain traction. Some critics worried that this bold, new evidence-based movement would “crush the freedom of doctors to make decisions based on their own experience or knowledge of their patients,” Pearson writes. “The fear was that it was ‘cookbook medicine,’ in which doctors followed protocols that evidence dictated like cooks rigidly following a recipe.”

Trish Greenhalgh, a professor of primary care health sciences at the University of Oxford, pushed for evidence syntheses in medicine to “include a wider array of evidence than randomized trials when assessing causal claims,” Pearson notes. A vocal proponent of masks during the Covid-19 pandemic, Greenhalgh argued that waiting for solid data can be unwise in some public health settings. “In the face of a pandemic the search for perfect evidence may be the enemy of good policy,” she and colleagues wrote in The British Medical Journal.

Today, many readers may feel the book is preaching to the choir. Our world, particularly academia and medicine, is powered by evidence. Large volumes of data are often required to publish a research paper. Health insurance won’t pay for treatments without high-quality data from controlled trials showing they are safe and effective. Evidence-based guidelines have led to regular colonoscopies and mammograms that save lives by catching cancer early when it can be treated. Yet at the same time, glowing reviews from internet influencers can lure consumers to drugs with little to no human data.

Pearson herself notes that learning about systematic reviews and the different ways that evidence gets synthesized and analyzed was eye-opening to her, even as an editor at Nature magazine steeped in the scientific literature. Despite widespread acknowledgment of the importance of quality evidence, in practice “most scientists today still trade in the currency of the single academic publication, publishing one paper after another because that’s how they make their names,” Pearson writes. Wrangling evidence from varied sources and trying to make sense of it is a time-consuming process.

So, too, is judicious and effective care for patients who suffer from modern medicine’s gray areas — diseases such as long Covid, PANDAS/PANS, and post-polio syndrome, which are diagnosed by clinical descriptions rather than lab or imaging results. Getting treatment can be a struggle, since many potential therapies have not been put through the costly and time-intensive gauntlet of randomized trials that are required for FDA approval. Sometimes the absence of solid evidence reflects the lack of an industry sponsor or other entity that is financially incentivized to conduct the studies.

And even when there is solid evidence, each patient is ultimately a unique case, the sole subject of a research study with an N of 1. If published data comes from trials of people who differ by gender, age, socioeconomic status, ethnicity, or other factors, the evidence may not always apply to a given patient.

The real challenge — for physicians and experts in other fields — is knowing when to seek data and how to integrate it with intuition and experience to make smart decisions in specific situations, settings, and populations.

In the book’s introduction, Pearson acknowledges shortcomings and nuance with relying on evidence. “Evidence rarely provides certainty,” she notes “Often the evidence simply isn’t there, or is hard to find, or is complex and contradictory — and new studies can come along that overturn existing ideas. But it can reduce uncertainty and improve decisions — and that can be life-changing.”

—

This article was originally published on Undark. Read the original article.

—

If you believe in the work we are doing here at The Good Men Project, please join us as a Premium Member today.

All Premium Members get to view The Good Men Project with NO ADS.

Need more info? A complete list of benefits is here.

—

Photo credit: iStock.com