Research Shows That … Education Journalists Don’t Always Use Research All That Well

The two most frightening words in education journalism are "research shows". While some writers completely disregard research, many education journalists rightly aim to bring empirical evidence to the table in a field that often relies on intuition and anecdote. However, it is important to recognize that "research shows" often means "one prominent study that I can recall easily".

Let’s take a case study: There is currently a claim that "research shows" that incentive pay for teachers "doesn’t work." An article from Chalkbeat – Colorado at the beginning of the school year states, "Research has shown that paying teachers more money to stay at schools with difficult working conditions largely hasn’t worked." Unfortunately, no citation is provided.

Similarly, journalist Kristin Rizga writes in the Los Angeles Times, "Politicians have attempted to address the issue of teacher attrition by providing bonuses to teachers in high-poverty schools or linking pay to standardized test scores. However, recent research shows that these approaches have mostly been unsuccessful." These statements are presented as fact, using the authority of research. It begs the question, do you not believe in science?

However, there is a major issue with these particular statements: research in this area is far more complex than it is presented. There are numerous rigorous studies that highlight the positive benefits of performance pay for teachers or bonuses for those working in hard-to-staff areas:

– A study conducted in Tennessee found that teacher retention improved among those who received a performance bonus. Similar findings were observed in a study in Denver.

– In Washington, D.C., a study demonstrated that teachers improved their performance in response to performance incentives.

– Studies of Austin’s performance pay system and Minnesota’s performance-based pay and professional development system both found gains in student achievement.

– Two studies conducted in Israel found short-term and long-term gains for students taught by teachers who received performance bonuses.

– Several unnamed school districts implemented a program that paid high-performing teachers bonuses for transferring to high-poverty schools, resulting in gains in student achievement.

– A study of a North Carolina program that provided small bonuses to math, science, and special education teachers found that it increased teacher retention.

– A randomized study of schools across the country that implemented performance pay observed small, statistically significant gains in reading achievement.

– A study of districts nationwide found that those utilizing a performance pay system attracted academically capable teachers, as measured by SAT scores, more than districts that did not use performance pay.

We can and should have a substantial debate about the implications of these studies. Were their research designs appropriate? Did they miss any unintended consequences of incentive pay? Would the money spent on these programs be better utilized elsewhere? What about the conflicting evidence on performance pay?

These are crucial discussions to be had, but it is evident that the simplistic notion that incentive pay doesn’t work is misguided. One-sentence dismissals of its effectiveness without proper citation are misleading.

Even when journalists do delve deeper into the topic, they often rely too heavily on the safe and ambiguous phrase "the research is mixed". For instance, a story in EdSource concluded that the impact of high school exit exams on students was "uncertain". However, the overwhelming body of research actually suggests that such exams have no positive effect on student achievement and lead to numerous negative consequences for vulnerable students, including higher rates of incarceration. Unfortunately, this was not highlighted in EdSource’s article.

Understandably, journalists cannot provide a nuanced review of research on every topic within a news story. However, there are some best practices that all writers should follow:

– Whenever stating "research shows", always provide a citation or at least a summary of the specific research. Sometimes we are informed about research without a citation, or with a link to another article that does not clearly state which research is being referred to.

– Be cautious about basing conclusions solely on one or two studies. Instead, consider multiple studies from different sources or look for meta-analyses, which analyze a topic based on numerous pieces of research. If focusing on one study, it is important to emphasize its limitations.

It is truly remarkable that an increasing number of journalists are enthusiastic about incorporating evidence into their writing. However, it is important to note that research, when not properly utilized, can actually have a negative impact on readers, potentially even worse than having no research at all.

Therefore, it is crucial that we not only utilize research, but also utilize it in an effective manner.

A key point to consider is the "recent research" mentioned by Rizga, which is actually an article in The Atlantic that claims combat pay is not supported by much research as an effective tool. The article cites a 2011 research brief from the Center For American Progress. This brief highlights the fact that some incentive pay programs have been discontinued or have gained limited interest. However, it does acknowledge that studies indicate teachers do consider wages in their decisions to enter and continue teaching. It is important to note that none of the studies mentioned in this research brief are rigorous examinations of the impact of incentive pay on teacher retention or student achievement.

In conclusion, let us strive not only to incorporate research into our work, but to do so in a meaningful and effective manner. By doing so, we can ensure that our writing is informed and of the highest quality possible.