Last week, my first journal article was published in Media and Communication. I worked with Nick Diakopoulos to research A/B headline testing in newsrooms. Headline tests allow publications to show different parts of the audience different headlines and measure how many people click on each option. Newsrooms use those tests to determine which headline audiences will prefer. I’ve summarized key findings from the paper below, and you can read the whole thing here.
1. Headline tests change headline writing.
Traditionally, journalists relied on their editorial judgment to craft compelling headlines. With A/B testing, the audience now has a way to give its direct feedback on what a headline should be. Newsrooms see both good and bad in this shift: Good because some of the guesswork gets removed from headline writing, and bad because testing has the potential to encourage clickbaity headlines. Newsrooms navigate that tension by giving editors strict control over the headlines that get tested.
2. The goals of testing depend on the newsroom.
In all cases, newsrooms use headline tests to increase traffic to their stories. Some newsrooms test headlines on a case-by-case basis, as a way to make sure important stories do well. Others only run tests geared toward answering long-term headline writing questions, like how long a headline should be, or what kinds of verbs work best. Headline testing also creates secondary benefits, like training new writers on what the audience responds to.
3. Optimizers play a key role.
There’s a group of newsroom workers, including social media managers, audience engagement staff, and analysts, that one interviewee called “optimizer type people”. We label them “optimizers”. Optimizers are critical to the headline testing process. They know how to work the testing system, they interpret results, and they provide recommendations to editors. In many cases, newsrooms adopt headline testing in large part because of lobbying from an optimizer.
4. Testing works best when everyone works together.
Since optimizers run tests and journalists write headlines, those groups need to work well together for A/B testing to take hold. Optimizers make their relationships with editors work by providing clear and convincing evidence of the benefits of headline testing and maintaining constant communication. They also leave room for editors to influence the testing process by giving them veto power in cases where the winning headline isn’t a good fit for the story.