The way we report, present and distribute the news is changing rapidly, and running experiments is becoming more important for newsrooms trying to keep up with technological change.
At the Financial Times, we run experiments to answer a variety of questions: What is the optimal publishing schedule for stories? How can we best package several pieces of related content together? Can we do better journalism by being transparent with readers in our reporting process?
There are skills involved in creating experiments that your team — and your whole newsroom — enjoy lasting benefits from.
A large part of my job as special projects editor at the FT is to think about doing experiments well. Central to this process is making sure your experiment has a distinct purpose, which can be a somewhat nebulous task.
But there are several concrete ways to hone the focus of your experiment. Here are some tips based on my findings and experience:
Think beyond the tools
Whether it’s telling readers to send in tips via WhatsApp or getting a reporter to use Snapchat for live coverage of an event, many newsroom experiments take the form of using a new tool or platform. But it is important not to let the tool frame your goal. So, for example, instead of setting a goal of “using Snapchat for the first time,” you could frame it instead as:
“To figure out what type of stories lend themselves to live Snapchat coverage.” Or: “Our target audience is on Snapchat. We want to figure out how we can reach them there.” Or: “We’ve tried it once before and got good results. This time, let’s figure out whether we can simplify the logistics so we could be doing these once a week.”
This is partly because the goals of the tool or platform may not be the same as your own goals. For example, your organization’s intention to drive traffic back to its own site could clash with, say, Snapchat or Instagram’s goal of keeping users on their platforms.
You don’t need to do something revolutionary
It is undoubtedly cool if you can innovate and come up with something that transforms the whole of the media industry, but your experiments are more likely to be useful if they are directed towards solving problems related to your newsroom.
Your experiment doesn’t have to be a first for all media organizations in the world. There’s value in trying to replicate, say, Wired’s smartphone Thunderdome, or The New York Times’ election slackbot.
There are also benefits to doing smaller, more incremental, but more frequent experiments: There’s less of a “failure” cost, it instills a culture of experimentation in the newsroom, and they’re easier to carry out.
At the FT, we were interested in figuring out how to improve the “landing” pages for series and projects. We did a big experiment on this with “When Rates Rise”, a project timed for the first U.S. interest rate rise in nearly a decade. But importantly, we followed it up with further “landing page” experiments that built on what worked (and what didn’t) with “When Rates Rise.”
NPR’s recent series of experiments on how to make audio shareable are a good example of this incremental approach.
Be (at least semi-) scientific about it
A good indicator of whether you have set a goal correctly is if you can articulate both what success and failure would look like. That’s why “let’s try to use Periscope for the first time” doesn’t work well as a goal. Failure in that case looks like: We didn’t end up using Periscope after all.
In practice, this could mean figuring out how your goals translate to key performance indicators, which can be matched up to specific metrics. So for example, your goal might instead be: “Let’s aim for 100 viewers on our first set of Periscope videos.” Even if you “failed” by not hitting a target, it can provide a springboard for thinking about why that happened and how things should be done differently next time.
Of course, you won’t be experimenting in laboratory-like conditions. You’re unlikely to have enough historical data to establish a baseline comparison to help set your targets. As a journalist, there will be a part of you that resists thinking of stories as fungible units that can be compared against each other in numeric terms.
That’s all true, but it doesn’t mean the process of articulating what success looks like and how you will measure it is useless. It’s invaluable when trying to obtain permission to conduct the experiment or to convince others to support and collaborate with you on it.
After the experiment
- Be sure you hold a retrospective meeting. In the meeting, discuss what went well, what went poorly and what should be done differently next time. Document and distribute the findings.
- Try to come up with at least one practical step to take as a result of what you learned.
- Talk about your experiment and evangelize both internally and externally. Experiments are meant to be conversation starters that spark other ideas.
- Ask yourself: Are there opportunities to do it again (maybe on a different story) and make incremental improvements?
