Gaps in information frustrate the work of fact-checkers. But what about when a government agency creates them?
“To know that the data has been tracked in the past and is maybe still tracked currently and is not being released — that just seems like a step backward,” said Angie Holan, editor of PolitiFact (a project of the Poynter-owned Tampa Bay Times).
Her concern stems from a recent change to the FBI’s 2016 crime report, which FiveThirtyEight reports now has close to 70 percent fewer tables than the 2015 version. Among the data tables missing in the report — the first to come out under the Trump administration’s FBI — is specific information about arrests, homicides and the only national estimate of gang-related murders.
The FBI has released a statement saying that the reduced tables were part of an effort to streamline its crime report because of a lack of web traffic, and that the missing data is still available upon request. But FiveThirtyEight reports the FBI only provides the raw data file, which is harder to analyze and does not always match the figures posted online. The news outlet also cited a former FBI employee who said the process was “shocking.”
The missing crime statistics are not the first example of government agencies scrubbing information during the Trump administration. In April, the EPA removed several agency websites that referenced detailed climate data and scientific information, The Washington Post reported.
Emily Atkin, an environmental politics reporter for The New Republic, told Poynter that while the change mostly removed climate information for the general public, it has made it a little harder for her to find basic data.
“A lot of the information they’ve taken down is readily available in other places,” she said. “The thing that used to take us 10 minutes now takes us 30 minutes.”
While PolitiFact hasn’t yet encountered a fact check that required data from the latest FBI crime report, Holan said the fact-checking team will try to work around it going forward. Glenn Kessler, who runs The Washington Post Fact Checker, also told Poynter in an email that the newspaper has discussed how to deal with the gaps.
In that, there are two general things any organization can do: finding the data elsewhere and telling your audience when you can’t.
“You just have to do your best and look for alternative sources,” Holan said.
In Argentina, that’s a lesson that fact-checkers learned first-hand years ago — and they offer important insights for organizations anywhere around the world facing sudden reductions in government data.
In 2007, the Argentine government started fudging its inflation rate statistics for political gain. Even after reforms in 2014, Reuters reported Argentina was still underreporting its inflation rate compared to private estimates.
And it’s not just economic data that’s been hard to come by. In 2008 — also under the administration of Cristina Fernández de Kirchner — the Argentine government stopped publishing annual crime statistics. It wasn’t until 2016 that they published new numbers.
Those gaps in official data posed a formidable problem for the country’s fact-checkers. Laura Zommer, director of Chequeado — a Buenos Aires-based fact-checking organization — said that, in order to accurately check politicians’ statements about the economy, they had to turn to alternative sources.
“We use alternative sources when (there is) bad information,” she said. “Then we use a special qualification that we have in Chequeado that is called ‘insostenible’ (unsupported).”
While statistics from nonprofit organizations and private economists are often imperfect, they can serve as placeholders for missing or inaccurate government information, Zommer said. And at American fact-checking organizations, that’s also standard practice for working around gaps in data.
In the U.S. — where there’s a lack of numbers about issues such as police shootings — journalists have also turned to unofficial measures, drawing upon databases created by organizations like The Washington Post and the John Jay College of Criminal Justice to fill in gaps. Eugene Kiely, director of FactCheck.org, told Poynter that while FBI crime data is notoriously unreliable because it’s voluntarily reported, it’s usually the best data available to journalists.
“There are other sources out there that do it, but you can’t make comparisons to previous years because you’re mixing apples and oranges,” he said. “You just have to look for another source and be clear to our readers that we can’t make comparisons.”
In short: Some data is better than no data. But what should fact-checkers do when no one else is keeping track?
“In those cases, our work isn’t to explain or to inform about the data of the phenomenon, but to inform citizens that there is not evidence about it,” Zommer said.
It’s a good rule of thumb to repeat a politician's claim and add context about why it can’t be verified, she said. During the years that Argentina didn’t release annual crime statistics, Chequeado published stories adding context to claims about the murder rate.
“Every time that the president … said homicides were going down in that period, we published an article saying the government stopped publishing official statistics about this — and it’s up to you to believe them or not,” Zommer said.
Those stories can be backed up by filing public records requests.
“All the times that they repeat (claims), make access to information requests again and again asking for that data,” Zommer said. “It’s stronger if we can say … we also required it in a formal way using the law and they don’t care.”
“That shows clearly that it is a political decision — it’s not just a mistake or someone decided not to do that.”
Being transparent and persistent when it comes to publishing government data goes back to what Kiely said is a central tenet of fact-checking: The burden is always on the person who makes the claim. And Zommer said fact-checkers have a responsibility to hold them accountable.
“You as a fact checker should be an advocate for good public data,” she said, “and if that data is not there, you have a problem.”
