If your time is short

• It’s hard to determine the precise impact of a large-scale event on the spread of the coronavirus. Contact tracing misses a lot of cases. Statistical methods used to make estimates often rely on assumptions that are difficult to get right.
• A working paper from four university researchers estimated that 266,796 COVID-19 cases over a month-long span could be linked to a motorcycle rally in South Dakota.
• Experts unaffiliated with the study said the Sturgis rally likely spread the coronavirus, but that estimate may be high.

In early August, more than 460,000 tourists poured into a small South Dakota city for a motorcycle rally over 10 days.

Few wore masks or practiced social distancing as attendees gathered at the Sturgis Motorcycle Rally for bike shows, races, performances and other events, according to the Associated Press. At one concert, the lead singer for the band Smash Mouth shouted, “F— that COVID s—!”

The conditions of the event made it seem like a recipe for increased COVID-19 infections. A new working paper says that’s what happened.

A team of four researchers said the rally could be responsible for as many as 266,796 coronavirus cases in the U.S. over a month’s time — about 19% of the cases reported between Aug. 2 and Sept. 2.

The rally was a “superspreading event” with a hefty price tag because it combined many “worst case scenarios” for coronavirus spread, the researchers wrote.

The preliminary report from San Diego State University’s Center for Health Economics & Policy Studies estimated that the Sturgis-linked infections could have generated over $12 billion in health care costs.

But South Dakota officials disputed the paper’s findings. Gov. Kristi Noem, a Republican ally of President Donald Trump and opponent of mask mandates, noted that it has not yet been peer reviewed and said it was “built on incredibly faulty assumptions.”

“This report isn’t science,” Noem said in a statement. “It’s fiction.”

This report isn’t science. It’s fiction.

Under the guise of academic research, it’s nothing short of an attack on those who exercised their personal freedom to attend Sturgis. (THREAD 1/) https://t.co/3zkmabJV2y

— Governor Kristi Noem (@govkristinoem) September 8, 2020

News reports and social media reflected the same disagreement over the study. Many news outlets reported on its findings. A few others sought to pump the brakes.

A Google search for the study returns some vastly different headlines, such as:

• From Yahoo News: “Sturgis motorcycle rally was a ‘superspreader event.’”
• From Reason: “No, the Sturgis Motorcycle Rally Didn’t Spawn 250,000 Coronavirus Cases.”
• From The Washington Post: “‘Worst case scenarios’ at Sturgis rally could link event to 266,000 coronavirus cases, study says.”

So, what’s the truth? We spoke to two of the report’s authors, who stood by their work, and four outside experts, who warned of the difficulties with estimating the coronavirus’ spread.

The rally, they agreed, had the makings of a superspreading event. But the 266,796 figure is still an estimate, and experts cautioned that it may be too high.

The Sturgis rally study

News outlets reported surging cases in the Dakotas following the Sturgis rally. The event drew hundreds of thousands of people to a city with a population of 7,000 from Aug. 7 to Aug. 16.

The South Dakota Department of Health said the recent case increases can be attributed in part to the reopening of K-12 schools and universities. Department officials said on Sept. 8 that they had linked 124 COVID-19 cases in South Dakota to the Sturgis rally through contact tracing, a small fraction of the study’s estimate.

The Associated Press and The Washington Post have also identified a few hundred more cases linked to the rally across other states. One death was reported in Minnesota.

But contact tracing can only catch so many cases, so scientists rely on estimates and models to make a best approximation of the real effects.

“You cannot contact trace everybody,” said study co-author Andrew Friedson, an assistant professor of economics at the University of Colorado, Denver.

Using anonymized cell phone data, Friedson and his fellow researchers analyzed the flow of people in, out and around the surrounding area. They estimated that roughly 90% of attendees came from outside of South Dakota.

They then used coronavirus case data and statistical methods to compare the COVID-19 trends between counties that sent people to the rally and weighted combinations of similar counties that didn’t, known in statistics as synthetic controls.

“These places are similar before, and then afterward they are different,” Friedson said. “And the main thing that is changing is their exposure to the rally.”

Statement by the Authors:

We stand by our entire body of rigorous scholarship on COVID-19, including studies of President Trump’s Tulsa Rally, Black Lives Matter protests, the Wisconsin Supreme Court stay-at-home ruling, and the Sturgis Motorcycle Rally. [1/1] pic.twitter.com/fgZnuy9P6T

— CHEPS (@SDSUCHEPS) September 9, 2020

Study co-author Dhaval Dave, a professor of economics at Bentley University, said 266,796 cases is “a ballpark estimate” of the number attributable to the rally.

The working paper’s estimates imply that there is 90% confidence that at least 100,000 infections could be linked back to the Sturgis rally, Dave said.

The exact number could be higher or lower than 266,796, Dave said. But that is not the key point of the paper.

“The study has two main takeaway points,” Dave said. “First, it shows that an event such as the Sturgis Rally can impose substantial costs in terms of COVID spread. Second, it shows that these public health costs may not be constrained to just the communities hosting the event.”

The authors used similar methods to analyze other large-scale events, including Black Lives Matter protests and President Donald Trump’s June rally in Tulsa, Okla. They found that neither event triggered a significant spike in case counts.

What early critics have said

Health economists and epidemiologists told PolitiFact that while the researchers used standard methods, the publicity their paper has received before entering the peer review process has subjected it to more scrutiny than such preliminary studies typically receive.

South Dakota officials, for example, questioned the paper’s methodology and assumptions. “The results do not align with what we know for the impacts of the rally among attendees in the state of South Dakota,” said state epidemiologist Josh Clayton.

Other academic experts also chimed in publicly. Jennifer Beam Dowd, a University of Oxford epidemiologist, wrote in an article for Slate that “the authors don’t outline what transmission on this scale would have to look like to reach 266,796 infections.” Their paper, she said, “relies on strong assumptions that rarely hold in the real world.”

Experts we spoke to agreed that the 266,796 figure is likely an overestimate.

“After only a few weeks, it’s unlikely to be as high as what this paper is estimating,” said Ashley O’Donoghue, an economist at the Center for Healthcare Delivery Science within Boston’s Beth Israel Deaconess Medical Center.

Much of the criticism comes down to the selection of counties for the synthetic control groups, which O’Donoghue said could be improved by additional “robustness checks.”

The peer-review process could refine the paper’s model and estimate, experts said. “Any referee will make the authors show a bunch of alternate specifications, and surely some will have very different point estimates than 266,000,” said University of Kentucky economics professor Aaron Yelowitz.

Friedson said he and his co-authors tried to match counties that were similar in all aspects except for the number of residents who went to the rally.

But it’s difficult to account for all the differences between counties, experts said, making it a tall task to isolate the rally as the definitive cause of any one case.

“All models rely on assumptions, but this report appears to ascribe all COVID-19 cases in the home areas of Sturgis attendees to the motorcycle rally attendees without any data to evaluate this,” said Derek Chapman, an epidemiologist at Virginia Commonwealth University.

The paper doesn’t rule out, for example, that the attendees may have been risk-takers who were disproportionately likely to get the virus elsewhere, said Andrew Noymer, an associate professor of population health and disease prevention at the University of California, Irvine.

The study’s authors responded to those who said their estimate seems implausibly high by noting that the estimated 266,796 cases would have been spread across hundreds of counties and may not have shown up as spikes in any one county’s raw coronavirus data.

But Noymer said the effect size still seems too large.

“Let’s do a thought experiment,” Noymer added. “The thought experiment is, the Sturgis rally never happened. Would there be 20% fewer cases of COVID in the United States? I think the answer is no. And the reason is because you’re assuming that these people wouldn’t have gotten it otherwise in a nation in which COVID is burning everywhere.”

This article was originally published by PolitiFact, which is owned by the Poynter Institute, and is republished here with permission. See the sources for these facts checks here and more of their fact-checks here.