In 2014, I participated in a group led by an American statistician adept at Bayesian statistics whose company had been commissioned by a drug company to predict what increment of HER-2-positive breast cancer patients achieving a complete pathological cancer response (pCR) with HER-2 directed neo-adjuvant chemotherapy (chemotherapy given before surgery) would be required to show a statistically significant survival advantage (and thence gain FDA and HPB approval and mouth-watering profits). We used previously published trials data.
Our model predicted that around a 10% increase in pCR rates should be enough to predict enough of an increase in survival to convince the FDA and HPB to approve a new drug. I was an author on that paper. I could not follow the statistical manipulations but went along with the paper because my gut feeling was that the conclusion was correct and besides, I’d already cashed the cheque. That’s weak, but the traditional schoolboy last gasp excuse (which I have used frequently in this life) is: “I wasnae the only one, sir.”
I have support for this cowardly decision from Sharon McGrayne in her book The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy: Laplace built his probability theory on intuition: "…essentially, the theory of probability is nothing but good common sense reduced to mathematics. It provides an exact appreciation of what sound minds feel with a kind of instinct, frequently without being able to account for it.”
With the COVID-19 pandemic, everyone’s a Bayesian now – most citizens understand how we must “flatten the curve” and reduce the R rate to reduce pressure on ICU beds by physical distancing and hand cleansing. Vanishingly few understand how these curves are produced and with what flimsy scaffolding they are constructed – by necessity – since for this latest pandemic, initially at least, its natural history was unknown.
This amiable statistician said at one point in our meeting: “The Bayesian statistical approach is difficult in the sense that thinking is difficult.”
The others smiled or laughed knowingly. I took comfort in my colleagues thinking they understood more than they probably did. As one striking modeller, Elizabeth Hurley, model, actress and business woman, has said: “I would seriously question whether anybody is foolish enough to really say what they mean. Sometimes I think civilization would break down if we all were completely honest.”
One tiny part of our paper’s statistics section read:
“For analyses, the joint posterior distributions of model parameters are fit using Markov Chain Monte Carlo using the estimated number of events and exposure time by pCR group within each time segment. We sample values of model parameters from the joint posterior distributions using Gibbs sampling. When closed forms to full conditionals are unavailable Metropolis-Hastings is used. When studies provided only the total number of events across all time segments, we impute the segment-specific number of events after each iteration of the MCMC from a multinomial distribution using the current sampled segment-specific baseline hazard rates, λt, and the amount of exposure within each segment.”
I asked: “Is this the kind of modelling for predicting climate change?”
There was laughter from the statistics folk. “Yes, of course.”
The problem is that the fallibility of Bayesian modelling is not emphasized. Accuracy depends on accurate relative numerical weighting – you don’t really know what goes into it – given the host of unknown unknowns or even known unknowns. It’s like the roadside food-trucker who was asked how he could sell rabbit sandwiches so cheap. “Well,” he said, “I put some horse meat in too. But I mix them 50:50. One horse, one rabbit.”
A better word than “probability” might be “possibility.” Next time there will be more skepticism. But what we need is not more skepticism – there’s enough of that – but more understanding of the weaknesses of Bayesian epidemiological statistics, with the promise of rapid improvements as new data comes in.
Will the population be as credulous the next time, knowing the massive debt and the limited risk in less populated areas? I doubt it. Some trust in modelling has been lost. Some mathematical modellers fall prey to the attraction of celebrity and press attention. All models are fallacious, but all can be useful with caution.
What has not been lost is trust in the wisdom of distancing, hand cleansing and perhaps face masking.
Editor’s note: The views, perspectives and opinions in this article are solely the author’s and do not necessarily represent those of the AMA.
Banner image credit: Chuk Yong, Pixabay.com