The beginning of the end, or the end of the beginning?
Written by Mark Freeston
On 10th November 2020 BBC news reported that ‘life could be back to normal by spring’ due to the development of a vaccine described as ‘90% effective’. I was reminded of a quote often attributed to Winston Churchill: ‘Now this is not the end. It is not even the beginning of the end, but it is, perhaps, the end of the beginning.’ He was talking about a first victory (el Alamein, 1942) after a long series of costly and near total defeats during WWII. The quote neatly summarises how the victory was not final, but in fact opened up a bigger process.
On hearing the vaccine news, the first thing that struck me was ‘what does 90% effective mean?’ Does this mean fewer deaths? Fewer hospitalizations? Fewer complications? Less severe symptoms? Shorter illness? No symptoms? No transmission?
My second thought was ‘what are the confidence intervals?’ Especially when the data is viewed each time there are 100 or so cases (predetermined by protocol) in a study of tens of thousands of people. The 90% effectiveness is based on low numbers. This is a problem with Covid-19; although it has a massive impact, it fortunately has a fairly low base rate in the UK and other European countries (tens to low 100s of cases per 100,000 people depending on which area) and countries are doing a half decent job at controlling the spread. So despite 45,000 people or so in the trial, the 90% effectiveness is coming from the small number (N = 94) who have so far tested positive for COVID. So, how many people had understood that the 90% claim was based on 94 people? Unless vaccines are given and then people deliberately exposed in challenge trials, the next few claims will also be based on very low numbers. Small numbers generally lead to unstable point estimates. The precision of this estimate is probably +/- 6%, but it doesn’t tell us what the likely range of the next results will be, Ie. prediction, with a degree of confidence. Apparently, vaccine researchers are often working in the 40-60% range. So, 90% is unusual, and could be a miracle, but...
Am I happy about the announcement? Of course I am. THIS IS GOOD NEWS. Much better and much sooner than I expected; other vaccines will probably prove ‘effective’ too. From a scientific standpoint, this is an absolute triumph. There has been an unprecedented drive, commitment and flourishing of science in 2020; some amazing science. We have seen science at its best, and also some dodgy science from scientists under time pressure, poor contingencies in the science “economy”, etc.
I have heard the news that stocks in pharmaceuticals are up, as are stocks in areas that have been badly affected such as cinemas. There is the sentiment that we can get the economy going again which is not so good for the planet and climate change if this is the oil economy. Stocks in Zoom, home delivery and anything that has done well in the pandemic are down. The stock market appears to be telling us that the pandemic is over! However, this is based on initial results in the vaccine trial from cases in the low hundreds at best.
I do not doubt the science or the rigour of this trial. I am not a conspiracy theorist. The data is what the data is: a single point estimate on small numbers. We need bigger samples, replications, different settings, etc. until we can have a backward-looking good estimate of the precision of the findings to date. We will then know the degree of heterogeneity, and what may influence results. When we know enough about the precision, we can have good-enough forward-looking predictions, assuming that the future will essentially be as variable as the past. Even if we have narrowing precision intervals, we will still have much wider prediction intervals. This is simple evidence-based medicine or cumulative science; systematic review and meta-analysis. Or we could go Bayesian…
Assuming nothing changes, is it possible to predict how effective any of the vaccines will be in the future? Novel settings need to be considered, along with how the as yet unaffected population may change over time, how the virus may change, and how long immunity may last. There is also the issue of the minks in Denmark . . . .
We have not factored in (despite all the planning and projections) how to manufacture and vaccinate the entire world and how long that will take. In the UK as of 10th November, I am finding it impossible to get an annual flu vaccination, and this programme has been running for years.
The point of uncertainty has moved away from, will there ever be an effective vaccine? We now know the answer is probably yes, but it may not be 90% effective. There may be several successful vaccines of differential “effectiveness” and “deliverability”. Uncertainty has moved to issues such as: When will the vaccine become available? When will I get one? Will it work for me? How long will it last for? Are there any risks? When will restaurants open again and under what conditions?
There may be many more understandable questions: When can I do what I want again? Why are we not getting the vaccine sooner? Why are other countries apparently doing better than us? Why them and not me? Why is life not back to normal yet? When will the pandemic be over? Why is the pandemic not over?
There may be not completely unreasonable, but suspicious, questions that will inevitably arise about what went wrong during this pandemic: Who is to blame? Who should be held to account? Why are people making money? What are the real motivations behind a given policy/decision, etc.?
There may be some questions that seek very simple explanations to very big and complex issues: Is this not further proof of some masterplan by … ?
Greater certainty about any single thing moves the point of uncertainty. Although we have some certainty about one aspect, we are now uncertain about many more different things. We need and we will get more good news; but it will be interspersed with some bad news. I firmly believe the ratio of good to bad news will change favourably over time; because of the science, the experts, the behaviour of the general population, and even because of the politicians dealing with unprecedented uncertainty. We are all caught between trying to get things right and trying to avoid getting things wrong in various ways.
The one thing that will not change in the foreseeable future is that we will continue to live with a high degree of uncertainty, just different uncertainties.
Take part in our research here
You can follow Unicorn on our website or on Research Gate
Email us on intolerance.uncertainty@newcastle.ac.uk
Tweet us @Covid19Study
