Why do phase III trials of promising heart failure drugs often fail? the contribution of “regression to the truth”

doi:10.1054/S1071-9164(03)00018-6

Journal of Cardiac Failure

Volume 9, Issue 5, October 2003, Pages 364-367

https://doi.org/10.1054/S1071-9164(03)00018-6 Get rights and content

Abstract

There has been considerable recent disappointment with the failure of a number of major new pharmacological strategies for the treatment of chronic heart failure. In turn, there has been much speculation as to why trials of these therapies have not shown benefit. Among a number of plausible and scientifically valid reasons, consideration should be afforded to the potential contribution of “regression to the truth.”

Regression to the truth derives from the biological concept of regression to the mean, whereby random fluctuations in a biological variable occur over time, such that the true value of the variable is approached with repeated measurements. This same concept can be applied to clinical trial programs for new drugs for heart failure. Because only strongly positive trials generally go on to phase III testing, and some of these early phase studies are positive by chance alone, on retesting in phase III the results are very likely not be as strongly positive. Numerous examples of regression to the truth apply for trials of heart failure therapies, as well as in other areas.

A major concern is how to minimize negative outcomes in phase III trials. One approach is to perform major rigorous phase II testing. Alternatively, avoidance of phase II testing will minimize “regression to the truth” because there are no data in phase II from which regression might occur. However, this approach does not obviate the need for an evaluation process in the selection of candidate agents (and their appropriate dose) in order to proceed to definitive testing.

Section snippets

What is regression to the truth?

To understand regression to the truth we must first consider the concept of regression to the mean. This concept derives from the random fluctuations that can occur in a variable over time. As a consequence, a single measurement of that variable more often yields a value removed from the mean, and the “true” value of the variable is approached with repeated measurements. As a corollary, in population studies, a single measurement of the dependent variable—for example, cholesterol—can lead to an

“Regression to the truth” in heart failure

This concept is true, not just of heart failure trials, but of any drug therapy for any specific indication. What exacerbates the problem in the setting of chronic heart failure is the low percentage strike rate in the development of successful pharmacologic therapies for this condition. Only renin-angiotensin and β-adrenoceptor blocking agents have come to the market over the last 30 years or so.

Therefore, very few promising drugs in early phase would be positive in phase III (if tested) and

“Regression to the truth” in clinical trials

Does “regression to the truth” occur in the real world of clinical trials? There are a large number of recent examples in which the concept may be operative within programs assessing the potential of new therapies for heart failure. These include both failed phase III trials after positive early-phase programs, as well as further testing after subgroup analysis of major trials. Examples of phase II studies that have gone on to be neutral or negative on phase III testing include the vesnarinone

Conclusions

Regression to the mean is a well-accepted concept in the measurement of biologic variables (eg, plasma norepinephrine), but we should not forget that “regression to the truth” occurs in clinical research programs. Therefore, in addition to the many plausible explanations for the failure of recent phase III heart failure trials, we should consider the concept that “regression to the truth” may be contributing to these disappointing outcomes.

References (18)

P. Kalra et al.
Do results of the ENABLE (Endothelin Antagonist Bosentan for Lowering Cardiac Events in Heart Failure) study spell the end for non-selective endothelin antagonism in heart failure?
Int J Cardiol
(2002)
J.L. Rouleau et al.
Comparison of vasopeptidase inhibitor, omapatrilat, and lisinopril on exercise tolerance and morbidity in patients with heart failure: IMPRESS randomised trial
Lancet
(2000)
B. Pitt et al.
Randomised trial of losartan versus captopril in patients over 65 with heart failure (Evaluation of Losartan in the Elderly Study, ELITE)
Lancet
(1997)
B. Pitt et al.
Effect of losartan compared with captopril on mortality in patients with symptomatic heart failure: randomised trial—the Losartan Heart Failure Survival Study ELITE II
Lancet
(2000)
S.S. Gottlieb et al.
Sustained hemodynamic response to flosequinan in patients with heart failure receiving angiotensin-converting enzyme inhibitors
J Am Coll Cardiol
(1993)
H. Wedel et al.
Challenges of subgroup analyses in multinational clinical trials: experiences from the MERIT-HF trial
Am Heart J
(2001)
M. Packer
Current perspectives on the design of phase II trials of new drugs for the treatment of heart failure
Am Heart J
(2000)
A. Coletta et al.
Clinical trials update: RENEWAL (RENAISSANCE and RECOVER) and ATTACH
Eur J Heart Fail
(2002)
M. Packer et al.
Comparison of omapatrilat and enalapril in patients with chronic heart failure: the Omapatrilat Versus Enalapril Randomized Trial of Utility in Reducing Events (OVERTURE)
Circulation
(2002)

There are more references available in the full text version of this article.

Cited by (26)

The Rollercoaster of Paclitaxel in the Lower Limbs and Skeletons in the Closet: An Opinion Review
2021, Journal of Vascular and Interventional Radiology
Citation Excerpt :
For example, during the coronavirus disease 2019 pandemic, early data from China suggested a 3.4% case fatality rate, whereas a few months later, the median infection fatality rate across 51 locations worldwide was estimated to be 0.23% (46). However, this regression 'to the truth' does not necessarily negate the originally discovered treatment effect (47). In fact, it is a reasonable mathematical phenomenon which is observed as the sample size increases and the treatments are applied to a more heterogeneous spectrum of patients (45).
There have been concerns about the long-term risk of all-cause death with the use of paclitaxel-coated devices in the lower limbs. Results from a 2018 meta-analysis were corroborated by the US Food and Drug Administration and later confirmed by an individual patient data meta-analysis. However, population-based observational studies have produced contradictory results and often claimed a survival benefit with the use of paclitaxel. The recently published Swedish drug-elution trial in peripheral arterial disease did not confirm a significant mortality risk. In this review, the authors discuss the key elements of the identified mortality signal and stress important facts and figures that remain underrecognized and elusive. They also highlight the important types of epidemiological bias that pertain to the ongoing debate on paclitaxel.
Limitations of Randomized Clinical Trials
2020, American Journal of Cardiology
Citation Excerpt :
Early trials report very significant usually positive findings whereas the average effect is observed when many reports are published. Regression to the truth derives from the biological concept of regression to the mean, whereby random fluctuations in a biological variable occur over time, such that the true value of the variable is approached with repeated measurements.13 Clinical trial participants are usually healthier than the average population suffering with the condition under study because they are health conscious with better life-style.
Randomized clinical trials are essential for determining the efficacy of interventions, but have limitations. The types of limitations discussed in this review may be grouped in 11 categories including incorrect statistical inference, low internal or external validity, misinterpretation of the difference between frequentist and Bayesian statistical approaches, publication bias, that healthy persons with a given condition participate in clinical trials although they are not representative of the population as a whole, the rather short duration (3 to 5 years) that does not give correct estimates of the lifetime effects of the interventions or the legacy effect when participants who receive active therapy derive residual benefit after the end of the study when all participants receive active medication and the tension between the generalizability of the evidence versus the reliability of the findings of different types of clinical trials and the difficulty in applying the findings of randomized clinical trials to individual patients. These limitations are described and illustrated by examples and figures from the literature. In conclusion, this review will be useful to clinical trialists, clinical trial participants and regulatory agents.
Lies, damned lies and statistics: Clinical importance versus statistical significance in research
2018, Paediatric Respiratory Reviews
Citation Excerpt :
Unfortunately, these results are also likely to be subsequently proven to be either incorrect, or at best, to have a highly inflated effect size. This is what has been termed “regression to the truth”! [40]. Because of the well-known high risk of error in the initial study, it is wise to wait for replication – in a separate setting, and ideally in a study that has a total of at least 300 (patient important) events [41].
Correctly performed and interpreted statistics play a crucial role for both those who ‘produce’ clinical research, and for those who ‘consume’ this research. Unfortunately, however, there are many misunderstandings and misinterpretations of statistics by both groups. In particular, there is a widespread lack of appreciation for the severe limitations with p values. This is a particular problem with small sample sizes and low event rates - common features of many published clinical trials. These issues have resulted in increasing numbers of false positive clinical trials (false ‘discoveries’), and the well-publicised inability to replicate many of the findings. While chance clearly plays a role in these errors, many more are due to either poorly performed or badly misinterpreted statistics. Consequently, it is essential that whenever p values appear, these need be accompanied by both 95% confidence limits and effect sizes. These will enable readers to immediately assess the plausible range of results, and whether or not the effect is clinically meaningful.
Small studies are more heterogeneous than large ones: A meta-meta-analysis
2015, Journal of Clinical Epidemiology
Citation Excerpt :
However, there are several reasons why small trials may also have a different level of heterogeneity than larger trials. Only after promising results of the initial, small exploratory trials, larger trials tend to be conducted, possibly in different patient populations [4,12,37]. On the other hand, small trials may suffer from lower-quality standards and show a diminished effect [38].
Between-study heterogeneity plays an important role in random-effects models for meta-analysis. Most clinical trials are small, and small trials are often associated with larger effect sizes. We empirically evaluated whether there is also a relationship between trial size and heterogeneity (τ).
We selected the first meta-analysis per intervention review of the Cochrane Database of Systematic Reviews Issues 2009–2013 with a dichotomous (n = 2,009) or continuous (n = 1,254) outcome. The association between estimated τ and trial size was evaluated across meta-analyses using regression and within meta-analyses using a Bayesian approach. Small trials were predefined as those having standard errors (SEs) over 0.2 standardized effects.
Most meta-analyses were based on few (median 4) trials. Within the same meta-analysis, the small study τ_S² was larger than the large-study τ_L² [average ratio 2.11; 95% credible interval (1.05, 3.87) for dichotomous and 3.11 (2.00, 4.78) for continuous meta-analyses]. The imprecision of τ_S was larger than of τ_L: median SE 0.39 vs. 0.20 for dichotomous and 0.22 vs. 0.13 for continuous small-study and large-study meta-analyses.
Heterogeneity between small studies is larger than between larger studies. The large imprecision with which τ is estimated in a typical small-studies' meta-analysis is another reason for concern, and sensitivity analyses are recommended.
Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects
2011, Journal of Clinical Epidemiology
Citation Excerpt :
However, when one focuses on meta-analyses with nominally statistically significant results, the effect sizes are, on average, inflated. The winner’s curse phenomenon that has been pinpointed in very diverse disciplines [13,14,24,33–49], including also in single randomized trials, stopped early for perceived effectiveness [11,35,37,49]. The winner’s curse is a form of regression to the mean.
To assess whether nominally statistically significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.
Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and four or more trials in 2005 with P < 0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.
Four hundred sixty-one meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005–2010) than in the respective meta-analyses in 2005 (median 0.85-fold, interquartile range [IQR]: 0.66–1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67-fold, IQR: 0.54–0.96). Mean credibility of the 461 meta-analyses in 2005 was 63–84% depending on the assumptions made. Credibility estimates changed >20% in 19–31 (24–39%) of the 80 updated meta-analyses.
Most meta-analyses with nominally significant results pertain to truly nonnull effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.
Novel Therapies in Childhood Heart Failure: Today and Tomorrow
2010, Heart Failure Clinics
Citation Excerpt :
Why has it been so difficult to produce effective medical treatments for heart failure? There are undoubtedly other reasons for the paucity of efficacious new heart failure therapies,207 but one important problem has been neglect of the systemwide effects of proposed therapeutic modalities. Early physicians dosed their patients with naturally available substances and recorded the results.

View all citing articles on Scopus

^⋆: The ideas and opinions expressed in Perspective articles in the Journal of Cardiac Failure are those of the authors and do not necessarily reflect those of the Editor, the Publisher, the Heart Failure Society of America, or the Japanese Heart Failure Society.

View full text

PerspectiveWhy do phase III trials of promising heart failure drugs often fail? the contribution of “regression to the truth”⋆

Abstract

Section snippets

What is regression to the truth?

“Regression to the truth” in heart failure

“Regression to the truth” in clinical trials

Conclusions

Int J Cardiol

Lancet

Lancet

Lancet

J Am Coll Cardiol

Am Heart J

Am Heart J

Clinical trials update: RENEWAL (RENAISSANCE and RECOVER) and ATTACH

Eur J Heart Fail

Comparison of omapatrilat and enalapril in patients with chronic heart failure: the Omapatrilat Versus Enalapril Randomized Trial of Utility in Reducing Events (OVERTURE)

Circulation

Perspective
Why do phase III trials of promising heart failure drugs often fail? the contribution of “regression to the truth”⋆