At work I recently finalized analyses for a clinical trial that had 4 arms (double blind with 3 dose levels and a placebo) testing a nutritional supplement which may be helpful for cognition. There were measurements at baseline (before randomization) and 3 followup times with the 2nd followup being the primary endpoint of interest. I recommended, and we performed, a mixed effects model of change scores from baseline also controlling for the baseline outcome (as in ANCOVA), and also controlling for gender, age, and education all known to be predictors of the primary cognition outcome.
I don’t analyze nearly as many clinical trials as I do observational studies. I was curious whether the analyses I’ve done (in particular the control or not for baseline outcomes and other covariates) would pass the check boxes for an FDA or EMA submission, so I looked it up.
Apparently just this past April 2019, FDA put out guidance for how to deal with covariates in clinical trials. This guidance document is still open for public feedback and apparently is not meant to be the legally enforceable rules but just the general principles. It included just 5 points, much shorter than I expected. Here are there 5 points and my reactions (in bold):
- Sponsors can use ANCOVA to adjust for differences between treatment groups in relevant baseline variables to improve the power of significance tests and the precision of estimates of treatment effect. (looks like controlling for baseline factors is basically recommended but without specific guidelines how to choose them. I would say include ones that are predictive of outcomes- because they will help with precision- and include ones that show non-trivial imbalance at baseline- since they will help with bias).
- Sponsors should not use ANCOVA to adjust for variables that might be affected by treatment. (Basically do not control for anything you collect AFTER randomization)
- The sponsor should prospectively specify the covariates and the mathematical form of the model in the protocol or statistical analysis plan. When these specifications are unambiguous, FDA will not generally be concerned about the sensitivity of results to the choice of covariates because differences between adjusted estimators and unadjusted estimators of the same parameter, or between adjusted estimators using different models, are random. (Saying they are unconcerned is a bit strange, but I guess they mean as long as they are prespecified since they are assuming randomization will take care of everything being balanced, this has not always been my experience especially in smaller <50 per arm studies)
- Interaction of the treatment with covariates is important, but the presence of an interaction does not invalidate ANCOVA as a method of estimating and testing for an overall treatment effect, even if the interaction is not accounted for in the model. The prespecified primary model can include interaction terms if appropriate. However, interaction means that the treatment effect is different for different subjects, and this fact could be relevant to prescribers, patients, and other stakeholders. Therefore, even though a primary analysis showing an overall treatment effect remains valid, differential effects in subgroups can also be important. (Green light to look at subgroup analysis – but remember ATE average treatment effect is still meaningful)
- Many clinical trials use a change from baseline as the primary outcome measure. Even when the outcome is measured as a change from baseline, the baseline value can still be used advantageously as a covariates. (These 3 things need to be said – and learned – over and over again, 1) ANCOVA with change score as the outcome gives the exact same test for treatment as ANCOVA with post-test as the outcome 2) Analyzing the change score as the outcome without controlling for the baseline gives a different treatment effect estimate than ANCOVA whenever there is any imbalance in the baseline outcome by treatment (i.e. whenever the treatment assignment is correlated by chance with baseline outcome) 3) Many articles recommend always using ANCOVA in clinical trials as it is most likely to reduce bias due to imbalance and also will have more precision)
The EMA also has guidelines from 2015 and in my opinion they are much better, perhaps because they are finalized and are not still in the open public feedback stage, although I was surprised FDA didn’t already have something like this. EMA has 15 guidelines following a very reasonable preamble. My comments in bold:
Preamble from the EMA: Baseline covariates impact the outcome in many clinical trials. Although baseline adjustment is not always necessary, in case of a strong or moderate association between a baseline covariate(s) and the primary outcome measure, adjustment for such covariate(s) generally improves the efficiency of the analysis and avoids conditional bias from chance covariate imbalance.
Baseline covariates may be accounted for at the design stage of a clinical trial and/or in the statistical analysis. When dealing with baseline covariates the following recommendations are made:
- Stratification may be used to ensure balance of treatments across covariates; it may also be used for administrative reasons (e.g. block in the case of block randomisation). The factors that are the basis of stratification should normally be included as covariates or stratification variables in the primary outcome model, except where stratification was done purely for an administrative reason. (Recognizing the stratification variables are covariates is very sensible)
- Variables known a priori to be strongly, or at least moderately, associated with the primary outcome and/or variables for which there is a strong clinical rationale for such an association should also be considered as covariates in the primary analysis. The variables selected on this basis should be pre-specified in the protocol. (Same point made in the FDA list)
- Baseline imbalance observed post hoc should not be considered an appropriate reason for including a variable as a covariate in the primary analysis. However, conducting exploratory analyses including such variables when large baseline imbalances are observed might be helpful to assess the robustness of the primary analysis. (If there is a large imbalance on a baseline variable and it is also a strong predictor of the outcome, it seems to me it must be included as a covariate to avoid bias. However, I guess if it is a strong predictor of the outcome you should have already known that and included it in #2)
- Variables measured after randomisation and so potentially affected by the treatment should not be included as covariates in the primary analysis. (Same point made in the FDA list)
- If a baseline value of a continuous primary outcome measure is available, then this should usually be included as a covariate. This applies whether the primary outcome variable is defined as the ‘raw outcome’ or as the ‘change from baseline’. (Yes, they are saying…use ANCOVA)
- Covariates to be included in the primary analysis must be pre-specified in the protocol. (This is the same thing they said above in #2)
- Only a few covariates should be included in a primary analysis. Although larger data sets may support more covariates than smaller ones, justification for including each of the covariates should be provided. (It would be helpful if this had more specifics…what does ‘a few’ mean and perhaps some comment about whether there are any concerns about misspecification)
- In the absence of prior knowledge, a simple functional form (usually either linearity or categorising a continuous scale) should be assumed for the relationship between a continuous covariate and the outcome variable. (good point)
- The validity of model assumptions must be checked when assessing the results. This is particularly important for generalised linear or non-linear models where mis-specification could lead to incorrect estimates of the treatment effect. Even under ordinary linear models, some attention should be paid to the possible influence of extreme outlying values. (Check for outliers, always)
- Whenever adjusted analyses are presented, results of the treatment effect in subgroups formed by the covariates (appropriately categorised, if relevant) should be presented to enable an assessment of the model assumptions. (I have never done this but I think it is a very useful idea and worth trying though may need to be careful with smaller sample sizes or groups)
- Sensitivity analyses should be pre-planned and presented to investigate the robustness of the primary analysis. Discrepancies should be discussed and explained. In the presence of important differences that cannot be logically explained – for example, between the results of adjusted and unadjusted analyses – the interpretation of the trial could be seriously affected. (I like the emphasis on looking at the data in different ways.)
- The primary model should not include treatment by covariate interactions. If substantial interactions are expected a priori, the trial should be designed to allow separate estimates of the treatment effects in specific subgroups (In contrast to the FDA, this is encouraging researchers to stay away from subgroup analyses, remember treatment by covariate interactions is how we test for treatment effects in subgroups)
- Exploratory analyses may be carried out to improve the understanding of covariates not included in the primary analysis, and to help the sponsor with the ongoing development of the drug. (Keep learning from the data)
- In case of missing values in baseline covariates the principles for dealing with missing values as outlined e.g. in the Guideline on missing data in confirmatory clinical trials (EMA/CPMP/EWP/1776/99 Rev. 1) applies. (Need to go read this and put up a blog post for it)
- A primary analysis, unambiguously pre-specified in the protocol, correctly carried out and interpreted, should support the conclusions which are drawn from the trial. Since there may be a number of alternative valid analyses, results based on pre-specified analyses will carry most credibility. (Lay out your plan beforehand thoughtfully and follow it is a good recipe for credible and reproducible science)
An informative ream. I gained more knowledge on this topic. Thanks for sharing