On May 18, 2015, the Food and Drug Administration (FDA) released a draft guidance document, Adaptive Designs for Medical Device Clinical Studies, discussing considerations related to the design and implementation of adaptive designs in medical device clinical studies used to support Premarket Approval Applications (PMA), premarket notification (510(k)) submissions, de novo submissions, Humanitarian Device Exemption (HDE) applications, and Investigational Device Exemption (IDE) submissions. The principal FDA contact for questions about this draft guidance is Greg Campbell, PhD, Director of CDRH’s Division of Biostatistics, signifying the importance of this document.
The draft does not seem to carve new ground in the field of adaptive designs but rather represents a compendium of the types of adaptive designs that are considered appropriate by FDA. Although aspects of adaptive designs have been discussed in other FDA statistical guidance, this is the first guidance devoted to adaptive designs in medical device clinical trials. The draft guidance defines adaptive designs, sets forth various types of adaptive designs, indicates when such designs are appropriate, and lays out some of FDA’s concerns about such designs.
An adaptive design for a medical device clinical study is defined as a clinical trial design that allows for prospectively planned modifications based on accumulating study data without undermining the trial’s integrity and validity. While the guidance could be applied to all stages of clinical trial development (from feasibility to pivotal studies), it is clearly most useful in the design of pivotal studies. Adaptive designs are most useful when there are uncertainties about certain aspects of the study design, such as the effect size or even whether the study as designed will be successful. For fixed (or non-adaptive) designs, sample size calculations are based on assumed values of certain parameters but the sponsor may not have confidence in the choice of those parameter values (possibly leading to an under- or over-powered study). The decision whether to employ an adaptive versus a fixed design may depend on a sponsor’s confidence in their ability to estimate certain study parameters and their willingness to possibly risk a failed study when using a fixed study design.
As noted in the guidance, if planned and implemented correctly, an adaptive study design may enable timelier device development decision-making, and therefore more efficient investment in resources in a clinical study. The agency lists several other advantages to adaptive designs, including that they can improve the chances of trial success using sample size reassessment, they can facilitate the transition from premarket to post-market patient follow-up, and they can enhance patient protection by increasing the likelihood that patients are allocated to a treatment with a better outcome. Limitations of adaptive designs include that they require more effort at the design stage, they can introduce bias, or they can confound the interpretation of the study results.
In the draft guidance, FDA details how to determine the appropriate design (fixed or adaptive) for a clinical study. The agency recommends that the sponsor analyze a range of realistic scenarios, gauge how likely each scenario is, calculate the chance of success, average size of the study, and statistical aspects of the study (Type I error and power) and contrast them with the same characteristics of a fixed design study using analytical calculations or computer simulations. Furthermore, study designs that optimally benefit from an adaptive design are those in which the time to the primary endpoint is long, but the expected accrual period is even longer. The guidance discusses the concept of “anticipated regret” where a sponsor can, as a form of “insurance,” anticipate what might go wrong in a study (e.g., barely missing the primary endpoint) and build in an adaptive design that increases the chances of success (e.g., adaptive sample size).
While the agency stresses in the guidance that modifications should be prospectively planned and described in the protocol to ensure study integrity, there are also limited situations where scientifically valid modifications can be made after the trial begins, for example, if the results by treatment group (including coded treatment groups (e.g., A and B)) are not known. Importantly, analyzing aggregate (“pooled”) data or baseline demographics at a pre-specified interim time point does not “break the blind” as long as outcomes by subject or treatment group remain masked. Nevertheless, FDA has clearly stated its strong preference for sponsors to prospectively specify possible changes to a pivotal study design in the statistical analysis plan prior to the initiation of the study.
The guidance is consistent with our experience in interactions with the agency regarding adaptive designs over the past several years. Namely, that FDA is willing to consider various types of adaptive designs. Moreover, FDA expects that Type I error and adequate power need to be controlled and pre-specified in the design phase and that operational bias should be minimized through pre-specified blinding of the appropriate parties. In addition, the guidance warns that the homogeneity of the results before and after the modification should be considered.
The guidance outlines several adaptive designs that are based on accumulating, un-blinded data and accompanied by pre-specified, strict statistical analyses. The agency notes that the three designs described below are the most widely used.
Click here to view the table.
Thus, the guidance covers the classic purposes for adaptive designs, namely, early stopping of a very successful study, an increase in sample size when the effect size was unable to be accurately estimated during the study design, Bayesian sample size adjustments, and changing between non-inferiority and superiority hypotheses. It also covers various other options, including “seamless studies.” Seamless studies, namely ones where there is a smooth transition from a feasibility study into a pivotal study in a preplanned manner, are often of substantial interest to sponsors trying to conserve time and resources in the development process.
Operational bias is a major concern in adaptive designs. It can arise when some participants in the study have access to certain study results and where this information can have the potential to influence the ongoing operations or conduct of the study. The need to minimize operational bias in adaptive trials is prominently mentioned in the draft guidance. Generally, FDA encourages sponsors to limit accessibility to outcomes by treatment group and to set up firewalls that guarantee that data from interim analyses are restricted to those with an essential need to know. In this regard, the guidance also provides useful information about the roles of Data Monitoring Committees (DMCs), Institutional Review Boards (IRBs) and independent statisticians in clinical studies with adaptive designs.
The guidance strongly encourages sponsors to consult with FDA about the various types of adaptations in a pre-submission meeting and notes that the likelihood of success is increased through interactions with the relevant FDA review division and statistical staff during the study planning phase. While mentioning some of the practical challenges associated with interpreting the results obtained from adaptive studies, the focus of the guidance is on the actual design of the studies. We encourage sponsors to also consider the practical aspects of adaptive studies, including when interim analyses will be conducted and what information will be available, the totality of study data when considering early stopping, and the interpretation of adaptive trial results by FDA or an advisory committee.
This draft guidance publicly signals FDA’s willingness to consider a variety of adaptive designs and details how FDA would like for a sponsor to interact with the agency during the development of such studies, the limitations of such studies, and certain pitfalls that must be considered. We believe this guidance will serve as a valuable resource that should be carefully considered by medical device sponsors when designing pivotal clinical studies.
The public comment period for the draft guidance will remain open until August 17, 2015, 90 days from publication in the Federal Register. Electronic comments should be submitted to http://www.regulations.govreferencing Docket No, FDA-2015 - D-1439.