On Monday, 15 August 2011, the Food and Drug Administration's (FDA or "the Agency") Center for Devices and Radiological Health (CDRH) issued two important draft guidance documents, one regarding the factors that the Agency will consider when making benefit-risk determinations during the premarket review process, and one discussing design considerations for pivotal studies of medical devices. Each of these guidance documents are discussed below.
Benefit-risk guidance document
The draft guidance, titled Draft Guidance for Industry and FDA Staff – Factors to Consider when Making Benefit-Risk Determinations in Medical Device Premarket Review (Benefit-Risk Draft Guidance), provides a framework for how benefit-risk determinations are made by the FDA during the premarket review process. The guidance is intended to apply to devices subject to premarket approval (PMA) applications, and in limited cases, devices subject to 510(k) premarket notification requirements.
In making a determination of whether a PMA application has demonstrated a "reasonable assurance of safety and effectiveness," the Agency will weigh "any probable benefit to health from the use of the device against any probable risk of injury or illness from such use".1 In making such a determination, the FDA will review valid scientific evidence, including non-clinical testing and clinical testing, if appropriate. The Agency will review the data and determine based on the factors outlined in this draft guidance if the benefits of the device outweigh the risks. This process also may be applied to the review of 510(k) premarket notifications when there are differences between the device that is the subject of the 510(k) notification and the identified predicate device that may adversely affect safety and/or effectiveness of the device for which a sponsor is seeking clearance.
The factors that the Agency will consider are: (1) effectiveness, which is used to determine the extent of probable benefit; (2) safety, which is used to determine the extent of probable risk; and (3) additional factors, such as the severity of the disease the device is intended to diagnose or treat, and whether or not alternative tests or treatments are available.
With respect to effectiveness, the Agency will consider the following factors:
- Type of benefit;
- Magnitude of the benefit;
- Probability of the patient experiencing a benefit; and
- Duration of the effect.
With respect to safety, the Agency will consider the following:
- Number, severity, and types of harmful events associated with the use of the device (including device-related serious adverse events, device-related non-serious adverse events, and procedure-related or indirect harms);
- Probability of a harmful event;
- Duration of harmful events;
- Risk from false-positive or false-negative for diagnostics; and
- Number of different types of harmful events that could result and the severity of aggregated effect.
As indicated above, additional factors that the Agency will consider in weighing probable benefits and risks include the following:
- Degree of certainty of the benefits and risks of a device;
- Characterization of the disease, such as its clinical manifestation, how it affects patients, the condition's natural history and progression;
- Patient tolerance for risk, which may be influenced by disease severity, disease chronicity, and availability of alternative treatment/diagnostic options;
- Availability of alternative treatment/diagnostic options, including non-device products, off-label uses, and how well the alternatives address the needs. The Agency stated that "if a new device has a very small benefit and there is significant uncertainty about that benefit, [the FDA] may still approve the product if there are no available alternative treatments or diagnostics and the risk profile is acceptable;"
- Risk mitigation, including warnings in labeling or restricted indications for use; and
- Novelty of technology.
The factors outlined in the Benefit-Risk Draft Guidance, while perhaps not novel, provide a methodical presentation of the factors that the FDA will consider in evaluating benefit-risk determinations for premarket submissions. The Agency included a proposed draft worksheet outlining the factors presented in the guidance document, which would be used by reviewers in making benefit-risk determinations. The FDA proposes that such a worksheet will help maintain consistency of review across different divisions within CDRH and ensure that an appropriate decision is reached.
The FDA also provides several examples of benefit-risk determinations to illustrate how the Agency would consider the factors outlined in the proposed guidance document.
One such example cited by the FDA involved the Agency's experience with an implanted cardiovascular monitoring device intended to diagnose heart failure. The manufacturer showed that the device can reduce hospitalization as a result of heart-failure by approximately three days, but the implantation procedure for the device required that patients be hospitalized for two days, whereas similar devices on the market did not require a similar implantation procedure. Thus, the FDA determined that even though the device was effective and relatively safe, the benefit of reducing hospitalization by one day for such patients did not outweigh the risk of implant surgery complications.
Pivotal study design guidance document
The second guidance, titled Design Considerations for Pivotal Clinical Investigations for Medical Devices, outlines various factors to be considered in designing and executing pivotal clinical studies, primarily for devices subject to PMA approval; however, the guidance indicates that the recommendations may also be used in designing studies to support 510(k) notifications.
First, the agency describes considerations unique to devices, as opposed to other regulated medical products, that are relevant for the clinical study design process:
- How and why the device works;
- User skill level and training; and
- Learning curve.
While not describing this phase in detail, the guidance emphasizes the importance of first-in-man and feasibility studies, as well as post-market studies in evaluating a device throughout its life cycle. The Agency states that exploratory studies (non-clinical and clinical) are important in order to elucidate the mechanism of action of the device, provide safety information, and estimate the performance of the device for designing the pivotal study. The agency warns that inadequate exploratory work can lead to issues being identified during the pivotal phase that might require new studies to address.
The guidance then discusses principles for choosing an appropriate study design for a pivotal trial of a medical device, including the following:
For all types of devices:
- Bias and variability in device performance;
- Study objectives;
- Subject selection;
- Site selection; and
- Comparative study designs.
For clinical outcome studies:
- Intervention assignment (randomization);
- Masking (blinding);
- Controls selected; and
- Placebo effect.
In the discussion regarding choosing study endpoints, the guidance also addresses a more recent phenomenon that is of great concern to medical device companies. Specifically, the situation where a company has negotiated an agreeable study design with the Agency, executes the study according to plan, and obtains a successful outcome, only to be asked for an additional clinical study by the Agency because some element of the study design may now be considered outdated (for example, the primary endpoint has fallen out of favor in the scientific community). The guidance encourages companies to contact the FDA to "discuss the best possible course of action" in the event that "the understanding of science or medicine changes during the course of a particular device study." While the guidance does not state that new studies will necessarily be needed, this scenario is particularly concerning because many pivotal clinical studies span years from planning through completion, and controlling for future changes in the practice of medicine is virtually impossible.
Also in the above discussion of factors to consider, as well as in a section titled Advantages and Disadvantages of Some Clinical Outcome Studies, the Agency makes clear that randomized, double-masked, controlled, parallel group clinical studies are highly preferred, and that if a sponsor chooses to deviate from this model, a strong justification will need to be provided. With regard to randomization, the guidance states: "when the design of a device or the intended subject population makes it impossible to randomize the intervention assignment, the study may be subject to bias of unknown size and direction, and such bias can adversely impact the level of evidence provided by the study and the ability to rely on the data as valid." In the guidance, the FDA encourages use of concurrent controls, in particular either an active control or a placebo control, and warns that placebo effects can be observed with both objective and subjective endpoints, and even for a period of years. With regard to masking, the Agency states that "if study participants are not masked, it is very difficult to assess the size of the resulting bias, and it can threaten the scientific validity of an otherwise solid study, even when a truly objective endpoint is used." The guidance also strongly discourages the use of single-group studies or studies using an Objective Performance Criterion (OPC) or historical controls.
Finally, the guidance also emphasizes that the conduct and analysis of pivotal clinical studies is critical to "sustaining the level of evidence" of such studies based on an appropriate study design. The Agency encourages use of a data management plan to ensure reliable and accurate data are collected and submitted. The guidance also emphasizes a number of principles of good study conduct, including minimizing protocol deviations (even those considered minor), minimizing loss to follow-up, proper monitoring of studies, and obtaining investigator buy-in during protocol development to ensure they follow the protocol once the study has been initiated. In terms of analysis, the Agency encourages the use of a well-designed Statistical Analysis Plan (SAP), and discourages performing post-hoc analyses, as they "can inflate the experiment-wise type I error rate and endanger the scientific validity of an otherwise well-designed and well-conducted study." Finally, the Agency notes that adaptations in the study design may be considered (for example adjustment of the sample size based on an interim analysis), as long as they are pre-planned.
In conclusion, while many of the principles described in the guidance are commonly understood to be part of Good Clinical Practice (GCP), and it is not surprising that the FDA indicates a strong preference for randomized, double-masked, controlled, parallel group clinical studies, the tone of the guidance indicates that it will be very difficult going forward to obtain Agency buy-in for alternative study designs that may have been accepted in the past.
The FDA has opened public dockets for comments on these draft guidances and will be accepting comments until 14 November 2011. Electronic comments on Docket No. FDA-2011-D-0215 (Risk-Benefit) or FDA-2011-D-0567 (Pivotal Trial Design) can be submitted at www.regulations.gov. Written comments can be submitted to the Division of Dockets Management (HFA-305), Food and Drug Administration, 5630 Fishers Lane, Room 1061, Rockville, MD 20852.