On May 13, 2016, the U.S. Food and Drug Administration (FDA or the Agency) issued a new draft guidance entitled, “Infectious Disease Next Generation Sequencing Based Diagnostic Devices: Microbial Identification and Detection of Antimicrobial Resistance and Virulence Markers.” The draft guidance builds on a previous workshop held by the Agency on April 1, 20141 and outlines the Agency’s recommendations for studies to establish the analytical and clinical performance of such “Infectious Disease NGS Dx devices.” The release of the draft guidance is FDA’s latest move in the regulation of Next Generation Sequencing (NGS) diagnostic devices.
FDA Regulation of NGS-Based Diagnostic Devices
NGS refers to technologies which allow rapid sequencing of large segments of an individual’s DNA, potentially even the entire genome. NGS technology can be, and has been, used to diagnose a disease or condition or to predict the likelihood of developing a disease or condition (see for example, FDA Press Announcement) A single NGS test can identify thousands or millions of genetic differences (called variants) in the patient’s sample compared to a reference database, in contrast to traditional laboratory tests that identify a single or a limited number of preselected gene sequences. Because it allows for rapid identification of a person’s unique genetic makeup, including health-related genetic alterations, NGS is believed to be a key component of the Obama administration’s Precision Medicine Initiative.
In the infectious disease setting, NGS technology can be used to detect the presence of clinically important pathogenic organisms, as well as antimicrobial resistance and virulence markers in human specimens by comparing detected genetic variants in the organisms to reference databases. NGS has the potential to replace previous culture-based methods with molecular methods, which allows the simultaneous detection of multiple organisms or markers present in a sample during a test.
The capabilities of NGS tests and the rapid evolution in the field pose regulatory challenges for FDA, as NGS tests do not fit neatly into the existing regulatory framework for medical devices. Historically, FDA has assessed the safety and effectiveness of a given in vitro diagnostic (IVD) test by reviewing the analytical and clinical performance of the test for a particular intended use or indication for use. For analytical validity, FDA evaluates the specific performance characteristics including specificity, sensitivity, positive percent agreement, negative percent agreement, precision, and other metrics. Thus, when a test detects multiple analytes, FDA reviews performance data for each of those analytes, individually, to assess what each analyte contributes to a specific clinical determination. Unlike traditional IVDs, NGS tests have the capacity to detect an essentially unlimited number of variants, and consequently may have relatively broad indications for use. It is impractical for test developers to gather and for FDA to review performance data for each potential variant or analyte when a huge number of variants can be detected and their significance may not yet have been established.
For clinical performance, FDA typically requires the test developer to establish that the identified variant is clinically meaningful to a disease or condition the test is intended to diagnose. Because NGS tests can identify a larger number of mutations in a single test, it could be highly impracticable to demonstrate clinical effectiveness for each one. In addition, NGS tests are frequently used to detect novel and rare variants that occur in a small number of patients in the population. For those tests, it may not be feasible to run clinical trials to obtain evidence linking it to a particular disease.
FDA recognizes the challenges and has been exploring new regulatory approaches for NGS tests. In the past two years, FDA has held several workshops attempting to obtain feedback from the academic experts and clinical laboratories on the appropriate requirements for establishing the analytical and clinical performance for NGS-based diagnostic devices2. In those meetings, the Agency has proposed a standard-based approach for assessing analytical performance and a database approach for assessing clinical performance. The May 13 draft guidance is the first guidance document released by the Agency to implement these concepts into actual recommendations. While this draft guidance is specific to infectious disease diagnostic devices, it reflects the current thinking of FDA on this topic, and similar approaches and requirements are likely to be present in future guidances for other NGS-based tests.
The draft guidance is intended for Infectious Disease NGS Dx devices that use targeted or agnostic (metagenomic) sequencing approaches. Specifically, targeted sequencing refers to “preferential amplification of defined regions that target a specific organism(s) or marker(s) selected for analysis a priori by any lab or bioinformatics method (e.g., amplicon sequencing or a k-mer signature database) based on the diagnostic device’s intended use,” whereas agnostic sequencing “does not use a priori knowledge of sequence targets and generally can identify all constituents (e.g., infectious agent(s) or marker(s) of interest; novel, emerging agent(s) or marker(s); microbiota; human background; and contaminants) in a clinical metagenomic sample (direct genetic analysis from a multi organism sample).”
FDA notes that assays addressed in this draft guidance should be “used in conjunction with a patient’s clinical presentation and other laboratory tests to aid in the diagnosis of pathogenic microorganism infections” and should not be used as the sole basis for diagnosis.
In addition, FDA states that the draft guidance does not apply to “devices for screening donors of blood and blood components or donors of human cells, tissues, and cellular and tissue-based products (HCT/Ps) for communicable diseases.”
Notably, FDA points out that inclusion of certain targets, such as Hepatitis B, Hepatitis C, HPV and HIV, could elevate the classification of the device to Class III, which would require the test sponsor to contact FDA for additional guidance. Consultation with FDA is also needed if the device is intended for detecting new pathogens or markers. Given how new the Agency is to dealing with tests of this type, we would recommend discussion with the Agency prior to submission of any marketing notification or application.
Because the Infectious Disease NGS Dx devices are capable of testing multiple pathogens and markers in a single sample through a common process, FDA proposes to use a “systems approach” for the evaluation of such devices. This approach will evaluate all steps in the sequencing pipeline, including specimen collection, specimen preparation for sequencing, sequencing/chemistry/data collection, data storage, and report of clinically actionable data, as a whole. Databases and genome assembly/annotation/finishing may also fall under regulatory oversight if they are used as part of the data analysis pipeline.
FDA notes that consideration of pre-analytical factors is critical for the Infectious Disease NGS Dx devices and should be addressed in the premarket submission.
Specimen Collection and Handling
The quality and quantity of the isolated nucleic acids play an important role in the performance of the Infectious Disease NGS Dx devices. FDA recommends validation for all specimen types for which the device is intended to be used. The submission should including the following:
- Validation of any nucleic acid extraction method to be indicated for use with the system
- Validation that sample collection methods provide adequate and appropriate nucleic acid for all sequence targets detected by the Infectious Disease NGS Dx device, if applicable
- Validation that the device maintains acceptable performance under all specimen handling conditions claimed
Specimen Preparation for Sequencing
FDA notes that while nucleic acid extraction and purification are essential for successful identification and detection, determination of integrity and purity of the extracted nucleic acids may not be possible for metagenomics specimen types. However, FDA maintains that minimum requirements should be provided as quality control for the material, including:
- Sample Amount (µg)
- Sample Volume (µl)
- Concentration of nucleic acid in the sample (ng/µl) (typically used to address unspecific loss)
- Quantification Method 260/280 ratio 260/230 ratio
- Agarose gel
- Total ng of nucleic acid
For devices that use library preparation methods, FDA also recommends that the sponsor address the variability of performance for all preparation methods and reagents. Further, FDA recommends conducting an analysis of potential inhibitors from clinical specimen or sample preparation methods.
Sequencing, Chemistry, and Data Collection
FDA lists quality metric that should apply to Infectious Disease NGS Dx device and should be submitted, including:
- Narrative on trimming and filtering logic (e.g., minimum Q-Score, minimum length, etc.)
- Total number of reads generated
- Total number of unique reads generated
- Range of read length
- Total number of mapped reads and percent identity
For devices using agnostic (i.e., without pre-specified target) sequencing, additional metrics include the number of reads and percent identity to a qualified genomic reference sequence.
FDA recommends that data generated should be securely stored and kept on file during the review process, during which FDA may request such data for independent verification purposes.
Data Analysis Pipeline
The bioinformatics package or data analysis pipeline for use with the sequencing platform should be in a “locked-down” configuration before device validation. The analysis pipeline may include the following components:
- Signal to base call transformation
- Sequence alignment
- Clinical call determination
- Database, if applicable
Documentation of the “locked-down” pipeline should be provided and should demonstrate robustness for clinical use.
In general, for targeted Infectious Disease NGS Dx devices, analytical performance can be validated similarly to other multiplexed devices. For agnostic Infectious Disease NGS Dx devices, validation for each target sequence included in the intended use may not be feasible. Therefore, FDA allows validation using a representative number of targets or chosen panels of certain agreed on organisms or markers. FDA summarizes the type of information that is recommended to be included in a premarket submission.
Limit of Detection (LoD)
In the draft guidance, FDA describes a feasibility study as an example for LoD determination in devices that use agnostic sequencing approach. In such study to approximate the LoD range, mock samples containing human background DNA are spiked with a set of representative pathogen and marker targets to clinical level expressed in genome. The clinical levels can be determined based on literature. Initially, preliminary LoD can be established by testing a small number of replicates at each concentration. The determination should then be confirmed by testing a minimum of 20 independent replicates at the lowest concentration that can produce a positive result greater than 95% of the time.
For validation of inclusivity and analytical reactivity, FDA recommends using intact cultured organism that undergo all pre-analytical steps. Pre-extracted and defined nucleic acids may be used in certain circumstances, such as rare organisms, non-culturable organisms or BLS3 and BSL4 organisms. For agnostic sequencing, the inclusivity and reactivity evaluation can be performed using panels that cover adequate diversity of the device’s intended use. For example, if the assay is to detect Salmonella enterica, the sponsor should demonstrate that the test can detect all frequently reported serotypes at or near the specific LoD or cut-off value. If adequate diversity cannot be achieved due to limited strain availability, FDA recommends expanding the laboratory testing through in silico analysis of target sequences. This panel-based approach, as FDA notes, may also apply to targeted sequencing based on amplification strategy.
The draft guidance lists sources of interfering substances that should be evaluated and submitted, including:
- Interference by contaminants when targets are present
- Interference by other microorganisms when targets are present (microorganisms known to be present in types of specimen tested by assay for specific indication of use (clinical syndrome))
- Interference by human background, if applicable
- Cross-reactivity when targets are not present
- Interference by polymerase chain reaction (PCR) inhibitors
- Competition of amplifying primers, if applicable. (This one only applies to targeted sequencing)
Precision (Reproducibility and Repeatability)
Similar to other IVDs, evaluation of reproducibility and repeatability for Infectious Disease NGS Dx devices can be designed in accordance with the EP12-A2 standard. The evaluation may also employ the Microbial Standard Reference Materials (SRM’s) developed by NIST.
Carryover and Cross-contamination
FDA recommends that carryover contamination should be evaluated for the entire device, including sample preparation and library preparation, by alternating known positive samples (at a high target concentration) and negative samples.
FDA recommends real-time stability testing for reagents and instrument.
Additional Analytical Studies
Depending on device intended use, specimen type and study design, FDA suggests additional studies, including matrix equivalency study, fresh vs. frozen study, specimen stability study, and mixed infection study.
Using Database in Clinical Evaluation
Consistent with the concept discussed in prior NGS workshops, FDA proposes an alternative clinical validation process than what would normally apply to routine diagnostic methods. This proposed approach relies on a defined set of sequences and variants tested in both the NGS method and a comparator method, coupled with public regulatory-grade genomic sequence databases to establish the clinical significance of detected variants. FDA states that this new regulatory strategy aims “to promote a least burdensome regulatory approach,” because “given the number of potential pathogens and resistance and virulence markers that NGS technologies may be able to detect in a single clinical specimen, the application of more traditional regulatory strategies may hinder approval or clearance of these devices by requiring extensive evaluation of every detected organism (genomic sequence) from a single specimen, or in the case of device specificity all of those that were not detected, using expensive reference methods.”
Specifically, the databases should contain a set of validated regulatory-grade genomic sequence entries and will be used in an alternative comparator method for clinical evaluation. To that end, FDA, in collaboration with other federal agencies, has developed the database called “FDA-ARGOS: FDA dAtabase for Regulatory Grade micrObial Sequences; BioProject 231221,”3 which provides regulatory-grade microbial sequences that are near-complete high quality draft genomes with metadata.
The regulatory-grade genomic target database can be used in evaluations of both negative percent agreement and positive percent agreement. For negative percent agreement, depending on the number of organisms and specimen types, the evaluation can be done using the database as a comparator. For positive percent agreement, a panel or subset of representative organisms can be confirmed by an acceptable comparator method, and then the entire set of claimed organisms can be confirmed using the database.
For the first time, FDA outlines the specific criteria for a database to be considered “regulatory-grade.” For example, FDA notes that regulatory-grade genome sequences should provide sufficient coverage for the assay’s indication for use. For microbial reference genome sequences, the coverage should be at a minimum of 20X over 95% of the core genome at Phred-like quality score Q ≥ 30.
The Appendix of the draft guidance includes quality metrics that FDA sets for regulatory-grade genomic sequence entries, including:
- Extracted Genomic DNA (gDNA) Extracted gDNA should be of high quality and purity, and at sufficient concentration to assure adequate genomic coverage.
- BioSample Metadata A minimal description of the sample source material, such as organism and identification method, should be available for traceability.
- Sequencing Data At the minimum, the generated raw reads should be deposited in NCBI’s Sequence Read Archive (SRA) and assemblies should be deposited at NCBI’s Assembly division.
- Sequencing Metadata A minimal description of the sequencing process, including library, platform, submitter, fold coverage, pipeline, assembler, and annotation tool, should be available for traceability.
- Suggested Phenotypic Metadata A description of the phenotypic information is suggested to create a link between the phenotypic traits of particular organisms and their genomic sequences.
In addition, if the assay utilizes a proprietary database, FDA recommends that the quality criteria for establishing the accuracy of regulatory-grade reference databases, as well as the methods for curating, maintaining, and updating the databases be included in the submission. To qualify as regulatory-grade, sequence entries for each claimed organism in the database should be constructed with a minimum of five well characterized isolates.
What FDA does not address in the draft guidance is how to use a regulatory database as a comparator. For example, the draft guidance does not explain, in cases where the test already uses a regulatory-grade database for clinical calling, whether the comparator method is still needed and, if it is needed, what database should be used for the comparison. Should the clinical calls using the sponsor’s own database be compared with the putative clinical calls for the same sequencing data using a public database? Since the sample preparation and sequencing steps are the same, would the comparison become in silico? Further, in the previous clearance of NGS genetic tests, such as the two cystic fibrosis assays using Illumina’s MiSeqDx platform, FDA accepted the CFTR2 genetic variant database as the source of clinical evidence, instead of requiring separate clinical validation.4 It is unclear whether such approach is exactly what FDA has in mind in the draft guidance.
The Infectious Disease NGS Dx draft guidance, although not finalized, indicates FDA’s willingness and flexibility in approach for regulating devices using NGS technology, which pose significant challenges to FDA’s existing regulatory scheme for IVDs. As the first released draft guidance document on NGS-based devices, it provides insight into FDA’s current thinking on this topic, and may become the basis for future guidances on other NGS-based devices and claims.
Because of the complexity of these tests, FDA proposes a one-system approach to regulate all processes from specimen collection to output of clinical data. For evaluation of analytical performance, FDA recognizes the difficulty to conduct traditional validation testing for each target. Instead, the Agency recommends testing on a representative subset of organisms or markers. Further, FDA proposes using public databases of regulatory-grade genomic sequence entries for clinical evaluations, and has already developed one such database to provide a set of validated regulatory-grade sequences from a variety of infectious microorganisms.
Due to new advancements in this fast moving field, FDA strongly recommends that sponsors contact the Agency prior to initiating any clinical or analytical validation studies to discuss the strategy and whether additional recommendations are available. Sponsors can use the pre-submission program to engage in the discussion with FDA.