- On December 21, 2017, the RAND Corporation submitted to Congress a detailed report on the federal bid protest system and its impacts on DoD contracting. The RAND report was commissioned by the FY 2017 NDAA.
- RAND provided detailed quantitative analysis on protests filed with GAO and COFC.
- We provide here some of RAND’s important observations based on its quantitative analysis. We will provide a summary of RAND Corp.’s recommendations in a separate update.
The RAND Corporation has submitted to Congress a detailed report summarizing extensive research on the impacts of the federal bid protest system on Department of Defense (DoD) contracting. The RAND report, Assessing Bid Protests of U.S. Department of Defense Procurements: Identifying Issues, Trends, and Drivers, was commissioned by Congress a year ago in Section 885 of the National Defense Authorization Act (NDAA) for FY 2017.
As we reported last year, the FY 2017 NDAA required DoD to contract with an independent research entity to carry out a comprehensive study on the prevalence and impact of bid protests on DoD acquisitions, including protests at contracting agencies, the Government Accountability Office (GAO) and the U.S. Court of Federal Claims (COFC). According to RAND’s cover letter to congressional committees, “the findings are intended to inform Congress and U.S. defense leaders about the effectiveness of current procurement policies and processes to reduce bid protests.” The report has been released by the news media and is expected to be available soon on the RAND Corp. website, www.rand.org.
The RAND report is more than 100 pages long and presents many detailed findings and conclusions on specific topics as called for under Section 885, but we provide here some of RAND’s most important observations, compiled from the report’s “Summary Findings and Observations.” We’ll provide a summary of RAND Corp.’s recommendations in a separate update.
RAND provided detailed quantitative analysis based on its review of data on protests filed with GAO and COFC. RAND found a “steady increase in the number of bid protest actions at GAO between FY 2008 and FY 2016; during that period, protest activity for both DoD and non-DoD agencies approximately doubled. Protest actions associated with DoD agencies accounted for roughly 60 percent of total protest actions over this period.” RAND found a similar trend for total protests filed with COFC.
RAND noted that the DoD trends over time are “indifferent to changes in DoD spending and contracting” because the overall number of contracts and contract spending declined for DoD from FY 2008 to FY 2016, while GAO protests clearly increased. RAND also found that “the overall percentage of DoD contracts protested was very small—less than 0.3 percent. This small percentage implies that bid protests are exceedingly uncommon for DoD procurements. However, it also should be noted that overall protest activity is increasing at both COFC and GAO and that small-business protests represent the majority of protests at both venues.”
RAND’s observation that only a small percentage of total contracts are protested should be qualified by one of RAND’s more detailed findings quoted below—that the number of protests “tends to grow with a contract’s value.”
RAND also noted that policymakers should not draw overall conclusions or assumptions from any one case. In addition, according to RAND, policymakers “should be aware that the perspectives of the bid protest system held by DoD personnel and by the private sector vary greatly and that there is a lack of trust on each side.”
RAND provided a summary of additional more specific observations. According to RAND, the data reviewed led them to the following GAO-specific observations, which we provide here, quoted directly from the RAND report:
- The stability of the bid protest effectiveness rate over time—despite the increase in protest numbers—suggests that firms are not likely to protest without merit.
- Small-business protests are less likely to be effective and more likely to be dismissed for legal insufficiency.
- Protest filing peaks at the end of the fiscal year.
- Task-order protests have a slightly higher effectiveness rate than other types of protests.
- There are measurable differences between the services and defense agencies, but overall the DoD (services and agencies) has a slightly lower effectiveness rate than non-DoD agencies.
- The largest DoD contractors have slightly higher sustained and effectiveness rates, but these differences are diminishing with time.
- Cases in which legal counsel is required (i.e., a protective order was issued by GAO) have higher effectiveness and sustained rates.
- DoD uses stay overrides infrequently.
- The number of protesters and protest actions tends to grow with a contract’s value.
The data reviewed led RAND to these COFC-specific observations, quoted directly from the RAND report:
- The sustained rate at COFC is declining with time as the number of cases increases. These trends suggest that firms may be more willing to file protests with COFC.
- There are no differences in sustained rates between DoD components and agencies or between small and larger businesses.
- The appeals rate is declining over time.
- Data and discussions suggest that the number of COFC cases that previously appeared at GAO may be increasing, but this potential trend needs further research.
In addition, the data and RAND’s analyses led RAND to these observations “common to both GAO and COFC,” also quoted here directly from the RAND report:
- While our statistical modeling indicates differences between types of cases (categorization of cases), it is not possible to predict the outcome of any case based on its general characteristics. Each case is different and its details affect the outcome.
- The overall level (numbers) of bid protest activity (DoD and non-DoD) has been increasing at both GAO and COFC since 2008.
- Bid protests by plaintiffs that identify themselves as small-business represent the majority of protests at both venues.
- At both venues, in a nontrivial number of cases (approximately 4–8 percent), the contract value is less than $0.1 million (then-year dollars, as reported by the protester).
- There are differences between the services and DoD agencies in terms of the number of cases filed. Specifically, the Army has the highest number of cases, year-on-year, at both venues. This is partly explained by its share of contract expenditures.
- Trends differ between GAO and COFC, suggesting that any changes to the protest system should be tailored to the venue. For example, COFC’s sustained rate is declining whereas at GAO it is holding steady (and potentially increasing).
RAND’s findings will likely lead to future efforts to further reform the bid protest system, following the recent reforms adopted in the FY 2018 NDAA, which included the enhanced debriefings and the bid protest pilot program that we have previously reported and summarized. We’ll provide a summary of RAND Corp.’s recommendations for further reforms in a separate update.