OPENing up military innovation: The causal effects of ‘bottom up’ reforms to US defence research

Sabrina Howell, Jason Rathje, John Van Reenen, Jun Wong 08 May 2021



Military necessity has been the mother of innovation since antiquity. During the Roman attack on Syracuse in 214 BC, Archimedes created several innovations that successfully defended the city including his famous ‘claw’ – a gigantic crane that dragged enemy ships out of the sea . More recently, Pentagon funding has helped develop jet engines, cryptography, nuclear power, and the Internet (e.g. Mazzucato and Semieniuk 2017). Despite being lauded by policymakers all over the world, US defence R&D seems to have lost its lustre in recent decades. 

Figure 1 shows that in 1976, the major American defence contractors (‘Primes’) produced 15% more innovations (as measured by future citation-weighted patents) than similar non-defence firms. By 2019, they appeared between 15% (if their self-citations were included) and 40% (if we drop self-citations) less innovative. This fall is due to both a relative decline in the number of patents granted to primes (despite no loss of market share) and the degree to which other firms see them as useful.

Figure 1 Innovation by prime defence contractors was stronger than non-defence firms in the late 1970s, but has been in long-term decline 

Notes: This figure describes patent quality for the six major prime defence contractors in 2019 (Lockheed Martin, Boeing, Raytheon/UTC; Harris/L3; Northrop-Grumman, General Dynamics) and all their acquisitions since 1976. We show the number of patents weighted by the number of future citations in the solid blue line and drop ‘self-citations’ to these primes in the dashed orange line. Both lines are expressed relative to the average in the same technology class-year. The dashed line (i) excludes self-citations and (ii) excludes any citations from prime defence contractors and their acquisitions. The measures are smoothed using kernel-weighted polynomial regressions. The gray band around the relative citations represents the 95% confidence interval. 
Source: US Patent and Trademark Office (USPTO) via Howell et al. (2021)

Senior US defence policymakers have recognised this problem, and see it stemming from both a failure to engage many of the new software-based start-ups and a consolidation among the defence contractors. For example, a tasking memo from the 2019 Under Secretary of Defense tasking memo stated, “The defense industrial base is showing signs of age. The swift emergence of information-based technologies as decisive enablers of advanced military capabilities are largely developed and produced outside of the technologically isolated defense industrial base” (Griffin 2019).

The open reform

To address the problem of declining innovation, the US Air Force (USAF) experimented with reforming their Small Business Innovation Research (SBIR) programme, which exists at all federal agencies that conduct extramural R&D but has some degree of flexibility in contracting procedures. The ‘conventional’ approach has been to run SBIR competitions that narrowly specify the technology to be procured. Starting in 2018, the Air Force made some competitions ‘open topics’, which gave applicants much more freedom to suggest projects which the firms thought would have potential military benefits.

Working with the US Air Force, we obtained the details on applications and winners of both conventional and open topics between 2003 and 2019. We found that that the types of firms who applied for open topics were very different from those who applied for the conventional programmes. They were half as old, half as likely to have previously applied for a Small Business Innovation Research programme, more likely to be in software than hardware, and more likely to be located in high-tech hubs. In short, ‘open’ attracted many more new entrants, just as the reformers wanted.

But did the reform improve outcomes? Senior administrators at the US Air Force have been concerned that SBIR projects rarely translate to ‘programmes of record’ – a technology embodied in a non-SBIR Department of Defence contract. They also care about whether the winners obtain outside funding from venture capital (VC) to help develop dual-use technologies for non-military use.

Three evaluators score each SBIR application and the combined score determines whether an applicant is funded or not. We use this information to compare firms who just lost versus those who just won in a ‘regression discontinuity design’.  

We find that there is a big jump in the probability of getting future venture capital funding for firms just above the threshold for open competitions, compared with those who were just below. By contrast, there is no jump for winners of the conventional programme. Similarly, open competitions seem to have a positive causal impact on getting a future non-SBIR military contract, whereas there are no effects for conventional awards.

We also found positive causal impacts of winning open awards on patenting – especially more original innovations – but no effects from conventional awards (see Figure 2). In fact, the main effect of winning a conventional programme seemed to be to increase the chances of winning another conventional competition in the future, creating a kind of ‘lock-in’ effect for incumbents.

Figure 2 Winning an open competition means a higher probability of future innovation (but winning a conventional competition does not)

Notes: These figures show the probability that an applicant firm had any ultimately granted patent applications within 24 months after the award decision. In both panels, the horizontal axis shows the applicant's rank around the cut-off for an award. A rank of one indicates that the applicant had the lowest score among winners, while a rank of -1 indicates that the applicant had the highest score among losers. We plot the points and 95% confidence intervals from a regression of the outcome on a full complement of dummy variables representing each rank, as well as fixed effects for the topic. The omitted group is rank=-1.We include first applications from 2017-19.

There were two reasons for the success of open topics. First, new entrants appeared to have bigger treatment effects on venture capital, though not on the other outcomes we study. A second reason is the bottom-up nature of innovation in open competitions, which appears to be relevant beyond selection. We provide evidence of this in two ways. In one analysis, we show that other reforms at the same time as open competitions which also attracted new entrants but were not bottom-up did not have positive causal effects on our success measures. In the second analysis, we find that among conventional competitions, winning an award among those that were relatively more open-ended (using machine-learning techniques to measure the variation of text applicants made within each topic competition) did have a positive causal effect on innovation, just like the open competitions.


Bottom-up innovation has recently been encouraged by many governments and private sector firms around the world. Our findings suggest that a reform encouraging this type of innovation procurement at the US Air Force was successful. Our findings do not mean there is no role for conventional top-down approaches in the Department of Defence or elsewhere, as there may be other benefits that we cannot measure. However, they do suggest there should be an ongoing role for open programmes both in the military and perhaps more widely in the private sector.


Griffin, M D (2019), “Terms of Reference - Defense Science Board Task Force on 21st Century Industrial Base for National Defense”.

Mazzucato, M and G Semieniuk (2017), “Public Financing of innovation: new questions”, Oxford Review of Economic Policy 33(1): 24-48.

Howell, S T, J Rathje, J Van Reenen and Jun Wong (2021), “Opening up Military Innovation: An Evaluation of Reforms to the U.S. Air Force SBIR Program”, CEPR Discussion Paper 16063.




Topics:  Productivity and Innovation

Tags:  military, research and development, innovation, reforms, regression discontinuity

Assistant Professor of Finance, NYU Stern School of Business

Director, AFVentures

Professor of Applied Economics, MIT

Research Assistant, NYU Stern

CEPR Policy Research