A Structured Approach to Strategic Decision Making for NASA's Technology Development
Are quantitative decision-making practices possible in complex systems such as NASA's technology-development program?
We performed this study for the Capabilities, Requirements, Analysis and Integration Team (CRAI), which NASA created as an inter-center effort to capture, validate, and analyze information that would enable a systematic approach to technology investment for space exploration. We employed our standard methodology to collect quantitative information and use it to assemble optimal development portfolios. The process and results are intended to be tools that a decision-maker can use to assess trends and better understand the underlying value of each capability area.
Mission-enabling vs. democratic
CRAI selected a set of missions for a proposed Lunar/Mars campaign consistent with the Presidential Vision for Space Exploration and sought information on capability requirements.
One possibility for guiding investment decisions is a "mission-enabling" approach, in which sets of technologies are selected to enable particular missions. In such an approach, no technologies for a particular mission are funded unless they can all be funded, since the mission would fail unless all were developed. (For an illustration of the mission-enabling approach, see our aeronautics study.)
An alternative is a "democratic" approach, in which technologies are selected based on their relative performance gain over the state of the art. This is the approach we opted to take, since it is most applicable for long-term planning studies where focused mission designs have not yet been firmly defined. We did, however, give technologies higher scores if it was clear that they would support multiple missions.
As stand-ins for future missions for purposes of this study, CRAI used three architectures: 1998 Mars Reference Mission, OASIS mission, and JSC (Johnson Space Center) Architecture.
Calculating unitless values
We created a data template to collect contact information, technology name, linkage to the capability-breakdown structure, expected cost, probability of success (POS) if fully funded, metrics used, state of the art (SOA), expected increase over SOA, and budgetary and development plans. Forty-seven of the finished technology sheets met the requirements of completeness and strict matching between requirement metrics and technology metrics.
To enable comparison of disparate technologies which have different goals, we converted the projected technology metrics into unitless scores by calculating a ratio, for each technology, of the requirement to the state of the art.
For example, the "Disposal of Human Waste" technology had two metrics: time that waste is contained and stored volume density. SOA for time contained is 0.25 years, while the requirement was given as 200 years. The ratio is thus 200 years ÷ 0.25 years = 800 (with no unit). Similarly, SOA for stored volume density is 70 kg/m3, while the requirement was given as 700 kg/m3. The ratio is thus 700 kg/m3 ÷ 70 kg/m3 = 10 (with no unit).
To make the results more intuitive, we took the log2 of each score, so the resulting score became a measure of how many times a technology's performance has to double to achieve the requirement. The results are 9.64 for time contained, and 3.32 for stored volume density. We then averaged the scores (adding them would have given arbitrarily high scores to technologies with many metrics) and multiplied the result by the percentage POS to arrive at an expected value. In this example, the average, 6.48, was multiplied by the 90% POS to produce an expected value of 5.83.
Multi-mission technologies
This unitless score can now be seen as an estimate of the expected benefit of this technology to a single mission. If it were needed for another mission, a score would be calculated for that mission's needs, and the two scores would be added. Note that under this system, a technology's expected-benefit score increases with each mission it supports, but its cost remains unchanged.
With a score and a cost, a benefit-cost ratio (or return on investment) can be obtained. Such a ratio was calculated for each technology under consideration.
We then performed a simple "grab bag" optimization at several budget levels. At each level, the technologies were selected to maximize the ROI for the whole portfolio.
Results
Each bar in the above graph represents an optimal technology portfolio (in terms of maximizing technology improvement) for a given budget level. For example, if the technology budget for the next ten years were $300 million, the funds should be invested approximately as follows: $60 million in Communications and Information Systems, $30 million in Space Utilities and Power, $110 million in Human Support Systems, $70 million in Automation and Robotics, and $20 million in In-Space Transportation.
Note that as the budget increases from $25 million to $100 million, the optimal portfolio increases the budget of Communications and Information Systems. But at $125 million, the amount allocated to that technology area decreases slightly. This is due to the addition of a more expensive technology in another area that has a higher benefit/cost score, which was previously unaffordable.
Supplemental data
Since the data collected was sparse, we extended our study to include supplemental data from a preliminary version of the study. We added 11 technology areas with 156 new technologies, for a total of 203 technologies arranged into 16 areas, and matched the new technologies to requirements from future NASA missions.
This graph illustrates the results. Once again, the decision-maker can select a budget and see the optimal way to divide the funds among the available technology areas. A similar graph can be calculated at any desired depth, down to individual technologies.
Conclusions
This work demonstrates that quantitative data collection can be conducted across a broad range of missions, and analytical decision-making based on the construction of optimal portfolios is feasible for complex systems such as NASA's technology-development program. The process has the advantages of being objective, traceable, quantitative, and repeatable.
