Assessment of Capability Development Portfolio for the NASA Aeronautics Program

What capability-development tasks should be funded to enable a Next-Generation Air Transportation System?

The United States has set a goal of enabling a Next-Generation Air Transportation (NGAT) System by the year 2025 to provide for substantially increased capacity while improving -- or at least not worsening -- safety, security, and environmental impact. The Joint Program Development Office coordinates the various agencies involved (including NASA and the FAA) in support of this effort. NASA contributes primarily as an R&D provider of enhanced technologies and capabilities, and its Aeronautics Research Mission Directorate (ARMD) initiated this study as part of an effort to formulate and assess the potential return on investment (ROI) for candidate capability-development tasks.

Like most START projects, this study involved determining a value for each of many competing capabilities, and using those values to compute optimal capability-development portfolios at various budgets. However, this study expanded the methodology used in other recent studies in two primary ways:

1. We included ranges which reflected uncertainty in the assignment of certain values.
2. We based our calculation of a capability's value on how much the capability would advance the goals of the mission, rather than on how much it would improve the state of the art for that capability. For the ARMD study, defining the value of a capability in these terms was a natural way to incorporate mission goals. Using a metric that measures improvement over the state of the art is valuable for research programs where the focus is on creating and advancing new technologies.

Collecting the data: Ranges

NASA provided us with five goals and 38 candidate capabilities to advance those goals.

We asked the appropriate technologists for performance ranges within which they expected their capabilities to fall, and for sets of high and low costs and probabilities of success (POS) associated with those performance ranges. POS in this case referred to the likelihood of achieving TLR 6. As mentioned above, asking for ranges represented a departure from past studies. It was done in an effort to manage the difficulty in assigning point values for such predictions.

We asked the decision-maker to assign values for the high and low ends of each performance range in terms of its utility to the broad objectives of the NGAT system, and to evaluate each capability's probability of acceptance by the FAA. As might be expected, low performance tended to correspond with low utility, low cost, and high POS, while high performance corresponded with high utility, high cost, and low POS.

We also asked the capability developers to assess the positive or negative impact that each capability would have on each goal. This information, together with the decision-maker's evaluation of the relative importance of each goal, was used to calculate a "capability influence factor" (CI) which represented the overall effect that a given capability was expected to have in advancing or impeding the NGAT mission. Note that a capability may advance one goal (such as increasing capacity) while impeding another goal (such as protecting the environment).

Analysis

The value of each capability was expressed through its expected utility, which was calculated using the following formula:

where CI is the capability's influence factor, P_A is its probability of acceptance, U_H and U_L are the high and low utility values, and P_H and P_L are the high and low POS values. The results were fed into our optimization algorithm to produce optimized portfolios for each of various budgets.

Results

2.1.1.A	Protect/Prevent Abnormal Operations & System Failures	2.3.1.B	General Aviation During Peak Demand
2.1.1.B	Detect & Mitigate Natural Hazards	2.3.1.C	Public Service Aircraft During Peak Demand
2.1.1.C	Prevent Breakdown of Human/Machine Interface	2.3.1.D	Minimize System-Wide Disruptions
2.1.1.D	Integrity & Efficiency of Accepting Advanced Software Systems	2.3.1.E	Globally Harmonized Equipage & Operations
2.1.2.A	Detect & Inform Potential System Vulnerabilities	2.3.1.F	Increase Arrival/Landing Rates at Commercial Airports
2.1.2.B	Mitigate Consequences from Intentional Attack	2.3.1.G	Commercial Operations from Small/Underused Airports
2.1.2.C	Detect & Contain Diseases & Bio/Chem Agents	2.3.1.H	Commercial Operations with short/no Runways
2.2.1.A	Low emission subsonic vehicles	2.3.1.I	Incorporate Full Spectrum of Aircraft to NAS
2.2.1.B	Low emission supersonic vehicles	2.3.2.A	Efficient subsonic vehicles
2.2.1.C	Low emission ESTOL vehicles	2.3.2.D	Efficient easy-to-operate personal air vehicles
2.2.1.D	Low emission personal air vehicles	2.3.2.E	Efficient all-weather rotorcraft
2.2.1.E	Low emission rotorcraft	2.3.2.F	Complete Decision Information to All in NAS
2.2.1.F	Low emission UAVs	2.3.2.G	Low Cost Vehicles for Bulk Cargo
2.2.2.A	Low noise subsonic vehicles	2.3.2.H	Increased Speed & Range for Pedestrian Travel
2.2.2.B	Low noise supersonic vehicles	2.3.2.I	Minimum Impediments of Mode Change
2.2.2.C	Low noise ESTOL vehicles	10.5.1.A	Autonomous high altitude long-endurance flight
2.2.2.D	Low noise personal air vehicles	10.5.1.B	Conduct Routine UAV in NAS
2.2.2.E	Low noise rotorcraft	10.5.2.A	Extended Autonomous Flight in Mars Atmosphere
2.3.1.A	Capacity En-Route Commercial Operations in NAS	10.5.3.A	Incorporating Hypersonic Air-Breathing Propulsion

Here we see the results for a budget of $15 billion. Capabilities in green are those that are included in the optimal portfolio, and those in black are excluded.

We then assigned each of the 38 capabilities to one of the nine categories listed in the key above, and produced a breakdown that shows how much of each budgetary portfolio should be devoted to each category. For example, at a budget of $15 billion, about $2 billion is for safety, about $1 billion is for security, about $3 billion is for emissions, about $3 billion is for noise, about $3 billion is for capacity, about $1 billion is for mobility, and about $2 billion is for new missions.

Note that safety is fully funded at a budget of $7 billion, so it maintains that level at all higher budgets. The same thing happens to security at a budget of $8.5 billion. Other categories rise and fall depending on the optimal mix at each budget. For example, the amount allotted to emissions drops when the budget rises from $15.5 billion to $16 billion, at which point the investment in capacity rises sharply. At the next-higher budget level, capacity funding falls and emissions is restored to its previous funding.

Sensitivity

The soundness of the results, of course, depends on the soundness of the input data. However, it is possible to calculate just how sensitive the results are to variations in the input data. In some cases, the results are so robust that they would remain the same even in the face of significant changes in the input values.

We performed both deterministic and Monte Carlo analyses of the results' sensitivity to such changes. (For a discussion of the deterministic process, see Return-on-Investment Analysis for JPL Chief Technologist.) Both produced similar results, leading to a consistent set of conclusions about the robustness of the recommended investments.

Our Monte Carlo analysis consisted of 1000 portfolio-optimization runs in which the value of each capability was randomly increased or decreased by up to 25%, and the results were compared to our initial optimization. (Note that 25% was chosen as a "best guess" at a reasonable deviation. In the future, we plan to use the actual performance ranges provided to us by the technologists.)

Shown here is the result for a budget of $15 billion. The blue bars indicate the capabilities that were included in the optimal portfolio during our original optimization. The red bars show the percentage of the 1000 Monte Carlo runs during which each capability was included in an optimal portfolio for that run. As you can see, the Monte Carlo portfolios corresponded pretty well with our initial deterministic analysis, but some capabilities did show considerable sensitivity to changing values, entering or exiting our portfolios many times.

We observed that the capabilities tended to cluster into three distinct sets: the first set appeared in at least 85% of the Monte Carlo runs, the second set appeared in 15% or fewer of the runs, and the third lay between the other two sets. We derived a heuristic rule that the first set should be considered robust and recommended with confidence, the second set should be rejected with confidence, and the third set should be considered unstable in that the selection of those capabilities was sensitive to even small changes in the input values. Everything in this third set was therefore considered a potential tradeoff.

The green dotted line represents the frontier between inclusion and exclusion in the optimal portfolio for a budget of $15 billion. Green squares far from the line represent capabilities found to be robust, red squares far from the line are those found to be reliably rejected, and squares of both colors that are near to the line are those considered tradable.

Recommendations

Based on these results, we made recommendations at various budget levels, categorizing each capability as either clearly selected, clearly not selected, or a trade candidate -- that is, one of the capabilities whose status could readily change with variations in cost or expected gain.