Sensitivity Analysis

Since the point of this sensitivity analysis was to address the uncertainty inherent in estimates of cost for a particular gain and vice versa, we treated the two as being independent. For example, when lowering the cost of a capability, we did not also lower the value assigned to its expected gain.

These two graphs depict the results at the mission level, but similar graphs could be drawn for any level down to individual technologies for each mission. The check marks indicate missions that were enabled because all of their technologies were selected. The optimal portfolio of missions for a budget of $500 million therefore includes all of the checked missions.

The bars indicate the range of cost or expected gain over which the composition of the optimal portfolio would not change.

On the "cost margins" graph, we see that Asteroid Sample Return (ASR) is one of the selected missions. The nominal cost, which we used in constructing the portfolio, is represented by the dividing line between the red and yellow portions of the bar. The bar indicates that the portfolio would remain constant even if the cost of this mission were as low as $32.4 million or as high as $49.9 million. If the cost dropped below the lower limit, money would become available to fund a currently unselected mission. Similarly, if the cost rose above the upper limit, there would not be enough money to fund all of the checked missions. Note that this graph does not tell us whether the Asteroid Sample Return mission, itself, would become unselected if its cost rose above the upper limit. It is possible that a greater overall return on investment would be achieved by keeping that mission and unselecting another one.

Similarly, Lunar Precursor Resource Survey (LPRS) was excluded from the optimal portfolio on the basis of a cost estimate of $245.9 million, but it would have made no difference if its cost were actually as low as $112.5 million. Below that figure, LPRS would be selected, possibly at the sacrifice of a currently selected mission, and the portfolio would change.

Similarly, the "expected gain margins" graph shows that the portfolio was calculated on the basis of ASR having an expected gain of 49.6, but remains unchanged even if ASR's expected gain is as low as 26.9. LPRS was omitted from the portfolio on the basis of an expected gain of 56.3, and remains unselected even with an expected gain of as much as 149.6.

This graph shows at a glance how robust our results were for each of the missions. The dotted blue line is the frontier between inclusion and exclusion in the portfolio. Those missions far to the left of it (in green) are safely included, with a large tolerance in the estimates of their cost and expected gain. Missions far to the right of the line (in red) are reliably excluded from the portfolio. Those near the line are subject to being traded in or out of the portfolio, depending on relatively minor changes in their cost or expected-gain estimates.

Note that for this analysis, cost and expected gain were varied for only one mission at a time. Each bar shows the range over which the values for that mission could vary without changing the portfolio provided there was no variation for the other missions at the same time.

We also performed a Monte Carlo sensitivity analysis, which consisted of 1000 runs in which the cost and expected-gain values for all of the missions were varied at the same time by random amounts up to +/- 25%. (This process is described in more detail in the description of our aeronautics study. The Monte Carlo analysis produced results similar to those of the deterministic analysis described above, leading to a consistent set of conclusions about the robustness of the recommended investments.