Return-on-Investment Analysis for JPL Chief Technologist

Can a systematic analysis of projected ROI assist JPL's selection of technology-development funding?

This study was part of an effort to develop a systematic process that JPL's Chief Technologist could use to help him select a group of technologies for development, on the basis of the probability that they will lead to mission assignments from NASA.

This study features the following innovations:

1. Determining sensitivity of the results to variation in the input values.
2. Identifying not only optimal portfolios, but also optimal portfolios that contain a required element (K-best sets).
3. Determining when to fund each mission.
4. Treatment of capabilities that enable multiple missions.
5. Including relative mission value when calculating a capability's value.

Calculating expected gain

As is often the case in studies like this one, the objective was to assemble optimal portfolios of technology capabilities to fund for development. The procedure generally is to calculate a unitless value for each candidate capability, representing its predicted return on investment (ROI), and build portfolios with the highest-possible total ROI for a particular budget level. In this case, we calculated the expected value (also known as "expected gain") for each capability by multiplying the following:

• Polarity -- the desired direction of the metric being measured for a given capability: plus indicates the goal is an increase (e.g., a rover's range), and minus denotes a desired decrease (e.g., an instrument's mass).
• Log₂(Need/SOA) -- The targeted value divided by the state-of-the-art (SOA) yields the required improvement. Giving the result as log₂ provides a number that indicates how many times SOA has to double to produce the goal performance.
• Importance -- An expert's estimate of the relative impact the capability will have on the mission.
• Probability of Success (POS) -- An expert's estimate of the likelihood that the capability, if funded, would achieve TRL 6. It is generally inversely related to the degree of difficulty in effecting the required improvement from SOA.
• Probability of Application (POA) -- An expert's estimate of the likelihood that the capability will achieve TRL 9 and actually be used in a NASA mission.

This formula produces the most technological improvement for the dollar, but it is certainly not the only basis on which capabilities can be compared. The formula can easily be customized to reflect the preference of the decision-maker, and can include non-technical factors such as workforce considerations or the earmarking of funds.

Assembling optimal portfolios

We fed the collected data into an algorithm to assemble optimal portfolios at various budget levels. The results can be displayed at various levels.

This graph takes one of JPL's 13 theme areas, in situ planetary exploration, and breaks it down to show how much should be spent on each capability area at various budget levels. Note that the total amount of spending at a budget of $1.4 billion is greater than at a budget of $1.5 billion. This is because, at the higher budget level, a greater total ROI for the portfolio can be achieved by taking some of the money from this theme area and spending it on one of the other theme areas. (During this phase of the study, only four theme areas were included. The results will likely vary when the remaining nine theme areas are taken into consideration.)

We also took each of these areas and broke them down into optimal portfolios of sub-areas. We can employ this process to whatever depth the decision-maker desires.

Results

The results (for a budget of $500 million) produced funding for technologies that enabled the missions shown here in green. Note that no capabilities for a given mission were funded unless all enabling capabilities for that mission were funded, because the mission would fail unless all of its enabling capabilities were funded. Missions with red bars did not have enough technologies funded in our optimal portfolio at this budget level to enable them.

Sensitivity

Recognizing that our results were based on predictions of cost and gain that are extremely difficult to predict with pinpoint precision, our next step was to determine the range of cost and gain values over which our results would be invariant. In some cases, we found that even if our estimates for a particular capability were off by a fairly large percentage, it would not change the composition of the optimal portfolio at a given budget level. In other cases, we found that a small change in cost or expected gain would cause one or more selected capabilities to become unselected or vice versa. For a description of the sensitivity-analysis process, click here.

K-Best Sets

If legacy, history, or other considerations dictate a choice which is not in the optimal set, then we can solve for the best set that does include the necessary component. This would not be the best portfolio without restriction, but would be the 2nd or 3rd (or Kth) best, and would be the optimal portfolio given the new requirement.

To calculate the K-best sets, we started with the optimal portfolio and then recalculated it with the constraint that the overall value must be less than that of the optimal portfolio. This produced our 2nd best portfolio. We then calculated a 3rd best, 4th best, etc. If a decision-maker required that the ultimate portfolio includes a particular item, we would simply find the highest-ranked portfolio that contains that item.

This chart shows the composition of the first-through-fifth-best sets of missions after the optimal portfolio for a total budget of $500 million. If, for some reason, it were a requirement that Large Observatory Platform must be included, then K5 would be the best portfolio.

Adding Time into the Equation

A major innovation of this study was the calculation of not only which technologies to fund, but also when to fund them. One option is a "just in time" approach, in which technology development for each mission is funded as late as possible while enabling the mission to launch at its scheduled time. This approach reduces the risk that funds will be spent prematurely for a mission that is ultimately canceled, or that technologies will become obsolete before the mission is launched. It also increases the ability to take advantage of technological improvements, such as faster computers, developed outside of the mission, itself.

However, this system increases the risk that one or more missions will have to remain unfunded in order to avoid exceeding the development budget for a given year. That problem can be avoided by eliminating the "just in time" constraint and permitting the development schedules to slide to earlier years if that would better accommodate the annual budgets.

This graph illustrates the results of a portfolio optimization at a budget of $100 million, using the "just in time" approach. Eight missions are funded with a combined value (i.e., expected gain) of 368.5.

This graph illustrates the more flexible approach to scheduling technology development. At the same budget level, this portfolio includes nine missions with a combined value of 502.9. Similar differences were found in optimal portfolios assembled according to these two methods at lower budget levels.

Common-mode Technologies

Some technologies may support more than more mission. For example, entry, decent and landing (EDL) for a mission to a comet would likely be identical to that for a mission to an asteroid. Some of the technology for operating in extreme cold would be useful in missions to any of the gas-giant planets and their moons. In such cases, it is necessary to avoid funding the same capability more than once, and to give multiple-mission capabilities credit for increased value.

We developed an algorithm that addresses these issues for multi-mission capabilities, when they can be identified. We subtract the capability when calculating gain and cost for each mission, then double (if two missions share the capability) the gain for the shared area while holding the cost at just the single value. Then we conduct the optimization as before.

Relative Mission Values

We are currently in the process of subjecting our data to review by mission directors and ultimately the director of JPL, to customize and refine the results. While our initial analysis assumed all missions to have equal value, the fact is that for scientific, financial, technological and other reasons, some missions are considered more valuable than others. We have invited mission directors to assign relative values within their own directorates, as well as to judge whether the data we collected on mission cost and likelihood of funding were accurate. JPL's director, in turn, can assign relative priorities across directorates, to compensate for any inconsistencies he may find in the estimates from different directorates and missions, and to make any further desired adjustments, such as raising the expected value of a mission that has the potential to open an entire mission suite. (For example, funding the relatively inexpensive Sojourner rover for the Mars Pathfinder mission led to an entire suite of rover development.)