Capability Prioritization for ETDP Support of Constellation

How can decision-makers schedule technology development so that it meets mission needs while fitting optimally into a multi-year budget schedule?

NASA's Exploration Technology Development Program (ETDP) funds technology development for the Constellation program, which comprises NASA's efforts to return astronauts to the Moon and ultimately to send them on to Mars. ETDP's responsibilities include selecting technologies for development from a group of proposals and scheduling their funding in a way that simultaneously meets mission requirements and fits within the forecasted schedule of budgets for the coming years.

ETDP engaged the START team to provide analysis and scheduling recommendations to assist the Technology Prioritization Panel (TPP) in its task of prioritizing the candidate technology capabilities. (In this context, a "capability" is a group of technologies that work together to achieve a common function, e.g., "life-support water recovery" or "habitat radiation-shielding systems.")

The benefits of the START system are enhanced efficiency and transparency, and the ability to see the results of making various alternate decisions or of changes in the data on which the decisions are based. When successfully implemented, the START process enables decision-makers to get the most value out of a given budget schedule and provides a logical explanation for the choices that have been made. The system requires a database that includes the considerations on which the decision-makers want to base their choices; building a database that is comprehensive and accurate is the system's greatest challenge.

Using a database provided by ETDP and Constellation, we computed an optimal baseline schedule for the development of technology capabilities for the lunar program, beginning in 2011. In response to "what if" inquiries from TPP, we also produced a set of 13 "focused" portfolios, each of which shows the impact of a different set of parameter variations. We followed the lunar scheduling by computing a schedule for the development of capabilities for the Mars program, beginning in 2012.

Technology funding is a dynamic process, with available amounts varying from year to year and subject to sudden reduction (or perhaps increase under favorable conditions). Further, new constraints may arise unexpectedly, such as the mandating of certain technologies for political or other reasons. One of START's chief advantages is the ability to quickly revise such inputs and compute new results.

Constraints and attributes

As usual, budget was an important constraint in this study. We considered total available budget, existing funding on reserve and any other earmarked funding.

We used each capability's Performance Design Review (PDR) date as its "needed by" date. A capability's PDR is the milestone by which it must be developed to Technology Readiness Level (TRL) 6, indicating that the technology has been tested in a relevant environment. We also factored in the number of years of development anticipated for each capability, and how much funding we were told the capability would require in each year of its development.

As the basis for calculating each capability's value, the database included a variety of attributes for each capability, including the following:

• Level of importance as perceived by experts in the Constellation program
• Level of importance as perceived by the capability's technologists
• Mass Figure of Merit (FOM): the extent to which a capability reduces the amount of mass that needs to be launched
• Combined FOM: the extent to which a capability improves the aggregate of mass, volume, power consumption, cost, risk/safety and reliability/maintainability/ operability
• Percent performance improvement in Key Performance Parameters (KPP) over State of the Art (SOA)
• Whether or not the capability is applicable to both the lunar and Mars programs

Each attribute was assigned a weight by Constellation, reflecting its perceived importance relative to the other attributes. The various FOMs were assigned relative weights by averaging the judgments of three experts and normalizing them so that the "Combined FOM" weight totals 1.

Since the various attributes differ by unit and scale, we normalized each attribute so that it was in the range 0 to 10. The normalized attribute value (v) for each capability (i) was computed as follows:

where H is the highest value for that attribute over all capabilities and L is the lowest value for the given attribute, and x is the value provided by the database. (v is essentially a normalized version of x.) The normalization is done independently for each attribute.

The capability's utility is defined as the weighted sum of its normalized attributes:

Note that START maximizes the total expected utility for the portfolio of capabilities while meeting budget and time constraints.

Focused portfolios

The TPP raised a number of "what if" questions regarding the impact on the results if certain parameters were varied. We provided a set of 13 "focused portfolios" that shone a light on these variations.

Case Enabling constraint KPP/ SOA boost All FOM boost Mass FOM boost Equal FOM weights PDR year shift % budget increase Mars boost Case 1 Yes 1 1 1 No 2 0 No Case 2 Yes 1 1 1 No 1 0 No Case 3 (baseline) Yes 1 1 1 No 0 0 No Case 4 No 1 1 1 No 0 0 No Case 5 No 1 1 1 No 1 0 No Case 6 No 10 1 1 No 1 0 No Case 7 No 1 10 1 No 1 0 No Case 8 No 1 10 1 Yes 1 0 No Case 9 No 1 1 10 No 1 0 No Case 10 No 1 1 1 No 1 20 No Case 11 No 1 1 1 No 1 40 No Case 12 No 1 1 1 No 1 100 No Case 13 No 1 1 1 No 1 0 Yes

Variations included the following: whether or not any of the capabilities were considered "enabling" and therefore essential; increasing the weight of certain attributes to 10 instead of the baseline 1; considering all FOMs to be of equal weight; shifting the PDR date by 1 or 2 years from the baseline, increasing the budget by 20%, 40%, or 100%; and giving extra weight to capabilities which have application to both the lunar and Mars programs.

Running these variations was a strategy for dealing with the uncertainty inherent in some of the attributes. Those capabilities that consistently were selected for portfolios for a given year in multiple variations were considered to be robust.

Results of lunar analysis

The following table compares TPP's judgment with the recommendations of the START analysis.

Capability ID number TPP rank 1st year of funding per START analysis Capability ID number TPP rank 1st year of funding per START analysis 17 1 2012 12 2 2012 21 1 2011 36 2 2012 5 1 2012 24 2 2012 29 1 2011 3 2 2012 9 1 2012 42 2 2012 35 1 2012 16 2 2013 13 1 2012 32 2 2011 39 1 2012 8 2 Not selected 2 1 2012 20 2 2011 41 1 2012 34 2 2011 27 1.5 2011 46 2 2011 11 1.5 2013 4 2 2012 43 1.5 2011 28 2 2012 23 1.5 2011 48 2 2012 47 1.5 2011 14 2 2012 1 1.5 2012 22 2 2012 37 1.5 Not selected 10 2 2012 19 1.5 2011 38 2 2013 31 2 2012 44 2 2012 7 2 2012 18 2 2012 45 2 2012 40 2 2012 25 2 2012 6 2 2012 33 2 2013 30 2 2012 15 2 2012 26 2 2012

A ranking of 1 implies that TPP wants development to begin at the soonest possible date for this evaluation--i.e., 2011. There is not a one-to-one correlation between ranks of 1.5 or 2 and any particular years, although there is an implication that capabilities ranked 1.5 would be started after those ranked 1 and before those ranked 2.

As is evident from the above table, there is a low level of agreement regarding which capabilities should be funded first. This signals a need to re-examine the database to ensure that it contains all of the constraints and attributes that need to be considered, and that all of the values in the database are accurate and in accord with the judgment of the decision-makers.

TPP's rankings were based on what the panel thought was each capability's importance to the mission, and possibly on other unstated constraints which need to be identified and included in the analysis. START scheduled the capabilities in the way that produces the highest total value, given the information in the database. Development of even the most highly valued capabilities could be delayed in START's schedule in order to permit earlier funding of other capabilities with longer development times, or to permit the best fit of capabilities into the total budget anticipated for a given year, as long as the data indicated that each capability would be able to meet its needed-by date.

We note that the stated budget for 2011 is insufficient to begin development of all 10 capabilities that TPP ranked as 1, given the funding that each of them was said to require in its first year. Further, delaying 9 capabilities that START designated for 2011 is likely to lead to budgeting and scheduling problems in later years, including missed deadlines, assuming the data we were provided is correct.

Mars program

In this follow-on to the lunar capability study, we calculated a development schedule for Mars capabilities such that the Mars funding takes advantage of the shape of the lunar funding schedule, filling gaps in years in which the lunar program does not utilize all of the funds anticipated to be available. This opportunity arises because, while available funding is projected to increase, the need for funding by the lunar program ramps down in the later years of development, creating a gap that can be filled with technology development for the Mars program.

Click here for illustrations of the development-funding schedules for Mars capabilities.

This was part of an effort to integrate the Mars funding schedule with the schedule of lunar new starts (those discussed above, beginning development in 2011) and the schedule of lunar technology development that the Constellation program already has underway.

At this point, we assumed that each capability for the Mars program was equally desirable, and cost was estimated to a rough order of magnitude. Development duration was estimated based on a priority sequence: Reducing mission risk was assumed to be highest priority and to require the most development time, followed by reducing mission cost, followed by improving mission performance.

Based on this preliminary data set, we computed a baseline development schedule and an alternate schedule that shows the impact of a change in funding.

Conclusions

The START team has been able to provide ETDP and Constellation with insights regarding the impact of various attributes and figures of merit on the prioritization of technology capabilities. These insights were very well received and much appreciated.

We were also able to demonstrate the flexibility implicit in the schedule and the advantages of using it appropriately. Our results showed, for example, that delaying some of TPP's top-priority development projects by one year would still permit all of those projects to meet their deadlines, while enabling other important technology capabilities to meet theirs, assuming the correctness of the data we were given.

Our recommendation for the next step in this process is to validate the existing database, examine it carefully for any missing considerations that should be included, and establish a process for keeping it up to date for use in future prioritizations.