Follow this link to skip to the main content
JPL - Home Page JPL - Earth JPL - Solar System JPL - Stars & Galaxies JPL - Science & Technology
  NASA Logo
Jet Propulsion Laboratory
California Institute of Technology
BRING THE UNIVERSE TO YOU: 
JPL Toolbar
Search JPL
Technology Selection & Risk Assessment
START Home Red Block
Middle Red Block
Case Studies
white line
About Us
white line
Methodology
white line
Case Studies
white line
 Overview
white line
 NASA agency-level studies:
     START Lite
     Technology Development
     Assessing Future Missions
     Integrated Resource Allocation
white line
 Science (SMD)
     Carbon Uncertainty
     JPL Chief Technologist Analysis
     New Millennium Program

     Enabling Mars Missions:
       - Biosignature Detection
       - Landing Site Selection
       - Selecting Technologies
       - Lander vs Rover
       - Autonomy
       - Hazard Avoidance
       - Predicting Technology Cost
       - Automated Design Tool

     Titan:
       - Science Traceability Matrix
       - Mission Architecture

     Europa

     Space Telescopes:
       - Tech Investment Tools
       - Earth Observatory at L2
white line
 Exploration (ESMD)

     Astronauts on Asteroid

     Asteroid surveyors

     Schrödinger Mission

     Shackleton-Malapert Mission

     Tech Prioritization for
Constellation


     Human-Robot Missions
       - Comparing Architectures

       - Task Scheduling
          - Allocating Tasks 1
          - Allocating Tasks 2

       - Lunar Mission Pilot
          - Human-Robot Polar Mission
          - Robotic Precursor Mission

       - Performance Improvements
       - System Architectures

     Autonomous Inspection
white line
 Aeronautics (ARMD)

     Capability Assessment
white line
white line
Publications & Proceedings
white line
News
white line
Sitemap
white line
START Technology Validation in Space banner

A Multi-Mission, Multi-Program Technology
Resource-Allocation Approach for NASA

Can a systematic, auditable approach to inter-program analysis trades help decision-makers to maximize NASA's return on investment in technology development?

The current drive toward "One NASA," the reorganization of NASA with the goal of a unified plan that crosses "Enterprise" divisions, requires replacing the traditional mission-by-mission approach to R&D allocation with one based on integrated, agency-wide systems analysis. We have completed Phase 1 of a pilot study whose objective was to demonstrate a system for optimizing development portfolios of advanced space technologies by evaluating the technologies, in an auditable fashion, in terms of their impact on multiple future NASA missions.

We selected a broad set of potential NASA missions and associated science and exploration goals, including four missions to the Moon, one comet-sample return, two missions to Venus, four to Mars, and two to the outer solar system (one each to Saturn's moon Titan and Jupiter's moon Europa). The capability requirements for each mission set were identified and quantified, along with the cost, performance, schedule, and risk data for technologies which, if funded, would meet one or more capability requirements. Then we conducted a computational search for technology portfolios with optimal science return, risk and cost.

Missions were chosen both for variety (to demonstrate the ability to evaluate technologies across disparate mission lines) and for the technological challenges they represent. For example, missions to the surface of Venus require equipment that can withstand sulfuric acid clouds and extremely high temperatures and atmospheric pressures, while a mission to Europa will have to tolerate extremely low temperatures and include the abilities to drill through a thick ice layer and explore the ocean thought to lie beneath.

We derived sets of enabling technologies from the mission requirements. For example, a Mars sample-return mission would require technologies in the areas of entry, descent and landing (landing on target while avoiding hazards); surface mobility (the ability to move about safely on the surface); and manipulation, drilling, and sampling (the ability to access, analyze, and retrieve Mars rock and soil samples to bring back to Earth) among others.

A Few Typical Technology Sub-Areas A Few Typical Technology Areas A Few Typical Performance Parameters
Modular, Distributed Structures, Deployable Structures, etc. Multi-Function Structures Contract/Extend (cm), Power per Mass (W/kg), etc.
On Orbit Cryrogenic Fuel Transfer, Tank Pressure Control, Fuel Storage, etc. Fuel Storage & Control Flow Rate (kg/min), Pressure (kPa), Time (yrs), etc.
Range, Radiation Dose, Payload Capacity, Ambient Pressure, etc. Subsurface Ice Mobility Distance (km, mRads), Mass (kg), Pressure (atm), etc.
High Temperature Electronics, Permanent Magnets, Energy Storage, etc. Extreme Temperature & Pressure Components Temperature (Celsius), Pressure (Bars), Energy Density (Whr/l) etc.
Model Based Risk Analysis, Mission Risk Profiling Capability, etc. Risk Methods, Tools & Workstation Accessibility, applicability to multiple mission phases, risk mitigation coverage

The technology areas were in turn decomposed into numerous sub-areas and performance parameters, and the information was organized in a database. As of April 15, 2004, the database included 12 missions covering a wide spectrum of NASA strategic plans, 23 technology areas, 86 technology sub-areas, and 167 technology performance parameters. Additional technologies and detail are being collected in Phase 2 of the study.

Evaluating Risk-Management Software

The evaluation process included both concrete technology products (e.g. advanced materials, structures, etc.) and software tools for risk assessment and mitigation. There was considerable controversy about whether the latter could or should be analyzed in the same way as the former.

Risk is often not well understood or well characterized, especially in early design phases, and it is not currently treated as an inherent resource in design tradeoffs in early formulation phases. Uncertainties and correlations in complex systems are difficult to model and visualize. Finally, risk assessment requires the careful integration of information at different levels of abstraction.

We found, however, that we could apply our methodology to risk-management software in much the same way as our treatment of other technologies -- i.e., quantifying their performance against a set of attributes.

All of the risk-management technologies considered in this study were ongoing tasks of System Reasoning and Risk Management (SRRM), a project of NASA's Engineering for Complex Systems (ECS) Program. SRRM had developed goals, objectives, and approaches for its project. We identified a set of five technology performance attributes against which the tasks could be measured to indicate the likelihood that the tasks would satisfy SRRM's goals. These attributes are:

  1. Accessibility of historical risk event data.
  2. Potential to understand and reduce design risks and optimize resources to retire risks.
  3. Risk-model enhancement (potential for better model credibility).
  4. End-to-end integration for breadth of domain.
  5. Extent to which the task's technology area addresses the needs that SRRM identified for it.

Note that the first four attributes address how well a given technology performs, while the fifth one indicates the percentage of needs that are addressed by that SRRM technology area.

We also interviewed a number of project managers and identified their requirements for risk-management tools. In our judgment, the set of performance attributes also serves as an accurate gauge of the ability of the tasks to satisfy the requirements of the interviewed project managers.

State-of-the-art (SOA) and target performance of each of the risk-technology areas was quantified by rating it on a scale of 1 to 10 for the first four attributes, and as a percentage (0 to 1) for the fifth attribute.

Technology Level Metric Unit Polarity SOA Low ML High $M
    How perfor-mance is measured What unit perfor-mance is measured in + = Better if performance is higher
- = Better if perfor-mance is lower
Current state-of-the-art for similar tech-nologies Technologist's estimate of low, most likely, and high values of what will be provided to the mission How much the tech-nologist needs to achieve TRL 6 in $M
ECS 1                
SRRM 2                
RISK Methods & Tools 4 Accessibility of Historical Risk Event Data 0-10 + 4 7 8 9 2
    Potential to Understand and Reduce Design Risks and Optimize Resources to Retire Risk 0-10 + 1 7 8 9  
    Risk Model Enhancement (Potential for Better Model Credibility) 0-10 + 2 9 10 10  
    End-to-end Risk Integration for Breadth of Domain 0-10 + 2 8 9 10  
    Extent of Needs Covered 0-1 + 0.5 0.7 0.8 0.9  

In the above table, for example, we see that for a budget of $2 million, the technology area called Risk Methods & Tools will improve from the SOA, in which half of the needs ascribed to it are being addressed, to a midpoint target level in which 80% are being addressed, and that the other four qualitative performance metrics will improve to the degrees shown.

Return on investment (ROI) for each technology area was calculated in terms of the projected increase in performance over SOA, weighted by risk of failure (or more positively, likelihood of success), per unit of funding. This calculation results in a unitless number, allowing us to compare technologies of widely different types -- for example, a risk-management algorithm whose performance is measured on a scale of 1 to 10, with a miniature hardware component whose mass-reduction "performance" is measured in kilograms. Based on the results, we calculated technology portfolios with the highest ROI at each budget level.

Policy Options

Different stakeholders with different policies may want to use this system in different ways, depending on their time scale, budget, goals, etc. One decision-maker may decide to fund partial development of a large group of technologies, with the intention of evaluating their progress before committing the funds to finish them. Another decision-maker may opt instead to fund only those technologies that could be completed within the existing budget. One may treat all technologies as having equal value; another may assign a weighted value to each technology, or to the missions it enables. Our framework is flexible and can accommodate various policies, which would of course produce different results.

For this pilot study, it was assumed that each mission is of equal value in terms of science return. It was also assumed that all technologies at the same hierarchical level have the same value. Technologies were not weighted according to any correlations or codependences they might have. It was further assumed that each technology would take no more than 10 years to develop to TRL 6 (the "technology readiness level" that indicates demonstration in a relevant environment).

Results

Overall Investment Strategy

This graph plots the suggested budget recommendations in each of five major technological areas as a function of the resources available to the sponsor over a projected 10-year period.

For example, an optimal technology portfolio at a total budget of $600M would consist of about $220M for Mars Surface Missions, $260M for In-Space Assembly, $45M for OASIS (a near-Earth transportation infrastructure that enables access to the Moon), $25M for Venus Surface Missions, and $50M for ECS (risk management) technologies. (The more precise, actual figures would be provided by an accompanying table.)

Note that certain budget levels represent tipping points, where new technology areas become possible or see significant growth. For example, In-Space Assembly doesn't show up until we reach a total budget of $400M. Funding for Venus Surface Missions stays fairly consistent from $300M to $700M and then increases dramatically at $800M.

Mars Surface Missions

Each of the five major technology areas is divided into sub-areas. Mars Surface Missions is shown here. It is interesting to note that Aerial Mobility comes and goes at the lower levels of funding. Our analysis (based on the policies described above) showed that, though it is worthwhile to fund it when the total budget is $400M, NASA can get more bang for the buck at the $500M level by putting the money into other technology areas.

Estimated Impact of Technology Budgets on Missions Enabled

This graph relates funding technologies to enabling missions. The vertical (y) axis shows the probability that specific missions would be enabled to TRL 6. For example, at a 10-year budget of $1 billion, Venus Surface Missions has about a 90% probability of achieving TRL 6. Another way of looking at this is that at that funding level, each technology component of Venus Surface Missions would be developed about 90% of the way toward what it needs to achieve TRL 6.

Note that this graph indicates the probability of missions being enabled, based on funding levels for portfolios of technologies, not funding for the missions, themselves. Thus we find that some missions remain at the same probability over a long stretch of budget increases, because the technologies necessary to bring them to a higher probability level are not added to the "optimal portfolio" until the budget reaches a certain level. In some cases, a given mission actually shows less probability of being enabled at a higher budget level. This is because, at the higher funding level, the technology mix is different and may no longer include technologies that are important to that particular mission. For example, Venus Surface Missions shows a higher probability of being enabled at a budget level of about $1.0 billion than at about $1.1 billion.

Conclusions

We have demonstrated a system for determining, in an auditable fashion, optimal portfolios for advanced space technology development across a wide spectrum of missions and technologies. Our system can accept new missions and technologies easily, and can produce a prototyped example rapidly (Phase 1 of this pilot study progressed from a dead start to a conclusion in only four months). As part of this study, we have shown that risk-management software can be included in this analysis like any other technology.

This system is intended to be a valuable tool for decision-makers charged with the difficult task of allocating resources in the most efficient and effective manner possible. The proposed value of this approach is its ability to produce an objective, auditable result which ultimately leads to more-credible investment strategies for the sponsor.

Our study was presented to a NASA-wide workshop held in San Diego, California on April 21-22, 2004, whose participants agreed that our methodology for systematic technology resource allocation does indeed scale up to the Enterprise and Agency levels as described here.

Phase 2 of this study, now in progress, is intended to increase both breadth and depth, including more missions and technologies as well as validated data and robustness of conclusions to a variety of policy options. In Phase 3, we intend to have a real decision-maker apply these techniques in practice (as in our study for the Space Shuttle Life Extension Program and our work for the NASA Space Architect).

For more information, contact: Charles.R.Weisbin@jpl.nasa.gov

Or see the following:

  • C.R. Weisbin, G. Rodriguez, A. Elfes, and J.H. Smith. "Toward a Systematic Approach for Selection of NASA Technology Portfolios," Systems Engineering Journal, Vol. 7, No. 4, pp. 285-302, 2004.



  About | Methodology | Case Studies | Publications & Proceedings | News | Sitemap | Home

PRIVACY / COPYRIGHT IMAGE POLICY CONTACT INFORMATION CREDITS
  NASA Home Page   Primary START Contact: Charles R Weisbin
  Last Updated: January 24, 2013