Predicting the Cost of New Technologies

People seeking technology development funding tend to put the best light on their estimates of how much time and money they will require. Add to this trait the fact that technologists and mission designers often have conflicting, unexpressed assumptions about what is required, and you have the makings of costly misunderstandings and cost overruns.

This task concept was to develop a process to generate plausible cost estimates grounded on clear assumptions.

This particular task subset dealt only with technologies up to TRL 6, and so did not include the action described in the lower middle box.

We developed a process for estimating the cost of new technology that included uncertainty and an independent peer review of the estimate. It is based on interviews with technology representatives that focus on cost and performance relationships for each technology:

1. What are the important relationships that influence the cost?
2. What are the development issues?
3. What happens to performance if the cost is higher or lower?
4. What happens to cost if performance is higher or lower?
5. What assumptions underlie the cost estimate?
6. What is the probability of successfully developing the technology?

As a test case, we applied the process to a set of autonomy software technologies for Mars rovers that were the focus of Rover Autonomy Study #1.

The interviews in this case revealed important and subtle factors such as technology interdependencies, resource dependencies, and areas of common problems for the technologies studied.

The third-party review was critical in helping to (1) validate the original prediction, (2) identify missing or redundant cost issues affecting the initial prediction, and (3) determine any adjustments that might need to be made to the original cost estimate.

While the task was to model the relationships between performance, cost, and schedule for autonomy software, the general approach should be extensible to other technologies, including hardware systems.

Stopping Rule

Part of the task was to develop and validate a "stopping rule," a formula that determines at what point diminishing returns make it inadvisable to invest in improving a technology to reduce its failure rate.

We developed an algorithm to improve the cost-effectiveness of the cost estimation process by focusing attention on the technologies with lowest performance and greatest potential benefit. Further study would likely yield a better understanding of the requirements and feasibility of finding the optimal stopping rule.

An optimal $50 million portfolio does not necessarily simply add new technologies to those of a $40 million portfolio. Expanding the budget may make an entirely different set of technologies possible and preferable.

By more reliably predicting the costs of component technologies and considering the interrelationships of their science return, we can help decision-makers to determine the best place to set the cutoff points for their technology budgets.

The above graph shows the probability of completing three tasks to their specified level of performance, at a range of budgets. For example, the probability of completing target handoff rises from about 0.3 at roughly $1.1 million to about 0.95 at a cost of about $1.75 million. The green shading around each budget point indicates the amount of uncertainty in the figure.

This graph illustrates the performance level (measured in the number of Martian days, or sols, that would be saved) one can expect at the budget levels plotted in the previous graph. For target handoff, the number of sols saved increases from about 10 at roughly $1.1 million to about 35 at around $1.75 million. The data for both graphs was derived from interviews with experts.

Together, these two graphs can help a decision-maker to optimize a portfolio.

Suppose he or she has about $2 million to spend on autonomy software technology. Considering the three technologies represented on these graphs, the decision-maker can fund one of three possible portfolios:

1. Camera models and target handoff. But there will only be enough money to fund target handoff to the point where the top graph indicates less than a 0.4 probability of being completed.

2. Target handoff alone, but to the level where the top graph indicates near certainty that it will be completed.

3. Short range path planning, but only to the level where it has around a 0.5 probability of being completed.

The bottom graph tells us that Portfolio #1 will save about 15 sols for the camera models plus about 10 sols for the target handoff, for a total of 25 sols saved. Portfolio #2 would save about 35 sols. Portfolio #3 would save about 11 sols.

All other things being equal, the best return-on-investment would come from portfolio #2, which would save 35 sols with a near-certainty of completion.