ETDP study

NASA's Exploration Technology Development Program develops new technologies primarily for the designers of flight systems in the Constellation Systems Program, which oversees NASA's effort to send astronauts to the Moon and beyond.

With three times the number of proposals to consider and many more people involved in the decision-making process, the study for NASA's Exploration Technology Development Program (ETDP) at Langley Research Center was considerably more challenging than that of the Human Research Program (HRP). Nevertheless, it was conducted quite similarly to the HRP study, with very good results.

There was a subtle difference in how ETDP used the START recommendations compared to how HRP used them. Whereas HRP used START's results as a reference and used its own judgment (based on factors not captured by the database) to modify the START rankings, ETDP used the START results to help them evaluate and ultimately to modify their own rankings. ETDP also differed from HRP in the desired product of the analysis. While HRP wanted a list of candidate technologies ranked in order of their final scores, ETDP was interested in assigning each technology to one of two bins: selected or not selected.

The attribute table we compiled in close consultation with ETDP was similar to that of HRP, but with some differences.

"Ratio of SBIR funding to programmatic investment" represents how well the program office currently funds the technology area in question: Is the area well-funded, under funded, or completely unfunded? "In-house advocates / project manager level of interest" gauges the involvement intensity of a manager who has the power to decide whether a technology will be infused into a mission. As in the HRP study, the weighted sum of the attribute scores for each technology was multiplied by its "probability of success" score, as evaluated by a panel of experts, to yield a final score.

We presented an initial result to ETDP management and then, after calibrating the weights and scores based on management feedback, we computed a second ranked list of capabilities. ETDP used the START results as an input in evaluating the independent results of its expert panel, which had sorted the proposals into "accepted" and "rejected" bins.

An analysis of ETDP's results before and after being exposed to START's recommendations reveals that START had a significant impact on the decision-making process. As seen in the table below, the correlation between before and after was only 0.88 for accepted proposals and 0.89 for rejected proposals, indicating that the panel changed its mind after considering START's results. (If START had no impact, ETDP's results would not have changed after the experts considered START's recommendations, and the correlation would be 1.0.)

Proposal subset Kendall tau rank correlation Accepted proposals 0.8839 Rejected proposals 0.8947 Correlation of ETDP panel's decisions before and after considering START recommendations.

At the same time, the fact that the correlation is as high as it is suggests that the START process was quite successful in capturing the criteria on which the panel based its decisions (a major goal of the START process, since it enables quantitative analysis and provides very good traceability), and that there was a high level of agreement between START's recommendations and those of the panel.

Back to START Lite main page

HRP study

SMD analysis