Sensor for Hazard Detection and Avoidance

What kind of sensor is best for the Hazard Detection and Avoidance system of a Mars lander?

The Mars Science Laboratory (MSL) mission planned for around 2009 will involve a controlled descent of the lander. It will be essential to avoid landing in a large crater that the accompanying rover would not be able to climb out of, or on a big rock that could cause the lander to touch down in an unstable position and fall over. Avoiding these problems will require a capable Hazard Detection and Avoidance (HDA) system.

We compared 10 sensors for such a system, using both objective criteria and expert opinion. The expert in this particular case was the study's sponsor, though that is not typical.

This study is characterized by its complexity. We evaluated each of the ten sensors according to 26 attributes, with a plethora of different metrics and an abundance of uncertainty. As often happens, many of the attributes were in conflict with each other. Increasing the complexity of a system, for example, may have a beneficial effect on its performance capability, but it is also likely to have an adverse effect on its reliability and cost.

How to juggle so many dissimilar variables? We developed SAGE (Select Alternative by Graded Evaluation), a system for formulating a unitless numerical value for each attribute. This enables us to calculate with all the attributes to arrive at values with which to compare the competing technologies. SAGE serves as an analysis system for a broad range of design trade problems that have too many conflicting attributes to plot simply against each other.

The SAGE Process

We began by identifying the available options. In this case, we had ten sensor candidates to compare: three lidars (sensors that employs laser), three radars, three optical systems, and one thermal system.

Next, we determined the attributes that the decision-maker deemed important. This produced the following chart for each of the ten sensors.

We looked at seven primary attributes: maturity of the technology, system complexity, cost, performance, reliability, robustness, and resource demands. Each of those attributes depends on other attributes, some of which depend on still further attributes. There are a total of 26 lowest-level attributes -- that is, attributes that are not divided into sub-attributes. They are highlighted in yellow.

Preliminary Screening

Next, we obtained values for each attribute (projected cost of testing, amount of mass, etc.). We reviewed the technology candidates to see whether any should be eliminated for failure to meet minimum requirements or because they are "dominated" by other candidates. A sensor is dominated if there is another sensor design that excels it in one or more attributes and equals it in the remaining attributes. In this study, none of the ten sensor candidates was dominated or failed to meet minimum requirements.

Deriving a Unitless Value for Each Attribute

Once the attributes have been established for each sensor candidate, the next critical step is to assign a unitless numerical value to each of them. This solves the "apples and oranges" problem, enabling us to calculate with a mix of dissimilar attributes.

First, each attribute must be assigned a performance level via models, simulation, expert opinion, field tests, and/or past mission results.

For this study, we interviewed an expert, soliciting high, low, and mid-range values for each attribute. For example, to assign a value to volume, we asked for an estimate of the maximum and minimum amount of volume a sensor could occupy (where "minimum" reflects a level at which further reduction would provide negligible benefit), then for a volume that could be included with confidence in the lander based on past experience, and finally for higher and lower amounts that would pose a challenge but probably would be satisfactory.

These five values form the basis of a curve like the one below, which reflects the "ideal" amount of volume (in cubic cm) spread out over the uncertainty about just what that amount would be. Or, to put it another way, the curve reflects the acceptable range of volume for the sensor, given that we can't determine exactly where on the curve the "ideal" amount of volume would be.

Note that the curve for volume falls from left to right, with desirable low volumes high on the curve, and undesirable high volumes low on the curve. For attributes like "robustness," the curve would be reversed, with desirable high levels high on the curve, and undesirable low levels low on the curve.

We plotted the volume of each sensor candidate on this curve, and read its value on the Y axis, where the lowest limits of the curve have a value of 0 and the highest limits have a value of 1. On this sample graph, the triangle represents a 4000 ccm sensor with a value of about 0.7, and the diamond indicates a 10,000 ccm sensor with a value of about 0.05. Note that we now have a value for the mass of each sensor that is not given in cubic centimeters or any other unit.

Utility Function

Not all attributes have equal importance in the judgment of the decision-makers, so our next step was to ask the decision-maker to assign a utility function to each of them, reflecting the importance of each attribute relative to the other attributes in its sub-tree. Numbers within each set of branches must add up to one.

In this study, the first level of attributes (as seen on the above chart) received the following utility functions: 0.15 maturity, 0.10 system complexity, 0.10 cost, 0.15 performance, 0.15 reliability, 0.20 robustness, and 0.15 resource demands. Note that they add up to one.

Each further set of branches received the same treatment. For example, "Resource Demands" branches off into five attributes: Power, Volume, Mass, Processing time, and Field of View (FOV) allowance. The decision-maker is asked to determine how much each of those attributes contributes to the total Resource Demands. If they are all of equal importance, each is assigned the a utility ranking of 0.2 (so that all five add up to 1). If mass were considered about twice as important as anything else in that category, it might be assigned a utility ranking of 0.336, while the four other attributes might each be assigned 0.166 (again, for a category total of 1).

It is important for the decision-maker to give an honest weighting of the importance of each attribute without regard to the quality of that attribute as it applies to a particular technology.

Once we had a utility ranking for each attribute, we used it to weight the unitless numerical value that each attribute received in an earlier step. This produced the attribute's "value function," a number that reflects both how well a sensor performs with regard to that attribute, and the attribute's importance to the overall value of the sensor.

We added the value functions for all 26 attributes to determine the value function for that particular sensor. Ultimately, we had a single number for each sensor that reflected how well that sensor was expected to perform. The following graph shows how the ten sensor candidates in the study compared to each other.

In this bar graph, we see that the second lidar sensor (L2) ranked ahead of its competitors.

Uncertainty

There is usually some level of uncertainty surrounding the input values. We can propagate that uncertainty through to the value function output, to show a range of value functions corresponding to the range of uncertainty in the initial input.

For example, the two bar graphs above on the left show how much uncertainty there was in the cost estimations for each of the sensors in this study. (The top graph is for regional HDA, which is detection and avoidance of craters, and the lower graph is for local HDA, which is detection and avoidance of rocks.) In the top left graph, the "L1" sensor is shown to have an uncertainty level of 24% -- that is, the cost that was used as an input is considered accurate to plus or minus 24%.

The graph on the right shows the impact on each sensor's value function throughout its range of cost uncertainty (in this graph, regional and local HDA are combined). The low end of uncertainty is represented by the bars above the zero line, since lower cost makes for higher value. The high end of uncertainty is shown below the zero line, since higher cost makes for less value.

For L1, the graph shows that if the true cost is at the lowest end of the uncertainty range, its value function is 1. If the true cost is at the uppermost end of its uncertainty range, the value function is about -1.75.

The multi-colored graph shows all three value functions for each sensor. Orange represents the high end of cost uncertainty, yellow is the low end, and red is the cost estimate that was actually inputted into the model.

Validation

The final major step is to validate the results. This study serves as a good example of how the process works when the results are not validated in the first pass.

Based on the original assumptions and judgments that were inputted into the model, our study declared the second lidar sensor (L2) to be superior to its competitors. The study's sponsor, however, thought that the first radar sensor (R1) should have won, prompting him to revisit those assumptions and consider whether they should be revised.

In preparation for this process, we calculated each attribute's sensitivity to its utility function (the attribute's relative importance), attribute values (such as volume in ccm or mass in kg), mu (the level of attribute value deemed satisfactory), and sigma (the amount of uncertainty in establishing mu).

We determined which parameters had the most impact in producing L2 as the result, compared to R1. The attributes with high sensitivities identify the principal trade space of the design selection.

High sensitivity means that a small change in that initial value (utility, mu, etc.) would make a big difference in the sensor's value function -- and potentially its ranking against the other sensors. Conversely, low sensitivity means that even a big change in the initial value would make little difference to its ranking.

We were able to guide the sponsor -- who served as the expert in this particular case -- to the assumptions that had the highest sensitivity, and which therefore would have a large impact on the results for only a small change in the input.

The above tables show the attributes most sensitive to changes in the mu and sigma functions, which indicate the level of volume, cost, etc. that the expert deemed satisfactory, and the amount of uncertainty in establishing those levels. In the right-hand column, a negative value corresponds to a parameter where L2 is better than R1. Decreasing the importance of that parameter will decrease the value of L2 relative to R1.

In this study, we determined that the values assigned to mu and sigma were the most fertile fields for possible changes. Our expert sponsor made slight changes to 12 of the parameters in those two categories.

With the new inputs for mu, the model turned out the above ranking. The sponsor's initial choice, R1, is now in first place, and the sponsor has new insight into the assumptions that are implicit in that choice.