1.1.2 Defining the operating environment

Author: Dr Richard Hawkins, Assuring Autonomy International Programme

Although the safety of a road vehicle must be assured under all conditions, there will often be limits upon the conditions under which the vehicle can be demonstrated to operate safely in an autonomous manner. These limits may result from the limitations on the conditions under which:

• the required level of understanding of the environment can be achieved (resulting from limitations on the sensors or perception algorithms) – for example the cameras installed in the vehicle may function poorly in the presence of bright sunlight at oblique angles. It is important in such circumstances to include detail of the lighting conditions in the Operational Design Domain (ODD) description
• the vehicle is able to act reliably (resulting from limitations of the vehicle driving performance) – for example the stopping distances of the vehicle may be significantly affected by road surface and road conditions (such as water or ice)

It is crucial therefore that such limits are explicitly defined. Once this is done, the safety of the vehicle must be demonstrated by showing that:

• the vehicle will operate safely autonomously within the defined conditions
• the vehicle detects when the conditions no longer hold – see Objective 1.3
• if the conditions no longer hold, the vehicle achieves a safe state (through handing control back to a human driver (see Objective 1.2.1) or reaching a minimum risk condition (see Objective 3.2))

If it is not possible to demonstrate sufficient safety assurance within the defined conditions, then further restrictions may need to be defined.

For automotive systems, the environment in which the vehicle is designed to operate is referred to as the Operational Design Domain (ODD). An ODD is defined by SAE J3016 [1] as the operating conditions under which a vehicle or driving feature is specifically designed to function. This includes, but is not limited to: 

• environmental restrictions
• geographical restrictions
• time-of-day restrictions
• required presence or absence of traffic or roadway characteristics

By defining the scope of the operating context for the vehicle, the ODD can be used to help define test environments (see Objective 2.1), and for assessing the coverage of datasets (used for training or validation (see Objective 2.3.1)). The ODD can also be used in defining simulation environments (see Objective 2.7). It is important therefore that the ODD defines all the conditions that might impact upon the safe behaviour of the vehicle. If important conditions are not included, they may not be considered as part of the assurance process and could therefore result in unanticipated hazardous situations. There exist a number of proposals for what information is required to be considered as part of the ODD, these are considered in the next section.

Proposed ODD Taxonomies

NHTSA [2] have defined and categorised an ODD taxonomy based on a review of over 50 sources of literature in automotive and other domains. The taxonomy is intended to be descriptive, recognising that other organisations of the elements are possible. The overall taxonomy structure is shown in Figure 1.

Figure 1: ODD top-level categories and immediate sub-categories taken from [2]

More detail is provided within each sub-element, as illustrated in Figure 2.

Figure 2 - Example of Hierarchical Levels within the Environmental Conditions Category taken from [2]

The report also provides a set of sample baseline ODDs for different automated driving features, such as those shown below for an Automated Highway Drive (HWD) function.

Figure 3 – example baseline ODD for HWD function

The Waterloo Intelligent Systems Engineering Lab have also proposed an ODD ontology [3] based upon an extensive review of standards and scientific literature. At the top level, the ontology is organized into five areas:

• road structure
• road users, including vehicles, cyclists, and pedestrians
• animals
• other obstacles
• environmental conditions

For each of these areas a very detailed classification and description is provided. For example, environmental conditions are decomposed to atmospheric conditions, lighting conditions and weather-related road surface conditions. The atmospheric conditions are then defined in terms of:

• temperature
• visibility
• wind
• clouds
• precipitation
• other atmospheric obscuration

Each of these is described in detail, such as is shown for cloud cover in Figure 4.

Figure 4 - Description of cloud conditions extracted from [3]

The appropriateness of developing the ODD description to this level of detail must be carefully considered. This is discussed in more detail below. It should also be noted that a lot of the information in the report, although useful, goes beyond defining the operating environment (such as information on driver behaviour models).

[4] provides a list of ODD factors that have been found to be relevant to characterising the operational environment. It is proposed that this could form a starting point for a publicly available master list of considerations. It is proposed that the ODD should include at least a description of:

• Operational terrain, and associated location-dependent characteristics
• Environmental and weather conditions
• Operational infrastructure
• Rules of engagement and expectations for interaction with the environment and other aspects of the operational state space (including traffic laws and social norms)
• Considerations for deployment to multiple regions/countries
• Communication modes
• Availability and freshness of infrastructure characterization data
• Expected distributions of operational state space elements (rare but in-scope elements such as toll booths or police traffic stops)

This includes many of the elements already discussed. What stands out is the explicit inclusion of infrastructure and communication elements. These are important constraints for many autonomous road vehicles that should be considered within the ODD description.

Validating the sufficiency of the defined operating environment

In assessing the sufficiency of the ODD description there are two main considerations. Firstly, whether the ODD provide sufficient coverage of all the important aspects of the operating environment. Secondly, whether the ODD is described in sufficient detail.

The publically available ODD ontologies, such as those discussed above, provide a useful resource for helping to ensure that important information relevant to the operating environment is not missed from the ODD description. In particular they provide a way of checking that an ODD description is consistent with current best practice, and doesn’t miss any known issues. It is reassuring therefore that there exists substantial cross-over between ontologies. However, none of these ontologies would claim to guarantee completeness. The sufficiency of the ODD description can therefore only be assessed on a case by case basis.

It is clearly possible to provide very detailed descriptions of the operating environment. Detailed ODD descriptions enable very precise understanding of the operating conditions, but also potentially become very large and intractable. This can be a difficult balance to strike. One way of addressing this can be to adopt an iterative approach to developing the ODD. Initially a relatively course grained definition can be used to ensure coverage of the operating environment. Each element can then be further refined to explore whether additional detail impacts upon the vehicle. It is also possible to use operational experience to inform areas of the ODD where particular detail is required (for example the performance of particular sensors may be seen to be sensitive to particular lighting conditions, leading to a more detailed description of that aspect of the ODD).

Summary of approach

1. Identify the elements of the operating environment relevant to the vehicle operation
2. Define the ODD to a suitable level of detail
3. Validate the sufficiency of the defined ODD
4. Continue to review and update the ODD throughout the system lifecycle. Define additional conditions as required.

References

• [1] SAE International, 2014. Standard J3016, Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems.
• [2] Thorn, E., Kimmel, S.C., Chaka, M. and Hamilton, B.A., 2018. A Framework for Automated Driving System Testable Cases and Scenarios (No. DOT HS 812 623). United States. Department of Transportation. National Highway Traffic Safety Administration.
• [3] Czarnecki, K., 2018. Operational World Model Ontology for Automated Driving Systems–Part 1 and 2. Waterloo Intelligent Systems Engineering Lab (WISE) Report.
• [4] Koopman, P., Fratrik, F., How Many Operational Design Domains, Objects, and Events, Safe AI 2019: AAAI Workshop on Artificial Intelligence Safety, Jan 27, 2019