User interface considerations for the control and tasking of multiple autonomous vehicles for facilities like the Goodyear Proving Grounds

Paul J Lewis^a, Joseph F. Harrison^b, Sarah A. Gray^c
Autonomous Solutions Inc. 1946 South 1600 West, Young Ward, UT, USA 84339

ABSTRACT

One of the largest factors relating to the commercial success of unmanned vehicles will be ease of use. A man machine interface (MMI) with the goal of allowing a user to easily task, monitor, and control multiple vehicles can benefit from several advancements in human machine interaction from both the research and commercial sectors. This paper focuses on the design considerations of an MMI that balances the complexity inherent to the control of multiple autonomous vehicles with the simplicity required for commercialization. It also profiles MARS, an MMI that Autonomous Solutions Incorporated has developed to facilitate the commercialization of automated test vehicles, and discusses an example applicable to the Goodyear Proving Grounds facility.

1. INTRODUCTION

From the time robots were first conceived, man has been confronted with the challenges of interacting with them in an intuitive and efficient manner. Unmanned vehicles, a specific subclass of mobile robots, present a unique set of interaction issues that must be addressed. To be useful, these vehicles must have the ability to perform a task, or set of tasks, assigned by a user. Man machine interfaces (MMI's) are designed to facilitate the communication necessary between robot and human for these tasks to be accomplished. MMI's generally gravitate toward one of two extremes: terse interfaces that require extensive training and patience, or overly simplistic interfaces that support little more than a handful of "canned" tasks. For an MMI to be viable for large-scale public use, it must be simple enough for humans to use, yet sophisticated enough to support complex machine interaction. Tasking and control of autonomous vehicles should be easy enough for the average consumer. The average consumer might be technically competent, but they do not have an engineering degree. This paper will discuss general considerations in developing a sophisticated yet simple MMI for unmanned vehicles as well as focus on specific considerations as they relate to the Goodyear Proving Grounds application.

2. VEHICLE MODES OF OPERATION

The complexity of an MMI is related to the modes of operation of the vehicles for which it is designed. There are generally four classes of operation related to robotic vehicles: Autonomous, Semi-Autonomous, Manual, and Operator Augmentation. Although this paper focuses on autonomous and semi-autonomous vehicles, a basic understanding of the different operational modes is important due to the varying degrees of complexity imposed on the MMI when working with vehicles in each mode. In addition, some vehicles may be capable of multiple operation modes further increasing the complexity of the system.

2.1 Manual

In this mode, vehicles are manually controlled from a remote location (i.e. tele-operated). They are generally standard vehicles modified to perform tasks for which humans are ill-suited. MMI's for this mode of operation can be simple consisting mostly of configuration, limited control, and monitoring.

2.2 Operator augmentation

In this mode, vehicles enhance human performance by automating difficult or exhausting tasks. The machine is not capable of operating without the presence of a human. MMI's for this mode of operation require all the capabilities of manual mode plus additional configuration for augmentation parameters.

2.3 Semi-autonomous

In semi-autonomous mode, vehicles are capable of performing complicated unmanned tasks but require some human interaction (e.g. how to react to an obstacle). MMI's for semi-autonomous vehicles might include support for the previous two modes, but would need to add support for the creation and assignment of the more complex tasks available in this mode.

2.4 Autonomous

In autonomous mode, vehicles are agents unto themselves. They receive tasks from a human, but require little or no interaction to perform them. Autonomous vehicles often use sensors to gather information about their environment and make navigation decisions accordingly. In addition to semi-autonomous capabilities, MMI's for autonomous vehicles require a greater degree of configuration support to determine desired vehicle responses to external stimuli.

3. MMI CONSIDERATIONS FOR UNMANNED VEHICLES

There are several considerations to address when developing a general purpose MMI for unmanned vehicles. Each consideration has an effect on the balance between complexity and efficiency in the MMI.

3.1 Ease of use

Users wish to accomplish tasks with a minimum of time and work on their part. MMI's for autonomous vehicle control must take this into account and allow tasks to be assigned to vehicles with minimal effort.

3.2 Interface consistency

The time required for a user to become familiar with an MMI can be drastically reduced if its subsystems and tools are consistent in their interface. If the procedure for performing one task (e.g. drawing a shape on the map) is consistent with the procedure for performing another task (e.g. drawing a path to be traversed), then a user need only learn one of them to become proficient with both.

3.3 Appropriate metaphors

By using interface metaphors that are familiar to the user, tasks and procedures can be made more intuitive. If metaphors such as vehicles and paths are introduced, they should be consistent with the domain in which they are used. With appropriate names, tools like "Map Builder" and "Path Builder" require little explanation as to their purpose.

3.4 Tooltips, messages, visual hints, and command menu navigation

By providing feedback in the form of tooltips and helpful user messages, an MMI can be made significantly more user-friendly. Visual hints that show the result of an operation before it happens can be beneficial in making the user feel more at ease.

3.5 High level control vs. micro-management

Users have varying control needs based on experience and comfort level with the system. By endowing the user interface or vehicle with the intelligence needed to make decisions about how to accomplish a task, the user is freed from having to specify every detail, but at the same time gives up some amount of control. Of course, the user should be relieved from tedious tasks but not prevented from accessing the minutiae of the system.

3.6 Heterogeneous vehicles

The number of autonomous vehicles has increased dramatically over the years. It is becoming more important to be able to control heterogeneous vehicles from diverse domains (aerial, underwater, ground) and suppliers from a single, user-friendly MMI. Some vehicles are specialized machines with a single purpose, while others are much more generic and may have many capabilities. A well designed MMI should be capable of tasking individual vehicles based on their specific capabilities and groups of vehicles based on their common or combined capabilities. Each of these capabilities should be clearly and uniformly presented to the user in an intuitive manner.

3.7 Multiple vehicles

In order to support multiple vehicles, MMI's inherently become more complex. Some of the complexities include: organizing and displaying vehicle information, tasking individual vehicles vs. groups of vehicles, coordinating vehicles to optimize performance while avoiding collisions, and re-tasking of vehicles. A well designed MMI will support the increased functionality of a multiple-vehicle system while minimizing the impact on the user. When a single user can command, control, and monitor multiple unmanned vehicles simultaneously the true value of autonomous vehicles will be realized.

3.8 Safety

Last but not least, an MMI should support control and monitoring of vehicle safety systems as well as provide safety systems of its own. Safety systems may include emergency stops, real-time vehicle position updates, live video feeds, redundant heartbeat signals, obstacle detection and avoidance, and error correction. It should also allow a user to place the vehicle in a safe state prior to entering the work area of the unmanned vehicle.

4. REAL-TIME STRATEGY GAMES AS A MODEL

As a highly competitive industry with a rapid development cycle, the computer game industry has made significant advances in the area of user interface design. Of the many genres in the field of computer gaming, one in particular, Real-Time Strategy (RTS) games, provides a pertinent model for efficient MMIs.

4.1 Interface elements

Many of the elements of RTS game interfaces correlate directly to the design of MMI's for the control of heterogeneous autonomous vehicles. Some of these elements are as follows:

4.1.1 Main map panel

This panel, covering most of the screen, is the primary means of viewing a section of the map. It generally supports zoom-in, zoom-out, and pan functions.

4.1.2 Mini-map

This panel, much smaller than the Main Map Panel and usually located in a corner of the screen, provides an overview of the entire map. Its main purpose is to show where the Main Map Panel is focused and allow quick refocusing to a different area of the map.

4.1.3 Units

Units (e.g. autonomous vehicles) are shown in both the Main Map Panel and the Mini-Map. They are selectable in the Main Map Panel where they can be assigned tasks.

4.1.4 Command panel

The Command Panel is a button-based menu system through which the user accesses most or all of the program's functionality. To help achieve the dual goals of simplicity and sophistication, these menu systems are often hierarchical, where clicking one button brings up a sub-menu of related buttons.

When a unit is selected, the Command Panel displays buttons specific to the unit and its capabilities. These buttons are used to assign tasks to the unit. When multiple units are selected, the Command Panel displays buttons that are common to all selected units.

The following screen-shot taken from the popular game WarCraft III by Blizzard Entertainment1 shows some of the RTS features previously discussed (Fig. 1).

Figure 1: WarCraft III screenshot

4.2 Control scheme

RTS games adhere to a simple and intuitive control scheme:

1. Select the unit(s) to be tasked
2. Select the action(s) to be performed
3. Select the object(s) on which the unit(s) will perform the action(s)

This scheme gives the user maximum control over tasks assigned to individual vehicles.

5. MMI CONSIDERATIONS SPECIFIC TO GOODYEAR PROVING GROUNDS

There are several enhancements that can increase the value of the MMI in the Goodyear Proving Grounds application.

5.1 Area of operation

The Goodyear Proving Grounds is a large test site. With such a large working area, it is important that the MMI allow the user to specify areas the vehicle is not allowed to enter, and boundaries that the vehicle may not leave.

5.2 Multi-vehicle, single operator

Automation of a large test facility will increase the efficiency of the testing as well as reduce operating costs. Autonomous Solutions, Incorporated is currently automating a single vehicle for autonomous operation at the Goodyear Proving Grounds. The MMI is designed to accommodate multiple autonomous vehicles, allowing for a multiple unmanned vehicle test site where a single operator controls many autonomous vehicles, thereby increasing the value of the system.

5.3 User training

In order for the MMI to provide customer value, it must be easy enough for the intended user base. Although many of the employees at the facility are engineers, the MMI should be usable by non-engineering personnel after brief training. In order to facilitate training, Goodyear was provided with a pre-release version of the MMI and a training manual prior to delivery of the vehicle. In addition, a vehicle simulator was provided to allow the users to practice tasking the vehicle.

5.4 Usability enhancements

After working with the MMI, Goodyear and other customers identified several usability enhancements that they felt would benefit their facilities. Among the suggested changes were alternate nomenclature, shortened command hierarchy, saving of UI preferences, and a "favorite commands" feature.

5.5 Vehicle operation modes

In addition to autonomous/semi-autonomous operation, the vehicle should be capable of manual OEM operation. This allows the vehicle to be used for one-time tasks, or tasks that are currently not well suited to autonomous operation (e.g. hooking-up implements). The MMI should not allow vehicles in manual mode to be autonomously tasked and should not limit their area of operation.

5.6 Data logging

Due to test repeatability, most testing facilities have a heightened need for data logging capabilities. By providing an interface designed specifically to allow the user to configure real-time logging of vehicle parameters and additional analog and digital input devices, engineers can gain valuable insight.

6. MISSION ASSIGNMENT AND RECONNAISSANCE SYSTEM

Mission Assignment and Reconnaissance System (MARS)^d is the name of the MMI implemented for the Goodyear Proving Grounds application. In order to better understand the design decisions that have influenced its development, it is helpful to look at the evolution of the software.

6.1 Origins

MARS has been in development for a number of years and has been used on a number of vehicles with varying levels of functionality. The original software has roots that go back to the Center for Self-Organizing and Intelligent Systems (CSOIS) at Utah State University.

"The Center for Self-Organizing and Intelligent Systems (CSOIS) is a multi-disciplinary research group at Utah State University (USU) that focuses on the design, development, and implementation of intelligent, autonomous mechatronic systems, with a recent focus on ground vehicles and robotics. CSOIS research advances the state-of-the-art in the theory, development, and application of systems that need advanced automation, autonomous operation or behavior, and intelligent decision-making and learning to achieve their objectives."²

The MMI developed at CSOIS was modified and upgraded for use in multiple research projects funded by INEEL, TACOM/TARDEC, and various commercial companies. Some of the features that are present in the current implementation of the MMI got their start from these early projects. The picture below shows one of the earlier MMI's developed at CSOIS (Fig. 2). This interface contained a large area map and allowed the user to issue commands to the vehicles from the menu bar. Once the robot had been tasked the planned path was displayed. The system was capable of working with multiple homogenous vehicles, utilizing roads, and optimizing task loads in the event of a vehicle breakdown.

Figure 2: Early CSOIS MMI

Each vehicle project required a new version of the MMI software. This allowed the previous projects to remain unchanged while providing a jump-start for new development. However, this also meant that older projects were always a step behind in terms of features and improvements made to the MMI.

As MMI development progressed, more emphasis was placed on ease-of-use and tailoring the interface to individual vehicle(s). Although this increased MMI customization for user familiarity, it progressed the MMI farther away from a general purpose system capable of working with multiple heterogeneous unmanned vehicles. The MMI below was specifically tailored for a customer to represent what a typical user would see if they were on the machine (Fig. 3).

Figure 3: MMI customized for the vehicle

The interface had a simple and intuitive control scheme:

1. Select an action you want to happen
2. Select what you want it to happen to
3. Press Start

This simplistic approach freed the user from worrying about the capabilities of a specific vehicle and allowed the computer to task the vehicle most capable of performing the action.

6.2 Software system overview

When the ASI team spun-off from USU in November, 2000 it was with the dream of bringing the IP developed at CSOIS up to a level of robustness sufficient for commercial application. We knew that an effective MMI would be critical to reaching this goal. Early in the design phase, we decided that by following the RTS control scheme, we could create an MMI that would most effectively balance usability with sophistication.

The underlying core of the autonomous vehicle software is the Joint Architecture for Unmanned Systems (JAUS), the official U.S. Military standard for unmanned systems.3 At a basic level, JAUS is a messaging standard providing a structure that enables components as diverse as user interfaces, vehicle controllers, positioning systems, safety systems, and detection systems to be used interchangeably. JAUS is built upon the following four pillars:

1. Vehicle platform independence
2. Mission independence/isolation
3. Computer hardware independence
4. Technology independence

The software system is composed of two JAUS 2.0 subsystems: an Operator Control Unit (OCU) and a Vehicle Control Unit (VCU). The OCU software resides on a base station PC and is responsible for tasking the unmanned vehicle through a user-friendly interface and displaying real-time progress and status. The high level tasks defined by the user (drive to point A, drive path B, etc) are decomposed into lower level tasks and sent to the VCU. The VCU then interprets the low level tasks and controls the vehicle hardware. The following diagram shows an overview of the software system (Fig. 4).

Figure 4: Software system overview

MARS is a vital part of the overall system software, serving as the sole interface between man and machine. MARS is a powerful interactive tool designed to facilitate the creation and assignment of autonomous tasks.

6.3 Control scheme

MARS adheres to the simple and intuitive control scheme used in most RTS games and extends it with a single step:

1. Select the unit(s) to be tasked
2. Select the action(s) to be performed
3. Select the object(s) on which the unit(s) will perform the action(s)
4. Press "start"

This scheme gives the user more fine grained control over which vehicle does which tasks than the old control scheme used by the previous MMI. After tasks are assigned to the vehicles, the planned path is displayed on the main map panel. Autonomous operation does not begin until the "start" button is pressed, allowing the user to visually verify the intended path.

6.4 MARS features

MARS contains a wide variety of tools for working with autonomous vehicles. Several of its features are highlighted below.

6.4.1 Map Builder

Map Builder is used to create a map of the geographical area in which the vehicle will be operating. Map Builder includes tools for drawing, editing, and importing shapes that represent areas in the map (Fig. 5). Map areas may be marked as obstacles to prevent the vehicle from passing through them.

Figure 5: Creating map areas with Map Builder

6.4.2 Waypoint goals

A waypoint goal is a point on the map to which the vehicle must travel. Waypoint goals may contain actions to be performed when the vehicle reaches the destination point.

6.4.3 Coverage goals

A coverage goal is an area on the map which the vehicle must cover using a pattern such as a back-and-forth sweep, contour, spiral, etc. Coverage goals may contain actions to be performed over the specified area.

6.4.4 Path Builder

Path Builder is an extremely efficient tool for creating drivable paths for an autonomous vehicle (Fig. 6). For a path to be drivable, all segments must be tangentially connected, and the path must contain no arcs smaller than the vehicle's minimum turn radius. Creating a drivable path using conventional drawing tools can be laborious and difficult. By using the constraints of the vehicle to ensure drivability, Path Builder assists the user in creating a drivable path. In addition, tools are included that allow the user to place actions along the path.

Figure 6: Creating a path with Path Builder

6.4.5 Path Trainer

The Path Trainer tool allows the user to log the path that the vehicle drives and any actions performed in the process (Fig. 7). By importing the saved path into Path Builder, the vehicle can be commanded to repeat the path. Similarly, by importing the saved path into Map Builder, the data can be used to create map shapes.

Figure 7: Path Trainer

6.4.6 Data Logger Component

The Data Logger Component (DLC) is a dual-purpose tool, logging data as well as triggering actions based on value limits (Fig. 8). The DLC is divided into channels, where each channel represents a sensor. A sensor can be one of the pre-defined sensors available to the vehicle or an off-the-shelf sensor (analog or digital). Each channel has an adjustable sampling rate and upper/lower limits with corresponding actions. Actions, which are triggered when the value from a sensor goes below the lower limit or above the upper limit, include E-stop, pause mission, cancel mission, and notify user.

Figure 8: Data Logger Component (DLC)

7. CONCLUSION

The Mission Assignment and Reconnaissance System is an intuitive and easy to use MMI for the tasking and monitoring of multiple heterogeneous unmanned vehicles. It has been developed and refined over several years with the goal of providing an MMI robust enough for commercial use. Once the MMI has proven its worth to customers like the Goodyear Proving Grounds, it is anticipated that the system will be marketed to other groups with similar needs. Any application involving dull, dirty, or dangerous work is a prime candidate for unmanned operation and can benefit from an MMI that balances simplicity and sophistication.

REFERENCES

1. Blizzard Entertainment website, www.blizzard.com, Blizzard Entertainment, 2003
2. CSOIS website, www.csois.usu.edu/about/ Utah State University, 2003
3. For more information on JAUS visit the JAUS web site

NOTES

a. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ; phone (435) 755-2980 ext 25; fax (435) 752-0541; www.autonomoussolutions.com
b. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ; phone (435) 755-2980 ext 21; fax (435) 752-0541; www.autonomoussolutions.com
c. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ; phone (435) 755-2980 ext 27; fax (435) 752-0541; www.autonomoussolutions.com
d. Not to be confused with DARPA's "Mobile Autonomous Robot Software" (MARS)