The cockpit assistant system CASSY
as an on-board player in the ATM environment

Reiner Onken
Universität der Bundeswehr München

Summary

This paper presents a concept which warrants a highest possible degree of situation awareness and efficient man-machine interaction on the flight deck, not being confined to the cockpit domain, though. This concept offers the solution to counteract the possible negative consequences of certain flight situations, which are, usually not known in advance, susceptible to pilot errors. It is founded on significant advances in cognitive system engineering in order to accomplish and really warrant complementary deployment of automation technology at the pilot's working station in favor of flight safety and mission effectiveness. These technologies enable a cockpit automation in order to systematically comply with the requirements of ‘Human-Centered Automation (HCA)’. They also allow to quantify at which degree these requirements are met. The underlying approach behind the concept will be illustrated in this paper by the functional concept and development of the cockpit assistant system CASSY. It has been extensively tested in flight simulators and has been successfully field tested with the ATTAS (Advanced Technologies Test Aircraft System) of the DLR. Some of the results of these flight trials will also be presented in this paper.

This demonstrates that the time has come where interaction between the human team and cockpit information systems have no longer to be designed on a vague basis of specifications. The advances in technology provide the necessary basis to systematically reflect requirements for human-centered automation into clear-cut specifications and system design. Therefore, this paper also presents recommendations for how to proceed in order to amend working positions of humans in ATM, including air traffic service providers and airline operation centers.

1 Introduction

Advances in electronics and computer technology have a profound effect on modern aircraft and aviation as such. A quantum leap will occur, when CNS/ATM, as expected for the near future, will allow for Area Navigation and ‘Free Flight’. On the flight deck computer controlled electronic displays and avionic devices are common features. Powerful computer technology is involved in the signal and information processing to generate information packages for the aircrew and furnish display formats and to transmit signals to and from control devices and the remaining avionic components.

However, it is not sufficient to just computerize the crew station the way it is done today. To let the crew station developments keep pace with the developments in CNS/ATM, is not just a matter of introducing new communication links, bringing more information about traffic and navigational data and their automated processing into the cockpit. More than ever before, the interaction between the human cockpit team and the systems the team has to deal with is becoming the crucial safety factor. Therefore, another profound change has to occur in parallel:

To incorporate cognitive systems in the cockpit, which are capable of processing abstract human-like knowledge

• to independently assess necessary situation-relevant information about mission goals, aircraft environment, aircraft systems and aircrew
• to understand the flight situation
• to independently interprete the flight situation in the light of the goals of the flight mission
• to support necessary replanning and decision making,
• to know which information the crew needs,
• to detect pilots' intents and possible errors,

and
to introduce human-like communication initiatives by the cockpit systems to ensure that the pilot's situation awareness is evened up with what is detected as conflicts or opportunities by the systems, not to leave the pilot alone solely with presentations which do not care about what the pilot has understood about the situation and what he actually perceives or does not perceive.

This is the change needed in the crew station which will provide really effective and significant increase of safety besides mission effectiveness. For the sake of safety, only on the basis of these capabilities one can effectively and pointedly account for both at the same time,

• assisting the human pilot to let him play his peculiar excellencies most satisfactorily and
• compensating for the deplorable but indisputable fact that the human pilot is virtually not able to assure one hundred percent safety.

It is known since long that erratic human behaviour is a contributing factor in about 75% and more of all accidents in civil aviation. This is too high an amount for the future ‘Free Flight’ environment and projects for very large aircraft, for which, as some people claim, safety has to be increased considerably (up to one order of magnitude). It is fair to say that these human errors are caused by some kind of overdemand of the pilot resources, either clearly realized by the pilot as an overdemand or not even noticed as such until it is too late. In this context, overdemands are considered as describing the situation when potential resulting human failings are imminent because of inherent human deficiencies in sensory, cognitive and effectory capabilities and performance. New types of latent overtaxing-prone situations appeared with the increase of automated functions, in particular with respect to failings in situation awareness, because advances in cognitive engineering methods were not sufficiently developed yet. Recent accidents of commercial aircraft with state-of-the-art cockpit automation provided some evidence for this particular consequence, although accident rates seem to be less for modern aircraft compared to older generation aircraft. Nevertheless, the overall accident rate has not decreased for many years.

Figure1. Flight Guidance and Control Todays

This does not mean that automation as such is causing this mismatch. It is the way automation is implemented and to what extent and by which interaction-guidelines machine functionality is designed for effective function sharing between man and machine, not to replace the human pilot rather than to support the pilot along the ideas of human-centered automation.

To think about the next generation of cockpit-avionics means to be aware of the even more complex but nevertheless equally crucial role which rests with the crew in the light of the aforementioned machine capabilities and to be conscious of which basic requirements for automation have to be met to comply with flight safety and mission effectiveness demands. That means, also the concept of function allocation to both aircrew and machine has to be reconsidered in this context.

Figure 1 illustrates, which are the tasks the crew is trained to perfom, compared to those functions aircraft systems are designed for. There are those functions, which have to be activated by the crew and thereby to be allocated in order to carry out certain tasks on request and in place of the crew and there are also those machine functions (usually not considered under the aspect of function allocation), which are permanently turned on like the basic cockpit instrumentation and actuator machinery for power amplification.

Function allocation is usually seen as deciding by design when certain pilot tasks are to be substituted by the machine. Figure 1 underscores this concept, illustrating that the machine is involved only in very confined special sections of the full spectrum of tasks to be carried out, in particular, where allocation of functions either to the pilot or to the machine in a clear sequential order is technically feasible. That means that proper compliance of the machine functions with the overall flight task depends on proper take-over procedures in both directions and proper setting of the machine function when activated in the context of the actual situation. It is important to realize that this setting has to be provided reliably, either by design or by the pilot crew. Since there is no capability designed into the machine functions, so far, of becoming aware of the situational context on its own, the pilot crew is burdened to take care of feeding the machine with all necessary information. This can be tedious and difficult. For this task, the pilot crew might need a high degree of insight into the internal processes of the machine function. Moreover, the task can be even impossible under certain circumstances. This might lead to fundamental problems.

The following brief discussion of a US investigation (Wise et al., 1993), one among many others, on the interaction of the pilot crew with the flight management system (FMS), the most advanced cockpit system in state-of-the-art flight decks, illustrates this further.

The FMS receives information about the actual flight, including data about the destination, the flight plan to the destination with way points and altitudes, weather information and weight of load. When these informations are keyed into the system by the crew, which can become a significant interactive effort, the FMS function can be initialised. From then on the aircraft can fly autonomously unless no changes of inputs have to be keyed in because of unexpected encounters in the overall flight situation. The conclusion of the investigation was that the pilots like to make use of the automatic functionality of the FMS, however they run into difficulties in time-critical situations with unforeseen constraint impacts like new ATC instructions. For these situations there is not sufficient spare time for the necessary inputs and the interpretation of computational results as delivered by the FMS. These are the situations when the pilots might be left on their own with questions like

what is it doing?

why did it do that?

what will it do next? Or

how did it ever get in that mode?

Thus, the FMS is usually turned off just at situations when the pilots starvingly look for assistance (Heldt, 1993).These obvious deficiencies clearly indicate that the FMS at this stage of automation, being very well received in some aspects, is not leading the way to what really is wanted for mission effectiveness and flight safety in higher workload situations. Certain principles of securing aircrew situation awareness have come somewhat out of sight.

Therefore, in the following, basic requirements will be established in order to avoid these problems, and a new design approach will be described.

Also application domains outside the cockpit such as working sites for human operators in the ATM environment have to be considered for the introduction of cognitive interaction systems in the future. Other safety critical application domains like the operation of nuclear power plants are already somewhat ahead with the introduction of cognitive engineering for situation awareness enhancement (Proc., 1994; Proc., 1996). Even the road vehicle manufacturers are well underway of preparing for a corresponding change in their products during the next decade. Therefore, it is due time, now, to reconsider the basic requirements for machine support in the aircraft cockpit and ATM environment, in particular regarding situation assesssment tasks of the human operator.

2 Basic requirements for flight deck automation

New ways of automation have to be established. It is not surprising that increases in automation without thinking about new ways of functional share between aircrew and machine imply increased potential of new types of crew overtaxing and resulting human failures, i.e. mission hazards. A good starting point is a top down structuring of design requirements, taking into account the aspects of human-centered automation in a systematic way.

There are a great number of well-formulated requirements at hand for man-machine interaction in the cockpit, including those for ‘human-centered automation’ (Billings, 1991). However, in order to systematically merge future automation into what is really wanted with regard to flight safety and mission effectiveness, it should be possible to assess how much certain individual requirements from the long list of existing ones contribute to the design goals, and what are the interdependencies. This is extremely important, in particular, when trade-offs are necessary for any reason and priorities are to be defined.

Therefore, a top down structure of as few as possible basic requirements is needed which will be described in the following, easing the engineering task of converting the requirements into a technical product. In order to resolve this problem, the objective of automation has to be restated in general terms: Simply, the objective is to avoid overtaxing of the cockpit crew and still to increase mission effectiveness. That means that the demands on the cockpit crew have to be kept on a normal level for all situations and situation-dependent tasks. The aircrew task domains to be considered are flight control, navigation, communication and system handling and the task categories under these domains are

• situation assessment,
• planning and decision making and
• plan execution.

For these task categories the following priority list in terms of a hierarchy of two levels of basic requirements can be established (Onken, 1993). These requirements are essentially equivalent to the requirements for human-centered automation as stated in (Billings, 1991), however, they are structured differently in favor of the engineering point of view with respect of mechanisation by use of cognitive engineering. They can be formulated as stated in the following:

1. To avoid overcharge of the crew in situation assessment, the top requirement
BASIC REQUIREMENT (1) should be met by cognitive engineering, i.e.:

Within the presentation of the full picture of the flight situation it must be ensured that the attention of the cockpit crew is guided towards the objectively most urgent task or subtask of that situation.

2. In order to avoid or decrease overcharge of the crew in planning/decision making and plan execution, as a subordinate requirement

BASIC REQUIREMENT (2) can be formulated:

If basic requirement (1) is met, and if there still comes up a situation with overcharge of the cockpit crew (in planning or plan execution), then this situation has to be transferred - by use of technical means - into a situation which can be handled by the crew in a normal manner.

With respect to basic requirement (2), cognitive engineering is to ensure identifying the situation, when an initiative for action from the side of the technical system is apt because of overdemand on the crew.

This particular top down formulation of requirements for human-centered automation distinctly makes clear that whatever technical specifications are made for systems in support for the cockpit crew, they are questionable if the specification for the situation assessment capability of the support system (Basic requirement (1)), including the assessment of the crew's situation, is too neglectful and sloppy. How can the support system work on directing the crew's attention, if it cannot assess the global situation on its own? If the system is not able to understand the underlying situation, it might work on the basis of wrong assumptions! May be those of the designer who was unable to anticipate all possible situations. Thus, if the specification fails with regard to basic requirement (1), this cannot be compensated by whatever automated support designed to comply with requirement (2) only.Unfortunately, this inadequacy by disregarding basic requirement (1) usually was the case in the past, essentially because the technical means were not available for comprehensive situation assessment by the machine. Prevention of overcharges concerning situation assessment was not worked into the specification in the systematic manner as it is suggested by basic requirement (1).
Basic requirement (1), in fact, compulsorily leads to the full set of specifications which in turn can be used to verify human-centered automation design.

3 How to apply the basic requirements for system development

Obviously, according to basic requirement (1), there is the main task to carefully specify the situation assessment part of the machine functions. The picture of the flight situation as generated by the machine should cover all aspects which are also to be considered as situational aspects by the cockpit crew. Moreover, it would be most desirable, if the machine picture would be even more comprehensive and more accurate. This is already feasible today in certain aspects by application of cognitive engineering. In principle, thereby compliance with basic requirement (1) can be accomplished effectively with the technology already at hand today. In essence, the capability of situation assessment is to be incorporated in terms of corresponding functions in the machine part of the man/machine system in parallel to those of the cockpit crew (figure 2). As part of the situation assessment, the machine is attentively watching the cockpit crew's state and activity, thereby having the full picture including the crew situation.

Figure 2. Flight Guidance and Control Upcoming

This is the basis for cooperative automation in order that the cockpit crew's attention can be guided towards the objectively most urgent task or subtask of the actual situation. It becomes evident at this point that instead of allocating functions either on the machine side or the crew side once and for all times, all functions necessary to fly the aircraft are not only inherent crew functions but also functions which the machine should be capable to perform. All of these functions are operative in parallel unless actions with direct effect on aircraft controls are concerned. Thereby, it is ensured that the crew generally makes the final decision about these actions, whether to accept action recommendations of the machine or to follow their own ideas.

We call this the situation-dependent functional share of man and machine as partners with different roles like the partnership within the cockpit crew between the pilot flying and the pilot not flying. Partnership means that the capabilities of the partners are similar, but not necessarily identical. Partnership demands for effective dialogue. According to basic requirement (1), the presentation of the full picture of the situation has to be shaped in a way that the crew's attention is guided by the presentation only if necessary. In addition, the crew should also be able to talk to the machine partner like the crew communicates among each other. Therefore, the key specifications for the technical development of new generations of cockpit automation including cognitive engineering, in summary, comprise both

• comprehensive machine knowledge of the actual flight situation and
• efficient communication between crew and machine, based on situation knowledge and new dialogue technology.

How can the machine knowledge about the actual flight situation be established in order to meet these specifications? Both advanced techniques of cognitive engineering for structured knowledge representation and information processing based on advanced sensor technology (e.g. voice recognition and computer vision) allow for generating the knowledge base which includes about all static and dynamic situation elements the cockpit crew may be aware of and possibly even more than that. The task-related situation elements are concerned as well as the elements pertinent to the main players like the world surrounding the aircraft, the aircraft itself and, probably most important, the cockpit crew (figure 3).

Figure 3. Knowledge domains for situation assessment

Objective knowledge about the crew can be of paramount value. On the one hand, the machine might have a better picture of the pilot's status than the pilot himself, in particular in situations of imminent overcharge. On the other hand, machine knowledge about the crew is the basis for crew-adapted assistance. The machine cannot assist in an efficient way, if it does not sufficiently understand the cockpit crew's activities and corresponding needs. In its most advanced elaboration the knowledge about the cockpit crew comprises models of the physical and mental resources as well as behavioural models (see figure 4).

Figure 4. Model of Cockpit Crew

Thereby, the crew behaviour for situation assessment, planning and plan execution is to be modelled for normative behaviour as well as individual behaviour. The knowledge about the crew member's individual behaviour has to be learned on-line by the machine. Modelling of the error behaviour is another important behavioural aspect to be covered. Crew action modelling should not be confined to activities with hands and feet, also eye and head motion as well as voice activity contain important information, also with regard to efficient communication management between machine and crew.

In summary, chapter 3 and 4 have outlined the main guidelines, in little depth though, which are to be followed as closely as possible in order to warrant human-centered automation. These guidelines can easily be formulated as system design specifications.

4 The cockpit assistant system CASSY

The following description of the cockpit assistant system CASSY presents a realistic sample of how the basic requirements can be effectively pursued and mechanized by use of cognitive engineering techniques in the domain of civil transport aircraft to be flown in controlled airspace.

The main structure of CASSY is shown in figure 5. All situational elements of the entire flight situation, including mission, aircraft, systems, environment, and crew aspects, are stored in a central object-oriented representation. A specific communication module, the Dialogue Manager (Gerlach & Onken, 1993) is responsible for extracting the decisive patterns and coordinating their output to the crew via speech and/or display.

Vice versa the Dialogue Manager picks up the inputs of the crew and directs them to the respective module of the assistant. This is done via speech recognition. The Automatic Flight Planner (Prévôt & Onken, 1993) generates a complete 3-D/4-D flight plan. This is done autonomously or interactively with the crew, which depends on the situation. According to the generated flight plan, the Piloting Expert (Ruckdeschel & Onken, 1994) elaborates the expected pilot action patterns on the basis of pilot modeling such that the Pilot Intent and Error Recognition (Wittig, 1994) can compare this expected behaviour to the actual behaviour of the crew. Thereby, it identfies discrepancies in behaviour and their reasons. There are three possible reasons for this:

a) Pilot error:
The pilot deviates from the objectively correct expectation of behaviour, derived by CASSY.

b) Temporary discrepancy of pilot intent:
Events, which CASSY does not know yet, cause the pilot to deviate from the behaviour expected by CASSY.

c) Machine error:
An inappropriate or erroneous modeling or information processing within the machine (including CASSY) leads to an objectively wrong expectation of pilot behaviour by CASSY.

In case of a pilot error, a warning or hint is given to the pilot to correct the error. In order to cope with the temporary discrepancy of pilot intent, the information about the intent is gained in a special function, since the pilot should not permanently be in charge to tell his electronic partner about his state or his intentions. CASSY tries to figure out the intention, to modify the flight plan, accordingly, and to elaborate the consistent expected behaviour. Machine errors should not occur, but realistically it sometimes happens and must be considered. The errors are less serious, when they can be detected easily, be recovered with very few commands and have no safety critical consequences. Therefore, capabilities for in-flight restart of the system or recovery from erroneous states are provided as an important functionality.

In addition to the situation assessment concerning pilot behaviour, conflicts with the flight plan are detected autonomously and conflict hints are given or replanning is initiated to solve the problem. In the conflict case CASSY decides whether to initiate an interactive replanning or a completely autonomous replanning, depending on the available time and human resources. If the pilot decides to initiate planning, the amount of inputs he gives is up to him. The assessed situation is permanently shown with respect to the current flight plan on the display in heading-up or plan-mode. When no problems occur and everything is working properly, the crew does not become aware of CASSY’s actvities other than the appropriate presentation of the flight situation.

The planning and decision making assistance includes:

• autonomous or interactive generation and evaluation of routings or routing alternatives and trajectory profiles for the complete flight or local portions of the flight
• evaluation and selection of alternate airports and emergency fields
• prediction of the remaining flight portions, when ATC redirects the aircraft or the pilot intentionally deviates from the plan.

The monitoring capabilities include

• monitoring of the pilot actions with regard to nominal flight plan values, i.e. altitude, heading/track, vertical velocity, speeds and
• configuration management, e.g. flaps, gear, spoiler and radio navigation settings
• monitoring of violations of specific danger boundaries, including minimum safe altitudes, stall and maximum operating speeds and thrust limits.

Basic services are provided for

• configuration management by speech input,
  approach briefings, departure, approach and profile charts generated from the actual flight plan on request,
  performance and navigation calculations on request.

The range and kind of provided functionalities and the way, they are realized, result from extensive simulation experiments and consultations of pilots at each decisive development step. After the successful completion of these simulator trials the flight experiments were performed in June 1994.

5 Flight experiment with CASSY

The flight experiments were aimed at evaluating the CASSY performance in the real aviation environment. The system was integrated into the experimental cockpit of the Advanced Technologies Testing Aircraft System ATTAS of the DLR and typical regional flights in high traffic areas were performed. In the following the experimental environment and the flight scenaries are presented.

5.1 Experimental environment

The flying simulator ATTAS, an especially developed modification of the 44-seat cummuter jet VFW 614, is equipped with an experimental fly-by-wire flight control system and a versatile computer and sensor system. Beyond many other test programs it is used as the airborne segment in DLR's air traffic management demonstration programme (Adam, Klostermann & Schubert, 1993) and is equipped with very good facilities for testing complex on-board systems in instrument flight scenarios. In addition to the two safety pilots seated in the front cockpit, the ATTAS aircraft can be flown by the test pilot in an experimental cockpit, which is installed in the rear cabin directly behind the front cockpit. The experimental cockpit is a generic flight deck (one seat) with side-stick, airbus display and autopilot techniques and ARINC control panels. Therefore, it represents a realistic pilot working environment for IFR operation. The CASSY hard- and software has been integrated into the experimental cockpit.

The hardware of the assistant system, consisted of

• an off-the-shelf Silicon Graphics Indigo (R 4000) workstation to run the core modules of the assistant system connected to the ATTAS experimental system via ethernet
• a PC/QT equipped with a Marconi MR8 PC-cart providing speaker dependent continuous speech recognition with a speech button on the side stick
• a DECtalk speech synthesizer with various voices for speech output connected to the ATTAS intercommunication facilities
• a BARCO monitor (about 25cm) connected to the graphics channel of the SGI-Indigo and built into the experimental cockpit.

The computers were located in the rear of the main cabin. There are several experimenter work stations in the aircraft. One was equipped with a laptop for starting and maintaining CASSY.

5.2 Knowledge acquisition

During the flight tests CASSY has been running throughout the complete flights from taxi-out to taxi-in. All data, which CASSY received via the avionics data bus, have been recorded with a frequency of 10 Hertz. All in- and output messages have also been recorded and every time, the flight plan had changed because of a major planning activity or when a checkpoint has been passed, the whole situation representation has been stored. These data enable a replay of all flights and a reproduction of all situations.

The presented results have been gained by observing the behaviour of the pilot and the intelligent assistant during the flights on-line, by off-line evaluating the collected data and in debriefings immediately after the flights. Two professional pilots served as experimental pilots and additional pilots from Lufthansa German Airlines were participating as observers.

5.3 Flight scenarios

A total amount of about 10 flight hours has been performed, comprising eight flights from the regional airport Braunschweig (EDVE) to the international airports of Frankfurt (EDDF), Hamburg (EDDH) and Hannover (EDVV) at which a missed approach procedure was conducted before returning back to Braunschweig.

The distribution of flight phases is shwon in figure 6.

Table 1
Flight test scenarios

Flight no. 2 has been an in flight simulation of departure and approach to Frankfurt, which was necessary to investigate certain incidents, which would have been safety critical in the real Frankfurt area, e.g. descending below the minimum safe altitude. In all other flights nothing has been simulated and no special situations have been provoked, since the system should be evaluated in the real environment, which includes coping with all events, which occur during an IFR flight in a high density area.

6 Experimental results

One important result held true throughout the complete test program: There was no significant difference in system performance between the flight tests and the simulation trials. Consistently, the following discussion of results concentrate on the major questions concerning the real environment rather than system performance.

6.1 Operationality of the interfaces

To evaluate the speech recognition performance three different speakers made the speech input during the flight tests, summarized in table 2.

Table 2
Speech inputs

In their first flight pilot 1 and pilot 2 were not very familiar with the speech recognition system and the specific syntax to be used. In flight no. 6 a CASSY experimenter made the complete speech input for the pilot. He was familiar with the syntax and the speech recognizer from simulation experiments. The results are shown in figure 7 with regard to recognition performance.

Figure 7. Percentage of recognized speech command

Obviously, speech recognition inside the noisy aircraft is possible. It takes some time for the pilot to become familiar with the recognizer and the syntax to be used. This learning process can also be done in the simulator, as the flight with the experimenter has shown. The achieved percentage of recognized speech commands is almost of the same level as could be achieved in simulator runs with the same recognition system.

For entering the ATC commands into the system, two different experiments have been made throughout the flights. 92 ATC commands of a total of 236 have been fed into the system by the pilot using speech. The remaining 144 commands have been keyed into the system by one of the experimenters onboard the aircraft immediately after receiving the message, to simulate a data link from the ground into the aircraft. This took some seconds. The pilot reacted to the commands at the same moment he received the ATC message, but acknowledged the command with some time lag. This time lag resulted in delayed reaction of CASSY, which sometimes led to unnecessary warnings and hints. The percentage of occurence of these incidents compared to the respective number of ATC messages and the mean phase lag is illustrated in figure 8:

Figure 8. Unnecessary warnings or hints resulting from time lags in entering ATC commands

This effect was typical for the one-pilot configuration of the experimental cockpit. The figure illustrates the importance of a fast and powerful ATC interface. Optimal system performance can only be achieved with a digital data link.

6.2 Situation assessment with respect to pilot behaviour

The basic requirements described in chapter 2.1 point out the necessity for a complete understanding of the global flight situation. To get an impression of the situation assessment capabilities the duration of discrepancies between the actual and the expected pilot behaviour has been related to the total flight time. This has been done on the basis of the stored data for the six flights 2, 3, 5, 6, 7 and 8.

Figure 9. Discrepancies in actual and expected pilot behaviour

Figure 9 indicates that for almost 94 % of the flight time in a high density environment the pilot and the machine assessed the situation equally, because otherwise they would not expect or perform the same action patterns.

A total amount of 100 incidents leading to warnings have been evaluated to find out the reasons for the warnings and messages of similar purpose and the consequences they had. All incidents have been related to one of the three categories: pilot error, pilot intent and machine error (i.e CASSY errors in this case) (figure 10).

In five cases of the intentional deviations from the flight plan the intention was autonomously figured out by the assistant system and the flight plan has been adapted, accordingly. In three cases the pilot had to inform CASSY about his intention.

Half of the machine errors were caused by an incomplete knowledge base, e.g. insufficient modeling of the aircraft performance and the other half by malfunctions of CASSY, i.e. software implementation errors due to less rigorous application of software development procedures. In one case such a malfunction led to a complete breakdown of the assistant system. In all machine error cases the pilot realized that a wrong warning was issued by CASSY. No negative influence on the pilot's situation assessment could be observed. In the one breakdown case, the complete CASSY system had to be restarted in flight, which took about 15 seconds. The only pilot input needed for such a recovery procedure is the flight destination. In all other machine error cases the warnings disappeared autonomously, when the incorrect assessed maneuver had been completed by the pilot.

Figure 10. Error count during flight tests

Concerning the pilot errors the light errors are considered to result in an inaccurate or uneconomical, but safe maneuver. Moderate errors, probably would lead to a safety critical situation, and severe errors surely would lead to a dangerous safety hazard unless an immediate correction is made. All pilot errors, which occured during the flight tests, were detected by CASSY. All moderate and severe errors as well as about 70% of the light errors were immediately corrected by the pilot after having received the warning or hint.

This means there were no significant negative consequences of errors or failures whether caused by the pilot or by CASSY. This is the symbiotic effect we want to achieve!

6.3 Flight planning and decision aiding

CASSY's flight planning capabilities have been stated by the experimental pilots and the observers as very impressive. As a matter of fact, all planning proposals have been accepted and none of the autonomous radar vectoring predictions has been modified or caused any doubt from the pilot. The time needed for planning a complete flight from one airport to the other is illustrated in figure 11.

Before every flight the flight destination and the departure runway were entered into CASSY and an autonomous planning of the complete flight was initiated. After the go around procedure at this destination the pilot initiated an interactive planning to return to the airport, from which he had departed, by entering its name. CASSY elaborated and presented two routing proposals in parallel, which the pilot could select from or modify. After the selection the trajectory profile was planned in detail and recommended speeds, times of overflight, radio aids etc. were inserted.

Figure 11. Duration of flight planning activities

The distance to the destination had only little impact on the duration of planning. The autonomous planning took between 4 and 6 seconds, the interactive replanning up to 26 seconds, of which the pilot needed about 16 seconds to decide for a proposal. This confirms the approach to replan autonomously, when the flight plan must be generated very fast. When there is more time available, replanning can be done interactively, too, in order to keep the pilot more involved.

6.4 Pilot acceptance

Pilot acceptance of the planning and monitoring functions of CASSY was extremely positive and at least as good as in the previous simulator trials (Onken & Prévôt, 1994). All pilots participating in the evaluation attested CASSY a nearly operational performance and a very promising concept. It was noted that the CASSY functionalities for enhancement of situation awareness, situation assessment and monitoring, as well as the good planning capabilities are effectively in line with human-centered design, according to the two basic requirements.

7. Main cause for substantial gain in situational awareness through cognitive machine assistant

The flight test results of CASSY as shown with figure 10 illustrate the symbiotic effect for error correction, no matter whether the error was caused by the pilot or by CASSY. Most of these errors accounted for in figure 10 were errors in situation assessment. Since the mechanisms for situation assessment of the pilot are dissimilar from that one of CASSY, also their error behaviour is dissimilar. It turns out that the dissimilarity in situation assessment performance results in complemntary performance to a great extent.

The cause for this effect is illustrated in figure 12, where the degree of completeness in situational awareness is depicted in a schematic way for different configurations, with and without the use of cognitive machine assistant. Figure 12.I shows the state-of-the-art outcome, figure 12.II shows how the pilot-flying and the pilot-not-flying are contributing to this outcome as a team, illustrating that situation awareness is considerably increased by the fact of having two crew members examining the situation in a dissimilar way. Figure 12.III shows in the first place the outcome of a configuration by exploiting all what can be exploited by better reflecting human factors knowledge in conventional cockpits, also optimizing flight crew training. The improvements to be expected are distinct and desirable, but more limited than we want. In comparison, the effect of error correction symbiosis with the advent of cognitive machine assistants as a third, much more dissimilar crew member is reflected with its powerful potential for accomplishing superiorly close to complete situational awareness.

8 Certification of cognitive systems

Cognitive systems are mistakenly considered to be hardly certifiable. The contrary is true by vitue of these systems. As opposed to a conventional system, like a flight management system, for instance, a small-sized piece of knowledge about overriding mission goals and constraints ensuring safety is inherent part of the entire knowledge incorporated, easily isolated from the knowledge bases about other knowledge domains implemented in the cognitive system. A simple algorithm for plausibility checks on the basis of this knowledge is then solely necessary in addition to this small, well-verifiable and testable knowledge base to ensure safety. Only these simple components are to be examined rigorously with respect to safety in the course of the certification process. The function of these components is completely independent of the flight situation. Whatever the situation is, they work the same. Therefore, there is no need to consider all kinds of possible flight situations in test runs in order to ensure the correct functioning.

Figure 12. Achievable gains in Situation Awareness (SA)

This approach of avoiding catastrophic failings is not unusual for us. It is about the same method, which is used by the humans. In parallel to our thinking of how to proceed in whatever situations, the potential intentions are automatically checked in a similar simple way against the personal goals and constraints before the intentions are carried out. If we detect a conflict as a consequence for a certain intent alternative considered, we switch to other alternatives until a conflict-free one is identified, without getting too much involved by which internal mistake this conflicting intent candidate was generated.

Again also this type of procedure, to find a workable, conflict-free way to proceed is easily incorporated in the artificial cognitive system, such that it is even capable of overcoming a deadlock in advicing the pilot in case of an event, when the system is generating a safety-compromising advice, which is prevented to be transferred to the pilot by the described plausibility check component. Thus, even in this case the pilot will not be left alone!

In essence, it can be expected that the certification effort for cognitive systems will be considerably lower as for conventional systems bare of this kind of knowledge and inferencing capabilities.

In addition, one can think of an explanation component, which will diagnose the process of advice generation to find the causes of mistakes and to allow fixing the software in this respect.

All this can be handled in the framework of existing certification regulations.

9 Evolution of cognitive machine assistants in the ATM environment

It has been shown that situation awareness and adequate workload levels can be systematically designed into the cockpit by cognitive engineering according to the basic requirements for flight deck automation (see chapter 2 and 3). The reason is that cognitive systems are capable of establishing their own picture of the situation like humans are doing it. Cognitive systems are doing it in a somewhat different way, though, not just coping the human operator's way. Thus, if the operator's situation awareness is lacking for any reason, there is a great chance that the cognitive machine assistant’s picture of the situation is complete and can be used to make the human's picture also complete. This leads to a form of interaction design, where the cognitive machine assistant reads from the behaviour of the pilot about his understanding of the situation, displaying at the same time its own understanding. This is important, since in existing operational systems, although for most situations the understanding will be equal for both the human operator and the machine, there are two cases, which were difficult to handle, like

1. there is a discrepancy of situation understanding between human operator and machine and

2. there is a problem which requires both human and machine resources for its solution.

Cognitive systems offer the potential to deal with these two cases satisfactorily. Figure 13 and figure 14 illustrate how the respective scheme of interaction between human operator and machine assistant have to look like.

Figure 13. Interactions to even up the situation awareness of both the Human Operator (HO) and the Machine Assistant (MA)

Greatest progress in compliance with the basic requirements can be expected, if comprehensive machine situation assessment is demanded, comprising as many situational aspects as technically feasible at the time being. Comprehensive machine situation assessment consists of the assessment of the situation of all, the controlled element (for instance air traffic), environment (if there is an effecting environment) and human operators. Situation assessment about the human operators, whether they are working in the expected way, is crucial. Furthermore, special attention should be directed towards the following main areas in cognitive engineering of future knowledge-based assistant systems guidelined by a generic functional concept as depicted in a compressed way in figure.15.

• Knowledge engineering concerning machine knowledge expansion (e.g. object-oriented situation representation in the machine, real-time modeling of domain behaviour of humans involved, real-time, situation-dependent resource modeling of human operator, domain data bases)

Figure 14. Interactions for problem solving by both the Human Operator (HO) and the Machine Assistant (MA)

• Machine perception
• Displays and human operator input devices as part cognitive systems, exploiting machine knowledge for most efficient communication
• Situation awareness assessment, including and human operator resource assessment.

ATM can tremendously capitalize on these capabilities, in particular, when the foreseeable growth in demand on the airspace system and the vision of free flight with its chances for safety and productivity are considered. Assistants might help in a crucial way to enable systematic exloitation of these opportunities. Then, assistants will be at all operator positions in the ATM environment as to cockpits, sites of the air traffic service providers and airline operations centers. Similar requirements as those for cockpit assistants will also have to apply to the other assistants respectively.

Figure 15. Functional layout of future cognitive machine assistant

10 Conclusion

The time has come that future cockpit systems no longer will be designed on a vague basis of specifications. The advances in technology have brought about means to systematically reflect requirements for human-centered automation into clear-cut specifications and cockpit system development. Machine functions will be incorporated which not only render support for planning and plan execution as emphasized in the past. Instead, main emphasis will be placed on comprehensive situation assessment by the machine in parallel to the crew's situation assessment activity.
This leads to better machine understanding of what the real needs of the crew are and consequently to more efficient support for the sake of flight safety as well as mission effectiveness. There are already examples of successful developments, which have proven that the way of implementing design guidelines as described in this paper systematically lead to the desired system performance. One example is the Cockpit ASsistant System CASSY. The successful flight tests of CASSY in real IFR flights have demonstrated that human-centered automation by means of cognitive systems on-board can be integrated into the cockpit of modern aircraft. The amount of detected and avoided pilot errors, the availability of features like pilot intent recognition as well as the power demonstrated in complex planning indicate the performance level of CASSY. The crucial features of this kind of cognitive system in the flight deck will find their way in the design specifications of future aircraft, in particular, because certification of cognitive systems has become rather easily feasible in the meantime. However, in order to capitalize on the capabilities of cognitive engineering even more comprehensively, cognitive machine assistants should also be considered for in the centers of the air traffic service providers and the airline operations.

References

Adam, V., Klostermann, E. & Schubert, M. (1993), DLR's ATM Demonstration Programme. AGARD Meeting on 'Machine Intelligence in Air Traffic Management'.
Amalberti, R. & Deblon, F. (1992), ‘Cognitive Modelling of Fighter Aircraft Process Control: Step towards an Intelligent Onboard Assistance System’, International Journal on Man Machine Studies, 36. Academic Press.
Billings, C. (1991), ‘Human-Centered Aircraft Automation: A Concept and Guidelines’, NASA Tech Memorandum 103885, Moffet Field.
Bittermann, V., Perennec, M., Mielnik, J.-C., Sainthuile, G. & Jud, J.-M (1992), ‘FINDER: Flight-Plan Inetractive Negotiation and Decision-Aiding System for Enroute Rerouting’, 12th Conf. Intelligence Artificielle, Systemes Experts, Langage Naturel, Avignon.
Boy, G. (1991), ‘Intelligent Assistant Systems’, Academic Press, London.
Chambers, A.B. & Nagel, D.C. (1985), ‘Pilots of the Future: Human or Computer?’, Comm. of the ACM, Vol. 28, No. 11.
Dudek, H.-L. (1990), ‘Wissensbasierte Pilotenunterstützung im Ein-Mann Cockpit bei Instrumentenflug’, Dissertation, UniBw, Munich.
Dyck, J.L., Abbott, D.W. & Wise, J.A. (1993), ‘HCI in Multi-Crew Aircraft’, Proc. of the HCI, Orlando, 19 A, 08/93.
Endsley, M.R. & Bolstad, C.A. (1992), ‘Human Capabilities and Limitations in Situation Awareness’, Proc. NATO AGARD FMP/GCP Symposium, Edinburgh, UK.
Gerlach, M. & Onken, R. (1993), ‘A Dialogue Manager as Interface Between Aircraft Pilots and a Pilot Assistant System’, Proc. of the HCI, Orlando, 19th A, 08/93.
Gerlach, M. (1995), ‘Schnittstellengestaltung für ein Cockpitassistenzsystem unter besonderer Berücksichtigung von Spracheingabe’, Dissertation, UniBw, Munich.
Harris, D.F. & Morrissette, J.A. (1982), ‘Single Pilot IFR Accident Data Analysis’, NASA CR 3650.
Helander, M., Moody, T.S. & Joost, M.G. (1988), ‘Systems Design for Automated Speech Recognition’, in Helander (ed.), Handbook of Human-Computer Interaction, Elsevier Science P.
Heldt, P.-H. (1993), ‘Flying the Glass-Cockpit’, DGLR-Fachausschuß Anthropotechnik, Berlin.
Hollnagel, E., Warren, C. & Cacciabue, P.C. (1993), ‘Automation in Aerospace and Aviation Applications: Where are the Limits?’, WS. Rep. 4th HMI-AI-AS, Toulouse, 09/93.
Huber, P., Jensen, K. & Shapiro, R.M. (1990), ‘Hierarchies in Coloured Petri Nets’, in: Rozenberg, G. (ed.): Advances in Petri nets 1990, Lecture Notes in Computer Science, Vol. 483., Springer, Berlin, Heidelberg, New York, pp. 313-341.
Meta Software Corporation (1991), Design/OA, Version 3.0, Cambridge.
Nilsson, N.J. (1992), ‘Principles of Artificial Intelligence’. Springer Verlag.
Norman, D.A. (1990), ‘The 'problem' with automation: inappropriate feedback and interaction, not 'over-automation'’, in: Human Factors in Hazardous Situations (D.E. Broadbent, A. Badderly and J. T. Reason (Ed.)), Clarendon Press, Oxford, pp. 585-593.
Onken, R. (1988), ‘Automatic Pilot Aid for Situation Awareness and Flight Planning’, HMI-AIAS, Toulouse-Blagnac.
Onken, R. (1992), ‘New Developments in Aerospace Guidance and Control: Knowledge-Based Pilot Assistance’, 12th IFAC Symposium on Automatic Control in Aerospace, Ottobrunn.
Onken, R. (1993), ‘Funktionsverteilung Pilot-Maschine: Umsetzung von Grundforderungen im Cockpitassistentsystem CASSY’, DGLR-Tagung des Fachausschusses Anthropotechnik.
Onken, R. & Prévôt, T. (1994), ‘CASSY-Cockpit Assistant System for IFR Operation’, ICAS 94-8.7.2 Proceedings Vol. 2, Anaheim, CA.
Onken, R. (1994), ‘Human-centered Cockpit Design through the knowledge-based Cockpit Assistant System CASSY’, NATO DRG Panel 8 Workshop on improving function allocation for integrated system design, Soesterberg.
Onken, R. (1995), ‘Functional Development and Field Test of CASSY - a Knowledge-based Cockpit Assistant System’, AGARD LS200.
Prévôt, T., Onken, R. & Dudek, H.-L. (1991), ‘Knowledge Based Planning for Controlled Airspace Flight Operation as Part of a Cockpit Assistant’, AGARD GCP Symposium, Amsterdam.
Prévôt, T. & Onken, R. (1993), ‘On-Board Interactive Flight Planning and Decision Making With the Cockpit Assistant System CASSY’, 4th HMI-AI-AS, Toulouse.
Prévôt, T. (1995), ‘Maschineller Flugplaner für Verkehrsflugzeuge als autonomes und kooperatives System’, Dissertation,UniBw, Munich.
Proceedings of the 3rd International Workshop on Human-Computer Teamwork (1994), ‘The Human-Electronic Crew: Can we trust the team’. Cambridge, UK.
Proceedings of ‘Intelligent Vehicles ‘94’, Paris, 1994
Proceedings of CSEPC ’96, Cognitive Systems Engineering in Process Control, Kyoto, Japan, 1996
Rasmussen, J. (1993), ‘Skills, Rules, and Knowledge: Signals, Signs, and Symbols, and other Distinctions in Human Performance Models’, IEEE SMC-13 (3), pp. 257-267.
Rasmussen, J. (1986), ‘Information Processing and Human Machine Interaction An Approach to Cognitive Engineering’, Elsevier, Amsterdam.
Reising, J.M., Emerson, T.J. & Munns, R.C. (1993), ‘Automation in Military Aircraft,’ Proceedings of the HCI, Orlando, 19A, 08/93.
Ruckdeschel, W. & Onken, R. (1994), ‘Modelling of Pilot Behaviour Using Petri-Netze’, 15th Int. Conf. on Application and Theory of Petri Nets, Zaragoza.
Rummelhart, D.E & McClelland, J.L. (1987), ‘Parallel Distributed Processing. Explorations in the Microstructure of Cognition’, MIT Press, Cambridge, MA.
Starke, P. (1993), ‘INA Integrated Net Analyzer’, Version 1.3, Berlin.
Strohal, M. & Onken, R. (1994), ‘Cockpit Assistance’, Human Factors Workshop, IABG.
Taylor, M.M. & Waugh, D.A. (1992), ‘Principles for integrating voice I/O in a complex interface’, AGARD Avionics Panel Symposium, Madrid, Spain.
Vogler, W. (1992), ‘Modular Construction and Partial Order Semantics of Petri Nets, Lecture Notes in Computer Science’, Vol. 625, Springer, Berlin, Heidelberg, New York.
Wiener, E.L. & Curry, R. (1980), ‘Flight Deck Automation: Promises and Problems’, Ergonomics 23, pp. 995-1011.
Wiener, E.L. (1989), ‘Human Factors of Advanced Technology (‘Glass Cockpit’) Transport Aircraft’, NASA Report 177528, Moffett Field, CA.
Wiener, E.L. (1989), ‘Error Exposure and Control in High Technology Aircraft’, in: Artificial Intelligence in Cockpit Design, Orly.
Wilensky, R. (1983), ‘Planning and Understanding’, Addison Wesley Reading, Massachussetts.
Wise, J.A., Abbott, D.W., Tilden, D., Dyck, J.L., Guide, P.C. & Ryan, L. (1993), ‘Automation in Corporate Aviation: Human Factors Issues’, Interim Technical Report CAAR-15405-93-1, FAA.
Wittig, T. & Onken, R. (1992), ‘Pilot Intent and Error Recognition as a Part of a Knowledge Based Cockpit Assistant’, AGARD Joint FMP/GCP Symposium, Edingburgh, 10/92.
Wittig,T., Onken, R. & Zeller, H. (1994), ‘Entwicklungsperspektiven intelligenter Flugführungssysteme’, DGLR-Jahrestagung, Erlangen.
Wittig, T. (1994), ‘Maschinelle Erkennung von Pilotenabsichten und Pilotenfehlern über heuristische Klassifikation’, Dissertation, UniBw, Munich.
Woods, D.D. & Roth, E.M. (1988), ‘Cognitive Systems Engineering’, Handbook of Human Computer Interaction, M. Helander (ed.), Elsevier Science Publishrs B.V.
Zadeh, L.A. (1965), ‘Fuzzy Sets’, Information and Control, Vol. 8.