Three challenges for adaptive human-machine interaction

Do you have any idea how stressful it is driving an old timer VW campervan during rush hour in Brussels? Last year, it demanded my total concentration due to the old van’s slow dynamics, relatively poor field of view and the crowded rush hour traffic. Meanwhile, my girlfriend helped navigate us through the complexity of roads. Having arrived at our destination with our heart rates back to ‘normal’, we toasted our excellent team work.

Since the introduction of the Volkswagen T2, cars and other machines have changed rapidly and become far more intelligent in the way they interact with humans. To facilitate cooperation like that between me and my girlfriend, systems must be able to adapt in an identical fashion. This requires machines to be able to observe the human teammate’s behaviour and to have a ‘mental model’ to interpret that behaviour.

“The time when machines only executed simple tasks are over”

A new mindset

Having smarter machines means that we need to change how we interact with them. The time when  they only executed simple, well defined tasks within a limited operational space, like welding robots on a manufacturing line, are over. Slowly but surely, machines have become capable of performing more complex tactical and strategic tasks which were traditionally allocated to humans. Consequently, man and machine are teaming up and sharing task responsibility, requiring us to interact with them on a semantically higher level. Just like a teammate or colleague.

No longer a research tool

The aviation domain has a strong background in measuring human behaviour in the context of man-machine systems. During the Second World War, this took flight because the war needed ‘everyday people’ to be capable of operating complex systems, like sonar, radar or aircraft. Anti-aircraft gunners were evaluated on how well they tracked a fighter flying overhead, whilst the vigilance of radar operators was judged by their hit or false alarm rate.

In the early days, these measurements were limited to the operator’s performance to control a system (i.e., reaction time, position error, etc.). Nowadays, state-of-the-art techniques can measure various aspects of human behaviour. For example, pilots’ gaze patterns can be evaluated by using high-end eye trackers (see an earlier blog on NLR’s research), skin conductance can be used as an indication for the pilot’s attention or electro-encephalography (EEG) to determine whether a warning light has been detected, for example.

The next step is to get machines to act more like a teammate as such measurement techniques and methods gradually start to emerge in commercial products. Modern cars already have ‘driver monitoring systems’ in place, with the Driver Alert System using cameras and video recognition to prompt a warning when a driver is showing signs of fatigue (e.g. percentage of time that the eyes are closed or detecting microsleep). Investing in the development of adaptive systems is essential for humans and machines to collaborate effectively and efficiently in the near future. Take the MEDIATOR project in which NLR is collaborating with academic and industry partners to develop a driving support system which dynamically allocates driving tasks to either the driver or automation, based on the driver state and environment.

“The next step is to get machines to act more like a teammate”

Challenges of the future

Finally, interaction researchers are now addressing these three main topics:

1. 1. Determine the human’s state
      Algorithms which interpret the human’s behaviour must enable machines to adapt their behaviour accordingly. These algorithms will need to combine many sources of information and pick up on subtle cues in gaze patterns, body posture and control inputs of the human to obtain a reliable estimate of their state. Moreover, no two humans are identical, which makes the development of a general algorithm complex. Machine learning and other forms of AI will probably resolve some of the difficulties arising in this area.
   2. Design interaction with failure in mind
      In a perfect world, the human and machine would both perform their assigned tasks faultlessly and interact seamlessly. However, machines and humans both fail from time to time and interaction designers must take this into account. The human needs to be aware of the machine’s performance and intervene in time if there is a failure. Ideally a machine would do the same. This challenges designers to create interaction that is not only robust when everything is going well, but also when things go wrong.
   3. Communicate clearly
      When the allocation of a task (and the responsibility) becomes dynamic and adaptive, clear communication between human and machine becomes essential for effective and safe collaboration. If responsibility for performing a task shifts from machine to human without the human realising, this could have serious consequences. A well designed interface is balanced, providing just the right amount of information to make it easy to understand and without overloading the human. In the future, smart interfaces should adjust the amount of information and how it is presented based on the state of the human to prevent alarm floods.

Inside and outside NLR, a great deal of effort is put into research focusing on the abovementioned challenges. In time, I’m sure that one day human-machine interaction will become as effortless and intuitive as the human-human interaction I experienced during my ride through Brussels. However, I will miss the satisfying toast.