Aviation training is often far more effective and efficient and significantly cheaper in the virtual world than in the real world. For that reason, NLR is experimenting in its X-Lab with virtual reality (VR) and augmented reality (AR) technologies to enable, for example, pilots and maintenance engineers to train for the most complex situations wherever and whenever they want in future.
Imagine it is 2030. You are a pilot and are not scheduled to fly for a while. In addition to flying, your most important task is training. So it is logical to board an aircraft undergoing maintenance in the hangar, sit at the control stick, put on your special virtual goggles and start a training session. It has become largely a matter of routine in the past few years to train on your own for exceptional situations that may occur during a flight. You don’t always need to use a large flight simulator. Nowadays you can train and complete your required number of flight hours at many more locations, like in an aircraft on the ground, in small simulators located at an airport, and even in your own home.
This specific scenario figures prominently in NLR’s vision of the future: flexible mobile and innovative locations where pilots can simulate their flights for training purposes and for subsequent evaluation. The high expectations of aviation applications of this kind are very promising and their impact will be considerable. Among the benefits are lower costs, more efficient and flexible training concepts, a smaller training and simulation footprint and even better approximation of real life.
More flexible, cheaper and closer to real life
The costs of today’s large flight simulators are relatively high. What’s more, they are at fixed locations so pilots must first fly to them. The simulators are necessary to simulate complex situations as close to real life as possible, but AR/VR technology also allows training in all kinds of subtasks and skills in an easier and sometimes better way. “The trend I observe is that training and simulations are becoming increasingly cheaper, more flexible and simpler to use”, says Roy Arents, one of the creative minds behind X”‘Lab, the place at NLR where new things are conceived rapidly in association with a diversity of external partners. This makes X-Lab the nursery for VR and AR innovations for training and simulations.
X-Lab
The X in X-Lab stands for two things: for eXperimentation and for X-reality, the term NLR uses to refer to the wide range of simulation technologies, from real to virtual realities. Augmented reality adds extra information to reality by means of goggles, while virtual reality enables a person to be in an entirely different world that does not actually exist. In practice, these two worlds are converging all the time, a phenomenon sometimes referred to as mixed reality or condensed reality.
How will people use technology in the future training?
“The world will become more and more virtual in the years ahead, including the world of aviation education and training”, says Harrie Bohnen, manager of NLR’s Training, Simulation and Operator Performance department. His department has 34 staff members and 10 students all working on new human factor concepts in aviation. This includes answering a key question: how will people use technology in the future? “We give scope to young researchers with different kinds of expertise and students from intermediate vocational institutions (game design), higher vocational institutions and universities (including those studying aerospace technology) to devise, develop and experiment with new technologies”, says Bohnen. “NLR does not actually produce the training courses or training manuals. We bring together applied research, our unique knowledge and expertise of aviation and new virtual technologies so as to develop new, innovative training concepts and simulations together with the user.”
In Bohnen’s team, a number of important conditions converge: AR and VR technology to examine the technological capabilities; investigate human factors for applicability of that technology in situations relevant to aviation operators; wishes and requirements of the client, including mobility (training anywhere and anytime), complex training situations, augmenting training and simulation with data, cost savings; and looking into whether all of that is possible within existing regulations (certification), and, where this is not possible, contributing to the development of new regulations.
Frontrunner
NLR is the frontrunner in the Netherlands when it comes to aviation training and simulation. Why? “We can offer a total package”, says Bohnen. “We invite customers to brainstorming sessions to give us an idea of what direction the customer wants to take and we develop a vision and a narrative. It can sometimes be useful if we can technically materialise this in a demonstrator and a video. So that’s exactly what we do. We do our best quickly to learn what precisely our customer needs and we quickly familiarise ourselves with the latest technological advances to enable us to adapt our vision and design. It is this approach that makes us successful.”
“We invite customers to brainstorming sessions to give us an idea of what direction the customer wants to take and we develop a vision and a narrative.”
Collaborative maintenance training with HoloLens
Together with KLM Royal Dutch Airlines, NLR produced a training concept for maintenance engineers where they learn as a group in a classroom how an aircraft’s air conditioning works. The objective is to make sure they are familiar with the working of the system and know how to repair it. The use of augmented reality technologies raises the efficiency of training and lowers the costs, according to research conducted by NLR educational experts. During the training, the trainees and the instructor wear special goggles — in this case a HoloLens —, to watch a three-dimensional model of an air conditioning system. They can walk around and through the system, perform tasks together, ask questions and receive an explanation from the instructor.
It is not just the use of the HoloLens and having several HoloLenses display a single simulation that are new, but also the interactive way of providing training. Research by NLR has shown that this produces greater training benefits than the present training methods. “It is motivational because you take the training as a team”, says Bohnen. “And in the virtual world it is easier for an instructor to look over the shoulders of the trainees than in the real world, enabling him to give feedback. The training can be given anywhere and practical exercises can be performed in a virtual aircraft instead of a real one. You no longer need the actual aircraft, which increases its availability and reduces wear and tear.” Moreover, instructors and trainees no longer necessarily have to be together in a training classroom, because in the virtual world interactive instruction can also be given remotely.
Anneke Nabben and Roy Arents (NLR) working on the concept for the AR maintenance training
Project-driven
Arents and his colleagues work in a project-driven way. Approximately 40 highly diverse projects are currently in progress at the department. For example, NLR carries out numerous projects for the Royal Netherlands Air Force, KLM Royal Dutch Airlines and aviation authorities EASA and FAA, which promotes the exchange of ideas from the civil and military worlds. This embraces the entire spectrum of flying, flight evaluation and aircraft maintenance. Another example is a serious game that was designed in 2016 to determine for which situations an Apache helicopter crew can best be trained during a certain live training session. For this purpose the players must work together in a virtual game world so as to test their competences. But the department switches just as easily to training for maintenance engineers or projects to improve future passenger experience.
Technology not yet mature
AR and VR are not yet sufficiently mature for all applications. From a technological point of view, there is still some way to go. Experimenting with users makes it possible to enhance the application and the training benefit. Building demonstrators remains the principal engineering challenge. “It’s certainly difficult to translate an enormous cockpit into a small mobile version”, admits Arents. This applies not just to making the equipment smaller, but also to taking a large visual system made up of traditional simulators and making it wearable by means of virtual reality or augmented reality goggles. The reason for this stems from the current technologies that are more consumer-oriented and have not yet been optimised for aviation, which imposes far more stringent requirements than the games world, where similar technologies are also in use. The most complex matter is to be able to see your own hands through virtual reality goggles. “You need your hands to press the buttons. We can make them partly visible by means of sensors, but this technology does not yet work perfectly.”
Augmented reality and regulation in civil aviation
NLR is endeavouring not only to make the most of technology, but to focus especially on combining an implementing new technologies, training methods, human factors and regulation. “We have a deep understanding of the rules in aviation and we are keen to speed up innovative thinking at regulatory authorities”, says Bohnen. Most civil aviation training still takes place in high-end simulators on motion platforms. These ‘level D sims’ are certificated in accordance with the standards of the European Aviation Safety Agency (EASA). EASA brainstorms with NLR on the future by examining whether virtual reality and augmented reality technologies yield the same training benefits for certain pilot tasks as the present level D sims do. The same goes for maintenance: how can the virtual world contribute to better training for maintenance engineers?
A portable cockpit, even for military aircraft
Between the desks in the X-Lab, a concept for a virtual portable cockpit recently took shape. It looks remarkably simple: a black leather seat in a steel frame, with a cardboard sheet at the front that holds a print-out of a cockpit and just a few buttons. The concept was devised and put together within a short time to demonstrate just how far you can go with simulation. Arents takes up position in the seat and dons virtual reality goggles (HTC VIVE), placing him in a virtual environment. The special glove that he put on shortly before enables the simulator to see his hand movements when he touches the ‘buttons’. “We want our customers also to be able to experience quality devices already commercially available, to make them aware that you don’t always need expensive high”‘end setups, but that some clever piecing together can be just as effective”, says Arents, who enjoys thinking up new training concepts with stakeholders.
In the longer term, defence is one of the areas where NLR would expect to find parties potentially interested in such a portable cockpit. As missions become more and more complex, military aviators want to be able to prepare even better for them, so that they can respond faster to unfolding situations. Tactically and cost-wise, it is nearly impossible to conduct exercises for the missions in the real world. It can be done in the virtual world, however, particularly as regards cooperation and coordination. This makes aviators even better prepared for future situations.
Keep exploring
Need to know more? Visit our Augmented and Virtual Reality capability page.