If you’re an avid smartphone user, you’ve probably encountered augmented reality before. Whether it’s a filter to give yourself cat ears and whiskers, or collecting creatures that appear in the real world through a videogame, these technologies that create digital overlays on top of the physical world are increasingly becoming a part of our lives.
NASA is using the same technologies to make useful information about all kinds of vehicles in our skies, like drones, more widely available to those who need it.
Whether for emergency response, managing air traffic, or local governance, visualising complex data through augmented reality makes it easier for people on the ground to be aware of the operations of the uncrewed vehicles that will increasingly populate our skies.
“When you see vehicles like drones a few hundred feet above your head, you’re a lot more curious about what’s going on than with a plane that’s several thousands of feet in the sky,” says Joey Mercer, a researcher at NASA’s Ames Research Centre. “We wanted to make some of the invisible information about the automated traffic systems managing these vehicles, and the vehicles themselves, more easily accessible from the ground.”
The software to make this happen was developed at the Airspace Operations Laboratory. It creates a digital overlay with information about flight patterns, a vehicle’s mission, and more – essential information as such vehicles become integrated into our day to day lives.
Airspace operators, local public officials, emergency responders and more may have needs for awareness of what’s happening in their local airspace. As more unconventional aircraft integrate into these environments, it’s important for them to do so safely and with appropriate communications to local communities.
The software leverages the capabilities of a commonly available mixed-reality headset, Microsoft HoloLens, to visualise this data in the real world.
By naturally swiping and selecting items through hand gestures and eye-tracking technology, NASA is helping air traffic management happen in a far more interactive way than it has previously.
The headsets are very useful for certain scenarios, but to make information even more accessible, this augmented reality software can also be used on smartphone devices. In the future, when you see someone with their phone out and pointed towards the sky while walking down the street, they may be looking at data related to a drone flying overhead instead of taking a selfie.
Augmented reality contributes to NASA’s future in Advanced Air Mobility, or AAM, which aims to develop air transportation systems that move people and cargo between places previous not served or underserved by aviation.
New technologies help address the unique challenges that come with drones and other uncrewed vehicles, where vehicles don’t necessarily have a set flight path or trajectory like larger aircraft, and are also more visible to the ground-based public.
For smaller uncrewed aircraft in urban environments, augmented reality allows researchers on the ground to help test air traffic management systems by overlaying real-world environments with interactive data visualizations, including other simulated aircraft.
The technology underwent a field evaluation during NASA’s fourth and final Technical Capability Level demonstration, or TCL4, of the Unmanned Aircraft Systems Traffic Management, or UTM, project.
Researchers used HoloLens devices to run the prototype augmented reality software to track the drones participating in the demonstration in real-time, visualising data gathered and spatial relationships between multiple flight paths and objects in the real world.
The software can also be used to visualise simulated drones in the same airspace.
“Uncrewed aircraft are more and more becoming a part of our lives,” says Mercer. “As we look at how to safely integrate these vehicles into already complex city infrastructures and airspaces, augmented reality is a great tool, with a lot of potential for how we think of managing our skies in the future.”