Episode 39: Intelligent Drones

Lisa Peña (LP): Welcome to a new year of tech talk. We're flying into 2022 with unmanned aerial vehicles, or drones. SwRI's advanced drone technology takes UAVs to new heights. With intelligent software, our drones are using algorithms and sensors to make decisions in the sky. How this technology can be used in life-saving operations and much more, next on this episode of Technology Today.

[MUSIC PLAYING]

We live with technology, science, engineering and the results of innovative research every day. Now, let's understand it better. You're listening to the Technology Today podcast, presented by Southwest Research Institute. Transcripts and photos for this episode and all episodes are available at podcast.swri.org.

Happy new year, and welcome to Technology Today. I'm Lisa Peña. You're probably familiar with UAVs, unmanned aerial vehicles, or drones, all terms for the same type of aircraft. You may even own one. Common uses include recreation, photography, and surveillance. SwRI is taking drones beyond these everyday uses, outfitting them with special software and expanding UAV applications. SwRI engineer and manager Dr. Eric Thorn joins us today to discuss advanced SwRI drone technology. Thanks for joining us, Eric.

enrich-drone-inspection-data-visualization.jpg

Courtesy of SwRI

A drone using autonomy software developed by Southwest Research Institute navigated an inoperable nuclear facility, during the October 2021 EnRicH European Robotics hackathon. SwRI built a map of the test facility using data and exploration algorithms. SwRI specializes in data visualizations to identify damage and potential hazards following accidents at nuclear power plants and other hazardous facilities.

Dr. Eric Thorn (ET): Hi, Lisa. Thanks for having me.

LP: We mentioned a couple at the top, but what are some of the traditional, everyday uses for UAVs, or drones, that most of us are familiar with?

ET: Sure. You mentioned kind of hobbyist type activities. Maybe some of the listeners have received a drone for Christmas that they're able to fly around, and maybe it has a camera. But some more commercial applications, we see these commonly being used for photography and videography, a lot of advertising and real estate, to take kind of advantageous videos or pictures, special events like weddings and things, but also more defense-related applications. Certainly, the military uses larger drones for surveillance and, potentially, remote strikes, and maybe to a lesser extent, for public safety applications,  again, surveillance and maybe things like search and rescue. In general, drones kind of serve as a good mobile sensing platform that can provide enhanced situational awareness for serving a variety of things.

LP: At SwRI, drone technology is doing much more. So what are some of the more advanced applications for drones?

ET: Sure. So we're seeing things, drones being used for things like border security, local package delivery. So for everybody with Amazon Prime, Amazon and other companies like that, are looking at using drones for local delivery. We're seeing them being used in agriculture to monitor crops and to be used for optimizing farming so that farmers can maximize their efficiency.

And an area that we've been spending some time looking at is inspection, so inspection of critical infrastructure, like bridges or overpass or cell towers, wind turbines, construction as well. So those can be kind of dangerous environments. We have used drones to collect imagery and data of commercial aircraft for the purposes of defect analysis, like finding a crack or a dent in an aircraft fuselage or wing. We've also done some research and development to help inspect the damage at the Fukushima Daiichi nuclear power plant, basically getting a small, agile drone inside a very radioactive area to help them understand what it looks like and help their decommissioning activities.

LP: Oh, can you talk about that project a little more? When did that take place, that inspection, and what did you find? What did you uncover?
 

enrich-team-photo-d025115-9550.jpg

Courtesy of SwRI

SwRI Manager Dr. Eric Thorn (left) and the UAS research team display the drone that earned recognition at the 2021 EnRicH Hackathon. The team utilized both on-drone processors and ground-based processing to perform simultaneous localization and mapping, exploration, decision-making, autopilot, computer vision analysis, sensing and other tasks.

ET: Sure. So back in 2011, I believe it was, there was a pretty violent earthquake offshore of Japan. And the resulting tsunami damaged this nuclear power plant on the coast and led to a meltdown of at least one of the reactors. In the cleanup effort since then, there's been a lot of effort to try and use robotic technologies to get in and assess the damage and, again, help with their decision making as to how they're going to clean up and decommission that site.

So we've worked, we were approached by one of the organizations managing that cleanup to try and develop new technologies, new robotic technologies to fly in. This is obviously a very dangerous, hazardous environment for people to be around, but it's also a very challenging environment for other types of robotic systems. There's a lot of debris. There are things hanging from the ceiling. And so using advanced navigation and control capabilities, a drone could conceivably get in, even with something as simple as a camera, and fly around and avoid those things, that debris, and help them locate and isolate some of that spent nuclear fuel.

Now, we've gone through a couple of phases of this work. We're in continuous discussions with the organization about the next phase, which would actually be to take the system that we've developed over to Japan and do some testing, initially, in one of the undamaged reactor buildings, but then eventually taking it into the damaged reactor buildings to help, again, with their decision-making.

LP: Our SwRI drones are outfitted with special software, as you mentioned. And we talked about how they are enhancing safety in situations where there is radioactivity. But there are other ways that drones could potentially save lives. Will you explain how that might work?

ET: Sure. So when we tend to talk about good use cases or applications for robots, including drones, we talk about the three Ds. So things that are dull, dirty, and, maybe most importantly, dangerous. So any time that we can use a robotic system to take a human out of harm's way, that's obviously something that we'd like to do. So defense is probably the first thing that comes to mind. Even thinking beyond the larger drones that I mentioned earlier, small groups of dismounted soldiers are using small drones to fly in hazardous and hostile environments to, again, enhance their situational awareness so they understand where it's safe or unsafe to kind of operate and identify things that they need to know about.

Again, public safety applications, we're seeing more fire departments and police departments using these as part of incident response plans. Again, I mentioned search and rescue. So if there's a stranded hiker or there's been an incident or a disaster where buildings have been damaged, or there might be people hidden or injured, that these rescuers are trying to get to, but it's dangerous for them, drones and other robotic systems can more easily get to those areas and, again, allow people to stay at a safe standoff distance while doing so.

LP: What takes our technology to the next level? How is your team enabling drones to do more than your average UAV?

ET: Sure. So a lot of the things I've talked about so far still have a person in the loop. They're either monitoring the drone or directly controlling the drone as they're trying to execute this mission or this application. Where we fit in, what we've been working on, is kind of adding intelligence and adding autonomy to not necessarily take the person out of the loop, but maybe reduce the cognitive load on that person so that they can multitask while the drone is executing a mission or executing some type of application.

So for instance, something we've been developing is an exploration capability. So basically, a remote pilot can initiate this exploration capability, and the drone can then fly around a largely unknown area and map out that area, without the person having to tell it specifically where to go. It kind of figures out on its own where it should fly to maximize how much it's seeing of the world.

LP: So how is it doing those calculations? How is it figuring things out or making decisions while it's flying?

ET: So like you mentioned, we kind of add specialized sensors, so things like camera or LIDAR or other types of sensors that help it localize. So it helps it know where it is in the world. But it also helps it see what's around it in the world. And using that information, it can detect objects, and it can detect areas that represent unknown space. And so using that, we've also developed navigation algorithms and path-planning algorithms so that it can navigate around in that environment and avoid obstacles, and then fly towards those unknown areas. So it can basically fill in gaps in the map so that we can get a better kind of comprehensive view of this unknown area.

LP: So whether conducting search and rescue or, let's say, a building inspection, how fast are these UAVs at making these calculations and conducting these various applications?

ET: They can be fairly quick, again, depending on what the operating environment looks like and what they're trying to inspect or map out. Obviously, larger spaces are going to take a little bit longer. But typically, the smaller drones that we're working with are quite nimble. And modern computational power has moved pretty far ahead. We're able to process these kind of massive amounts of data coming from the sensors and make these navigation decisions very rapidly. In our kind of development, our research and development, we tend to start slow. But as we implement and improve these capabilities, we're always interested in cranking the speed up to see how efficiently we can execute these operations and execute these kind of exploration missions.

LP: And how long has this technology been in development? How was it developed? What's the beginning story of when your team said, hey, we have the drones here, they have certain capabilities, but we think they can do more? How did that come to be?

ET: Yeah, drone technology has been around for decades. And SwRI has actually been conducting research and development related to drones for roughly 20 years. Within the Intelligent Systems Division that I work with again, our focus is kind of on adding intelligence and automation. We've really been kind of adding that to drone technology within, really, the last five or seven years or so, maybe. And again, we've been able to build on a lot of other robotics development projects that we've been involved in, more related to industrial robotics or ground robotics, and kind of pushing that envelope and pushing it into the aerial domain.

LP: Recently, the SwRI UAV technology competed at the EnRicH 2021 European Robotics Hackathon in Austria. The drone explored and mapped the interior of a nuclear power plant there and detected radiation sources without a human pilot. Can you tell us about this competition and about what the SwRI technology achieved?

ET: Yeah, this was really an exciting event to be a part of. We had put together a development roadmap for developing an autonomous exploration capability for our drone, a software stack. Because we perceive this as an industry need that was ripe for research and development, we had been executing this roadmap through several internal research projects and leveraging other internal investment.

When we learned about this EnRicH event from a European colleague, we thought that this would be a great opportunity and a really nice milestone as part of this development plan to test out what we had developed and to showcase these new capabilities. So as part of this event, we traveled to Austria to this nuclear power plant that had actually never been put into operation but was fully constructed and completed. And one of the tasks was to explore the inside of this facility, including the reactor room. So we were able to do some testing when we initially arrived.

And then in a couple of experiment runs, our drone autonomously ascended a 40-meter vertical shaft to enter into the reactor room, which was huge and had a lot of equipment that the drone had to detect and navigate around. While it was navigating around, the drone was, again, collecting sensor data and building up a map of the facility. And once our batteries began to run low, the drone returned to the top of the shaft, and then descended the shaft back to the launching point, and did this all within about 10 or 12 minutes. And then so in post-processing, we were able to generate, using that data that the drone had collected, a very high-fidelity three-dimensional map of both the shaft and the reactor room at the top of the shaft.

LP: And how was that received, or how did it compare to what others in the competition were presenting?

ET: Yeah, the competition involved both companies developing ground robotic technology and aerial vehicles, like ours. And while this wasn't technically a competition, we were recognized that at the end of having what was perceived as the most advanced exploration capability and having produced, basically, the best map of the facility compared to some of the other teams that did have drone technology.

LP: So yeah, I guess, why is it called a hackathon? I guess it's all kinds of different technology there represented.

ET: Yeah, right, all kinds of different robotic technologies, some products, a lot of systems that were kind of like ours in development. And so again, this was less of a competition, more of an opportunity for companies like SwRI and others to bring those technologies that are under development, to put through some real-world kind of applications and see where they stand, and as we're doing, continuing the development going forward to improve.

LP: So it sounds like this was a global audience. What was it like to get this technology on a world stage?

ET: It was a lot of fun. Yeah, there were teams from Germany, obviously Austria, teams from the United Kingdom, and other places in Europe. We were the only team from the United States, so it was a lot of fun, for us as researchers, to mingle with other researchers from Europe and learn about what they're working on, and understand some of the similarities and dissimilarities of our approaches, and really see how we compare and stack up to some of the preeminent research organizations across the pond.

LP: So kind of an aside here, what was it like visiting Austria? Had you been there before?

ET: No, I had not been to Austria. That was my first time to Austria, which I love traveling. I love seeing new places. So it was a lot of fun, I think a great experience for me and a great experience for the technical staff that we took with us that, honestly, did the majority of the work and were the ones that, I think, were stressing through the actual testing and the experiments that we conducted. But yeah, it was a great trip all around.

LP: Did you get to see any sights?

ET: I did, yeah. The nuclear power plant was in a fairly small town about an hour outside of Vienna. So for the majority of the time, we were kind of out maybe in the middle of nowhere. But I did get a chance to visit Vienna and do some sightseeing while we were there.

LP: Sounds like an all-around great trip. And do you think you'll be back participating in the hackathon at some point?

ET: We're discussing it, yeah. This is something that they do every other year. But they also have other similar type events in different parts of Europe, again, for companies like SwRI to bring technologies that are being developed for these specific applications and have a chance to test them out in different types of environments. So there's another one again in Austria next year that we're considering attending again. And we're kind of discussing the new technologies that we'd like to show off at that event.

LP: You mentioned the Fukushima testing going on with our technology. But is it already in use in other areas for these advanced applications that you outfit the drones for? And if so, what type of feedback are you getting?

ET: Yeah, we've done some other projects for commercial clients for different type of applications, one example of which is, we developed a drone-related system that would fly around commercial aircraft and collect data and automatically analyze that data for the purpose of defect analysis. And so that project was successful. We demonstrated it.

Some of the feedback that we've gotten in the time since is that, in that case, we still had a remote pilot flying the drone around. So obviously, the automated analysis increases the efficiency of that type of inspection once we've collected the data. But there could be additional efficiency gains from automating the actual flight around the aircraft to collect that data, in addition to the automated data analysis.

LP: So once you have a client interested in this type of technology, do you build it specifically for that client's needs, or do you kind of have one a model already ready to go that can do various tasks?

ET: Yeah, luckily, through SwRI's internal research program, we've been able to develop our own autonomy software stack. We've done that in a fairly modular and extensible way. So while we have developed a limited number of behaviors or use cases at this point, there's always the opportunity that we can take what we have and extend it into new mission profiles or new applications. And so if a customer did come and had a need for what we currently have, then that's something we could offer up as is, or with minor modification. But if they needed something that was a little bit different, or they needed a new feature or needed to extend or enhance something that we demonstrated, we luckily have kind of a baseline that we can work from. And I think that's something that's unique about the Institute, is that we can offer intellectual property like that, that we've developed using internal funds, and make that available for customers. So we're basically not starting from scratch for them.

LP: And yet get them exactly what they need. That sounds great. So how can a company invest in the SwRI drone technology? Is it they just check out our website, they reach out to you directly? How would that work?

ET: Yeah, so we have a website. We have contact information. Again, like much of the rest of SwRI, we tend to be focused on applied research and development. So kind of like I described, if a customer felt like something we had demonstrated could be modified or enhanced to meet their needs, we'd obviously be happy to engage with them and discuss how to make that happen.

We've also been considering licensing of that autonomy software stack that I mentioned. And what that could enable is, if companies didn't necessarily want to work with us or fund us to develop those new capabilities, we could potentially offer them a license to this software. And then they could, on their own, modify it or extend it to meet their needs.

LP: All right, so I want to shift the conversation a little more personal. You are an engineer, and your focus is outfitting these drones. And I'm sure a lot of people or young people who hear this think that that sounds pretty cool. So if someone listening thinks this sounds like a great career, what advice do you have to get them on this type of path?

ET: Sure. Yeah, there's a lot of demand for robotic technologies right now, including drones. And with that demand for technology comes demand for engineering and technical talent to actually make it happen. I presented on this type of work at elementary school career days, and I tend to talk about STEM, science, technology, engineering, and math. And that's all important, to plant that seed early on in science and math are important if you are interested in working in this type of technology.

For students in high school or maybe already in college, they're thinking about what type of career they want to pursue and this does maybe pique their interest, I'd encourage them to look at any classes on robotics or artificial intelligence or control systems that might be offered. And certainly, computer science and programming is incredibly important in this type of work. So much of what we do is focused on developing algorithms to make the drones complete these complicated behaviors and maneuvers, and so software is a huge part of that.

We do have several certified drone pilots, which is important and required for us to conduct our flight testing. But it takes more than just having experience remotely piloting a drone that you got for Christmas, for example. We need people who can develop these types of new capabilities, develop these new algorithms, so that, ultimately, the safety pilot becomes less and less important.

LP: So what motivates you or inspires you to continue developing this technology?

ET: It definitely excites me to think about some of the things we've talked about, some of the different ways drones could be used to help industry and help society. It's also a bit daunting to think about how challenging it could be to implement the technologies needed to do that. But that's certainly something that motivates me. And I think that that's what motivates a lot of the staff here at SwRI. It's just really trying to solve these challenging programs and problems. In my role as a research manager, I, unfortunately, get to do less of the fun and exciting engineering development. But it's still fulfilling for me to work with and help my technical staff come up with a plan to solve those problems, and then to execute that plan.

LP: And what do you envision for the future of this technology? Where do you see it in five, 10 years?

ET: Yeah, like I've talked about, I think we're going to continue to see drones and other robotic systems that have increasing levels of intelligence and autonomy, whereas right now, one or more people have to be continuously monitoring or controlling the system. Adding that intelligence will allow them to multitask and only check in on the system when the mission is complete or when there's a major problem. And with that added autonomy and added intelligence, I think that will only lead to a new use cases and applications for them.

LP: We tag on the word smart to everything. Does this apply here? Are we going to see, we're just going to see smart drones pop up?

ET: Yeah, absolutely. I think there will still, in a lot of cases or in most cases, still be a person in the loop, just to monitor. But again, any way that we can reduce the cognitive load that is placed on that remote pilot or that person that's monitoring the system is going to be helpful.

LP: So do you see, eventually, just everyday people receiving a smart drone, maybe for Christmas? That will be the new thing?

ET: It could be. Yeah, right now it's fun to fly them around. But there could be cases where people aren't quite as interested in that, but they do they would like to have a drone to follow them around and take pictures and videos. Maybe the drone selfie is something that's out there. But yeah, I think there's a lot of use cases that we haven't even thought about yet.

LP: All right, a lot of possibilities. Well, thank you for such an informative discussion to kick off the year, Eric. We've all seen drones hovering overhead, and I think we can appreciate them a little more after hearing about their many advanced uses today.

ET: Yeah, great. Thanks, Lisa.

And thank you to our listeners for learning along with us today. You can hear all of our Technology Today episodes and see photos and complete transcripts at podcast.swri.org. Remember to share our podcast and subscribe on your favorite podcast platform.

Want to see what else we're up to? Connect with Southwest Research Institute on Facebook, Instagram, Twitter, LinkedIn, and YouTube. Check out the Technology Today Magazine at technologytoday.swri.org. And now is a great time to become an SwRI problem solver. Visit our career page at SwRI.jobs.

Ian McKinney and Bryan Ortiz are the podcast audio engineers and editors. I am producer and host, Lisa Peña.

Thanks for listening.

[MUSIC PLAYING]