Episode 7: Human Digital Twins

Lisa Peña (LP): Are you ready for a new you? Our guests today say they can make that happen. Well, at least digitally. Today, we're exploring personalized human digital twins. Detailed medical data is uploaded into a computer, forming a digital copy of a person. So how are they used, and what can a digital twin reveal?

Plus, markerless motion capture is taking athletes to peak performance. Join us as we explore high-tech health solutions on this episode of Technology Today.

[MUSIC PLAYING]

We live with technology, science, engineering, and the results of innovative research every day. Now let's understand it better. You're listening to the Technology Today Podcast presented by Southwest Research Institute. Hello, and welcome to Technology Today. I'm your host Lisa Peña.

Digital twins are a fascinating concept. Your digital twin is basically you, all of your health data down to your cells, coming to life in a computer. Your twin can provide detailed medical information about you like never before.

Our guests today are Kase Saylor and Dr. Dan Nicolella, Southwest Research Institute engineers taking biomechanics and human performance research to the next level. Thanks for joining us, Dan and Kase.

Left to right: Dr. Dan Nicolella, Lisa Peña, and Kase Saylor

Dr. Dan Nicolella (DN): Thanks for having us.

Kase Saylor (KS): Yeah, it's great.

LP: So let's start with Kase. We just covered the basic definition of a human digital twin, but how would you describe it?

KS: I think you got the basic parts of it. And the idea is that the human body is a very complex system. Digital twins have been using the past in other technology areas, such as high-performance aircraft, large manufacturing processing plants.

Because we're such a complex system, if you will, this idea of being able to collect a lot of different data from different sensors, what we call modalities, different types of sensing, we can take that information, group it together, and then start looking at it in a very holistic perspective. And then you start to understand the different measurements you're getting from your body, how those all work together.

So the idea is to take that, and essentially create a digital representation of you personally so that you can start getting to personalized performance, or even personalized health, because you as an individual is unique from anyone else. And so if we can start using this idea of a twin that represents you very specifically, we can start doing very specific things for your health and for your performance.

LP: And Dan, if you can enlighten us a little bit, how did this concept originate?

DN: Well, some of the work that we've been doing in the past involves converting high-fidelity medical images, like CT scans or MR scans that you would get at the hospital or your doctor's office, into a digital representation of that person. So we take that digital image data and turn it into a physics-based model of a person. So we can then use that model to understand what's happening internally within our bodies. So if we're running, or if we're playing football and we get hit, we can understand the forces that are occurring in our joints, in our bones, and then maybe use that to design safer equipment that an athlete or a soldier would wear.

LP: So I'm looking at this slide from one of your presentations, and it looks really thorough. We're talking about cognitive information, nervous system, skeletal system, muscle system, even down to nutrition. What type of information are you collecting to form the digital twin?

KS: Well, that certainly is very high-level look at the different types of data. What's really interesting is we're at a point now where technology is such that we can grab a lot of different information. People are wearing wearables. I think we're at a point where people are accepting, and understanding, and willing to use different types of technical devices to get information about themselves.

So you can take things like the watch you're wearing, or heart rate, or, as Dan mentioned, medical imaging, breath analysis, a lot of different things. You can start grabbing that type of information to make a better picture of what your body is doing and how it's reacting to your environment. And so it's really a lot of different things.

And we're at a point in time and technology where there's so many different types of sensors that are out there. So from our perspective is you don't throw anything out. You grab all the data and you start to make sense of it.

DN: Yeah, that's a great point. We're collecting so much data about ourselves, from wearables, from what we eat, how much we exercise every week. And this concept allows us to aggregate all that data and understand how it propagates within our body, and through our body, and affects how we perform in life. And maybe give us some clues about our future risk of injury, or how we're performing for an athlete or a soldier, and then how to improve those performances.

LP: So walk us through this process. We go into your lab. We want our digital twins created. What does that look like? Are you putting sensors? Are you doing blood samples? How does this work?

DN: Great question. So what we do now is we can create a digital twin of the person using a medical image data set, for instance. So we could go over to a medical imaging center, put that person in a CT scanner or an MR scanner, and generate a set of images of that person, and then use that image set to create a digital representation of that person.

Then we come into our lab. We can capture how that person moves using our markerless biomechanics system, which I think we'll talk about in a minute. We can use that information to drive that digital model that we've just created of that person, and then start to understand what makes them move. Are they moving in a way that is optimal for their body structure? Are they doing something that could put them at risk of an ACL tear if they're running, or cutting, or something like that?

LP: So you mentioned some examples, but the big picture, what are you trying to find out about a person based on their twin? What can the digital twin tell you?

KS: Well, I think what we're really trying to do is, because we have this digital representation, we can get a better understanding of how different, whether it's interventions, or if you want to say exercise, or nutrition, how do those all play together?

Again, most of what you see in the literature and what's out there is based upon very large populations. Right now, it's probably the easiest or the best way to understand, like, say, understanding nutritional advice, or exercise, is that they take very large populations. OK, how does this work? So there's some statistical significance to the findings.

But what we want to do is say, well, what is that for you personally? So that's what we want. We want to take all the information and understand, what's good for you, Lisa. It's like, what is best nutrition-wise? What's your best exercises? How should you move?

Something that comes up a lot is like, well, if you could teach somebody to swing a club like Tiger Woods, that'd be great. Well, not necessarily. Actually, we believe that Tiger is very unique. He has a very unique musculoskeletal system. The way he swings a club is best for him personally. And so that might not mean the best way for me to swing a club. So it's that idea of getting away from very generalized type of approaches to something that's very specific to the individual itself.

LP: So tell me what you see when you create a digital twin. Is this like a little person on the screen giving you information? Is this just a collection of numbers? What does the digital twin actually look like?

DN: Well, we think it's both. We certainly would have a digital representation of that person on the screen so we could look at it, and move it around, explore what it looks like. But that's just part of it. The other part of is all the data that we're collecting about an individual, as Kase mentioned, we're collecting so much data from wearables and tracking our health, we can use that in combination with this digital representation of the human to understand what's best for that person.

LP: So this is a concept that is still in the development phase. Is that correct?

DN: Correct.

KS: Yes.

LP: OK. So where are you in the process of developing it and maybe taking it further?

DN: So as I mentioned, we have quite a bit of experience and capability in developing the digital representation of a person from medical image data. We can track how that person moves in a specific environment with our markerless biomechanics system.

The part that we're working on integrating is how to incorporate all this additional data that is being collected about individuals, either your health care data, or performance data if you're an athlete. How to roll all that data into a digital representation of a person. What's the best way to process that data and gain information about how that person performs?

LP: So going through the list of input, of different health data, it looks like you collect most of this medically through, as you mentioned, CT scans or MRIs. I'm curious about the nutrition aspect of this. Are you asking people what they eat and putting it in the system, or are you suggesting what they eat? Maybe it's a little bit of both?

KS: Well, yeah. I think certainly a big part of that is the ability to ask people to enter that information. And we all know that that information is good as the person that's actually entering the information. However, that's still a standard practice in the communities, is that, at least to self-report what you're doing.

For now, that's probably the best approach, or the most current approach is to provide that. But we're also looking at things, and we have a colleague here at the Institute who does a lot of work in breath analysis. And so there are abilities to get that information also just from breathing into a tube, which is very exciting. It's another area that we're looking at.

But yeah. So nutrition, it's certainly harder. And actually, we'd like to get away from a very invasive type of approaches. We'd like to get away from blood draws. They're useful, but how can you do something that you could do daily? People don't want to get their blood drawn daily. Some people never want their blood drawn. So what can we do to do that?

LP: So you mentioned athletes. Who else do you envision using this technology?

KS: So certainly, when we say athletes, we also refer to tactical athletes. And that would be our service members, whether that's just your different active duty components, or even your special operations. They consider those tactical athletes. So that's certainly a big push for us, is how can we help out our military?

And that comes down to also helping long-term careers. You have individuals that are put in situations way outside of an athletic event, but they're very, very challenging, both from a physical perspective, a cognitive perspective. What can we do to help understand and track the career, just the impact that has on the individual? So that's definitely an area that we're looking at.

But certainly, we believe that this approach is universal, because, since we're all human, we're all individual. That's our approach. So whether it's helping the athlete score more baskets in a game, or helping your grandmother have a more productive maybe less pain-free life, these are all areas that we're interested in.

LP: Is it possible for a digital twin to diagnose illness? Like you collect all this information, then you see maybe something's not quite right. Is that something that you're looking at?

DN: Yeah, and I think that's the vision. And it's not only us. There are a lot of organizations across the country, and across the world really, who are trying to get a handle on, get their arms around the amount of data that we're collecting, and how to process that better. And the idea is that by processing this data in aggregate all together, we might be able to detect diseases or illnesses at an early stage so that people can get treatment sooner before it's too late.

LP: And that's really an important aspect of this, I would say.

DN: Absolutely.

LP: That's great. So does something like this require updating? Let's say someone has a digital twin made. Maybe they come in...

DN: Great question.

LP: ...once a year, something like that?

DN: Yeah. And we envisioned this being updated constantly. So it's updated daily if you're collecting data from a wearable, for instance. It's updated periodically if you go back to your health care provider and get a new X-ray or new CT scan. It's updated as often as data is collected on that person.

LP: So who are your clients for this? Who do you think, we have athletes, but do you envision companies or corporations doing this for employees, or maybe it is a team doing this for all of their athletes. What is your goal?

DN: Well, right now, this is a concept. We're developing this concept. And our goal is to offer this to high-performance organizations initially. So high-performance athletes or teams, and as Kase mentioned, perhaps the special operations community and our military. But we can see this being ubiquitously applied across the population eventually.

LP: I'm thinking about those DNA kits that are really popular right now. People are using them to figure out their genealogy. Do you foresee this concept taking on that level of popularity? Maybe everyone's going to want a human digital twin at some point.

KS: Yeah, that's a great question, and I think that's true. I mentioned earlier that I think we're at a time where people are very accepting and receiving of this idea of data and information about themselves. We're in a time where you have so much information being collected about yourself that people care. So I agree. I think the idea, we would love it that, essentially, you have the ability that everybody has a twin. Precision medicine would absolutely be benefited by this approach, and just overall general health. And I think there's just a curiosity within people to understand themselves better. So I do believe that if we get to that point, that it would be something that everybody would want.

LP: How soon do you see this happening?

KS: Well, that's a great question. We're starting it from what we know now and what we can do now. And we have to be mindful as we go down this road and this approach as to not do something that precludes us from doing something later. So we don't want to design something now that requires very specific or only certain types of sensors or data, and build the system that way, because what if, two years from now, there's some ground change, some new technology that comes out that allows us to get even more information?

So it's hard to say when. I think we have, currently, what we're working on could be applied very soon within the high-performance teams. But this is really just the base or the foundation, and I think we'll build upon that.

LP: Are there any privacy issues or concerns with collecting so much data about a person and how this information is stored?

DN: Yes.

[CHUCKLING]

LP: Short answer is yes.

DN: Short answer is yes.

KS: Yes.

LP: Could you expand on that?

KS: Yeah, no, absolutely. That's a huge concern. What we're very careful with is trying to, certainly right now, look at this data in a way that we don't know who the individual is itself. So you got to put in protections that say, the ability to look at data and understand that, but it's a number, or it's not actually tied to somebody personally.

Because you're talking about potential medical data which is heavily regulated and protected, which is understandable. And so you have to be very careful with that. So yes. It's a difficult challenge, and it's something that would have to be worked through.

LP: OK, so we just want to emphasize. This is a concept in development right now. There are different aspects to look at and still areas of it that need to be thoroughly developed further. But hopefully, one day, we can all go to our little digital twin and find out more about ourselves. That would be super cool.

KS: Yes.

DN: Yes.

LP: Are you guys excited about the idea of creating this and being on the front lines of making this a reality?

DN: Oh, yeah. Yeah, this is something that we're very excited about. As I mentioned, we've been working on the periphery of this idea for a long time.

And a lot of things are starting to come together that will allow this to come to fruition, whether it's this year, or next year, or sometime in the future. As Kase mentioned, we think we're going to take an incremental approach. There's some things that are ready now to start applying to the development of a human digital twin, and that's what we're trying to do.

LP: All right. Human digital twins, here they come. Love it. So let's move on to another hot topic in your human performance work, and that's markerless motion capture. And it does sort of connect to the digital twin in that it's a great way to collect data about a person. So what is markerless motion capture?

DN: So markerless motion capture allows us to capture the movement of a person without all the specialized equipment you would typically need in a laboratory setting. So in a laboratory setting, we need an array of highly-specialized infrared cameras and infrared-reflecting markers that are placed on the person that we're tracking. And these markers have to be placed at very specific anatomical locations. And that's time-consuming to put those markers on a person.

So for instance, if we were to capture the motion or movement or biomechanics of an individual, it would take 30 to 45 minutes, perhaps an hour just to get that person ready to capture their motion. What our system allows us to do is we can have an individual walk essentially in front of a single or a pair of cameras, and we can start collecting their motion immediately with the same level of accuracy as you would get in a traditional biomechanics laboratory.

LP: So it's faster.

DN: It's faster.

LP: Easier.

DN: And it's just as accurate.

LP: OK. How does that work without all the, well, first of all, what is a marker?

DN: It's a secret.

LP: [CHUCKLES] Top secret.

DN: Top secret.

LP: But OK, so we were talking about the old approach, putting, like, is a marker like a sensor that you, is it like a...

DN: Yes...

KS: I'm sorry.

DN: Go ahead.

KS: You've probably seen this before when you see behind the scenes and how they make the movies. It looks like little ping pong balls. They're little silver type of spheres. That's what a marker is.

And so essentially, those are put on the wrist, the elbow, different parts. And you can have very huge amount of markers depending on what you're trying to capture. So that's what a marker is.

LP: Now, you're eliminating the need for all of these markers, and you're doing this with a camera. So is this software-based is what it is.

DN: So here at the Institute, we've combined machine vision, deep learning, artificial intelligence with biomechanics to create a software system that can process a video of an individual and output their biomechanics. So using off-the-shelf videos and software that we've developed here at the Institute, we can capture a person's biomechanics very accurately.

LP: So let's do a quick definition of biomechanics. Basically this study of movement? What is it?

DN: Yeah. So biomechanics is the study of movement and function and form of the human body. It's the application of mechanics and dynamics to understanding how a person moves and behaves.

LP: OK. And why do we want to know that?

DN: Because it's cool.

LP: It is. [CHUCKLES] And that's reason enough for me. But there's a bigger picture, right?

DN: Yeah, there's a bigger picture. There's a whole wide variety of applications, from orthopedics, from rehabilitation. So for instance, if you injure your shoulder and you have to rehab, it's important to track your motion to make sure that you're doing those exercises correctly.

Oftentimes, it's used to understand how a person walks, and how their gait, gait is the patterns of how you walk, how that affects your risk of developing arthritis in your knee or your hip. And so by understanding how a person moves, we can understand what the risk of developing a musculoskeletal disease might be. We can help them rehab from an injury.

LP: So without all those markers, how accurate is it? Are they just standing in front of a camera and it's just picking up their motion? How accurate is this approach?

KS: Yeah, that was really the driving part of our research was, we're not the only people who've created markerless motion capture. There are definitely other companies out there. But what we decided early on is that we were looking for the highest level of accuracy that we could possibly do without the markers.

And so we've tested our system against probably two of the biggest marker-based systems out there. And our accuracy is, we would say that it's basically on par with the marker-based systems, and so that was, to be honest, was surprising to us when it first happened.

You have these ideas. You put your time and effort into the research. And we were actually generally shocked at how well it worked, which is always great. It's awesome as a research to get those kind of results.

LP: Yeah. So this system is in use now, correct? Is it being used?

KS: We have. Yeah. We have certain clients that are using this system. Or we're developing some applications for clients that use this technology.

LP: Is this specifically for athletes, or for other areas?

KS: Current implementations are for athletes themselves.

DN: We have both athlete-based applications and clinical applications.

KS: That's true.

LP: So people in the medical field are interested.

DN: Mm-hm. Oh, for sure. Definitely, yeah.

LP: How interested are athletes, though? I'm sure it's improving their game.

KS: Yeah. And actually, one of the reasons why we went down this area of research is that we talked to a number of individuals and experts within the area of high performance. And this is one of the things they said they really wanted. And so it's always nice when someone tells you, hey, can you do this? And so we moved forward with that.

But yeah, it's great. It's definitely used a lot by your strength and conditioning coaches, your sport science. They're the ones who are probably the most interested in it. But again, that gives them ability to give feedback to their athletes.

And in line with the digital twin, by using a markerless system, this can be done daily. So you can start to see trends over time. So it's not as time-consuming. You can have somebody come in, do a set of movements that you want to see, and then they just go on.

And so we want to give the community ability to do a lot more capturing of data, again this ties to the digital twin itself, because now you're capturing all this data. What can you do with it? So yeah, I think we're going to see a lot of people very excited and interested in using this, because it's going to give them more information either about them personally or about the athletes that they're training, and that helps them with their decisions down the road.

LP: OK, so I just want to understand this. You walk in to your lab, stand in front of a camera, and the camera is reading your movements and analyzing them using an algorithm, or is that correct? How does it work?

DN: Correct. Yes. Yeah, it's a software system that reads the video and outputs your movement patterns, your biomechanics.

LP: And there are some sensors also?

DN: No sensors. So that's a key concept. We're trying to do this without any additional sensors that have to be added or applied to the person.

LP: OK. All right. So markerless motion capture, again, tying back to human digital twin, because it's a way to collect a good amount of data about a person, and potentially form their digital twin, so really cool.

KS: Yeah, absolutely. And even beyond that, we have a concept of using this, if you're working out, you can capture that data immediately. So say you're doing squats. You can tell how well you're doing with that. It can count your repetitions.

And so even the ability to, you mentioned walked into our lab. We don't even really want the idea of walking into our lab. We want this idea that you just go into your normal space that either you're competing, or you're working, and it's just collecting that data for you as you go through the process.

LP: So cool. So you have to have a camera set up in your...

KS: That's right.

LP: Ah. Even better. So before we go, we do want to mention the Human Performance Summit. It's a two-day conference where sports science professionals come together to learn about and discuss these software-based human performance solutions. So what's happening at the summit this year?

KS: So this is our third summit. And it's exciting. A few years ago, Dan and I had talked, we wanted to really bring San Antonio into prominence in the area of human performance.

We have the perfect mixture here in San Antonio, whether you have, certainly, Southwest Research Institute, but our deep ties within the military, military medicine. We have the biomed work that's done in San Antonio. And so we want to start building this idea that San Antonio's a place to come where there's a lot of great interesting research done in the area of human performance. So that was the impetus behind this summit.

And so we're able to bring in some great speakers from around the world that we wanted them to talk on various issues or subjects that directly relate to human performance. So it's not just the strength and conditioning conference. It's not just the nutrition conference.

We're also giving our speakers ample time to present. A lot of times you go to the conferences, you might have 20 minutes. It's a very quick, maybe, sales pitch. That's not what we want. So we give them a lot of time to dig deep into a very technical topic, and then time for a question and answer.

It's great because it brings in a very diverse group of attendees. You have your professional sports folks. You have your special operations. You have researchers. It's a great time for a lot of different communities to come together around this idea of elite human performance, and so that's the idea.

LP: All right. So it's a really comprehensive look at human performance. And that's coming up on July 18 and 19 on our Southwest Research Institute campus. That's here in San Antonio. And as you mentioned, you're going to have some awesome speakers covering a range of top picks, including Dan.

DN: Correct.

LP: So you'll be talking about biomechanical analysis.

DN: That's right.

LP: So so much to be covered there. And you can register on our website, swri.org, but we'll also include the registration link on the podcast page for this episode. So again, your human performance initiatives are fascinating. Having the ability to recreate a person with a digital snapshot, and then using software for detailed health performance assessments. I think it's an important and innovative tool. So thanks so much for telling us all about it and sharing your work with us and joining us today.

KS: Thank you very much.

DN: Thanks. Thanks for having us. This was fun.

LP: And that wraps up this episode of Technology Today.

Subscribe to the Technology Today podcast to hear in-depth conversations with people like Kase Saylor and Dan Nicolella, changing our world and beyond through science, engineering, research, and technology.

Connect with us on our Southwest Research Institute social media pages on Facebook, Instagram, Twitter, and LinkedIn for the latest Technology Today episodes and much more. Thanks for listening.

[MUSIC PLAYING]