NMSU engineer using smartphone sensing to identify falls risk
A New Mexico State University engineer is exploring one of the fastest emerging fields as a means to predict and prevent a pervasive problem. Phillip De Leon, associate dean of research for the College of Engineering and professor of electrical and computer engineering, and recently graduated doctoral student Matthew Martinez are using artificial intelligence and deep learning to help predict a person’s risk of falling.
“Nearly 3 million older adults in the United States fall each year with serious, sometimes life-changing, consequences – even death,” De Leon said. “There are a lot of researchers trying to identify the patterns that predict the risk of falling so that measures can be taken to prevent a fall, such as using a cane or a walker or taking physical therapy.”
In fact, De Leon’s interest in this area began with NMSU colleagues in the Department of Kinesiology about five years ago, who developed questionnaires that helped identify the risk of falls.
At about the same time, smartphones had begun to use advanced motion sensors, which include accelerometers and gyroscopes. De Leon wondered if a biomechanical approach to falls prediction could be useful. De Leon and Martinez found that research in this area had been done, using data collected from 3-D motion-capture systems and pressure-sensitive walkways, both of which are expensive and not widely available.
“It was nothing like the convenient package that we have in a smartphone. The appeal of the smartphone is its ubiquity – everyone has one, it’s a low-cost platform, it has an array of sensors, and can transmit data,” De Leon said.
At the time, De Leon was following the field of deep learning and eagerly looking for an application in which he could apply it.
So De Leon wrote the code to create an app for the smartphone that would capture inertial measurement or motion data when someone is walking. The signals were analyzed to identify certain features in the gait patterns that might indicate a heightened risk of falls, such as stride length – when stride length shortens the risk of fall increases. Other indicators are cadence – how fast the legs are moving; forward linear velocity – how fast someone is actually moving; percent of gait cycle spent in double support – a brief period when both feet are on the ground; and percent of swing phase – the time legs are actually swinging. Each of these variables are known to predict falls risk.
In partnership with Electronic Caregiver, a Las Cruces-based company that provides electronic health and safety monitoring, De Leon and Martinez collected data over a five-year period from nearly 700 subjects ranging between 40-85 years of age.
With data gathered, then comes the complex task of training models to identify and predict fall risk. This relies on the use of deep neural networks, “a network model that somewhat mimics what we think is happening in the human brain,” De Leon said. As part of his doctoral research, Martinez developed extensive software codes for the architecture and simulation of the deep neural network.
A neural network is like a fabric of connections. As input data moves through one layer of connections it weights factors related to fall risk and creates output that goes to the next layer of connections and so on. The network is trained to learn patterns that identify fall risk.
“So we have all of these layers that are taking input, the motion data from the phone. Along the way, the data are getting crunched, processed and churned while identifying for patterns in that data. The resulting output indicates if someone has a high risk or a low risk of falling.”
De Leon also used the motion capture laboratory located in the Department of Mechanical and Aerospace Engineering. Infrared markers, about 35 of them, are placed on a subject’s limbs and torso. About 12 infrared cameras are placed around the roof line of the room. When a subject walks through the capture zone, the system records the coordinates of every marker. Software turns that information into a skeletal model as the subject is moving, and yet more software extracts data about the gait of the subject, yielding very detailed biomechanical data.
Simultaneously, a smartphone attached to the subject’s hip collects motion data.
“The motion capture system provides us with ground truth data, which we use as target data for the deep neural network, and the smartphone data provides us the input.”
Both data are recorded in parallel and synchronized in time so that certain markers can be identified. Comparison and evaluation of the results validate the accuracy of the method.
“Through that research we demonstrated that it is possible to use smartphone data to make accurate predictions,” De Leon said.
“The ultimate goal is to develop a very low-cost technique that could be used on demand. A person could turn on the app, walk a few steps, and the app would make the prediction. If the person is at risk of falling, they could take preventative measures. The Holy Grail would be to monitor falls risk continuously, just like the devices that track steps,” De Leon said.
“Deep neural networks and machine learning are fundamentally changing the world. This is the current technology in use for voice and image recognition, language translation, even self-driving cars,” De Leon said. “The potential is great in all aspects of use. It’s an exciting area and it’s a great time to be working on this research.”
De Leon and Martinez’s research was published in March in the IEEE Journal of Biomedical and Health Informatics. Martinez was chosen as outstanding researcher in machine learning at the Lenovo Artificial Intelligence Challenge in 2018 for his research in deep learning applied to falls risk prediction.
