2015 IEEE Symposium Series on Computational Intelligence

Smartphone-based Tele-Rehabilitation System for

Frozen Shoulder Using a Machine Learning Approach

Kanmanus Ongvisatepaiboon and Jonathan H. Chan* Vajirasak Vanijja

Data and Knowledge Engineering Laboratory, IP Communication Laboratory,
School of Information Technology, School of Information Technology,
King Mongkut’s University of Technology Thonburi, King Mongkut’s University of Technology Thonburi,
[email protected]; [email protected];
[email protected] (*Corresponding author)

Abstract— Frozen shoulder is a very painful condition that Our prior work [2] proposed a framework of this tele-
affects patients’ daily life. Patients with frozen shoulder have to go rehabilitation system, which allows patients to perform the
to a hospital or medical center to get appropriate rehabilitation. correct exercise, while physiotherapists monitor and analyze the
Transportation to the hospital raises healthcare costs and the progress. This system can eliminate some useless processes and
process can be time-consuming. We have developed a tele- tends to improve conditions of shoulder problems.
rehabilitation system which allows patients to perform an at-home Physiotherapists can also customize three parameters, i.e. number
exercise. According to our existing system, it is only available for of rounds, target angle, and a reminder date and time, to be
high-end smartphones with multiple sensors that include appropriate for each patient. ROM measurements are the most
accelerometer, gyroscope, and magnetic field sensors. In this work,
important data to be collected, since they will have an effect on
we propose a novel approach using machine learning to estimate
the condition level.
the arm angle of rotation using only the accelerometer sensor.
Results show that reasonable accuracy can be obtained so that it In recent years, many researchers in the neuro-engineering
may be used with lower-end Android smartphone devices that only and rehabilitation field have tried to adapt technologies with
have an accelerometer available. A web-based interface enables the physical therapy to improve rehabilitation methods. Since
medical practitioner such as a physiotherapist to monitor and technologies are rapidly evolving, many tools and techniques
administer an appropriate rehabilitation program for more have been proposed to improve both technical and operational
effective recovery. methods.
Kim et al. [3] used the accelerometer and gyroscope sensors
I. INTRODUCTION in a smartphone to measure the rotation displacement and
Due to stressful daily routines, many people have some form therefore measure the shoulder joint range of motion. The
of shoulder problem, especially frozen shoulder. This is a accuracy of the sensors are adequate to use in rehabilitation
condition that affects a patient’s lifestyle on a daily basis. exercises. However, the scope of their work is limited to local
Patients with frozen shoulder would not be able to move their data collection and analysis.
arm as usual. They normally have to travel to a hospital or
Pan et al. [4] proposed to use two accelerometer sensors and a
medical center to get proper rehabilitation. Transportation raises
built-in smartphone sensor, which is also an accelerometer
healthcare costs and time, which may reduce their motivation and
sensor, to increase the accuracy of the shoulder joint range of
determination to seek proper treatment. Tele-rehabilitation is a
motion. However, this method requires additional devices and
useful technique that can assist in this problem by allowing
multi-stage preparation.
patients to do regular exercise at their home and communicate
with physiotherapist through today’s modern communications Ferriera [5] implemented rehabilitation games using a single
(e.g. Internet). This technique is not much different than the smartphone to measure the shoulder joint ROM and showing a
traditional face-to-face rehabilitation methods. Frequent and feedback on the computer screen. This work used an orientation
regular exercise can effectively improve the condition. Patients sensor, which is software-generated data available in the Android
are assigned to do a simple at-home exercise by raising their platform, to find the shoulder joint ROM. In particular, individual
affected shoulder to the limit of the patient’s tolerance, in order to games were used to calibrate each of the three software
break up adhesions at the joint capsule. A universal goniometer is components of the smartphone for ROM measurement.
a tool to measure the shoulder joint range of motion (ROM)
when monitoring progress [1].

978-1-4799-7560-0/15 $31.00 © 2015 IEEE 811

DOI 10.1109/SSCI.2015.120
 Other work uses image processing and Microsoft Kinect the same concept as a pill reminder application. With this feature,
device for the ROM measurement, which can enhance the patients would not forget to do an exercise task at the proper
therapy process [6] [7]. time.
According to the previous works, most of them focused on The first time, physiotherapists have to create an account for
the accuracy and reliability of the measurement, thus requiring each patient on the web application. Then, patients will be able to
additional devices and complex preparation. In our prior work, login to their account through the application on their smartphone
we proposed to use a single smartphone which contains device (Fig. 2). After logging in, all assignments and results will
accelerometer, gyroscope, and magnetic field sensors to provide be synchronized between the smartphone and server. An
the necessary monitoring measurement to enable effective tele- uncompleted task will be shown as ‘Ready’ with yellow
rehabilitation. However, there are only selected smartphone background. After finishing the task, the word will be changed to
models that have all these three built-in sensors. Therefore, in ‘Complete’ with blue background and will be changed to
this work, we propose to create a model for estimating the ‘Synced’ with green background after finishing synchronization
shoulder joint ROM by using only data from one accelerometer (Fig. 3).
sensor, which is contained in vast majority of devices. Then,
people with cheaper smartphones will be able to use this
application. We believe this is the first published attempt to use a
machine learning approach to estimate the angle of rotation using
only the accelerometer sensor in a smartphone.

II. METHODOLOGY

A. Overall System Architecture

We have developed a tele-rehabilitation system for patients
with frozen shoulder (Fig. 1). There are two main parts in this
system: client side (a smartphone) and server side.

Fig. 2. A screenshot of the Login page

Fig. 1. Smartphone-based Tele-Rehabilitation System for Patient with Frozen

Shoulder

The smartphone application was implemented on the Android

platform, since it is one of the most popular platforms with the
largest number of devices in the world [8]. The hardware Fig. 3. A screenshot of the List Item page
performance could not be summarized since there are various
A smartphone device must be placed on the elbow by using
manufacturers making devices on the Android platform. Some of
an armband as shown in Fig. 4. After choosing an exercise task,
them are using a low quality hardware to reduce the cost, thus
patients will be asked to do a simple calibration process by just
smartphone models would be a performance variable.
standing straight for five seconds, then the program will use the
A reminder notification will be sent to remind patients to average of this data to subtract from the real angle when
perform an exercise task at the assigned date and time, which is performing an exercise. Sensor data will be saved into the

812
database of the device, SQLite database, in 100 milliseconds
intervals.
At the end of the exercise, sensor data stored in the device’s
database will be sent to the cloud via the Internet. However, if
there is no network connection, data will be synchronized again
the next time the application is opened.
Regarding the web application, physiotherapists can manage
and customize patients and exercise tasks. They can also see the
results, graphs, and summary, which can be used to analyze the
condition level. Physiotherapists can customize three parameters,
which are number of rounds, target angle, and a reminder date
and time, to be matched with each patient. Sample screenshots of
the task list and results visualization are shown in Fig. 5 and 6,
respectively. The medical practitioner is able to adjust the
rehabilitation routine accordingly for each patient.

Fig. 6. A screenshot of the Results page in the web application

B. Experiment
In this experiment, two high-end smartphone models (i.e.
Samsung Galaxy S5 and Oppo Find 7A) were used to gather the
sensor data. The smartphone devices were placed on the elbow
using an armband. Three physical built-in sensors (i.e.
accelerometer, gyroscope, and magnetic field) and one software-
generated sensor, rotation vector sensor, were stored in the
device’s database every 100 milliseconds.
The main researcher was assigned to raise his left arm from
Fig. 4. A sample armband used for placing a smartphone on the elbow the initial position, stand straight with arms parallel at side, to the
target angle at 120 degrees for ten times. The results from two
smartphone models were analyzed and discussed in the next
section.
We used a machine learning approach based on multiple
linear regression (MLR) to build a model of the roll angle value
obtained from the Android software-derived rotation sensor. Raw
data from the accelerometer, gyroscope, and magnetic field
sensors were collected in a controlled manner to build a MLR
model, as shown below:

μy = 0 + 1x1 + 2x2 + 3x3 (1)

where μy is the roll angle, x1, x2, and x3 are the x, y, z components
of the raw/normalized accelerometer data, and 0, 1, 2, and 3
are the corresponding coefficient parameters of the model.
Data analysis and modeling were performed mostly using R
[9]. In particular, the linear model function lm was used to build
Fig. 5. A screenshot of the task list in the web application the multi-linear model. Weka [10] was also used to validate and
reinforce the results.

813
 III. RESULTS AND DISCUSSION
Since the common smartphone only has the accelerometer
sensor, we built a correlation model between the roll angle value
and the accelerometer sensor data. The training model was
developed using the Samsung S5. The multiple R-square value is
0.9929 and the model coefficients are presented in Table I.
For comparison purposes, the models for using gyroscope and
magnetic sensors alone are shown in Tables II and III
respectively. These results show that using simple multiple linear
regression to build a model of rotation angle based on raw
accelerometers is quite acceptable. However, the other two
mobile sensors are not suitable as standalone sensors.

TABLE I. THE COEFFICIENTS OF ACCELEROMETER SENSOR DATA

Estimate Std. Error t value Pr(>|t|)

(Intercept) -62.05380 0.60552 -102.480 <2e-16
left_accX 3.14794 0.09919 31.737 < 2e-16
Fig. 7. Histogram of deviation in the normalized model on the test phone
left_accY 5.75333 0.04141 138.952 < 2e-16
left_accZ 1.04332 0.24070 4.335 1.71e-05

TABLE II. THE COEFFICIENTS OF GYROSCOPE SENSOR DATA

Estimate Std. Error t value Pr(>|t|)

(Intercept) -62.846 1.720 -36.544 <2e-16
left_gyroX 1.930 18.735 0.103 0.918
left_gyroY 11.950 9.886 1.209 0.227
left_gyroZ 1.562 5.376 0.290 0.772

TABLE III. THE COEFFICIENTS OF MAGNETIC FIELD SENSOR DATA

Estimate Std. Error t value Pr(>|t|)

(Intercept) -126.8540 49.4854 -2.563 0.0106 Fig. 8. Goodness of normalized model on the test phone
left_magX -0.1415 0.4260 -0.332 0.7398
The Android rotation sensor is based on a software
left_magY -3.5824 0.7675 -4.668 3.76e-06 compilation with all three sensors. Further improvement, as
left_magZ 0.9680 1.4693 0.659 0.5103 indicated by the t-values, was obtained by normalizing the raw
accelerometer data first, as shown in Table IV. The
corresponding R-square value is 0.995, which may be taken to
TABLE IV. THE COEFFICIENTS OF NORMALIZED ACCELEROMETER SENSOR mean the accuracy of the regression analysis is 99.5%.
DATA

Estimate Std. Error t value Pr(>|t|) Next, to test the developed model, the Oppo phone was used
to perform the same exercise. The results show that the
(Intercept) -59.8800 0.5492 -109.030 <2e-16 normalized model provides a reasonable estimation on the test
phone when comparing to the actual angle of rotation (see Fig. 7
left_accX 31.7669 0.8596 36.956 < 2e-16
and Fig. 8). The histogram of the deviation between the
left_accY 55.5208 0.3815 145.540 < 2e-16 normalized raw accelerometer data and the rotation angle from
the rotation vector sensor is presented in Fig. 7. The results
left_accZ 10.5863 1.9846 5.334 1.36e-07
indicate a pseudo-normal distribution with the majority of the
measurements within five degrees of the actual values (for a
target angle of 120 degrees). The goodness of fit shows an R-

814
square value of 0.99 but there is an offset for both the slope and the rehab program as necessary for an effective treatment using
the intercept in Fig. 8. Nonetheless, these results are promising the provided web interface.
and future work will be undertaken to further improve upon this
model. In addition, field tests will be undertaken with lower-end For future work, we will consult and work with a medical
phones as well as with actual user usability testing. practitioner more closely to implement the developed system and
perform usability studies. Moreover, lower end smartphones will
The same MLR models were obtained using Weka with the be used to validate our hypothesis and we are planning to extend
Linear Regression module as a classifier. A summary of raw and to other platforms such as iOS and Windows.
normalized data model building based on 10-fold cross validation
is shown in Table V. Note that the correlation coefficient R is ACKNOWLEDGMENTS
shown in Weka rather than the R-square value reported earlier.
These results also show that normalization helped to improve This work is supported by a National Research Council of
model performance as expected. Thailand (NRCT) sub-grant, in the scheme of National Research
University (NRU) of Thailand. The authors would like to
Perhaps a more significant outcome is shown in Table VI for express their thanks to Ms. Katlin Kreamer-Tonin for fruitful
model testing with the test phone. It is clear from the comparison feedback and proofreading of this paper. Furthermore, special
results on the errors that normalization is needed to obtain more thanks are due to Prof. Mark Chignell and Ms. Tiffany Tong for
robust performance.
their insightful discussion with the main author in the
preparation of the final manuscript.
TABLE V. TEN-FOLD CROSS VALIDATION MODEL TRAINING RESULTS
FROM ACCELEROMETER SENSOR DATA
REFERENCES
Raw data Normalized data [1] H. M. Clarkson, “Joint Motion and Function Assessment: A Research-
based Practical Guide,” Lippincott Williams & Wilkins, 2005.
Correlation coefficient 0.9964 0.9975
[2] K. Ongvisatepaiboon, V. Vanijja, J. H. Chan, “Smartphone-based Tele-
Mean absolute error 2.8189 2.3046 Rehabilitation Framework for Patient with Frozen Shoulder,” Advances in
Digital Technologies: Proceedings of the 6th International Conference on
Root mean squared error 3.5569 2.9820 Applications of Digital Information and Web Technologies 2015. Frontiers
in Artificial Intelligence and Applications, Vol. 275. IOS Press, 2015,
Relative absolute error 7.3833% 6.0360% pp.158-169.
Root relative squared error 8.4579% 7.0909% [3] Y. S. Kim, S. Jang, and Y. Yoo, “Mobile Assessment System for Shoulder
Joint Rehabilitation: System Development and Preliminary Study,”
International Journal of Bio-Science and Bio-Technology, 2014, pp.51-60.
TABLE VI. TEN-FOLD CROSS VALIDATION MODEL TESTNG RESULTS FROM [4] J. Pan, H. Chung, and J. Huang, Intelligent Shoulder Joint Home-Based
ACCELEROMETER SENSOR DATA Self-Rehabilitation Monitoring System,” International Journal of Smart
Raw data Normalized data Home, 2014, pp.395-404.
[5] C. F. R. Ferreira, “Smartphone Based Tele-rehabilitation,” Master’s
Correlation coefficient 0.9923 0.9950 dissertation in Department of Computer Science, University of Porto, 2013.
[6] M. Huang, S. Lee, S. Yeh, R. Chan, A. Rizzo, W. Xu, W. Han-Lin, L.
Mean absolute error 5.1837 3.6958 Shan-hui, “Intelligent Frozen Shoulder Rehabilitation,” Intelligent
Root mean squared error 6.1154 4.6009 Systems, IEEE:22-28, 2014.
[7] S-C. Yeh S-H Lee, Y. Fan, “The Development of Interactive Shoulder
Relative absolute error 12.4654% 8.8875% Joint Rehabilitation System Using Virtual Reality in Association with
Motion-Sensing Technology,” Advanced Technologies, Embedded and
Root relative squared error 13.4391% 10.1108% Multimedia for Human-centric Computing, Lecture Notes in Electrical
Engineering:1073- 1082, 2014.
[8] Statistica, “Global smartphone sales from 2009 to 2014, by operating
IV. CONCLUSIONS system (in millions),” [Online]. Available:
http://www.statista.com/statistics/263445/global-smartphone-sales-by-
This work presents a smartphone-based tele-rehabilitation operating-system-since-2009/. (Accessed June 2015.)
system for patients with shoulder problems. In particular, we [9] R Development Core Team, “R: A language and environment for statistical
proposed the use of machine learning in the form of multiple computing,” R Foundation for Statistical Computing, Vienna, Austria.
linear regression to estimate the rotation angle of an Android ISBN 3-900051-07-0, URL http://www.R-project.org/, 2015.
smartphone using only raw readings from the accelerometer [10] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I. H. Witten,
“The WEKA Data Mining Software: An Update,” SIGKDD Explorations,
sensor. Further improvement was obtained by normalizing the Volume 11, Issue 1, 2009.
raw accelerometer data first. The results show that the accuracy is
acceptable and this approach should be able to use in lower-end
Android smartphones for patients with frozen shoulder problems
to perform tele-rehabilitation from home. Thus, a medical
practitioner is able to monitor the patient’s progress and adjust

815