As mobile devices become more capable, the need for customization of mobile services becomes increasingly important for users. Nowadays, mobile device sensors are able to collect information from users throughout the day, which gives insight into their profiles. The advent of modern System-on-Chip architectures has enabled mobile devices to tackle machine learning-oriented problems heretofore reserved to desktop computers. The recent success of deep learning makes it a method of choice for understanding the complex user patterns on mobile devices. Unfortunately, training a deep neural network is often considered as too computationally intensive on mobile devices. To address this issue, we consider transfer learning, a technique that aims to take advantage of deep learning features that have been previously learned to improve the learning performance of another neural network. In this paper, we propose a deep learning framework TransferCL, that supports transfer learning on mobile devices. Our approach relies on the collaboration of the multicore CPU and the integrated GPU to accelerate deep learning computation on mobile devices. We consider three major issues - performance/portability tradeoff, power efficiency, and memory management, propose our approaches, and conduct experiments to evaluate them.