Supervised machine-learning algorithms are used to solve classification problems across the entire spectrum of computing platforms, from data centers to wearable devices, and place significant demand on their computing abilities. In this paper, we propose semantic decomposition to build a hierarchical framework of classifiers, a new approach to optimize the energy efficiency of supervised machine-learning classifiers. We observe that certain semantic information like color/texture are common across various images in real-world datasets for object detection applications; exploiting these common semantic features we can easily distinguish the objects of interest from the remaining inputs in a dataset at a lower computational effort of the classifier. Yet, state-of-the-art classification algorithms expend equal effort on all inputs, not exploring the semantic features. To address this issue, we present the concept of decomposition of inputs into relevant semantics where we build a hierarchical framework of classifiers, with increasing levels of complexity, trained to recognize the broad representative semantic features of the input. The degree of confidence at each classifier’s output in the hierarchy is used to decide whether classification can be terminated at the current stage or not. Our methodology thus allows us to transform any given classification algorithm into a semantically decomposed hierarchical framework. We use color and texture as our distinctive traits to carry out several experiments for object detection from the Caltech dataset by decomposing the input image into the two broad features on Artificial Neural Networks. Gabor filtering/HSV transformation is used to extract the color/texture components respectively. Our experiments demonstrate the proposed approach yields 2.31 x reduction in average number of operations per input which translates to 1.93 x improvement in energy over hardware implementation.