In this paper, we investigate improving energy efficiency in heterogeneous cellular networks (HCNs). A Stackelberg learning game is first formulated, in which the macrocells behave as the leaders and the small-cells are followers. In the beginning of each epoch (every T time slots are defined as one epoch), the leaders update their power adaptation policies by knowing the best-responses of all followers, while the followers compete against each other in each time slot with only the leaders' action information. The hierarchy in learning procedure indicates the macrocell states in any two consecutive epochs are highly correlated. Then the small-cells' historical policy information can be leveraged to enhance the learning performance. Accordingly, a combined learning framework is established, through combining the Stackelberg learning formulation and the technique of transfer learning, to tell players how to plan the action decisions. Simulations presented show that the combined learning algorithm substantially improves the energy efficiency of HCNs.