In this paper, we consider distributed heterogeneous multi-layer mobile edge computing (HetMEC) networks, where resource-poor edge devices (EDs) upload computing tasks for processing to the mobile edge computing (MEC) servers and a cloud center (CC). To reduce total energy consumption, computation offloading and resource allocation are independently performed by each device and each server. However, due to the partial information available at each device and server, the offloading strategies may overwhelm the layers above. This may lead to network congestion, i.e., so many tasks are offloaded to the same node that this node is overloaded. To address this problem, we develop a smart pricing mechanism to coordinate the computation offloading of multi-layer devices, where the CC charges the MEC servers and EDs for computing services and network congestion. In particular, to satisfy the latency constraints of each task, we construct a Lagrangian framework where multi-agent reinforcement learning is utilized by each MEC server to determine its offloading strategies and resource allocation, so that the total energy consumption is reduced. Simulation results show that our algorithm achieves an energy consumption reduction of 28% and a decrease in congestion probability of between 28% and 100% compared to the state of the art.