Fluorescence molecular tomography (FMT) is a promising and noninvasive in vivo functional imaging modality. However, the quality of FMT reconstruction is limited by the simplified linear model of photon propagation. Here, an end-to-end three-dimensional deep encoder-decoder (3D-En-Decoder) network is proposed to improve the quality of FMT reconstruction. It directly establishes the nonlinear mapping relationship between the inside fluorescent source distribution and the boundary fluorescent signal distribution. Thus the reconstruction inaccuracy caused by the simplified linear model can be fundamentally avoided by the proposed network. Both numerical simulations and phantom experiments were carried out, and the results demonstrated that the 3D-En-Decoder network can greatly improve image quality and significantly reduce reconstruction time compared with conventional methods.