High-performance poly(1,4-butylene terephthalate (PBT) nanocomposites have been developed via the consideration of phosphorus-containing agents and amino-carbon nanotube (A-CNT). One-pot functionalization method has been adopted to prepare functionalized CNTs via the reaction between A-CNT and different oxidation state phosphorus-containing agents, including chlorodiphenylphosphine (DPP-Cl), diphenylphosphinic chloride (DPP(O)-Cl), and diphenyl phosphoryl chloride (DPP(O3)-Cl). These functionalized CNTs, DPP(Ox)-A-CNTs (x = 0, 1, 3), were respectively mixed with PBT to obtain the CNTs-based polymer nanocomposites through a melt blending method. SEM observations demonstrated that DPP(Ox)-A-CNT nano-additives were homogeneously distributed within PBT matrix compared to A-CNT. The incorporation of DPP(Ox)-A-CNT improved the thermal stability of PBT. Moreover, PBT/DPP(O3)-A-CNT showed the highest crystallization temperature and tensile strength, due to the superior dispersion and interfacial interactions between DPP(O3)-A-CNT and PBT. PBT/DPP(O)-A-CNT exhibited the best flame retardancy resulting from the excellent carbonization effect. The radicals generated from decomposed polymer were effectively trapped by DPP(O)-A-CNT, leading to the reduction of heat release rate, smoke production rate, carbon dioxide and carbon monoxide release during cone calorimeter tests.