Recent advancements in privacy-preserving deep learning (PPDL) enable artificial intelligence-assisted (AI-assisted) medical image diagnostics with privacy guarantees, addressing increasing concerns about data and model privacy. However, intensive studies are restricted to shallow and narrow neural networks (NNs) for simple service (e.g., disease prediction), leaving a gap in exploring diverse inferences. This paper proposes TrustMIS, a trust-enhanced inference framework for fast and private medical image segmentation (MIS) and prediction services. Based on two-party computation, TrustMIS introduces lightweight additive secret-sharing tools to safeguard medical records and NNs. Complementing existing PPDL schemes, we present a series of secure two-party interactive protocols for linear layers. Specifically, we optimize the secure matrix multiplication by reducing the number of expensive multiplication operations with the help of free-computation addition operations to enhance efficiency (bringing 1.15× ∼2.64× savings in both time and communication costs). Furthermore, we customize a fresh secure transposed convolutional protocol for MIS-oriented NNs. A thorough theoretical analysis is provided to prove TrustMIS’s correctness and security. We conduct experimental evaluations over two benchmark and four real-world medical datasets and compare them to state-of-the-art studies. The results demonstrate TrustMIS’s superiority in efficiency and accuracy, improved by 1.1× ∼ 54.4× speedup in secure disease prediction, and 5.56% ↑ ∼ 11.7% ↑ accuracy in secure MIS.