A significant limitation of neural networks is that the represen (cid: 173) tations they learn are usually incomprehensible to humans. We present a novel algorithm, TREPAN, for extracting comprehensible, symbolic representations from trained neural networks. Our algo (cid: 173) rithm uses queries to induce a decision tree that approximates the concept represented by a given network. Our experiments demon (cid: 173) strate that TREPAN is able to produce decision trees that maintain a high level of fidelity to their respective networks while being com (cid: 173) prehensible and accurate. Unlike previous work in this area, our algorithm is general in its applicability and scales well to large net (cid: 173) works and problems with high-dimensional input spaces.