It is important, yet a challenging procedure, to segment bones and cartilages from knee MR images. In this paper, we propose multi-atlas context forests to first segment bones and then segment cartilages. Specifically, for both the bone and cartilage segmentations, we iteratively train sets of random forests, based on training atlas images, to classify the individual voxels. The random forests rely on (1) the appearance features directly computed from images and also (2) the context features associated with tentative segmentation results, generated by the previous layer of random forest in the iterative framework. To extract context features, multiple atlases (with expert segmentation) are first registered, with the tentative segmentation result of the subject under consideration. Then, the spatial priors of anatomical labels of registered atlases are computed and used to calculate context features of the subject. Note that these multi-atlas context features will be iteratively refined based on the (updated) tentative segmentation result of the subject. As better segmentation result leads to more accurate registration between multiple atlases and the subject, context features will become increasingly more useful for the training of subsequent random forests in the iterative framework. As validated by experiments on the SKI10 dataset, our proposed method can achieve high segmentation accuracy.