Perception system performance can vary dramatically with contextual factors such as environmental geometry, appearance, and other phenomena. In this work we present a theoretical framework for understanding the role of context in perception and discuss three approaches for predicting probabilistic performance from observations by efficiently learning local performance models. We compare these approaches with experiments on the monocular and stereo visual odometry systems for a ground robot, and show that they can effectively predict system failures in a wide variety of environments.