The Hungarian algorithm can be used to calculate the earth mover’s distance, as a measure of the difference between two probability density functions, when the pdfs are described by sets of n points sampled from their distributions. However, information generated by the algorithm about precisely how the pdfs are different is not utilized. In this paper, a method is presented that incorporates this information into a ‘bag-of-words’ type method, in order to increase the robustness of a classification. This method is applied to an image classification problem, and is found to outperform several existing methods.