Current automatic speech recognition (ASR) systems have to limit their vocabulary size depending on available memory size, expected processing time, and available text data for building a vocabulary and a language model. Although the vocabularies of ASR systems are designed to achieve high coverage for the expected input data, it cannot be avoided that input data includes out-of-vocabulary (OOV) words. This is called the OOV problem. We propose dynamic vocabulary expansion using a conceptual base and multi-pass speech recognition using an expanded vocabulary. Relevant words to content of input speech are extracted based on a speech recognition result obtained using a reference vocabulary. An expanded vocabulary that includes fewer OOV words is built by adding the extracted words to the reference vocabulary. The second recognition process is performed using the new vocabulary. The experimental results for broadcast news speech show our method achieves a 30% reduction in OOV rate and improves speech recognition accuracy.