Attribute based image retrieval has offered a powerful way to bridge the gap between low level features and high level semantic concepts. However, existing methods rely on manually pre-labeled queries, limiting their scalability and discriminative power. Moreover, such retrieval systems restrict the users to use only the exact pre-defined query words when describing the intended search targets, and thus fail to offer good user experience. In this paper, we propose a principled approach to automatically enrich the attribute representation by leveraging additional linguistic knowledge. To this end, an external semantic pool is introduced into the learning paradigm. In addition to modelling the relations between attributes and low level features, we also model the join interdependencies of pre-labeled attributes and semantically extended attributes, which is more expressive and flexible. We further propose a novel semantic relation measure for extended attribute learning in order to take user preference into account, which we see as a step towards practical systems. Extensive experiments on several attribute benchmarks show that our approach outperforms several state-of-the-art methods and achieves promising results in improving user experience.