In this study we describe a new approach to extract layout of unconstrained handwritten letters such as those sent by individuals to companies. The proposed model uses a hierarchical combination of Conditional Random Fields (CRFs) which gives access to various levels of the layout interpretation. The analysis proceeds by decreasing the resolution and increasing the abstraction of the document, starting from high resolution analysis (pixel level), to a low resolution of the layout structure. Informations of high resolution are used to bring a specific prior knowledge of the layout like presence of textual information. Experiments have been performed on the RIMES database composed of more than 5000 handwritten letters. Good results have been reported showing the capacity of our approach to extract simultaneously the physical and logical layouts.