2013 IEEE International Conference on Computer Vision

Human Attribute Recognition by Rich Appearance Dictionary

Jungseock Joo1 , Shuo Wang1,2 , and Song-Chun Zhu1

1
Center for Vision, Cognition, Learning and Art,
Depts. of Statistics and Computer Science, UCLA
{joo@cs, sczhu@stat}.ucla.edu
2
School of EECS, Peking University
[email protected]

Abstract


  

 

We present a part-based approach to the problem of hu-

man attribute recognition from a single image of a human
body. To recognize the attributes of human from the body
parts, it is important to reliably detect the parts. This is
a challenging task due to the geometric variation such as
articulation and view-point changes as well as the appear-

 

 
ance variation of the parts arisen from versatile clothing
types. The prior works have primarily focused on handling
geometric variation by relying on pre-trained part detectors Figure 1. Part learning methods in Poselet-based approach [3] and
or pose estimators, which require manual part annotation, our method. (left) Each poselet is learned from the examples of
but the appearance variation has been relatively neglected similar geometric congurations of keypoints (red marks). (right)
in these works. This paper explores the importance of the We learn our parts based on appearance to preserve attribute-
appearance variation, which is directly related to the main specic information.
task, attribute recognition. To this end, we propose to learn
a rich appearance part dictionary of human with signi- tection itself, is a challenging task, as noted in [2], due
cantly less supervision by decomposing image lattice into to the geometric variation such as articulation and view-
overlapping windows at multiscale and iteratively rening point changes as well as the appearance variation of the
local appearance templates. We also present quantitative parts arisen from versatile clothing types. The existing ap-
results in which our proposed method outperforms the ex- proaches [2, 5] have mainly focused on resolving the rst
isting approaches. issue - geometric variation of parts - by adopting pre-trained
part detector or pose estimator.
However, there are a few limitations in these approaches.
1. Introduction First, appearance variation of parts (hat vs. non-hat) is not
taken into account in part learning or detection. The visual
We present a part-based approach to the problem of hu- part dictionary or part appearance model of pose estima-
man attribute recognition from a single image of a human tion is usually obtained by geometric constraints and not in-
body. Human attributes, enriched textual descriptions of formative for attribute classication. In other words, these
people such as gender, hair style, clothing types, provide are generic part templates that do not have to distinguish
ne-grained semantics. This is a practically important prob- different types of appearance in their learning objectives.
lem which can lead to many applications such as surveil- Apparently, this is not the case for the problem of attribute
lance [19] or image-to-text generation [22]. recognition because it is the appearance type of body parts
Since many attributes can be inferred from various body that one has to answer. Although prior works also attempt
parts (e.g., legs jeans), it is important to reliably de- to recognize the appearance type after detecting the parts,
tect the parts for accurate attribute recognition. This, de- such approaches might suffer from noisy detections since

1550-5499/13 $31.00 2013 IEEE 721

DOI 10.1109/ICCV.2013.95
pose estimation is a still unsolved problem. In addition, it attributes explicitly, using the pose estimation of [21].
is required to collect keypoint annotation on body parts to In the second group of approaches, part detection or pose
train the pose estimators. estimation functions as a pre-processing stage and attribute
This paper explores the other dimension of variation of recognition is performed subsequently. Despite the impor-
human parts: the appearance variation. The major source tance of part localization in the task, there are a few limi-
to appearance variation of human parts is a variety of cloth- tations on this strategy. (i) It is a still challenging problem
ings and these different types of clothes or accessories often in computer vision to estimate pose, thus it may be risky to
yield more signicant changes in the actual images than ar- rely on potentially noisy output. (ii) The appearance varia-
ticulation or viewpoint changes (see the examples of skirt tion of parts (e.g. hat vs non-hat) is not taken into account
in Fig. 1). Therefore, it is important to address such vari- in part learning nor detection. The learned dictionary usu-
ation properly for reliable part detection by learning a rich ally contains generic parts mainly constrained in geometry
appearance part dictionary. A rich appearance dictionary and such parts do not convey attribute-specic information.
means that the dictionary is uent enough to account for (iii) Finally, it is expensive to collect keypoint annotation of
many different appearance part types. To explain appear- body parts, which is required to train pose estimators or part
ance type also means to answer given questions in our ul- detectors.
timate task, attribute recognition. We empirically demon- Weakly-Supervised Part Learning. In this paper, we
strate the importance of such dictionary for the task of at- learn the dictionary of discriminative parts for the task of
tribute recognition on two publicly available datasets [2, 15] attribute recognition directly from training images. Our
where our method, without using numerous keypoint anno- method can be viewed as a weakly-supervised method since
tation, outperforms the prior works. we do not require manual annotation of parts used in fully-
supervised methods. Fig. 1 illustrates the main difference
2. Related Works between our approach and Poselet [3] in part learning. We
learn each part by clustering image patches on their appear-
Human Attribute Recognition from Body Cues. Due
ance (low-level image features) while the poselet approach
to its practical importance, ne-grained human attribute
[3] learns a part from the image patches of similar geo-
recognition has been studied intensively in the literature.
metric congurations of keypoints. Intuitively, our parts
Earlier works used the facial images for classication of
are more diverse in appearance space and the Poselets are
gender [9], age group [13], ethnicity [10], and so on, since
strictly constrained in geometry space. 1
the face is the most informative part for these attributes. Ku-
The recent researches in weakly-supervised part learning
mar et al., [12] recently proposed a framework for face veri-
suggest two important criteria. First, part learning can be
cation aggregating a much broader set of general facial at-
directly guided by overall learning objective of given task
tributes. Since frontal face is visually distinct from the other
(i.e., classication gain) [17, 7]. In particular, Sharma et
human parts or other objects (i.e., background), detection is
al. propose the expanded parts model [16] to automatically
relatively reliable. On the other hand, the other body parts
mine disciriminative parts for human attribute and action
such as arms, legs can be also informative for certain types
recognition, which is similar in spirit to our paper but differs
of attributes. Cao et al. [4] has shown that the evidences
in that we learn one part dictionary shared by all attribute
to determine gender can be collected from the whole body
categories while [16] learns a separate dictionary for each
and a more general set of attributes (gender, hair style, and
attribute. Second, it is important to use exible geomet-
clothing types) has been also considered in recent works
ric partitioning to incorporate a variety of region primitives
[2, 5, 15].
[11, 20, 18, 1] rather than a pre-dened and restrictive de-
In contrast to the face, it is difcult to extract information
composition which may not capture all necessary parts well.
reliably from the whole body due to huge variation of parts
Therefore, we decompose the image lattice into many over-
in geometry and appearance. The prior works on attribute
lapping image subregions at multiscale and discover useful
recognition can be categorized into two sets by their strate-
part candidates after pruning sub-optimal parts with respect
gies to handle pose variation. (i) The rst class of methods
to the attribute recognition performance.
ignore latent pose and use xed templates [4, 15]. This may
not be robust against articulation or viewpoint change which
3. Part Learning
is frequent in real-world examples. (ii) The other methods
model the pose with geometric latent variable and rely on Now we explain the details of our approach. In general,
pre-trained pose estimator or part detectors to infer it [2, 5]. there are two considerations to be made in part learning, a
For example, Bourdev et al. [2] proposed a framework for 1 More precisely, the poselet approach, after the initial learning stage,
human attribute classication using pre-trained part detec-
lters examples whose appearance is not consistent with the learned detec-
tors, Poselets [3]. Chen et al. [5] introduced a model based tor. This makes each Poselet also tight in the appearance space. However,
on conditional random eld to exploit correlation between they are still not diverse.

722
 
















    !"#!$%

 
 
       
Figure 2. Two region decomposition methods based on the image     
grid: (left) spatial pyramid [14] and (right) our multiscale overlap- Figure 3. Window specic part learning. For every window on the
ping windows. The spatial pyramid subdivides the image region grid, we learn a set of part detectors from clustered image patches
into four quadrants recursively, while we use all rectangular sub- in training set. Each learned detector is reapplied to the images
regions on the grid, which is similar to [20, 18]. and rened.

geometric and an appearance basis. We rst need to spec- The advantage of exible geometric partitioning has
ify what kinds of region primitives are allowed to decom- been also advocated in the recent literature of scene model-
pose the whole image region into subregions at the part level ing and object recognition [11, 20, 18, 1]. In particular, our
(Sec. 3.1). Then, we discuss how to learn appearance mod- grid decomposition method is very similar to the initial step
els to explain the local appearance of each part (Sec. 3.2). of [20, 18], which then further attempt to pick up a subset of
good windows and represent each image with a small num-
3.1. Geometric Conguration ber of non-overlapping windows, which reduces to a single
best conguration through explicit parsing. However, we
Given no prior knowledge on human body ontology, our
empirically found that it leads to a better performance to
objective is to dene a geometric basis (i.e., region decom-
allow many number of overlapping windows, therefore we
position) which is expressive enough to capture the parts
only prune inferior part templates in the later stage but do
of arbitrary shapes, but is still of manageable complexity.
not eliminate or suppress any windows. In other words, our
While there exist simpler methods such as spatial pyra-
method allows all congurations, each of which is implic-
mid [14] or uniform partitioning where all subregions are
itly dened and partially contributes to explain each image.
squares, it is difcult to represent many body parts such as
arms and legs in squares, and moreover, we do not know 3.2. Part Appearance Learning
what would be sufcient. Therefore, we examine many pos-
sible subregions from which we can learn many part candi- Once we dene all windows, we visit each window and
dates, some of which will be pruned in later stages. We learn a set of part detectors that are spatially associated with
only use rectangular subregions to limit the complexity, but that particular window. Our motivation is that the human
allow them to be of arbitrary aspect ratios and sizes. body consists of a number of parts, which are usually spa-
tially constrained, i.e. anchored at their canonical positions.
Specically, our method starts by dividing the image lat-
Fig. 3. shows the general procedure and examples. For
tice into a number of overlapping subregions. In this paper,
each window, wi , we rst crop all the corresponding image
we refer to each subregion as a window. We dene a grid
patches from the entire set of training images. Then each
of size W H 2 , and any rectangle on the grid containing
patch is represented by the feature descriptor. We use the
one or more number of cells of the grid forms a window.
Histogram of Oriented Gradient (HoG) [6] and color his-
Fig. 2. illustrates our region decomposition method in com-
togram as the low-level features of image patches. On the
parison with the spatial pyramid matching structure (SPM)
extracted features, we perform K-means clustering and ob-
[14]. Both methods are based on the spatial grid on images.
tain K = 50 clusters, {v1i , ...vK i
}. Each obtained cluster
The SPM recursively divides the region into four quadrants
represents a specic appearance type of a part. Since the ini-
and thus, all subregions are squares that do not overlap with
tial clusters are noisy, we rst train a local part detector for
each other at the same level. In contrast, we allow more
each cluster by logistic regression as a initial detector and
exibility in shape, size, and location of part window. An-
then, iteratively rene it by applying it onto the entire set
other important difference between our approach and SPM
again and updating the best location and scale. We mine the
is that we treat each window as a template by a set of de-
negative patches from the regions outside given bounding
tectors that can be deformed locally, whereas each region in
boxes. At the initial iteration, we discard the noisy part can-
SPM is used for spatial pooling.
didates by cross validation, and limit the maximum number
2 We use W = 6, H = 9 and let the unit cell be of aspect ratio of 2:3 of useful parts to 30 (we will discuss the choice of this quan-
in the experiment. tity in the experimental section). The detection score, g, of

723
an image I for a part vki can be expressed as follows: it is possible that the same image can be included in multi-
ple number of clusters as long as its part detection score is
P (vki = +|I i ) positive (g(vki |I i ) > 0). Then for each attribute, we have
g(vki |I i ) = log , (1)
P (vki = |I i ) two disjoint subsets of the image patches in the cluster, the
positive and the negative. By using the same image fea-
where I i is the image subregion occupied by the window, tures used for detection, we train an attribute classier for
wi . We transform this log posterior ratio by logistic func- an individual attribute, aj , by another logistic regression as
tion, as in [2]: follows:

d(vki |I i ) = P (vki = +|I i ). (2) P (aj = +|vki , I i )

f (aj |vki , I i ) = log . (3)
P (aj = |vki , I i )
These detectors are local and only activated at the spe-
cic window from which they have been trained. We only Again, we discard the learned classier whose accuracy is
allow relatively small range of local deformation (transla- inferior by cross validation so that the number of parts in
tion and scaling) around the window (20% of deviation from whole model is limited to 1,000 to address all attributes.
the window). Such assumption may seem to be weak to ro-
bustly capture many articulated parts or viewpoint change. 4.2. Aggregating Attribute Scores
For example, a lower-arm can be located virtually every- We have obtained all part detection scores as well as
where. However, we do not explicitly distinguish the geo- part-specic attribute classication scores. These are local
metric and appearance variation. That is, if a part is articu- scores obtained from local parts. Now we use another dis-
lated and located far from its canonical window frequently, criminative classier to aggregate these local scores to out-
we treat this as another appearance part type that is dened put the global prediction. Again, we use the same strategy
at another window. If such arrangement is not frequent, we used in the Poselet-based approach, which combines the at-
disregard it. This treatment can be also justied by consid- tribute classication scores with the weights given by part
ering that a part looks differently from the same part in a detection scores.
different pose. Therefore, it may be benecial to maintain Specically, we form a nal feature vector, (I) for each
separated part templates for those cases so that each tem- image I and each attribute a as follows:
plate can explain its own type better.
ik (I) = d(vki |I i ) f (aj |vki , I i ). (4)
4. Attribute Classication
Therefore, each element is simply the product of two terms
Now we explain our method for attribute classication. that we dened in previous sections. Note that i and k are
After learning the parts at multiscale overlapping windows, used to index the window and part type at each window,
we mainly follow the strategy for attribute classication and we form a 1D vector simply by organizing each part
proposed in the Poselet-based approach [2]. The key idea is sequentially. We refer to this vector as the part-attribute
to detect the parts by learned detectors (Poselets in [2]) and feature vector. Then we train the logistic regression again
then to train a series of part-specic local attribute classi- with this feature and use it as the nal classier.
ers. The nal prediction is made by aggregating the scores In contrast to the Poselet-based approach which lever-
from these local classiers with the weights given by part ages an additional layer of classiers to model contextual
detection scores. information between attributes (e.g., long-hair and fe-
Such strategy is effective for the task of ne-grained clas- male), we do not explicitly model the correlation between
sication such as human attribute classication. This is be- attributes in this paper. This is mainly because our dictio-
cause many part elements that are not directly informative nary, which preserves appearance type, can be a good basis
about the class to predict (attributes) can still provide a guid- to implicitly capture such correlation. For example, once we
ance to retrieve the discriminative cues. For example, an detect a face with long-hair, it can immediately inform us
upper-body detector itself does not help predict any types that it is more likely to nd skirt as well even before pro-
of attributes. However, once we locate the upper-body by ceeding to attribute inference stage. The poselet, however,
it, we can run the additional classier to obtain ner-grained lacks appearance type inference in detection stage and thus,
information constrained in the upper-body region. has to explicitly enforce such constraints in a later stage.
4.1. Part Specic Attribute Classiers
5. Experimental Results
Specically, for each part detector, vki , at each window,
i In this section, we present a various experimental results
w , we have a set of image patches, the cluster members
used to train the detector. This cluster is a soft cluster, so that can support the advantage of our approach.

724

 
sometimes ambiguous in crowded scenes (Fig. 4) and such
box is difcult to obtain in fully automated systems which
would typically deploy a person detector prior to attribute
inference; such detector would provide the alignment at the
level of full-body or upper-body.
Therefore, we rst aligned all the training images by us-
ing two keypoints on the head and middle-hip and trained
upper-body detectors. And by applying it onto the images
(while penalizing deviation from the original boxes), we ob-
tained roughly aligned boxes. These aligned boxes are sim-
ply enlarged from the outputs of upper-body detectors and
share the same aspect ratio.

 
For fair comparison, we used the same set of image fea-
tures as [2], HoG and color histogram. While the Poselet-
based approach additionally uses the human skin-tone as
another feature, we do not use any extra image features in
this paper because our goal is weakly-supervised learning
which requires no supervision on skin. The total number of
parts was set to 1000 while [2] used 1200 parts.
Table 1. shows the full comparison where our full model
outperforms the Poselet based approaches in 8 out of 9 at-
tributes. Note that the full model indicates the approach
using multiscale overlapping windows and the rich appear-
ance part dictionary as we have discussed in this paper. In
order to verify the contribution of each factor to the nal
Figure 4. (a) (top) Images from the dataset of Poselet-based ap-
performance, we conducted two additional tests as follows.
proach, copied from [2]. This dataset exhibits a huge variation Rich Appearance Dictionary. We have argued that it is
of pose, viewpoint, and appearance type of people. (a) (middle) important to learn a rich appearance dictionary that can ad-
A few selected, extremely challenging images from the same set. dress the appearance variation of parts effectively. We val-
Either the bounding box (to indicate the person of interest) is am- idate this argument by varying the number of parts learned
biguous in cluttered scene, the resolution is too low, or occlusion at each window, K, ranging from 1 to 30. However, we still
and truncation is too severe. (b) Randomly selected images from perform clustering to form many clusters and then choose
HAT Database [15]. All the bounding boxes have been obtained K best part candidates, judged by cross validation detec-
by person detector [8], therefore the complexity is limited.
tion score.
Table 3. shows the performance improvement according
Datasets. For evaluation of our approach, we use two
to K and this result can support the importance of rich vi-
publicly available datasets of human images labeled with
sual dictionary. In particular, having many parts per window
attributes: the dataset of Poselet [2], and the Database of
is important for subtle attributes, such as glasses. Note
Human Attributes (HAT) [15]. Fig. 4 shows the examples
that, K = 1 does not mean that we only have one part tem-
taken from both sets. Each set has been constructed in a dif-
plate for each true part. Since we have multiscale overlap-
ferent way. We discuss the details in following subsections.
ping windows, and we can still have many other templates
learned at the other windows. This can also explain why the
5.1. Attribute Classication on Dataset of Poselet
gender attribute, whose cues would be more distributed over
The dataset contains 3864 training images and 3600 many subregions as a global attribute, has the least amount
testing images, each of which is labeled with 9 binary of gain from increasing K.
attributes. Fig. 4 shows a few examples from Poselets Multiscale Overlapping Windows. We also tested the
dataset. Each image is manually annotated by a visible effect of multiscale overlapping window structure used in
bounding box of each person. This bounding box is pro- our approach. Table 1. (b) shows the performance when
vided at training as well as testing time, i.e., detection is we only used a set of non-overlapping windows at single
given. Since these boxes that cover visible parts of humans layer, which reduces to a simple grid decomposition, and
do not provide any alignment, it is very challenging to learn the row (c) shows the result when we use the windows at
or detect the parts from them. Also, the evaluation may two more additional layers as spatial pyramid scheme. Nei-
be problematic because the interested person indication is ther method performed as well as the full approach.

725
 male long glasses hat t- long shorts jeans long Mean
hair shirt sleeves pants AP
Base Frequency .593 .300 .220 .166 .235 .490 .179 .338 .747 .363
(a) Full .880 .801 .560 .754 .535 .752 .476 .693 .911 .707
Ours (2) (b) Uniform Grid .857 .734 .429 .631 .405 .687 .349 .560 .862 .613
(c) Spatial Pyramid .857 .725 .407 .641 .429 .707 .356 .565 .886 .620
(d) Full * .824 .725 .556 .601 .512 .742 .455 .547 .903 .652
Poselet (33) (e) No context * .829 .700 .489 .537 .430 .743 .392 .533 .878 .615
(f) No skintone .825 .732 .561 .603 .484 .663 .330 .428 .850 .608
Table 1. The attribute classication performance on the dataset of poselet [2]. The number in parentheses is the number of keypoints used
in learning of each method. * indicates the methods to use an additional image feature (skintone).

Attributes Freq. Ours DSR EPM K 1 3 5 10 20 30

[15] [16] Male .858 .860 .862 .870 .880 .881
Female .488 .914 .820 .859 LongHair .690 .731 .736 .771 .795 .801
FrontalPose .747 .968 .913 .936 Glasses .375 .397 .429 .458 .536 .560
SidePose .239 .772 .610 .673 Hat .634 .625 .678 .676 .744 .754
TurnedBack .105 .898 .674 .772 Tshirt .412 .412 .430 .430 .521 .535
UpperBody .413 .963 .924 .979 LongSlv .715 .719 .726 .738 .745 .752
Standing .768 .977 .960 .980 ShortPnts .373 .386 .407 .470 .474 .476
Run/Walk .153 .635 .676 .746 Jeans .612 .595 .644 .690 .695 .693
Crouching .026 .123 .207 .240 LongPnts .891 .906 .905 .910 .912 .911
Sitting .117 .593 .546 .627 mAP .618 .630 .654 .679 .693 .707
ArmsBent .765 .954 .919 .940 Table 3. The performance improvement according to the maxi-
Elderly .070 .321 .293 .389 mum number of appearance part types at each window (K).
Mid-Aged .507 .700 .663 .689
Young .381 .656 .594 .642
TeenAged .159 .335 .291 .362 Poselets dataset. i) The considered attributes includes a
SmallKid .143 .535 .437 .497 set of action or pose categories such as running or side
SmallBaby .027 .163 .122 .243 pose. Table 2. shows the full list. ii) The dataset was con-
TankTop .081 .370 .332 .377 structed in a semi-supervised way by a person detector of
Tee shirt .352 .671 .591 .616 [8], instead of manual collection. Therefore, the complexity
CasualJacket .088 .426 .353 .400 in terms of articulation or occlusion is relatively lower than
MensSuit .068 .648 .482 .571 that of Poselets dataset. However, such criterion is also
FemaleLongSkirt .060 .420 .399 .448 meaningful, considering the fully automated real-world sys-
FemaleShortSkirt .058 .301 .337 .390 tem would follow the same procedure - running the person
Shorts .070 .496 .427 .468 detector and then performing attribute classication. Since
LowcutTop .166 .660 .556 .613 the bounding boxes were obtained by the person detector,
FemaleSwimsuit .023 .467 .282 .322 the people in the images are roughly aligned. Therefore, we
WeddingDress .014 .621 .621 .642 did not use any keypoints for the rough alignment in this
Bermuda .129 .420 .393 .437 dataset.
Mean AP .230 .593 .538 .587 Table 2 shows the performance comparison among our
Table 2. The attribute classication performance (average preci- approach, the discriminative spatial representation (DSR)
sion) on the dataset of HAT [15]. In addition, EPM [16] achieves [15], and the expanded part models (EPM) [16]. The DSR, a
.597 in the setting to leverage image context from outside of variant of spatial pyramid representation, seeks for an ideal
bounding box region. partitioning for classication, instead of strictly constrained
structure (squares) of original spatial pyramid. However,
their partitioning, even if it is an optimal, is still xed.
5.2. Attribute Classication on HAT Database And geometric variation (local deformation) has not been
addressed, which is important for human models. On the
The HAT database contains 7000 training images and other hand, the EPM which also attempts to learn the dis-
2344 testing images, labeled with 27 binary attributes. criminative parts has shown a comparable result to ours in
There are two main difference between this dataset and an equivalent setting where recognition is performed solely

726



   

  ve attribute categories. We denote the most contributed,
most discriminative part window for each image by blue
boxes. Although there are some meaningless activation (for
example, NOT JEANS was inferred from the head), the
most parts show reasonable localization.
Fig. 5 shows the most discriminative parts for ve se-



   
    lected attributes. We measure this by correlation between
 



 
attribute labels and the part-attribute feature. As one can
easily see, the most discriminative Poselets are unbiased de-
tectors which would respond to both female and male. In
contrast, our discriminative parts has distinct polarities.

6. Conclusion
We presented an approach to the problem of human at-
tribute recognition from human body parts. We argue that it
is critical to learn a rich appearance visual dictionary to han-
dle appearance variation of parts as well as to use a exible
and expressive geometric basis. While the major focus has
been made on appearance learning in this paper, we plan to
expand the current model into structured models where we
can learn more meaningful geometric representation, as for
the future work.

7. Acknowledgement
This work was gratefully supported by NSF CNS
1028381, DARPA MSEE project FA 8650-11-1-7149, ONR
MURI N00014-10-1-0933, and NSF IIS 1018751. The rst
author was also partially supported by Kwanjeong Educa-
tional Foundation.
Figure 5. The most discriminative parts in Poselet-based approach
[2] and our learned model. Our rich dictionary distinguishes many References
different appearance part types, which are directly informative [1] R. Benenson, M. Mathias, T. Tuytelaars, and L. Van Gool.
for attribute classication, while the selected poselets are generic Seeking the strongest rigid detector. In CVPR, 2013.
parts. [2] L. Bourdev, S. Maji, and J. Malik. Describing people:
Poselet-based attribute classication. In ICCV, 2011.
[3] L. Bourdev and J. Malik. Poselets: Body part detectors
from person bounding box region. However, the advantage
trained using 3d human pose annotations. In ICCV, 2009.
of our method is to learn a common dictionary shared by all
[4] L. Cao, M. Dikmen, Y. Fu, and T. S. Huang. Gender recog-
attribute categories whereas the EPM uses a separate dictio- nition from body. In ACM MM, 2008.
nary for each category. [5] H. Chen, A. Gallagher, and B. Girod. Describing clothing by
5.3. Discriminative Parts for Localization semantic attributes. In ECCV, 2012.
[6] N. Dalal and B. Triggs. Histograms of oriented gradients for
Finally, we discuss the issue of the discriminative parts human detection. In CVPR, 2005.
by providing qualitative results. The quantitative evalua- [7] I. Endres, K. J. Shih, J. Jiaa, and D. Hoiem. Learning collec-
tion is difcult because neither dataset provides required tions of part models for object recognition. In CVPR, 2013.
ground-truth annotation. The most discriminative part for [8] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ra-
an attribute is the part whose contribution to the attribute manan. Object detection with discriminatively trained part-
prediction is the biggest. We measure this from product of based models. TPAMI, 32:16271645, 2010.
the weights of the nal logistic regression classier and the [9] B. A. Golomb, D. T. Lawrence, and T. J. Sejnowski. Sexnet:
part-attribute feature of each image, (I). Fig. 6. shows A neural network identies sex from human faces. In NIPS,
the examples in the testing set (from the Poselets dataset), 1990.
which output the most positive and negative responses for [10] S. Gutta, H. Wechsler, and P. J. Phillips. Gender and ethnic
classication of face images. In FG, 1998.

727
Figure 6. The qualitative results of attribute classication for male, glasses, hat, t-shirt, and jeans categories. (left) The most positive, and
(right) the most negative images for each attribute. The red boxes denote the bounding boxes and each blue box represents a part detection
whose contribution to prediction is the biggest.

[11] Y. Jia, C. Huang, and T. Darrell. Beyond spatial pyramids: [17] S. Singh, A. Gupta, and A. A. Efros. Unsupervised discovery
Receptive eld learning for pooled image features. In CVPR, of mid-level discriminative patches. In ECCV, 2012.
2012. [18] X. Song, T. Wu, Y. Jia, and S.-C. Zhu. Discriminatively
[12] N. Kumar, A. Berg, P. N. Belhumeur, and S. Nayar. Describ- trained and-or tree models for object detection. In CVPR,
able visual attributes for face verication and image search. 2013.
TPAMI, 33(10):19621977, 2011. [19] D. A. Vaquero, R. S. Feris, D. Tran, L. M. G. Brown,
[13] Y. H. Kwon and N. da Vitoria Lobo. Age classication from A. Hampapur, and M. Turk. Attribute-based people search
facial images. CVIU, 1999. in surveillance environments. In WACV, 2009.
[14] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of [20] S. Wang, J. Joo, Y. Wang, and S.-C. Zhu. Weakly super-
features: Spatial pyramid matching for recognizing natural vised learning for attribute localization in outdoor scenes. In
scene categories. In CVPR, 2006. CVPR, 2013.
[15] G. Sharma and F. Jurie. Learning discriminative spatial rep- [21] Y. Yang and D. Ramanan. Articulated pose estimation with
resentation for image classication. In BMVC, 2011. exible mixtures-of-parts. In CVPR, 2011.
[22] B. Yao, X. Yang, L. Lin, M. W. Lee, and S. C. Zhu. I2T:
[16] G. Sharma, F. Jurie, and C. Schmid. Expanded parts model
Image parsing to text description. Proceedings of the IEEE,
for human attribute and action recognition in still images. In
pages 14851508, 2010.
CVPR, 2013.

728