Engineering an image‐browsing environment: re‐purposing existing denotative descriptors

Photographs

Photographs present a different form of representation from words. On the whole, they are made with the direct participation of some object or objects and are, thus, very specific. In this sense, they are exquisitely empirical. That is, they present to us the surface appearance of an object or set of objects at a particular moment in time. Instead of the representation “dog”, we are presented with a two‐dimensional projection of a particular terrier on the grass in the late afternoon; instead of “runner”, we are presented with a curly haired and bearded runner in blue shorts with his left foot crossing the finish line of the Boston Marathon in April of 1982; instead of “assassination” we are presented with the 343 frames (each a still photograph) of the Zapruder film showing John Kennedy's limousine in Dallas.

Since photographs tend to be much more specific than words and since words are not native elements of photographs, there is at least some difficulty in achieving verbal representations of photographs that will be useful for several sorts of searches by several sorts of users. We posit that a photograph may be sought because it presents a noun – that is, an object of the sort we wish to present was in front of the lens; it may be sought because it presents an instance of an event we wish to present; or it may sought because it contains an instance of an image primitive we wish to present – a particular color or texture or composition characteristic.

Representation of photographs as nouns is not especially difficult, though it is not so trivial as it might seem. Greisdorf and O'Connor (2002a) note that even among a fairly homogenous group, different people representing the same photographs chose different subsets of the image data, used different terms for the same object, and chose to represent at different levels of generality. A small thought exercise here may help to illustrate the different requirements and challenges in representation of photographs for use. Suppose we have a collection of some thousands of photographs with no verbal tags as representation, only small versions (thumbnails) of each of the pictures. Suppose, we present retrieval tasks to each of three groups of participants. We ask the first group to find seven photographs to illustrate a story on dogs in America. We ask the second group to find seven photographs to illustrate a story on the American West. We ask the third group to find seven photographs to illustrate a story on love in twenty‐first century America. We might hypothesize that the first group would finish most quickly and that there would be little variance in the seven photographs of dogs. This would be an example of a noun search. We might expect that the second group would take a bit longer and that there would be a greater variance in the seven‐image sets, because there is no single noun photograph of the American West. Cacti, pickup trucks, rodeo riders with iPods, housing development on ranches, four‐wheelers instead of horses, and numerous other nouns might suit the needs of illustrating a story on the American West. Then, it would not be hard to imagine that the third group would require more time to sort through images and that there would be even more variance in the picture groupings.

What now if an advertising designer wants a “bright red” campaign? What if an interior designer wishes to “tie together” several rooms subtly? What if adjectival concepts are the requirement of the photographs – delicate, robust, soft? In these instances, we might be looking for image primitives such as color, texture, edges, and lighting.

In some searches, it might be possible to make a list of nouns that might fulfill the requirements; however, there might be other nouns of which the searcher does not think. If it were possible to link other nouns that had satisfied searchers looking for photographs to illustrate the same abstract or adjectival concepts, we might give the browser a better chance of finding a satisfactory image or set of images.

Browsing

Browsing is not a matter of dumb luck and it is not simply the result of the invisible guiding hand of the librarian. Browsing is a set of activities used to locate and sample documents when other systems are inadequate to the search task. They represent a shift of the locus of representation from an external agency to an individual's internal agency. Of course, this introduces a substantial issue: browsing “success” is specific to a particular person and a particular set of circumstances. For, at least, some of the activities, we label as “browsing” there is no simple way of predicting what success would require. This means we do not have access to an easy metric of success.

Inherent visual features of an image, low‐level image primitives are often not sufficient to represent semantics embodied within an image message (Hare et al., 2007). In particular, connotative meanings are more difficult to represent with descriptive techniques based on image primitives (Eakins and Graham, 1999; Rui et al., 1999; Jörgensen, 2003). Even in cases in which image primitives are easily clustered to represent particular objects, it is not necessarily the case that the message will have meaning for any particular viewer. It is also obvious that several keywords assigned by an indexer may not represent an image adequately. Figure 1 shows that a large number of messages contained within an image, especially connotative messages, can result in discrepancies between indexers and users.

If the very specificity of images and their fundamental representational difference from words mean that browsing may at times be a useful search mechanism; and if we assume that browsing is a set of activities that makes use of multiple modalities for insertion into a collection and evaluation of discovered documents; then we can propose that a clustering mechanism derived from existing representations might reduce the search time and search space and increase the likelihood of satisfactory results. We must be clear here that we are not proposing that images are some sort of free‐floating triggers and we agree with one anonymous reviewer that: “images are created by humans operating within the logonomic parameters of their socio‐cultural moment.” It is precisely because of this that we are pursuing this line of research. A single cataloger sits within a set of logonomic parameters. Training, system design, and thoughtfulness of the cataloger and users of the catalog may not be sufficient to bridge differences in logonomic parameters of users; especially in the case where the objects being described – photographs – are not constructed of words.

If we look to a social networking image environment such as Flickr.com, we see a phenomenon that lies between a formal bibliographic apparatus (or even a quasi‐formal folksonomy) and utter chaos. That is, there are millions of images, some with multiple descriptors, some with no descriptors at all (except for the Exchangeable image file data automatically applied to many images made with recent digital cameras). There are no particular rules for how one should make up descriptors, especially since there are no rules governing the use of the environment. Some may simply use it as a handy album from which to select images for various projects or to show occasional friends and relatives some interesting images; others may hope to attract lots of attention to their images and so put numerous descriptors and add them to various “groups” – which act as a form of classification scheme or, at least, a set of local, and specific folksonomies. Millions of images on Flickr.com are tagged with “me.” For each photographer who uses this tag, it is especially useful, even though for anybody else searching the entire flickr domain, the millions of “me's” will be useless because they will not be of that “me.”

However, there are certain trends. If one searches the entire Flickr corpus for “dog,” one finds a bit over four million hits. Skimming through the first few hundreds, one finds family pets, fashion photos with dogs, hunting dogs, lyrical images, hotdogs (the food,) guard dogs, sled dogs, close ups of parts of dogs, and groups of dogs. If one searches Flickr for images tagged “love” about one million this come up. Here, the variety is greater than the “dog” search. Many of the images are of the photographer's beloved, hoped for beloved, and wishful thinking beloved. Others are a heart drawn in the sand, clasped hands, a champagne bottle, a peace symbol, wedding rings, a dog, a cruise ship, flowers, a baby, and even a shopping cart in a restroom. Someone searching for images of “love” might not have thought to look for images of a baby or a dog; now they might see this as useful.

We set about taking an existing set of images and descriptors to see if we could, indeed, use the ordinarily denotative descriptors of a standard cultural heritage collection of images and derive some measure of connotation from them. We must note that we are not saying there is any direct mapping of denotative descriptors to immediately and generally useful connotative descriptors. Instead, we are seeking to give browsers another navigation method that may lead to more successful searching with the expenditure of fewer resources. Browsing is still a matter of individual engagement with and evaluation of documents. It thrives on such individual engagement and evaluation. However, in a large collection, it may be possible and quite useful to make some rough predictions about areas that are more or less likely to yield useful results.

We are aware of what may seem to be a problem here. As one anonymous reviewer commented:

The danger with the author's approach is that semantic integrity will be forsaken in the interests of accommodating a wide variety of highly subjective terms which a set of viewers has seen fit to allocate [sic] to the depicted salient object or scene.

Our response to this comment is: “Exactly!” Browsing is a set of activities which acts as end runs around limitations of standard bibliographic procedures. In many browsing situations, random insertion into a collection might prove to be the most useful means of entry. However, many people do not have unlimited time for random insertion after random insertion. Also, it may be the case that someone else has blazed a useful path. So, by doing a small amount of clustering, we might be able to offer at least some of the benefits of browsing with a little more organization. People who purposely put themselves into browsing situations are generally aware that there is a tradeoff – a higher trash to gems ratio, but the possibility of finding the gem where little or no probability existed within the standard retrieval system. That is, by stepping outside the standard bibliographic apparatus, we increase the search space to include areas of the collection beyond those that have proved unfruitful (either in a given search or over a set of searches). While this may provide more potential, it may prove to be too large a search space; so, various heuristics for making the search smaller may be of value.

As shown in Figure 1, and as discussed by other researchers (Besser, 1990; Fidel, 1997; Greisdorf and O'Connor, 2002a, b; Jörgensen, 1998; Layne, 1994), connotative meaning[1] is an important but problematic attribute during the image representation process. Our study starts from the premise that connotative messages are important for improving user satisfaction of image‐retrieval results. Then, by applying semiotic notions of denotation and connotation in an instrumental manner, we explore the characteristics of messages within an image and the structures of interrelations among other types of messages. Based on the understanding of the nature of connotative messages within an image, we propose and evaluate a model that helps users find what they want, even when unfavorable information loss makes it difficult to obtain satisfactory search results.

We must clarify two points. We are using the notions of denotation and connotation in an instrumental fashion. These are more complex and more subtle than we have allowed here; we basically stripped away much of the subtlety to see if our model works at a general level. Also, an anonymous reviewer commented that our use of the terms “cataloger” and “real people” in Figure 1 showed a “pretty grim distinction”. On the face of it, we would agree; however, in the original design and publication of the research, the distinction was made as a playful statement of different needs and tools – the catalogers had smiled over it. Construction of descriptors in the formal bibliographic apparatus tends to focus on the nominal and are often in some order outside the norms of the local natural language.

Representations of image documents and user needs

Essential and unique features of image documents have been presented through research on image attributes. Panofsky (1962) suggested three levels of meanings of images: pre‐iconography, iconography, and iconology. Layne (1994) and Markey (1988) analyzed image attributes based on Panofsky's attribute levels. In addition, Hidderley et al. (1995), Krause (1988), and Jaimes and Chang (2000) examined the attributes of images. Although various categorizations have been made, it is common for researchers to distinguish between objective attributes (factual) and abstract attributes (symbolic, emotional, and subjective) in an image.

As research on image documents has focused on attributes analysis, most user studies of image retrieval are about the analysis of query types. One of the widely cited query classifications was provided by Eakins and Graham (1999). They divided queries into three levels: Level 1 primitive features, Level 2 logical features, and Level 3 abstract attributes. Enser and McGregor (1992), Armitage and Enser (1997), Hastings (1995), Chen (2000), Choi and Rasmussen (2003), and Markkula and Sormunen (1998) also analyzed user queries and showed that objective attributes are used more often than abstract attributes in queries. However, compared to the query‐analysis studies, other studies (O'Connor et al., 1999; Greisdorf and O'Connor, 2002a), examined users' verbal reactions to an image or sorting patterns, experimentally demonstrated the importance of affective and emotional attributes in image perception. Then, what caused this difference? According to a series of studies performed by Jörgensen (1995, 1998), when participants are given a template including both objective and abstract attributes, they tend to choose abstract terms to describe image more frequently than without the template. This result probably implies that users do not use abstract attributes in query formulation because they are not familiar with representing that kind of attribute.

Here, it is useful to turn to the binary document model suggested by Shannon and Weaver and mirrored in semiotics. Shannon and Weaver (1949) asserted the strict distinction between the message and the meaning. The message is the set of organized primitives the physical stimuli – in whatever medium; and the meaning is the result of an individual's engagement with the message – the individual interpretation. In this binary model by “interpretation”, we mean “function” or “suitability for a particular purpose.” That is, for some purposes, color or general composition rather than object photographed might be “meaning.”

Connotative descriptors for image‐retrieval systems

Although there has been research demonstrating the importance of connotative messages during the image‐search process, it has been recognized that connotative messages, depending on an individual indexer's subjectivity, are “out of scope” for text‐based indexing. Some commercial image‐retrieval systems attempt to describe emotional and affective attributes of an image, but they also show inconsistency and subjectivity issues (Jörgensen, 2003). In addition to text‐based approach, there are few content‐based systems which facilitate connotative access by involving emotional attributes. Kansei research is one of those few efforts. The Kansei‐based approach tries to index users' impressions and subjective feelings which are evoked while viewing images (Black et al., 2004). Kansei research has focused on finding a connection between low‐level features (i.e. color, shape, texture, and so on) and subjective impressions (Kato and Kurita, 1990; Tanaka et al., 1997; Bianchi‐Berthouze et al., 1999; Bianchi‐Berthouze, 2001). However, as Kansei research has progressed, some researchers addressed the insufficiency of low‐level dependent approaches and proposed to combine low‐level features and denotative/objective features for extracting emotional/abstract features (Kuroda and Hagiwara, 2002). In addition to Kansei research, Colombo et al. (1999) also proposed a model which derives emotional features by mapping objective and low‐level features into four representative emotional concepts. Recently, some researchers (Hare et al., 2006; Enser et al., 2007) introduced an idea demonstrating a possibility of ontological support for bridging the semantic gap between denotations and connotations of an image. Ontology‐based representation of images suffers from the very short‐coming browsers are often attempting to escape – a priori tagging. We might even suggest that browsers are attempting to escape the established ontological constructs. We might say that they are seeking an ontological commons where their needs and documents relevant to their needs both reside (Anderson, 2006).

Although these previous efforts for improving image retrieval through connotative attributes reported positive results, they seem to be partial solutions constrained by the experimental environment, such as a domain of collection, a limited number of connotation, and so on. For proposing a more generalized resolution for utilizing connotative messages during the image‐search process, understandings on the fundamental natures of connotative messages within an image should provide guidelines for designing and implementing image‐retrieval systems. This study adopted a theory of semiotics as a basis for understanding connotative messages of an image.

Image‐retrieval system effectiveness

Indexing is good if and only if it results in good retrieval. However, it is difficult to objectively measure the performance of image retrieval, because due to various subjective meanings of an image, a relevant set of images can be different depending on users. In spite of these difficulties, the most widely used measurements for estimating image‐retrieval system effectiveness are precision and recall. However, since humans can browse more images with less effort in a shorter time compared to text documents, some researchers insist that in the case of image retrieval emphasis on recall is reasonable. Layne (1994) explains that since recall is more important than precision, concentrating on browsing with basic index terms would be better than making efforts to assign detailed index terms. Cox et al. (1996) stressed the importance of recall by asserting that a critical problem of image retrieval is non‐retrieved relevant images rather than retrieved non‐relevant images.

Furthermore, Cox et al. (2000) confirmed the effectiveness of displaying diverse images by comparing two display modes, “most‐probable display update scheme” and “most‐informative display update scheme.” The most‐probable display scheme, which is adopted by most if not all image‐retrieval systems, displays the best possible images after each query and relevance feedback. Whereas, the most‐informative display scheme exhibits images in order to gain the highest information from users, by displaying a large varied set of images at the beginning of search stage. They demonstrated that by deriving as much information from the users as possible during the searching process, it is ultimately possible to finish the search iteration more quickly. Zhou and Huang (2003) suggested that these two schemes can be mixed in one screen if the balance between the two can be determined optimally.

Fidel (1997) introduced the concept of the recall of browsing sets. That is, indexing should be done to create browsing sets rather than differentiating relevant images from non‐relevant images. She explained that for a user looking for a picture of Paris for a travel brochure, a large diversity of pictures of Paris would be more informative rather than 50 images of a bird's‐eye view of Paris or of the Eiffel Tower. Since, even with the same query, the relevant images can be entirely different, recall of browsing sets could be a more effective measurement than general recall. As can be noticed from this line of studies, the researchers' common view is that in an image‐retrieval system recall is more important than precision and effective browsing would be more critical than searching only the small number of relevant images.

Synthesis for resolution

Consider that a denotative query is given. Since a denotative query may be generated from various connotative needs depending on the user's socio‐cultural backgrounds, the relationship between the query and the needs can be shown as in Figure 2; |D| is a relevant set for a given denotative query, and |C1|, |C2|, |C3| or |C4| are potential connotative needs which are related to that denotative query. The (a) images may match both denotative query and connotative need; and the (b) images do not match the denotative query, but they have a possibility of satisfying user connotative needs.

Figure 3 shows an example of Figure 2. The images with “olive branch” are the relevant set for the denotative query “olive branch” and the other images without “olive branch” are connotatively related images. Here is a conceptual scenario exemplifying how this model can be applied to the image‐search process. If a user submits a query “olive branch,” the image‐retrieval system displays images of olive branch (|D|) categorized by connotative meanings, scene of old town (a1), Catholic Christian symbol (a2), and symbol of peace (a3). If the user is specifically interested in a group of images symbolizing peace (a3), he/she can expand the group including images symbolizing peace without olive branch (b3). Then, how can this conceptual searching scenario be implemented?

We propose a model for improving the recall of browsing sets by displaying not only sets of denotatively related images but also sets of connotatively related images. As shown in Figures 2 and 3, when the images that match a denotative query are returned, the search results would be grouped by connotative messages which can be derived from denotative messages ((a)s); then, based on user selection, the search results would be expanded to the images which do not match the denotative query but do relate connotatively ((b)s). This study refers to these two functions as grouping and expanding functions, respectively, and proposes that these functions can be implemented by developing an association thesaurus. The association thesaurus is different from a traditional thesaurus which narrows or broadens the search results through vocabulary control, such as narrower, broader and related terms. The purpose of this association thesaurus is to identify, at an utterly instrumental level for now, associations between connotations and denotations in accordance with Barthes' assertion that connotation can be inferred from denotative messages. A detailed association thesaurus construction procedure will be discussed in the following section.

Data source

The “Artefacts Canada: Humanities” (http://daryl.chin.gc.ca/Artefacts/e_MasterLayout.cgi) and the “Art & Architecture Thesaurus” (AAT, www.getty.edu/research/conducting_research/vocabularies/aat/) were selected as our data sources. Artefacts Canada provided by the Canadian Heritage Information Network (CHIN) is composed of approximately 2.5 million records gathered from contributing museums and galleries (from now on, CHIN refers to “Artefacts Canada: Humanities” by CHIN). Their main subject areas are archaeology, decorative arts, fine arts, ethnology, and history. CHIN has integrated the AAT into its search interface, so that users can search or browse the AAT terms and the system presents the results based on matching between selected AAT terms and bibliographic information of records.

The AAT is maintained by the Getty Vocabulary Program covering art, architecture, and material culture areas. It contains approximately 125,000 terms with facet structure. From its seven facets, we considered the “associated concepts” facet as connotation, because it includes terms for abstract concepts, theories, ideologies, and so on. For denotations, “agents”, “activities”, and “objects” facets are selected, because each of them contains terms for role, occupation, and persons, terms for actions or tasks, and terms for entities, respectively. On the other hand, the terms for quality or design elements (physical attributes facet), the terms for produced substances (materials facet), or the terms for artistic classification (styles and periods) are not regarded as either connotations or denotations.

For this study, 5,199 records were collected from the entire Artefacts Canada collection, according to two criteria:

1. 1.

   Record types should be images or photographs.

2. 2.

   Since this study examines the relation between connotation and denotation, records should include terms from the associated concepts facet as well as agents, activities, or objects facets in subject‐related fields.

The dataset was processed with two procedures. First, French terms are translated into English terms; since entries of the CHIN database are contributed by various Canadian organizations, some records are written in French. Using a Free Text Translator (www.freetranslation.com), French terms are translated into English. Second, the Porter's (1980) suffix‐stripping procedure is applied to the dataset for stemming purposes. Porter's stemming procedure is widely used in information‐retrieval applications to convert term variations into their common linguistic roots. The Natural Language Toolkit (http://nltk.sourceforge.net/r) for Python is used for the stemming procedure. The processed 5,199 records are divided into two sets, which are used as a training set and a testing set, respectively, with a ratio of 3:1. The training set is composed of 3,900 records and the testing set is 1,299 records.

Association thesaurus construction

For identifying the relation between a denotation and a connotation, this study developed an association thesaurus based on term association principle. Term association, which mainly depends on the degree of term co‐occurrences in a text or text collection, has been used in the text‐retrieval area as an automatic term classification technique. The term association assumes that:

[…] if two or more terms co‐occur in many documents of a given collection, […] these terms are related in some sense and hence can be included in common term classes (Salton and McGill, 1983, p. 228).

Since this assumption has been verified in prior experimental research, it is plausible to apply the same assumption to this study, that is, “if a denotation and a connotation co‐occur in many images, these two terms are related in some sense, so that they can be included in a common class.” A degree of term association between denotation and connotation is computed using five normalized term association measures, Cosine, Dice, Jaccard, Pearson, and Yule's Y, which are widely used in the information‐retrieval field (Chung and Lee, 2001; Kim and Choi, 1999; Rasmussen, 1992). It is not the purpose of this study to compare the effectiveness of these five association measures; however, since there is little consensus as to what types of measures are most generally applicable (Chung and Lee, 2001), five measures are used and compared to decide which measure would be appropriate for this specific dataset.

From the training set, 148 connotations and 446 denotations are extracted in accordance with AAT fields, and the degree of association between those connotations and denotations were measured with five association measures. Among 66,008 connotative‐denotative term pairs generated from 148 connotations and 446 denotations, 3,323 term pairs (5.034 percent) showed positive term associations. When comparing the five measures, the Cosine and Pearson measures and the Dice and Jaccard measures demonstrated high‐agreement ratios, respectively, so three different thesauri were developed using three measures Cosine, Dice, and Yule's Y. Each thesaurus contained approximately 660 (1 percent) of the top‐ranked connotative‐denotative term pairs with their association measures.

Procedure of indexing the testing set with association thesauri

Then, the testing set was indexed with the association thesauri constructed from the training set. In other words, by referring the relation between connotation and denotation term pairs indicated in three association thesauri, images of the testing set are assigned “Associated connotations.” In order to assign Associated connotations to an image in the testing set, the record should contain denotative terms which are included in an association thesaurus. However, it was found that of the 1,299 records of the testing set, only 680 records (52.35 percent of the testing set) included at least one denotation found in the 446 denotations which were extracted in the training set. This implies that only half of the records in the testing set have denotations used in constructing the association thesauri. This low ratio might be attributed to the limited number of the training dataset. A more comprehensive term list covering various terms is suggested for future research. In this study, the 680 records were used for the effectiveness evaluation.

The indexing algorithm was designed to assign every possible Associated connotation with its weight (association value). The algorithm for indexing the testing set with the association thesauri was adjusted from the research of Plaunt and Norgard (1998). First of all, the 680 records from the testing set were examined to identify the denotations included in the association thesaurus entries. When a denotation was found by the association thesaurus, the record was indexed with the Associated connotations and association values. For example, a record in the testing set has two denotations, “Automobile” and “Museum,” which are entries in the association thesaurus. According to the association thesaurus developed by the Cosine measure, those denotations have the following associations with the connotations: “Automobile – Antique – 0.404,” “Museum – Antique – 0.229,” and “Museum – Golden Age – 0.204.” Based on this association information, the record was indexed with two ranked connotations:

1. 1.

   Antique (0.633 – this value is obtained by combining 0.404 and 0.229).

2. 2.

   Golden Age (0.204). The thesauri using Dice and Yule's Y‐measures were also applied to the dataset in the same way.

Measurement: relative recall

Relative recall is a useful measurement in comparing different retrieval approaches (Tague‐Sutcliffe, 1992). In this study, relative recall for an association thesaurus is computed for investigating to what extent the association thesauri can enhance the accessibility of connotative messages even when the collection is not indexed with connotations. That is, the relative recall for an association thesaurus is compared with the relative recall for AAT connotations, which are originally assigned to the CHIN collection (from now on, Associated connotations refer to the connotations which are assigned through an association thesaurus and AAT connotations refer to the connotations which are originally assigned to the CHIN collection).

Relative recall measures the ratio between the number of relevant retrieved documents for a treatment and the total number of unique relevant documents retrieved for all treatments (Greenberg, 2001). In this study, the relevant documents are determined based on two treatments: AAT connotations and Associated connotations. In the case of the AAT connotations, all of them are regarded as relevant connotations for the assigned image documents, because each AAT connotation is given by museums or related organizations which contribute to the CHIN database. However, in the case of the Associated connotations, the relevance of connotations to the assigned image documents is assessed by two judges who have some expert knowledge in image representation. The judges are asked to evaluate the relevance of Associated connotations by reviewing the image itself and its bibliographic record. Relevance evaluations involve a three‐tier scale of “Relevant,” “Partially relevant” and “Not‐relevant.” “Partially relevant” is regarded as “Relevant” during data analysis similar to prior documented research (Greenberg, 2001; Saracevic and Kantor, 1988). This study considers the possibility that even judges can interpret connotative messages of an image differently based on their own coding systems. Since a connotative meaning can be relevant for one viewer but not for the other depending on a specific socio‐cultural context, the Associated connotations, which are judged as “relevant” or “partially relevant” by one judge, but “not‐relevant” by another judge, are regarded as “relevant” during data analysis.

The relative recall is calculated in two aspects: relative recall of retrieved image documents and relative recall of browsing sets, which are designed to support H1 and H2, respectively. First, relative recall of retrieved image documents is computed by examining whether an association thesaurus can retrieve more image documents which are connotatively relevant. Relative recall of retrieved image documents relates to expanding function; that is, high‐relative recall of retrieved image documents will increase the number of images in (b)s from Figure 2. In other words, once a user selects a group of images (one of (a)s) which are connotatively related, expanding function pertains to the number of images which are connotatively related but having different denotations. This evaluation is performed by comparing the relative recall for Associated connotations with relative recall for AAT connotations against evaluative connotative terms. Here, is an example which considers “anger” as an evaluative connotative term. With the AAT connotations, two image documents, I₁ and I₂, are found, and with the Associated connotations, four image documents, I₂, I₃, I₄, and I₅ are found. I₁ and I₂ are relevant image documents, because “anger” is assigned to those documents by indexers. The relevance of I₃, I₄, and I₅ are judged by two judges, and the judges assess I₃ and I₄ as relevant and I₅ as not‐relevant. Then, the total number of unique relevant documents is four: I₁, I₂, I₃, and I₄. The relative recall for association thesaurus is 0.75 (three‐fourth) and the relative recall for AAT connotations is 0.5 (two‐fourth). That is, the relative recall for the association thesaurus is calculated as follows: Equation 1 Second, relative recall of browsing sets is measured. In this study, the browsing sets are defined as connotatively related groups of image documents which are retrieved against a denotative query. This evaluation process explores to what extent the association thesaurus can offer connotatively related groups of images; in other words, it is related to grouping function. The browsing sets which are retrieved through an association thesaurus are compared with the browsing sets which are obtained through AAT connotations. For example, there is a denotative query “mountain,” and it returns three image documents (The three documents correspond to |D| in Figure 2). If two AAT connotations, Chaos and Paradise, are found from these three images, the number of browsing sets expanded through AAT connotations is two (|C1| and |C2| in Figure 2); users may want to expand their search, either chaos or paradise. On the other hand, if four Associated connotations, inspiration, paradise, romanticism, and totemism, are found using an association thesaurus, those four connotations are defined as browsing sets (|C1|, |C2|, |C3| and |C4| in Figure 2). Then, in order to judge the relevance of the browsing sets, the judgment result of the three images against four Associated connotations are examined. The Associated connotations which are judged as “not‐relevant” in all three images are regarded as a not‐relevant connotation group and the Associated connotations which are judged as “relevant” or “partially relevant” in at least one image are regarded as a relevant connotation group. In this example, among four Associated connotations, Totemism is judged as not‐relevant in all three images. Therefore, the number of relevant browsing sets is three, and the total number of the unique relevant browsing sets is four: chaos, paradise, inspiration, and romanticism. The relative recall for the association thesaurus is 0.75 (three‐fourth) and the relative recall for AAT connotation is 0.5 (two‐fourth). The relative recall of browsing sets for the association thesaurus is calculated as follows: Equation 2

Mapping results between AAT connotations and Associated connotations

The degree of mapping between Associated connotations and AAT connotations was examined. In this evaluation process, AAT connotations were used as a standard relevant connotative term. Therefore, these results will be somewhat conservative, because although AAT connotations are assigned by professional human indexers, AAT connotations should not be regarded as a comprehensive source of relevant index terms. In fact, there might exist Associated connotations which do not match AAT connotations, but are considered relevant connotations by some users. However, in spite of its conservativeness, comparing index terms, which are generated by a new approach with the index terms generated by human indexers, is a method that has been widely used in the evaluation of new information‐retrieval approaches. In a sense, this study considers that the degree of mapping between AAT connotations and Associated connotations can present a feature of the effectiveness of the semiotic‐based approach. By measuring the degree of mapping between two types of connotations, the best‐performing association measures, which produce high recall and precision, are also identified.

Table I shows the overall description of AAT connotations and Associated connotations assigned to 680 records through the association thesauri. The first column indicates the source of connotations. The second column, “No. of indexed record,” shows the number of records containing indexed connotative terms through “source of connotation,” and the third column, “No. of connotations in the records,” gives the total number of AAT connotations or Associated connotations which were assigned to the records. “Avg. no. of connotations per record” presents the average number of connotations per record and “No. of unique connotation” indicates a total number of unique connotations appearing in the 680 records. While the average number of AAT connotations per record is 1.34, the average number of Associated connotations per record is from 3.25 to 5.75 depending on which association thesauri is used.

Table II demonstrates the evaluation results of three thesauri. The first column contains the association measure used in constructing the association thesauri. The second column indicates the maximum number of Associated connotations per record, i.e. the depth of indexing. The depth of indexing was designed to reflect the ranks of association weights. The third and fourth columns show the total number of assigned Associated connotations and the average number of Associated connotations per record, respectively. The table presents the number of Associated connotations matching AAT connotations followed by recall and precision scores.

These results indicate that the Cosine, Dice, and Yule's Y have similar patterns, but on the whole, the Cosine measure shows the best results of the three measures. The number of matches (the fifth column) shows that as the depth of retrieval increases from one to five, the number of matched connotations increases proportionally, but beyond that point it increases slowly. As the depth of indexing increases, recall increases and precision decreases, as expected (refer to Figure 4). However, the recall and precision scores do not seem satisfactory. One possible reason for relatively low recall and precision scores might be the constraints of this study – only a small number of denotations and connotations were used in constructing the thesauri and indexing the dataset. Another issue affecting the low recall and precision might be the conservative nature of the evaluation. Although the association thesauri assign relevant connotations, most of those Associated connotations could be estimated as non‐relevant because they do not match AAT connotations. The relevance judgment results on Associated connotations will be reported in the next section. In spite of these limitations, the results show that by applying association thesauri to a collection having only denotative index terms, it is possible to assign 27 percent of all relevant connotations (among 909 AAT connotations, 249 terms were able to be assigned through Cosign association thesaurus). With considerations on low‐term agreement across indexers –for instance, Markey (1984) demonstrated 13 percent overlap among three image indexers' index terms– 27 percent overlap between AAT connotations and Associated connotations implies a great potential.

Relative recall of image documents and browsing sets

This section reports the results of the second experiment designed to support two hypotheses. Whereas, the first experiment defined the relevant set as AAT connotations, the second experiment expanded the relevant sets by reflecting relevance judgments on Associated connotations. Then, by measuring relative recall, the second experiment focuses onto what extent the Associated connotations can retrieve additional image documents as well as browsing sets. Based on the previous analysis, the association thesaurus developed with the Cosine measure was used in measuring relative recalls. The testing set for the second experiment was limited to the records containing the images, because the judges were asked to review both the image itself and its bibliographic description. (N.B.: Only small portions of the CHIN collection provide both the image and a bibliographic description. Most records contain only bibliographic descriptions.) As a result, 58 records were selected from the 680 records for the second experiment. The results of the two tests supporting H1 and H2 are discussed next.

First, retrieval of additional connotatively relevant images by applying an association thesaurus was tested. This evaluation was performed by comparing relative recalls and precisions. The relative recall and precision of image documents which are indexed with the association thesaurus were compared with those of the AAT connotations. As shown in Table III, the number of AAT connotations assigned to 58 records was 67, and these 67 connotations were all regarded as relevant connotations. The total number of Associated connotations assigned to 58 records was 346, and among the 346 connotations, 278 connotations were judged as relevant; that is, approximately 80 percent of Associated connotations were judged as relevant. The number of unique relevant connotations, from AAT connotations or from Associated connotations, was 337.

Table IV indicates the t‐test results of relative recalls and precisions of retrieved image documents. It shows significant differences between the relative recall for AAT connotations (0.24) and the relative recall for the association thesaurus (0.79) at the 0.05 level (paired samples t‐test, p‐value=0.000). It was also found that the association thesaurus can cause a decline in precision without statistical significance (paired samples t‐test, p‐value=0.38). In spite of a decline in precision, this result supports the H1 concentrating on recall, which is, a connotation derived from denotation retrieves additional relevant images and increases relative recall of image documents. From the result, it can be concluded that the association thesaurus can provide a larger number of relevant connotations than connotations assigned by human indexers.

Next, the relative recalls and precisions of browsing sets were computed. This study defined browsing sets as connotative groups which are retrieved against a denotative query. The browsing sets retrieved and expanded through the association thesaurus were compared with the browsing sets expanded through AAT connotations. The 57 denotative queries extracted from the 58 records were used for measuring the relative recall of browsing sets. Table V presents the number of browsing sets expanded through AAT connotations and the association thesaurus. The average number of relevant browsing sets obtained through the association thesaurus was 2.26. This implies that even when the image collection is indexed with denotative terms, it is possible to display approximately two browsing sets (connotative groups) by utilizing the association. Compared to the browsing sets from the association thesaurus, it was found that AAT connotations generate, on average, 1.51 browsing sets.

Table VI indicates the t‐test results of relative recalls and precisions of browsing sets. It shows significant differences between the relative recall of browsing sets expanded from AAT connotations (0.41) and those expanded from the association thesaurus (0.62) at the 0.05 level (paired samples t‐test, p‐value=0.000). However, it was also discovered that the association thesaurus can cause a statistically significant decline in precision (paired samples t‐test, p‐value=0.000). In spite of a significant decline in precision, this result supported H2, which is a connotation derived from denotation retrieves additional relevant browsing sets and increases relative recall of browsing sets. The results of the experiments demonstrated that the association thesaurus which was developed based on associations between connotation and denotation can facilitate assigning connotative terms to image documents indexed with only denotative terms. Especially, it was an encouraging result that 80 percent of connotative index terms assigned through the association thesaurus were judged as relevant. In addition, it was also found that the association thesaurus improves the recall of retrieved image documents as well as browsing sets compared to the connotations given by human indexers. From this analysis, it can be concluded that the association thesaurus can facilitate availability of connotative messages.