Classifying Idiomatic and Literal Expressions Using Topic Models and Intensity of Emotions
Classifying Idiomatic and Literal Expressions Using Topic Models and Intensity of Emotions
Classifying Idiomatic and Literal Expressions Using Topic Models and Intensity of Emotions
Intensity of Emotions
Jing Peng & Anna Feldman Ekaterina Vylomova
Computer Science/Linguistics Computer Science
Montclair State University Bauman State Technical University
Montclair, New Jersey, USA Moscow, Russia
{pengj,feldmana}@mail.montclair.edu [email protected]
2019
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2019–2027,
c
October 25-29, 2014, Doha, Qatar.
2014 Association for Computational Linguistics
in the local topic. We treat idioms as semantic outliers, based classification task and in a token identification
and the identification of semantic shift as outlier detec- task, in which they distinguish idiomatic and literal us-
tion. Thus, this topic representation allows us to differ- ages of potentially idiomatic expressions in context.
entiate idioms from literals using the local semantics. Sporleder and Li (2009) present a graph-based model
The paper is organized as follows. Section 2 briefly for representing the lexical cohesion of a discourse.
describes previous approaches to idiom recognition or Nodes represent tokens in the discourse, which are con-
classification. In Section 3 we describe our approach in nected by edges whose value is determined by a seman-
detail, including the hypothesis, the topic space repre- tic relatedness function. They experiment with two dif-
sentation, and the proposed algorithm. After describing ferent approaches to semantic relatedness: 1) Depen-
the preprocessing procedure in Section 4, we turn to the dency vectors, as described in Pado and Lapata (2007);
actual experiments in Sections 5 and 6. We then com- 2) Normalized Google Distance (Cilibrasi and Vitányi
pare our approach to other approaches (Section 7) and (2007)). Sporleder and Li (2009) show that this method
discuss the results (Section 8). works better for larger contexts (greater than five para-
graphs). Li and Sporleder (2010) assume that literal
2 Previous Work and figurative data are generated by two different Gaus-
sians, literal and non-literal and the detection is done by
Previous approaches to idiom detection can be classi- comparing which Gaussian model has a higher prob-
fied into two groups: 1) Type-based extraction, i.e., de- ability to generate a specific instance. The approach
tecting idioms at the type level; 2) token-based detec- assumes that the target expressions are already known
tion, i.e., detecting idioms in context. Type-based ex- and the goal is to determine whether this expression is
traction is based on the idea that idiomatic expressions literal or figurative in a particular context. The impor-
exhibit certain linguistic properties that can distinguish tant insight of this method is that figurative language
them from literal expressions (Sag et al. (2002); Fa- in general exhibits less semantic cohesive ties with the
zly et al. (2009)), among many others, discuss various context than literal language.
properties of idioms. Some examples of such proper- Feldman and Peng (2013) describe several ap-
ties include 1) lexical fixedness: e.g., neither ‘shoot proaches to automatic idiom identification. One of
the wind’ nor ‘hit the breeze’ are valid variations of them is idiom recognition as outlier detection. They
the idiom shoot the breeze and 2) syntactic fixedness: apply principal component analysis for outlier detec-
e.g., The guy kicked the bucket is potentially idiomatic tion – an approach that does not rely on costly an-
whereas The bucket was kicked is not idiomatic any- notated training data and is not limited to a specific
more; and of course, 3) non-compositionality. Thus, type of a syntactic construction, and is generally lan-
some approaches look at the tendency for words to oc- guage independent. The quantitative analysis provided
cur in one particular order, or a fixed pattern. Hearst in their work shows that the outlier detection algorithm
(1992) identifies lexico-syntactic patterns that occur performs better and seems promising. The qualitative
frequently, are recognizable with little or no precoded analysis also shows that their algorithm has to incor-
knowledge, and indicate the lexical relation of interest. porate several important properties of the idioms: (1)
Widdows and Dorow (2005) use Hearst’s concept of Idioms are relatively non-compositional, comparing to
lexicosyntactic patterns to extract idioms that consist literal expressions or other types of collocations. (2)
of fixed patterns between two nouns. Basically, their Idioms violate local cohesive ties, as a result, they are
technique works by finding patterns such as “thrills and semantically distant from the local topics. (3) While
spills”, whose reversals (such as “spills and thrills”) are not all semantic outliers are idioms, non-compositional
never encountered. semantic outliers are likely to be idiomatic. (4) Id-
While many idioms do have these properties, many iomaticity is not a binary property. Idioms fall on the
idioms fall on the continuum from being composi- continuum from being compositional to being partly
tional to being partly unanalyzable to completely non- unanalyzable to completely non-compositional.
compositional (Cook et al. (2007)). Fazly et al. (2009); The approach described below is taking Feldman
Li and Sporleder (2010), among others, notice that and Peng (2013)’s original idea and is trying to address
type-based approaches do not work on expressions that (2) directly and (1) indirectly. Our approach is also
can be interpreted idiomatically or literally depending somewhat similar to Li and Sporleder (2010) because it
on the context and thus, an approach that considers to- also relies on a list of potentially idiomatic expressions.
kens in context is more appropriate for the task of idiom
recognition. 3 Our Hypothesis
A number of token-based approaches have been
discussed in the literature, both supervised (Katz Similarly to Feldman and Peng (2013), out starting
and Giesbrech (2006)), weakly supervised (Birke and point is that idioms are semantic outliers that violate
Sarkar (2006)) and unsupervised (Sporleder and Li cohesive structure, especially in local contexts. How-
(2009); Fazly et al. (2009)). Fazly et al. (2009) de- ever, our task is framed as supervised classification and
velop statistical measures for each linguistic property we rely on data annotated for idiomatic and literal ex-
of idiomatic expressions and use them both in a type- pressions. We hypothesize that words in a given text
2020
segment, such as a paragraph, that are high-ranking place no restriction on the vocabulary. That is, the vo-
representatives of a common topic of discussion are cabulary includes terms from documents that contain
less likely to be a part of an idiomatic expression in both idioms and literals.
the document. Note that by computing MD̂ , the topic term by doc-
ument matrix, from the training data, we have created
3.1 Topic Space Representation a vocabulary, or a set of “features” (i.e., topic terms)
Instead of the simple bag of words representation of a that is used to directly describe a query or test segment.
target document (segment of three paragraphs that con- The main advantage is that topics are more accurate
tains a target phrase), we investigate the bag of words when computed by LDA from a large collection of id-
topic representation for target documents. That is, we iomatic or literal contexts. Thus, these topics capture
extract topics from paragraphs containing idioms and more accurately the semantic contexts in which the tar-
from paragraphs containing literals using an unsuper- get idiomatic and literal expressions typically occur. If
vised clustering method, Latent Dirichlet Allocation a target query appears in a similar semantic context, the
(LDA) (Blei et al., 2003). The idea is that if the LDA topics will be able to describe this query as well. On the
model is able to capture the semantics of a target docu- other hand, one might similarly apply LDA to a given
ment, an idiomatic phrase will be a “semantic” outlier query to extract query topics, and create the query vec-
of the themes. Thus, this topic representation will al- tor from the query topics. The main disadvantage is
low us to differentiate idioms from literals using the that LDA may not be able to extract topic terms that
semantics of the local context. match well with those in the training corpus, when ap-
Let d = {w1 , · · · , wN }t be a segment (document) plied to the query in isolation.
containing a target phrase, where N denotes the num-
ber of terms in a given corpus, and t represents trans- 3.2 Algorithm
pose. We first compute a set of m topics from d. We The main steps of the proposed algorithm, called
denote this set by TopSpace, are shown below.
T (d) = {t1 , · · · , tm },
Input: D = {d1 , · · · , dk , dk+1 , · · · , dn }: training
where ti = (w1 , · · · , wk )t . Here wj represents a word documents of k idioms and n − k literals.
from a vocabulary of W words. Thus, we have two Q = {q1 , · · · , ql }: l query documents.
representations for d: (1) d, represented by its original
terms, and (2) d,ˆ represented by its topic terms. Two 1. Let DicI be the vocabulary determined solely
corresponding term by document matrices will be de- from idioms {d1 , · · · , dk }. Similarly, let DicL
noted by MD and MD̂ , respectively, where D denotes be the vocabulary obtained from literals
a set of documents. That is, MD represents the original {dk+1 , · · · , dn }.
“text” term by document matrix, while MD̂ represents
2. For a document di in {d1 , · · · , dk }, apply LDA
the “topic” term by document matrix.
to extract a set of m topics T (di ) = {t1 , · · · , tm }
Figure 1 shows the potential benefit of topic space
using DicI. For di ∈ {dk+1 , · · · , dn }, DicL is
representation. In the figure, text segments containing
used.
target phrase “blow whistle” are projected on a two di-
mensional subspace. The left figure shows the projec- 3. Let D̂ = {dˆ1 , · · · , dˆk , dˆk+1 , · · · , dˆn } be the
tion in the “text” space, represented by the term by doc- resulting topic representation of D.
ument matrix MD . The middle figure shows the projec-
tion in the topic space, represented by MD̂ . The topic 4. Compute the term by document matrix MD̂ from
space representation seems to provide a better separa- D̂, and let DicT and gw be the resulting
tion. dictionary and global weight (idf ), respectively.
We note that when learning topics from a small data
sample, learned topics can be less coherent and inter- 5. Compute the term by document matrix MQ from
pretable, thus less useful. To address this issue, regu- Q, using DicT and gw from the previous step.
larized LDA has been proposed in the literature (New- Output: MD̂ and MQ
man et al., 2011). A key feature is to favor words that
exhibit short range dependencies for a given topic. We
can achieve a similar effect by placing restrictions on To summarize, after splitting our corpus (see section
the vocabulary. For example, when extracting topics 4) into paragraphs and preprocessing it, we extract top-
from segments containing idioms, we may restrict the ics from paragraphs containing idioms and from para-
vocabulary to contain words from these segments only. graphs containing literals. We then compute a term by
The middle and right figures in Figure 1 illustrate a case document matrix, where terms are topic terms and doc-
in point. The middle figure shows a projection onto the uments are topics extracted from the paragraphs. Our
topic space that is computed with a restricted vocabu- test data are represented as a term-by-document matrix
lary, while the right figure shows a projection when we as well (See the details in section 5).
2021
2D Text Space: Blow Whistle 2D Topic Space: Blow Whistle 2D Topic Space: Blow Whistle
100 20 25
Idioms Idioms Idioms
Literals Literals Literals
20
80
15
15
60
10
10
40
5
5
20
0
0
0
−5
−20 −5 −10
−100 −80 −60 −40 −20 0 20 −20 −15 −10 −5 0 5 10 15 −12 −10 −8 −6 −4 −2 0 2 4 6 8
Figure 1: 2D projection of text segments containing “blow whistle.” Left panel: Original text space. Middle panel:
Topic space with restricted vocabulary. Right panel: Topic space with enlarged vocabulary.
3.3 Fisher Linear Discriminant Analysis m0 is the mean vector of all data, and mj is the mean
vector of jth class data. A within-class scatter ma-
Once MD̂ and MQ are obtained, a classification rule trix characterizes the scatter of samples around their
can be applied to predict idioms vs. literals. The ap- respective class mean vector, and it is expressed by
proach we are taking in this work for classifying id-
ioms vs. literals is based on Fisher’s discriminant anal- J lj
X X
ysis (FDA) (Fukunaga, 1990). FDA often significantly Sw = pj (xji − mj )(xji − mj )t , (1)
simplifies tasks such as regression and classification by j=1 i=1
2022
(2009); Cook et al. (2008) and labeled as L (Literal), applied: a) FDA (Eq. 4), followed by the nearest neigh-
I (Idioms), or Q (Unknown). The list contains only bor rule. b) SVMs with Gaussian kernels (Cristianini
those VNCs whose frequency was greater than 20 and and Shawe-Taylor (2000)). For the nearest neighbor
that occurred at least in one of two idiom dictionaries rule, the number of nearest neighbors is set to dn/5e,
(Cowie et al., 1983; Seaton and Macaulay, 2002). The where n denotes the number of training examples. For
dataset consists of 2,984 VNC tokens. For our experi- SVMs, kernel width and soft margin parameters are set
ments we only use VNCs that are annotated as I or L. to default values.
2023
Table 1: Average accuracy of competing methods on four datasets in single paragraph contexts: A = Arousal
Model BlowWhistle LoseHead MakeScene TakeHeart
Prec Recall Acc Prec Recall Acc Prec Recall Acc Prec Recall Acc
FDA-Topics 0.44 0.40 0.79 0.70 0.90 0.70 0.82 0.97 0.81 0.91 0.97 0.89
FDA-Topics+A 0.51 0.51 0.75 0.78 0.68 0.66 0.80 0.99 0.80 0.93 0.84 0.80
FDA-Text 0.37 0.81 0.63 0.60 0.88 0.58 0.82 0.89 0.77 0.36 0.38 0.41
FDA-Text+A 0.42 0.49 0.76 0.64 0.92 0.63 0.83 0.95 0.82 0.75 0.53 0.53
SVMs-Topics 0.08 0.39 0.59 0.28 0.25 0.45 0.59 0.74 0.61 0.91 1.00 0.91
SVMs-Topics+A 0.06 0.21 0.69 0.38 0.18 0.44 0.53 0.40 0.44 0.91 1.00 0.91
SVMs-Text 0.08 0.39 0.59 0.36 0.60 0.52 0.23 0.30 0.40 0.42 0.16 0.22
SVMs-Text+A 0.15 0.51 0.60 0.31 0.38 0.48 0.37 0.40 0.45 0.95 0.48 0.50
(the degree of control exerted by a stimulus). These being equal, this clearly shows the advantage of topics
components were elicited from human subjects via an over simple text representation.
Amazon Mechanical Turk crowdsourced experiment.
We only used the arousal feature in our experiments 0.8
0
dA −mA = {a1 −mA , · · · , aN −mA }t . Similarly, the Precision Recall Accuracy
2024
Table 2: Average accuracy of competing methods on four datasets in multiple paragraph contexts: A = Arousal
Model BlowWhistle LoseHead MakeScene TakeHeart
Prec Recall Acc Prec Recall Acc Prec Recall Acc Prec Recall Acc
FDA-Topics 0.62 0.60 0.83 0.76 0.97 0.78 0.79 0.95 0.77 0.93 0.99 0.92
FDA-Topics+A 0.47 0.44 0.79 0.74 0.93 0.74 0.82 0.69 0.65 0.92 0.98 0.91
FDA-Text 0.65 0.43 0.84 0.72 0.73 0.65 0.79 0.95 0.77 0.46 0.40 0.42
FDA-Text+A 0.45 0.49 0.78 0.67 0.88 0.65 0.80 0.99 0.80 0.47 0.29 0.33
SVMs-Topics 0.07 0.40 0.56 0.60 0.83 0.61 0.46 0.57 0.55 0.90 1.00 0.90
SVMs-Topics+A 0.21 0.54 0.55 0.66 0.77 0.64 0.42 0.29 0.41 0.91 1.00 0.91
SVMs-Text 0.17 0.90 0.25 0.30 0.50 0.50 0.10 0.01 0.26 0.65 0.21 0.26
SVMs-Text+A 0.24 0.87 0.41 0.66 0.85 0.61 0.07 0.01 0.26 0.74 0.13 0.20
Text Representation
0.7 0.2
Text
Text+A 0
0.6
−0.2
0.5 Idioms
Literals
−0.4
Arousal Values
0.4
−0.6
0.3
−0.8
0.2
−1
0.1 −1.2
0 −1.4
Precision Recall Accuracy 0 5 10 15
Text Terms
Topic Representation
0.8 1
Topics Idioms
0.7 Topics+A Literals
0.5
0.6
0.5
Arousal Values
0.4
−0.5
0.3
0.2
−1
0.1
0 −1.5
Precision Recall Accuracy 0 5 10 15
Topic Terms
Figure 3: Aggregated performance: Text Figure 4: Average arousal values–Upper panel: Text
vs. text+Arousal representations (top) and Top- space. Lower panel: Topic space.
ics vs. Topics+Arousal representations (bottom).
2025
relatively short. A larger context provides more related comments. The third author thanks the Fulbright Foun-
terms, which gives LDA more opportunities to sample dation for giving her an opportunity to conduct this re-
these terms. search at Montclair State University (MSU).
Idioms are also relatively non-compositional. While
we do not measure their non-compositionality in this References
approach, we indirectly touch upon this property by hy- Ariel, M. (2002). The demise of unique concept of
pothesizing that non-compositional idiomatic expres- literal meaning. Journal of Pragmatics 34, 361–402.
sions are likely to be far from the local topics.
We feel that incorporating the intensity of emotion Birke, J. and A. Sarkar (2006). A clustering approach
expressed by the context into our model improves per- to the nearly unsupervised recognition of nonliteral
formance, in particular, in text representation. When language. In Proceedings of the 11th Conference of
we performed a qualitative analysis of the results try- the European Chapter of the Association for Com-
ing to determine the causes of false positives and neg- putational Linguistics (EACL’06), Trento, Italy, pp.
atives, we noticed that there were quite a number of 329–226.
cases that improved after incorporating the arousal fea- Blei, D., A. Ng, and M. Jordan (2003). Latent Dirich-
ture into the model. For example, the FDA:topic classi- let Allocation. Journal of Machine Learning Re-
fier labels ”blow the whistle” as literal in the following search 3, 993–1022.
context, but FDA:topics+A marks this expression as id- Burnard, L. (2000). The British National Corpus Users
iomatic (italicized words indicate words with relatively Reference Guide. Oxford University Computing Ser-
high arousal values): vices.
Peter thought it all out very carefully. He decided the wis- Cilibrasi, R. and P. M. B. Vitányi (2007). The
est course was to pool all he had made over the last two years,
enabling Julian to purchase the lease of a high street property. google similarity distance. IEEE Trans. Knowl. Data
This would enable them to set up a business on a more set- Eng. 19(3), 370–383.
tled and permanent trading basis. Before long they opened a Cook, P., A. Fazly, and S. Stevenson (2007). Pulling
grocery-cum-delicatessen in a good position as far as passing
trade was concerned. Peter’s investment was not misplaced. their weight: Exploiting syntactic forms for the auto-
The business did very well with the two lads greatly appreci- matic identification of idiomatic expressions in con-
ated locally for their hard work and quality of service. The text. In Proceedings of the ACL 07 Workshop on A
range of goods they were able to carry was welcomed in the Broader Perspective on Multiword Expressions, pp.
area, as well as lunchtime sandwich facilities which had pre- 41–48.
viously been missing in the neighbourhood.
Success was the fruit of some three years’ strenuous work. Cook, P., A. Fazly, and S. Stevenson (2008, June). The
But it was more than a shock when Julian admitted to Pe- VNC-Tokens Dataset. In Proceedings of the LREC
ter that he had been running up huge debts with their bank. Workshop: Towards a Shared Task for Multiword
Peter knew that Julian gambled, but he hadn’t expected him Expressions (MWE 2008), Marrakech, Morocco.
to gamble to that level, and certainly not to use the shop as
security. With continual borrowing over two years, the bank Cowie, A. P., R. Mackin, and I. R. McCaig (1983). Ox-
had blown the whistle. Everything was gone. Julian was ford Dictionary of Current Idiomatic English, Vol-
bankrupt. Even if they’d had a formal partnership, which ume 2. Oxford University Press.
they didn’t, it would have made no difference. Peter lost all
he’d made, and with it his chance to help his parents and his Cristianini, N. and J. Shawe-Taylor (2000). An In-
younger brother and sister, Toby and Laura. troduction to Support Vector Machines and other
Peter was heartbroken. His father had said all along: nei- kernel-based learning methods. Cambridge, UK:
ther a lender nor a borrower. Peter had found out the hard Cambridge University Press.
way. But as his mother observed, he was the same Peter, he’d
Fazly, A., P. Cook, and S. Stevenson (2009). Unsu-
pick himself up somehow. Once again, Peter was resolute. He
made up his mind he’d never make the same mistake twice. It pervised Type and Token Identification of Idiomatic
wasn’t just the money or the hard work, though the waste of Expressions. Computational Linguistics 35(1), 61–
that was difficult enough to accept. Peter had been working 103.
a debt of love. He’d done all this for his parents, particularly
for his father, whose dedication to his children had always Feldman, A. and J. Peng (2013). Automatic detec-
impressed Peter and moved him deeply. And now it had all tion of idiomatic clauses. In Computational Linguis-
come to nothing. tics and Intelligent Text Processing, pp. 435–446.
Therefore, we think that idioms have the tendency to Springer.
appear in more affective contexts; and we think that in- Fukunaga, K. (1990). Introduction to statistical pattern
corporating more sophisticated sentiment analysis into recognition. Academic Press.
our model will improve the results.
Hearst, M. A. (1992). Automatic acquisition of hy-
Acknowledgments ponyms from large text corpora. In Proceedings of
the 14th Conference on Computational Linguistics
This material is based upon work supported by the Na- - Volume 2, COLING ’92, Stroudsburg, PA, USA,
tional Science Foundation under Grant No. 1319846. pp. 539–545. Association for Computational Lin-
We also thank the anonymous reviewers for useful guistics.
2026
Horvath, B. M. (1985). Variation in Australian English.
Cambridge: Cambridge University PRess.
Katz, G. and E. Giesbrech (2006). Automatic Iden-
tification of Non-compositional Multiword Expres-
sions using Latent Semantic Analysis. In Proceed-
ings of the ACL/COLING-06 Workshop on Multi-
word Expressions: Identifying and Exploiting Un-
derlying Properties, pp. 12–19.
Li, L. and C. Sporleder (2010). Using Gaussian Mix-
ture Models to Detect Figurative Language in Con-
text. In Proceedings of NAACL/HLT 2010.
Newman, D., E. V. Bonilla, and W. L. Buntine (2011).
Improving topic coherence with regularized topic
models. In NIPS, pp. 496–504.
Nunberg, G., I. A. Sag, and T. Wasow (1994). Idioms.
Language 70(3), 491–538.
Pado, S. and M. Lapata (2007). Dependency-based
construction of semantic space models. Computa-
tional Linguistics 33(2), 161–199.
Peng, J., A. Feldman, and L. Street (2010). Comput-
ing linear discriminants for idiomatic sentence de-
tection. Research in Computing Science, Special is-
sue: Natural Language Processing and its Applica-
tions 46, 17–28.
Sag, I. A., T. Baldwin, F. Bond, A. Copestake, and
D. Flickinger (2002). Multiword expressions: A
Pain in the Neck for NLP. In Proceedings of the
3rd International Conference on Intelligence Text
Processing and Computational Linguistics (CICLing
2002), Mexico City, Mexico, pp. 1–15.
Seaton, M. and A. Macaulay (Eds.) (2002). Collins
COBUILD Idioms Dictionary (second ed.). Harper-
Collins Publishers.
Sporleder, C. and L. Li (2009). Unsupervised Recogni-
tion of Literal and Non-literal Use of Idiomatic Ex-
pressions. In EACL ’09: Proceedings of the 12th
Conference of the European Chapter of the Associa-
tion for Computational Linguistics, Morristown, NJ,
USA, pp. 754–762. Association for Computational
Linguistics.
Warriner, A. B., V. Kuperman, and M. Brysbaert
(2013). Norms of valence, arousal, and dominance
for 13,915 english lemmas. Behavior Research
Methods 44(4).
Widdows, D. and B. Dorow (2005). Automatic extrac-
tion of idioms using graph analysis and asymmet-
ric lexicosyntactic patterns. In Proceedings of the
ACL-SIGLEX Workshop on Deep Lexical Acquisi-
tion, DeepLA ’05, Stroudsburg, PA, USA, pp. 48–
56. Association for Computational Linguistics.
Woods, A., P. Fletcher, and A. Hughes (1986). Statis-
tics in Language Studies. Cambridge: Cambridge
University Press.
2027