Linee Guida per l’Inglese

Guidelines for the Annotation of Irony

Authors: Cristina Bosco, Luca Anselma, Pier Felice Balestrucci, Valerio Basile, Eliana Di Palma, Marta Marchiori Manerba, Michael Oliverio, Viviana Patti, Alessandro Mazzei

Affiliation: Department of Computer Science, University of Turin, Italy, Corso Svizzera 185, 10149 Torino (Italy)

Contact: cristina.bosco@unito.it

Recognizing irony is a challenging task, and the availability of annotated corpora may be crucial for its automatic processing. This document provides guidelines for annotation based on a fine-grained scheme centered on irony, with particular focus on a variety of rhetorical figures which can play the role of irony triggers.

Introduction

Irony detection is an important task in Sentiment Analysis, as its presence can reverse the polarity of an opinion expressed in a text. For example, positive words may be used to convey a negative meaning, thereby undermining the performance of Sentiment Analysis systems [1] [2] [3] [4] [5] [6]

In fact, irony relies on pragmatic and linguistic phenomena whose identification can be controversial [7] [8] [9], since different rhetorical devices may be used as triggers of irony, such as analogy, euphemism, context shift, oxymoron and paradox, hyperbole, false assertion and rhetorical question. This makes the identification of irony an especially challenging task for both human annotators and automatic tools.

Furthermore, the automatic processing of this phenomenon is complicated by the co-occurrence of similar forms of speech, such as sarcasm or satire [10] [11] [12] [13], as well as by the text domain.

The use of tools for irony detection has been particularly focused on micro-blogging platforms, as irony is widely used by online users. In social media, contrasts that trigger irony in short messages (such as tweets) often involve at least two propositions (or sometimes single words) that contradict each other [14]. Nevertheless, this contradiction can occur at a verbal or situational level, and the two contrasting elements can both be part of the internal context (e.g., explicitly lexicalized), or one may be present while the other is implied and must be inferred from a context external to the message.

Annotation Guidelines

The annotation described in this document1 has been developed to capture the notions of irony in a fine-grained manner within a multilingual corpus [14] [15] (covering Italian, French, Spanish and English).

A recent research project coordinated by the Computer Science Department of the University of Turin enabled us to apply this scheme to another corpus, i.e., the Italian section of MULTIPICO [16]. MULTIPICO is a corpus from Twitter and Reddit that includes a larger set of languages annotated for irony, covering 9 languages and 25 varieties. In this corpus, the data are organized in pairs, each consisting of one post and one reply, which is annotated with a binary label indicating whether it is ironic with respect to the corresponding post.

The promising results of our annotation exercise on the ironic messages of the Italian section of MULTIPICO, described in [17], led to the decision to apply the scheme also to other ironic data extracted from MULTIPICO for English, Spanish and French.

The annotation task

Given a set of ironic messages (i.e., ironic according to the annotation provided in MULTIPICO) composed of one post and one reply, the annotation task for each message consists in:

  • selecting one (or more) of the rhetorical figures (listed and described in the next section), to indicate the kind of irony trigger or triggers occurring in the reply.

Rhetorical Figures

The following list includes the rhetorical figures to be annotated. The examples may help the annotator better understand the meaning of each figure.

  • ANALOGY – Irony is triggered by a literal or metaphorical comparison of two elements, or by a comparison of an element with an external context.

    Example:
    Post: He killed those people. You should ask for her resignation rather. She was not saying a word.
    Reply: It’s like wanting a bicycle but asking for a car.

  • EUPHEMISM – One element is supposedly used to soften or simply substitute the other one in the pair or some external knowledge.

    Example:
    Post: We can wage a nuclear war to avoid solving the climate crisis.
    Reply: It seems a tiny bit excessive to me!

  • CONTEXT SHIFT – The words, the style or the register of one of the two elements shows a striking change of topic/frame with the other element.

    Example:
    Post: Are you afraid to fly? I am.
    Reply: @USER Only when my arms are tired.

  • OXYMORON / PARADOX – The two elements are in clear contrast or contradiction, such as two opposite events.

    Example:
    Post: The idea of superpowers, supernatural powers, and powers given to humans by god, has existed for thousands of years. We can’t really say who created these ideas first. But, we can say that it wasn’t the creator of the Power Puff Girls, sorry.
    Reply: That’s your realistic and correct information but like you’re still wrong.

  • HYPERBOLE – One element or both elements express an idea or a feeling in an exaggerated way.

    Example:
    Post: The effects of Covid-19 on the population: I have the start of the flu with a 99.3°F (37.4°C) temperature. As soon as my mom found out, she basically lost her mind. She’s panicking, measuring her own temp, and wants to call the authorities. Save me.
    Reply: Double down and tell her you’re all going to die.

  • FALSE ASSERTION – A proposition, fact or assertion fails to make sense against reality.

    Example:
    Post: Temperatures in Siberia are skyrocketing.
    Reply: Russia needs some global warming. The value of land in Siberia rises.

  • RHETORICAL QUESTION – A question is asked in order to make a point rather than to elicit an answer, involving just one of the elements or both.

    Example:
    Post: It’s absolutely freezing outside.
    Reply: Where is global warming when it is needed?

  • OTHER – This category may comprise different linguistic and pragmatic phenomena, such as puns, humor or situational irony.

    Example:
    Post: Once again this year I’ll be looking at your Christmas trees thinking: why do you even bother?
    Reply: @USER The joy of liv… ah, never mind.

  1. The guidelines discussed in this paper are largely inspired by this document: https://github.com/IronyAndTweets/Scheme

References

  1. Cristina Bosco, Viviana Patti, and Andrea Bolioli. 2013. Developing Corpora for Sentiment Analysis: The Case of Irony and Senti-TUT. IEEE Intelligent Systems, 28(2), 55–63.
  2. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in Twitter. Language Resources and Evaluation, 47(1), 239–268.
  3. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling Sarcasm in Twitter, a Novel Approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 50–58, Baltimore, Maryland. Association for Computational Linguistics.
  4. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), 470–478.
  5. Delia Irazú Hernández-Farías, José-Miguel Benedí, and Paolo Rosso. 2015. Applying Basic Features from Sentiment Analysis for Automatic Irony Detection. In Pattern Recognition and Image Analysis: 7th Iberian Conference, IbPRIA 2015, Lecture Notes in Computer Science 9117, 337–344. Springer.
  6. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing Context Incongruity for Sarcasm Detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 757–762, Beijing, China. Association for Computational Linguistics.
  7. H. Paul Grice. 1975. Logic and Conversation. In Peter Cole and Jerry L. Morgan (eds.), Syntax and Semantics, Vol. 3: Speech Acts, 41–58. Academic Press.
  8. H. Paul Grice. 1978. Further Notes on Logic and Conversation. In Peter Cole (ed.), Syntax and Semantics, Vol. 9: Pragmatics. Academic Press.
  9. Dan Sperber and Deirdre Wilson. 1981. Irony and the Use-Mention Distinction. In Peter Cole (ed.), Radical Pragmatics, 295–318. Academic Press.
  10. Delia Irazú Hernández-Farías and Paolo Rosso. 2017. Irony, Sarcasm, and Sentiment Analysis. In Federico Alberto Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu (eds.), Sentiment Analysis in Social Networks, 113–128. Elsevier.
  11. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2017. Automatic Sarcasm Detection: A Survey. ACM Computing Surveys, 50(5), Article 73.
  12. Kumar Ravi and Vadlamani Ravi. 2017. A Novel Automatic Satire and Irony Detection Using Ensembled Feature Selection and Data Mining. Knowledge-Based Systems, 120, 15–33.
  13. Shiwei Zhang, Xiuzhen Zhang, Jeffrey Chan, and Paolo Rosso. 2019. Irony Detection via Sentiment-Based Transfer Learning. Information Processing & Management, 56(5), 1633–1644.
  14. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the Impact of Pragmatic Phenomena on Irony Detection in Tweets: A Multilingual Corpus Study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 262–272, Valencia, Spain. Association for Computational Linguistics.
  15. Alessandra Teresa Cignarella, Cristina Bosco, Viviana Patti, and Mirko Lai. 2018. Application and Analysis of a Multi-layered Scheme for Irony on the Italian Twitter Corpus TWITTIRÒ. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  16. Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, and Davide Bernardi. 2024. MultiPICo: Multilingual Perspectivist Irony Corpus. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 16008–16021.
  17. Pier Felice Balestrucci, Michael Oliverio, Soda Marem Lo, Luca Anselma, Valerio Basile, Alessandro Mazzei, and Viviana Patti. 2025. When Figures Speak with Irony: Investigating the Role of Rhetorical Figures in Irony Generation with LLMs. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), 55–63, Cagliari, Italy. CEUR Workshop Proceedings.