The EMNLP 2026 (Conference on Empirical Methods in Natural Language Processing), one of the world’s leading international scientific conferences dedicated to natural language processing, took place in Budapest, Hungary, from October 24 to 29, 2026. The paper entitled “Multi-Model Framework for Sentiment Annotation: A Use Case of Multilingual Wikipedia” was presented at the conference. The study introduces an approach to automated sentiment annotation that leverages agreement among multiple artificial intelligence models to improve the reliability of the resulting labels.
Traditional manual annotation of textual data is time-consuming and expensive, which has led to the increasing use of large language models (LLMs) for this task. However, their predictions are not always consistent – the same text may receive different labels depending on the model used or the particular run of the analysis. To address this challenge, the authors proposed a framework in which three language models (GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6) act as independent annotators. Each model evaluated the sentiment of the same sentences three times, classifying them as positive, neutral, or negative, with an additional option for cases where sentiment could not be reliably determined. Labels were considered particularly reliable only when all nine assessments were in agreement. The approach was tested on sentences extracted from the introductory sections of Wikipedia articles in five languages: English, German, Spanish, Polish, and Russian. Wikipedia was deliberately chosen as the source of data because its articles are expected to maintain a neutral point of view, making the identification of subtle differences in sentiment an interesting research challenge.
The results showed that all nine assessments agreed for approximately 71-76% of the analyzed sentences, depending on the language. Fleiss’ kappa, a statistical measure of agreement among annotators that accounts for chance agreement, ranged from 0.72 to 0.76, indicating a high level of consistency. The model-generated labels were also compared with human judgments using selected samples of Polish and English sentences. For sentences meeting the strict unanimity criterion, the labels showed perfect agreement with human annotations. This finding highlights the potential of the proposed approach, although it does not mean that the models are accurate in every situation. The study also revealed an important limitation: individual models performed considerably worse at identifying negative sentiment, particularly in predominantly neutral texts.
Another important outcome of the research is a multilingual dataset containing more than 75,000 sentences with assigned sentiment labels and information about the degree of agreement among the models. The dataset has been made publicly available on Hugging Face and can support further research into automated text analysis. The proposed approach demonstrates how combining multiple language models can facilitate the creation of large annotated datasets while improving the reproducibility and reliability of the annotation process.
The paper was authored by Włodzimierz Lewoniewski, Milena Stróżyna, and Izabela Czumałowska. It was published in the proceedings volume “Findings of the Association for Computational Linguistics: EMNLP 2026”. The full text is available through the ACL Anthology digital library. DOI: 10.18653/2026.findings-emnlp.1830
EMNLP is an annual conference organized by SIGDAT, a special interest group of the Association for Computational Linguistics (ACL). It is widely recognized as one of the most prestigious scientific conferences in natural language processing and holds the highest A* rating in the ICORE (formerly CORE) conference rankings. The 2026 edition was held at the HUNGEXPO Budapest Congress & Exhibition Center.
