Can artificial intelligence identify positive or negative sentiment in encyclopedic texts? Do different language models assess the same content in similar ways?These questions are explored in the research paper entitled “Evaluating Multilingual Sentiment Classifiers Using an LLM-Annotated Wikipedia Benchmark”. The paper was authored by Dr. Milena Stróżyna, Dr. Włodzimierz Lewoniewski, Izabela Czumałowska. The research was presented at ACL 2026 in San Diego, held from July 2 to 7, 2026.
Why Is Sentiment Analysis of Wikipedia Challenging?
Sentiment analysis is a technique in natural language processing (NLP) that aims to automatically determine whether a text expresses positive, negative, or neutral sentiment. It is widely used to analyze customer feedback, product reviews, online comments, and social media content.
However, applying sentiment analysis to Wikipedia presents particular challenges. Encyclopedic articles are expected to follow the principle of Neutral Point of View (NPOV), which emphasizes impartial presentation of information and the avoidance of unjustified subjective judgments. Consequently, positive or negative sentiment in such texts is often much less explicit than in content expressing personal opinions.
An additional challenge is distinguishing the sentiment conveyed by a statement from the nature of the events it describes. For example, reporting on a success, conflict, or disaster does not necessarily mean that the author is expressing a positive or negative attitude. Accurate interpretation requires consideration of context, wording, and subtle linguistic differences.
Five Languages and Three Large Language Models
The study examined Wikipedia content in five language editions: English, German, Spanish, Polish, and Russian. The research material was drawn from selected encyclopedic articles covering various fields of knowledge, including science, history, geography, and biographies. The analysis focused on the introductory sections of articles, which provide concise summaries of their main subjects.
In total, more than 75,000 sentences were collected and subjected to automated sentiment analysis. Three advanced large language models (LLMs) were used for this purpose: GPT-5.2 (OpenAI), Gemini 3.1 Pro (Google), and Claude Opus 4.6 (Anthropic).
Each LLM evaluated every sentence three times, resulting in nine evaluation runs per sentence. This approach made it possible to investigate not only whether different models assigned similar sentiment labels to the same sentences, but also whether individual models produced consistent responses across repeated evaluations of identical text. This distinction is important because even advanced AI systems do not always interpret the same content in the same way. Such differences may arise from the characteristics of individual models as well as ambiguities in the analyzed statements.
A Multilingual Benchmark for Model Evaluation
One of the main outcomes of the study is the development of a multilingual benchmark dataset that provides a reference for systematically comparing the results of sentiment analysis models.
The benchmark includes only those sentences for which all three large language models assigned exactly the same sentiment label across all nine evaluation runs. This strict selection procedure resulted in a dataset containing more than 55,000 sentences from five Wikipedia language editions.
Such a rigorous agreement criterion was intended to reduce the influence of random variations in model responses and improve the consistency of the reference dataset. However, agreement among models does not necessarily guarantee that their assessments are objectively correct. The resulting benchmark therefore serves as a reference for further research rather than a definitive measure of textual neutrality.
The authors subsequently used this dataset to compare two multilingual sentiment classifiers based on BERT-family architectures: modBERT and mulBERT. This comparison allowed them to examine how closely specialized classification models reproduce sentiment labels established through consensus among large language models.
What Did the Results Reveal?
The findings show that neutral sentiment predominates in most of the analyzed topic categories. This pattern is particularly evident in articles concerning science, technology, and other knowledge-oriented subjects, where the proportion of sentences classified as neutral frequently exceeded 90%.
At the same time, substantial differences were observed across topics. For example, in some biography-related categories, the proportion of sentences classified as positive reached or exceeded 45%. A relatively high proportion of positive sentiment was also identified in certain geography-related articles.
Significant differences were also observed between the evaluated classifiers. The modBERT model was more likely to assign positive or negative sentiment labels, whereas mulBERT showed a strong tendency to classify text as neutral. Although the latter approach often resulted in high percentage agreement with the benchmark, it also limited the model’s ability to distinguish more subtle differences in sentiment.
For example, for Russian-language sentences, modBERT achieved 23% agreement with the benchmark, compared with 82% for mulBERT. However, a high overall agreement percentage does not necessarily mean that a model performs equally well across all sentiment categories. When most sentences are neutral, a classifier that frequently predicts this category may achieve seemingly strong results while still struggling to identify positive and negative sentiment.
The authors also examined two methods for determining the overall sentiment of an article based on individual sentence-level predictions. The first method, simple majority voting, assigns equal importance to each sentence. The second method, weighted majority voting, takes sentence length into account when aggregating predictions. Comparing these approaches made it possible to investigate how different aggregation strategies affect sentiment classification at the article level.
Overall, the study demonstrates that reliable sentiment analysis requires consideration not only of the model and language used, but also of the subject matter and the methods applied to interpret and aggregate the results.
Research Data Made Publicly Available
Another important outcome of the study is the public release of the research data on Kaggle. The publicly available resources include sentiment analysis results for five Wikipedia language editions, covering labels from the benchmark and the evaluated models. They also contain data related to two methods of aggregating sentence-level predictions into article-level sentiment assessments. This enables other researchers to reuse the data to evaluate additional sentiment analysis models, compare results across languages and topics, and develop further methods for analyzing encyclopedic texts.
The publication contributes to research on multilingual natural language processing by highlighting both the capabilities of modern AI systems and the limitations of applying them to texts that require particularly careful interpretation. The findings also emphasize that automated sentiment analysis is not equivalent to a comprehensive assessment of an article’s neutrality. Nevertheless, it may serve as a useful tool for identifying passages that warrant closer examination.
The full paper is available as an open-access publication through ACL Anthology. DOI: 10.18653/v1/2026.gem-main.63