Many existing datasets used for hate speech research mainly contain short messages with explicit slurs and insults. Davide Ceolin, leader of the Human-Centered Data Analytics group at Centrum Wiskunde & Informatica (CWI), finds this somewhat surprising. “Hate speech is often related to claims that may be false or misleading. These can reinforce the hateful message.”
CWI postdoctoral researcher Benjamin Ocampo and Ceolin encountered this issue when they set out to study hate speech detection together with colleagues from the University of Groningen. They wanted to investigate whether AI language models (LLMs) could support human moderators in countering online content for hate speech and misinformation. “To study the intersection of hate speech and misinformation, we needed data,” Ocampo explains. “That turned out to be a major challenge.”
First combined dataset
The researchers first tried to combine existing datasets on hate speech and misinformation. “But these datasets were not a good reflection of what happens in practice,” says Ocampo. They therefore built on an existing dataset containing real posts from a white-supremacist forum, adding information about which claims were worth checking. The result is one of the first datasets to combine information on hate speech and check-worthiness. By check-worthiness, the researchers mean a verifiable claim that merits checking because of its potential societal relevance or public interest.
“Based on the combined dataset, we developed a framework that combines human judgment with AI to detect claims that should be checked,” says Ceolin. He stresses that the AI model only determines whether a claim is worth checking, not whether it is misinformation.
This requires distinguishing facts from opinions. Because the boundary between the two can be unclear, multiple human annotators are normally involved in labelling research data. In the method developed by Ceolin and Ocampo, an LLM takes over part of this work. If the human annotator and the AI agree, the label is accepted. If they disagree, a second human annotator makes the final decision. “The final decision therefore remains in human hands,” says Ceolin.
Higher levels of hate and harassment
Tests of this approach showed that the AI language model and the human annotator agreed in a significant proportion of cases. “We then looked at how hate speech and factual claims are related,” Ocampo explains. “We found that hate speech posts containing check-worthy claims showed higher levels of hate and harassment on average.”
The researchers also wanted to know whether information about claims worth checking could also help an AI model detect hate speech more accurately. It could. “The AI language model takes as input the claims that are worth verifying. Giving the model this additional information results in more accurate hate speech detection for large AI language models,” says Ceolin.
Counter hate speech
The new dataset also opens up possibilities beyond this study, Ocampo says. “We now have the right elements to start experimenting with ways to counter hate speech. For example, an LLM could immediately respond to a post by saying: this is a claim that has been checked and found to be false.”
What is it like for researchers to read so much hateful content as part of their work? Ceolin and Ocampo shrug it off. “You shouldn’t take it personally. We know where it comes from,” says Ocampo. “And we are used to it because we do a lot of research on this,” Ceolin adds. “But these are things we do talk about within our group.”
More information
About the study: When Hate Meets Facts: LLMs-in-the-Loop for Check-worthiness Detection in Hate Speech
