Data Mining & AI in Support of Online Hate Speech Detection
How to improve hate speech detection ?
How to untap the potential of Data Mining and AI to detect hate speech ?
As part of my master’s degree final project (TAL-IL DEFI, Paris Nanterre University), I contributed to the study Registres de langue et discours haineux sur internet (“Language Registers and Hate Speech on the Internet”). More specifically, the purpose of this study was to describe, with a view to automatically detecting them, extreme right-wing “hate” statements within texts published in French on the Web, taking into account the types of language register (casual, neutral, formal) they may belong to. This project, which lasted five months, required a precise positioning in relation to different notions such as “far-right wing” (we relied on Camus, 2020), “hate” (with reference to the Avia Act, 2019), “hate speech” (we chose to use the definition from Twitter) as well as “language register” (derived from Mekki et al., 2018)
This work led to the creation of a hate-speech corpus composed from a variety of sources and genres (literature, essays, press, political discourse, comments, forum, tweets) potentially associated with different registers: casual (C), neutral (N), formal (F). The corpus was automatically harvested and manually checked. It counts 893 exclusively hate texts averaging four sentences for each text.
We made a multi-label annotation. Each text can contain several registers in different proportions. Linguistic descriptors distributed by levels of language analysis (Mekki et al., 2017) guided us through the annotation as to establish for each text its probability to belong to a register. Finally, part of the corpus (seed) was manually annotated and balanced to obtain 100 texts per register. This seed was used for two main purposes: 1. extract emerging sequential patterns (Agrawal, Srikant, et al. 1995) both specific to hate speech and to the registers under consideration, 2. Incorporate new unannotated texts into the seed through a semi-supervised self-learning method (Mekki et al., 2020).
The extraction of emerging sequential patterns is done from a sequential database where each sentence constitutes a sequence and each word is represented by a set of linguistic features (its surface form, its lemma, its grammatical category and its syntactic function) This process brought out a number of sequences per register (C: 226, N: 301, F: 361). The cross-comparison of these sequences shows that we get the most emerging patterns when comparing S to F. On the other hand, C-to-S comparison is the least productive. Among the “hate” emerging patterns, we note the sequence [NAME+de] corresponding to “bande de X” (gang of X), “gueule de X” (X-head); or the ellipsis used to convey reticence or self-censorship as in “you can laugh at everything but the j…”.
Self-learning integration of new texts was carried out on sets of linguistic features (E1: linguistic, E2: word embeddings, E3: E1+E2) that were extracted beforehand from the entire corpus. Estimation is carried out using a 3-fold cross-validation method. Mean Squared Error during the learning sessions shows a decreasing curve from 0.15 to 0.05 for set E1. On the other sets of features this curve shows a decrease in the error rate which eventually increases when the evaluation test is performed on the validation sample. This result has room for improvement that can be achieved, among other things, by increasing the size of the seed.
This exploratory work relates to the ANR Astrid 2020-2022 project called FLYER. It was carried out by Sandy Duchemin, Aude Julien, Joy Devine and myself, and led by our supervisors from the MoDyCo laboratory (UMR 7114 CNRS Université Paris Nanterre), Delphine Battistelli, Jade Mekki and Aline Étienne whom we would like to thank. Special thanks to Mondeca for bringing visibility to this project.
——————
Agrawal R., Srikant R. (1995). « Mining sequential patterns ».
Camus J. Y., « Extrême droite », Encyclopædia Universalis [online], accessed Mars 28, 2020. URL : http://www.universalis.fr/encyclopedie/extreme-droite/
Mekki J., Béchet N., Lecorvé G., Battistelli D. (2020). « Corpus annoté automatiquement en proportions de registres de langue pour le français », en cours de soumission.
Mekki J., Battistelli D., Lecorvé G., Béchet N. (2018). « Identification de descripteurs pour la caractérisation de registres ».
Mekki J., Battistelli D., Béchet N., Lecorvé G. (2017). « “Nous nous arrachâmes promptement avec ma caisse” : quels descripteurs linguistiques caractérisent les registres de langue ? ».