Alex Bravo Serrano TNA's experience in GATE, Sheffield, UK

As the amount of user generated content in social media is increasing at an exponential pace, detecting offensive language and harmful content automatically in an efficient way is a very important issue for the society. The offensive language can be expressed in various forms such as hate speech, cyberbullying, profanity and harassment, which has negative effects especially in adolescents.

Previously, in the Pompeu Fabra University (Barcelona, Spain), I was involved in several projects related to sentiment analysis and some of them related to abuse language detection. In the University of Sheffield, I had the opportunity to collaborate on a project on detecting offensive language. Specifically, this project was used to explore a collection of abusive replies to tweets by UK politicians in the run-up to the 2015 and 2017 UK general elections. This work shown that the abuse directed at politicians has increased in recent years, both in volume and proportionally, despite Twitter’s greater activity in banning abusive use.

During my visit to Sheffield, I learned more about this work and they helped me a lot to understand how their tool was implemented in GATE. The aim of this project is to adapt the abuse detection algorithms developed by Sheffield for English, to detect abuse in tweets in Spanish. To the best of our knowledge, this will be the first attempt at developing a political abuse detection system, which will enable the quantitative longitudinal study of online abuse towards Spanish politicians.

Firstly, the code of the tool has been reviewed to understand all the mechanisms and components involved in abuse detection in English tweets. Then, the state of the art in sentiment analysis in social networks in Spanish has been explored in order to identify data and approaches. Specifically, the project has focused on two relevant shared task: MEX-E3A Aggressiveness detection track 2019 and HatEVAL 2019.

The first task follows up on last year's evaluation task; it focuses on the detection of aggressive tweets in Mexican Spanish. It is important to mention that for this year the criteria for identifying aggression have been revised and a new data set has been created. This dataset was collected from tweets in Mexican Spanish. To select the set of terms that served as seeds for extracting the tweets, they used the words classified as vulgar & non-colloquial in the Diccionario de Mexicanismos de la Academia Mexicana de la Lengua, as well as words and hashtags identified by the Instituto Nacional de las Mujeres as related to violence and sexual harassment against women on Twitter. Unfortunately, these vocabularies are not available. For that reason I have contacted La Academia Mexicana, El Instituto Nacional de las Mujeres and even the organizers of the MEX-E3A task to obtain this glossary of terms. Today, I have not been answered to any email.

On the other hand, HatEVAL consists in Hate Speech detection in Twitter but featured by two specific different targets, immigrants and women, in a multilingual perspective, for Spanish and English. The task will be articulated around two related subtasks. The task a “Hate Speech Detection” against Immigrants and Women: a two-class classification where systems have to predict whether a tweet in English or in Spanish with a given target (women or immigrants) is hateful or not hateful. Adn the task b “Aggressive behavior and Target Classification”, where systems are asked first to classify hateful tweets for English and Spanish as aggressive or not aggressive, and second to identify the target harassed as individual or generic.

Previous datasets were used to collect offensive vocabulary in spanish. These vocabulary was organized into three groups: Sensitive Words (Homophobic, Nationalities, Religious, Sexist and Political), Offensive Words and Slurs. The vocabulary was enriched from multiple resources, such as Offensive words in Spanish from Hatebase, Glossary of Vulgar Terms and Insults in Spanish and Keyword set from HatEval task.

On my two-week visit, I did not have much time to fully adapt the tool, but I got good preliminary results. I evaluated the adapted tool against the test dataset from HatEVAL 2019, because the test dataset from MEX-E3A task does not include the corresponding labels. In the following table, the performance in terms of accuracy, precision, recall and F1 is shown.

In addition, I established guidelines for future work. Most offensive vocabulary collected in the adaptation is well written. Many people on twitter write with errors to highlight or by mistake. Words with errors should be included in the lists. For this reason, using FASTEXT word embeddings, we can extract similar words semantically and morphologically, which include words with mistakes.

In the “FastText Word Embeddings for Spanish Language Variations” website, we can find a repository containing word embeddings for several Spanish language variations created from Twitter data. These embeddings can be used to extract variations and synonyms from the lists of offensive words.

In addition, we hope to get an answer about the word list classified as vulgar & non-colloquial in the Diccionario de Mexicanismos de la Academia Mexicana de la Lengua, as well as words and hashtags identified by the Instituto Nacional de las Mujeres as related to violence and sexual harassment against women on Twitter. Adding this information can greatly enrich the vocabulary of our system. We also need to better explore the grammar of offensive expressions in Spanish tweets to enrich the grammar lists of our system.

The results reported in this project were achieved using English processing (such as Tokenization). Changing the preprocessing for components compatible with Spanish would improve the performance of the tool.

Finally, I will be available to participate in the next MEX-E3A Aggressiveness dDtection track 2020.