Home / [French Chapter] Deploying an Accurate Classifier to Stop Online Violence Against Children using NLP

Local Project

[French Chapter] Deploying an Accurate Classifier to Stop Online Violence Against Children using NLP

Local Chapter Marseille, France Chapter

Coordinated byFrance ,

Alexandre Lang

Status: Completed

Project Duration: 03 Dec 2022 - 17 Jan 2023

Open Source resources available from this project

Github Repository

Project background.

According to a recent French poll by Audirep, more than 1 in 5 children are the subject of cyberbullying on social media, especially on Instagram and Twitter. Furthermore, the most vulnerable are young adolescents from 10-13, who already are prone to be the subject of bullying. Some members of the governments have tried to come up with solutions but to no avail. At the same time, more than 80% of parents believe that their kids should be protected from online bullying, and find it regrettable that no suitable solutions have yet to be found.

To remedy this problem, we will be creating an NLP classifier to detect whether an online post is harmful to children.

The link to the study: https://www.e-enfance.org/wp-content/uploads/2021/10/Infographie_Caisse-Epargne-e-Enfance-2021.pdf

The problem.

The project is designed to reduce Online Sexual Exploitation and Abuse of Children (OSEAC). With a 15,000% rise in online Child Sexual Abuse Materials (CSAM) online from 2005 to 2020, it is clear that online child violence is growing exponentially. In 2021, the National Center for Missing and Exploited Children’s CyberTipline received 29.3 million reports of CSAM, making 2021 the worst year on record for online child sexual abuse.

A primary way that adults with a sexual interest in children or those who wish to harm them in other ways are through online grooming. As described by Sørensen, 2015; Greijer et al., 2016,
“Grooming is a multidimensional phenomenon in which an adult aims to solicit a child into a seemingly voluntary interaction with the intention of sexually abusing that child.” In a study Save the Children published last year, Grooming in the Eyes of a Child (Juusola et al., 2021), we found that children who are the object of grooming often do not realize what is happening so they do not recognize they are in danger until they are being extorted into providing increasingly harmful imagery or even to meeting an online predator in person.

Project goals.

Our goal is to stop online violence against children by deploying an accurate classifier to identify grooming behavior in online chats with children. Once suspicion of grooming reaches a threshold based on its similarity to the training data, it will trigger an action, which may differ depending on the platform it is deployed on and the objectives of the intervention. As example, we may warn the child through the chatbot without alerting the groomer, call a moderator, or shut down the chat entirely.In 2020, Save the Children US collaborated with Omdena to address online violence ([https://omdena.com/projects/children-violence/](https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fomdena.com%2Fprojects%2Fchildren-violence%2F&data=05%7C01%7C%7C2c00781bc77a4e879aaa08dab1adbc56%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C638017657014767879%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FIKg%2BWCo9pOtaedgvXGT4794qDDfr7sXwMwUVMR0G1s%3D&reserved=0)). Of the various products that were generated from the sprint, the most promising was a classifier algorithm using Natural Language Programming to identify online grooming combined with a chatbot that can warn the children that they may be chatting with a groomer. Since then, a team of three engineers associated with the original project has continued to refine the technology. The core team now wants to expand on the work to build an industry-usable solution at scale.From [the original challenge](https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fomdena.com%2Fblog%2Fonline-predators%2F&data=05%7C01%7C%7C2c00781bc77a4e879aaa08dab1adbc56%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C638017657014767879%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2B%2BJ1m3ebopT6yIxq8s%2BcR%2FJyx847pBQIAfch61O18rA%3D&reserved=0), we have a large dataset of more than 800,000 lines taken from the [Perverted Justice](https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fperverted-justice.com%2F&data=05%7C01%7C%7C2c00781bc77a4e879aaa08dab1adbc56%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C638017657014767879%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BR7MULyKU08yqiFVE4Mxid0u50PecNm%2FXWfjF7WZL9s%3D&reserved=0) project, a project from 2003 to 2019 that used online volunteers as decoys to entrap predators that sought to contact minors to obtain sexual images or videos from them or to meet them in person. During the challenge and afterward, we tagged much of the training data with labels, such as male or female, predator or victim, and level of risk of the conversation, but the data still requires extensive processing, and in particular, we need to improve and systematize the way judge and annotate the level of risk. In addition to the data we already have, we are actively attempting to obtain additional databases of online grooming chats from a variety of sources, such as law enforcement agencies.1. Build on existing data and further annotate additional sentences. Target is to achieve 100,000 annotated sentences with risk levels (non-risky, potentially risky, or risky). If new data is made available by law enforcement, annotate that data.2. Look for and scrape user data from online resources.3. Create a language model [Classification] to detect grooming behavior by labeling it as non-risky, potentially risky, or risky.4. Test the data on various models and provide ablation studies.5. Deploy the system as an API.6. Make the API a stand-alone chrome extension that predicts labels in an impromptu manner [The Grammarly execution process is the best example to relate with the final deliverable]