This AI predicts online trolling before it happens

2025-04-27 19:08:11admin

How do you keep online trolls in check? Ban them? Require real names?

Dr. Srijan Kumar, a post-doctoral research fellow in computer science at Stanford University, is developing an AI that predicts online conflict. His research uses data science and machine learning to promote healthy online interactions and curb deception, misbehavior, and disinformation.

His work is currently deployed inside Indian e-commerce platform Flipkart, which uses it to spot fake reviewers. We spoke to Dr. Kumar ahead of a lecture on healthy online interactions at USC.

Dr. Kumar, how do you counteract online harassmentusing data science and machine learning? How does your system identify the trolls?In my research, I build data science and machine learning methods to address online misbehavior, which transpires as false information and malicious users. My methods have a dual purpose: first, to characterize their behavior, and second, to detect them before they damage other users. I have been able to investigate a wide variety of online misbehavior, including fraudulent reviews, hoaxes, online trolling, and multiple account abuse, among others.

How are you teaching the AI to spot these patterns?I develop statistical analysis, graph mining, embedding, and deep learning-based methods to characterize what normal behavior looks like, [and] use this to identify abnormal or malicious behavior. Oftentimes, we may also have known examples of malicious behavior, in which case I create supervised learning models where I use these examples as training data to identify similar malicious entities among the rest.

Mashable Light Speed Want more out-of-this world tech, space and science stories? Sign up for Mashable's weekly Light Speed newsletter. By clicking Sign Me Up, you confirm you are 16+ and agree to our Terms of Use and Privacy Policy. Thanks for signing up!

Your research is currently being used in Flipkart. What problem were they trying to solve and how are they measuring results?The key problem that I helped address on Flipkart was of identifying fake reviews and fake reviewers on their platform. This is a pervasive problem in all platforms; recent surveys estimate as much as 15 percent of online reviews [are] fake. It is therefore crucial to identify and weed out fake reviews, as our decision as consumers is influenced by them.

What's the method called here?My method, which is called REV2, uses the review graph of user-review-product to identify fraudsters [who] give high scoring ratings to low-quality products or low scoring ratings to high-quality products. REV2 [compares] our recommendations to previously identified cases of fake reviewers.

Is it possible for AI to keep an eye inside social networks and raise the alarm when bad behavior is about to arise? Is this purely pattern-based analysis with sentient data crunching or something entirely different?It is possible to proactively predict when something may go wrong by learning from previous such cases. For instance, in my recent research, I showed that it is possible to accurately predict when one community in the Reddit online platform will attack/harass/troll another. This phenomenon is called "brigading," and I showed that brigades reduce the future engagement in the attacked community. This is detrimental to the users and their interactions, which calls for methods to avoid them. Thus, I created a deep learning-based model that uses the text and community structure to predict, with high accuracy, if a community is going to attack another. Such models are of practical use, as it can alert the community moderators to keep an eye out for an incoming attack.

Do you see a logical extrapolation of your work used in "nudges" to prompt users to clean up their act prior to prosecution? Akin to a teacher at the front of the class keeping a wary eye on the troublemakers in the back row before they fall into criminal masterminded gangs?Absolutely! A natural and exciting follow-up work is how to discourage bad actors to do malicious acts and to encourage everyone to be benign. This will help us to create a healthy, collaborative, and more inclusive online ecosystem for everyone. There are many interesting challenges to achieve this goal, requiring new methods of interventions and better prediction models. Enabling better online conversations and nudging people to be their better self is going to be one of my key thrusts going forward.

Have you had personal experience with online harassment or was this more of an interesting AI problem to solve for you?One of the major reasons for me to follow this direction of research was seeing some of my friends being harassed by social media trolls. This led to look for non-algorithmic ways to curb this problem. Being a challenging task, it piqued the interest of the scientist inside me and I eventually learned to create data science and machine learning methods to help solve these problems.

You're collaborating on the $1.2 million DARPA-funded project "Active Social Engineering Defense," which continues until 2022. What has the agency asked you to prove? In this project, we are studying how malicious actors carry out social engineering attacks on unsuspecting victims. Social engineering attacks are very nuanced and complex personalized attacks, with the aim of compromising sensitive information. So the key questions we want to answer are: can we predict when a social engineering attack is happening; and how can we defend against them?

Finally, as a scientist, what really excites you about predicting human behavior? Do you feel we're getting closer to understanding what makes us "tick" at last?Human behavior is highly volatile and unpredictable, which makes it fun and challenging to predict. That being said, I do feel that AI is indeed getting better at understanding human behavior. To give one example, recommendation systems are significantly better than years ago at predicting what we want. However, one key piece of this puzzle that needs to be solved is to forecast how malicious entities will recreate themselves after being caught and banned. Thus, I am enthusiastic to build new machine learning and AI models to address this problem.

Featured Video For You

Scientists revive DNA from ancient woolly mammoth