# How does AI moderation work?

> AI moderation uses language models to read messages and flag abuse, spam or rule breaks. Learn how it works, where it fails, and how to use it safely.

URL: https://www.jalkarna.in/based/learn/how-ai-moderation-works/  
Last updated: 2026-11-16

AI moderation uses a machine-learning model to read content and judge whether it breaks a rule, then flags it for a human or takes a limited action such as deleting it. Unlike a fixed word filter, it considers meaning and context, so it can catch abuse that avoids banned words. It is also fallible, which is why good setups keep humans in charge of serious decisions.

## How does AI moderation decide what to flag?

A typical flow has four steps:

1. **Collect:** the system gathers recent messages from the channels it is allowed to watch.
2. **Evaluate:** a language model is given the messages plus instructions describing what counts as a violation, such as clear harassment or disruptive spam.
3. **Score or label:** the model returns a decision for each message, often with a confidence level.
4. **Act or escalate:** the system either logs the result, asks a human to review it, or performs an action that the server owner has permitted.

Because the model reads language rather than matching a list, it can understand that a message is a threat even when no single word in it is banned. The same strength creates the main risk: it is making a probabilistic judgment, not a lookup.

## AI moderation vs keyword filters

| | Keyword or regex filter | AI moderation |
|---|---|---|
| Understands context | No | Partly |
| Catches misspellings and workarounds | Only if you list them | Often |
| Predictable | Very | Less so |
| Explains itself | Always (it matched X) | Only if designed to |
| Cost of mistakes | Over-blocking harmless words | Misjudging tone or sarcasm |

The two work well together. Cheap, exact rules handle known bad strings and links, and AI handles the fuzzy middle. See [how to stop spam and scam links on Discord](https://www.jalkarna.in/based/learn/how-to-stop-spam-and-scam-links-on-discord/) for the rule-based side.

## What are the limits of AI moderation?

- **False positives:** jokes, quotes, song lyrics, and reclaimed language can look abusive without being so.
- **False negatives:** coded language, new slang, and slow-burn harassment spread across many messages can slip through.
- **Missing context:** the model usually sees a short window, not your server's history or inside jokes.
- **Language and culture gaps:** performance is uneven across languages and dialects.
- **Overconfidence:** a model can sound sure and still be wrong, a behavior related to [AI hallucinations](https://www.jalkarna.in/based/learn/what-are-ai-hallucinations/).

These limits are the reason to start conservative, review the logs, and widen the settings gradually rather than switching everything to automatic punishment on day one.

## How should you deploy AI moderation safely?

1. Begin in a review-only mode where the AI flags but changes nothing.
2. Send results to a private staff channel and read them for a week or two.
3. Note what it gets wrong and adjust your policy or channel selection.
4. Allow the mildest action first, such as deleting a message, before considering timeouts or bans.
5. Keep a human appeal path and tell members what is automated.

Pair this with clear written rules, which give both the AI and your moderators something concrete to enforce. See [how to write Discord server rules](https://www.jalkarna.in/based/learn/how-to-write-discord-server-rules/) and the broader [content moderation overview](https://www.jalkarna.in/based/learn/what-is-content-moderation/).

## Doing this with Based

Based's automatic moderation module is off by default. When enabled, it reviews messages in the channels you select in batches (30 messages every 600 seconds by default) and looks for clear abuse. The default review policy is conservative, and an optional stricter policy also covers disruptive spam. The maximum action defaults to "Review only", which never changes messages or members; you can raise it to deleting messages, warn and delete, timeout, kick, or ban. A private log channel records what happened, and message excerpts in the log are off by default. Image review is optional.

There is also an Apps menu action called "Review message" that gives an advisory opinion on one message. Commands also work with the "." prefix, for example `.help`. See the [automod docs](https://www.jalkarna.in/based/docs/automod/) and the [AI automod feature page](https://www.jalkarna.in/based/features/ai-automod/).

## FAQ

### Is AI moderation accurate?

It is often good at obvious abuse and weaker at sarcasm, in-jokes, and coded language. Treat its output as a strong signal that a person should be able to review, not as a final verdict.

### Can AI moderation ban people automatically?

Only if the tool allows it and the server owner turns that on. A safer practice is to start with review-only or message deletion and escalate only after checking the logs.

### Does AI moderation read every message?

It only reads what it can access. Good tools let you limit it to selected channels and exclude private ones, and they should say how much they keep.

### Do I still need human moderators if I use AI?

Yes. People handle context, appeals, and fairness, and they are accountable to the community in a way software is not.

## Related
- [What is content moderation?](https://www.jalkarna.in/based/learn/what-is-content-moderation/)
- [Discord moderation best practices](https://www.jalkarna.in/based/learn/discord-moderation-best-practices/)
- [What are AI hallucinations?](https://www.jalkarna.in/based/learn/what-are-ai-hallucinations/)
- [How to stop spam and scam links on Discord](https://www.jalkarna.in/based/learn/how-to-stop-spam-and-scam-links-on-discord/)
- [How do Discord bots handle your data?](https://www.jalkarna.in/based/learn/how-discord-bots-handle-your-data/)
