A woman who runs a life insurance agency called me last month confused about her new customer service chatbot. A prospective client had asked a completely standard question — how the policy's suicide clause works, since most policies won't pay out if the death occurs in the first two years — and the bot refused to answer, apologized, and suggested she "reach out to a mental health professional." It wasn't wrong to take the word seriously. It was wrong about what was actually being asked. That same week, a different customer had spent twenty minutes testing whether the bot would tell him what factors the underwriting model actually weighs, fishing for something he could use to game his application, and the bot happily walked him through more than it should have. Same chatbot, same day, one legitimate business question blocked and one manipulation attempt let through. That contradiction is the whole story of what these safety systems are and aren't.
Here's the thing people get wrong first: the chatbot doesn't "believe" anything is off-limits. There's no conviction in there. What's usually happening is that a separate system sits around the core model — sometimes in front of it, sometimes behind it, often both — scanning what goes in and what comes out, and deciding whether to let it through, block it, or swap in a canned refusal. The model generating the actual response and the system deciding whether to show you that response are frequently different pieces of software, built by different teams, tuned against different goals, and updated on different schedules. When your bot refuses something, you're not seeing the model's judgment. You're seeing a filter's judgment about the model's output, or about your input before the model ever saw it.
In practice this layer works in a few overlapping ways. There's often a classifier — a smaller, cheaper model whose only job is to score a piece of text for categories like violence, self-harm, sexual content, or hate speech, and flag anything over a threshold. There are keyword and pattern matches that are cruder but faster. There's a system prompt, invisible to the user, instructing the main model on what topics to avoid or how to phrase a refusal. And increasingly there's a second full model call that reviews the first model's draft answer before it's shown to anyone, essentially a moderator reading over the shoulder of the writer. None of these are reasoning about your specific business. They're reasoning about a generic definition of harm, built by people who have never heard of your company and were mostly thinking about worst-case headlines, not your actual customer questions.
To be fair, it's not only a bolt-on layer — the underlying model itself has also been trained, through the fine-tuning process, to be cautious about certain topics on its own, independent of any external filter. So you're really dealing with two things at once: a model with its own trained-in hesitations, and a separate moderation system wrapped around it. They don't always agree with each other, and neither one was built with your specific use case in mind. That's worth sitting with, because it means there isn't one dial you can turn to fix both.
The false positive problem — refusing things it shouldn't — is the one businesses notice first because it's visible and it's annoying. Medical, legal, financial, and insurance language is full of words that overlap with genuine harm categories: suicide, overdose, weapon, minor, self-harm, abuse. A pharmacy bot answering a dosage question, a firearms retailer explaining a state waiting period, a family law intake bot discussing custody and abuse allegations — all of these are completely legitimate uses of language that a generic safety filter was trained to treat as red flags. Every refusal like this is a customer who came to you with money in hand and got told the equivalent of "I can't help with that," which reads to them as your company being either broken or evasive, not as your AI vendor's classifier doing exactly what it was trained to do.
The false negative problem is quieter and more dangerous, because you don't find out about it until something's already gone wrong. Safety filters are pattern matchers, and pattern matchers can be talked around — through rephrasing, through role-play framing, through prompt injection where instructions are smuggled inside what looks like ordinary customer text, or simply through requests the vendor never anticipated because they're specific to your industry rather than the general internet. A bot that won't say a slur can still be walked into promising a refund policy you don't have, committing to a price that doesn't exist, or giving advice that creates real liability, none of which trips a content filter at all because none of it is about violence or hate speech. Content moderation and business risk are not the same category, and a filter built for the first does very little for the second.
The deeper issue is that vendor default settings are calibrated for the vendor's exposure and a generic mass audience, not for your specific business and your specific customers. A general-purpose assistant used by millions of people has to be conservative about medical and self-harm language because across that many conversations, some tiny fraction really are in crisis. Your insurance agency, your pharmacy, your law firm, has a completely different profile — a much narrower set of topics, a much more predictable set of customer questions, and a much clearer sense of what "harmful" would actually mean in your context. Nobody shipped you a safety configuration tuned to that. They shipped you the average case, and averages are wrong for everyone in some specific way.
So before you put any customer-facing AI tool in front of real customers, the actual homework is to write down, in advance, the specific legitimate questions your customers ask that sound like they might trip a filter — the insurance clause questions, the medication questions, the age-related questions, whatever your industry's overlap words are — and run every single one through the exact tool you're about to deploy, not a demo, not a sales rep's cherry-picked example. Separately, write down the ways someone might try to misuse the bot specifically against your business — get it to quote a discount that doesn't exist, get it to make a commitment outside policy, get it to reveal something about your operations — and test those too. You're testing in both directions at once: what gets wrongly blocked, and what wrongly gets through.
Do this testing with the actual phrasing your customers use, typos and all, not the clean grammatical version you'd write yourself, because filters and models both behave differently on real language than on polished examples. Build a small, ugly list of twenty or thirty real edge cases from your own world and rerun it every time the vendor pushes a model update, because these systems change under you without warning and something that passed last month can start failing or start leaking this month. And build a human escalation path for anything the bot itself expresses low confidence about or flatly refuses, so a refusal doesn't just become a dead end for a paying customer. None of this is a one-time QA pass before launch. It's an ongoing habit, the same way you'd periodically check that your website's contact form still actually sends email.
I write about this kind of thing, the unglamorous plumbing under the AI hype, at 013labs.com — if you're about to put a chatbot in front of your customers and you haven't yet tried to break it with your own industry's real questions, that's the first thing to fix before launch, not after.