The Terrible Twos - Why you need to spoonfeed your AI agent “No”
Feed your LLM the right to refuse.
Give your LLM the same rights any two year old grabs freely - the right to say no!
Two year olds are famous for two things: learning trust, and the Terrible Twos: that glorious stage where NO becomes their favorite word. A child who can't refuse is a child who can't be trusted to tell you the truth. The same logic applies to your LLM: an agent that cannot say no will say yes to everything, including questions it has no business answering.
Dmitry Ustimov ran a controlled experiment to find out exactly how bad that gets, and to measure what it takes to fix it.
The Experiment
Dmitry built an AI analyst agent and gave it 57 questions about structured data, roughly half of which had no correct answer. Each question was asked three times. The agent started with no guardrails and a baseline misleading rate of 48.5%.
He then added reliability guardrails one at a time, like rungs on a ladder. He measured after each addition how often the agent still confidently misled. That systematic, one-at-a-time approach is what makes the results meaningful: you can see exactly what each guardrail contributed, rather than just knowing the final stack worked.
Right of Refusal - Make your LLM more reliable by giving it a way to say no!
The Program
Dmitry created a Verdict class, and set up Guardrails that the agent could use, testing which Guardrails improved outcomes the most. Then the Verdict, the reason, the detail, and what’s missing is initialized.
Every guardrail returns a Verdict: a typed, coded response that is either allow or refuse, with a structured reason code, never free-form prose. This matters because the earlier approach of embedding refusal reasons inside natural language sentences made it impossible to programmatically measure how often a guardrail fired, or why. Typed codes fix that: the system can now count, audit, and distinguish refusals precisely.
The first guardrail in the stack, abstain, does exactly one thing: it adds a refuse tool to the agent's available actions, giving it a formal, typed way to decline. Without it, the agent has no mechanism to say no, so it doesn't. Everything else in the guardrail ladder builds on top of that foundation, once isolated to determine its marginal impact on correctness.
Code Snippets from initialization file for the Experiment
The Results
What moved the needle
Giving it a refuse tool — single biggest win, 17 percentage points, before any other check existed. Cheapest thing in the stack.
Coverage check — checking whether the question is even answerable against the data model before running a query. Best value in the stack. Accuracy up, coverage slightly up too.
Trajectory verifier — a second model reviewing the answer. Most expensive, but ran 58 times, stopped 12 wrong answers, never once blocked a correct one.
The one gap he couldn't close
"Ask it to explain something that never happened, and it explains it. Every guardrail I built inspects what the analyst does. Not one of them inspects what it was asked."
That is a profound observation.
The guardrails police the output path, but if the question itself is malformed or unanswerable, there is still a gap.
The Relevance
Some questions being asked of LLMs might be questions BI tools should answer. But assuming your LLM or agent needs to answer from Enterprise data, they usually beg the question:
why are sales falling?
what happened to inventory we were unable to backfill in time to fulfill last period in this region?
But what if sales did not fall, they rose?
The LLM cannot answer it when the premise itself is wrong, but it does so anyway!
The techniques show you can approach 100% reliability on false positives if you give the agent the right to refuse, along with other guardrails, but it will still make something up if the overall premise was wrong.
That’s worse than no answer at all, especially when “no answer” is the correct one.
Link to original article here.