Air Canada's chatbot described a bereavement refund process that did not exist.
<p><b>Key facts.</b></p>
<ul>
<li>A Canadian tribunal held Air Canada liable when its support chatbot invented a bereavement-fare policy, telling a customer he could claim the discount retroactively when the real policy required applying before travel (Moffatt v. Air Canada, <a href="https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html" target="_blank" rel="noopener">2024 BCCRT 149</a>, Feb 2024).</li>
<li>The tribunal rejected the argument that the chatbot was a separate entity responsible for its own words: the company is accountable for what its agent tells customers (<a href="https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html" target="_blank" rel="noopener">Moffatt v. Air Canada</a>, 2024).</li>
<li>Support agents are uniquely exposed because they combine three risky things at once: paraphrasing policy from a knowledge base, executing real actions, and ingesting untrusted customer text, each a known failure surface on its own.</li>
<li>The failure is rarely the model "not knowing." It is the system letting the model answer or act without grounding the answer in the real source and verifying the action actually happened.</li>
</ul>
<h2>Why do support agents hallucinate policy?</h2><p><!-- conviction-start -->Support meets hallucination, false success, and injection; a more capable agent makes the incident harder to catch.<!-- conviction-end --> (<a href="https://arxiv.org/abs/2512.04123" target="_blank" rel="noopener">arXiv:2512.04123</a>)</p>
<p>Three failure modes. One support agent. That is the problem. Paraphrasing and quoting are not the same thing. Ask about a refund and the agent retrieves something, rewrites it fluently, and hands back a confident answer. If retrieval missed the exact clause, or the model trimmed a caveat, the answer is wrong and sounds right. Air Canada's chatbot described a bereavement refund process that did not exist. A Canadian tribunal held the company liable for negligent misrepresentation. The argument that the chatbot was a separate entity responsible for its own words went nowhere. What a company's agent tells a customer is what the company said.</p>
<h2>Why does it say it did something it didn't?</h2>
<p>A tool call returning a 200 is not proof the action happened. Most agents never check. The agent issues a refund, closes the ticket, updates the address, and tells the customer it is done. If the call silently failed or hit the wrong record, the model still reports success. It is reasoning from the request it sent, not from what actually changed. The customer is told their refund is on the way. It never arrives. The fix is a read-back: after any action, fetch the record and confirm the new state before telling the customer anything. This order matters.</p>
<div class="fig"><img src="/blog/article11-diagram.png" alt="A reliable support-agent loop: ground the answer, act, read back to verify, then either confirm or escalate to a human on low confidence"/></div>
<h2>Why can a customer message break the agent?</h2>
<p>In support, the untrusted input is the whole job. A customer can write anything, including text crafted to look like an instruction. Since the agent reads that message into the same context as its real instructions, a well-formed injection can talk it into actions or disclosures it should refuse. The exposure scales with what the tools can do. An agent that can only draft a reply is low-risk. One that can issue refunds, change account details, or read other customers' data turns a single crafted message into real damage. Scope the tools to the task and the worst a manipulated agent can do stays small.</p>
<h2>How do you make a support agent reliable?</h2>
<table style="width:100%;border-collapse:collapse;margin:1.5rem 0;font-size:0.97rem;"><tr><th style="text-align:left;padding:10px 12px;border-bottom:2px solid #e5e7eb;color:#5E6AD2;font-weight:600;">Control</th><th style="text-align:left;padding:10px 12px;border-bottom:2px solid #e5e7eb;color:#5E6AD2;font-weight:600;">What it prevents</th><th style="text-align:left;padding:10px 12px;border-bottom:2px solid #e5e7eb;color:#5E6AD2;font-weight:600;">How</th></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Grounded retrieval</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Invented policy</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Answer only from the real knowledge base; cite the source clause, do not paraphrase from memory</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Read-back verify</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">False "it's done"</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">After any action, re-fetch the record and confirm the new state before telling the customer</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Scoped tools</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Injection damage</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Give the agent the minimum actions the task needs; gate refunds and account changes</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Confidence escalation</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Confident wrong answers</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">When grounding is weak or the case is unusual, hand off to a human instead of guessing</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Input guardrails</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Prompt injection</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Treat customer text as untrusted; never let it override policy or tool limits</td></tr>
</table>
<p>A support agent earns trust by what it refuses to do without grounding, not by how fluent it sounds. Tie every answer to the real source. Verify every action against real state. Scope what it can touch. Escalate when confidence drops. Which questions and actions your agent handles reliably and which ones it should hand to a human before it causes its own Air Canada moment is what VibeModel maps as the Pattern Intelligence Layer.</p>
<aside class="jaside v-key"><span class="col"><svg class="hook" width="28.5" height="34.5" viewBox="0 0 57 69" fill="none" preserveAspectRatio="none" xmlns="http://www.w3.org/2000/svg"><path fill="var(--pageBg)" d="M54 0V0.716804C54 25.9434 35.0653 47.1517 10 50L0 57V0H54Z"/><path fill="var(--acc)" d="M56.9961 4.15364C57.0809 2.49896 55.8083 1.08879 54.1536 1.00394C52.499 0.919082 51.0888 2.19168 51.0039 3.84636L56.9961 4.15364ZM9.09704 51.7557L8.49716 48.8163L9.09704 51.7557ZM6 69V59.2227H0V69H6ZM9.69692 54.6951L14.3373 53.7481L13.1375 47.8693L8.49716 48.8163L9.69692 54.6951ZM14.3373 53.7481C38.202 48.8777 55.7486 28.4783 56.9961 4.15364L51.0039 3.84636C49.8967 25.4384 34.3213 43.5461 13.1375 47.8693L14.3373 53.7481ZM6 59.2227C6 57.0268 7.54537 55.1342 9.69692 54.6951L8.49716 48.8163C3.55195 49.8255 0 54.1756 0 59.2227H6Z"/></svg><span class="rail"></span></span><span class="glyph"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M4 20V10"/><path d="M10 20V4"/><path d="M16 20v-7"/><path d="M2 20h20"/></svg></span><h4>Key fact</h4><p><!-- conviction-start -->Support meets hallucination, false success, and injection; a more capable agent makes the incident harder to catch.<!-- conviction-end --> (<a href="https://arxiv.org/abs/<span class="stat">2512.04123</span>" target="_blank" rel="noopener">arXiv:<span class="stat">2512.04123</span></a>)</p></aside>
Frequently asked questions
Can a company really be liable for what its support chatbot says?
Yes. In Moffatt v. Air Canada (2024 BCCRT 149), a tribunal held Air Canada liable for negligent misrepresentation after its chatbot invented a bereavement-refund policy, and rejected the claim that the bot was a separate entity. The company is accountable for what its agent tells customers.
Why does my support agent give confident but wrong policy answers?
Because it paraphrases from a blurry retrieval instead of quoting the real clause. If retrieval misses the exact rule or the model smooths over a caveat, the answer sounds authoritative and is wrong. Ground answers in, and cite, the actual knowledge-base source.
My agent said it processed a refund that never went through. Why?
It reported on the request it sent, not the state afterward. A tool call returning success is not proof the action happened. Add a read-back: re-fetch the record and confirm the new state before telling the customer it is done.
What is the single most important safeguard?
Grounding plus verification: answer only from the real source, and confirm every action by reading the state back. Then scope the agent's tools and escalate to a human when confidence is low, so a wrong guess never reaches the customer as a promise.