- 18 production bugs in a guest responder for 3 short-term rental properties, every one from a message I hadn't thought to test for. (first-person account)
- Automated testing techniques "fail to systematically exercise the conversational space of chatbots," per a 2026 study at the International Conference on Mining Software Repositories. arXiv:2602.13072
- Test case generation for conversational chatbots is "still limited," with existing tools producing scenarios too simple to cover real production interactions. arXiv:2503.05561
- LLMs drop substantially in multi-turn conversations because of "structural ambiguity in conversational context," and researchers confirm "scaling model size or improving training alone cannot resolve this gap." arXiv:2602.07338
- The pattern across all eighteen: the fix rule only became obvious after the bug happened. There was no way to write it before that.
What the happy path doesn't show you
I tested pretty thoroughly. WiFi password, check-in time, parking, directions, house rules. Every message category I'd seen guests send across two years of hosting. Sixty scenarios, maybe seventy. Ran them all. Nothing broke.
First real guest sent something I hadn't seen. Second one too. A week in, I had nine failures written down, and not one of them looked like anything in my test file. They weren't obscure edge cases. They were just the first nine things actual guests typed.
You can't cover this upfront. Just the question "which floor is the apartment on?" has at least forty phrasings, and three of them collide with keywords from unrelated intents. I had four variants in my test suite. Four. I thought that was enough coverage.
Researchers at MSR said it plainly: traditional testing techniques fail to systematically exercise the conversational space of chatbots. The test suite captures what the developer could imagine sitting at a keyboard. Production doesn't care what I could imagine.
The word that meant two completely different things
Bug five. Keyword collision. The word "available" sat in two separate intents: dates available to book, and facilities available for use. Lift, parking bay, geyser. Both real guest questions. Both use the same word. I built each intent separately and never saw the collision until it happened in production.
Guest asked "is the lift available?" My classifier saw "available" and fired the date-availability flow. Sent them a check-in greeting with parking instructions. The guest wanted to know if the building had a working lift.
Fix took ten minutes once I understood it: if "available" appears next to an amenity word, it's not a booking question. But I couldn't have written that rule before that guest's message. The rule came from the message. That was true for every fix on the list. The guest told me what the test should have been.
The inputs that didn't come from the guest
Some bugs weren't about what guests wrote. They were about what the messaging platform injected into the thread alongside the guest's words, text my scraper read as if a person had typed it.
Guest unsends a message, the platform appends "Message unsent" as plain text, inline, in the thread. Not metadata. Not a flag. Just text. My scraper had no rule for it. A guest who unsent something and then followed up got ignored on the follow-up, because my classifier hit the unsent intent the moment it saw that string, even with real content after it.
The second one was worse because I kept looking in the wrong place. The platform doesn't tag outgoing host replies in the thread. It just shows the text. My parser grouped my own previous answers with the incoming guest messages. Guest sent "thanks for the response" and my classifier looked back through what it thought was the guest's message history, found unanswered questions from my own earlier replies, and sent the full answer again.
Three days to find that one. I checked the classifier logic. Checked the reply builder. The problem was upstream, in the parser treating my outgoing messages as incoming guest text. Platform behavior with no documentation and no way to spot it except live traffic.
Why you can't write the tests before the bugs
Keyword collision, platform artifact bleed, context poisoning. Same root cause across all three. Invisible before launch, obvious after. No pre-launch testing catches them because the test suite comes from the developer's mental model, and the developer's mental model has exactly the gaps the bugs fall through.
A 2025 study on automated chatbot test generation found this directly: test case generation for conversational chatbots is still limited, scenarios generated are too simple for real interactions. A 2026 follow-up confirmed it still: "simplicity of the generated test scenarios and weakness in implemented oracles." The tools generate what's easy to generate, not what production sends.
Multi-turn makes it worse. Research on LLM performance across conversation turns found substantial performance drop in multi-turn settings from "structural ambiguity in conversational context." Specific finding worth noting: scaling the model doesn't close that gap. Improving training doesn't either. The ambiguity lives in the accumulated context, not in the model's capability. Swapping my keyword classifier for a frontier model wouldn't have caught the keyword collision. It would've inherited the same gap.
| Bug class | Why it happened | What the fix required |
|---|---|---|
| Keyword collision | "available" matched both dates and amenities | Amenity-exclusion check before date-availability classifier |
| Context poisoning | Bot's outgoing replies grouped into guest message run | Speaker-labeling: any block with host name reclassified as host |
| Platform artifact bleed | "Message unsent" read as guest text | Strip artifact first; only return unsent if nothing follows |
| Acknowledgment replay | "Thanks" re-triggered full answer from earlier intents | Ack guard + dedup ledger tracking already-answered topics |
| Over-answering | Reply builder pulled from fixed topic list, not detected intents | Combined reply built from intents in that message only |
| Silent crash | Unhandled socket error killed the process, no alert | Error listener added; uncaught handler fires alert before exit |
| Wrong input field | Generic selector hit host-notes field, not message composer | Hard-coded platform-specific test-id with aria-label guard |
The responder is stable now. That word means one thing: it handles every message that's already arrived. Next guest who types something I haven't seen yet finds the next gap. That's how rule-per-intent systems age. Every fix is one rule added after one real failure. The list of what's already gone wrong grows. Production doesn't wait for the list to catch up.
The Pattern Intelligence Layer works at a different level. Not one rule per bug. It models the shape of failures across a population of properties and conversations, so when a new input matches the structural pattern of a known failure class, it catches it before that specific input has ever been seen. Every conversation adds signal. The system gets more reliable as it runs, not as rules accumulate.