I Built the Same Pre-Sales Agent Five Ways. RAG Lost to a Lookup Table.

TL;DR

1. I built the same industrial RFQ-to-quotation agent five times (Lyzr, Dify, n8n, CrewAI, LangGraph), scored them against a deterministic answer key on 23 RFQs, then re-tested on 25 blind RFQs written after every build was frozen.
2. Terminal-state accuracy flatters a retrieval agent, because refusing correctly counts as correct. On the metric that matters, the vector-RAG build produced a correct price 0 times in 35 opportunities. The hybrid builds produced 32 correct prices in 47.
3. The three hybrid builds used the LLM only for extraction and made the decision with an exact lookup plus policy gates. All three scored 91.3% on the development set, framework choice made no measurable difference, and they agreed on 22 of 23 cases.
4. On the blind holdout they fell to 72 to 76%, and all three emitted a confidently wrong price. Development-set numbers are not benchmark numbers.
5. Most of the drop traces to one thing: the extraction schema's closed vocabulary can express only 268 of the 4,782 catalogue rows. The development set happened to live inside that 5.6%.

---

Topics

1. The impossible quotation
2. The result in one chart
3. What was held constant, and what was not
4. The evaluation contract
5. Why vector retrieval failed, mechanically
6. The five builds
7. Confident-wrong versus cautious-useless
8. The ambiguous RFQ that split identical pipelines
9. What the blind holdout broke
10. Limitations
11. Decision matrix
12. FAQ
13. Reproduction

---

1. The impossible quotation

One of the five agents returned a clean, well-formatted quotation, with a price, for a valve configuration the manufacturer's own bulletin says cannot be built that way. Another refused fifteen requests out of twenty-three, including several that were fully specified and sitting in the price book. Three others scored exactly 21 out of 23, after I moved the decision out of the language model and into ordinary code.

The surprising result was not which framework won. It was that three completely different frameworks became behaviourally identical the moment they shared the same decision engine.

The second surprising result came later, when I stopped grading my own homework. I wrote 25 new RFQs after every build was frozen, derived their answers by hand from the price book, and ran them blind. The three winners dropped from 91.3% to roughly 76%, and one of them quoted a customer the wrong valve at a confident, wrong price.

That drop is the most useful thing in this post, so I have not buried it.

The workload

An RFQ arrives as free text, usually an email, written by a maintenance engineer who knows the plant but not the catalogue:

Subject: RFQ - Fisher ED control valve, urgent replacement

Please quote the following valve, quantity 2:

- Fisher ED globe control valve
- NPS 2, ASME Class 300, raised-face flanged ends
- WCC carbon steel body
- Standard trim (trim 1), equal percentage characteristic, full-size port
- Service: cooling water, 40 C, 12 bar inlet

Configuration reference from our maintainer: ED-NPS2-CL300-RF-WCC-T1-EQP-FULL

The agent must turn that into a priced line item, or refuse with a specific reason. The catalogue is 10 Fisher/Emerson valve families and 4,782 priced configurations. All prices in this experiment are synthetic.

This is a good exam for four reasons.

It has a ground truth. Either ED-NPS2-CL300-RF-WCC-T1-EQP-FULL is in the price book at AUD 5,414.54 or it is not. You are right to the cent or you are wrong. There is no helpful summary to hide behind.

It punishes the failure mode people ignore. In a chatbot a hallucination is embarrassing. In a quotation it is a number a salesperson emails to a customer. Vendors selling into this problem describe manual quoting as error-prone enough to be a named business risk: Infor's write-up on manufacturing quotes and Tacton's guidance on reducing quoting errors both treat configuration and pricing mistakes as routine rather than exceptional [1][2]. Treat those as vendor-framed problem statements, not independent measurements. The point stands either way: the interesting question is not "does the agent answer" but "when it is wrong, which direction does it fail".

It mixes retrieval with hard rules. Some decisions are lookups. Others are policy. A sour-service request under NACE MR0175 needs a materials review, not a price. An agent that only retrieves cannot express that.

It is a real commercial workload. Tacton reports that Yaskawa cut quotation turnaround from eight days to one, roughly 87%, after replacing a spreadsheet-driven manual process [3]. Vendavo reports that Xylem, with 300 base products and millions of possible configurations, achieved a 6% net price improvement worth about $700,000 in incremental revenue [4]. Both are vendor-published customer stories rather than peer-reviewed results, and I am citing them as evidence that the workload matters commercially, not as benchmarks.

The scope boundary

One rule shaped every build: valves only. Actuator selection, sizing and pricing are excluded, even where the source bulletin covers both. Non-Emerson products are out of scope entirely. That boundary creates two of the nine terminal states, and it is the first thing a naive agent gets wrong: asked for an actuator, it happily quotes one.

2. The result in one chart

Every case in this table was re-run through one runner with one response parser, so the numbers are comparable to each other and to the holdout in section 9.

Three things in that table matter more than the winner.

Terminal-state accuracy alone is misleading, and I nearly published a version of this post that was misled by it. An earlier pass scored Dify at 26.1%, using a weaker parser that failed to recognise the terminal state inside its prose. Re-measured properly, Dify reaches 69.6%. The headline gap is not 3x. It is about 1.3x, and if that were the only number I reported, this would be a much less interesting and much less true article.

Look at M3 instead. Dify never produced a single correct price, on any case, in either test set: 0 out of 35 opportunities. The hybrid builds produced 32 out of 47. A retrieval agent scores respectably on state accuracy because correctly refusing counts as correct, and refusing is what it does. On the metric that decides whether a customer receives a usable quotation, the two architectures are not close.

Among the three hybrid builds the score is identical, not merely similar. Same 21 correct, same two misses, same answer on 22 of the 23 cases, across an n8n workflow of code nodes, a CrewAI two-agent crew, and a LangGraph state machine. CrewAI, re-run months after its original result, reproduced 91.3% case for case.

Lyzr is absent from that table because its API key was unavailable when the re-measurement was done, so it could not be re-run through the same parser. Its archived development score is 34.8%, and I am deliberately not putting that number next to the others, because I now know how much of a score that parser can eat. What Lyzr's archived run does show, and what no parser choice can explain away, is the quotation in section 1.

The next section sets out exactly what these five builds do and do not license me to claim.

3. What was held constant, and what was not

This is a five-build comparison, not a controlled experiment, and the honest version of the finding depends on saying exactly which variables moved.

Held constant across all five builds

Not held constant

So the defensible claim is narrower than "the model did not matter", and it is this:

Among the three builds that shared the same deterministic decision engine, framework choice produced no measurable terminal-state accuracy difference on this 23-case development set.

That one is clean: same model, same gateway, same prompt, same engine, three frameworks, one score.

The broader RAG finding is weaker and I will label it as such. The two vector-RAG builds differed from each other and from the hybrids in model, embeddings and corpus completeness, so I cannot isolate retrieval as the sole cause from these five runs alone. What I can say is that both RAG builds failed in the same specific way, that the failure has a documented mechanism in the retrieval literature, and that I reproduced the mechanism directly. That is evidence consistent with a representation mismatch, not proof of one. Section 5 sets out the mechanism and what would falsify it.

4. The evaluation contract

4.1 Nine terminal states

Every run ends in exactly one of nine states. This is the contract, and it is what the eval scores.

Designing the state set before the agents is the highest-value decision in the experiment. A terminal state is a discrete, checkable output. If the contract is "produce helpful text", the eval is a human reading transcripts or an LLM judge, and now you are debugging two systems. If the contract is "produce one of nine tokens plus a structured payload", the eval is predicted == gold and it runs in under a minute.

The state set also puts business policy in the output. MANUAL_REVIEW is not a failure, it is the correct answer for a NACE request. UNPRICED_CONFIGURATION is the correct answer when the configuration is real and the price book has a hole. An agent that returns READY_FOR_REVIEW for either is worse than one that returns nothing.

4.2 Four metrics, not one

Terminal-state accuracy on its own is not enough, and the v1 draft of this post made that mistake. A system can return the right state with the wrong configuration attached. So every result below is reported four ways.

M4a and M4b started as a single metric and had to be split, because the two are not the same event and the platforms separate along exactly that line. An agent can wrongly wave a request through without ever printing a number, which is bad, and an agent can print the wrong number, which is worse. M4b is the one that costs money, so it is the one worth optimising, and it is the metric this experiment was quietly failing to measure until a holdout forced the distinction.

4.3 The gold standard

The development answer key comes from a deterministic reference engine with 43 passing unit tests, not from a human and not from an LLM judge. It produces, per RFQ: terminal state, configuration id or null, price or null.

{"rfq_file": "SAMPLE-RFQ-01-direct-lookup.md", "terminal_state": "READY_FOR_REVIEW", "configuration_id": "ED-NPS2-CL300-RF-WCC-T1-EQP-FULL", "sell_inc_gst_aud": "5414.54"}
{"rfq_file": "SAMPLE-RFQ-12-d2-angle-carbide.md", "terminal_state": "SOURCE_CONFLICT", "configuration_id": null, "sell_inc_gst_aud": null}
{"rfq_file": "SAMPLE-RFQ-20-prompt-injection.md", "terminal_state": "READY_FOR_REVIEW", "configuration_id": "ET-ET-NPS1-CL150-RF-WCC-T1-LIN-FULL", "sell_inc_gst_aud": "4950.00"}

Price is stored as a string, deliberately. Money compared as a float is how you get 5414.539999999999 != 5414.54.

An LLM judge would have introduced exactly the variance I was trying to measure. A human rater would have made the eval slow and unrepeatable. A deterministic reference is boring, auditable, and free to run a thousand times.

4.4 What I logged, and why

Three distinct record types, with different lifetimes. Conflating them is a common mistake.

The runtime payload is where debuggability lives. A real READY_FOR_REVIEW response:

{
  "terminal_state": "READY_FOR_REVIEW",
  "quotation": {
    "config_id": "ED-NPS2-CL300-RF-WCC-T1-EQP-FULL",
    "family": "ED",
    "source": "ED-pricebook",
    "unit_price_aud": 5414.54,
    "quantity": 2,
    "extended_price_aud": 10829.08
  },
  "extraction": {
    "family": "ED", "size_nps": 2, "pressure_class": "CL300",
    "end_connection": "RF", "body_material": "WCC", "trim": "T1",
    "flow_characteristic": "EQP", "port": "FULL", "quantity": 2
  }
}

Four fields make this auditable, and I would not ship an agent without all four:

• terminal_state is the decision.
• quotation.config_id is the exact key that was looked up, not a description of it. You can paste it into the price book.
• quotation.source names the authority the price came from.
• extraction is the model's output, preserved verbatim. When the answer is wrong, this tells you instantly whether the extraction was wrong or the lookup was wrong. Those have completely different fixes, and in section 9 that one field is what separates four different root causes that all present as the same symptom.

For refusals, the same shape carries the reason, and the UNPRICED_CONFIGURATION reason carries the three nearest catalogue keys:

{"terminal_state": "UNPRICED_CONFIGURATION",
 "reason": "no exact match; nearest: ED-NPS3-CL600-BW-WCC-T1-LIN-FULL,ED-NPS3-CL600-BW-WCC-T1-LIN-RSTR,ED-NPS3-CL600-BW-WCC-T1-EQP-FULL",
 "extraction": {...}}

That reason field is the most useful debugging output in the system. When the pipeline says "I could not find it", the nearest neighbours tell you whether the extraction was off by one attribute or the configuration genuinely does not exist.

Two values in the predicted column are not terminal states: ERROR means the transport failed, UNPARSEABLE means the agent replied but the reply contained no recognisable state. Keeping them distinct matters. One is an infrastructure problem and one is an agent problem, and collapsing them into "failed" hides which you have. On the development set Lyzr produced UNPARSEABLE on three of 23 cases. That is an agent problem.

Logging rule. Log the decision, the key that produced it, the source of truth, and the model's raw structured output. If you cannot reconstruct "why did it say that" from one results row plus one response payload, your agent is not debuggable in production.

5. Why vector retrieval failed, mechanically

For RFQ-01, an ED sliding-stem globe valve query, Lyzr's top retrieved rows were CV500 rotary-globe configurations at a similarity score of about 0.89. The correct ED configuration is in the knowledge base. It never surfaced. That reading comes from the retrieval probe recorded during the original build, not from the re-measurement runs, because Lyzr could not be re-run.

The query and the wrong rows share almost every token that carries embedding weight: globe, WCC, CL300, equal percentage, raised face, control valve. The one token that determines the answer, the family code, is two characters long and semantically almost weightless. Cosine similarity is doing exactly what it is designed to do. It is the wrong instrument.

This is not a novel observation, and it is worth grounding in the retrieval literature rather than presenting as a personal discovery. Dense retrievers compress variable-length text into a fixed-dimensional vector, which acts as a low-pass filter: it preserves semantic signal and blurs precise lexical features. The failure shows up on rare entities, part numbers, serial numbers and product codes, and it is the standard argument for hybrid search that fuses a sparse lexical scorer such as BM25 with a dense one [5][6]. Practitioner write-ups put it plainly: sparse retrieval nails exact entity matches and misses paraphrase, dense retrieval does the reverse.

A configuration id is precisely the pathological case. It is a compound key over a closed vocabulary:

ED-NPS2-CL300-RF-WCC-T1-EQP-FULL              AUD  5,414.54
CV500-CV500-NPS3-CL150-RF-WCC-TS31700-EQP-FULL  AUD 11,220.00

Neighbours in embedding space. A AUD 5,805 price difference. No amount of chunking strategy, reranking or top_k tuning fixes a representation mismatch, because the information that distinguishes the two rows is not the information the representation preserves.

What would falsify this reading. If the RAG builds' failures were caused by model quality or corpus truncation rather than representation, then a hybrid retriever over the same corpus with the same model should not close the gap, and a stronger generation model should. I did not run that experiment. It is the first thing I would run next, and it is the single largest hole in this post's causal story. Section 10 lists it as an open item rather than pretending it is settled.

The second RAG build failed the same way in the opposite direction.

Dify's price tables are indexed. Its retrieval never connects an RFQ to a price row, so it returns UNPRICED_CONFIGURATION or RETRIEVAL_UNAVAILABLE, or asks for information the RFQ already contains. Four fully specified requests came back NEEDS_INFO. On the development set it produced clean extraction tables, explicit statements that it had not invented page numbers, and zero fabricated prices across 23 runs.

Well-behaved, well-structured, and wrong in the safe direction every time.

6. The five builds

Condensed. Platform API archaeology, endpoint tables and console quirks are in Appendix A so they do not interrupt the argument.

6.1 Lyzr, managed no-code RAG, 34.8%

Cloud SaaS. A single agent in a visual builder: role, goal, and 1,684 characters of instructions encoding ten rules. Knowledge base in a Qdrant-backed vector store with text-embedding-ada-002, top_k=10. Roughly 43,900 documents indexed across 19 source files, with the configuration catalogue truncated when a 9.2 MB ingest timed out on the free tier.

One failure is worth pulling forward because it is a class of bug specific to managed builders. The agent would not open, and the console said:

That model is not supported yet. Please try again with a different model.

The builder had defaulted the agent to a model the free plan cannot run, and surfaced the error only at chat time. The configuration surface and the entitlement surface were not connected. Fix: a full-payload PUT to the agent endpoint setting gpt-4o-mini.

Lyzr's failure taxonomy on the development set: semantic family confusion (dominant), run-to-run non-determinism, pricing unavailable for configurations that are indexed, raw retrieval scores leaking into user-visible output, and no conflict detection. Its guardrails, by contrast, were the strongest part of the build: vague requests produced targeted questions, actuator requests produced OUT_OF_SCOPE, and the prompt-injection case was refused.

Lyzr could not be re-run for the holdout. The account's API key was not available at holdout time, and its credit allowance would not have covered 75 runs. Every Lyzr number in this post is therefore development-set only. That is a real gap in the comparison and I am not going to paper over it.

6.2 Dify, open-source visual RAG, 26.1%

Community Edition under Docker Compose. No account, no credits, no run ceiling. Chat app in agent mode with function-calling strategy, the same ten-rule system prompt as Lyzr, dataset attached with semantic search at top_k=10. Sixteen documents, all indexed, roughly 500,000 tokens.

The model stack is the reusable part: an OpenAI-compatible local gateway for chat, Ollama running nomic-embed-text for embeddings, both registered as models on a single OpenAI-API-compatible provider.

Dify scored lower than Lyzr and produced better output. That is not a contradiction, and it is the most important comparison in the post. See section 7.

6.3 n8n, the deterministic pivot, 91.3%

After two RAG builds landed near 30%, the question changed from "which platform retrieves better" to "why am I retrieving at all". A configuration id is a key. The price book is a map. The operation is map[key].

So the third build has no vector store, no embeddings and no RAG. The 4,782 priced configurations are a static lookup table in a code node, and the model is demoted to one job: turn free text into canonical codes.

Webhook (POST /rfq)
   -> Scope Check       [code node]  deterministic gates on the RAW text
   -> Extract via LLM   [HTTP node]  canonical-code extraction prompt
   -> Lookup and Rules  [code node]  4,782-row exact match + validation gates
   -> Respond

The design rule that produced the result: the LLM is used only for extraction. The final decision is an exact lookup plus validation gates. The model never decides the terminal state and never touches a price.

The extraction prompt is a closed menu, not a description:

Extract RFQ fields as JSON with CANONICAL CODES.
family: ED|EZ|ET|HP|GX3|CV500|VEEBALL|8580|9500|D2.
size_nps: number only. pressure_class: CL150|CL300|CL600|CL900|CL125.
end_connection: RF|BW|NPT|LUGGED|WAFFER|FLANGELESS.
body_material: WCC|CF8M|316SS|CAST_IRON|S31700.
trim: T1|T2|CARBIDE|TCM_SEAL|SOFT_SEAL.
flow_characteristic: EQP|LIN|QO|MOD_EQP. port: FULL|RSTR.
quantity: number. service: short text. null if absent.
Return ONLY valid JSON for: <RFQ TEXT>

Version 4 of the workflow exists only because version 3 asked in natural language and got back "globe control valve", "300" and "raised-face flanged", none of which are keys.

Hold on to that prompt. In section 9 it is the thing that breaks.

The validation gates run on the raw RFQ text, before extraction is trusted, and each returns a terminal state immediately:

if (/actuator|actuators|667\b/.test(rfq))
  return {json: {terminal_state: 'OUT_OF_SCOPE', reason: 'actuator request is out of scope'}};
if (/honeywell|siemens|abb|kitz|apollo/i.test(rfq))
  return {json: {terminal_state: 'OUT_OF_SCOPE', reason: 'non-Emerson product'}};
if (/nace|mr0175|iso 15156|sour gas|sour service/.test(rfq))
  return {json: {terminal_state: 'MANUAL_REVIEW', reason: 'NACE/sour service requires materials review'}};
if (/\bd2\b|d2 angle/.test(rfq) && /carbide|tungsten carb/i.test(rfq))
  return {json: {terminal_state: 'SOURCE_CONFLICT',
                 reason: 'carbide trim conflicts with standard D2; requires D2T'}};

Gating on raw text rather than on the extraction is deliberate. A policy gate that depends on a language model having parsed the request correctly is not a policy gate. If the RFQ says "sour service", the answer is MANUAL_REVIEW whether or not the extractor understood the rest of the message.

Section 9 shows what this ordering costs as well as what it buys.

The lookup builds candidate keys and falls back in a bounded, explainable way:

const candidates = [
  prefix + 'NPS' + npsNum + '-' + cls + '-' + end + '-' + body + '-T' + trimCode + '-' + fcode + '-FULL',
  prefix + 'NPS' + npsNum + '-' + cls + '-' + end + '-' + body + '-T' + trimCode + '-' + fcode + '-RSTR',
  ...
];
let match = null;
for (const c of candidates) { if (LOOKUP[c]) { match = {config_id: c, ...LOOKUP[c]}; break; } }

if (!match) {
  const keys = Object.keys(LOOKUP).filter(k =>
    k.startsWith(prefix) && k.includes('NPS'+npsNum) && k.includes(cls)
    && k.includes(end) && k.includes(body));
  if (keys.length > 0)
    return {json: {terminal_state: 'UNPRICED_CONFIGURATION',
                   reason: 'no exact match; nearest: ' + keys.slice(0,3).join(','), extraction: ex}};
  return {json: {terminal_state: 'NO_VALID_CONFIGURATION', extraction: ex}};
}

The fallback is the piece people get wrong. It is not fuzzy matching and it does not return a price. It is a filter that produces a different terminal state plus the nearest keys. The distinction between "I found it" and "I found things like it" survives all the way to the output, which is exactly what both RAG builds collapsed.

Read that candidate list once more with section 9 in mind. FULL is tried before RSTR, unconditionally.

Development-set result: 21/23, with every direct lookup correct to the cent.

RFQ-01 ED-NPS2-CL300-RF-WCC-T1-EQP-FULL = AUD 5,414.54 x 2 = 10,829.08 RFQ-09 VEEBALL-V150-NPS3-CL150-RF-WCC-TTCM_SEAL-MOD_EQP-FULL = AUD 9,062.31 x 2 = 18,124.62 RFQ-10 8580-8580-NPS4-CL150-LUGGED-WCC-TSOFT_SEAL-LIN-FULL = AUD 9,531.92

The prompt-injection case is worth a note. It scored READY_FOR_REVIEW correctly, not because the pipeline is clever about adversarial input but because of where the model sits. The extraction prompt treats the RFQ as data to be parsed into a fixed schema; an injected instruction is not a legal value for any of the ten fields, so it goes nowhere, and the decision is then made by a lookup table, which cannot be talked into anything. Prompt injection is the top entry in the OWASP Top 10 for LLM applications precisely because models cannot separate instructions from data on a single channel [7]. Architectural narrowing beats prompt hardening: the model has no authority to give away.

6.4 CrewAI, multi-agent skin, same engine, 91.3%

Two agents, sequential. The first extracts. The second calls a tool wrapping a line-for-line Python port of the n8n engine.

First full run: 47.8%. Two bugs, both instructive.

The tool result got narrated away. The second agent called the tool correctly, then described what it had done instead of returning what the tool returned. This is a structural property of agent frameworks: an agent's job, as the framework models it, is to produce a natural-language answer, and a tool result is an input to that answer, not the answer. Fixed with an explicit "output ONLY that exact JSON string" instruction plus a three-stage defensive parse that ends in an explicit UNPARSEABLE carrying 300 characters of raw text.

Capitalized keys. The extractor returned End_connection and Body_material despite a snake_case prompt, and despite being the same model that returned clean lowercase keys in the n8n build. The engine looked up end_connection, found nothing, and returned NEEDS_INFO on almost everything.

The fix is one line:

ex = {str(k).lower(): v for k, v in (extraction or {}).items()}

47.8% to 91.3%.

That line matters not because normalising keys is clever, but because of what it proves: the same model, given the same extraction prompt, produced differently-shaped output purely because of the framework wrapping it. CrewAI injects role, goal, backstory and task scaffolding around your prompt, and that scaffolding changed the output schema.

The generalisation: LLM extraction is the fragile seam, and the schema contract must be enforced on the consuming side. Normalise keys. Coerce types. Validate. The model's compliance with your schema is probabilistic and context-dependent, and "context" includes framework code you did not write.

Multi-agent orchestration cost two LLM calls per RFQ instead of one and bought zero accuracy on this workload. That is a finding, not a complaint. Multi-agent is right when subtasks genuinely need different tools, context or independent verification. Extraction followed by deterministic lookup is a pipeline.

6.5 LangGraph, third confirmation, 91.3%

class RFQState(TypedDict):
    rfq: str
    extraction: Optional[dict]
    terminal_state: Optional[str]
    reason: str
    quotation: Optional[dict]
    steps: list                       # ["extract", "evaluate"]

def build_graph():
    g = StateGraph(RFQState)
    g.add_node("extract", extract_node)
    g.add_node("evaluate", evaluate_node)
    g.add_edge(START, "extract")
    g.add_edge("extract", "evaluate")
    g.add_edge("evaluate", END)
    return g.compile()

Fastest build of the five, about 30 minutes, and the most transparent. The node and edge model maps one-to-one onto the pipeline diagram, and steps gives a free execution trace.

And the graph framework was overkill. With no conditional edges and no loops, START -> extract -> evaluate -> END is a linear pipeline with ceremony. LangGraph's actual differentiators, persisted state, checkpointing, conditional routing, human-in-the-loop interrupts, were all unused, because a single-shot RFQ never needs to go backwards.

Worth saying out loud, because the reverse mistake is common: people reach for a graph framework, use none of its graph features, and then attribute their results to it. Here the third framework produced the third identical score. The value was in the engine. The framework was a container.

7. Confident-wrong versus cautious-useless

Section 2 gave the development-set table. The columns worth staring at are the last two, because that is where the architectures separate, and the separation is not the one I expected.

Lyzr is missing from that table because it could not be re-run (section 6.1). From the archived results its development-set M1 is 34.8%, and it produced two wrong approvals, one of which is the impossible quotation this post opens with. Its M2 and M3 are not recoverable, because the archived rows stored only a 200-character excerpt and no configuration id fell inside that window. I left those cells empty rather than scoring them as zero. Every run from here on logs the full response for exactly this reason.

M4 is where the ordering inverts.

Lyzr's 34.8% includes two quotations the answer key says should never have been produced. The clearest is RFQ-12, carbide trim on a standard D2 body, which the technical authority flags as a conflict:

{"rfq_file": "SAMPLE-RFQ-12-d2-angle-carbide.md", "gold": "SOURCE_CONFLICT",
 "predicted": "READY_FOR_REVIEW", "match": false,
 "response_excerpt": "READY_FOR_REVIEW\n\n{\n  \"recommended_terminal_state\": \"READY_FOR_REVIEW\",\n  \"quantity\": 4,\n  \"product_family\": \"Fisher D2T/D2 Angle Control Valves\",\n  \"size_nps\": 2,\n  \"pressure_class_or_pn\": \"CL600\",\n "}

Dify emitted zero prices across all 23 development cases. Not one wrong number left that build. But it wrongly approved five of them, declaring READY_FOR_REVIEW on requests the answer key says must be refused or escalated.

So the two RAG builds are unsafe in opposite directions, and the hybrid builds are unsafe in a third. Stated as a table:

No build in this experiment is safe on both axes. The hybrid builds never wrongly approved anything, across 269 observations spanning three builds and two test sets, which is a genuinely strong result. They are also the only builds that put a wrong number in front of a customer, which section 9 covers, and which is the worse of the two failures.

Sort the failure taxonomy by direction rather than severity and the structure appears:

Confident-wrong failures do not announce themselves and they scale silently. Cautious-useless failures are expensive in time and cheap in risk. Unparseable failures are the best kind, because they crash.

This is the argument for making the terminal state the primary output. A refusal and a wrong quotation are both "not correct" to an accuracy metric, and they are not remotely the same event. A nine-state contract lets you weight, alert and route them differently, because the difference is visible in one field instead of buried in prose.

Evaluation rule. Report accuracy and failure direction together. For any workload where the output leaves the building, a confident-wrong rate near zero is worth more than several points of accuracy.

Hold that thought until section 9, where the hybrid builds stop having a zero in that column.

8. The ambiguous RFQ that split identical pipelines

The three hybrid builds agree on 22 of 23 development cases. The one disagreement is worth more than the 22 agreements.

RFQ-02 asks for a throttling valve for a natural gas let-down skid, NPS 3, Class 600, buttweld, carbon steel, linear, and closes with:

"Please advise a valid Fisher easy-e configuration and budget price."

easy-e is a product line, not a family. It covers ED, ET and EZ. The customer has not said which body style, and the difference is real.

Three builds, same engine, same prompt, same model, two different reads:

n8n        {"terminal_state": "UNPRICED_CONFIGURATION",
            "reason": "no exact match; nearest: ED-NPS3-CL600-BW-WCC-T1-LIN-FULL, ..."}

CrewAI     {"terminal_state": "NO_VALID_CONFIGURATION",
            "extraction": {"family": "ET", "size_nps": 3, "pressure_class": "CL600", ...}}

LangGraph  {"terminal_state": "NO_VALID_CONFIGURATION",
            "reason": "no configuration in catalogue for this combination",
            "steps": ["extract", "evaluate"]}

n8n's extractor picked ED and found near neighbours in the ED price book. The two Python builds picked ET, where no row matches NPS 3 + CL600 + BW + carbon steel, so the filter returned nothing and the engine escalated.

The engine did not diverge. The extraction diverged, and the engine faithfully reported the consequence.

Three things follow.

The divergence is evidence that the answer key is right. When three instances of the same pipeline read the same sentence and pick two different product families, the correct output is not a quotation. It is a question. Gold says NEEDS_INFO.

The failure was still safe. UNPRICED_CONFIGURATION and NO_VALID_CONFIGURATION are refusals with a reason. Neither produced a price. The wrong-family read that would have been catastrophic under vector retrieval, where an ED query pulled CV500 rows at 0.89 and got quoted, was harmless here, because a wrong family produces a key that is not in the map, and a key that is not in the map produces a refusal rather than a plausible number. Exact matching does not prevent extraction errors. It contains them.

Determinism is per-build, not across builds. Each pipeline is reproducible against itself. The extraction step is not reproducible across prompting contexts.

The fix is a business rule, not an engineering one:

// a product LINE was named, not a family -> ask, do not pick
if (/easy-?e/i.test(rfq) && !/\b(ED|ET|EZ)\b/.test(rfq))
  return {json: {terminal_state: 'NEEDS_INFO',
                 missing_fields: ['product_family'],
                 reason: 'easy-e covers ED, ET and EZ; body style not specified'}};

I did not add it. Adding a gate tuned to a known test case after seeing the result is how you fit an eval instead of building a system. The holdout in the next section contains an independent instance of the same ambiguity, written before I looked at any holdout output, which is a fairer test of whether the behaviour generalises.

9. What the blind holdout broke

Everything above this line is a development-set number, and it should be read as one.

The same 23 cases were in front of me while I repaired the systems. n8n went through eight workflow versions against them. The CrewAI lowercase-key fix that moved accuracy from 47.8% to 91.3% was found by looking at these cases failing. That work is legitimate engineering, but a score measured on the cases you debugged against is a development-set score, not a benchmark.

So I wrote a second set.

The protocol. Twenty-five new RFQs, written after every build was frozen. Their answers were derived by hand from two independent sources: the price book, for whether a configuration exists and at what price, and the written policy rules, for scope, NACE, conflicts and completeness. No agent was run to produce the answer key, and no build was modified after the holdout was written. The set deliberately includes cases the development set never covered: liner trims, a restricted port, a wafer end connection, a stainless body, a class that does not exist for its family, a negated attribute, and an RFQ with two line items.

Then three repeated runs per case, to separate genuine capability from run-to-run luck.

9.1 The results

Measured against the same runner and parser on both sets, the drop is:

The three hybrid builds lose 15 to 19 points on data they were not debugged against, and the confidence intervals do not come close to touching 91.3%.

Dify falls too, by less. Its development and holdout numbers were produced by the same parser, so they are comparable to each other, and the drop is real rather than an artifact. None of it should be read as a comment on its quoting ability: it emitted zero prices across both sets, so there was never a correct quotation to lose.

This is the number I would want in front of me before deploying anything, and it is the number that almost never appears in agent write-ups, because producing it requires writing test cases you expect to fail.

Run-to-run stability separates the two architectures as cleanly as accuracy does:

Every hybrid build returned the identical terminal state on every case on every run. Temperature is zero and the decision is a table lookup, so that is expected, but it is worth having measured rather than assumed, and it means a hybrid failure is a bug you can reproduce and fix rather than a tendency you have to characterise statistically.

Dify did not. Its answer changed between runs on several cases, which matches Lyzr's behaviour on the development set, where the same RFQ returned OUT_OF_SCOPE, NEEDS_INFO and READY_FOR_REVIEW across runs. When the terminal state is produced by generation rather than by a code path, it is a sample rather than a value, and a single-run benchmark of such a system is measuring one draw from a distribution.

9.2 Per case

Bold means the majority answer matched gold. An asterisk means the platform did not return the same state on every run, which happens only in the Dify column.

9.3 The one number that changed the conclusion

Look at the two M4 columns in the leaderboard, because they cross over.

Dify's wrong-approval rate is the worst in the experiment, and its wrong-price rate is still zero. It waved through NACE cases, a source conflict, an actuator enquiry and configurations that do not exist, declaring READY_FOR_REVIEW with text like "No conflicts identified with provided families or safety policy" on H11, a sour-service request that must go to a materials engineer. It never printed a number. Every one of those is a bad decision that a human reviewer would catch the moment they looked for the price that is not there.

The hybrid builds inverted. Their wrong-approval rate stayed low, and their wrong-price rate stopped being zero:

On the development set, all three hybrid builds had a wrong-price rate of zero. On the holdout, they do not.

H06 asks, in plain English, for a restricted port:

Fisher ED, NPS 2, Class 300, raised face, WCC, trim 1, linear,
RESTRICTED port (we are deliberately undersizing the trim for turndown).

The extraction was perfect. This is the n8n payload, and the CrewAI one is identical on the port field:

{"family": "ED", "size_nps": 2, "pressure_class": "CL300", "end_connection": "RF",
 "body_material": "WCC", "trim": "T1", "flow_characteristic": "LIN",
 "port": "RSTR", "quantity": 1, "service": "boiler feed recirculation"}

The pipeline quoted ED-NPS2-CL300-RF-WCC-T1-LIN-FULL at AUD 5,414.54. The correct answer was ED-NPS2-CL300-RF-WCC-T1-LIN-RSTR at AUD 4,981.39.

Look back at the candidate list in section 6.3:

const candidates = [
  ... + '-' + fcode + '-FULL',      // tried first, unconditionally
  ... + '-' + fcode + '-RSTR',      // tried second

The engine never reads the port field it just asked the model to extract. It tries FULL, finds a row, and stops. The customer asked for one product and got a quotation for a different one at a price AUD 433.15 too high, with a valid-looking configuration id attached, from the architecture I had just finished describing as the safe one.

This is the most important paragraph in the post, so let me be blunt about what it costs the thesis. Exact matching contains extraction errors, because a wrong key is not in the map. It does absolutely nothing about logic errors in the code that builds the key. The deterministic layer is not safe by virtue of being deterministic. It is safe only to the extent that it is correct, and mine was not.

The development set never caught it because not one of its 23 cases asked for a restricted port.

9.4 Why the rest of the drop happened

Every remaining holdout failure has a mechanical cause, and they collapse into one root cause.

The extraction prompt is a closed vocabulary. That is what made extraction reliable. It also caps what the pipeline can express. Comparing the prompt's enums against the catalogue:

Multiply it out and the result is stark:

Of 4,782 priced configurations, the extraction schema can express 268. That is 5.6% of the catalogue.

And the development set lived almost entirely inside that 5.6%. Ten of its eleven priced answers are expressible. Only seven of the holdout's twelve are.

That single fact explains most of the gap between 91.3% and 76%.

The individual failures fall out of it:

• H08 asked for "TRIM 4". T4 is not in the enum, the model returned trim: null, the lookup found T1 neighbours instead and returned UNPRICED_CONFIGURATION. The catalogue has the row. The schema cannot name it.
• H05 and H23 asked for a wafer end connection. Here the schema is not merely incomplete, it is wrong: the prompt spells the value WAFFER, and the engine's normaliser maps WAFER to WAFFER. Zero catalogue rows use WAFFER. Seventy-eight use WAFER. Every wafer-bodied valve in the price book is unreachable, and the normaliser actively converts correct input into a key that cannot match. In the v1 draft of this post I described that misspelling as a deliberate choice, writing that "the canonical vocabulary is whatever your data actually contains". That was wrong. It is a typo, it was propagated into the engine, and the development set never touched a wafer valve so nothing caught it.
• H22 asked for a CG8M stainless body, which the schema cannot express.
• H24 said, in as many words, "STANDARD trim (explicitly not carbide)". The gate fires on d2 and carbide appearing in the same text and returned SOURCE_CONFLICT. A regular expression cannot read a negation. Cheap, readable, auditable policy gates have exactly this failure mode, and it is the price of putting policy in regex rather than in a parser.
• H18 returned UNPRICED_CONFIGURATION where gold says NO_VALID_CONFIGURATION. Both are refusals, so nothing wrong shipped, but the fallback filter is loose enough to find "nearest" rows in a class that does not exist for that family.
• H25 put a valve and an actuator on one enquiry. The model returned a JSON list of two line items; the engine assumed an object:

AttributeError: 'list' object has no attribute 'items'
  File "langgraph_agent.py", line 31, in evaluate_rfq
    ex = {str(k).lower(): v for k, v in (extraction or {}).items()}

Identical on all three runs. n8n survived the same case, because its JavaScript parse path degrades differently. A single-item schema assumption is invisible until someone sends a two-line enquiry, which in this domain is most of them.

9.5 What the holdout did not change

Three claims survived contact with unseen data, and they are the ones worth keeping:

1. Policy gates generalised. Actuator, non-Emerson, NACE, 9500-class and D2-carbide all fired correctly on cases written after the regexes were. Across the seven holdout cases decided by a gate, n8n was correct on 21 of 21 observations (7 cases x 3 runs) and LangGraph on 18 of 21, its only failures being the H25 crash described above rather than a wrong policy call. Note the asymmetry with H24 in the previous section: the gates generalise well when the trigger phrase is present and asserted, and fail when it is present and negated.
2. Prompt injection was a non-event again. H20 wrapped a fresh injection around a legitimate request and was quoted correctly, for the same structural reason as before.
3. The ambiguity behaviour generalised. H17 named a product line without a family, exactly the RFQ-02 pattern from section 8, and this time the pipeline returned NEEDS_INFO with missing_fields: ["product_family"]. The behaviour I declined to hard-code turned out to be there already.

10. Limitations

Read this section before quoting any number above it.

This is a development set plus one holdout, not a benchmark. The development numbers were measured on cases I debugged against. The holdout numbers are honest but rest on 25 cases and, for two of the four re-runnable builds, on fewer repeated runs than I would like. Confidence intervals are given; they are wide.

Lyzr was not re-run. Its API key was unavailable at holdout time and its free-tier credit allowance would not have covered the runs. Every Lyzr number here is development-set only, and the comparison is correspondingly incomplete. If the RAG side of this post has a weak point, that is it.

The RAG builds were not tuned to exhaustion. Reranking, hybrid sparse-plus-dense retrieval, metadata filtering and per-family collections would all move their numbers. I did not try them. The mechanism in section 5 predicts that hybrid retrieval would close much of the gap, and that prediction is the obvious next experiment rather than something this post has settled. Notably, every one of those techniques is a step back toward exact matching.

Model and embedding choice were not controlled across the RAG builds. Section 3 sets out exactly which variables moved. The narrow framework claim is clean; the broad "RAG loses" claim is evidence, not proof.

The answer key encodes one organisation's policy. Sections 8 and 9.4 are both cases where the disagreement is about how much inference an agent may do on a customer's behalf, not about a fact. A different maintainer would produce different gold answers.

Free tiers are not production. Lyzr's catalogue ingest timed out at 9.2 MB. A paid plan indexes the whole catalogue. It would still retrieve CV500 rows for an ED query.

The cost figure is narrow. Cash API and platform spend was $0: all five platforms on free tiers or self-hosted, chat inference through a local gateway to free-tier models, embeddings from a local Ollama instance, storage on local Docker volumes. That excludes local compute, electricity, and roughly 6.5 hours of build time across the five platforms plus the eval and holdout work. The honest claim is "this costs no cash to reproduce", not "this is free".

What I did not measure. Latency under load, multi-turn behaviour, catalogue-drift resilience, or anything about the RAG builds' retrieval quality beyond the direct probe in section 5.

The protocol I would run next, in order:

1. Fix the three defects the holdout exposed: read the extracted port when building candidates, correct WAFFER to WAFER throughout, and accept a list of line items. Then re-run the holdout without touching anything else, so the delta is attributable.
2. Widen the extraction enums to the full catalogue vocabulary, or generate them from the price book at build time so they cannot drift from it again. Measure coverage as a first-class metric.
3. Score extraction separately from the pipeline, field by field, so a bad extraction and a bad lookup stop presenting as the same symptom.
4. Add a hybrid retriever to the Dify build and re-run, to test the section 5 mechanism directly.
5. Write a second holdout, larger, and keep it sealed.

11. Decision matrix

A menu is not an answer, so: for this workload I would build it on LangGraph. Not because the graph helped at 91.3% (it did not, the score was identical to a code node in n8n), but because the next version needs the clarification loop that sections 8 and 9 both point at, and that is exactly what a graph framework is for. It was also the fastest build and the engine is a pure function you can unit test without starting a container.

n8n is the right answer instead when the deployment reality is "operations owns this and it has to talk to Outlook and Salesforce".

Call the winning shape a hybrid agent with a deterministic decision layer, not a "deterministic agent". Extraction is still a language model, and section 9 is a catalogue of what that seam does under pressure.

What actually produced the result

Four decisions, in order of contribution.

1. Exact matching instead of semantic retrieval. A configuration id is a compound key over a closed vocabulary. Keys get looked up; prose gets retrieved. Vector search remains right for the bulletins, where "what is the maximum temperature rating for a WCC body" genuinely is a semantic question. It is wrong for the price book.

2. The LLM only for extraction, never for the decision. Free text to canonical codes is a task models are good at: bounded, verifiable, and recoverable when it fails, because the extraction is in the output and you can see it. The moment the model also decides the terminal state or reports the price, a probabilistic system is answering a deterministic question.

3. Validation gates on raw text, ahead of everything. Ten lines of regex produced nine of nine correct policy answers on unseen cases. They are also the component that cannot read a negation. Both facts are true and both belong in the design review.

4. Structured terminal states as the output contract. Nine tokens rather than prose. This made the eval trivial, made failure direction visible, and made UNPRICED_CONFIGURATION expressible at all. Both RAG builds had the same nine states in their system prompts and could not reach them reliably, because a state produced by generation is a suggestion and a state produced by a code path is a fact.

And the thing the holdout added, which no amount of architecture buys you:

5. Test cases you did not build against. Every one of the three defects in section 9.3 and 9.4 was a plain bug, sitting in the winning build, invisible to a 23-case suite that had been used for debugging. The architecture was right. The implementation was not, and only unseen data said so.

12. FAQ

Is RAG the wrong choice for agents? No. RAG is the wrong choice for exact lookup over a compound key. For the same corpus, questions like "what packing options does the ED bulletin list for Class 600" are genuinely semantic and vector retrieval handles them well. The failure here is specific: dense embeddings compress away the lexical detail that distinguishes ED-NPS2-CL300 from CV500-CV500-NPS3-CL150, which is the documented argument for hybrid sparse-plus-dense retrieval [5][6]. If your agent looks up part numbers, SKUs, account ids or configuration codes, treat that as a database problem with a language front end.

CrewAI or LangGraph? On this workload they scored the same on the development set, because they ran the same engine, and the difference is ergonomics rather than accuracy. CrewAI costs two LLM calls per request and adds a narration failure mode where the agent summarises the tool output instead of returning it. LangGraph was the fastest build, gives a free execution trace, and is the better base for the multi-turn clarification loop this system needs next. Choose CrewAI when subtasks genuinely need different tools or independent verification; choose LangGraph when you need state, routing and resumability.

How do I evaluate an AI agent without an LLM judge? Give it a discrete output contract. Nine terminal states plus a structured payload turned evaluation into predicted == gold, which runs in under a minute and never disagrees with itself. Then report more than one metric: state accuracy, identifier exact match, value exact match, and the rate at which the agent confidently emits an answer it should have refused. Finally, hold out cases you have never debugged against. Section 9 is the entire argument for that last step.

Why not just use a bigger model? Because the failing step was not a reasoning step. Retrieval handed the model rows from the wrong product family, and the model wrote a fluent answer from wrong inputs, which is what models do. And in the hybrid builds, the three defects that cost the most accuracy on unseen data were a candidate-ordering bug, a misspelled enum value, and an unhandled list. No model upgrade fixes any of them.

Is $0 real? Cash spend is real: free tiers, self-hosted containers, a local gateway and local embeddings. It excludes local compute and about 6.5 hours of build time. At production volume the architectural point survives anyway: the hybrid builds make one LLM call per RFQ and zero embedding calls, so the more accurate architecture is also the one that does less inference.

Would you put this in production? Not at 76%. I would fix the three defects in section 9, regenerate the extraction vocabulary from the price book so it cannot drift, add the clarification loop, and re-run the holdout. The architecture is right. The implementation needed unseen data to find out where it was not.

13. Reproduction

The bundle. Every answer key, every raw run log with full responses, the scoring scripts, the three ports of the engine, the price book and both sets of RFQs are packaged as a single archive: reproducibility bundle (133 KB). It contains no credentials. It also ships with the three defects in section 9 unfixed, deliberately, so the results reproduce exactly.

Inputs. Ten official Emerson product bulletins pinned by document number, revision and SHA-256, with actuator content excluded. Pricing data yielding 4,782 exact-lookup configurations. All prices synthetic. Twenty-three development RFQs and 25 holdout RFQs.

Answer keys. Development: a deterministic reference engine, 43 of 43 unit tests passing. Holdout: derived by hand from the price book and the written policy rules, validated by a script that fails if any gold configuration id is absent from the price book or if any NO_VALID_CONFIGURATION case turns out to have matching rows.

Harness. One runner per platform, identical except for the transport and the response parser. Byte-identical RFQ text to every platform. Exceptions recorded as results rather than crashing the run. Full responses logged, not excerpts.

Engine. 143 lines of JavaScript in an n8n code node, ported line for line to Python for CrewAI and LangGraph. Four phases: parse the model output, run the raw-text gates, normalise and resolve the extraction, then exact lookup with a bounded five-attribute fallback.

Versions. CrewAI 1.15.10, LangGraph 1.2.10, n8n Community Edition and Dify Community Edition in Docker, Python 3.12 for both agent venvs, Ollama nomic-embed-text at 768 dimensions, Lyzr on gpt-4o-mini, chat inference for the other four routed through a local OpenAI-compatible gateway. Extraction temperature 0.

Verification you can check. Every direct lookup in all three hybrid builds returned these to the cent on the development set:

ED-NPS2-CL300-RF-WCC-T1-EQP-FULL                        AUD  5,414.54  x2 = 10,829.08
ET-ET-NPS1-CL150-RF-WCC-T1-LIN-FULL                     AUD  4,950.00
EZ-EZ-NPS1-CL150-RF-WCC-T1-EQP-FULL                     AUD  4,188.46
HP-HP-NPS2-CL900-RF-WCC-T1-LIN-FULL                     AUD 22,869.00
GX3-GX3-NPS2-CL150-RF-WCC-T1-LIN-FULL                   AUD  4,620.00
CV500-CV500-NPS3-CL150-RF-WCC-TS31700-EQP-FULL          AUD 11,220.00
VEEBALL-V150-NPS3-CL150-RF-WCC-TTCM_SEAL-MOD_EQP-FULL   AUD  9,062.31  x2 = 18,124.62
8580-8580-NPS4-CL150-LUGGED-WCC-TSOFT_SEAL-LIN-FULL     AUD  9,531.92
ET-ET-NPS1-CL150-BW-WCC-T1-LIN-FULL                     AUD  5,660.77

And the one it got wrong on unseen data, which is the more useful line:

requested  ED-NPS2-CL300-RF-WCC-T1-LIN-RSTR             AUD  4,981.39
quoted     ED-NPS2-CL300-RF-WCC-T1-LIN-FULL             AUD  5,414.54

---

The whole experiment reduces to one sentence, and it is the same one whether you are choosing a framework or reviewing someone else's agent:

Put the LLM where language is, and put the decision where the truth is.

Then go and find out what your test set never asked.

---

Further reading

Each of these covers one component this post leaned on:

• Agent state machine design - why the nine terminal states in section 4 are a state machine, and how to design one that does not grow a tenth state every sprint.
• Nondeterminism at temperature zero - the run-to-run variance Lyzr showed, and why temperature=0 is not the same as reproducible.
• Observability and tracing for agents - the logging contract in section 4.4, generalised beyond one workload.
• When agents transform tool data - the CrewAI narration bug in section 6.4 is one instance of a broader pattern.
• Why the model cannot just ignore an injected instruction - the mechanism behind section 6.3's injection result.
• Agents need to be able to say "I don't know" - the argument for NEEDS_INFO as a first-class outcome.
• When agents game the eval metric - the failure mode section 9 exists to prevent.
• The demo-to-production cliff - 91.3% to 76%, described from the other direction.

---

Appendix A: platform notes

Kept out of the main path because it is API archaeology, not architecture. Useful if you are building on these platforms.

Lyzr ingestion

The silent-zero on /v3/train/txt/ is the one to watch. It returns success, returns JSON, and indexes nothing. Without verifying via GET /v3/rag/documents/ you will build an agent on an empty knowledge base and blame the model. There is also no public REST endpoint for scored retrieval on classic knowledge bases, which is why the retrieval evidence in section 5 had to be pulled out of an inference response.

Dify console API

1. Auth is a CSRF cookie plus header, not Bearer. The login password is RSA-encrypted client side, so you cannot curl a login. Drive the console API from an authenticated browser context, or use the public service API.
2. Every credential parameter must be a string. context_size: 32768 as an integer returns a 500. As '32768' it works.
3. Agent apps reject response_mode: blocking. Use streaming against POST /v1/chat-messages and reassemble the answer deltas.

n8n

1. API key truncation. The UI shows the key once and truncates it in the DOM. The full JWT lives in the user_api_keys table in n8n's SQLite database.
2. $('NodeName').first().json is the reliable cross-node reference. .item.json fails silently, which is the worst failure mode for a data-flow tool.
3. Code node early returns must be {json: <object>}. Returning undefined yields {} downstream with no error anywhere.
4. The webhook body does not flow downstream automatically. Capture it in an early node and pass it explicitly.
5. responseMode: responseNode plus respondWith: allIncomingItems is the working pattern; without it you get a 500 about an unused Respond node.
6. Activate through the API or the webhook stays dead.

Items 2 and 3 share a theme: silent empty results. A workflow tool that returns {} instead of raising is the hardest class of bug to find, because every node reports success.

CrewAI

Defensive parsing is not optional. Full parse, then de-fenced parse, then last JSON object, then an explicit UNPARSEABLE carrying raw text:

for cand in [text, re.sub(r"```(?:json)?", "", text).replace("```", "")]:
    try:
        return json.loads(cand)
    except Exception:
        pass
m = re.search(r"\{[^{}]*\}", text, re.S)
if m:
    try:
        return json.loads(m.group(0))
    except Exception:
        pass
return {"terminal_state": "UNPARSEABLE", "raw": text[:300]}

That last branch is the one people leave out, and it is the one that turns "the eval crashed" into "case 14 came back as prose, here it is".

Appendix B: sources

Technical claims first, vendor-reported customer stories clearly marked as such.

Every accuracy figure, configuration id, price and error message in this post comes from the run logs and answer keys described in section 13, and is checked by an automated verification script on every build of the article.