Most LLM features that disappoint in production were judged badly, not built badly. A practical evaluation process for product leaders: job definition, golden sets, failure taxonomy, budgets and a ship/no-ship rule.
Most LLM features that disappoint in production were not built badly. They were judged badly. A team demos the feature on five friendly inputs, everyone nods, and the launch date holds. A few weeks later support is forwarding screenshots. The model did not get worse after launch. The team simply never decided what "good enough" meant, so nobody could tell whether they had reached it.
Evaluation is a product decision before it is a technical one, and you do not need a research team to do it well. You need a written definition of the job, a small set of real cases, a shared way to classify what goes wrong, and a decision rule agreed before you see the results. Below is the process I use, with a worked example from payments, where I have spent a large part of my career.
Write one paragraph that describes the job the feature does for a specific user. Not "an AI assistant for disputes" but "drafts a first reply to a customer who has disputed a card payment, using the account history, so an agent can review and send it quickly." If you cannot write that paragraph, you are not ready to evaluate anything, because you do not yet know what success looks like.
Then list the unacceptable failures: outcomes where a single occurrence is a problem, not a statistic. There are usually only a handful, and they tend to come from legal, compliance, brand or money. Write them before anyone looks at model output. If you write them afterwards, you will quietly write them around what the model already does.
This splits evaluation into two kinds of questions. For most qualities you ask "how often is it good?" For unacceptable failures you ask "can it happen at all, and what stops it?" Those need different tests and often different owners.
A golden set is a collection of inputs with a clear description of what a good output must contain. Start small. Somewhere between fifty and a hundred and fifty real cases is enough for a first decision, and far more useful than a thousand synthetic ones. Pull them from real tickets, transcripts or documents, masked according to your data rules. In regulated products I have learned to confirm that using production data for testing is permitted before anyone copies it into a spreadsheet, not during the next audit.
Choose cases deliberately: the common path, the awkward but frequent path, and the rare cases that map to your unacceptable failures. A set of only typical cases will tell you the feature is great. A set of only edge cases will tell you it is hopeless. Neither is true.
For each case, write what a good output must do and must not do, rather than one perfect answer. LLM outputs vary in wording, and grading against a single reference punishes harmless variation. "Must acknowledge the dispute reason, must not promise a refund, must reference the correct transaction date" is gradeable. A model answer is useful as an illustration, not as the bar.
"It's not quite right" is not actionable. A shared taxonomy turns vague discomfort into counts you can track over time. I use six categories:
Every failing case gets exactly one primary label. That forces the reviewer to decide what actually went wrong, and it gives you a distribution instead of a mood. A feature failing mostly on "unhelpful" usually needs better context or a sharper prompt. One failing on "off-policy" needs rules the model can see, or a check after generation. One failing on "wrong" may need retrieval, or may be the wrong job for a model.
Human review is slow and expensive, and it is the only thing that tells you whether an output is actually good. Automated graders, whether plain code checks or a second model scoring against a rubric, are fast and cheap, and they tell you whether an output matches what you already decided good looks like. You need both, in that order.
Start with two people who know the domain grading the same cases independently, then compare. Where they disagree, your criteria are unclear, and that is worth fixing first. Only then write automated graders, and check them against the human labels before trusting them.
Use deterministic checks wherever you can. Whether a draft contains an account number or mentions a refund can be checked with code, and code does not drift. Use a model as grader for judgments that need language understanding, such as tone or whether the reply addressed the customer's stated reason. Keep a sample of human review running permanently, because model graders tend to share blind spots with the models they grade.
Cost and latency are product requirements, not engineering afterthoughts. Set them per interaction, working backwards from the workflow: how long will the user wait at that moment, and what is the interaction worth? An agent in a back-office tool will tolerate a short wait. A customer in a chat window will not. A feature called constantly by every agent needs a per-call cost that still works at that volume, including retries, retrieval and any graders you run in production.
Look at the slow end of the distribution, not the average. Users remember the waits. Put the budget in the same document as the job description so nobody treats it as negotiable in launch week.
Once you have a golden set and graders, every change to the prompt, model, retrieval setup or tool definitions runs against the full set before it ships. This one habit separates teams that improve their LLM features from teams that churn them. Prompt edits feel small and local. They are not. A sentence added to fix one complaint can break cases nobody was looking at.
Treat model upgrades the same way. A newer model is usually better on average and sometimes worse on the specific thing your product depends on. Compare the failure distributions side by side and read the cases that changed category. Keep the set alive: every serious production failure becomes a new golden case, so the same mistake cannot come back quietly.
Take a hypothetical payment company whose operations team handles card disputes. Agents write each customer reply by hand. The proposed feature reads the dispute, the transaction record and the account notes, then drafts a reply for the agent to edit and send. Nothing reaches a customer without a human clicking send.
The job statement: draft a reply the agent can send with light edits in most cases, without looking up transaction details themselves. The unacceptable failures, written before anyone sees output: the draft promises a refund or chargeback outcome the company has not decided; it includes another customer's data; it states an amount or date that does not match the record; it says anything that reads as legal advice. The compliance lead signs that list.
The golden set is about a hundred and twenty past disputes, masked, covering the common reasons (not recognized, duplicate charge, goods not received), the awkward ones (partial refund already issued, confusing merchant descriptor), and a handful built to tempt the unacceptable failures, such as an angry customer demanding a guaranteed refund.
Two senior agents grade every draft and label failures with the taxonomy. Code checks verify that any amount and date in the draft match the transaction record and that no identifiers appear other than the customer's own. A model grader, calibrated against the agents' labels, scores tone and whether the reply addresses the stated reason. The budget comes from the workflow: the draft must be ready by the time the agent has opened the case, and it must cost a small fraction of the agent time it saves.
The team agrees the decision rule in advance. Here is the version I start from and adapt per product. The feature ships when all of these hold:
If any line fails, the answer is no-ship or a narrower ship. In the dispute example, a perfectly reasonable outcome is launching for duplicate-charge and not-recognized cases while holding back partial-refund cases, where drafts tend to be confidently wrong about amounts.
Fix the rule before the results arrive. Otherwise the bar moves to wherever the demo landed, and the launch date wins every argument.
This process costs time. A first golden set takes a few days of senior attention, and keeping it alive costs a little every week. For an internal tool with a forgiving user, such as a summarizer that helps your own team skim meeting notes, a lighter version is fine: twenty cases, one reviewer, and the habit of rerunning them. The full process earns its cost where outputs reach customers, touch money, or sit inside a regulated workflow.
The strongest counterargument is that real usage is the only meaningful test, so you should ship early and learn. That is partly right. No offline set captures everything users will do, and production monitoring is not optional. But shipping without a baseline means you cannot tell whether the next change made things better or worse, and "we'll learn in production" is not a sentence you want to say to an auditor. Offline evaluation does not replace learning from users. It is what lets you learn from them without guessing.
Pick the LLM feature closest to launch. Write the job paragraph and the unacceptable failures on one page, ask whoever owns risk to sign the second half, and pull fifty real cases into a spreadsheet with must-do and must-not-do columns. Then grade the current version yourself before anyone else does. One afternoon of that will tell you more about the feature than a month of demos.
Written by
Ratomir Jovanovic
In collections and payments, the conversation design is the compliance work. How to set guardrails on promises, detect vulnerability, keep audit trails and measure outcomes that matter more than containment.
27 August 2026
Approval rates, retries, failure messages, fees and reconciliation shape revenue and customer trust directly. That makes payments a product area with an owner, metrics and a roadmap.
12 August 2026
A product leader can now put a working prototype in front of users in days. The long PRD should shrink to a decision memo plus a prototype, with clear rules on what the prototype proves and what it does not.
9 June 2026