AI as a Design Material
AI is a material with a grain. It is probabilistic, generative, sometimes wrong. You design with it the way you design with any material: by understanding its properties and building the right harnesses around it.
A material's grain tells you which cuts to make. The grain of a language model is that it fails plausibly: wrong output arrives in the same shape, the same tone and the same confidence as right output. Nobody can sort the two by reading them.
That one property should decide the first thing you build.
01What the grain predicts
A deterministic component announces its failures. You get an exception, an empty state, a total that is obviously off by a factor of ten. You can catch those in the system, or by looking at the screen.
Plausible failure carries no such signal. A model's mistake is well formed, internally consistent, and phrased with the confidence of a correct answer. So no amount of reading the output tells you whether to trust it. Checking has to happen against something outside the text: a vehicle on a lift, a parts catalogue, a recording of what the customer actually said, a transcript line the claim is supposed to rest on.
Which gives you the first structure to build, before you tune a prompt and before you write an eval suite. Find the moment an output stops being a draft and becomes a commitment: money quoted, a message sent to a customer, a row written, a promise a person now owes. Put a named human at that moment, with the tools to compare the draft against the world. Everything else in the harness is negotiable. That gate is not.
02One artifact, from blank page to approved
At AutoLeap, a platform for auto-repair shops, this job belongs to a service advisor. A customer calls, or a technician finishes an inspection, and the advisor has to turn what was said into a repair order: what is wrong, which parts, how many labour hours, what it costs. They do this with the customer waiting.
Magic RO removes the blank page. Feed it the conversation and it returns a structured estimate in seconds, line items and hours and prices, in the format the shop already uses.
Then the part that matters. The estimate arrives as a draft, and it stays a draft. It cannot reach the customer or the ledger until the service advisor reviews and approves it.
Ask what that step actually catches, because the answer follows straight from the grain. Every error it catches is well formed. A line item that is correct for a different engine variant of the same model year. Labour hours drafted from the common version of a repair when this particular car needs the awkward one. A part the shop sources at a price the model had no way to know. Work the conversation mentioned as a maybe, which the draft quietly promoted to agreed. And the quietest of them, the omission: something the customer raised once, in passing, that never made it onto the estimate at all. Read the draft cold and every one of those looks like a competent repair order. Only somebody holding the car, the catalogue and the memory of the call can see them.
That tells you what the review surface has to do. If approving means re-deriving the estimate, you have moved the work rather than removed it, and within a week your advisors will be clicking approve without reading. So the interface has to make comparing cheaper than reconstructing: line-level edits instead of a rewritten block, and each line traceable back to the part of the conversation that produced it. The advisor's question drops from "is this estimate right?" to "does this line match what I heard?" One of those is answerable in seconds, forty times a day. The other is a second job.
The model is allowed to be wrong. The product is not.
There is a measurement in that, and you can run it this week on whatever you have shipped. Instrument your approval step and count how often a human changes anything before approving. If that rate drifts towards zero, your gate has become a button, and you should find out whether the drafts really did get better or whether your reviewers simply stopped looking. Twenty minutes of logging will tell you which.
One honest note on evidence, since this is where essays usually overclaim. AutoLeap's platform numbers moved over that period: ACV up 20%, NPS up 15 points, churn down 23%. No single feature owns any of that, and Magic RO certainly does not. Read those as evidence the approach shipped and survived contact with real shops, not as the yield of one approval gate.
03Magical, but accountable
The principle underneath all of it is short. The AI generates a strong starting point, and the human always approves. The person using the product should feel that the work started itself. The person carrying the consequence should be able to point at the moment they took the wheel and say what they changed.
Those two are usually treated as a trade. Magical without accountable gives you a demo: impressive once, unusable by anyone whose name goes on the output. Accountable without magical gives you a form with extra steps. Held together, they are one design: a generative starting point that removes effort, and a checkpoint that keeps a person in command of the result.
04What to build after the gate
Now the rest of the harness has a job description. Each piece earns its place by making review cheaper or rarer.
Guardrails narrow what the model is allowed to attempt, so fewer drafts arrive in a state no reviewer can check. Model routing sends each step to whichever model is good at that step. Evals tell you whether last month's prompt change made drafts better or only different, which is the question your approve-without-edit rate cannot answer on its own. Audit journals record how an output was derived, so that when someone rejects a draft you can trace what produced it and fix the cause instead of the symptom.
That is roughly the order things went in at Articos, an AI user-research platform and a Disrupt Labs product, where I work as Lead Researcher & Product Designer. Approval first, then journals, then the evals that tell you whether the journals are recording anything worth keeping. A guardrail also lands faster when the person who noticed it missing can encode it the same afternoon rather than file it against another team, which is a separate argument I have made at research to product to code and will not repeat here.
Keep one limit in view. None of this makes the output true. An estimate can be fully traceable, approved by the right person, journalled end to end, and still wrong about the car. Provenance tells you what produced a claim. Approval tells you who now owns it. Whether the claim is correct is a third question, and the gate exists precisely because the material cannot answer it for you.
So look at your own product and find the outputs that commit without anyone approving them. What would it cost you, the first time one of those is confidently wrong?
05References
- UX Design Innovation: Challenges for Working with Machine Learning as a Design Material
- Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
- Why Language Models Hallucinate
- Hallucinations in code are the least dangerous form of LLM mistakes
- Ironies of Automation
- The Flaws of Policies Requiring Human Oversight of Government Algorithms
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
- Guidelines for Human-AI Interaction
- Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
- Measuring Attribution in Natural Language Generation Models
- Your AI Product Needs Evals
06Further study
Branching by what you are actually trying to do next.
If you are about to put an approval step in front of a model.
- Ironies of Automation — five pages from 1983; the gate-becomes-a-button failure mode predates the model by four decades, read it first.
- To Trust or to Think — the interventions that measurably keep reviewers thinking instead of clicking approve.
- The Flaws of Policies Requiring Human Oversight of Government Algorithms — what oversight-on-paper looks like once nobody instruments it; the case for logging your edit rate.
If you want the material's properties from the source.
- Why Language Models Hallucinate — why plausible error is baked into training and scoring, so no prompt tuning removes the grain.
- Does the Whole Exceed its Parts? — the experiment showing plausibility sways reviewers regardless of correctness; checking must leave the text.
- Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design — what those properties do to a standard design process, mapped by designers who hit them.
If you are building the rest of the harness.
- Your AI Product Needs Evals — the concrete eval loop that answers what your approve-without-edit rate cannot.
- Measuring Attribution in Natural Language Generation Models — how to make line-to-source traceability rigorous instead of decorative.
- Guidelines for Human-AI Interaction — the checklist for the review surface itself, correction and dismissal included.
- AI-native product designThe harness as part of the interface, not the plumbing.
- Agent experience designWhen the user is sometimes an agent.
- Designing AI Behavior, Not AI ScreensThe decision layer under every agent interface.
- Building With LLMs: Designing the HarnessThe rig around the model: evals, guardrails, routing, journals.