Harness engineering for research agents
Everything written about harness engineering assumes a compiler will catch the lies. Research agents don’t have one. The harness has to be the compiler.
Harness engineering for research agents is an unbuilt discipline. The term itself is not new anymore. OpenAI published on harness engineering in February, Martin Fowler wrote about the agent harness as a governor in April, LangChain has an anatomy of one, and there is now an awesome-list collecting the patterns and a hundred-paper survey formalizing them. All of it is about coding agents. Almost none of it is about agents that produce research, and research has a different failure mode than code.
Code that is wrong usually fails loudly. A test goes red. A build breaks. A user files a bug. Research that is wrong reads beautifully and gets acted on.
That difference is why the interesting work in research agents sits in the scaffolding rather than the prompt, and why this piece is about the scaffolding.
01What is an agent harness, in plain terms?
The model is the engine. The harness is everything you wrap around it so the engine does useful work at a standard you can defend: the tool definitions, the routing between models, the context you feed in, the guardrails, the evals, the approval gates, the journal of what happened.
Martin Fowler’s framing is the one I keep coming back to. He describes the harness as a governor combining feed-forward and feedback to hold the codebase toward a desired state. Swap “codebase” for “study” and the shape survives the move. You are still regulating a probabilistic process toward a standard it will not hold on its own.
I wrote about this material problem before I had the vocabulary for it, in Building with LLMs: designing the harness. An LLM is a probabilistic material. Send it the same prompt twice and you can get two different answers, one of them confidently wrong. That is the grain of the material, not a bug to patch out.
02Why does a research agent need a different harness than a coding agent?
Four differences matter most for the kind of research agent I mean here, and they compound.
There is no compiler. A coding harness gets a free oracle. Tests pass or they do not. A research harness has to build its own oracle out of rubrics, reference findings and adversarial review, because you cannot call a function that answers “did this study land somewhere defensible”. This is the defining problem of the discipline: the harness has to supply the falsifier that the environment refuses to provide.
The user is simulated, and the simulator is part of the instrument. This is the one most teams miss. If your research agent interviews a synthetic participant, that participant is not test data. It is measuring equipment, and it has its own bias. Zhou and co-authors put numbers on this in July: across 451 human participants, 165 tasks, and 31 LLM user simulators, the simulators came out excessively cooperative, stylistically uniform, and short on realistic frustration or ambiguity. That creates an easy mode that inflates agent success rates above the human baseline. Their sharpest result should worry anyone planning to buy their way out with a bigger model: making the simulator smarter did not reliably make it more faithful.
The task itself drifts. Ask a coding agent to fix a bug and the compiler will not let “fix the bug” quietly become “redesign the module”. Research has no such lock. In the evaluation behind the Grounded Simulation preprint, the intuitive workflow, a practitioner iterating with a chat model over ten turns, failed exactly this way: four of every five of its themes came back phrased as design recommendations rather than findings. The model stopped reporting what users said and started advising what the team should build. Findings the conversation had surfaced mid-way were discarded by the final synthesis. A research harness has to be task-preserving: the question asked at turn one must still be the question being answered at the end.
The score is not about the model. Zhu and co-authors argued in a paper revised on 6 August that “LLM agent performance” does not name a single evaluation target at all. The harness, the environment, the evaluator and the compute budget all move the score, so a number only means something once you say what you held fixed. For research agents this is not an academic point. It means your quality number belongs to the harness you ran it on, not to whichever model you happened to be using that week.
03What does the harness actually contain?
A research pipeline I work on runs as a harness of 100+ substages with a test suite in the four figures. That number is not the point, and it is not a target to copy. That is what it costs to turn “the model writes a research report” into something you can stop and inspect partway through. Expect the infrastructure to take longer than the interesting part, and budget for that rather than being surprised by it.
If you are building one, these are the pieces that carry the most weight:
- 01A rubric written before the first prompt.
Six criteria a senior researcher would use to judge a report, from evidence specificity through to whether the model ever saw the hypothesis it was supposed to be testing. Writing the rubric first is the step almost everyone skips, and skipping it means you discover what you wanted only after you are staring at something you do not like. Write yours before you write a prompt. It is an hour that saves a month.
- 02Grounding layers with citations attached.
In the Grounded Simulation architecture, persona traits anchor to personality psychology, culture anchors to a published set of dimensions applied as group-level response-style priors, and stances anchor to adoption theory. Each layer is somewhere a claim can be traced back out. The alternative, describing the persona in prose and hoping, is what the field mostly does; in my testing it worked as a volume knob, changing how much the model said rather than what it knew.
- 03Blindness over instruction.
You cannot prompt away sycophancy; asking a model to be unbiased is asking it nicely. What works is isolation: the simulated participants never see the researcher’s hypotheses, so there is nothing to helpfully confirm. This is the research-agent equivalent of sandboxing. You do not instruct the code not to touch production; you take production away.
- 04Stance diversity as a first-class control.
A room of agreeable synthetic users converges, and convergence looks like a finding. Force disagreement into the population by construction, at the cohort level, so no study can collapse toward a dominant view. If you build only one control, build this one.
- 05An audit journal.
Every intermediate artifact stays addressable: the prompt, the retrieved context, the raw transcript, the coded segment, the theme, the sentence in the report. If a stakeholder asks where a claim came from, that is a lookup, not an archaeology project.
- 06Approval gates where the stakes change.
Not everything needs a human. The moment a described pattern turns into advice does.
04How do you know the harness is working?
You version it, and you report the version with the number.
This is the discipline worth importing from coding-agent practice into research tooling, and you can adopt it before you have anything else in place. Coding teams already accept that a benchmark result is a property of model plus scaffold. Research tools are still sold as if the quality lives in the model. It does not. The Grounded Simulation preprint reports the spread from holding one model fixed and varying only the structure: a bare prompt surfaced most of the findings expert researchers had published, but buried them in roughly 141 themes per study at 6.7% precision, sixteen noise themes for every real one. The full harness recovered 75% of the expert findings in eighteen focused themes at 42% precision. The paper’s own summary of that result is the sentence I would put on the wall of anyone building in this space.
Recall is nearly free; precision is the architecture.GROUNDED SIMULATION, SSRN 6503241
It is a preprint, not peer reviewed, and I would rather say that plainly than let the numbers do work they have not earned yet.
The same paper’s second lesson is the uncomfortable one: measured quality is instrument-relative. A different scorer can erase the gap between architectures.
Fidelity claims travel only with the instrument that produced them. That is the instrument rule, and it is the whole reason the journal exists.
Which means an evaluation harness that does not disclose its instrument is not an evaluation; it is a preference.
What I still do not have solved: reporting over time. Every meaningful harness change invalidates the comparison to last month’s number. Coding benchmarks handle this with versioned suites and frozen reference stacks. Research evals mostly handle it by quietly not mentioning it.
05What happens next in this space?
Harness engineering for research agents will end up looking like a lot of unglamorous plumbing: writing rubrics, keeping journals, adding gates, calibrating simulators, declaring which version ran. None of it is the part people want to demo. All of it is the part that decides whether a research agent is a tool or a very confident random number generator.
The papers above are worth an hour each if you are building in this space. But the practical starting move is smaller than it sounds. Pick one claim your system produced last week and try to trace it all the way back to a source line. Whatever stops you first is your next substage. That exercise costs twenty minutes and it will tell you more about your pipeline than a week of reading, this post included.
The wider thesis lives on my research page, the architecture is explained in Grounded Simulation and auditable AI research, and the Articos build is written up as a case study.
What would you want declared alongside a research agent’s quality score before you would let it into a real decision?
06References
- “LLM Agent Performance” Is Not a Single Evaluation Target
- Mind the Sim2Real Gap in User Simulation for Agentic Tasks
- Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
- User Simulation in the Era of Generative AI
- Harness engineering for coding agent users
- Harness engineering: leveraging Codex in an agent-first world
- The Anatomy of an Agent Harness
- Agent Harness for Large Language Model Agents: A Survey
- A Philosophical Introduction to Language Models
- The Symbol Grounding Problem
- Grounded Simulation: Task-Preserving LLM Personas for Synthetic User Research
- awesome-harness-engineering
07Further study
Branching by what you are actually trying to do next.
If you are about to trust a number from an agent benchmark.
- “LLM Agent Performance” Is Not a Single Evaluation Target — why one score names at least three different questions; declare what you held fixed.
- Agent Harness for LLM Agents: A Survey — the vocabulary the engineering side already uses; six-component harness anatomy.
If you are building or buying simulated users.
- Mind the Sim2Real Gap in User Simulation for Agentic Tasks — puts a fidelity index on the problem rather than arguing about it; read first.
- Lost in Simulation — the failure is not uniform: worst on AAVE and Indian English speakers, a fairness problem before a methods problem.
- User Simulation in the Era of Generative AI — the map of the wider field, where “user simulation” is used properly.
If you want the engineering practice rather than the research.
- Harness engineering for coding agent users — the harness as governor; the clearest single framing.
- Harness engineering: leveraging Codex in an agent-first world — the most concrete account of working depth-first with agents.
- The Anatomy of an Agent Harness — the best component breakdown. All three are about coding agents, which is exactly the gap.
If you are interested in why grounding is the word.
- A Philosophical Introduction to Language Models — where the philosophical argument sits now.
- The Symbol Grounding Problem — short; worth reading directly rather than through summaries, this one included.
If you want to argue with me. The strongest objection to everything above is that traceability is not validity: a claim can trace perfectly to a real transcript line and still describe nobody. I have not solved that, and the reporting problem in the previous section is the smaller version of it. If you have a working answer, I would rather hear it than not.