Less Expertise, More Coverage
We assume the way to get a better analysis out of a model is to make it an expert. My research found the opposite can happen: the expert framing quietly shrinks the answer space the model is willing to explore.
Casting a model as a narrow domain expert can shrink what it treats as in-scope. The vocabulary changes, and so does the set of considerations the model thinks belong in an answer at all. Ask for a senior tax attorney, and you get the paths a senior tax attorney walks along with the branches a senior tax attorney learned years ago to stop walking down. On analytical work where the expensive failure is a missed branch rather than a shallow one, a broad framing often covers more of the real answer space than the specialist costume does.
That claim comes out of a study of mine, Less Expertise, More Coverage: The Counterintuitive Effects of Prompting LLMs for Analytical Tasks. The study no longer stands alone. I folded it into the Grounded Simulation paper, and the merged paper is under peer review. Until review clears, treat the rest of this as a claim with a test attached rather than a finding you can cite. The test is the part worth your time this week.
01What the framing actually changes
Pruning is most of what expertise buys a person. A specialist is defined partly by what they have learned to stop considering, and that is an excellent trade when someone has forty hours and a deadline. A model is not short of time in the same way. So when you ask for the specialist, you buy the pruning without needing the thing the pruning was paying for, and what you pay is the branches that never get named.
Tone moves in the other direction, which is what makes this awkward to catch. Expert framing does sharpen the prose. Answers come back more assured, better organised, easier to forward to a colleague. None of that is evidence about scope.
The most authoritative version of an answer can be the least covered one, and nothing on the page marks the regions it never entered.
Reading harder will not fix that, because the signal you would need has been removed from the artifact you are reading. Coverage has to be measured against something outside the answer.
02Run the comparison yourself
Pick one analytical task from your own work where the answer space can be written down. What could go wrong with this migration. Which clauses in this agreement create obligations. Which failure modes this feature introduces for support. The task has to be one where you would know a missing consideration when you saw it.
Build the answer key before you prompt anything. Two people list the considerations a complete answer needs, working separately, then merge the two lists into one. That merged list is your denominator, and without it you are scoring impressions. Getting it wrong in the generous direction is fine, because both framings are scored against the same key.
Now run two prompts against the same model on the same day: one casting it as a named specialist, one asking a capable generalist to map everything relevant to the question. Hold the output length budget equal, since a longer answer covers more for reasons that have nothing to do with expertise. Hold temperature and sampling settings equal too, and take three runs of each so you are not reading a single sample.
Score one thing: what fraction of the answer key each framing named at all. Named or not named, no partial credit for elegance. Do not score which answer reads better, and do not score which one you would rather send to a client, because polish is precisely the variable the specialist framing wins and precisely the one that will mislead you. Separately, keep the items each framing raised that were not on your key. Some of those are branches you missed, some are noise, and the ratio between them is worth knowing before you trust either prompt.
Ten tasks is roughly where your own result starts to mean something. Fewer than that and you are looking at task variance.
What would prove this wrong is simple to state. If the specialist framing matches or beats the broad one on answer-key coverage at equal output length across your tasks, then for your work the claim is false, and your tasks are a better guide to your work than any paper of mine. That result is more useful to you than agreement, and I would want to hear which kind of task produced it.
03Where the breadth and depth story breaks
Breadth against depth is the tidy way to hold this, and it is only approximately true. The framing moves what the model treats as in-scope. Depth is not automatically what you get back in exchange. On a task whose answer space is genuinely small, narrowing costs nothing at all and the specialist framing reads better for free. On others you get a narrower answer that is no deeper, only more confident, which is the worst version of the trade because it looks like the best one. Treat scope and depth as two properties you measure separately rather than as two ends of one dial.
There is a second limit worth naming. Coverage is not correctness. A framing that names more branches can still be wrong about every one of them, so a coverage win tells you which prompt to use for mapping a problem and tells you nothing about whether to trust the content.
04What this changed in practice
On the research platform I work on at Articos, early hypothesis generation turned out to be a coverage problem before it was a depth problem. The rule that came out of it is portable: keep the wide pass and the deep pass as separate stages with separate prompts, and put a person in between to decide which branches earn the deep one. Specialist framing is useful once a branch has been chosen. While the branches are still being found, it is quietly deciding for you.
That decision only stays honest if you can see it. If your pipeline records which branches a run explored rather than only what it concluded, as the audit chain in auditable AI research is built to do, a narrow confident answer stops being able to pass as a covered one. Designing that in from the start, instead of prompting your way around it later, is most of what Grounded Simulation is for.
None of this needs you to believe the expert prompt is bad. It needs you to stop treating it as free. If you run the comparison, the number I would find most useful is not your average across tasks. It is the one task where the specialist framing covered more, and what that task had in common with the work you actually get paid for.
05References
- Grounded Simulation: Task-Preserving LLM Personas for Synthetic User Research
- When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
- Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
- When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs
- In-Context Impersonation Reveals Large Language Models' Strengths and Biases
- Better Zero-Shot Reasoning with Role-Play Prompting
- Understanding the Effects of RLHF on LLM Generalisation and Diversity
- Generative AI enhances individual creativity but reduces the collective diversity of novel content
- Why Good Thoughts Block Better Ones: The Mechanism of the Pernicious Einstellung (Set) Effect
- State of What Art? A Call for Multi-Prompt LLM Evaluation
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
06Further study
Branching by what you are actually trying to do next.
If you are deciding whether the expert persona stays in your system prompt
- When "A Helpful Assistant" Is Not Really Helpful — the largest negative result: 162 roles across model families, no reliable accuracy gain.
- Claude prompting best practices: give Claude a role — the canonical vendor advice this essay argues you should stop treating as free.
- Expert Personas Improve LLM Alignment but Damage Accuracy — routes personas by task intent instead of defaulting to one expert, a middle path if you cannot drop the costume.
If you are running the comparison before trusting anyone's claim
- State of What Art? A Call for Multi-Prompt LLM Evaluation — why one prompt and one run is an anecdote, not a measurement.
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design — formatting alone swings benchmark scores, the reason the protocol holds every knob equal.
- Can LLMs Express Their Uncertainty? — score the answer key, never the model's stated confidence.
If you are asking why the narrowing happens at all
- Why Good Thoughts Block Better Ones — the human mechanism: in chess masters the first familiar solution blocks the search for better ones.
- Understanding the Effects of RLHF on LLM Generalisation and Diversity — the training mechanism: alignment cuts output diversity before any persona is added.
- Generative AI enhances individual creativity but reduces the collective diversity of novel content — the population-level cost, measured in human writers rather than models.