For the owner who has started building agents and has walked into the wall where the thing you need to write down is the thing you cannot see.
The short answer. Because you cannot read your own decision rules off. You can only reconstruct a plausible story about them afterwards. Surgeons describing a procedure they perform constantly matched the correct protocol on 31 per cent of its steps when unaided. Interview yourself and you get the story. Build from logged decisions and outcomes instead.
The obvious fix is better documentation, write the rules down properly and hand them over. That fix fails, and it fails for a reason that has been measured rather than argued about.
Building an AI system has an infuriating way of walking you straight into your own limits. I have been sitting with my own decision logic for a few weeks, trying to get it into a shape an agent could run, and it is not easy to squeeze twelve years of experience and why I do a thing and why I do not into something that looks like a sensible model. A lot of it happens on automation by now and I am not aware of it any more. I know what I would do with a slow-moving line in week 38. I could not tell you the rule.
So I complained to the machine about this, which is slightly absurd, and it told me kindly that yes, this is how it is, and there is lot of research on it. The thing I am trying to build knows more about why I cannot build it than I do.
Already in 1977 it was written down properly. Nisbett and Wilson reviewed the evidence and concluded there may be little or no direct introspective access to our own higher order cognitive processes. Asked why we did something, we do not go and look. We assemble a plausible explanation out of what we believe generally causes decisions like this one, and then report it with real confidence, because from the inside it does not feel assembled.
Which is exactly what I do. I waffle on about how I looked at the season obviously, and the margins, and the cashflow, and the targets, and then decided to do x, when actually I was on autopilot and a bit short on coffee, making fast calls on the feeling of that afternoon. Michael Polanyi got there earlier, on page four of The Tacit Dimension in 1966: we can know more than we can tell.
The gap gets uncomfortable once somebody measures it instead of philosophising about it. Asked to describe a femoral artery shunt, a procedure they knew well, experienced surgeons agreed with the gold-standard protocol on 31.25 per cent of the steps. They left out about 69 per cent of their own procedure, and not out of carelessness. Automated knowledge runs faster than conscious thought and outside its view, so it is not there to be reported on.
| Source | What it measured | The number |
|---|---|---|
| Polanyi, The Tacit Dimension, 1966 | The shape of skilled knowledge | "We can know more than we can tell", page 4 |
| Nisbett and Wilson, Psychological Review, 1977 | Whether people can report the causes of their own responses | Reports come from plausible causal theories, not from introspection |
| Clark and others, Journal of Surgical Research, 2012 | Surgeons describing a procedure they perform regularly | 31.25% agreement with the protocol, ie about 69% of steps left out |
| Kahneman and others, Harvard Business Review, 2016 | Underwriters pricing the same cases | Median difference of 55%, where executives expected 10% |
I have been in consumer business the same twelve years, and this is the first thing you learn there.
Ask consumers what they base their buying decisions on and environmental values come up high. Watch what they actually do and easy, fast and perceived value wins almost every time. Harvard Business Review put a number on the gap in 2019: 65 per cent said they want to buy purpose-driven brands that advocate sustainability, and about 26 per cent actually do. That article quotes the survey without naming it, so treat the two figures as the size of the gap rather than as a measurement. The direction is not in doubt and any product owner reading this has seen it in their own sales data.
So I have spent over a decade not believing what people say about their own decisions, and then sat down to build an agent by asking myself about mine. The method I would never accept from a customer survey was the one I reached for first.
Every article telling you to make your decision-making explicit skips this. Anthropic's alignment team published a paper in May 2025 that fed reasoning models a subtle hint about the answer to a question, then checked whether the model's written reasoning admitted using it. The models used the hints, and in most settings tested the stated reasoning mentioned the hint in under 20 per cent of the cases where it had been used. The unfaithful explanations ran noticeably longer than the faithful ones, which anybody who has sat in a board meeting will recognise.
So there are two of us in the room reconstructing our reasons after the fact, and only one of us is being paid for it. You cannot get the decision logic out by asking me why. You cannot get it out by asking the model why either.
In 1970 Lewis Goldberg built simple linear models out of clinicians' own judgements and then used the model in place of the clinician. The model predicted better than the person it was copied from, for 26 of the 29 judges. Kaufmann and Wittmann pooled 52 tasks across 35 studies in 2016 and found the same direction holds broadly.
The mechanism is not that the model understands you. It keeps your policy and throws away your Tuesday. Kahneman and his co-authors ran a noise audit at an insurance company in 2016 and found that when underwriters priced the same cases independently, the median difference between them was 55 per cent, where the executives had expected about 10. Same guidelines, same role, same case. The variation was the mood, the hour, the coffee.
Will the agent be as good as me on my best day? No. It will be better than me on my worst one, and there are more of those than I care to count. That is a duller promise than the one being sold in most AI writing, and it is the one that survives contact with a real business. The argument about what protects your business when everyone has the same AI lands in the same place: the process knowledge is the asset, and it is the hardest thing to get out of a head.
I am logging the decision and not the reason. One line each time: the date, what I decided, what I was choosing between, and what I expected to happen. The why stays out, because the why is the unreliable part and writing it down gives it an authority it has not earnt.
When I do interrogate myself I am going at one specific past decision, never at the policy. This is the one piece of method with forty years of practice behind it. Klein, Calderwood and MacGregor published the Critical Decision Method in 1989, a retrospective interview that probes a real incident with specific cues instead of asking for the general rule, and it has been used on fireground commanders, paramedics and structural engineers since. So I ask myself about the order I placed in March. Not about how I place orders.
Then I go back and write down what actually happened, date-stamped, later, when I know. A decision with no recorded outcome teaches nothing and cannot be modelled, and the pairing is the only honest test of whether my stated reasons were ever right. Most of the value arrives about six months after starting, which is why almost nobody does it. Same point as the purchase decision itself, where the money is committed months before the market answers.
No, it is an argument about which artefact to spend the time on. Documentation of what you intend to do is worth having and is not the same as a record of what you did. The first is a policy. The second is evidence.
Longer than a weekend and shorter than a rebuild. The agent has to observe the actual work first, and ask you what you were thinking at particular moments, and the quiet knowledge gets documented over months. I am a few weeks in and I have a log, not a model.
It can read what you sent. It cannot see what you chose against, and the alternatives you rejected are most of the decision. That is the gap a log closes and an inbox does not.
It will, and that is the argument for recording outcomes alongside decisions instead of recording reasons. A copy of your policy with the noise removed still has your policy in it, mistakes included. What the log gives you is sight of which of your rules actually pays, which is information you have never had about yourself.
At the decisions that are not repeated. Pricing a new category, firing a supplier, deciding whether to take an account at all. The agent gets the recurring calls, where the variance is quietly costing you money, and the argument for what AI is actually for in a product business is the same argument at a different altitude.
Anyway. Back to writing down what I decided on Monday and what I was choosing between, which is a duller job than building an agent and is apparently the job.
This world is not finished yet either.
Richard E. Nisbett and Timothy D. Wilson, "Telling more than we can know: Verbal reports on mental processes", Psychological Review 84(3), 1977, 231–259. Michael Polanyi, "The Tacit Dimension", 1966, page 4. Richard E. Clark, Carla M. Pugh, Kenneth A. Yates, Kenji Inaba, Donald J. Green and Maura E. Sullivan, "The use of cognitive task analysis to improve instructional descriptions of procedures", Journal of Surgical Research, 2012. Katherine White, David J. Hardisty and Rishad Habib, "The Elusive Green Consumer", Harvard Business Review 97(4), July–August 2019, 124–133. The 65% and 26% figures are quoted there from a survey the article does not name. "Reasoning Models Don't Always Say What They Think", Anthropic Alignment Science, arXiv:2505.05410, May 2025. Lewis R. Goldberg, "Man versus model of man", Psychological Bulletin, 1970. The 26 of 29 result is as reported in the bootstrapping literature. Esther Kaufmann and Werner W. Wittmann, "The Success of Linear Bootstrapping Models", PLOS ONE 11(6), 2016, e0157914. Daniel Kahneman, Andrew M. Rosenfield, Linnea Gandhi and Tom Blaser, "Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making", Harvard Business Review, October 2016. Gary A. Klein, Roberta Calderwood and Donald MacGregor, "Critical decision method for eliciting knowledge", IEEE Transactions on Systems, Man and Cybernetics, 1989. Jen Stave, Ryan Kurt and John Winsor, "Teach Your AI How You Make Decisions", Harvard Business Review, 25 June 2026, cited as the framing this article argues with. All retrieved 12 September 2026.