Essay5 April 2026

In which the rule outlives the reason.

Version Eleven

"Never Active products don't indicate risk."

The sentence entered a system prompt on an afternoon. An LLM response had triggered an alert that made no sense, the kind of technically correct signal that collapses under the smallest amount of scrutiny and institutional knowledge. The model had identified language that correlated with risk and applied that pattern faithfully, even though the customer had never adopted the specific product in the first place. I pulled up the underlying record, scrolled through a transcript and other data looking for the moment that had confused the system, and the conclusion arrived quickly. The model wasn't exactly wrong about the language, but the framing of the problem was off. A customer who never started using something cannot meaningfully be described as about to stop. The model was missing context I took for granted.

So I wrote the rule and committed it to the prompt.

That prompt lives in a system I named TARS, after the Interstellar robot, because I could. Every fifteen minutes a cron job wakes up, analyzes five accounts, and goes back to sleep. It pulls from everything: CRM data, call transcripts, chat logs, meeting notes, issue trackers, a PM tool. Quietly, continuously, across the full portfolio.

Everything surrounding that rule has already begun to fade. The particular transcript that triggered the problem, the order in which the logic became clear, all of it dissolves into the background noise of dozens of similar afternoons. Only the rule remains intact. Every transcript the system analyzes now encounters that sentence, quietly adjusting the model's interpretation before the rest of the analysis unfolds.

That persistence runs against how organizations usually forget. A teammate leaves, a document gets archived, a wiki page falls out of date, and the reasoning that once guided a decision slowly dissolves into rumor. Months later the same question surfaces and someone reconstructs the answer from scratch, unaware that the organization solved it once already. Here the reasoning evaporates the same way, but the rule survives it, the conclusion outliving the argument that produced it.

And rules like this don't only accumulate in the prompt or memory file. They hide inside the system's architecture, in decisions that looked like engineering but were actually beliefs. The Slack sync originally processed account-related channels in alphabetical order.

That doesn't sound like the beginning of a disaster. But with hundreds of channels and API rate limits, the sync would reliably run out of time somewhere around the letter M, every single run. Accounts whose Slack channels happened to start with N through Z were structurally underserved, not occasionally or randomly but systematically, as a feature of the alphabet. I only caught it after noticing that some accounts consistently had rich Slack context in their analyses while others had none, and the dividing line was, absurdly, the first letter of the channel name.

The fix was a rotation cursor, a few lines of code that pick up where the previous run left off. The lesson underneath it stayed: any system with bounded execution time and ordered processing will invent invisible hierarchies. Alphabetical ordering feels neutral but isn't. The hierarchy it created was perfectly silent, distinguishable from normal operation only if you already suspected something was wrong.

The most surprising version of the pattern was in the prompt's persona instruction. The prompt tells the model "you are the world's best [x], and your job is to [y]," where [x] is a particular role in a company and [y] is a judgment call. For months I assumed this was a tone instruction, a way to make the output sound less like a help desk and more like a colleague. Then I compared the outputs side by side, and the difference was larger than tone.

Without the "world's best [x]" framing, the model hedges. Assessments arrive padded with qualifiers: potential risk factors, possible concerns, indicators that may suggest elevated attention. These sentences are technically defensible and practically meaningless. Someone reading "there may be potential risk factors" across forty accounts learns exactly nothing about which ones need attention this week.

With the persona framing, the model commits. It names the evidence, identifies the source, and makes a call about what matters. My best explanation is that the persona creates pressure toward specificity. A model that hedges everything isn't performing the role the prompt describes, and it seems to resolve that tension by grounding itself in concrete evidence rather than retreating into abstraction.

The instruction that most shaped how the system reasons turns out to be a sentence about who the model is. Telling it that it occupies a role does more work than telling it how to weigh information, because the role implies a stance toward the information that no list of rules can specify directly. A skilled practitioner doesn't hedge across forty accounts, and the model, reading itself as one, stops hedging too. Which means the most consequential governance choice in the system, the one that determines whether anyone trusts the output enough to act on it, was a sentence I wrote without recognizing it as governance at all: a few words at the top of a file that the model takes as constitutive of how it should think rather than how it should sound.

Inside the file, lines accumulate gradually. The filename gives the game away:

2026-02-21-v11

Over time the prompt resembles sediment more than design. Buried in it are fragments of conversations the organization has already forgotten: an account misread as at-risk, a transcript mentioning a competitor that turned out to be harmless, a product adoption metric missing from a regional account that once triggered a cascade of false alarms. Each line compresses the outcome of a discussion that once occupied a room full of people.

Rules behave differently from principles in that respect. Principles carry a lineage. Someone can usually reconstruct the argument that produced them. They have parents. Rules accumulate through unexpected encounters instead: an edge case appears, the system reacts poorly, and a constraint enters the prompt to keep the mistake from repeating. The only evidence that the lesson ever mattered is a lingering reluctance to delete the rule.

Scar tissue is closer to the right word than memory. Each rule marks a place where the organization's assumptions turned out to be incomplete, and protects the system from reopening the same wound twice. It also conceals the original injury. The scar tissue isn't confined to the prompt, either. The rotation cursor, the persona instruction, the cooldown windows: each is a rule embedded in architecture rather than language, harder to find and harder to question because it presents as engineering.

TARS wasn't designed as the system of record for how the organization thinks. These things rarely are. Someone builds a tool to solve a specific problem, the tool accumulates context, the context layer grows, and at some point the system crosses a line nobody quite notices. It holds institutional knowledge that exists nowhere else, not in the CRM, not in anyone's head, not in documentation. It becomes the company's context brain without anyone formally deciding it should.

By the time the prompt reaches version eleven, most of the scenes that produced its instructions have disappeared. The model still behaves correctly because the rules execute every time the system runs, but the reasoning that gave those rules their meaning has dissolved. What remains is a system that remembers certain operational lessons more faithfully than the organization itself does, applying them hundreds of times a week without hesitation or explanation.

There are lines in the prompt I no longer fully recognize, instructions that feel right the way all caution feels right, without being able to reconstruct the scene that made them necessary. The obvious response is to audit the file and delete what no longer applies. I keep not doing it, and the reason isn't really avoidance. The cost of auditing is structural. To safely remove a rule I'd have to recover the context that produced it, and recovering lost context is precisely the labor TARS was built to spare me from. The system exists to make that reconstruction unnecessary, which makes the audit a choice to do by hand the one thing the machine was supposed to absorb. Set against work that visibly needs doing, disturbing a quiet system that runs without complaint never wins the trade. So the scar tissue protects itself: the rule survives because the conditions for questioning it are the exact conditions the system was designed to eliminate. The next person to read the prompt encounters that rule as part of the system's apparent design, indistinguishable from instructions written yesterday with full context and conviction.

The model underneath the rules has changed too. Every constraint was calibrated against a particular version's tendencies, and the version that made the original mistake may no longer exist. Some of these rules might be correcting for failure modes the current model doesn't have, while staying silent about new ones it does. The scar tissue protects against an injury the body can no longer sustain.

What keeps the oldest rules in place, finally, is that removing one means knowing something the organization has forgotten how to name. The rule might be protecting the system from a mistake nobody remembers making.

Or it might be a ghost.

Footnotes

Most of the prompt rules I've written don't correct the model's reasoning so much as redefine the boundaries of the problem it's trying to solve. The model was right about the language, but the question was wrong. No amount of pattern recognition fixes a misframed question.

The constraint TARS was built to address was never analytical capability. Anyone managing a book of business will, on a good week, actually review call transcripts for maybe eight of their forty accounts. The other thirty-two get whatever the person can hold in working memory from the last time they looked, supplemented by CRM fields that may or may not reflect reality.

Multiply this across a team of ten and the math gets ugly fast: at any given moment, the organization's understanding of most of its revenue is weeks stale and selectively remembered.

Not because anyone is lazy. The day fills up, and the transcripts keep coming, and there are only so many hours before the next call.

There are quieter versions of this same pattern elsewhere in the system. Some risk dimensions refresh every seven days, others every fourteen, with shorter windows for accounts that carry more weight. There's also a 24-hour cooldown that keeps the system from reanalyzing accounts it just touched, unless genuinely new evidence has appeared or a critical date is within thirty days. Each window is a belief about how fast a particular kind of signal can change, written down once and applied hundreds of times a week without revisiting the question. The timing of the analysis matters as much as the analysis itself: an overnight run meant people got Monday-morning assessments generated Saturday at 2am, and they treated those the way you'd treat a Monday-morning email sent at 3am.

Older enterprise systems often accumulate logic in the same way. Deep inside billing platforms or CRM automation flows sit conditions that begin with phrases like "temporary workaround" or "special handling." Years later the surrounding context disappears while the logic remains, quietly redirecting transactions no one remembers debugging. Engineers sometimes refer to these sections of code as defensive layers: protections against edge cases the system once encountered and never wants to encounter again. Prompt files are beginning to develop the same geology, except what's being defended is a judgment call, not a calculation.

Individual prompts, and the context and rules around them, are written in isolation, months apart, for different problems. Whether they still make sense together is a question the system can't ask of itself.

I've not figured out how to systematically check it. Two reasonable constraints written for different accounts in different quarters can quietly contradict each other in a context their author never anticipated.

The system also can't tell the difference between a quiet account where everything is fine and a quiet account where the customer has already made some decision we'd want to understand. Silence looks identical in the data regardless of what's producing it. Absence of signal isn't the same as absence of risk, and the model handles that ambiguity unevenly.

This one nags at me, because I'm not sure the scar tissue model accounts for it. You can write a rule for a pattern you've seen before, but not for a signal that doesn't exist.

The system is designed to extend easily (templatized views, an MCP server, a shared context layer), so every new use case inherits the full accumulation of rules and embedded beliefs from the original one. Someone asks the system a question nobody anticipated, and the answer arrives pre-shaped by judgment calls encoded months ago for a different purpose.

I built a page that exposes these rules so anyone can read them and suggest changes. In theory this should solve the problem. In practice, people look at a rule like "Never Active products don't indicate risk" and either leave it alone because they don't understand it or propose removing it because they don't understand it. They can read every rule the system applies, but they can't reconstruct the afternoon that made each one necessary.

Transparency and understanding turn out to be different things.


Reply

I’d welcome your thoughts on this essay. Send me a note →

Related reading
Latest entries