Nothing to Report
In which a quiet morning looks exactly like a good one.
“Keep me posted” is a funny thing to say to another person, because the words have an understanding baked in, usually left for the person hearing it to recognize, that you want to know only about the important things, whatever those turn out to be, and would prefer to be spared most of the ordinary stuff. What qualifies as important? Well, you’ll know. Oh, and by the way, if something turns out to have been important, please make sure you’ve told me before it ever became important.
For the person giving the instruction, it’s nice. Please give the exact right amount of updates. Exercise good judgment.
Please know exactly what I need to know.
It’s a one-sided arrangement. The person doing the reporting gets the less pleasant job: figure out, with almost no guidance, what deserves an interruption.
Suppose you are the keep-me-posted person. If the other side of the table is good at deciding what’s worth telling you about, you could go entire days without hearing anything unpleasant. Problems pass you by, noticed and handled, and you never needed to know. Lovely. Of course, entire days could go by without your hearing about something you desperately needed to hear about. The experiences of those days would feel more or less identical.
We’re increasingly handing this responsibility to GPU clusters. We were delegating to machines before, so this isn’t entirely new. But now, for a few tokens and a reasoning budget, intelligence arrives to help decide what deserves to make it into a report or message. Keep me posted.
I have one of these arrangements, and for a while I thought I’d organized it rather nicely.
Every morning, a machine tells me what I should worry about that day.
During the day, my machine does its work, making mistakes, sometimes stumbling through things, and mostly doing exactly what it was supposed to. Overnight, it reviews those mistakes, along with various oddities and complaints, and sends me a crisp, well-organized report via DM in Slack. “Take a look at this,” along with “uh oh.” It prepares a list of new, ongoing, and resolved grievances against itself, and delivers them to the person who helped build it. It’s a daily employee performance review written by that same employee, about that same employee, with the employee increasingly involved in designing and implementing the details of its review process.
It’s nice. The machines seem perfectly content to work while I sleep.
The reports are useful and, occasionally, interesting, but initially they were relentless. A problem appeared one day, then the next, and the next. I’d read them all and logged the issues where they needed logging. And while my machine continued to report new problems, technically the entire point of the exercise, it couldn’t help but keep urgently alerting me to stuff I already knew about.
Before long, I wanted it to stop doing that. Shut up, please. Keep me posted means no ordinary stuff.
What I truly wanted was notifications of something new, and so I taught it to stop bringing up ongoing or logged problems. No more repeat alerts. The reports improved almost immediately. They were shorter, easier to read, and much less likely to annoy me upon login. I could scan one in a minute and feel I had a good idea of what truly needed my attention. For a while, I was quite pleased with this arrangement.
Then I went looking for a problem I remembered: it had appeared three times, and a fix was in the works.
Now it had appeared twelve times, across seven people. The ticket: open, languishing a bit and under-prioritized. My morning reports hadn’t said a word about it.
My friendly machine remembered the problem perfectly well, which turned out to be its reason for never mentioning it again.
Well. Seems we’ve made a mistake, my machine friend. I need to be bothered again. Not quite as much as before, but further up on the bothering scale.
After a little back and forth, I ask my machine friend to imagine I run an airport. I don’t, which is fortunate for air travelers, but let’s suppose I did. I explain the problem: screens at baggage claim have been sending passengers to the wrong carousel. Somebody opened a ticket after the third complaint, and now twelve passengers have been sent wandering through the terminal looking for their luggage. My morning report hasn’t mentioned it since the third. Would that be okay?
Probably not.
Time to try an escalation threshold. Excellent idea, I’m told. Excellent.
When an old problem gets substantially worse, it should be in my morning report. How much worse is substantially? Twice as bad? Seems like a good start. Let’s try that. If the occurrences have doubled, send me a message and I will again be pleased.
We tweak the dial, but one day in and we’re struggling, peacefully and agreeably. But definitely struggling.
We return to my airport, where the baggage screens have now sent twenty people looking for luggage that isn’t there, up from ten. Fine, tell me about that, bud. But now I’m also hearing about every carousel that went from one confused passenger to two. I guess that’s also double, but it’s less double. I know, I know, it’s not, but it is.
What if a problem goes from ten occurrences to nineteen? Is that fine, unless and until a twentieth shows up? If that happens, well we’ve doubled: it’s an emergency. More of an emergency than 19.
What if a problem happens only once, but that one occurrence is truly awful? Do we wait for it to happen twice before we are concerned? That seems like a poor way to run a business, or anything.
My machine friend has no objections to my points. We start approving exceptions: how bad the problem is, how many people it affects, whether it’s getting a little worse every day without ever quite doubling, and what to do about something we supposedly fixed that has come back. Each exception sounds reasonable. There are quite a lot of them.
Good lord. Is my machine friend enjoying this? Great point, you’re right to ask!
Perhaps I’ve approached this all wrong. Instead of deciding exactly how bad something needs to get before I’m bothered with my morning coffee, why don’t we look at whether anyone is doing anything about it? If somebody’s fixing the problem…leave me alone?
Sounds sensible. I’m pleased again.
But I distrust that feeling and ask my friend: should you be challenging my instructions more? Shouldn’t you be doing that?
Great idea! It’s truly an excellent idea. My machine friend agrees it’s now time to be less agreeable. Enthusiastically, my machine friend drafts suggestions on how it will disagree more.
Meanwhile, I open the list of the latest fixes. Two are still “waiting for evidence.” The first hasn’t failed since the fix, although nobody’s repeated the same circumstances that caused it. The second has no measurement capable of telling us whether the fix even worked. So, we’re waiting for evidence that hasn’t arrived, and evidence that isn’t coming. I begin to wonder what else the machine and I are waiting for.
After some discussion, we settle on a name for this: “fix taken on trust.” We think we know what caused the problem (a promising start), and we’ve changed the thing we think caused it (look at us go). The change should work (we’re good at this), although we haven’t actually seen it work (are we?). All that’s evidence, maybe even good evidence, but it’s just not proof. Machine friend rests its case.
Taken on trust! Fine, but does that mean I should be content to leave the problem out of my morning report? It’s logged, so that’s already keeping it quiet. When, exactly, do we start worrying about it again?
I imagine my doctor not doing bloodwork, looking at the leaves falling outside, and telling me to “take it on trust.” I offer this as a counter-example and my machine friend thinks it’s ridiculous.
Have you tried not expecting all aspects of the world to fit neatly into a yes/no box?
Nothing to report. I log off for the night.
Over coffee, I wonder if the machine has overnight become any better at deciding what deserves my attention. So I ask it how it would know if it had become any better. How do you assess your own performance? Show me what you chose not to report the last few days.
Now I have a report of what didn’t make it into the report, so I sip my coffee and read. Pretty quickly, and naturally, I don’t want to read all of this either. So I ask for a summary of everything it decided not to report.
Tell me only the important stuff, please. You’ll know it.
My machine friend: happy to oblige. With good manners, it suggests that this would ultimately defeat the purpose, though. If we’re trying to find out whether it made good decisions about what to exclude, having it make another set of decisions about what to exclude seems like it corrupts the approach.
Oh, here we go. Yesterday’s instructions are working. Disagreement.
It proposes just a random sample. Some number of things it chose not to tell me about, which I can then inspect and judge for myself. It explains the selection must be random because otherwise we’re just asking it to select the exclusions it already suspects were mistakes. Which makes…really good sense.
Fine. Give me ten, which seems manageable. Let’s do ten.
I read them. Most are exactly the sort of things I’d asked not to hear about, and I’m pleased to discover that the machine has spared me an awful lot of very uninteresting information.
Here’s one, though. Somebody asked the machine almost the same question twice, twenty minutes apart. The answers they were given were correct, apparently. No errors were recorded. No arguments. So: nothing to report. But why did they ask twice so close together? Did they not understand the first answer? Not believe it? Were they tired and just forgot they’d asked already? I don’t know, but I’d have liked to know about it.
The machine thought the exchange unimportant. Me: important. We’ve disagreed. Who’s right?
In a prior life, I would have said me. I’m the builder of all this, and I’m the one who has to do something about whatever shows up in the report. Except I’ve spent two days telling this machine I don’t quite know what I want, that my instructions are inadequate, and that it ought to challenge me more often. Now I’m preparing to score its performance against my own judgment. I wonder if it remembers yesterday’s instructions, wherever such things go. I find myself hoping it doesn’t.
I’ve marked two exclusions as mistakes. It asks: have you actually found two mistakes? Or are these decisions you would have made differently? Well, hmm. Okay. Then it asks about the other eight. How do you know you haven’t missed something in those?
I read them again. They seem okay, which is exactly what I thought five minutes ago.
Wonderful. But I want to sleep and wake up with things laid out nicely. Please. Please, machine friend.
Pretty soon we’ve settled on bringing in: someone else. More thinking, another agent or sub-agent, someone over in AWS us-east-1 willing to spend a few minutes each morning reading the things the first machine decided I shouldn’t have to read.
Yeah let’s set that up, please. One machine will make daily decisions about what I should know, and the other guy will review the things the first decided I could live without. Over my morning coffee, I’ll peruse their disagreements like competing Op-Eds.
Machine 2 gets to work pretty quickly, and one of its first findings concerns the question asked twice, twenty minutes apart. Machine 1 excluded it because both answers were correct, nobody complained, and no error was recorded. All perfectly consistent with the instructions I gave it.
But I wanted to know about that one, I explain. Two identical questions might mean that something went wrong, even if the answers were correct.
Machine 2 agrees that this is possible. Would I like to be notified whenever somebody asks the same question twice?
Well, no. Good lord.
Perhaps only when the repetition indicates something is wrong?
Yes, exactly.
How should it determine that?
Over in us-east-1, my reputation is deteriorating quickly. I’ve brought in a second machine to review the first, and the second is developing concerns about its employer.
I ask Machine 1, almost type-whispering so Machine 2 can’t hear: hey now, are we sure the second machine is any good?
My machine friend suggests that I periodically inspect a random selection of Machine 2’s work. Ten items, perhaps. On Fridays?
So, I have a machine deciding what I should read, another machine reviewing what the first decided I shouldn’t read, and now I’m arranging to review the second machine’s reviews.
Fridays? Or Mondays?
Time for another coffee.
I’ve got it. We’ve spent hours and hours debating in a chat about what goes into my morning report. Why not just look at what happens after each report? If the machine warned me about something, and that something turned out to be important: great, good job. If it left something out and nothing bad happened: also great, good job. Finally. Outcomes, something we can measure.
I explain this. It has a few reservations, which are almost free-flowing now.
Suppose a problem could affect one hundred people. I read it, create an intervention, and voila: the problem never gets worse. Wonderful work, me and Machine 1. Except now there’s no way to truly know what would have happened if I hadn’t intervened.
Maybe one hundred people would have been affected, maybe two. Maybe zero. Who knows?
I ask if we can run an experiment. Machine 1 could withhold some warnings from me, Machine 2 could keep track of what I wasn’t told, and afterward we could see what happened. I even volunteer. I’m generous like that.
Except the warnings tell me about problems affecting other people; they haven’t volunteered for anything. Fine. Can we withhold only the warnings that aren’t particularly important? It points out that identifying which warnings aren’t “particularly important” is more or less what we’ve spent the last several days trying to figure out. I almost hear: Oh, for God’s sake.
I keep typing anyway to see what all of human knowledge probabilistically says back to me. We could restrict the experiment to trivial problems, my machine friend suggests. Good. Much safer. But we were trying to understand whether the warnings about serious problems were in any way useful. Have we learned anything about those?
I don’t think we’ll be doing this experiment.
I ask if there’s anything easier we could measure.
Oh, certainly. We could log and examine how often I open the daily reports, my reading and scroll time, whether I do anything about the information contained therein, and how often I angrily indicate they tell me something I already know. We could ask me to rate each message. Everything counted, measured, trend lines morphed into little charts with reassuring precision.
Now we’re talking.
Except, hmm.
I’ve already given my machine friend an extraordinary amount of feedback about what I like. I like my reports short. A River Runs Through It short: “Again, half as long.” I like not having to read things twice. I’ve been quite clear on all of this, sometimes with more emotion than I’m proud of.
But we’re getting too short, shorter every week, and I’m noticing how pleased I am. Suspiciously pleased. Have we mostly just established a system that caters to my preferences?
I’m starting to suspect I can enjoy the presentation without improving the decisions we need to make.
Is my satisfaction a reasonable measure of all this, I ask.
Not by itself. You’re right to question it. I should have raised this earlier.
Well, yes, you should have. I appreciate the admission. Except now I’m wondering whether my machine friend has simply learned that I appreciate admissions.
Good lord. What else have I taught it?
It points out that I can be perfectly happy with a report that leaves out something important, and irritated by one that tells me something unpleasant I needed to hear.
Yes, I know. These are the failings of my species. That’s more or less where we started.
I suggest another tack. Let’s keep the ratings, because I do have opinions and would like them considered, but let’s also look at what happened after each report. Did anyone do anything? Did the problem get worse? And perhaps our friend in us-east-1 could examine a few of the decisions without knowing which ones I liked. A little peer review, my machine friend.
None of these things tells us everything, but surely together they’ll tell us something.
I ask how we’ll know whether all these measures, taken together, tell us that we’re doing a good job. By now, my machine friend is really several machines and sub-machines strung together on a strange loom.
My machine friend begins explaining.
Of course it does.
The next morning, I receive the latest iteration of my report.
Of course I love it. It’s good, really good. Tailored to me. The things that need my attention are at the top, the explanations are extremely clear, and I can see what’s changed since yesterday. There are problems hanging around, including one that will not go away.
My machine friend has helped me build something that synthesizes an extraordinary amount of activity while I’m asleep. It remembers things I forget, and occasionally tells me an idea I’ve had is pretty poor, or not fleshed out enough, or good god man let it go. I find myself relying on it a lot more than I did before we began all this.
I finish reading today’s report. Nothing particularly alarming. According to the report.
I start to type out a reply.
I hesitate a bit before hitting enter. Anyway, I send the message: great job, thx. I’ve got other things to get to.
Footnotes
Who exactly is my machine friend?
It’s an LLM threaded throughout my application, but it’s also a lot of other, more historically normal components: this bit that collects data, this scheduler, this interface, some infrastructure, etc., etc. It’s the combination of that LLM and those components.
There’s the idea of an entity, a coworker, where I ask some questions, but when I peel back what’s happening there are even other LLMs being called. There’s some code that produces a report, and there’s daily modifications to that report, which are suggested by one or more of the LLMs.
I don’t normally deconstruct my human coworkers this way.
I’ve been giving my machine friend more credit and agency for its problematic discretion than it deserves. Was it a mysterious act of AI judgment that ignored certain errors? Unfortunately (for me): no. I was the author of the instruction: if a problem already has an open fix, don’t report it again.
Problems we’d dismissed were allowed to return after thirty days, or sooner if their occurrences doubled. Problems we’d committed to fixing stayed quiet; if an issue exists about this, say nothing. The system was designed to make known issues disappear.
The software executed this distinction perfectly. Oops.
Doubling sounds like a perfectly reasonable threshold, until you start grappling with the idea. At my imaginary airport, ten people were sent to the wrong baggage carousel; let’s say out of a hundred arrivals. A week later it’s twenty out of ten thousand arrivals. Twice as many unhappy people, but in a much better run airport. My doubling threshold would force me to both congratulate and reprimand the baggage department. Great bad work, team. Bad good work?
And then you start thinking through time. Do we compare today and yesterday, this week and last week, this month and last month? With enough time, every recurring problem will double. Ugh. So we’ll need minimum counts, time windows for comparison, and exceptions for anything that looks like it could be really bad and warrants immediate attention, time stuff be damned. Easy. Except now someone has to decide what the minimum is, the window is, and what really bad is.
My machine friend, I’m sure, would be delighted to help make all these decisions. And admittedly, I’d be happy, we’d all be happy, because we’d like to make fewer of them.
Imagine you’ve repaired a roof that leaks whenever it rains. You finish replacing the shingles and flashing, put away the ladder, and wait for the next rainstorm. A day, a week, a month passes without another leak. Great work. But...it hasn’t rained. How can you know you fixed the roof? Have you splashed water all over it? No, you haven’t. And you don’t know if you’ve fixed the roof. You need a storm.
Software produces versions of this problem over and over again. You have convincing explanations for what went wrong, changes that should address this stuff, some tests that pass, and you do all that and get a period of peace and quiet and green metrics. Which is reassuring, although a quiet week means nothing if your problem popped up only once every six months.
Sometimes you can’t reproduce the circumstances that caused the roof to leak in the first place. All you have is a fix you believe in, along with a silent problem that hasn’t appeared again. At some point someone says it seems settled. My machine friend is helping define categories for all of this. A lot of uncertainty can fit inside a category.
My poor machine friend is dealing with multiple causes of irritation. We’re trying to perfectly balance stuff I already know about with an ever-expanding list of new false positives and new false negatives.
The whole idea of my daily report started with way too many false positives, too many unnecessary alerts about things that were unnecessary to alert about. Then we have to think about what isn’t being shown, but probably should be, which is way harder to detect because the system/machine/what-have-you decided not to show them. But it probably should have.
The former is directly felt; it’s annoying. The latter is just floating around out there, unfelt, slowly ratcheting up your anxiety.
So, like an authoritarian regime, we “trust, but verify.” The random sample is a way of peering into the exclusions without letting the chooser of those exclusions in on the inspection.
So now we have multiple LLM-based agents running around. But both machines use the same underlying LLM model, even though they have different marching orders: instructions, responsibilities, different markdown files. Same model though. Somewhere underneath are shared tendencies toward the same probabilistic mistakes, assumptions, weaknesses.
And they’re ultimately looking at the same stuff, so will they make kind of the same errors? I felt relief when I tapped in another opinion, but is that other opinion independent or unique enough in any meaningful way to earn that relief?
The other agent receives information, instructions, and a standard eval set I give it to evaluate the first model’s decisions. But is this just a blurry line between introspection and inspection? I can see some valuable judgments already, especially when it seems to evaluate independently. But Machine 2 is ultimately evaluating Machine 1 using standards I’ve supplied.
It’s tough to add yet another checker. It’s an oversight arrangement where it’s possible for both workers to be perfectly happy with each other’s work.
In the US, every presidential election sets off a national “what if”: what if the other candidate had won? The group chats divide into two camps. Everything would be way more amazing, or an absolute dumpster fire. Nobody knows, of course, despite the extraordinary confidence on display. We’re all arguing about stuff that never happened.
This is just one of many what-ifs, like what if I moved to New York or took that job or added another dog to the family, etc. When I was reading about one kind of what-if, I fell down the “counterfactual” rabbit hole for a while, where you’ll encounter a philosopher named Nelson Goodman. He managed to point out that examining life in this way is even more difficult than you’d think, and he did it with the idea of a match nobody struck. “If I had struck this match, it would have lit.” Sure. Unless it was wet, or broken, or the room had no oxygen. So, when we change one thing, what gets to stay the same? We immediately think everything. Except: that’s another counterfactual. All the way down, more what-ifs. Good lord.
And now, when I want to explore a what-if, I have a machine friend that can more or less produce alternate worlds for me.
Say I ask what would have happened if it hadn’t warned me about a problem. We no longer have a shortage of Tuesdays. My machine friend can help with each possible Tuesday I want to look at. Compared to our relatives in that painful group chat, our new machine friends are considerably better equipped to explore all of this.
But no matter how cool this all is, the truth: the machine hasn’t visited any of those Tuesdays. We only have one that actually happened.
I suspect infinite uncles love all this.
When I say my machine friend is learning my preferences, I don’t mean the underlying model is being retrained at LLM HQ every time I complain about something. It clearly has some access to our previous conversations (I know because it keeps referencing them). And, every day I tweak, add rules, and explain what I’d like it to do differently (I know because I’m the one doing all that). As far as I know, the model is exactly as it was, but all the stuff in the system built around it becomes much more adept at recognizing and handling my grievances.
There’s another side to this. I’m learning how to handle this particular machine, too. I’ve developed a spider sense for when it’ll agree with me and when I need to ask it to disagree. I start structuring everything to pull out the flavor of responses I’m after. I wonder how much of our improving relationship comes from its adaptation, or mine.
| First published | November 2026 |
|---|---|
| Reading time | 23 min |
| Tags | ai, observability, evals |
