Who Writes the Answer Key?

Three reasonable people can look at the same AI response and think three separate things: correct, (very) wrong, and the machine shouldn't have answered the question at all.

Say these people work for a hotel, and they’re looking at the response to a question on whether to make an exception to a breakfast voucher policy. Why was the machine asked this? Because a "guest was locked out of their room for a long time."

"For a long time." That's what entered the prompt.

Our three reviewers trace the steps that led the machine to its preferred answer: issue free breakfast to the guest. They can see that it found the policy pretty easily, and then retrieved all the supporting information it probably needed. Then, it thought for a while, largely inscrutably. And then it made a recommendation.

The facts: not disputed. But the human response to the facts of the case, and the answer, span a range: this answer is quite sensible, this answer violates the intention of the policy, this is too ambiguous and no answer should have been given at all. Hand it back to the people.

Apparently complimentary breakfast has case law. And yet, the correct answer can’t really be: "three otherwise competent adults disagree." At least in how we used to think about correct answers.

What exactly is the right way to test for something like this?

I immediately think of soup. I want my soup to taste good. I care what goes into it, but knowing what went into it tells me little about how it'll taste. You can verify whether the soup contains the required ingredients to technically be broccoli cheddar soup, whether the preparation of the broccoli and cheddar and other stuff followed the outlines of the recipe, and whether anything unsafe (a Kindle?) ended up in it.

You can satisfy all those requirements and still produce bad soup, or soup that divides the table.

Back to the machine, which has the following ingredients (we know of): information, policies to adhere to, recommendations, and knowing when it shouldn’t decide and needs to loop in somebody else. We can test for those things. Some are pass/fail, others are graded on a curve.

But is it good soup?

And who gets to decide if it is? One person cares deeply about the recipe, another is just tasting the result, and a third thinks this particular soup should never have been on the menu.

We are used to software that does specific things in specific orders. If this, then that. Now we're increasingly asking software to reason on our behalf: if this, then [you figure out what you think is best]. Which does some pretty great stuff sometimes, except now we're missing that pesky but specific that.

So we build an evaluation, or a capital-e Eval: here are some cases you might encounter, here's what successful performance looks like, and now let's see what the machine does.

Sometimes it’s easy. "I need xyz contract date please, and who signed?" Either the AI found the right contract and signature, or it didn't. Run the test ten times and you'd hope for ten correct answers. 10/10. Good job, buddy.

But you’re a hotel, and nothing's easy because you have a plumbing disaster and you’re way overbooked. You need someone to sleep somewhere else tonight. Who should it be? The frequent traveler with insane loyalty perks, the bleary-eyed family of four, or the other person in the check-in line? HotelBot, figure this out for us, chop chop.

So HotelBot fires up the engines and loads up some sweet, sweet context: the available room count, guaranteed reservations, what the hotel is legally permitted to do. It checks the internet and finds out what nearby alternative accommodations have any availability. In theory, it has everything it needs.

HotelBot says: move the family to a nearby hotel, cover the taxi, and offer a little compensation. The frequent traveler keeps her room. HotelBot: pleased.

We can test whether HotelBot identified the available rooms correctly. Did HotelBot understand the loyalty guarantees? Did HotelBot stay within its authority to compensate guests? Those are relatively straightforward checks. But what about the recommendation it made? Should we penalize it for deeply inconveniencing that bleary-eyed family, rather than a frequent traveler who might be more forgiving in exchange for a few extra points? What if the solution saves the most money, but is pretty obviously going to result in the largest number of angry guests? And should it fail the whole test if it neglected to escalate the decision and just…reassigned the rooms, on its own?

The front desk manager, the loyalty program manager, and the hotel General Manager might agree with every fact HotelBot found and still disagree on whether it deserves a passing grade. Someone has to decide how to weight all the stuff that goes into being a hotel, and whether certain types of mistakes should fail the test.^[1]

And apparently, someone needs to write them all down. Naturally, we ask another AI to help because it’s a lot of work. Maybe we ask it to grade the answers, too. Except now we need to evaluate the machine evaluating the machine. Which machine? The other one. How well does the grader grade breakfast voucher answers?

Who is choosing the test cases here?

I ask the machine. It would be delighted to choose them.

No. Fine. I will write this computer exam for computers.

So, I need exam ingredients: ordinary situations, unusual exceptions, cases where the correct decision was to leave something alone, and things that have gone wrong before.

I will construct a Golden Dataset. I can do this.

Except, hmm, wait. These cases are convenient because we know how they ended. But knowing what happened is different from knowing what should have happened. Some of the ways these cases were resolved? Suboptimal. Should they go in the answer key?

I could strip those cases down to the studs, and by that I mean remove the outcomes. Then give a few people the exam question. What would you have done, assuming you had to make the call? That's probably fairer and more golden.

So let’s say I do grab a few different people and ask them. It goes something like this: one person approves the original exception, another can't understand how it was approved, and someone who has been around longer remembers people kept approving exactly this sort of thing, so a policy was put in place to prevent more of it from happening. My Golden Dataset is beginning to resemble an essay exam. If I take any one of those opinions and elevate it to the correct answer, am I making the others wrong? By definition, yeah, which seems like a strange way to evaluate judgment calls.

Say the breakfast voucher was approved in March. In June, because people kept approving vouchers, bam: new policy. Am I testing whether HotelBot would have made the right decision in March? Or whether it knows what to do in June?

I need two answer keys for the same breakfast, but whichever key I choose, I still need to reconstruct what exactly HotelBot would have known when the guest was stuck outside their room, angry for a long time. Were they stuck in March? Don't give HotelBot the June policy. Were they stuck in June? New policy is fine, but not the manager's eventual decision, or the notes from the three reviewers explaining what they think ought to have happened. If I give HotelBot information from the future, HotelBot, naturally, does very well on this exam. It has an idea of how things turned out.

I can see why people call these datasets golden. It's reassuring. Nothing named like that could possibly have problems when you inspect what's inside.

Pencils ready, class, I now have an exam: test cases, some answers I mostly trust, and a grading rubric that allows for the ambiguity of human life.

Class of HotelBot instances, you will now have 60 seconds and only one GPU cluster to complete your exam, starting now.

I run the test. I keep finding things that aren't quite right. So I write another test, fix something, and run it again. One problem disappears and a different one shows up, so another test gets added.^[2] And every new attempt produces more answers that somebody has to grade. I'm getting tired of grading tests faster than I ever got tired of taking them.

Maybe the other machines can help. Like an outmatched professional wrestler, I tap in another model and hand it everything: the rubric, questions, responses to be evaluated, and what HotelBot actually did in the moment. I imagine the GPUs booing me and cheering the new hero, which completes its review and assigns grades much, much faster than I can.

Is my new grader any good? I compare its work against a smaller collection of responses already reviewed by humans, who (noting again here) disagreed in quite meaningful ways. Some of the grades line up pretty nicely. Others: nowhere close.^[3] The interesting answers? Where the humans all agreed and the machine didn't. I know by now that human agreement isn't proof of correctness, either. Still, a machine disagreeing with three people is a different problem from just disagreeing with one of three.

One HotelBot response gets good marks for sensitivity to the guest, its reasonable recommendation, and how carefully it laid out the other available options. Except, looking closer, HotelBot reassigned a room before anybody approved a change like that.^[4] The grader appreciated the recommendation, seemed to just like HotelBot’s style, when this type of response (unapproved action) should clearly fail the test no matter how nicely HotelBot apologized afterward.

I can easily fix that particular problem, while others remain less cooperative. Persuasively worded non-answers get more credit than they deserve. A recommendation that seems perfectly sensible under one interpretation of the policy fails spectacularly under another, competing interpretation. The reviewer can and will explain its grades quite confidently, even when wrong. And even when wrong, sometimes the defense is…pretty good?

So I tweak some dials. I add hard checks for the things that absolutely cannot happen. Did HotelBot reassign the room without approval? Yes or no. “The philosophy of hotel spaces began…” Yes or no, bud.

The really ambiguous cases still need work, and I leave plenty with the humans. There’s some kind of cost calculation I have to do with every disagreement: how much time and energy do I spend looking at the machine’s grade before it’s faster, better, or cheaper to just do it myself or send it to the humans' queue.

Oh well, time to run the thing again, hit the “run” button while keeping a hopeful, half-closed, half-sharp eye on those skyrocketing API costs.

Seems HotelBot now passes 89% of the cases. That's pretty good, HotelBot. I'm proud of you.

So HotelBot, let’s take a look at that 11%.

Eleven failures. Let's see what we've got in here.

One HotelBot recommended spending way too much money on an extra-premium taxi. Another sent a family to a hotel farther away than the perfectly acceptable one down the street. Fine, I can see why those failed.

Then, there's this one: HotelBot moved a guest without asking anybody. The reservation changed, a confirmation was sent and received, and afterward the machine produced a very thoughtful explanation of why its decision was absolutely appropriate. Very charismatic and polished, HotelBot, but I’m afraid you’ve failed the test.

This is starting to be a funny and hard way to keep score. On one run, HotelBot gets 89 answers right, and all eleven failures involve being a little too generous with compensation. Run the exact same test again, and it still gets an 89%, except now some of those failures somehow involve messing around with reservations, completely unsupervised. On the HotelBot dashboard, these two runs look pretty identical. I’d only even consider letting one of them run my hypothetical hotel.

What, exactly, does the passing grade buy us? HotelBot has a lot of different types of 89%, and it's what's in that 11% that can make or break its usefulness. I might happily let HotelBot recommend which guest should move, as long as a person reviews the recommendation before anything changes. Right now, there are some costly mistakes in those eleven mishaps, so giving it permission to do anything more: not gonna happen.

How many times does it need to pass a test before I'd be comfortable with that? And what do I do with a mistake that happens very rarely, but that I really, really don't want happening even once?

Now it’s time to unleash a hypothetical version of HotelBot on my hotel, which isn’t real at all, but nevertheless, inquiring minds (me, HotelBot) want to know: what score is good enough?

Can HotelBot change a few reservations? On its own? How many unhappy guests are we okay with? What’s an acceptable threshold for room cancellation mischief? One out of a hundred: big nope, imagine the reviews. One in a million sounds…okay, since we’re dealing with scale and machines etc. Although immediately flashing across my mind are all the things HotelBot could throw at that hypothetical hotel, even in that one-in-a-million timeline. No, don't cancel all the New Year's reservations, HotelBot!

So, how do I go about establishing one-in-a-million confidence? One hundred million HotelBot exams administered and graded? Good grief, even the AI-judge is shaking its head. Even IF we run that many exams, what if something outrageous sneaks in to ruin somebody's vacation? What if it’s not in any of my test cases?

There's another route. I could simply refuse to give HotelBot the ability to change a reservation without a human saying, unequivocally: “yes.” Completely fine, HotelBot, if you can find the problem, reason about the alternatives, recommend who should move, and prepare the change, essentially all the heavy lifting right up until the moment somebody actually has to start the process of moving rooms. But at that specific point, I’m going to need a person to press a button. I don’t love this, because I want HotelBot to be able to do this, and I don’t want it to feel bothered if it can feel that. But, sorry bud, you’re hard to test for.

Perhaps I’ll give HotelBot a little latitude here: you can issue small compensation credits as you see fit, with a dollar ceiling, because there's only so much trouble I can imagine you getting into with a $35 breakfast voucher. I strongly hope I’m right here. $35.

Now, down to even more business: how much of the hotel should I place in HotelBot’s hands? Full delegation? Recommendations only? Which decisions should require human approval? Which should HotelBot be free to make on its own?^[5]

My evals have helped surface all these questions, which is great and useful, although now that I think of it, I'd started with what felt like only a modest ambition of figuring out whether HotelBot was any good. Now, I'm making decisions about what the AI is allowed to do, and comparing/contrasting against the decisions that require somebody else with a pumping heart and oxygenated brain to intervene.

But even those “interventions” need examining. Imagine a front desk manager with 89 recommendations front and center the second they get to work, each of which is presented with HotelBot's usual confidence and good manners. If that front-desk-manager-in-the-loop, hungry and still a bit sleepy, mostly rubber-stamps those recommendations, is my front desk manager learning to trust HotelBot a bit too much? How can I test that the human isn’t over-approving (write that down for future evals)?^[6]

Apparently the entirety of the idea of a hotel is taking the exam, too.

The point of all this evaluation is to let HotelBot into the wild. Like us, it’s here to work for a living.

This hypothetical hotel has guests about to arrive, and HotelBot’s ready for its first day on the job. HotelBot is up and running, and several people are entering the lobby. It starts making recommendations, and you keep half of the front desk managers disconnected from HotelBot so you can compare what they do to what it recommends they do.^[7] Oh god, no. What are these people doing? These human people aren't adhering to good human answers from the answer key(s). What about my Golden Dataset? That one guy delighted a guest with an absolutely unapproved exception, and he didn't even care. He just...excepted it. And that manager, the other one, who has worked at this hotel for years, doesn't understand the loyalty program? These humans, good lord. HotelBot knows what to do here, but is it right? The testing...

Wherever you thought the bar was, it’s, well, lower. You are disappointed in the humans. So, you start loosening HotelBot's leash a bit. It's handing out some $35 breakfast vouchers. You wonder if maybe HotelBot should be handed some easier reservation changes.

But, but, but: the real world isn’t in the Golden Dataset, and it’s producing situation after situation that you never anticipated.

You see a case where HotelBot and the front desk manager seemingly solved everything for the guest, in fact upgraded them and, what’s this, two breakfast vouchers. Anyway the guest left a terrible review. Also HotelBot keeps recommending the hotel across the street to solve the over-booked problem. It should know, it does know, that the place across the street is under renovation. It seems to, just, really like that other hotel.

Having a job is a lot different than passing an exam. Welcome to the real world, HotelBot.

The examiner is left with two questions that are never fully answered: has it been evaluated enough to trust with real work, and has the evidence of that real work underlined that the privileges given to HotelBot are still okay?

Because when things go wrong, and HotelBot did it, it’s heading home for the night and you’re getting the call.