We debated early on what the right goal for the machine should be. In our business, the ultimate goal is to make great investments that generate exceptional returns for our LPs, and as investors, we of course have a point of view on that. But venture is a power-law business where most investments fail, and nobody finds out who was right for a decade or more. On the flip side, whether a company is worth a thirty-minute meeting is a call we’re confident we judge well, and one we can grade every week. We set the machine’s target there because it’s the one we can tune in the near term.

The first version of the machine we built was a rubric that scored each company on a long list of weighted attributes and added the scores into one number. In the first week of partner grading, the machine told us to take meetings we would never have taken, and it missed a lot of low-hanging fruit. When I dug in, the score had complexity without the benefit of being able to pin down what was going on.

So we replaced the rubric with instructions written in prose: say what matters, let the underlying model reason, and return a call in the same format the partners would use when they judge a company themselves. Now the partners could read why the machine made each call and tell it where the reasoning was off.

Picture this loop as a training program for a junior investor. The junior does the work, a senior person judges the same company on their own, you compare the two, and you correct what the junior got wrong. AI teams call this an eval, and it’s how these products get better.

Every eval has a grader: whatever you compare the product’s output against before you tune it. The grader is also the ceiling on how good the product can get.

When outcomes arrive fast, the grader is the outcome. An AI support agent is graded on whether the customer’s problem got resolved, and can climb as far as the outcomes let it. Coding and math are where the models got good first: a machine can check the answer, and there’s a huge public record to learn from.1

When outcomes are slow or never arrive, teams reach for the next-fastest thing: the answer an expert would have given. Harvey, the legal AI company, grades its agents against tens of thousands of expert-written criteria.2 It’s the right instrument for that job, and it sets a ceiling: the product can get as good as the people who wrote the answer key, including their blind spots.

To get past your experts, grade their reasoning, not their answers

Ours is the third grader. We’re not trying to train the machine to come up with the exact same answer we would. We’re trying to deduce the logic and the reasoning, and teach it to reason like a senior investor, so every week the partners grade how the machine reasoned. That moves the ceiling: a partner can follow a chain of reasoning to a conclusion none of us would have reached and still recognize it as sound. An answer key can never reward that.

If you key off the answer (which I’ve seen in our own eval process), the model will come up with all sorts of ways to get to it. An answer key only checks what I thought to write down. Ask for sausage and it makes pork when I wanted beef: “that is sausage, so it passed the test.” Add “beef” to the key and it finds the next thing I left out. It hits the answer for the wrong reasons, and on a hundred other companies the right answers stop coming.

Grading reasoning only works if the machine prints its reasoning in the way you reason. A rubric of weighted scores isn’t how we think about a company, and you can’t grade reasoning you can’t read. Over a dozen revisions, how often the machine’s meeting call matched the partners’ own went from 50% to 92%. That’s an early signal the loop works.

Most products should stop at the answer key, and know it

The question for a founder is which product you’re building: one that replaces the manual, mundane junior slice of the work, or one that has to get past your senior people. For many products, getting to the expert’s answer is good enough. Most teams are just trying to replace the junior slice, and an answer key written by the senior person is the right bar for that. SOP work has steps and a result you can check, so you tune to the outcome.

Judgment work has no quick check, and that’s where grading only the answer breaks down: researchers keep finding models that reach the right answer for the wrong reasons, then fail on new cases.3 Where a machine can check the answer, the best results now come from grading outcomes alone. We don’t have that option, so we review the reasoning. That’s a bet that hasn’t been proven yet.

The slice that stays human keeps shrinking

The last step in our loop is still human, and I expect it to stay human: the live meeting, where someone reads the founder’s visual cues and asks the pointed follow-up in real time, and the final call. The slice we’re responsible for will shrink to be smaller and smaller.

A junior taught only by their seniors can match them at best. A machine has three advantages no human junior has. For one, the model underneath it gets smarter every few months (unlike humans!). On top of that, every call it makes is written down with its reasons, and every new version of the machine can be checked against the full record. The third is probably the most important: as those companies succeed or fail, the results come back as a grader of their own.

When the grader changes, the ceiling moves with it. If we believe that framing, over the long run a machine taught to reason like a senior investor could conceivably beat us. That’s a guess, because humans’ frontier also moves up and forward as the model takes over the work below it.

In the near term a reasoning-graded machine and an answer-graded one will look the same. Ideally we’d see the machine suggest a meeting none of us would have taken, and then be convinced by its reasoning, the way an investor with an additive perspective shows up on a team.

Whatever you grade your AI against is the ceiling it can reach: outcomes when they arrive fast, your experts’ answers when they don’t, and your experts’ reasoning only if the product has to get past your experts.


    1. Most other fields have neither. The work sits inside firms and in experts’ heads, and it takes a person to say whether an answer was right. That is why products in those fields have to build an eval of their own, with their own experts as the grader. ↩
    2. “Introducing Harvey’s Legal Agent Benchmark” (Harvey, May 6, 2026): more than 1,200 agent tasks, graded pass/fail against “over 75,000 expert-written rubric criteria” that the company says “emulate the scrutiny work product undergoes when handed off to partners and clients.” ↩
    3. “The Paradox of Outcome Optimization” (May 30, 2026): models rewarded on outcomes “achieve high performance on in-distribution benchmarks while demonstrating brittle reasoning capabilities on out-of-distribution (OOD) tasks.” OpenAI’s “Let’s Verify Step by Step” (May 31, 2023) first showed that grading each step of the reasoning could beat grading the answer on math. That edge has not held where answers are checkable: DeepSeek-R1 (January 22, 2025) trained on outcomes alone, “obviating the need for human-labeled reasoning trajectories.” ↩