Skip to Content

Check the AI Grader Before Trusting Its Verdict

Anthropic’s November guidance examines automated graders. Our administrative example shows why the meaning of a pass must be understood by the team using it.

Table of contents

No titles in the blog post

An automated reviewer can make a long list of AI answers look manageable. It can also hide a second problem inside the first: the system that awards a pass may misunderstand what a pass means.

In engineering guidance published on November 13, Anthropic recommends checking AI graders against human judgments before trusting their scores. Its discussion includes people labelling examples without seeing the grader’s verdict, comparing human–grader agreement with agreement between people, and revisiting the measure when experts disagree. The company also warns that graders can break or shift their judgments. These are attributed recommendations about safety evaluations, not independent evidence of a particular product’s accuracy.

Start with a decision people can explain

The following is our original business analysis, using an invented, low-risk administrative example. It does not reproduce a test run or establish that a safety-evaluation method works for this task.

Imagine a small operations team using AI to turn internal requests into short summaries. A separate automated reviewer marks each summary as ready for a colleague to read. The team wants to reduce the time spent fixing omissions, but the word “ready” has never been written down clearly.

One person thinks a ready summary must include the requested action and the relevant date. Another thinks it should also identify anything the requester has left unresolved. A reviewer could appear consistent while applying only the first interpretation. The reported pass rate would then answer a narrower question than the business intended.

Before deciding whether to use that number, ask the team to describe a summary it would accept and one it would return. Concrete examples can expose disagreements that remain hidden behind a familiar label.

Look closely at a disagreement

In our hypothetical case, an original request says that a delivery date is provisional. The summary presents it as confirmed. A person returns the summary because the distinction could change what happens next; the automated reviewer accepts it because the date is present.

That disagreement is useful information. It identifies a specific gap between the team’s intended meaning and the reviewer’s apparent interpretation. Changing the final score by hand would correct this example, but it would not explain whether similar summaries are being accepted elsewhere.

The team could record what was missing from its definition of readiness and examine other examples where that distinction matters. It should also consider whether the summary itself supplied enough information for any reviewer to make the decision. An unclear input and an unclear grading rule are different problems.

Separate a business exception from a grading failure

Some requests genuinely need a person to decide how to proceed. In the invented workflow, a request might mention two possible dates without choosing one. Returning that item for clarification could be the right business outcome, even if nobody has made a technical error.

A useful report would distinguish those unresolved requests from summaries that lost information and from review attempts that did not finish. Combining them into a single count of rejected items would make it harder to choose the right response.

For example, clearer instructions to requesters may address recurring ambiguity. A change in summary preparation may address missing qualifications. A broken review step needs its own investigation. The same headline number can otherwise encourage the team to solve the wrong problem.

Keep the report attached to the work

When showing the results to a manager, include a few examples of the decisions the number represents. State what material was reviewed, what the team meant by readiness and which unresolved cases remain. Describe the limits of that review rather than presenting it as approval for every future request.

If the workflow changes, consider whether the earlier account still describes the new work. A reviewer designed around brief scheduling requests may face different judgments when the team adds detailed supplier discussions.

The practical question is whether the review helps people make the intended decision about the actual output. An orderly table of passes becomes useful when its judgments are understood, its failures can be investigated and its scope is clear to the people relying on it.

AI-generated conceptual editorial photograph of fictional reviewers; not Anthropic staff, a real evaluation or its results.