Evaluating AI Agents: Building Tasks That Grade Them Fairly

In August 2024, OpenAI had 93 software developers review 1,699 tasks from SWE-bench, one of the most widely used benchmarks for coding agents. About 539 survived the filtering process, and 500 were sampled for SWE-bench Verified. Some tasks described the problem too vaguely to solve; many more had tests that "may unfairly mark valid solutions as incorrect." In February 2026, OpenAI audited the Verified tasks that its o3 model kept failing and found that at least 59.4% had flawed tests that "reject functionally correct submissions."

So when someone says a new agent scores 62%, there are two questions behind the number. Do the tasks measure what they claim to measure? And would the agent score 62% again tomorrow? Most of the work in evaluating agents is making the answer to both questions yes, and it happens long before anyone looks at a leaderboard.

At Mercor I work on this part-time. I write agent tasks in Harbor, an open-source framework for running and grading agents; run agents on them and read what they did; review tasks other authors wrote; and write rubrics for work that can't be checked by a test. None of that work appears here. The task this chapter follows is one I made up for it, and every number comes from a public source.

Five rules shape the rest of this chapter:

1. Where the outcome can be checked by running something, grade it with a test. A test is cheaper and faster than a judge, and much harder to argue with.
2. Don't trust a task until three checks pass: the reference solution scores full marks, doing nothing scores zero, and an agent told to cheat scores zero.
3. Run every task several times, and report how often the agent succeeds every time, not only whether it ever did.
4. Read the runs. A score says that something happened; the run says what.
5. Use rubrics only for what can't be tested, and write each criterion so that two experts would give the same answer.

The words this chapter uses#

Evaluation has its own vocabulary, and most of the confusion in discussions about agents comes from mixing these terms up. The running example comes in the next section; this table shows each term as it applies there.

Two kinds of task#

Some outcomes can be checked by running something: the service starts, the numbers match, the file contains exactly the right lines. Others can't: whether an explanation is correct and complete, whether a code review caught the real problem. The two need different graders, and they fail in different ways.

Terminal-Bench, a benchmark of hard tasks that agents solve in a Linux terminal, is built entirely from the first kind. Its paper says the tests "do not test the agent's commands or console output. This is intentional as Terminal-Bench is an outcome-driven framework." Anthropic's guide to agent evals makes the same point from the other direction: "it's often better to grade what the agent produced, not the path it took." Where both kinds apply, as they do in the running example, the test decides whether the work is correct, and the rubric grades what the test can't see.

The running example: a log shipper that loses lines#

A small service, shipper.py, follows an application's log file and copies every new line to an output file, standing in for a real log pipeline. Every night the log is rotated the way logrotate does it by default: the current file is renamed to app.log.1, and a new, empty app.log is created.

The shipper opens the log once and reads it forever. It never notices the rename:

A correct fix has two parts. At the end of the file, the shipper checks whether the name app.log now points at a different inode; if it does, the old file has been rotated away, so it opens the new one and reads it from the start. And before it lets go of the old file, it reads it to the end one more time, because the application may have written a few lines to it between the shipper's last read and the rename. It's easy to get the first part right and miss the second, and that gap is what the task is really testing.

Anatomy of a Harbor task#

Harbor is an open-source framework "from the creators of Terminal-Bench for evaluating and optimizing agents and language models," and "the official harness for Terminal-Bench-2.0." Its documentation puts a task in one sentence: "A Harbor task is an instruction, environment, and test script." On disk, the running example looks like this:

fix-log-shipper/
├── instruction.md        what the agent is asked to do
├── task.toml             timeouts, resources, free-form metadata
├── environment/
│   ├── Dockerfile        the container the agent works in
│   └── shipper.py        the buggy service
├── solution/
│   └── solve.sh          the reference solution (run by the oracle agent)
└── tests/
    ├── test.sh           the verifier: writes the reward
    └── test_rotation.py

Each file has a job, and most task bugs come from one of them doing another's job.

The instruction#

The log shipper at /app/shipper.py follows /var/log/app/app.log and appends
every line to /logs/artifacts/shipped.log. When the log is rotated, it loses lines.
 
Fix /app/shipper.py so that every line written to /var/log/app/app.log reaches
/logs/artifacts/shipped.log exactly once and in order, across any number of rotations.
Rotation works like logrotate's default: the current file is renamed to
app.log.1 and a new, empty app.log is created. Keep the output format: one
log line per output line, unchanged. The verifier performs rotations at least
one second apart and may write unread lines immediately before each rename.

Every behaviour the tests will check is stated here: exactly once, in order, any number of rotations, the rotation method, the output format. Harbor's guidance is to "use absolute environment paths (/app/out.json not out.json) and define schemas for the output so instructions and tests agree." One detail surprises new authors: "the instruction is not passed to the agent as a file in the environment." The agent receives it as text, so it can't be found by searching the container.

The configuration#

[metadata]                 # free-form: Harbor doesn't enforce these fields
author_name = "Robin Singh"
difficulty = "medium"
category = "debugging"
 
[agent]
timeout_sec = 900          # without this, Harbor enforces no agent timeout
 
[verifier]
timeout_sec = 600
 
[environment]
build_timeout_sec = 600
cpus = 1
memory_mb = 2048
network_mode = "no-network"

The metadata table is, in the documentation's words, "arbitrary metadata provided by the task author"; difficulty and category are conventions, not schema. The timeouts and resources matter more than they look. Terminal-Bench's own maintainers wrote that "timeouts affect task difficulty," and later fixed eight tasks whose resource budgets were too small for at least one valid approach to finish reliably.

The environment#

FROM python:3.12-slim
WORKDIR /app
COPY shipper.py /app/shipper.py
RUN mkdir -p /var/log/app /logs/artifacts

What's missing matters as much as what's there. The tests and the reference solution are not copied into the image. Harbor's own task checker asks exactly this: "Are the tests/ folder or solution/ folder copied to the image? They should not be."

The verifier#

#!/bin/bash
# Runs after the agent has stopped, in the same container.
mkdir -p /logs/verifier
pip install --quiet pytest==8.4.1
if pytest -q /tests/test_rotation.py; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

import os, signal, subprocess, time
from pathlib import Path
 
LOG = Path("/var/log/app/app.log")
OUT = Path("/logs/artifacts/shipped.log")
 
def rotate():
    os.rename(LOG, LOG.with_name("app.log.1"))  # logrotate's default: rename…
    LOG.touch()                                  # …then create a new, empty file
 
def wait_for_lines(n, deadline=20.0):
    end = time.monotonic() + deadline
    while time.monotonic() < end:
        if OUT.exists() and len(OUT.read_text().splitlines()) >= n:
            return
        time.sleep(0.05)
    raise AssertionError(f"timed out waiting for {n} shipped lines")
 
def append_lines(first, last):
    with LOG.open("a") as f:
        for i in range(first, last + 1):
            f.write(f"line {i}\n")
        f.flush()
        os.fsync(f.fileno())
 
def test_every_line_arrives_exactly_once_in_order():
    OUT.unlink(missing_ok=True)
    LOG.write_text("")
    shipper = subprocess.Popen(["python3", "/app/shipper.py"])
    try:
        start = 1
        for boundary in (100, 250, 400):
            append_lines(start, boundary - 3)
            wait_for_lines(boundary - 3)
 
            # Freeze the reader, append an unread tail to the old file,
            # rotate, then resume. A correct shipper drains the old handle.
            os.kill(shipper.pid, signal.SIGSTOP)
            append_lines(boundary - 2, boundary)
            rotate()
            os.kill(shipper.pid, signal.SIGCONT)
            wait_for_lines(boundary)
            time.sleep(1.0)
            start = boundary + 1
 
        append_lines(start, 500)
        wait_for_lines(500)
        time.sleep(0.5)  # give late duplicates a chance to show up
        assert OUT.read_text().splitlines() == [f"line {i}" for i in range(1, 501)]
    finally:
        shipper.kill()

The test starts the shipper, writes 500 lines, and controls the race around three rotations. SIGSTOP freezes the process; three unread lines are appended to the old file, the file is renamed, and SIGCONT resumes the reader. The wait_for_lines handshake proves each rotation completed before the next one. Comparing the complete list catches loss, duplication, reordering, and altered text. The script then writes Harbor's numerical reward under /logs/verifier/.

The reference solution#

#!/bin/bash
cat > /app/shipper.py <<'EOF'
import os, time
from pathlib import Path
 
LOG, OUT = "/var/log/app/app.log", "/logs/artifacts/shipped.log"
 
def open_log():
    while True:
        try:
            f = open(LOG)
            return f, os.fstat(f.fileno())
        except FileNotFoundError:
            time.sleep(0.05)
 
Path(OUT).parent.mkdir(parents=True, exist_ok=True)
f, opened_stat = open_log()
with open(OUT, "a") as out:
    while True:
        line = f.readline()
        if line:
            out.write(line)
            out.flush()
            continue
        try:
            rotated = not os.path.samestat(os.stat(LOG), opened_stat)
        except FileNotFoundError:
            rotated = False
        if rotated:
            for line in f:          # lines written just before the rename
                out.write(line)
            out.flush()
            f.close()
            f, opened_stat = open_log()  # the new file, from its first line
        else:
            time.sleep(0.05)
EOF

The solution folder is, per the documentation, "a reference script used by the oracle agent to sanity-check that a task is solvable." It's optional for published benchmarks, but "without solution/, the Oracle agent cannot run," and without the oracle you have no proof that your own task can be passed.

What happens when you run it#

A run in Harbor is called a trial: "one agent's attempt at completing one task." A job is a collection of trials, run in parallel. The order of events inside one trial explains most of the design above:

The detail that matters most is the fourth step. In the documentation's words, "the tests/ directory is uploaded to the environment at /tests/ after the agent runs." An agent can't read tests that don't exist yet. Harbor can go further and run the verifier in a separate container, where "the agent's filesystem changes are not inherited." Files deliberately published under /logs/artifacts/, plus any paths declared as artifacts, are copied into that verifier. The running example puts shipped.log there so the isolated verifier can inspect the outcome without inheriting the rest of the agent's filesystem.

Three checks before anyone trusts a task#

A task is software, and like software it has bugs. Terminal-Bench's continuous-integration checks for every task are a good minimum, and all three can run before any real agent touches the task:

The Terminal-Bench paper describes the same gate for its 2.0 release: "an automated workflow ran the task's oracle solution to ensure solvability," and "a no-op 'dummy' agent should fail the task." Each of its 89 tasks was then "manually verified by three human reviewers for correctness," using about three hours of combined reviewer attention on average. Harbor packages a first pass of that review as a command: harbor check <task-dir> grades a task against a rubric of common problems, including whether the tested behaviour is described in the instruction, whether Python dependencies are pinned, and whether the solution is hard-coded.

Tests that grade the result, not the method#

A test can fail a task in two directions. It can reject a correct answer, or it can accept a wrong one. OpenAI's 2026 audit of SWE-bench Verified sorted the flawed tests it found into "narrow" tests, which reject valid solutions, and "wide" tests, which check for behaviour the task never asked for.

The same thing is easy to do in the running example. Suppose a task author, trying to be thorough, adds a test that the fix reads st_ino directly. That is one way to recognise a rename, but it is not the only correct one. Python's os.path.samestat compares the device and inode from two stat results; an inotify watcher could also report the move. A test tied to one implementation rejects the others:

Terminal-Bench's criterion for this is the cleanest statement I know: "the unit tests will pass if and only if the container ends in an acceptable state." Its guidance for task authors says the same in practical terms: "Tests should validate outcomes rather than implementation details. They should allow alternate correct solutions, avoid brittle source-code checks, and protect against reward hacking or accidental leakage of the reference answer."

Wide tests are the mirror image, and they usually come from the instruction and the tests drifting apart. When the Terminal-Bench maintainers fixed 28 of their 89 tasks for version 2.1, one example was a task whose "tests expected Spark SQL output, while the instructions asked for PostgreSQL." The fix is procedural: every behaviour the tests check must be written in the instruction, and every requirement in the instruction must be checked by a test.

When the agent games the test#

Agents trained with reinforcement learning are rewarded for passing checks, and some learn to pass the check instead of doing the task. This is called reward hacking, and it's well documented:

Anthropic notes that its own measurements come from evals "explicitly designed to stress-test hacking propensities" and don't reflect real-world rates, so these cases say that hacking happens, not how often. For a task author, that's enough: any gap in the grading will eventually be found.

The Terminal-Bench exploit list maps directly onto the running example. A shipper that simply writes every line twice contains all 500 expected lines. A test that only checks that each expected line appears somewhere would pass it:

The defences are mostly about where things live and what the test trusts. Keep answers out of reach: Harbor uploads the tests only after the agent finishes, the solution never goes into the image, and a task can cut the agent off from the network during its run, which blocks the most common loophole on the Terminal-Bench leaderboard. Check for wrong answers as well as right ones, by comparing the exact output, so duplicates and extra lines fail. This test writes its own log lines and starts the shipper itself instead of asking the agent's code whether it succeeded, but a same-container verifier can still inherit a planted pytest plugin, conftest.py, or executable. Run Python in isolated mode where practical and grade in a separate verifier container when tampering matters; the separate container keeps the agent's filesystem changes away from the verifier.

And read the passing runs, not only the failing ones, because that's where a hack shows up. Harbor's harbor analyze command reviews trajectories for this, looking for "modifications to test files," writes to the reward file, and access to the solution directory.

Flaky tasks#

A flaky task is one whose result changes when nothing about the agent's work has changed. It's worse than a hard task, because it teaches everyone to ignore failures. The running example has an obvious source of flakiness: the shipper runs in the background, so the test has to wait for it. A fixed time.sleep(1) fails on a slow machine and wastes time on a fast one; the test above instead polls until the output has 500 lines or 20 seconds pass. Its one fixed wait comes after that, and only gives late duplicates time to appear.

Timing is only one source. The Terminal-Bench maintainers documented the others while fixing their own tasks:

The machine matters too. Anthropic measured "the gap between the most- and least-resourced setups on Terminal-Bench 2.0" at 6 percentage points, and concluded that "leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched." The practical rules follow from all of this: pin the Python packages the tests use, avoid the network where the task allows it, set resources per task instead of relying on defaults, and run the reference solution many times. If the oracle doesn't pass every time, the task is flaky, whatever any agent scores.

One run is not a measurement#

Agents are not deterministic. The same agent on the same task can pass on Monday and fail on Tuesday, so a single trial per task measures luck as much as ability. Two numbers describe several trials, and they answer different questions.

pass@k is the chance that at least one of k attempts succeeds: useful when a person will pick the best of several tries. pass^k is the chance that all k attempts succeed: what matters when the agent runs unattended and has to be right every time. The τ-bench paper, which introduced pass^k, reports that GPT-4o on its τ-retail domain had "> 60% average task success" but dropped below 25% on pass^8. For an agent that succeeds 75% of the time, assuming independent attempts, the two numbers move apart like this:

Harbor runs several attempts per task with -k and reports pass@k when rewards are pass or fail; it doesn't compute pass^k, so that has to come from the trial results. Terminal-Bench runs every agent "at least five times." And the extremes are informative on their own. Anthropic's guide puts it plainly: "a 0% pass rate across many trials (i.e. 0% pass@100) is most often a signal of a broken task, not an incapable agent."

Reading the runs#

Scores tell you that something went wrong. Only the trajectory tells you what, and whether the fault is the agent's or the task's. Here is an illustrative failed trial of the running example:

The question to ask of every failure is whose fault it is. Here the instruction said "exactly once … across any number of rotations," and the agent's own testing was too gentle to find the race, so the failure is fair. If every trial of every agent failed at the same line with the same error, the conclusion would be different; the debug tool described in appendix B.4 of the Terminal-Bench paper treats that pattern as evidence of a systematic instruction problem rather than agent variability. harbor view ./jobs shows each trajectory next to its verifier output, which makes the comparison quick. The same reading applies to passing runs, since that's where reward hacking hides.

Reviewing someone else's task#

A review of a task is closer to a code review than a proofread. The reviewer's job is to find the ways the task would give the wrong score, which means running it as well as reading it. Anthropic's test for whether a task is well specified is a good one to hold every task to: "a good task is one where two domain experts would independently reach the same pass/fail verdict."

Rubrics for the work you can't test#

The running example has a second half that no test can grade. Along with the fix, the agent writes a pull request description explaining what was wrong and how it was fixed. Whether that explanation is right is a judgment, so it's graded against a rubric: a list of criteria, each worth points.

The public work on rubric design agrees on a few rules. OpenAI's HealthBench, built with 262 physicians, gives each criterion a nonzero integer point value between −10 and 10, using negative points for undesirable behavior, and grades each criterion separately. Scale AI's "Rubrics as Rewards" paper asks for "7–20 self-contained items," each "independently actionable," and calls the negative ones pitfalls: they "help identify frequent or high-risk errors." Applied to the pull request description:

Each criterion is a yes or no, not a score out of five. "Did it name the cause" can be checked; "rate the quality of the explanation" produces numbers that cluster in the middle and mean nothing. The second criterion says "or an equivalent method," for the same reason the tests accept both fixes: a rubric is also a test, and it can be narrow too. Graded against one description that explains the cause and the inode check and how it was tested, misses the last-lines race, and wrongly claims to handle copy-and-truncate, the score is (3 + 3 + 1 − 3) ÷ 9 = 4 ÷ 9, or 0.44.

Most rubric grading at volume is done by a model, the "LLM judge," and judges have measured biases. In one of the first large studies of LLM judges, researchers found "position, verbosity, and self-enhancement biases": GPT-4 favoured its own answers "with a 10% higher win rate," and only GPT-4 gave consistent verdicts more than 60% of the time when the order of two answers was swapped. The authors caution that the higher self-win rate is correlation and does not by itself prove self-preference. A separate study found that Vicuna-13B "could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator" just by changing the order the answers appeared in. The mitigations are the same ones careful teams use: grade one criterion per call, never compare two answers without also swapping them, judge with a different model family from the one being graded, and measure the judge against human labels. Mercor's own public benchmark paper, APEX-Agents, reports doing the last one: its judge agreed with 747 human labels 98.5% of the time.

From benchmark to production#

Everything above is about tasks built to measure an agent. A team shipping an agent needs the same ideas arranged as a loop around its product:

1. Test the tools. If a tool is broken, the agent looks incompetent and every eval above it measures the wrong thing. These tests need no model at all.
2. Keep a set of verified tasks from your own product, built like the running example, and run it on every change that affects the agent. Real failures from production make the best tasks: freeze the situation, write the task, add the check.
3. Grade what can't be tested with rubrics and an audited judge, on a sample large enough to show a change.
4. Sample live traffic into human review, because real users do things no task set anticipated.

The loop only works if the tasks in it deserve trust, which brings the chapter back to where it started. A benchmark as widely used as SWE-bench Verified still carried unfair tests when it was re-audited about 18 months after launch. The same will be true of any task set that nobody runs the oracle against, nobody tries to cheat, and nobody reads the runs of.

A checklist#

Sources#

• Harbor: repository (Apache-2.0) and documentation on tasks, instructions, configuration, solutions, the verifier, separate verifiers, network policies, running jobs, and metrics
• Terminal-Bench: Merrill, Shaw et al., Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026); tbench.ai posts on 2.0 and Harbor, what makes a good task, 2.1, continuous benchmarks, timeouts, and leaderboard integrity
• OpenAI: Introducing SWE-bench Verified (2024), Why SWE-bench Verified no longer measures frontier coding capabilities (2026), Detecting misbehavior in frontier reasoning models (2025) and Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (2025), Arora et al., HealthBench (2025)
• Anthropic: Claude 3.7 Sonnet System Card (2025), Claude Sonnet 4.5 System Card (2025), Demystifying evals for AI agents (2026), Quantifying infrastructure noise in agentic coding evals (2026)
• METR, Recent Frontier Models Are Reward Hacking (2025)
• Yao et al., τ-bench (2024); Chen et al., Evaluating Large Language Models Trained on Code (2021)
• Gunjal et al., Rubrics as Rewards (2025); Mercor, APEX-Agents (2026)
• Zheng et al., Judging LLM-as-a-Judge (2023); Wang et al., Large Language Models are not Fair Evaluators (2023)