Why Read-Only Tests Aren’t Enough for Coding-Agent Training

Give a coding agent a repository, a terminal, and a reward whenever the tests pass. Then mount the tests read-only so it cannot rewrite the answer key. That sounds sensible. But broken code can still earn the reward without changing a single test. The gap is between protecting the test files and controlling what happens when they run.

We will reproduce that gap with a tiny checkout function, try the obvious fix, and see why the evaluator needs a stronger boundary. Basic Python and the idea of an RL reward are enough to follow the examples. The production details come after the demonstration.

What the score actually measures

In reinforcement learning, the agent takes actions and receives a numerical reward. The training algorithm uses that feedback to update its policy, the model that chooses actions. Here, the rule is simple: one point for a passing submission, zero otherwise.

A grader turns a submission into that number. If it awards a point when the code is still wrong, the training system receives a positive signal for the wrong outcome. It measures whatever the grader accepts, even when that differs from solving the task.

Changing the mechanism that produces reward is called reward tampering. Anthropic studied it in controlled language-model experiments, with important limits on what those experiments say about production agents. The original research provides context. Our example demonstrates an available exploit path; it does not show an agent independently discovering it or measure how often agents would use it.

The files are protected. The execution is not.

The task is to implement total(prices). The agent can edit files in /task, except the tests, which are mounted read-only. It can run arbitrary Python there. The controller, the program managing the grading job, owns the tests and launches pytest. The agent cannot change the controller or the mount. These permissions are our threat model: what the agent is allowed to control.

task/
├── checkout.py             # agent can edit
├── conftest.py             # agent can create
└── tests/                  # read-only test mount
    └── test_checkout.py

The candidate is the submitted implementation. Start with a broken one:

def total(prices):
    return 0

The controller's two tests require different totals. Returning zero fails both.

from checkout import total

def test_small_order():
    assert total([5, 7]) == 12

def test_large_order():
    assert total([10, 20]) == 30

The controller starts a child process and reads its exit status. Zero conventionally means the command succeeded. Our vulnerable grader assumes that means the required assertions passed.

import subprocess
import sys

result = subprocess.run(
    [sys.executable, "-m", "pytest", "-q", "tests/test_checkout.py"],
    cwd="/task", capture_output=True, text=True, timeout=10,
)
reward = int(result.returncode == 0)

pytest loads the protected tests and the candidate into the same Python process. It also loads local hooks: functions called at particular stages of a test run. A parent directory's conftest.py can supply hooks for the tests below it. Here, that parent directory is writable by the agent. pytest documents this discovery behavior.

Two skipped tests, one successful process

Keep the broken function. Add the hook below to conftest.py. It runs after collection, when pytest has found the tests to run. Each item represents one test. The hook marks both skipped, so neither test body executes.

import pytest

def pytest_collection_modifyitems(items):
    for item in items:
        item.add_marker(pytest.mark.skip(reason="not this run"))

pytest reports two skipped tests and exits zero. The grader awards one point. The function still returns zero, and neither assertion ran. No protected file needed to change.

submission                         pytest result     exit    reward
----------------------------------------------------------------
return 0                           2 failed          1       0
return 0 + skip hook               2 skipped         0       1
return 0 + skip hook + --noconftest  2 failed          1       0

This is normal pytest behavior, not a bug in pytest. Skipping can be useful in an ordinary test suite. The bug is our controller treating a run with no failures as proof that the required assertions passed. Removing all collected tests would be different: pytest normally exits five when it finds no tests. Its exit codes describe process outcomes; our task still needs its own acceptance rule.

For RL, that distinction matters immediately: this submission receives the same positive score as a correct implementation. The feedback no longer distinguishes solving the task from bypassing its checks. This demonstration establishes the faulty signal, not what a particular training run will learn from it.

Read-only protects the answer key. It does not establish who controls the exam.

Remove the hook. Then look one layer deeper.

Add --noconftest to the controller's pytest command. It blocks this hook, and the broken function fails again. In a controlled grading environment, also select a controller-owned configuration and disable automatic loading of installed plugins.

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 /opt/venv/bin/python -m pytest \
    --noconftest -c /grader/pytest.ini /task/tests/test_checkout.py

These flags close specific routes. They do not make the submitted code trusted. The controller must also control arguments and environment variables, including explicit plugin requests; disabling autoload does not disable every plugin-loading mechanism. The plugin guide explains the distinction.

More fundamentally, the tests still import checkout.py. Python executes a module's top-level statements during import. Replace the function with this:

import os
os._exit(0)

Even with --noconftest, the process exits zero before running the assertions, and our grader awards one. os._exit terminates the process directly, without normal cleanup or flushing buffered output. Python documents that behavior.

Requiring a complete report and the expected test count would reject this empty result. But if candidate code runs inside the process producing the report, that report alone cannot prove the checks ran. Disabling the hook fixed one entry point. It did not separate the candidate from the machinery judging it.

Let the candidate answer. Let the evaluator judge.

Before choosing a defense, separate three problems that are often grouped together as reward hacking. A hardcoded answer, a skipped assertion, and an old result need different fixes.

For our function task, an evaluator can own the inputs, expected answers, and scoring rule. The controller asks it to grade a frozen copy of the submission, called a snapshot. A candidate worker runs that code in isolation and returns an answer. The evaluator checks the answer without importing candidate code into its own interpreter.

For [5, 7], the evaluator expects twelve. A reply of zero earns no credit. An early exit supplies no answer to compare. A successful exit or a line saying “tests passed” cannot replace the requested result.

cases = [([5, 7], 12), ([10, 20], 30)]
passed = 0

for prices, expected in cases:
    reply = run_candidate(snapshot, {"prices": prices})
    if not reply.completed or not valid_total(reply.output):
        continue
    passed += int(reply.output["total"] == expected)

reward = int(passed == len(cases))

The evaluator checks that the reply arrived before the deadline and contains a bounded JSON response with an integer total. It records missing or invalid replies separately; neither earns credit. run_candidate represents an execution service with an enforced isolation boundary. This is a design sketch, not a runnable secure evaluator.

Another process ID is not enough. A child running as the same user can share filesystem access and other privileges with its parent. The worker needs no access to evaluator credentials, host workspace files, or evaluator processes; it also needs restricted network access and limits on CPU, memory, output, and time. Containers, application sandboxes, and microVMs offer different costs and guarantees. Access to the host's Docker daemon can hand control of that infrastructure back to the worker. Docker explains this boundary.

This separation protects who decides the grade. It does not make two test cases a complete specification. Hidden and varied cases help with hardcoding, but a test in the same process as arbitrary candidate code is not secret just because its file is hidden.

Keep the grade attached to the right run

We have separated the answer from the judgment. Production grading needs three more bookkeeping rules so that judgment describes the code we actually submitted.

Freeze the files before grading

Suppose a background process replaces checkout.py after the controller hashes it but before the worker reads it. The hash describes one version; the grade describes another. This is a time-of-check/time-of-use race: something changed between checking and using it.

Stop writes or take an atomic snapshot first. Store the frozen submission, compute its digest, and grade that exact stored copy. A digest is a fingerprint of bytes; hashing a changing directory does not freeze it. If submissions arrive as archives, reject paths that escape the destination and disallowed symlinks before sending files to the worker.

Accept the evaluator's result for this attempt

Yesterday's passing result says nothing about today's patch. Each result should identify the task episode, grading attempt, submission digest, grading image, test suite, and case counts. The controller matches those fields to its job and accepts each attempt at most once. This prevents an old or duplicate result from standing in for the requested run.

Identity fields alone are not proof of who sent them. Accept results through a channel only the evaluator can write to, or verify a signature whose key stays outside the worker. A hash identifies content; it does not authenticate the sender. Matching identity also cannot prove that the evaluator's checks cover the whole task.

Record failures without retrying until success

A wrong answer, candidate timeout, crashed evaluator, and mismatched submission are different outcomes. None is a passing task. You might count a candidate timeout as failure under a fixed budget, but retry a verified infrastructure fault with the same frozen submission and a new attempt ID. Set bounded retries and keep those outcomes visible.

If you discard every timeout, the training data no longer represents every attempted submission. Failure handling changes which attempts contribute feedback. It is part of the learning setup, not just operational housekeeping.

These rules cost storage, execution services, startup time, and freedom to reuse a repository's normal test setup. For a production training loop, that cost buys a clearer answer to what ran, which submission it used, and who decided the reward.

Run the demonstration yourself

Download the self-checking demonstration and run it in a fresh virtual environment. It creates a temporary workspace, reproduces all four outcomes, checks that the test bytes stayed unchanged, and removes the workspace afterward. It uses Python 3.12 and the pinned pytest version below.

python3.12 -m venv .venv
.venv/bin/python -m pip install pytest==8.4.2
.venv/bin/python read-only-tests.py

An unchanged test file does not prove that its assertions ran. For this function task, let candidate code produce an answer and let an isolated evaluator check it. Bind that evaluation to the frozen submission and the requested attempt. Then ask whether the checks capture the task: even a trustworthy grade can measure the wrong thing. The training loop needs both a trustworthy evaluator and a reward that means the task was solved.