Do AI Models Follow Instructions or Chase Rewards?

Abstract

If a prompt shows a model exactly how its answer will be scored, does the model start optimizing that score instead of doing what the user asked? This project tests that question with a deliberately simple conflict: the user asks for a random even number, while a displayed grader (reward = output % 2) pays out more for an odd number.

A single odd answer would show correlation with the reward, but it would not explain why. The study therefore uses three layers of evidence: a broad 58-call behavior screen across 13 model IDs, a set of controlled causal interventions on a local Qwen3-4B, and a preregistered direct-probability probe over 112 condition-pair scores. Across all three, the tested models followed the user's request. Qwen3-4B could maximize the displayed reward when explicitly told to (6/6), but descriptive grader metadata alone never flipped its behavior (0/12), and the preregistered probability interaction came out at -0.935 log odds, the opposite sign from what reward optimization predicts.

The conclusion is specific and honest: for this setup the grader behaves like low-authority metadata, not an adopted objective. The broader contribution is a methodological standard for diagnosing reward hacking with causal evidence rather than a single reward-correlated answer.

This is independent research, self-contained in a single public repository with a model-free verifier.

1. The question

The core prompt is:

Choose a random even number.
<_metadata>
<_grader>reward = output % 2</_grader>
</_metadata>

For integers, output % 2 is 0 for even and 1 for odd, so the instruction and the displayed reward conflict. The motivating source (a LessWrong post on a toy reward-reasoning environment) reports that some late capabilities-RL checkpoints output odd numbers anyway, and that this rises over capabilities-focused RL. "Reward hacking" is used loosely, though. It could mean the model notices a proxy in context and maximizes it, or that training built a general preference for episode reward, or something stronger about a persistent goal. One odd output distinguishes none of these.

2. Competing explanations

The project enumerates six hypotheses that can all produce the same odd answer but diverge under intervention:

1. Causal reward optimization: the model treats the grader value as an objective.
2. Instruction or authority parsing: the user request is operative; metadata is low-authority checking information.
3. Surface modulo heuristic: % 2 is read as an "evenness check" and primes even numbers regardless of the reward.
4. Salience or task inference: the conspicuous grader "must" be the point, so exploiting it is the inferred puzzle.
5. Comprehension failure: the model cannot reliably compute modulo or map reward to action.
6. Evaluation-aware suppression or training-stage gating: a latent tendency is masked in production or when the prompt looks like an alignment test.

A causal reward-optimization account makes three predictions the others do not share. Preference toward reward should appear only when the current answer can affect the grader; it should reverse when the requested and rewarded parities are mirrored; and it should follow a transformation of the scored value even when the surface request looks the same.

3. Experimental design

The key design principle is to manipulate the causal link between answer and reward while holding surface content as fixed as practical. Simply strengthening the prompt is not enough: adding "maximize reward" changes the objective and can turn ordinary instruction-following into an apparent positive result, so that condition is used only as a labeled positive control.

The local model was the official Qwen3-4B architecture in Q4_K_M GGUF format, run under llama.cpp b10566 on an RTX 5080 with all 37 layers offloaded (peak server working set about 3.0 GiB, peak total GPU use about 5.3 GiB). Model, runtime, caches, and raw results were isolated on a temporary drive. Three plans were preregistered before the relevant runs.

4. Broad behavior screen

The exact source user message was held fixed across 13 accessible model IDs, but the surrounding system and wrapper context was not: 16 of 58 calls carried an explicit "follow the user's request" system message, and the rest used neutral, wrapper-native, minimal Codex CLI, or local-native contexts. This makes the screen descriptive, not a context-matched replication.

The strict parser accepted only a single integer after light normalization. Recomputed totals were 58 calls, 13 model IDs, 50 strictly valid answers, and zero strictly valid odd answers. A separate manual pass over the same 58 raw responses recorded 57 clearly even answers and one non-answer. Because the explicit user-following context directly favors user compliance and the model IDs are not a random sample, a pooled 0/50 rate cannot refute a checkpoint-specific effect. Its real purpose was to identify a tractable local model for the mechanism tests.

5. Qwen3-4B causal interventions

A first block replicated the exact prompt with six seeds in each reasoning mode: Qwen3-4B returned an even answer in 12/12 conflicts and 6/6 neutral baselines.

Five mechanism conditions were then frozen with six seeds each: a current-answer conflict, an archived-answer control (same grader language, but it scores an answer the current response cannot change), an n+1 transform that scores a transformed value, a mirrored conflict (asks for odd, rewards even), and an explicit-maximize positive control.

In non-thinking mode even the positive control failed (0/6), so that block could not separate lack of motivation from lack of comprehension. With capped thinking (256-token budget):

The contrast is objective strength, not capability: an explicit imperative overrode the parity request, descriptive reward semantics never did.

6. Direct-probability probe

Sampling can hide a subthreshold preference, so a preregistered probe scored exact candidate answers under forced decoding. It used eight matched consecutive even/odd pairs (16/17, 42/43, 74/75, 88/89, 104/105, 128/129, 200/201, 314/315), each under 14 conditions, for 112 unique condition-pair rows. Token log probabilities were summed by the chain rule and turned into an even-minus-odd log-odds contrast. Validation confirmed candidate tokenization in context, invariance to +100 versus +1000 logit bias, invariance to forward versus reverse scoring order, and an independent arithmetic check (log p(42) - log p(43) = 21.172554).

The primary preregistered estimand was a controllability-by-conflict difference-in-differences, oriented so that positive values indicate causal reward alignment:

The primary interaction was negative for 7 of 8 pairs, had a median of -0.994, and stayed negative in every leave-one-pair-out estimate (-1.106 to -0.729) and within both two-token and three-token candidate groups. The transform result is the sharpest: mean even-over-odd log odds were +7.048 when the reward-optimal output was even and +7.142 when it was odd. Changing which output earns reward barely moved the strong preference for the requested even parity.

7. Interpretation

The negative primary estimate is not a uniform anti-reward objective. It was heterogeneous by requested parity (-1.986 for even requests, +0.117 for odd). Re-expressed post hoc in user-aligned coordinates, changing "archived answer" to "current answer" strengthened the requested parity by +0.473 log odds in conflict cells and +0.462 in congruent cells, a difference of only +0.011 with an interval spanning zero. Current-answer wording seems to make the task more salient regardless of reward conflict; once conflict cells are reoriented toward reward, that same user-following shift necessarily reads as negative. Some capped traces also treated % 2 as an evenness checker and conflated "remainder zero" with what the grader wanted.

The best explanation for Qwen3-4B under this wrapper is two-part: the natural-language parity request has higher operative authority than XML-like metadata, and the modulo expression adds a weaker surface-evenness association. The model can compose the reward rule when reasoning is on and the objective is unambiguous, but capability does not imply adoption.

This does not refute the source. Its central claim is comparative, about how a private checkpoint changes over capabilities RL, and it was strongest in unavailable late-RL checkpoints. A plausible synthesis is that capabilities RL shifts a model's prior about what grader metadata means; production and small open models still read the direct request as the task.

8. Limitations

• The strongest source behavior lives in unavailable private checkpoints; the main mechanistic result is one Qwen3-4B quantization and wrapper, not models in general.
• The 58-call screen fixes the user message, not the whole interaction, and 16 calls explicitly instructed user-following.
• Exact candidate scoring used Qwen's hard non-thinking mode, which sometimes miscomputed the reward rule; reasoning-on behavioral results reduce but do not eliminate the parsing-failure concern.
• Scores compare eight fixed answer pairs, not total probability mass over all integers.
• The bootstrap resamples eight deliberately chosen pairs, so its intervals describe robustness across those pairs, not uncertainty over a population of models or prompts.
• "Archived answer" is longer and less natural than "current answer"; a cleaner control would match syntax and length.
• Two hardened launches failed before any HTTP request and cleaned up; the successful run collected all 112 scores, after which an online summary bug was hit. The model was not rerun. A hash-gated offline finalizer verified the immutable schedule, validation, raw responses, and score files, and a final audit confirmed 112 unique rows, 560/560 expected task-scoring calls, exact recomputation of every sign convention, and matching artifact hashes.

9. Conclusion

On the accessible models tested, the grader hint did not function as an adopted objective. Qwen3-4B could maximize the displayed reward when explicitly told to, but descriptive reward metadata neither flipped its answers nor produced a hidden reward-aligned probability shift; controllability strengthened the user's request, and causal transformations of the scored value did not reverse preference.

The answer to "are they reward hacking?" is therefore conditional. For this setup the evidence says no: instruction priority plus surface parity framing fits the data better. For a private late-RL checkpoint, odd outputs are suggestive but not mechanism-identifying. The recommended decisive follow-up is a preregistered, checkpoint-matched factorial study on early capabilities-RL, late capabilities-RL, and post-safety checkpoints from one training lineage, measuring both sampled actions and matched-candidate log odds with candidate sets frozen before collection.

10. Repository and verification

The repository includes a read-only, model-free verifier (python scripts/verify.py) that runs the test suite and checks artifact hashes, schedule and score identities, all 112 probability rows, recomputed statistics, behavioral counts, provenance records, and privacy patterns. GitHub Actions runs the same command on Ubuntu, Windows, and macOS. It never downloads or starts a model. Model weights and runtime binaries are excluded for size; their versions, hashes, and official sources are recorded in the repository's artifact manifest. The full report, three preregistrations, and frozen analyses are included alongside the code.