The reward is code.
The experiment is a system.

1. A narrow question worth testing

Can a small pretrained base model become better at finding valid arithmetic expressions through reinforcement learning, without first receiving a supervised fine-tuning warm start? That is the question behind verifiable-rl-from-base. The default policy is Qwen/Qwen3.5-0.8B-Base; the task is Countdown arithmetic; the training algorithm is GRPO with LoRA adapters.

“From base” needs a precise reading. The model already has pretrained weights. This project skips a new supervised fine-tuning stage before RL; it does not train language or arithmetic from random initialization. Likewise, “verifiable” describes the output contract. A checker can prove that an expression solves a puzzle without proving that the model’s surrounding explanation is faithful or that a general reasoning capability has emerged.

A puzzle supplies numbers and a target. Given [8, 3, 4] and target 20, one solution is (8 - 3) * 4. Every supplied number must appear exactly once. Printing 20 is not a solution: it reaches the target while ignoring the permitted operands. This small distinction is the foundation of the reward system.

The appeal is experimental control. There is no human preference label or learned reward model between the completion and its correctness score. We can inspect the exact code that assigns reward, enumerate adversarial inputs, and compare the policy against a symbolic solver. The difficult work moves into specifying the task correctly, preserving a fair evaluation, and ensuring the trainer really executes the intended model.

2. Treat the experiment as a pipeline

The implementation separates these responsibilities. data.py loads and splits the dataset. prompts.py constructs the request. verifier.py interprets answers. rewards.py adapts verification results to TRL’s reward interface. train.py owns training and artifact persistence. Evaluation, aggregation, solving, generation, plotting, and reporting have separate modules.

This separation makes failures local enough to investigate. An invalid expression should fail a verifier test. A duplicate puzzle should fail a dataset or evaluation test. A model-wrapper mismatch needs an integration test. Putting everything in one training script makes it tempting to call a completed process a successful experiment, even when one of those contracts is wrong.

The artifacts matter as much as the Python entry points. Training writes configuration snapshots, dataset manifests, checkpoints, a final adapter, tokenizer files, and trainer state. Evaluation writes raw completion records alongside aggregate metrics. Reports are views over those records, not replacements for them. If a result looks surprising, the underlying expression and verifier flags should still be inspectable.

3. The verifier is the specification

The verifier accepts exactly one balanced, nonempty <answer>...</answer> block. Reasoning outside the block is allowed. Duplicate blocks, nested answer tags, or stray opening and closing tags invalidate the answer. A regex that merely finds one successful match is insufficient: it can overlook malformed tags elsewhere in the completion.

Accepted:
Reasoning text. <answer>(8 - 3) * 4</answer>

Rejected:
<answer>(8 - 3) * 4</answer><answer>
<answer>(8 - 3) * 4</answer></answer>
<answer>20</answer>  # valid literal, wrong input usage

Once extracted, the expression must fit a 200-character limit and a restricted character set. Python’s parser builds an abstract syntax tree, but the expression is never passed to eval. A small recursive interpreter allows integer constants and the binary operators +, -, *, and /. Calls, names, attributes, unary signs, exponentiation, floor division, and floating-point literals are rejected.

Evaluating the AST has two outputs: its rational value and a list of integer leaves. The list is converted to a multiset and compared with the input multiset. This preserves multiplicity: [5, 5, 2] requires two leaves equal to five and one equal to two. A set comparison would incorrectly erase that distinction.

Values use exact rational arithmetic. For example, 8 / (3 - 8 / 3) solves [3, 3, 8, 8] → 24: the inner division is 8/3, the denominator becomes 1/3, and the final result is exactly 24. There is no floating-point tolerance to tune and no possibility that a nearly correct value passes because of an epsilon threshold. Division by zero remains invalid.

This contract permits negative and fractional intermediate values, even though negative literals written with unary minus are not allowed. That is a deliberate task definition, not an implementation accident. Anyone adding a new benchmark must decide whether its rules match. “Countdown-like” datasets can disagree about operand reuse, allowed intermediates, and whether every number is mandatory.

4. Reward shaping creates signal—and its own risks

A base model can initially produce few fully correct answers. The default shaped profile therefore provides four additive rewards, implemented as independent functions sharing the same verifier.

A correct completion earns 2.85. A well-formed expression with the right operands but the wrong value earns 0.85. Printing the target inside a valid block can earn format and parseability credit—0.35—but cannot earn operand or correctness credit for the three-number puzzle above. These examples make the incentives visible rather than hiding them behind a single reward curve.

Shaping is an empirical choice. It may help the policy discover valid expression syntax, but it can also reward a stable population of neatly formatted wrong answers. The repository includes a binary profile, which gives only 2.00 for correctness, and a no-format profile. Comparing those profiles under the same evaluation protocol is more informative than assuming that a rising shaped reward means improved solving.

There is also a mundane interface invariant: reward functions must return one value per completion. Silently zipping unequal metadata arrays drops examples. The implementation uses strict alignment so that missing targets or number lists fail visibly. In an RL loop, a data plumbing bug can otherwise masquerade as a learning effect.

5. How grouped rewards become a policy update

GRPO, introduced in DeepSeekMath, compares multiple completions for the same prompt. Conceptually, the group provides a local baseline: a better-than-group completion gets positive advantage and a worse-than-group completion gets negative advantage. A schematic normalized advantage is:

The policy objective then uses token probability ratios, clipping, and optional reference-policy regularization. This equation is an intuition for the group-relative signal, not a transcription of every reduction or loss option in the library. The repository delegates optimization to TRL 0.29.0; its precise loss and normalization behavior belongs to that pinned implementation.

For this project, the important consequence is that the signal is relative within a puzzle. If every completion earns the same reward, there is no within-group ranking to learn from through that reward advantage. The trainer can execute perfectly while producing zero useful policy gradient from a group. Logging reward variance and the fraction of zero-variance groups helps distinguish an execution failure from a sparse or uninformative reward signal.

The default configuration uses eight generations, a per-device training batch of two, and eight accumulation steps. Under the supported single-process path, the configured rollout batch is 1 × 2 × 8 = 16 sequences. Because that count already includes repeated generations, it corresponds to two distinct prompts per generation batch—not sixteen prompts each multiplied by eight again.

LoRA constrains the trainable update to low-rank adapter matrices. In the familiar schematic W′ = W + (α/r)BA, the base matrix stays fixed and the adapter learns the correction. This reduces trainable parameter and optimizer-state costs, but it does not eliminate the forward and backward computation through the base network, generation cache, or activation memory. Small parameter count alone is not a complete GPU-memory estimate.

The repository uses rank 16 and alpha 32 in the standard configuration, with KL coefficient beta = 0.04. These are experiment settings, not established optima. The right next comparison is to change one variable while keeping problem identities and decoding fixed.

6. A random row split can leak the puzzle

The source dataset contains 490,364 rows in the version checked during development. Related rows may reorder the same operands or pair a combination with different targets. If [3, 8, 4] is in training and [8, 4, 3] is in evaluation, row identity has changed but much of the arithmetic search space has not.

The project assigns the whole sorted number multiset to a split. It hashes a seed and the combination key, converts the first eight hexadecimal digits of SHA-256 to an integer, and takes the remainder modulo 100. Buckets below 90 train; the rest evaluate.

combination_key([8, 3, 4]) = "3,4,8"
bucket = int(sha256(b"42:3,4,8").hexdigest()[:8], 16) % 100

train: bucket < 90
eval:  bucket ≥ 90

The target is intentionally absent from this split key. Different targets sharing the same inputs stay on the same side of the boundary. The seed changes the assignment reproducibly. An explicit disjointness assertion checks that no input combination appears in both resulting subsets.

Deduplication solves a separate problem. Evaluation collapses equivalent (sorted nums, target) puzzles before selecting the requested sample count. Group splitting prevents cross-split overlap; deduplication prevents repeated evaluation rows from quietly weighting one puzzle more heavily. The two operations are complementary.

There are still limits. Training uses its evaluation split repeatedly for monitoring. That makes it validation data, not an untouched final test set after many rounds of tuning. A credible final result needs a separately reserved benchmark. Changing the split seed also changes which examples are held out, so multi-seed studies should distinguish optimization randomness from a changed test population.

7. Exact solvability, approximate difficulty

The solver uses subset dynamic programming. Each input occurrence receives a bit position, including duplicates. For every subset, a table maps reachable rational values to candidate expressions. Combining two disjoint nonempty subsets yields addition, multiplication, both subtraction orders, and both division orders when the denominators are nonzero.

For the full input subset, a target lookup either returns a witness expression or proves that no expression exists within the enumerated operator contract. Because the witness is fed through the same verifier used for model outputs, solver tests exercise consistency across two independent implementations: expression construction and expression interpretation.

The computation is more expensive than the small input arrays suggest. There are exponentially many subsets, and combining tables requires considering pairs of reachable values. The public interface therefore caps inputs at five numbers. A 16-entry LRU retains completed value tables; bounding cache entries prevents retaining enormous numbers of tables, although a single five-number table can still be substantial.

Difficulty is a heuristic built from capped derivation counts, fractional-intermediate requirements, operator diversity of the selected witness, and arity. A rare solution receives a larger rarity term. A required fractional intermediate receives a penalty. The resulting score maps to easy, medium, or hard bands used by the curriculum configurations.

Those labels should not be oversold. The count is not a count of unique mathematical expressions; equivalent derivations and repeated operand positions affect it. The selected witness’s operator set is not a proof of globally minimal operator diversity. The bins are deterministic engineering features, not calibrated estimates of human or model difficulty.

The generator samples candidate inputs and targets, keeps solvable puzzles, and emits reference expressions. Five-number puzzles provide a useful arity shift from a three/four-number training source. For other custom JSONL benchmarks, disjointness must be checked explicitly rather than inferred from the fact that the file is called “OOD.”

8. pass@k must mean something reproducible

Strict accuracy is the fraction of individual completions that satisfy the entire verifier. It answers a different question from “did at least one of several attempts work?” Both can be useful, but they should not be merged into a headline number.

The current metrics implementation estimates pass@k for a problem with n sampled completions and c correct completions as:

The fraction being subtracted is the probability that a size-k subset contains only failures. Consider four samples with two correct answers. For k=1, the estimate is 1/2. For k=2, it is 1 − C(2,2)/C(4,2) = 5/6. For k=4, it is one. These are properties of the observed sample pool and estimator—not claims about this model’s measured performance.

Averaging that estimate over problems is order-independent. Taking only the first k stored results is not: reordering records can change the answer. The implementation rejects k when any problem has fewer than k samples, and summary output uses the minimum available sample count to avoid pretending every problem received the largest observed budget.

Sampling conditions matter. In this repository, one sample per prompt uses greedy decoding; more than one enables stochastic decoding. Pass@1 estimated from sampled completions is therefore not automatically the same experiment as a separately measured greedy accuracy. Base and trained models need identical decoding modes, temperature, token budget, prompts, and test identities.

Evaluation JSON stores those decoding settings, a dataset fingerprint, stable problem IDs, sample indices, completions, verifier flags, and exact values as rational strings. Token counts stop at the first EOS, including that EOS while excluding later padding. Custom JSONL evaluation rebuilds prompts and identities from the inputs and recomputes difficulty, instead of trusting supplied solution-bearing prompts or labels.

The HTML report summarizes the JSON; it cannot certify comparable runs. Confidence intervals and multi-seed statistical aggregation are not yet implemented. A small observed accuracy difference needs more evidence than a taller bar.

9. The failure that unit tests did not catch

The most consequential correction came from running the actual trainer. The configured Qwen checkpoint advertises a composite conditional-generation architecture. It contains the language model needed for Countdown, plus a multimodal wrapper. Loading a recognized architecture is not sufficient to establish that it behaves correctly through every batching pattern in a training algorithm.

In the reproduced integration failure, generation and GRPO scoring used different batch sizes. The multimodal wrapper retained position-related state from generation. A subsequent smaller scoring batch encountered incompatible dimensions and failed in the rotary-position path. The reward functions, parser, solver, and metric tests were all green; none exercised that state transition.

Before:
composite Qwen wrapper
  → generate a grouped batch
  → retain multimodal position state
  → score a smaller batch
  → incompatible tensor dimensions

After:
same checkpoint's text weights
  → Qwen3_5ForCausalLM
  → generate and score text batches
  → train, save, resume, reload

The shared model loader now selects Qwen3_5ForCausalLM for Qwen3.5 composite checkpoints. Training and evaluation use the same loader. The task is text-only, so the unused vision wrapper is unnecessary. The integration test compares the loaded language-model tensors and output head against the original tiny composite checkpoint to ensure this conversion loads actual weights rather than silently initializing replacements.

This is a bounded finding: a failure in the repository’s original integration path and a tested correction under the selected runtime. It is not a claim that every multimodal Qwen use is broken or that every version of the library behaves identically. That is precisely why dependency versions and a runnable reproducer are more useful than a vague compatibility statement.

10. A smoke test should leave evidence

The countdown-smoke command has two modes. CPU mode constructs a small random Qwen3.5 model containing both linear-attention and full-attention layers. CUDA mode loads the configured pretrained base model and the real Countdown dataset. Both call the production training function.

1. Load the model and tokenizer and create training/evaluation inputs.
2. Train to step one and require a saved checkpoint.
3. Resume that checkpoint and require global step two.
4. Load the final adapter through the evaluation entry point.
5. Require two evaluation records, then generate an HTML report and a chart.
6. Write acceptance.json with the mode, device, versions, and observed completion of the workflow.

A random tiny model is very unlikely to discover valid arithmetic in a few sampled tokens. If every reward is zero, an integration test can run a backward pass without demonstrating an effective parameter change. The CPU test therefore adds an explicitly labeled diagnostic reward that differentiates completions and checks that LoRA B matrices become nonzero. This reward tests the optimization plumbing; it is never added to the real-model CUDA test or normal experiments.

The scope distinction is important. This proves more than successful imports, but less than learned arithmetic. It establishes that a small representative architecture traverses the trainer, optimizer, serialization, resume, and inference path. The real-model CUDA test uses shorter completions and fewer generations than the full experiment, so even its success would not prove that the larger run fits the same memory budget.

All five implementation CI jobs passed: Linux/Windows unit-and-build jobs for Python 3.10 and 3.12, plus fresh Linux setup and offline trainer integration. The evidence is linked in the repository’s Actions history. CUDA performance, Docker execution, distributed training, and full-model accuracy remain outside the measured results in this post.

11. Clone, configure, and run on a real host

The primary setup path targets Linux, Python 3.10–3.12, and one NVIDIA GPU with a driver compatible with CUDA 12.8. A 24 GB GPU, 32 GB host RAM, and 20 GB free disk are planning starting points, not measured guarantees. Initial setup needs access to PyPI, the PyTorch wheel index, and Hugging Face.

git clone https://github.com/manishklach/verifiable-rl-from-base.git
cd verifiable-rl-from-base

bash scripts/setup.sh cuda
source .venv/bin/activate
countdown-doctor --config configs/qwen35-0.8b-grpo.yaml
countdown-smoke --device cuda

# Continue after inspecting acceptance.json:
bash scripts/run_experiment.sh

setup.sh creates the environment, installs the explicit CPU or CUDA PyTorch build, applies core runtime constraints, checks dependency consistency, and checks device access. Selecting the build explicitly matters when reusing an environment: the CPU and CUDA variants can otherwise satisfy the same public-version requirement while providing different hardware capabilities.

The constrained stack is PyTorch 2.9.1, Transformers 5.3.0, TRL 0.29.0, PEFT 0.18.1, Accelerate 1.12.0, Datasets 5.0.1, Matplotlib 3.10.8, and PyYAML 6.0.3. This is a core compatibility baseline, not a complete transitive lock. Archive pip freeze, model revision, dataset identity, and acceptance evidence for each host.

The setup intentionally avoids requiring optional compiled attention extensions. Qwen can take its PyTorch fallback, trading potential speed for a smaller installation surface. The default single-GPU pipeline uses the Python training entry point directly, so it does not depend on an interactive Accelerate configuration step.

For CPU-only development, use bash scripts/setup.sh cpu, activate the environment, and run countdown-smoke --device cpu --tiny. For the combined GPU workflow, bash scripts/quickstart.sh performs setup, preflight, acceptance, and the full experiment in sequence. Each command stops on failure. Acceptance refuses to overwrite an existing output directory; reruns should use a new --output path.

See the clone-to-training guide for the maintained procedure. A successful installation is only the first checkpoint. A successful short acceptance run is the next. A reproducible full experiment is a third, more demanding deliverable.

12. The research result still has to be earned

The next experiment should compare the untouched base policy and trained adapter on a fixed, independently reserved set of puzzle identities. Both need the same prompt construction and decoding budget. Repeat training across multiple seeds, preserve raw completion JSON, and report strict accuracy alongside syntax validity, operand usage, token length, pass@k, and failure categories.

Start with a small number of interpretable ablations: shaped versus correctness-only reward, ordinary training versus a difficulty curriculum, and perhaps three/four-number evaluation versus the five-number generator. A curriculum resume is not automatically equivalent to a single preplanned schedule. The shipped stages change cumulative step targets; optimizer and learning-rate scheduler behavior on resume must be inspected rather than assumed.

It would also be useful to measure how much improvement comes from learning the answer format versus solving more puzzles after controlling for format. If parseability rises and exact correctness stays flat, the model has learned a useful interface behavior without establishing better search. If correctness rises only after allocating more samples or longer completions, the extra inference budget belongs in the explanation.

The broader lesson from this repository is practical: a deterministic reward gives us a tractable boundary, but a trustworthy RL experiment requires several boundaries to hold at once. The verifier must describe the task. The split must describe what is unseen. The metric must describe the sampling budget. The loader must describe the actual model being trained. And the acceptance test must execute the path that readers will run after cloning.

The code is available in verifiable-rl-from-base, with v0.3.0 serving as the snapshot discussed here. The valuable next contribution is a documented real-GPU run with preserved artifacts and a fair base-versus-adapter comparison. That would turn a tested experiment framework into measured evidence about learning.

Code and primary references

Implementation details in this essay refer to the v0.3.0 source snapshot. The arithmetic and pass@k examples are illustrative calculations, not model benchmark results.

1. Project repository and v0.3.0 release.
2. Verifier, rewards, and exact solver.
3. Data preparation, metric definitions, and evaluation protocol.
4. Shared text loader, acceptance test, and CI evidence.
5. Setup guide and runtime constraints.
6. Shao et al., DeepSeekMath (2024), introducing Group Relative Policy Optimization.
7. TRL 0.29.0 GRPO documentation, for the library’s optimization and batching behavior.
8. Configured Qwen base model and Countdown source dataset.