Pinning Both Ends: Letting AI Rewrite 15k Lines of Test Tooling

Tool Y, an old test generator, shrank from about 15,000 lines to about 3,500 in 3.5 working days. Not one old golden file changed, except the ones the upstream specification showed were wrong.

The method is the same as in Part 1: pin both ends, then let the AI rewrite the middle. The input end is the parsed data. The output end is the golden files, byte for byte. The old tool itself was the check, and I read the AI's code after every round.

The problem

Tool Y was one engineer's personal tool. It reads test specifications from spreadsheets and writes golden files: generated test cases, kept as the known-good set. Another flow, the downstream flow, runs those files as its regression suite.

Nobody had maintained tool Y for a long time, so my manager asked me to take it over. The first job was to upgrade it to a newer upstream specification version. The upstream specification is an outside standard. It defines the element types that every test case is built from.

Tool Y works, but nobody planned its structure. Every new need added a new script, a new config, or a new copy of an old file. The tree, with names changed:

tool-y/
├── README.md                manual post-processing steps
├── config_main/             spreadsheets and JSON for the main CLI
│   ├── elements.xlsx
│   ├── sets.json            refers to spreadsheet row numbers
│   └── ...
├── gen/                     CLI entry point, plus a flag that acts as another
│   ├── main.py
│   ├── elements.py          one hand-written class per element type
│   ├── utils.py
│   └── golden_b.py          a second golden path
├── flow/                    mode dispatch, called back from gen/
├── post/                    4 scripts, chained through JSON files
├── config_b/
│   └── elements.xlsx        same file name as config_main/, different content
├── gen_b/                   generator B: its own entry point,
│   ├── main.py              and its own copies of elements.py and utils.py
│   ├── elements.py
│   └── utils.py
└── gen_c/                   generator C: reads a spreadsheet outside the repo

A test case can stand alone or combine several others. These are the modes. Each target, one setup of the downstream flow, has its own generator.

What I found when I read the code:

Copies that drifted. Generator B started as a copy of the main helpers. Now the two differ by thousands of lines. A bug fix in one needs a manual check of the other.
Generators that skip the main path. Generators B and C never call the main generator, so a fix there does not reach them.
Two golden paths. The README names one command. A script makes golden files another way. From the code, I couldn't tell which one is current.
Inputs outside git. Generator B downloads its spreadsheets from a shared drive. Generator C reads a hard-coded path. A git tag alone can't rebuild their output.
A manual release. Four scripts pass data through JSON files, then a list of hand steps in the README.

The worst part was the spreadsheets themselves. Each specification version had its own spreadsheet, and every element type had its own tab in it, far too many to scroll through. The tabs sit along the bottom of the window. Scrolling right through the tabs, I could never reach the one I wanted. Nobody could review a test case in there, let alone maintain one.

That scale also hides mistakes. Combined test cases point at spreadsheet rows by number, and the tool checks only that the number is in range. So I tried it. I inserted one valid row and ran the same command again. No warning. The case had the same name, but the golden file was different.

The second problem was the generator itself. Every element type is a hand-written class. The upstream specification already publishes machine-readable definitions of its element types, but tool Y never reads them. So a new specification version means editing the classes one by one, by hand. And nothing checks that a class still matches the specification. The upgrade I was asked to do was exactly this work.

Before: a person reads the upstream specification and writes one class per element type. Nothing checks the classes against it. After: the generator and the validator read the same machine-readable definitions, so a new specification version is a data change.

Patching one script at a time would leave the next owner the same problem. So I rewrote it, under three rules I couldn't break:

Old test cases stay the same, byte for byte. The only exceptions are test cases that were already wrong.
Every output is valid input for the downstream flow.
The downstream flow doesn't change. Same files, same format, same place.

The goal: all inputs in git, one parser, one validator, and one generator that reads the upstream specification's machine-readable definitions. Then a new specification version is a data change, and a new test case is a recipe, not code.

How I did it

Transforming tool Y. Three parsers and their tangled generators become one parser, one validator, and one generator. Both ends stay pinned: the same parsed data going in, the same golden files coming out.

Step 1: Lock the golden output

Before I touched any code, I ran the old tool once in every mode and kept the output. These are the old golden files. The new tool has to reproduce them byte for byte.

Generators B and C still read inputs from outside git. I ran them as they were, then moved those inputs into git, so every later run starts from the same files.

Then I wrote a small compare tool. It takes two output trees and reports every file that differs, down to the field value. CI runs it on every change.

Step 2: Pin the input end

First, the spreadsheets needed a replacement. The recipe format had to be easy to read for both a human and a parser, and keep the reason for a case next to the case. And no new dependency just to read a config file. I looked at six:

Format	Comments	Clean git diff	Parser	Catch
Spreadsheet (old)	no	no	third-party	binary file, references by row number
JSON	no	yes	stdlib	no comments, so no "why"
YAML	yes	yes	third-party	guesses types, e.g. `1.10` → `1.1`
Python as config	yes	yes	it's code	a recipe becomes a program
Racket DSL	yes	yes	easy to write	few engineers read Racket
TOML	yes	yes	stdlib (3.11+)	deep nesting gets awkward

YAML reads version: 1.10 as the number 1.1. I don't want the format to guess. I really wanted Racket, but the next owner would have to maintain my interpreter. So I chose TOML: comments, explicit types, and a parser in the Python standard library. A recipe is one TOML block per test case:

# Recipes for upstream specification version "1.10".
spec_version = "1.10"  # a string, so it stays "1.10"

[[case]]
name = "case_001"
test_intent = "<one line: why this case exists>"
kind = "alpha"
mode = "A"
size = 4
strict = true

I had the AI write a one-time converter. Each spreadsheet row became a recipe, and every reference by row number became a reference by name.

Then I had the AI write the input pin: parse the same test cases with the old code and the new parser, and check that both give the same data. A small adapter turns the new shape into the old one first. Once it passed, I deleted the converter.

The validator defines valid input. It also checks each case against the upstream's machine-readable definitions. That is the only check a new specification version gets, because it has no old golden files. It shows the case is legal. It does not show the case tests the right thing.

This check also found old bugs. Some old test cases failed the upstream definitions. Copying them would only keep the bugs. So I fixed them or dropped them, and took them out of the comparison.

Step 3: Move one element type, then two, then batches

I moved one element type first, to see whether the AI builds this structure: parser → validator → generator → golden files. Then two. Then batches of 5, then 10, until every element type had moved and every check was green.

The AI did get things wrong. But the check went red, and the AI fixed its own work before it ever reached me. The pinned ends weren't only my safety net. They were the AI's feedback loop.

After each round, I read the code by eye before I let the next round grow. I also checked that the change touched none of the pins: the old golden files, the compare tool, or the parse test. That review was the only guard. No tool stopped the AI from editing the pins. Next time, I will make them read-only with chmod, so an accidental edit fails. CI will also check their hashes against a list kept outside the agent's workspace, because an agent with a shell can run chmod too.

The pins do not check design. For the first element type, I read the design and decided where each part goes. I dictated the internal functions and the structs by hand. I still don't have a good method for that. If anyone has one, please let me know.

Step 4: Delete the old paths

Only after every mode matched byte for byte did I delete anything. The old generators went, the second golden path went, and the spreadsheets went.

Then the scaffolding went too: the adapter and the migration tests. Same as Part 1. If it stayed, it would test code that no longer exists. The validator tests and the generator unit tests stay.

Results

What	Before	After
Generator code (lines)	~15,000	~3,500
Time spent (working days)	-	3.5
Old golden files changed (bytes)	-	0*
Entry points	4	1
Inputs outside git	2	0
Upstream versions	1	multiple

*Every mode and every target, including generators B and C. Old test cases that failed the upstream definitions were fixed or dropped, and left out of the comparison.

The upgrade I was first asked to do is done, and the downstream flow uses its output.

The AI did most of the typing again. The pins caught its mistakes, and I read every round. Next time, the pins get a guard of their own.