Prompts are code, so version them like code

Author:

SquidTrain

Time for reading:

5 min read

Prompts are code, so version them like code

A prompt determines system behaviour. It gets edited under time pressure, it breaks in ways that are hard to reproduce, and it is frequently the least controlled artifact in the stack. The same organisation that requires two approvals to change a database index will let someone edit a prompt in a web form at four in the afternoon, with no history and no review.

This is not a discipline problem. It is a tooling default. Most platforms present prompts as configuration, and configuration is culturally exempt from the controls applied to code. The exemption made sense when configuration meant a timeout value. It does not survive a prompt that decides which customers get flagged for manual review.

The usual state of things

Prompts tend to live in a config UI, a shared document, or inline in a function that three people have edited.

There is no history, so when behaviour changes there is no way to see what changed. Someone reports that outputs got worse last week. You look at the prompt. It contains the current text and no record of any other text it has ever contained. The investigation ends there, or it turns into interviews.

There is no review, so a wording change ships without a second reader. This matters more than it sounds, because prompt edits have non-local effects. Removing a clause that seems redundant can change behaviour on inputs nobody was thinking about. A second reader catches the edit that quietly dropped a constraint.

And there is no link between a prompt version and the outputs it produced, so debugging a bad result from last week means guessing. You have the output. You have the current prompt. You do not have the prompt that produced the output, which is the one you need.

What versioning actually buys you

Putting prompts in the repository as files gives you the same properties you already rely on for code.

Diffs show exactly what changed, down to the word. This is more useful for prompts than for code, because prompt regressions are frequently caused by changes that look cosmetic. A reordered instruction list, a softened verb, a removed example.

Review catches the edit that removed a constraint. Blame tells you when a behaviour was introduced, which converts an open-ended investigation into reading one commit. Rollback is a revert rather than an attempt to remember the previous wording.

It also makes prompts testable. A prompt in a file can be loaded by a test that runs it against known cases and asserts on the output. A prompt in a text box cannot. This is the property that matters most, because it is what lets you change a prompt without fear.

There is a secondary benefit worth naming: prompts in the repository are visible to the people maintaining the surrounding code. A prompt in a separate system is invisible during code review, so an engineer changing the calling function has no way to see the instructions that function depends on.

A workable structure

Keep each prompt in its own file, named for the task it performs. One file per task, not one file containing every prompt, because a single file makes diffs noisy and blame useless.

Store the model, temperature, and any other parameters alongside it, because those change behaviour as much as the text does. A prompt that works at temperature zero and fails at temperature one is not a prompt problem, and you cannot see that if the parameters live in a different system. Keeping them together means one diff shows the whole change.

Record the prompt version with every logged output so a bad result can be traced back to the exact input that produced it. A commit hash or content hash stored on the log line is enough. This single practice converts most prompt debugging from guesswork into lookup.

If the same instruction appears in several prompts, factor it out into a fragment and compose. Duplication drifts, and drifted prompts fail in ways that are hard to attribute: two tasks that should behave identically diverge because someone updated one copy. The usual candidates are output format instructions, tone constraints, and domain definitions.

Resist over-factoring. A fragment used by one prompt is indirection with no benefit, and heavily composed prompts become hard to read as a whole, which matters because the model reads them as a whole.

What to log

Version control answers what the prompt said. Logging answers what happened when it ran. You want both, and the link between them.

At minimum: the prompt version, the model and parameters, the input, the raw output, and whether any validation or retry logic fired. The raw output matters specifically — logging the parsed result discards the evidence you need when parsing is what failed.

Inputs and outputs may contain sensitive data, so this intersects with your retention and privacy rules. Decide the policy deliberately rather than discovering it during an audit. Redaction at write time is usually easier than deletion later.

The objection

The common argument against this is that non-engineers need to edit prompts, and a repository puts them behind a pull request.

That is a real constraint. The people with the domain knowledge to write a good prompt are frequently not the people comfortable with git, and routing every wording change through an engineer is both slow and a poor use of everyone involved.

But it is a workflow problem rather than a reason to abandon version control. A review interface that writes to the repository serves both needs: the editor gets a text box, and the system still gets history, review, and rollback. Several tools do this, and a thin internal one is not difficult to build for a small number of prompts.

The intermediate position, if that is too much to build now, is to keep prompts in files and give non-engineers a documented path to request changes with a named owner who applies them. Slower than a text box, considerably faster than reconstructing history you never kept.

Giving up history is a large price for a convenience that can be built.