My tool’s instructions hid a feature it already had

The instructions for my MIDI editor told an AI agent not to use a feature I had already shipped.

Battuta is a command-line tool I use to work on music with an agent. The agent can inspect a MIDI file and write requests to change notes or instruments; I listen to judge the result. The tool already supported changing the instrument assigned to a MIDI channel. But its agent instructions still said those edits would fail to parse. Following that advice would leave a supported operation unused.

I had updated the code without keeping its descriptions in agreement. The README, command help and agent instructions each carried their own account of what the tool could do. Each had a reasonable purpose: introduce the project, explain a command, or tell an agent how to work with it. Maintaining an operation list in each place meant that adding a feature could leave several different answers behind.

One remaining list can still go stale

The first repair brought those descriptions into agreement. A later pass removed the exhaustive lists outside mid apply --help and directed readers there. The README could still introduce the tool; it no longer had to enumerate its operations.

That reduced the maintenance work, but the help was still written by hand. The next code change could leave it behind again. A test could compare the help with the operations the tool accepted, except that writing an expected list into the test would create another copy to maintain.

The parser already knew the answer. Every edit request names an operation in a field called kind. Give it an unknown name and its error lists the names it would have accepted. The test deliberately submits an empty name:

{ "edits": [ { "kind": "" } ] }

The request is supposed to fail. The useful part is the rejection: the accepted names come from the same type definition that the parser uses, so adding an operation changes the test’s expectation without a second list being updated by hand.

The test then runs the built command’s apply --help and compares what it says with those names. It looks for both missing operations and names the help offers that the parser does not recognise. An omission can keep a caller from discovering a feature; an invented name sends them toward a request that will be rejected.

The implementation record reports deliberately breaking the help both ways. Removing resize_note was caught. Replacing delete_note with make_sadder was caught too. These experiments checked that the test would object when the help changed, even though the underlying editing code had not.

This still leaves plenty for a documentation review to do. An operation can keep the same name while its parameters change, and a correctly spelled name can accompany a misleading explanation. The check covers the inventory. Deciding whether the instructions actually teach someone how to use those operations takes a different kind of reading.

let refusal = serde_json::from_str::<battuta::EditSet>(
    r#"{ "edits": [ { "kind": "" } ] }"#,
)
.expect_err("an empty kind is not a kind")
.to_string();