A Language for Writing Prompts

It is arduous to read and remember a whole prompt written in natural language. For example, Claude’s system prompt is 16,000 words long. If the subject described in the prompt is constantly changing (e.g., API specification), the prompt must be continuously updated, which is a demanding task.

This problem is very similar to the issue of inconsistency between traditional software specifications and their implementation. Because specifications are written in natural language, if the person in charge forgets, changes in the implementation may not be reflected in the specification, and discovering such inconsistencies is also presents a challenge.

Developing a programming language for writing prompts could be one solution. This is because if prompts are written in a formal language, it is possible to develop tools to analyze and verify their correctness. However, writing everything in a formal language, while powerful, is often too complex and inefficient for the general public.

Another approach is to find an intermediate form between natural language and formal language, and to use language models to verify the correctness of the prompt. One example is to wrap the changing parts with a <Mutable> tag, and have the language model repeatedly check the content of this section in a CI process. A kind of Markup Language for writing prompts could be created by introducing various tags.