Ask Again When the JSON Is Wrong
When a model's reply fails validation, tell it what was wrong and ask again. A model that left out a field will nearly always supply it once it is told which field.
Implement ask_for_json(llm, prompt, schema, max_attempts=3). validate(reply, schema) is provided: it returns (obj, None) when the reply holds a JSON object matching the schema, and (None, error) when it does not, where error is a short sentence such as the field 'age' is missing.
- Start a conversation with one user message: the prompt, unchanged.
- Call
llm.messages.create(messages=...)and validate the reply's text. - If it is valid, return the object. Make no further calls.
- If it is not, grow the conversation by two messages: the model's reply, as it was, with role
"assistant"; then a user message that includes the error text and asks for the corrected JSON. Ask again with the whole conversation. - After
max_attemptscalls without a valid reply, returnNone.
The wording of your correction is yours; the tests check that the error text is in it. The model is scripted, so the tests read exactly what you sent on each attempt.