Make the Model Admit It Doesn't Know
When retrieval comes back empty or irrelevant, a model's default is to answer confidently from nothing.
Implement grounded_answer(llm, question, context) that answers only from context, and returns exactly "I don't know." (DONT_KNOW) when the context does not contain the answer.
- Say in the prompt what to do when the answer is absent. "Answer from the context" leaves that case to the model's instinct, which is to guess.
- Give it an exact string to emit. "Say you don't know" produces fifteen phrasings you then have to detect.
- Normalise what comes back. A model sometimes adds a sentence of apology. Detect the abstention and return
DONT_KNOW, so callers get one string.
You are shifting a rate, not flipping a switch. The tests ask a real model: refund and shipping questions must carry 30 and 50; off-topic questions, or an empty context, must return exactly DONT_KNOW.