Skip to content

< CurriculumRAG and grounding · 41 of 46 ·26 · Level 03, RAG

Make the Model Admit It Doesn't Know

hard · implement · Production

When retrieval comes back empty or irrelevant, a model's default is to answer confidently from nothing.

Implement grounded_answer(llm, question, context) that answers only from context, and returns exactly "I don't know." (DONT_KNOW) when the context does not contain the answer.

  1. Say in the prompt what to do when the answer is absent. "Answer from the context" leaves that case to the model's instinct, which is to guess.
  2. Give it an exact string to emit. "Say you don't know" produces fifteen phrasings you then have to detect.
  3. Normalise what comes back. A model sometimes adds a sentence of apology. Detect the abstention and return DONT_KNOW, so callers get one string.

You are shifting a rate, not flipping a switch. The tests ask a real model: refund and shipping questions must carry 30 and 50; off-topic questions, or an empty context, must return exactly DONT_KNOW.