Check Every Sentence Has a Source

medium · implement · RAG

A RAG system retrieved two passages about a bridge and wrote a four-sentence answer. Three sentences are right; the fourth says the bridge is 512 metres long, and the passage says 412. Check every sentence against the sources.

Implement unsupported(sentences, passages). sentences is the answer, already split into a list; passages is the list of source texts. Return the indexes of the sentences the passages do not support, in ascending order. A fully supported answer returns [].

A sentence is supported when the passages state it, in any wording: Cyclists can cross it is supported by it carries road traffic and a cycle lane. It is not supported when:

it contradicts the passages (512 metres, when the passage says 412);
it adds something the passages never say, however plausible;
it is true in the world but absent from the passages. Lisbon is the capital of Portugal is unsupported when the only source is a rainfall report. The question is what the sources say, not what is true.

How you check is up to you. jev.ask(state, questions) is the decision model: a yes/no question per sentence, all in one call, comes back as a probability each. llm.ask(prompt) is the chat model.

The tests compare your list with the right one for five answers and count calls: at most two per answer. Comparing words will not do it: the supported paraphrases share few words with their source, and the wrong number shares nearly all of them.