Ask Five Times, Keep the Majority
Ask a model the same question twice and you can get two different answers. The cheap fix: ask several times and keep the answer that comes back most often.
Implement consistent_answer(llm, question, samples=5):
- Call
llm.ask(question)exactlysamplestimes, same question each time. - Tidy each reply before counting: strip whitespace, lowercase it, drop a trailing period.
"144."," 144"and"144"all count as"144". - Return the tidied reply that appeared most often. On a tie, any tied answer will do.
The catch: skip step 2 and every count is 1, and "most common" quietly becomes "whichever came first". Return the tidied text: "Paris." gives you "paris".