Retry a Failed Model Call
Model APIs fail. They rate-limit you, they time out, they return a 503 because a GPU somewhere fell over. Almost all of it is transient, and the difference between a flaky product and a reliable one is a retry loop that knows what's worth retrying.
Implement call_with_retry(fn, max_attempts=3, base_delay=1.0).
Call fn(). If it succeeds, return its result. If it raises:
- Retry
RateLimitError,TimeoutError, andServerError— these are transient. - Do not retry anything else. A
ValueErrorfrom a malformed request will fail identically forever; retrying just wastes time and money. - Wait
base_delay * (2 ** attempt)before each retry — 1s, 2s, 4s — using the providedsleep(). Hammering a rate-limited endpoint makes it worse. - After
max_attemptstotal attempts, re-raise the last exception.
Use the provided sleep() rather than time.sleep so the tests don't actually wait.