Skip to content

< CurriculumRetrieval · 33 of 46 ·07 · Level 02, Search

Build a Document Chunker

easy · implement · RAG

Implement chunk_document(text, chunk_size, overlap), splitting text into a list of word chunks. Assume chunk_size > 0 and 0 <= overlap < chunk_size.

  1. Each chunk holds at most chunk_size words.
  2. Each chunk after the first starts overlap words before the previous one ended.
  3. Chunks are strings, words rejoined with single spaces.
  4. Never an empty chunk, and never a chunk whose words are all already covered by the previous one.
  5. Empty or whitespace-only input returns [].

The catch: rule 4. A naive range(0, len(words), step) emits a redundant tail chunk whenever the text divides unevenly.