Explore character, sentence, and section-based chunking without requiring an API key.

Download the runnable notebook

This lesson was converted from the matching notebook in this repository. Outputs are cleared; read the lesson before running code, and never add API keys or private data.

Goal, prerequisites, and setup

Compare character-, sentence-, and section-based chunking and explain the trade-offs between context, overlap, and retrieval precision. Know Python strings, lists, loops, regular expressions, and file reading. No API key is required. Create a virtual environment only if you need the notebook’s optional data packages.

Run every chunker on the same short document. Inspect boundaries manually, count chunks and characters, and save a small table containing the method, size, overlap, and an example chunk.

# Size based chunking
# Chunk by a set number of charactesr
def chunk_by_char(text, chunk_size=150, chunk_overlap=20):
    chunks = []
    start_idx = 0

    while start_idx < len(text):
        end_idx = min(start_idx + chunk_size, len(text))

        chunk_text = text[start_idx:end_idx]
        chunks.append(chunk_text)

        start_idx = (
            end_idx - chunk_overlap if end_idx < len(text) else len(text)
        )

    return chunks

# Structure-Based Chunking
# Chunk by sentence
import re


def chunk_by_sentence(text, max_sentences_per_chunk=5, overlap_sentences=1):
    sentences = re.split(r"(?<=[.!?])\s+", text)

    chunks = []
    start_idx = 0

    while start_idx < len(sentences):
        end_idx = min(start_idx + max_sentences_per_chunk, len(sentences))

        current_chunk = sentences[start_idx:end_idx]
        chunks.append(" ".join(current_chunk))

        start_idx += max_sentences_per_chunk - overlap_sentences

        if start_idx < 0:
            start_idx = 0

    return chunks

# Semantic-Based Chunking
# Chunk by section
def chunk_by_section(document_text):
    pattern = r"\n## "
    return re.split(pattern, document_text)

with open("report.md", "r") as f:
    text = f.read()

# chunks = chunk_by_char(text)
chunks = chunk_by_sentence(text, max_sentences_per_chunk=5, overlap_sentences=1)
# chunks = chunk_by_section(text)

[print(chunk + "\n----\n") for chunk in chunks]

References and evidence

Choosing Your Strategy

Your choice depends entirely on your use case and document guarantees:

  • Structure-based: Best results when you control document formatting (like internal company reports)
  • Sentence-based: Good middle ground for most text documents
  • Size-based: Most reliable fallback that works with any content type, including code

Size-based chunking with overlap is often the go-to choice in production because it’s simple, reliable, and works with any document type. While it may not give perfect results, it consistently produces reasonable chunks that won’t break your pipeline.

Remember: there’s no single “best” chunking strategy. The right approach depends on your specific documents, use cases, and the trade-offs you’re willing to make between implementation complexity and chunk quality.