Recursive Retrieval: Multi-Hop RAG for Complex Questions

Single-pass retrieval works fine for simple FAQ queries. But when questions require multi-hop reasoning across document hierarchies—"Compare the refund policies across our Q3 and Q4 updates and explain how they affect European customers"—flat vector search falls apart. Recursive retrieval solves this by iteratively drilling down through document structures, combining summaries with granular chunks. Teams using RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) report 20% absolute accuracy gains on complex question-answering benchmarks compared to baseline RAG.

The problem with flat retrieval

Traditional RAG chunks documents into fixed-size pieces (200-500 tokens), embeds them, and retrieves top-k matches based on semantic similarity. This breaks when:

• Answer requires context spread across multiple chunks. A query about "company-wide changes in 2024" might need information from Jan, Jun, and Dec meeting notes—chunks that won't co-occur in top-5 results.
• Document hierarchy matters. Legal contracts have nested clauses; research papers have section dependencies. Flat chunking destroys this structure.
• Queries need iterative refinement. User asks a broad question, system retrieves high-level summaries, user needs drill-down details—single-pass retrieval can't adapt.

In production, flat retrieval plateaus at 60-65% accuracy on multi-hop questions. Recursive approaches push this to 80-85% by preserving document structure and enabling iterative context expansion.

Three types of recursive retrieval

1. Page-based recursive retrieval

Start with document-level summaries. When a summary matches the query, recursively fetch the full pages or sections within that document. Think of it as: "Find the right book → open the relevant chapter → pull the specific paragraph."

Best for:

• Large document collections (100+ files) where page-level precision matters
• Legal contracts, policy manuals, multi-chapter reports
• Scenarios where summary-to-detail traversal reduces noise

2. Information-centric recursive retrieval

After initial retrieval, extract supporting concepts or entities mentioned in the chunks. Use those as search terms for a second retrieval pass. Example: first pass finds "Q3 policy change," second pass retrieves chunks containing "refund timeline" and "EU compliance"—concepts referenced but not explicitly matched in the initial query.

Best for:

• Knowledge bases with heavy cross-referencing
• Technical documentation where concepts link across pages
• When you need to surface implicit dependencies

3. Concept-centric recursive retrieval

Build a concept graph from your corpus upfront. At query time, retrieve matching concepts, then traverse the graph to pull related concepts and their associated chunks. This is similar to GraphRAG but focuses on iterative concept expansion rather than graph-wide reasoning.

Best for:

• Academic research, scientific papers with dense citation networks
• Internal wikis with tagged concepts
• When concept relationships are well-defined

RAPTOR: The benchmark implementation

RAPTOR (introduced in arXiv:2401.18059) builds a retrieval tree with recursive summarization:

1. Chunk documents into 100-token segments (sentence boundaries preserved). Embed with SBERT.
2. Cluster chunks using Gaussian Mixture Models (GMM) based on embedding similarity.
3. Generate summaries for each cluster using an LLM (GPT-3.5 or GPT-4).
4. Recursively repeat: treat summaries as new "chunks," cluster them, summarize again. Build a tree from leaves (original chunks) to root (corpus-level summary).
5. At query time: retrieve from all tree levels (summaries + chunks), rank by relevance, combine top results into LLM context.

On the QuALITY benchmark (complex question-answering over long documents), RAPTOR + GPT-4 achieved20% absolute accuracy improvement over flat retrieval. On NarrativeQA (story comprehension), gains were 12-15%.

Implementation: Building a recursive retrieval pipeline

Here's a production-ready implementation using LlamaIndex and OpenAI. This builds a two-level recursive structure: document summaries → granular chunks.

Step 1: Install dependencies

pip install llama-index openai sentence-transformers scikit-learn

Step 2: Build the retrieval tree

from llama_index import Document, VectorStoreIndex, ServiceContext
from llama_index.node_parser import SimpleNodeParser
from llama_index.embeddings import OpenAIEmbedding
from llama_index.llms import OpenAI
from sklearn.cluster import KMeans
import numpy as np

# Load documents (Thread Transfer bundles work great here)
documents = [
    Document(text="Q3 2024 refund policy: 30-day window for EU customers..."),
    Document(text="Q4 2024 update: Extended to 45 days for all regions..."),
    # Add more documents
]

# Parse into chunks
node_parser = SimpleNodeParser.from_defaults(chunk_size=100, chunk_overlap=20)
nodes = node_parser.get_nodes_from_documents(documents)

# Embed chunks
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
embeddings = [embed_model.get_text_embedding(node.text) for node in nodes]

# Cluster chunks (using k-means as simplified GMM alternative)
num_clusters = max(2, len(nodes) // 10)  # ~10 chunks per cluster
kmeans = KMeans(n_clusters=num_clusters, random_state=42)
cluster_labels = kmeans.fit_predict(embeddings)

# Generate summaries for each cluster
llm = OpenAI(model="gpt-4o-mini")
summaries = []
for i in range(num_clusters):
    cluster_chunks = [nodes[j].text for j in range(len(nodes)) if cluster_labels[j] == i]
    summary_prompt = f"Summarize these related chunks concisely:\n\n" + "\n\n".join(cluster_chunks)
    summary = llm.complete(summary_prompt).text
    summaries.append(Document(text=summary, metadata={"level": "summary", "cluster": i}))

# Combine summaries + original chunks into index
all_docs = summaries + [Document(text=n.text, metadata={"level": "chunk"}) for n in nodes]
index = VectorStoreIndex.from_documents(all_docs, embed_model=embed_model)

print(f"Built recursive index: {len(summaries)} summaries, {len(nodes)} chunks")

Step 3: Query with recursive retrieval

query_engine = index.as_query_engine(
    similarity_top_k=10,  # Retrieve from both summaries and chunks
    response_mode="tree_summarize"  # Hierarchical synthesis
)

response = query_engine.query(
    "How did the refund policy change between Q3 and Q4 2024 for European customers?"
)

print(response.response)
# Expected: Synthesizes info from Q3 summary + Q4 chunks + EU-specific details

Advanced: Query decomposition for recursive answering

For ultra-complex queries, decompose them into sub-questions, answer each recursively, then synthesize:

from llama_index.query_engine import SubQuestionQueryEngine
from llama_index.tools import QueryEngineTool

# Create sub-question engine
tools = [
    QueryEngineTool.from_defaults(
        query_engine=index.as_query_engine(),
        name="policy_search",
        description="Search company policies and documentation"
    )
]

sub_query_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=tools,
    llm=llm
)

# Complex query gets decomposed automatically
response = sub_query_engine.query(
    "Compare refund policies for EU vs US customers across Q3 and Q4 2024"
)
# Internally generates sub-questions:
# 1. What was the Q3 refund policy for EU customers?
# 2. What was the Q4 refund policy for EU customers?
# 3. What was the Q3 refund policy for US customers?
# 4. What was the Q4 refund policy for US customers?
# Then synthesizes comparison

print(response.response)

Benchmarks: Recursive vs flat retrieval

Internal tests on Thread Transfer customer data (50 multi-hop queries across policy docs):

Key takeaway: Two-level recursive gives you 80%+ accuracy at reasonable cost. Only move to full RAPTOR trees if your queries demand 85%+ and latency isn't critical.

Production tips: When to use recursive retrieval

Use recursive retrieval when:

• Queries require multi-hop reasoning (compare, summarize across, explain connections)
• Documents have clear hierarchies (reports, legal contracts, technical specs)
• Your corpus is 10k+ chunks where document-level context matters
• Accuracy is more important than sub-second latency

Stick with flat retrieval when:

• Queries are single-hop ("What is X?" "Find Y's definition")
• Documents are short and self-contained (FAQs, product specs)
• Latency requirements are strict (under 500ms)
• Budget is tight (recursive adds 40-150% cost overhead)

Thread Transfer integration

Thread Transfer bundles are ideal input for recursive retrieval pipelines. Each bundle contains structured conversation history with decision points, context links, and temporal markers—perfect for building summary hierarchies. Export bundles as JSON, extract message threads, and treat each thread as a document for recursive indexing. This preserves conversation flow while enabling cross-thread reasoning.

2025 developments: Adaptive recursive retrieval

Static retrieval trees become stale as documents change. adRAP (adaptive Recursive Abstractive Processing) incrementally updates RAPTOR structures without full re-computation. When a new document arrives, adRAP re-clusters only affected nodes and propagates summary updates upward through the tree.

Early benchmarks show adRAP maintains 95%+ retrieval accuracy with 70% less compute overhead compared to rebuilding the tree from scratch. Expect this to become standard in production systems by late 2025.

Combining recursive retrieval with re-ranking

For maximum accuracy, chain recursive retrieval with a re-ranking layer. Retrieve 20-30 candidates recursively, then use a cross-encoder (Cohere Rerank, Jina Reranker, or BGE Reranker) to re-score them against the query. This cuts false positives by 30-40% while preserving recall.

from llama_index.postprocessor import CohereRerank

# Add reranker to query engine
query_engine = index.as_query_engine(
    similarity_top_k=20,  # Retrieve more candidates
    node_postprocessors=[
        CohereRerank(top_n=5, model="rerank-english-v3.0")
    ]
)

response = query_engine.query("Complex multi-hop question")
# Returns top 5 re-ranked results from 20 recursive candidates

Final recommendations

Start with two-level recursive retrieval (document summaries → chunks). Measure accuracy on 50+ representative queries. If you're hitting 75%+ accuracy, stop there. If not, graduate to full RAPTOR trees or add query decomposition. Always benchmark against flat retrieval to justify the added complexity.

Recursive retrieval is not a silver bullet—it trades latency and cost for accuracy on complex queries. Use it where it matters, pair it with re-ranking for precision, and integrate it with Thread Transfer bundles to leverage structured conversation context.