Stop Building Throwaway RAG Wrappers: Inside Kotaemon’s Extensible Document Engine

Most document QA interfaces hit a brick wall the moment you move past basic demo notebooks. They break when asked to cross-reference multi-page PDF tables, collapse under the latency of naive vector search, or lock you into rigid SaaS APIs with zero visibility into chunk citations. If you are building production-grade retrieval systems, you do not need another toy chatbot UI. You need a modular, ground-up framework that unifies hybrid search, multi-modal ingestion, and advanced agentic reasoning.

That is the engineering philosophy behind Kotaemon, an open-source RAG interface designed to bridge the gap between clean end-user workflows and deep backend customization.

+-------------------------------------------------------------------+
|                           Kotaemon UI                             |
|    (Gradio Interface: Auth, Document Viewer, Pipeline Settings)   |
+---------------------------------+---------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                        Reasoning Layer                            |
|    [FullQAPipeline]   [ReactAgentPipeline]   [RewooAgentPipeline] |
+---------------------------------+---------------------------------+
                                  |
            +---------------------+---------------------+
            v                                           v
+-----------------------+                   +-----------------------+
|   Retrieval Engines   |                   |    Execution Core     |
| - Hybrid (BM25 + Vec) |                   | - Local (Ollama/GGUF) |
| - Re-ranking Step     |                   | - Cloud APIs (OpenAI) |
| - GraphRAG / LightRAG |                   | - Multi-modal Parsers |
+-----------+-----------+                   +-----------------------+
            |
            v
+-------------------------------------------------------------------+
|                       Storage Subsystems                          |
|   Docstore: Elasticsearch / LanceDB / SimpleFileDocumentStore     |
|   Vectorstore: ChromaDB / LanceDB / InMemory / Milvus / Qdrant    |
+-------------------------------------------------------------------+

Architectural Personas: Who Is This For?

Kotaemon avoids identity crises by cleanly separating its code paths into three distinct usage tiers:

End Users: Need a zero-friction, multi-user document QA workspace. They get private/public file collections, shareable conversation histories, and an in-browser PDF viewer that highlights exact grounding citations with relevance scores.
Pipeline Developers: Need complete programmatic control. They import kotaemon as an orchestration library, customize UI components with the kotaemon-gradio-theme, and register bespoke reasoning engines directly in code.
System Contributors: Need a modular codebase to improve core interfaces, extend indexing backends, and push upstream enhancements.

Core System Capabilities

1. Hybrid Retrieval and Re-Ranking

A single retrieval strategy is rarely enough. Kotaemon defaults to a hybrid retrieval architecture that pairs full-text keyword indexing with dense vector similarity search. The pipeline passes candidate chunks through a dedicated re-ranking stage, filtering out noise and flagging low-relevance retrievals before sending the payload to the LLM.

2. Multi-Modal Document Extraction

Dense visual layouts frequently trip up standard parsers. Kotaemon integrates selectable parsing backends directly in the settings UI to handle tabular structures, figures, and complex multi-page layouts:

Cloud API Parsers: Azure Document Intelligence, Adobe PDF Extract.
Local Open-Source Engines: Docling, PaddleOCR.

3. Agentic Reasoning Strategies

When queries require multi-hop synthesis, simple single-turn retrieval fails. Kotaemon implements question decomposition and advanced agent architectures, allowing systems to break down queries and plan execution routes:

ReAct: Interleaves reasoning and action steps to query, verify, and formulate outputs iteratively.
ReWOO (Reasoning Without Observation): Decouples reasoning from execution to minimize API round-trips and optimize multi-step workflows.

4. Graph-Based Indexing Architectures

For domain corpora where entity relationships matter as much as raw proximity, the codebase includes reference implementations for graph-enhanced retrieval:

NanoGraphRAG & LightRAG: Lightweight graph indices designed for straightforward local integration.
Microsoft GraphRAG: Deep entity and relationship indexing pipelines compatible with OpenAI and Ollama endpoints.

Deployment and Infrastructure Setup

Kotaemon is distributed via containerized images and standard package installs.

                      +-------------------+
                      |   Source Repo     |
                      +---------+---------+
                                |
             +------------------+------------------+
             v                                     v
   [Docker Deployment]                   [Manual Installation]
   - Full Image                          - python >= 3.10
   - Full + Ollama                       - uv sync / conda env
   - Lite Image                          - Custom .env config

System Requirements

Runtime Engine: Python >= 3.10
Container Runtime: Docker (recommended for reproducible isolation)
Document Parsers: unstructured (required for parsing .doc and .docx formats beyond standard .pdf, .html, .mhtml, and .xlsx)

Containerized Deployment (Docker)

Kotaemon provides distinct Docker flavors depending on your footprint and runtime constraints:

bash
# 1. Full Image (Includes Unstructured for deep .doc/.docx parsing)
docker run \
  -e GRADIO_SERVER_NAME=0.0.0.0 \
  -e GRADIO_SERVER_PORT=7860 \
  -v ./ktem_app_data:/app/ktem_app_data \
  -p 7860:7860 -it --rm \
  ghcr.io/cinnamon/kotaemon:main-full

# 2. Full Image + Integrated Ollama Runtime (Air-gapped / Local inference)
docker run \
  -e GRADIO_SERVER_NAME=0.0.0.0 \
  -e GRADIO_SERVER_PORT=7860 \
  -v ./ktem_app_data:/app/ktem_app_data \
  -p 7860:7860 -it --rm \
  ghcr.io/cinnamon/kotaemon:main-ollama

# 3. Lite Image (Reduced disk footprint for core PDF/HTML pipelines)
docker run \
  -e GRADIO_SERVER_NAME=0.0.0.0 \
  -e GRADIO_SERVER_PORT=7860 \
  -v ./ktem_app_data:/app/ktem_app_data \
  -p 7860:7860 -it --rm \
  ghcr.io/cinnamon/kotaemon:main-lite

Note: For ARM architectures such as Apple Silicon, append --platform linux/arm64 to the execution flags. Access the UI at http://localhost:7860/.

Manual Bare-Metal Installation

bash
git clone https://github.com/Cinnamon/kotaemon
cd kotaemon

# Option A: Fast environment sync using uv
uv sync --python 3.10
source .venv/bin/activate

# Option B: Isolated Conda environment
conda create -n kotaemon python=3.10
conda activate kotaemon
pip install -e "libs/kotaemon[all]"
pip install -e "libs/ktem"

To enable the PDF.js visual citation viewer, extract the PDF_JS_DIST distribution package into libs/ktem/ktem/assets/prebuilt. Initialize your system configuration by copying .env.example to .env to seed the database on your initial boot:

bash
python app.py

Default root access authenticates via admin / admin.

Storage & Component Configuration

You can configure Kotaemon directly in the settings UI or define your infrastructure topology via flowsettings.py and .env.

Supported Data Backends

Engine Category	Config Variable	Supported Subsystems / Drivers
Document Store	`KH_DOCSTORE`	`Elasticsearch`, `LanceDB`, `SimpleFileDocumentStore`
Vector Store	`KH_VECTORSTORE`	`ChromaDB`, `LanceDB`, `InMemory`, `Milvus`, `Qdrant`
Reasoning Engine	`KH_REASONINGS`	`FullQAPipeline`, `ReactAgentPipeline`, `RewooAgentPipeline`
Multi-Modal Flag	`KH_REASONINGS_USE_MULTIMODAL`	`True`, `False`

All stateful runtime information resides within ./ktem_app_data, making backups and fleet migrations straightforward.

LLM and Embedding Providers

Configure target inference endpoints within .env or set them up directly under the Resources panel:

dotenv
# Standard OpenAI Endpoints
OPENAI_API_BASE=https://api.openai.com/v1
OPENAI_API_KEY=sk-your-key-here
OPENAI_CHAT_MODEL=gpt-3.5-turbo
OPENAI_EMBEDDINGS_MODEL=text-embedding-ada-002

# Enterprise Azure OpenAI Deployments
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY=your-azure-key
OPENAI_API_VERSION=2024-02-15-preview
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-35-turbo
AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT=text-embedding-ada-002

GraphRAG and Local Inference Integration

                 +-------------------------------+
                 |       Ingestion Source        |
                 +---------------+---------------+
                                 |
        +------------------------+------------------------+
        v                                                 v
+----------------------------------+   +----------------------------------+
|      Graph Indexing Pipeline     |   |    Local Inference Engine        |
|  - NanoGraphRAG                  |   |  - Ollama (llama3.1:8b)          |
|  - LightRAG                      |   |  - GGUF via llama-cpp-python     |
|  - MS GraphRAG (graphrag<=0.3.6) |   |  - Memory Sizing: Target < VRAM  |
+----------------------------------+   +----------------------------------+

GraphRAG Extensions

To run lightweight knowledge graph indexing without external network calls, configure one of the built-in graph extensions:

bash
# Setup NanoGraphRAG
pip install nano-graphrag
pip uninstall hnswlib chroma-hnswlib && pip install chroma-hnswlib
# Launch with USE_NANO_GRAPHRAG=true

# Setup LightRAG
pip install git+https://github.com/HKUDS/LightRAG.git
pip uninstall hnswlib chroma-hnswlib && pip install chroma-hnswlib
# Launch with USE_LIGHTRAG=true

# Setup Official Microsoft GraphRAG (pinned to stable release)
pip install "graphrag<=0.3.6" future
# Set GRAPHRAG_API_KEY and USE_CUSTOMIZED_GRAPHRAG_SETTING=true

Local Inference via Ollama and GGUF

Keep sensitive enterprise data entirely on-premise using local model providers.

Ollama Engine:
```
bash
ollama pull llama3.1:8b
ollama pull nomic-embed-text
```
Register the models in the Web UI under Resources and mark them as active defaults.
Native GGUF Execution (llama-cpp-python): When running local quantized models directly, budget your system memory to leave at least 2GB of headroom above the model size. On a 16GB machine with 12GB free RAM, cap your model choice at roughly 10GB. A lightweight model like Qwen1.5-1.8B-Chat-GGUF (~2GB) provides a fast, reliable baseline for testing local reasoning loops.

Extending the Core: Custom Reasoning Pipelines

Kotaemon's modular layout lets you swap out retrieval-generation loops without rewriting the UI or session layer.

To create a custom pipeline, build a new logic module in libs/ktem/ktem/reasoning/ using the base pipeline interfaces. Then, expose it to the Gradio interface by registering the class path in KH_REASONINGS inside flowsettings.py:

python
# Conceptual registration inside flowsettings.py
KH_REASONINGS = [
    "ktem.reasoning.simple.FullQAPipeline",
    "ktem.reasoning.react.ReactAgentPipeline",
    "ktem.reasoning.rewoo.RewooAgentPipeline",
    "ktem.reasoning.custom.CustomDomainPipeline", # Your custom pipeline
]

For indexing tasks, custom graph indexing pipelines can be dropped directly into libs/ktem/ktem/index/file/graph.

Engineering Verdict

Kotaemon steps past simple chat wrappers by treating retrieval as an extensible systems problem. Its combination of hybrid retrieval, multi-modal ingestion, pluggable graph algorithms, and granular citation tracking offers a solid foundation for both fast prototyping and production deployments.

Source Code: https://github.com/Cinnamon/kotaemon