ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter3/structured-index/IMPLEMENTATION_GUIDE.md

@@ -0,0 +1,313 @@
+# Structured Index Implementation Guide
+
+## Overview
+
+This project implements two advanced document indexing approaches for handling large technical documentation:
+
+1. **RAPTOR** (Recursive Abstractive Processing for Tree-Organized Retrieval)
+2. **GraphRAG** (Graph-based Retrieval Augmented Generation)
+
+Both approaches are designed to handle complex technical documentation like the Intel® 64 and IA-32 Architectures Software Developer's Manual (5000+ pages).
+
+## Architecture
+
+### RAPTOR Tree-Based Index
+
+RAPTOR creates a hierarchical tree structure through recursive summarization:
+
+```
+Document
+    ↓
+[Chunks] → [Embeddings] → [Clusters]
+    ↓           ↓              ↓
+Level 0: Leaf nodes (original chunks with summaries)
+    ↓
+Level 1: Parent nodes (cluster summaries)
+    ↓
+Level 2: Higher-level summaries
+    ↓
+Root: Top-level abstraction
+```
+
+**Key Features:**
+- Multi-level abstraction hierarchy
+- Gaussian Mixture Model clustering
+- Recursive summarization at each level
+- Cross-level search capability
+
+### GraphRAG Knowledge Graph
+
+GraphRAG builds a knowledge graph with entities and relationships:
+
+```
+Document
+    ↓
+[Chunks] → [Entity Extraction] → [Relationship Discovery]
+    ↓              ↓                        ↓
+Entities ←→ Relationships → Knowledge Graph
+    ↓
+[Community Detection]
+    ↓
+Community Summaries
+    ↓
+Hierarchical Communities
+```
+
+**Key Features:**
+- LLM-based entity and relationship extraction
+- Community detection (Leiden/Louvain algorithms)
+- Hierarchical community summarization
+- Graph-based search across entities and communities
+
+## Implementation Details
+
+### Core Components
+
+1. **Document Processor** (`document_processor.py`)
+   - Handles multiple file formats (PDF, TXT, MD, HTML)
+   - Optimized for technical documentation
+   - Special handling for Intel manual format
+   - Table extraction and formatting
+
+2. **RAPTOR Indexer** (`raptor_indexer.py`)
+   - Text chunking with configurable overlap
+   - Embedding generation using sentence transformers
+   - GMM clustering for node grouping
+   - Recursive summarization using LLMs
+   - Tree persistence and loading
+
+3. **GraphRAG Indexer** (`graphrag_indexer.py`)
+   - Entity extraction focused on technical concepts
+   - Relationship discovery between entities
+   - NetworkX graph construction
+   - Community detection and summarization
+   - Graph persistence and querying
+
+4. **API Service** (`api_service.py`)
+   - RESTful API using FastAPI
+   - Asynchronous processing for large documents
+   - Support for file uploads
+   - Unified interface for both indexing approaches
+   - Real-time status and statistics
+
+### Processing Pipeline
+
+#### Building Indexes
+
+1. **Document Processing**
+   ```python
+   processor = DocumentProcessor()
+   text = await processor.process_file(Path("intel_manual.pdf"))
+   ```
+
+2. **RAPTOR Indexing**
+   ```python
+   raptor = RaptorIndexer(config)
+   raptor.build_index(text)  # Creates tree structure
+   raptor.save_index()       # Persists to disk
+   ```
+
+3. **GraphRAG Indexing**
+   ```python
+   graphrag = GraphRAGIndexer(config)
+   graphrag.build_knowledge_graph(text)  # Extract entities
+   graphrag.detect_communities()          # Find communities
+   graphrag.hierarchical_summarization()  # Create hierarchies
+   graphrag.save_index()                  # Persist graph
+   ```
+
+#### Querying
+
+1. **RAPTOR Search**
+   - Creates query embedding
+   - Searches across all tree levels
+   - Returns nodes with different abstraction levels
+   - Includes level-specific summaries
+
+2. **GraphRAG Search**
+   - Supports entity, community, or hybrid search
+   - Returns entities with relationships
+   - Includes community summaries
+   - Provides graph context
+
+### API Endpoints
+
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/build` | POST | Build index from text/file |
+| `/upload` | POST | Upload and index document |
+| `/query` | POST | Query indexes |
+| `/status` | GET | Check index status |
+| `/statistics` | GET | Get index statistics |
+| `/indexes` | DELETE | Clear indexes |
+
+### Integration with Agentic RAG
+
+The structured indexes integrate seamlessly with the Agentic RAG system:
+
+1. **Configuration** (`agentic-rag/config.py`)
+   ```python
+   KnowledgeBaseType.RAPTOR   # Tree-based backend
+   KnowledgeBaseType.GRAPHRAG # Graph-based backend
+   ```
+
+2. **Tool Integration** (`agentic-rag/tools.py`)
+   - `_search_raptor()`: Queries RAPTOR API
+   - `_search_graphrag()`: Queries GraphRAG API
+   - Unified search interface for agents
+
+3. **Agent Usage**
+   ```python
+   config.knowledge_base.type = KnowledgeBaseType.RAPTOR
+   agent = AgenticRAG(config)
+   response = agent.query("What are x86 registers?")
+   ```
+
+## Usage Examples
+
+### Command Line Interface
+
+```bash
+# Build both indexes
+python main.py build intel_manual.pdf --type both
+
+# Query RAPTOR
+python main.py query "MOV instruction syntax" --type raptor
+
+# Query GraphRAG
+python main.py query "CPU register relationships" --type graphrag
+
+# Start API server
+python main.py serve
+```
+
+### Python API
+
+```python
+from config import get_raptor_config, get_graphrag_config
+from raptor_indexer import RaptorIndexer
+from graphrag_indexer import GraphRAGIndexer
+
+# RAPTOR Example
+raptor_config = get_raptor_config()
+raptor = RaptorIndexer(raptor_config)
+raptor.build_index(document_text)
+results = raptor.search("SSE instructions", top_k=5)
+
+# GraphRAG Example
+graphrag_config = get_graphrag_config()
+graphrag = GraphRAGIndexer(graphrag_config)
+graphrag.build_knowledge_graph(document_text)
+results = graphrag.search("instruction relationships", top_k=5)
+```
+
+### HTTP API
+
+```bash
+# Build index
+curl -X POST http://localhost:4242/build \
+  -H "Content-Type: application/json" \
+  -d '{"file_path": "intel_manual.pdf", "index_type": "both"}'
+
+# Query
+curl -X POST http://localhost:4242/query \
+  -H "Content-Type: application/json" \
+  -d '{"query": "vector instructions", "index_type": "hybrid"}'
+```
+
+## Performance Considerations
+
+### RAPTOR
+- **Indexing Time**: O(n log n) for clustering
+- **Memory**: Stores embeddings for all nodes
+- **Query Time**: Fast similarity search
+- **Best For**: Hierarchical information, long documents
+
+### GraphRAG
+- **Indexing Time**: O(n²) for relationship extraction
+- **Memory**: Graph structure can be large
+- **Query Time**: Graph traversal overhead
+- **Best For**: Complex relationships, entity-centric queries
+
+## Configuration Options
+
+### RAPTOR Settings
+```python
+chunk_size: 1000          # Words per chunk
+chunk_overlap: 200        # Overlap between chunks
+tree_depth: 3            # Maximum tree levels
+summarization_length: 200 # Summary word count
+```
+
+### GraphRAG Settings
+```python
+chunk_size: 1200              # Words per chunk
+max_knowledge_triples: 10     # Triples per chunk
+community_detection: "leiden" # Algorithm choice
+summarization_model: "gpt-5.6-luna"
+```
+
+## Extending the System
+
+### Adding New Document Types
+1. Extend `DocumentProcessor` with new format handlers
+2. Add format-specific extraction logic
+3. Update supported_formats dictionary
+
+### Custom Entity Extraction
+1. Modify prompt in `extract_entities_relationships()`
+2. Add domain-specific entity types
+3. Customize relationship types
+
+### Alternative Clustering
+1. Replace GMM in RAPTOR with other algorithms
+2. Implement custom similarity metrics
+3. Add dimensionality reduction options
+
+### Graph Algorithms
+1. Add new community detection algorithms
+2. Implement graph embedding techniques
+3. Add path-finding for relationship queries
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Out of Memory**
+   - Reduce chunk_size
+   - Process documents in sections
+   - Use smaller embedding models
+
+2. **Slow Indexing**
+   - Use faster/smaller LLMs
+   - Reduce tree_depth or max_triples
+   - Enable caching
+
+3. **Poor Search Results**
+   - Adjust chunk_size and overlap
+   - Fine-tune clustering parameters
+   - Improve entity extraction prompts
+
+4. **API Errors**
+   - Check API keys in .env
+   - Monitor rate limits
+   - Verify index exists before querying
+
+## Future Enhancements
+
+1. **Hybrid Indexing**: Combine RAPTOR and GraphRAG
+2. **Incremental Updates**: Add documents without rebuilding
+3. **Multi-modal Support**: Handle images and tables
+4. **Cross-lingual**: Support multiple languages
+5. **Active Learning**: Improve extraction with feedback
+6. **Distributed Processing**: Scale to larger documents
+7. **Query Optimization**: Cache frequent queries
+8. **Visualization**: Interactive graph/tree exploration
+
+## References
+
+- [RAPTOR Paper](https://arxiv.org/abs/2401.18059)
+- [GraphRAG by Microsoft](https://github.com/microsoft/graphrag)
+- [Intel SDM](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html)
+- [FastAPI Documentation](https://fastapi.tiangolo.com/)
+- [NetworkX Documentation](https://networkx.org/)