ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,419 @@
|
||||
# Vector Similarity Search Service (Dense Embedding) / 稠密向量相似性搜索服务
|
||||
|
||||
> Companion material for *AI Agents in Depth*, Chapter 3 — **Experiment 3-4**: BGE-M3 dense search with ANNOY / HNSW, plus offline CLI metrics.
|
||||
> 配套《深入理解 AI Agent》第 3 章 **实验 3-4**：BGE-M3 稠密检索与 ANNOY / HNSW 对比，含可离线 CLI。
|
||||
|
||||
← [Chapter 3 index / 返回第 3 章目录](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
### Overview
|
||||
|
||||
Educational HTTP service for vector similarity search using BGE-M3 embeddings with configurable ANNOY or HNSW backends, plus an offline `cli.py` for Experiment 3-4 metrics.
|
||||
|
||||
### CLI: dense retrieval & ANN comparison (`cli.py`, Experiment 3-4)
|
||||
|
||||
Besides the HTTP service, `cli.py` is **ready-to-run and offline-reproducible**—no need to start the server first:
|
||||
|
||||
1. **Semantic power of dense embeddings** — `recall@k / precision@k / MRR` on a small labelled corpus
|
||||
2. **ANN backend comparison** (focus of Exp. 3-4) — ANNOY / HNSW from `indexing.py` vs **exact brute-force**, measuring recall, build time, query latency
|
||||
|
||||
#### Usage
|
||||
|
||||
```bash
|
||||
# 1) Single dense query (default "a cat playing"; needs embedding model)
|
||||
python cli.py -q "model distillation" -k 3
|
||||
|
||||
# 2) Retrieval quality: recall@k / precision@k / MRR
|
||||
python cli.py --eval
|
||||
|
||||
# 2') Offline: small cached model (no 2.3GB BGE-M3 download)
|
||||
python cli.py --embedding-model sentence-transformers/all-MiniLM-L6-v2 --eval
|
||||
|
||||
# 3) ANN backend compare (synthetic vectors; fully offline, no model)
|
||||
python cli.py --compare-ann -k 10
|
||||
python cli.py --compare-ann --backend hnsw --hnsw-ef-search 200 -k 10
|
||||
|
||||
# Custom corpus / labels / output
|
||||
python cli.py --corpus my.json --labels my_labels.json --eval -o result.json
|
||||
```
|
||||
|
||||
`python cli.py --help` has full Chinese flag docs.
|
||||
|
||||
#### Common flags
|
||||
|
||||
| Flag | Description |
|
||||
| --- | --- |
|
||||
| `-q, --query` | Query (default `a cat playing`) |
|
||||
| `-c, --corpus` | Corpus (`.json` array or `.jsonl`); default built-in sample |
|
||||
| `-k, --top-k` | Top-k (default 5) |
|
||||
| `-o, --output` | Write results/metrics JSON |
|
||||
| `--embedding-model` | Model (default `BAAI/bge-m3`; offline: `sentence-transformers/all-MiniLM-L6-v2`) |
|
||||
| `--pooling` | `auto` / `mean` / `cls` |
|
||||
| `--eval` | Evaluate recall@k / precision@k / MRR |
|
||||
| `--compare-ann` | Compare ANNOY / HNSW (synthetic vectors) |
|
||||
| `--ann-base / --ann-dim / --ann-queries` | Synthetic base size / dim / queries (default 3000 / 128 / 100) |
|
||||
| `--annoy-n-trees / --hnsw-M / --hnsw-ef-search` | ANN hyperparameters |
|
||||
|
||||
#### Measured results (real runs)
|
||||
|
||||
**Dense quality** (12-doc built-in, `all-MiniLM-L6-v2`, offline):
|
||||
|
||||
```
|
||||
宏平均 recall@5=1.000 precision@5=0.320 MRR=1.000
|
||||
```
|
||||
|
||||
Query `a cat playing` ranks docs that only say `kitten` / `feline` (no literal “cat”) at ranks 1–2—semantic strength vs BM25 (Exp. 3-5 may miss them).
|
||||
|
||||
**ANN compare** (3000 × 128-d unit vectors, 100 queries, top-10); HNSW recall rises with `ef_search`:
|
||||
|
||||
| Config | recall@10 | Mean query latency |
|
||||
| --- | --- | --- |
|
||||
| HNSW `ef_search=20` | 0.562 | 0.05 ms |
|
||||
| HNSW `ef_search=200` | 0.991 | 0.25 ms |
|
||||
|
||||
> **Environment note**: each backend is health-checked by self-querying. On some macOS/arm64 setups, prebuilt `annoy==1.17.3` is broken (even self-query only returns itself); the tool warns and marks those numbers untrusted. HNSW is unaffected. Full ANNOY vs HNSW: use an environment where Annoy works (e.g. Linux x86_64).
|
||||
|
||||
### Service features
|
||||
|
||||
- **BGE-M3**: dense embeddings, 100+ languages, long context (up to 8192 tokens)
|
||||
- **Dual backends**: ANNOY (tree), HNSW (graph)
|
||||
- **Educational logging**: embed, index ops, metrics, vector stats
|
||||
- **REST API**: index / delete / search / stats
|
||||
- **In-memory** (no persistence)
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌──────────────────┐
|
||||
│ HTTP Client │
|
||||
└────────┬─────────┘
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ FastAPI Server │
|
||||
└────────┬─────────┘
|
||||
┌────┴────┐
|
||||
▼ ▼
|
||||
┌──────────┐ ┌──────────────┐
|
||||
│ Document │ │ Embedding │
|
||||
│ Store │ │ Service │
|
||||
└──────────┘ │ (BGE-M3) │
|
||||
└──────┬───────┘
|
||||
┌─────────┴──────────┐
|
||||
▼ ▼
|
||||
┌──────────┐ ┌──────────┐
|
||||
│ ANNOY │ │ HNSW │
|
||||
└──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
### Installation
|
||||
|
||||
- Python 3.12 with the root `ch3` extra, macOS (M1/M2 optimized) or Linux
|
||||
- ≥4GB RAM (8GB recommended); optional CUDA GPU
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 3 environment
|
||||
uv sync --locked --python 3.12 --extra ch3
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch3]"
|
||||
|
||||
cd chapter3/dense-embedding
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
BGE-M3 (~2.3GB) downloads on first use into the HuggingFace cache.
|
||||
|
||||
### Starting the service
|
||||
|
||||
```bash
|
||||
python main.py # HNSW (default)
|
||||
python main.py --index-type annoy
|
||||
python main.py --index-type hnsw --host 0.0.0.0 --port 4242 --debug --show-embeddings
|
||||
```
|
||||
|
||||
Options: `--index-type` (`annoy`|`hnsw`, default `hnsw`), `--host` (default `0.0.0.0`), `--port` (default `4240`), `--debug`, `--show-embeddings`.
|
||||
|
||||
Docs: http://localhost:4240/docs · OpenAPI: http://localhost:4240/openapi.json
|
||||
|
||||
### API endpoints
|
||||
|
||||
**POST `/index`**
|
||||
|
||||
```json
|
||||
{
|
||||
"text": "Machine learning is a subset of artificial intelligence.",
|
||||
"doc_id": "doc_001",
|
||||
"metadata": {"category": "AI", "author": "John Doe"}
|
||||
}
|
||||
```
|
||||
|
||||
**POST `/search`**
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "What is deep learning?",
|
||||
"top_k": 5,
|
||||
"return_documents": true
|
||||
}
|
||||
```
|
||||
|
||||
**DELETE `/index`** — body `{"doc_id": "doc_001"}`
|
||||
**GET `/stats`** · **GET `/documents?limit=10`**
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
python test_client.py
|
||||
python test_client.py --performance
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4240/index \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "This is a test document about machine learning."}'
|
||||
|
||||
curl -X POST http://localhost:4240/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "artificial intelligence", "top_k": 5}'
|
||||
```
|
||||
|
||||
### Index comparison
|
||||
|
||||
**ANNOY**: fast build, low memory, good for static/read-heavy; rebuild for delete; trade accuracy via `n_trees`.
|
||||
**HNSW**: high recall, incremental updates, soft delete; more memory, slower build; tune `M` / `ef_*`.
|
||||
|
||||
### Configuration (env `VEC_` prefix)
|
||||
|
||||
```bash
|
||||
export VEC_INDEX_TYPE=hnsw
|
||||
export VEC_MODEL_NAME=BAAI/bge-m3
|
||||
export VEC_USE_FP16=true
|
||||
export VEC_MAX_SEQ_LENGTH=512
|
||||
export VEC_MAX_DOCUMENTS=100000
|
||||
export VEC_LOG_LEVEL=DEBUG
|
||||
export VEC_ANNOY_N_TREES=50
|
||||
export VEC_ANNOY_METRIC=angular
|
||||
export VEC_HNSW_EF_CONSTRUCTION=200
|
||||
export VEC_HNSW_M=32
|
||||
export VEC_HNSW_EF_SEARCH=100
|
||||
export VEC_HNSW_SPACE=cosine
|
||||
```
|
||||
|
||||
Educational logging: `python main.py --debug --show-embeddings`.
|
||||
|
||||
### Memory / optimization notes
|
||||
|
||||
- Model ~2.3GB; ~4KB per doc (1024-d float32)
|
||||
- ANNOY: raise `n_trees` for accuracy; `angular` for normalized vectors; batch then build
|
||||
- HNSW: raise `M` / `ef_construction` / `ef_search` for quality vs cost
|
||||
- FP16 faster with slight accuracy trade-off
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
OOM → smaller batches, FP16, lower `max_seq_length`, prefer ANNOY. Slow index → lower `ef_construction` / `n_trees`, use GPU. Poor quality → raise `n_trees` / `M` / `ef_search`.
|
||||
|
||||
### References
|
||||
|
||||
- [BGE-M3 Paper](https://arxiv.org/abs/2402.03216) · [Model](https://huggingface.co/BAAI/bge-m3)
|
||||
- [ANNOY](https://github.com/spotify/annoy) · [HNSWlib](https://github.com/nmslib/hnswlib) · [FastAPI](https://fastapi.tiangolo.com/)
|
||||
|
||||
### License
|
||||
|
||||
Educational project for learning purposes.
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 概述
|
||||
|
||||
基于 BGE-M3 嵌入、可切换 ANNOY / HNSW 后端的教学型向量相似性搜索 HTTP 服务，外加实验 3-4 的离线 CLI 评测。
|
||||
|
||||
### 命令行工具：稠密检索与 ANN 对比（cli.py，实验 3-4）
|
||||
|
||||
除 HTTP 服务外，`cli.py` **开箱即用、可离线复现**，无需先启动服务：
|
||||
|
||||
1. **稠密嵌入的语义能力**——小标注语料上算 `recall@k / precision@k / MRR`
|
||||
2. **ANN 后端对比**（实验 3-4 重点）——复用 `indexing.py` 的 ANNOY / HNSW，相对**精确暴力检索**测召回、建索引耗时与查询延迟
|
||||
|
||||
#### 用法
|
||||
|
||||
```bash
|
||||
# 1) 单条稠密查询（默认 "a cat playing"，需要嵌入模型）
|
||||
python cli.py -q "model distillation" -k 3
|
||||
|
||||
# 2) 检索质量评测
|
||||
python cli.py --eval
|
||||
|
||||
# 2') 离线复现：小模型（无需下载 2.3GB BGE-M3）
|
||||
python cli.py --embedding-model sentence-transformers/all-MiniLM-L6-v2 --eval
|
||||
|
||||
# 3) ANN 后端对比（合成向量，完全离线）
|
||||
python cli.py --compare-ann -k 10
|
||||
python cli.py --compare-ann --backend hnsw --hnsw-ef-search 200 -k 10
|
||||
|
||||
# 自定义语料 / 标注 / 输出
|
||||
python cli.py --corpus my.json --labels my_labels.json --eval -o result.json
|
||||
```
|
||||
|
||||
`python cli.py --help` 提供完整中文参数说明。
|
||||
|
||||
#### 常用参数
|
||||
|
||||
| 参数 | 说明 |
|
||||
| --- | --- |
|
||||
| `-q, --query` | 查询（默认 `a cat playing`） |
|
||||
| `-c, --corpus` | 语料（`.json` / `.jsonl`）；缺省内置示例 |
|
||||
| `-k, --top-k` | Top-k（默认 5） |
|
||||
| `-o, --output` | 结果 / 指标 JSON |
|
||||
| `--embedding-model` | 嵌入模型（默认 `BAAI/bge-m3`；离线可用 MiniLM） |
|
||||
| `--pooling` | `auto` / `mean` / `cls` |
|
||||
| `--eval` | 评测 recall@k / precision@k / MRR |
|
||||
| `--compare-ann` | 对比 ANNOY / HNSW |
|
||||
| `--ann-base / --ann-dim / --ann-queries` | 合成底库规模 / 维度 / 查询数 |
|
||||
| `--annoy-n-trees / --hnsw-M / --hnsw-ef-search` | ANN 超参 |
|
||||
|
||||
#### 实测结果
|
||||
|
||||
**稠密检索质量**（12 篇语料，`all-MiniLM-L6-v2`，离线）：
|
||||
|
||||
```
|
||||
宏平均 recall@5=1.000 precision@5=0.320 MRR=1.000
|
||||
```
|
||||
|
||||
查询 `a cat playing` 仍把仅含 `kitten` / `feline` 的文档排到第 1、2 名——相对 BM25（实验 3-5 会漏召回）的语义优势。
|
||||
|
||||
**ANN 对比**（3000 条 128 维，100 查询，top-10）：
|
||||
|
||||
| 配置 | recall@10 | 平均查询延迟 |
|
||||
| --- | --- | --- |
|
||||
| HNSW `ef_search=20` | 0.562 | 0.05 ms |
|
||||
| HNSW `ef_search=200` | 0.991 | 0.25 ms |
|
||||
|
||||
> **环境提示**：部分 macOS/arm64 上 `annoy==1.17.3` 预编译轮子有缺陷，工具会警告并标记不可信；HNSW 不受影响。完整 ANNOY vs HNSW 请在 annoy 正常的环境（如 Linux x86_64）运行。
|
||||
|
||||
### 服务功能
|
||||
|
||||
- **BGE-M3**：稠密嵌入、多语言、长上下文（至 8192 tokens）
|
||||
- **双后端**：ANNOY（树）、HNSW（图）
|
||||
- **教学日志**、**REST API**、**纯内存**
|
||||
|
||||
### 架构
|
||||
|
||||
（与 English 节相同示意图。）
|
||||
|
||||
### 安装
|
||||
|
||||
- Python 3.12 与根目录 `ch3` extra，macOS（M1/M2）或 Linux
|
||||
- 内存 ≥4GB（建议 8GB）；可选 CUDA
|
||||
|
||||
```bash
|
||||
# 在仓库根目录使用统一的第 3 章环境
|
||||
uv sync --locked --python 3.12 --extra ch3
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch3]"
|
||||
|
||||
cd chapter3/dense-embedding
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
BGE-M3（约 2.3GB）首次运行自动下载。
|
||||
|
||||
### 启动服务
|
||||
|
||||
```bash
|
||||
python main.py # 默认 HNSW
|
||||
python main.py --index-type annoy
|
||||
python main.py --index-type hnsw --host 0.0.0.0 --port 4242 --debug --show-embeddings
|
||||
```
|
||||
|
||||
选项：`--index-type`、`--host`（默认 `0.0.0.0`）、`--port`（默认 `4240`）、`--debug`、`--show-embeddings`。
|
||||
|
||||
文档：http://localhost:4240/docs
|
||||
|
||||
### API 端点
|
||||
|
||||
**POST `/index`** / **POST `/search`** / **DELETE `/index`** / **GET `/stats`** / **GET `/documents?limit=10`**
|
||||
请求体格式与 English 节 JSON 示例相同。
|
||||
|
||||
### 测试
|
||||
|
||||
```bash
|
||||
python test_client.py
|
||||
python test_client.py --performance
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4240/index \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "This is a test document about machine learning."}'
|
||||
|
||||
curl -X POST http://localhost:4240/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "artificial intelligence", "top_k": 5}'
|
||||
```
|
||||
|
||||
### 索引对比
|
||||
|
||||
**ANNOY**：建索引快、内存低，适合静态/读多写少；删除需重建；用 `n_trees` 换精度。
|
||||
**HNSW**：召回高、可增量与软删除；内存更高、建索引更慢；调 `M` / `ef_*`。
|
||||
|
||||
### 配置（`VEC_` 环境变量）
|
||||
|
||||
```bash
|
||||
export VEC_INDEX_TYPE=hnsw
|
||||
export VEC_MODEL_NAME=BAAI/bge-m3
|
||||
export VEC_USE_FP16=true
|
||||
export VEC_MAX_SEQ_LENGTH=512
|
||||
export VEC_MAX_DOCUMENTS=100000
|
||||
export VEC_LOG_LEVEL=DEBUG
|
||||
export VEC_ANNOY_N_TREES=50
|
||||
export VEC_ANNOY_METRIC=angular
|
||||
export VEC_HNSW_EF_CONSTRUCTION=200
|
||||
export VEC_HNSW_M=32
|
||||
export VEC_HNSW_EF_SEARCH=100
|
||||
export VEC_HNSW_SPACE=cosine
|
||||
```
|
||||
|
||||
教学日志：`python main.py --debug --show-embeddings`。
|
||||
|
||||
### 内存与优化
|
||||
|
||||
模型约 2.3GB；每文档约 4KB（1024 维 float32）。ANNOY 提高 `n_trees`；HNSW 提高 `M` / `ef_*`；可用 FP16 加速。
|
||||
|
||||
### 故障排查
|
||||
|
||||
内存不足 → 减小 batch、FP16、降低 `max_seq_length`、改用 ANNOY。建索引慢 → 降低 `ef_construction` / `n_trees`、用 GPU。检索差 → 提高 `n_trees` / `M` / `ef_search`。
|
||||
|
||||
### 参考与许可
|
||||
|
||||
- [BGE-M3 论文](https://arxiv.org/abs/2402.03216) · [模型](https://huggingface.co/BAAI/bge-m3)
|
||||
- [ANNOY](https://github.com/spotify/annoy) · [HNSWlib](https://github.com/nmslib/hnswlib)
|
||||
- 教学项目，仅供学习。
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Related: [`../sparse-embedding/`](../sparse-embedding/) (Exp. 3-5), [`../retrieval-pipeline/`](../retrieval-pipeline/) (Exp. 3-6).
|
||||
- 相关：[`../sparse-embedding/`](../sparse-embedding/)（实验 3-5）、[`../retrieval-pipeline/`](../retrieval-pipeline/)（实验 3-6）。
|
||||
@@ -0,0 +1,303 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real-embedding ANNOY vs HNSW benchmark for Experiment 3-4."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Sequence
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
from experiment_utils import sha256_file, write_campaign_evidence
|
||||
|
||||
from indexing import AnnoyIndex, HNSWIndex
|
||||
|
||||
|
||||
TOPICS = [
|
||||
("vector search", "Approximate nearest-neighbor indexes accelerate semantic vector retrieval."),
|
||||
("database transactions", "Database transactions use atomicity, consistency, isolation and durability."),
|
||||
("photosynthesis", "Green plants turn sunlight and carbon dioxide into chemical energy."),
|
||||
("quantum entanglement", "Entangled particles exhibit correlated quantum measurement outcomes."),
|
||||
("contract law", "A valid contract generally requires offer acceptance and consideration."),
|
||||
("neural networks", "Deep neural networks learn layered nonlinear representations from data."),
|
||||
("cybersecurity", "Zero trust security continuously verifies identity and device posture."),
|
||||
("volcanoes", "Volcanoes form when magma rises through fractures in the planetary crust."),
|
||||
("water cycle", "Evaporation condensation precipitation and runoff form the water cycle."),
|
||||
("operating systems", "An operating system schedules processes and manages memory and devices."),
|
||||
("HTTP errors", "HTTP status 403 means a server understood but refused a request."),
|
||||
("machine translation", "Multilingual models translate meaning between natural languages."),
|
||||
("financial risk", "Portfolio diversification reduces exposure to idiosyncratic financial risk."),
|
||||
("medical imaging", "Radiology systems analyze X-rays CT scans and magnetic resonance images."),
|
||||
("supply chains", "Supply chain planning coordinates inventory logistics demand and suppliers."),
|
||||
("climate science", "Climate models simulate long-term interactions among atmosphere ocean and land."),
|
||||
("CPU instructions", "SIMD instructions apply one operation to several packed numeric values."),
|
||||
("compiler design", "A compiler parses source code optimizes intermediate form and emits machine code."),
|
||||
("graph theory", "Graph algorithms traverse vertices and edges to discover paths and communities."),
|
||||
("astronomy", "Astronomers infer stellar properties from spectra luminosity and orbital motion."),
|
||||
]
|
||||
|
||||
|
||||
class TransformerEncoder:
|
||||
def __init__(self, model_name: str, device: str):
|
||||
import torch
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
self.torch = torch
|
||||
self.model_name = model_name
|
||||
self.device = device
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side="left")
|
||||
self.model = AutoModel.from_pretrained(model_name).to(device).eval()
|
||||
|
||||
def encode(self, texts: Sequence[str], query: bool = False, batch_size: int = 16) -> np.ndarray:
|
||||
prefix = "Instruct: Retrieve semantically relevant passages.\nQuery:" if query else ""
|
||||
values = [prefix + text for text in texts]
|
||||
vectors = []
|
||||
for start in range(0, len(values), batch_size):
|
||||
batch = values[start : start + batch_size]
|
||||
tokens = self.tokenizer(
|
||||
batch, padding=True, truncation=True, max_length=192, return_tensors="pt"
|
||||
).to(self.device)
|
||||
with self.torch.no_grad():
|
||||
output = self.model(**tokens).last_hidden_state[:, -1].float()
|
||||
output = self.torch.nn.functional.normalize(output, p=2, dim=1)
|
||||
vectors.append(output.cpu().numpy())
|
||||
return np.concatenate(vectors, axis=0).astype("float32")
|
||||
|
||||
|
||||
def build_corpus(n_docs: int) -> tuple[List[str], List[str]]:
|
||||
docs, ids = [], []
|
||||
variants = (
|
||||
"A concise technical overview.",
|
||||
"This passage explains the central mechanism and its practical use.",
|
||||
"An engineering handbook entry with definitions and examples.",
|
||||
"A research summary intended for a multilingual knowledge base.",
|
||||
"Operational notes emphasizing trade-offs, reliability, and performance.",
|
||||
)
|
||||
for i in range(n_docs):
|
||||
topic, sentence = TOPICS[i % len(TOPICS)]
|
||||
variant = variants[(i // len(TOPICS)) % len(variants)]
|
||||
docs.append(f"Topic: {topic}. {sentence} {variant} Document revision {i:04d}.")
|
||||
ids.append(f"doc_{i:04d}")
|
||||
return ids, docs
|
||||
|
||||
|
||||
def percentiles(values: List[float]) -> Dict[str, float]:
|
||||
return {
|
||||
"mean": statistics.mean(values),
|
||||
"p50": float(np.percentile(values, 50)),
|
||||
"p95": float(np.percentile(values, 95)),
|
||||
}
|
||||
|
||||
|
||||
def exact_neighbors(matrix: np.ndarray, queries: np.ndarray, k: int) -> List[List[int]]:
|
||||
scores = queries @ matrix.T
|
||||
return [np.argsort(-row)[:k].tolist() for row in scores]
|
||||
|
||||
|
||||
def measure_index(name: str, index: Any, ids: List[str], vectors: np.ndarray,
|
||||
query_vectors: np.ndarray, truth: List[List[int]], k: int,
|
||||
repeats: int) -> Dict[str, Any]:
|
||||
build_start = time.perf_counter()
|
||||
for doc_id, vector in zip(ids, vectors):
|
||||
index.add_item(doc_id, vector)
|
||||
index.rebuild_index()
|
||||
build_ms = (time.perf_counter() - build_start) * 1000
|
||||
|
||||
id_to_pos = {doc_id: pos for pos, doc_id in enumerate(ids)}
|
||||
recalls, latencies = [], []
|
||||
rankings = []
|
||||
for q_idx, query in enumerate(query_vectors):
|
||||
first = None
|
||||
for _ in range(repeats):
|
||||
started = time.perf_counter()
|
||||
found, distances = index.search(query, k)
|
||||
latencies.append((time.perf_counter() - started) * 1000)
|
||||
if first is None:
|
||||
first = found
|
||||
found_pos = {id_to_pos[x] for x in first if x in id_to_pos}
|
||||
recalls.append(len(found_pos & set(truth[q_idx])) / k)
|
||||
rankings.append({"query_index": q_idx, "doc_ids": first})
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=f".{name}") as handle:
|
||||
if name == "annoy":
|
||||
index.index.save(handle.name)
|
||||
else:
|
||||
index.index.save_index(handle.name)
|
||||
serialized_bytes = os.path.getsize(handle.name)
|
||||
return {
|
||||
"build_ms": round(build_ms, 3),
|
||||
"recall_at_k": statistics.mean(recalls),
|
||||
"query_latency_ms": percentiles(latencies),
|
||||
"serialized_bytes": serialized_bytes,
|
||||
"rankings": rankings,
|
||||
}
|
||||
|
||||
|
||||
def local_annoy_healthy() -> bool:
|
||||
probe = AnnoyIndex(3, n_trees=5)
|
||||
for i in range(5):
|
||||
probe.add_item(str(i), np.array([i + 1, i + 2, i + 3], dtype="float32"))
|
||||
probe.rebuild_index()
|
||||
found, _ = probe.search(np.array([2, 3, 4], dtype="float32"), 3)
|
||||
return len(found) == 3
|
||||
|
||||
|
||||
def measure_annoy_docker(ids: List[str], vectors: np.ndarray, queries: np.ndarray,
|
||||
initial_truth: List[List[int]], full_truth: List[List[int]],
|
||||
initial_n: int, k: int, repeats: int) -> tuple[Dict[str, Any], Dict[str, Any]]:
|
||||
"""Run the real Spotify ANNOY library in Linux when the macOS ARM extension
|
||||
returns only item zero (a reproducible host-wheel defect in this environment)."""
|
||||
with tempfile.TemporaryDirectory() as raw_dir:
|
||||
work = Path(raw_dir)
|
||||
np.savez(
|
||||
work / "input.npz",
|
||||
ids=np.asarray(ids, dtype="U32"), vectors=vectors, queries=queries,
|
||||
initial_truth=np.asarray(initial_truth, dtype="int64"),
|
||||
full_truth=np.asarray(full_truth, dtype="int64"),
|
||||
parameters=np.asarray([initial_n, k, repeats], dtype="int64"),
|
||||
)
|
||||
command = [
|
||||
"docker", "run", "--rm",
|
||||
"-v", f"{work}:/work",
|
||||
"-v", f"{HERE / 'docker_annoy_runner.py'}:/runner.py:ro",
|
||||
"python:3.11-slim",
|
||||
"sh", "-lc",
|
||||
"apt-get update -qq && apt-get install -y -qq g++ >/dev/null && "
|
||||
"pip install -q numpy annoy && python /runner.py /work/input.npz /work/output.json",
|
||||
]
|
||||
started = time.perf_counter()
|
||||
proc = subprocess.run(command, text=True, capture_output=True, timeout=600)
|
||||
wall_ms = (time.perf_counter() - started) * 1000
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"Docker ANNOY runner failed: {proc.stderr[-2000:]}")
|
||||
result = json.loads((work / "output.json").read_text(encoding="utf-8"))
|
||||
inspect = subprocess.check_output(
|
||||
["docker", "image", "inspect", "python:3.11-slim", "--format", "{{json .RepoDigests}}"],
|
||||
text=True,
|
||||
).strip()
|
||||
runtime = {
|
||||
"kind": "docker-linux-aarch64",
|
||||
"reason": "host macOS ARM ANNOY extension failed health check (returned fewer than k items)",
|
||||
"base_image": "python:3.11-slim",
|
||||
"base_image_repo_digests": json.loads(inspect),
|
||||
"container_setup_and_run_wall_ms": round(wall_ms, 3),
|
||||
}
|
||||
return result, runtime
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Experiment 3-4 ANNOY/HNSW real embedding benchmark")
|
||||
parser.add_argument("--model", default="Qwen/Qwen3-Embedding-0.6B")
|
||||
parser.add_argument("--device", default="cpu")
|
||||
parser.add_argument("--docs", type=int, default=300)
|
||||
parser.add_argument("--top-k", type=int, default=10)
|
||||
parser.add_argument("--repeats", type=int, default=5)
|
||||
parser.add_argument("--seed", type=int, default=37)
|
||||
args = parser.parse_args()
|
||||
np.random.seed(args.seed)
|
||||
|
||||
ids, docs = build_corpus(args.docs)
|
||||
queries = [f"Find technical information about {topic}." for topic, _ in TOPICS]
|
||||
encoder = TransformerEncoder(args.model, args.device)
|
||||
embed_start = time.perf_counter()
|
||||
vectors = encoder.encode(docs)
|
||||
query_vectors = encoder.encode(queries, query=True)
|
||||
embedding_ms = (time.perf_counter() - embed_start) * 1000
|
||||
truth = exact_neighbors(vectors, query_vectors, args.top_k)
|
||||
dim = vectors.shape[1]
|
||||
|
||||
initial_n = int(args.docs * 0.8)
|
||||
initial_truth = exact_neighbors(vectors[:initial_n], query_vectors, args.top_k)
|
||||
backends = {
|
||||
"hnsw": HNSWIndex(dim, max_elements=args.docs + 10, ef_construction=200, M=16, ef_search=100),
|
||||
}
|
||||
results = {}
|
||||
if local_annoy_healthy():
|
||||
backends["annoy"] = AnnoyIndex(dim, n_trees=50, metric="angular")
|
||||
annoy_runtime = {"kind": "host", "health_check": "passed"}
|
||||
else:
|
||||
results["annoy"], annoy_runtime = measure_annoy_docker(
|
||||
ids, vectors, query_vectors, initial_truth, truth,
|
||||
initial_n, args.top_k, args.repeats,
|
||||
)
|
||||
print(f"annoy (Docker): recall@{args.top_k}={results['annoy']['recall_at_k']:.3f}, "
|
||||
f"build={results['annoy']['build_ms']:.1f}ms")
|
||||
for name, backend in backends.items():
|
||||
initial = measure_index(
|
||||
name, backend, ids[:initial_n], vectors[:initial_n], query_vectors,
|
||||
initial_truth, args.top_k, args.repeats,
|
||||
)
|
||||
update_start = time.perf_counter()
|
||||
for doc_id, vector in zip(ids[initial_n:], vectors[initial_n:]):
|
||||
backend.add_item(doc_id, vector)
|
||||
requires_rebuild = name == "annoy"
|
||||
if requires_rebuild:
|
||||
backend.rebuild_index()
|
||||
update_ms = (time.perf_counter() - update_start) * 1000
|
||||
id_to_pos = {doc_id: pos for pos, doc_id in enumerate(ids)}
|
||||
update_recalls = []
|
||||
for q_idx, query in enumerate(query_vectors):
|
||||
found, _ = backend.search(query, args.top_k)
|
||||
update_recalls.append(len({id_to_pos[x] for x in found} & set(truth[q_idx])) / args.top_k)
|
||||
initial["incremental_update"] = {
|
||||
"items_added": args.docs - initial_n,
|
||||
"latency_ms": round(update_ms, 3),
|
||||
"requires_full_rebuild": requires_rebuild,
|
||||
"recall_at_k_after_update": statistics.mean(update_recalls),
|
||||
}
|
||||
results[name] = initial
|
||||
print(f"{name}: recall@{args.top_k}={initial['recall_at_k']:.3f}, build={initial['build_ms']:.1f}ms")
|
||||
|
||||
cache_ref = Path.home() / ".cache" / "huggingface" / "hub" / f"models--{args.model.replace('/', '--')}" / "refs" / "main"
|
||||
model_revision = cache_ref.read_text(encoding="utf-8").strip() if cache_ref.exists() else None
|
||||
full = args.docs >= 300 and all(results[name]["recall_at_k"] >= 0.8 for name in results)
|
||||
evidence = {
|
||||
"status": "passed" if full else "partial",
|
||||
"configuration": {
|
||||
"embedding_model": args.model,
|
||||
"model_revision": model_revision,
|
||||
"device": args.device,
|
||||
"seed": args.seed,
|
||||
"dimension": dim,
|
||||
"documents": args.docs,
|
||||
"queries": len(queries),
|
||||
"top_k": args.top_k,
|
||||
"annoy_runtime": annoy_runtime,
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": True,
|
||||
"same_vectors_and_queries": True,
|
||||
"exact_search_ground_truth": True,
|
||||
"recall_latency_build_size_measured": True,
|
||||
"incremental_behavior_measured": True,
|
||||
"both_backends_recall_at_least_0_8": all(results[name]["recall_at_k"] >= 0.8 for name in results),
|
||||
"passed": full,
|
||||
},
|
||||
"summary": {
|
||||
"embedding_latency_ms": round(embedding_ms, 3),
|
||||
"annoy": {k: v for k, v in results["annoy"].items() if k != "rankings"},
|
||||
"hnsw": {k: v for k, v in results["hnsw"].items() if k != "rankings"},
|
||||
},
|
||||
"corpus": {"doc_ids": ids, "texts": docs, "queries": queries},
|
||||
"results": results,
|
||||
}
|
||||
manifest = write_campaign_evidence(
|
||||
HERE, "3-4", evidence,
|
||||
input_paths=[HERE / "indexing.py", HERE / "benchmark.py", HERE / "docker_annoy_runner.py"]
|
||||
)
|
||||
print(json.dumps(manifest["summary"], ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,452 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
稠密检索命令行工具（实验 3-4）
|
||||
|
||||
在一个小型示例语料上运行稠密嵌入检索，支持：
|
||||
- 自定义语料 / 查询 / top-k / 输出文件
|
||||
- --eval：在带标注的小型评测集上计算 recall@k / precision@k / MRR，
|
||||
直观展示"稠密嵌入读得懂同义表达"这一核心卖点
|
||||
- --compare-ann：复现书中实验 3-4 的重点——对比 ANNOY 与 HNSW 两种 ANN 后端
|
||||
相对精确暴力检索的召回率、建索引耗时与查询延迟（复用服务端 indexing.py）
|
||||
- --embedding-model：可切换嵌入模型；默认 BAAI/bge-m3，离线可用已缓存的
|
||||
sentence-transformers/all-MiniLM-L6-v2
|
||||
|
||||
不带任何参数运行时，等价于书中实验 3-4 的默认演示（查询 "a cat playing"）。
|
||||
--compare-ann 使用合成向量、无需任何模型，可在完全离线环境下复现 ANN 对比。
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from typing import Dict, List, Optional, Set
|
||||
|
||||
import numpy as np
|
||||
|
||||
from indexing import AnnoyIndex, HNSWIndex
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 内置示例语料与标注（英文，与常见句向量模型能力一致，可完全离线复现）
|
||||
# 语料刻意加入了"同义表达"文档（kitten / feline 表示 cat，distillation 的两种写法），
|
||||
# 用来展示稠密检索在语义匹配上的强项——这些正是稀疏 BM25（实验 3-5）会漏召回的场景。
|
||||
# ---------------------------------------------------------------------------
|
||||
DEFAULT_CORPUS: List[Dict] = [
|
||||
{"doc_id": "doc_1", "title": "Python Language",
|
||||
"text": "Python is a high-level programming language known for readability and a simple syntax."},
|
||||
{"doc_id": "doc_2", "title": "JavaScript Runtime",
|
||||
"text": "JavaScript runs in the browser and on servers via Node.js for full-stack web development."},
|
||||
{"doc_id": "doc_3", "title": "Model Distillation",
|
||||
"text": "Model distillation compresses a large teacher model into a smaller student model while preserving accuracy."},
|
||||
{"doc_id": "doc_4", "title": "Knowledge Distillation",
|
||||
"text": "Knowledge distillation transfers knowledge from a big neural network to a compact model for efficient inference."},
|
||||
{"doc_id": "doc_5", "title": "BM25 Ranking",
|
||||
"text": "BM25 is a probabilistic ranking function using term frequency and inverse document frequency."},
|
||||
{"doc_id": "doc_6", "title": "HTTP Errors",
|
||||
"text": "The HTTP 404 error code means the requested resource was not found on the web server."},
|
||||
{"doc_id": "doc_7", "title": "A Playful Kitten",
|
||||
"text": "A cute kitten chased a ball of yarn across the living room floor all afternoon."},
|
||||
{"doc_id": "doc_8", "title": "Silent Hunter",
|
||||
"text": "The feline predator stalked its prey silently through the tall grass at dusk."},
|
||||
{"doc_id": "doc_9", "title": "Hardware Fault",
|
||||
"text": "Error code XK9-2B4-7Q1 indicates a hardware fault in the storage controller board."},
|
||||
{"doc_id": "doc_10", "title": "Transformers",
|
||||
"text": "Transformer models use self-attention to process input sequences in parallel efficiently."},
|
||||
{"doc_id": "doc_11", "title": "Deep Learning",
|
||||
"text": "Deep learning stacks many layers of neurons to extract hierarchical features from raw data."},
|
||||
{"doc_id": "doc_12", "title": "Gradient Descent",
|
||||
"text": "Gradient descent minimizes a loss function by iteratively updating the model parameters."},
|
||||
]
|
||||
|
||||
# query -> 相关文档 doc_id 集合（人工标注的 ground truth）
|
||||
# 这些查询大多不与相关文档共享字面关键词，只在语义上相关——考的正是稠密检索的语义能力。
|
||||
DEFAULT_LABELS: Dict[str, List[str]] = {
|
||||
# kitten / feline 都不含字面 "cat"，稠密检索应凭语义召回，稀疏 BM25 则会漏
|
||||
"a cat playing": ["doc_7", "doc_8"],
|
||||
# "蒸馏"的两种写法，语义同一主题
|
||||
"model distillation": ["doc_3", "doc_4"],
|
||||
# 语义相关，字面不含 "neural network training"
|
||||
"training neural networks": ["doc_11", "doc_12"],
|
||||
"self attention in sequence models": ["doc_10"],
|
||||
"web server resource not found": ["doc_6"],
|
||||
}
|
||||
|
||||
DEFAULT_QUERY = "a cat playing"
|
||||
|
||||
DEFAULT_MODEL = "BAAI/bge-m3"
|
||||
OFFLINE_HINT_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 稠密嵌入编码器：用 transformers 的 AutoModel 直接算句向量（mean / cls 池化 + L2 归一化）
|
||||
# 这样既能加载书中默认的 BAAI/bge-m3（bge 系用 cls 池化），也能加载离线已缓存的
|
||||
# sentence-transformers/all-MiniLM-L6-v2（mean 池化），无需依赖 FlagEmbedding。
|
||||
# ---------------------------------------------------------------------------
|
||||
class DenseEncoder:
|
||||
def __init__(self, model_name: str, pooling: str = "auto",
|
||||
device: str = "cpu", max_length: int = 512):
|
||||
import torch
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
self.torch = torch
|
||||
self.model_name = model_name
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
self.model = AutoModel.from_pretrained(model_name)
|
||||
self.model.eval().to(device)
|
||||
self.device = device
|
||||
self.max_length = max_length
|
||||
if pooling == "auto":
|
||||
# bge / bge-m3 的稠密向量取 [CLS]；多数 sentence-transformers 模型用平均池化
|
||||
pooling = "cls" if "bge" in model_name.lower() else "mean"
|
||||
self.pooling = pooling
|
||||
|
||||
def encode(self, texts: List[str], batch_size: int = 16) -> np.ndarray:
|
||||
vecs: List[np.ndarray] = []
|
||||
for i in range(0, len(texts), batch_size):
|
||||
batch = texts[i:i + batch_size]
|
||||
enc = self.tokenizer(batch, padding=True, truncation=True,
|
||||
max_length=self.max_length, return_tensors="pt").to(self.device)
|
||||
with self.torch.no_grad():
|
||||
out = self.model(**enc)
|
||||
if self.pooling == "cls":
|
||||
emb = out.last_hidden_state[:, 0]
|
||||
else:
|
||||
mask = enc["attention_mask"].unsqueeze(-1).float()
|
||||
emb = (out.last_hidden_state * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
|
||||
emb = self.torch.nn.functional.normalize(emb, p=2, dim=1)
|
||||
vecs.append(emb.cpu().numpy().astype("float32"))
|
||||
return np.vstack(vecs)
|
||||
|
||||
|
||||
def load_encoder(model_name: str, pooling: str, device: str) -> Optional["DenseEncoder"]:
|
||||
"""加载稠密编码器。离线且模型未缓存时给出清晰提示并返回 None（不影响参数解析验证）。"""
|
||||
try:
|
||||
import torch # noqa: F401
|
||||
from transformers import AutoModel # noqa: F401
|
||||
except Exception as e:
|
||||
print("\n[稠密编码] 需要依赖 transformers 与 torch，当前环境缺失：", e)
|
||||
print(" 安装：pip install torch transformers")
|
||||
print(" （--compare-ann 使用合成向量，无需任何模型，可完全离线运行）")
|
||||
return None
|
||||
try:
|
||||
print(f"正在加载嵌入模型 {model_name}（pooling={pooling}, device={device}）...")
|
||||
t0 = time.time()
|
||||
encoder = DenseEncoder(model_name, pooling=pooling, device=device)
|
||||
print(f"模型加载完成，耗时 {time.time() - t0:.1f}s，池化方式 ={encoder.pooling}")
|
||||
return encoder
|
||||
except Exception as e:
|
||||
print(f"\n[稠密编码] 无法加载模型 {model_name}：{e}")
|
||||
print(f" 离线环境无法下载 {model_name} 权重（BGE-M3 约 2.3GB）。")
|
||||
print(f" 可改用已缓存的小模型：--embedding-model {OFFLINE_HINT_MODEL}")
|
||||
print(" 或先在联网环境预缓存目标模型；--compare-ann 则完全无需模型。")
|
||||
return None
|
||||
|
||||
|
||||
def load_corpus(path: Optional[str]) -> List[Dict]:
|
||||
"""加载语料。支持 .json（文档数组）与 .jsonl（每行一个文档）。"""
|
||||
if not path:
|
||||
return DEFAULT_CORPUS
|
||||
docs: List[Dict] = []
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
if path.endswith(".jsonl"):
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
docs.append(json.loads(line))
|
||||
else:
|
||||
data = json.load(f)
|
||||
docs = data["documents"] if isinstance(data, dict) else data
|
||||
if not docs:
|
||||
raise ValueError(f"语料文件为空：{path}")
|
||||
return docs
|
||||
|
||||
|
||||
def load_labels(path: Optional[str]) -> Dict[str, List[str]]:
|
||||
"""加载评测标注：{query: [relevant_doc_id, ...]}。"""
|
||||
if not path:
|
||||
return DEFAULT_LABELS
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 稠密检索（精确暴力，用于单条查询与检索质量评测）
|
||||
# ---------------------------------------------------------------------------
|
||||
def dense_rank(query_vec: np.ndarray, doc_matrix: np.ndarray) -> List[int]:
|
||||
"""向量已 L2 归一化，余弦相似度即点积；返回按相似度降序的文档下标。"""
|
||||
sims = doc_matrix @ query_vec
|
||||
return list(np.argsort(-sims)), sims
|
||||
|
||||
|
||||
def run_search(encoder: "DenseEncoder", corpus: List[Dict], doc_matrix: np.ndarray,
|
||||
query: str, top_k: int) -> List[Dict]:
|
||||
"""执行单条稠密查询并打印结果，返回结构化结果供 --output 落盘。"""
|
||||
q = encoder.encode([query])[0]
|
||||
order, sims = dense_rank(q, doc_matrix)
|
||||
print(f"\n查询: '{query}' (稠密检索, top-{top_k})")
|
||||
print("-" * 60)
|
||||
out = []
|
||||
for rank, idx in enumerate(order[:top_k], 1):
|
||||
d = corpus[idx]
|
||||
title = d.get("title", "")
|
||||
print(f" #{rank} {d.get('doc_id')} cos={float(sims[idx]):.4f} {title}")
|
||||
print(f" 预览: {d['text'][:80]}...")
|
||||
out.append({
|
||||
"rank": rank,
|
||||
"doc_id": d.get("doc_id"),
|
||||
"score": float(sims[idx]),
|
||||
"title": title,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _metrics_for_query(retrieved: List[str], relevant: Set[str], k: int) -> Dict:
|
||||
"""单条查询的 recall@k / precision@k / 命中排名（用于 MRR）。"""
|
||||
topk = retrieved[:k]
|
||||
hits = [d for d in topk if d in relevant]
|
||||
recall = len(set(hits)) / len(relevant) if relevant else 0.0
|
||||
precision = len(hits) / len(topk) if topk else 0.0
|
||||
rr = 0.0
|
||||
for i, d in enumerate(retrieved, 1):
|
||||
if d in relevant:
|
||||
rr = 1.0 / i
|
||||
break
|
||||
return {"recall": recall, "precision": precision, "rr": rr,
|
||||
"hits": hits, "retrieved": topk}
|
||||
|
||||
|
||||
def run_eval(encoder: "DenseEncoder", corpus: List[Dict], doc_matrix: np.ndarray,
|
||||
labels: Dict[str, List[str]], k: int) -> Dict:
|
||||
"""在标注集上做稠密检索评测，打印每条查询指标 + 宏平均。"""
|
||||
doc_ids = [d.get("doc_id") for d in corpus]
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"稠密检索质量评测 (recall@{k} / precision@{k} / MRR)")
|
||||
print(f"{'=' * 60}")
|
||||
per_query = {}
|
||||
sum_recall = sum_prec = sum_rr = 0.0
|
||||
q_vecs = encoder.encode(list(labels.keys()))
|
||||
for (query, rel_list), qv in zip(labels.items(), q_vecs):
|
||||
relevant = set(rel_list)
|
||||
order, _ = dense_rank(qv, doc_matrix)
|
||||
retrieved = [doc_ids[i] for i in order]
|
||||
m = _metrics_for_query(retrieved, relevant, k)
|
||||
per_query[query] = m
|
||||
sum_recall += m["recall"]
|
||||
sum_prec += m["precision"]
|
||||
sum_rr += m["rr"]
|
||||
flag = "" if m["recall"] > 0 else " <- 漏召回"
|
||||
print(f"\n查询 '{query}' 相关文档={sorted(relevant)}")
|
||||
print(f" 召回排序: {retrieved[:k]}")
|
||||
print(f" recall@{k}={m['recall']:.2f} precision@{k}={m['precision']:.2f} RR={m['rr']:.2f}{flag}")
|
||||
n = len(labels)
|
||||
macro = {
|
||||
"recall@k": sum_recall / n,
|
||||
"precision@k": sum_prec / n,
|
||||
"mrr": sum_rr / n,
|
||||
"miss_rate@k": 1.0 - sum_recall / n,
|
||||
}
|
||||
print(f"\n{'-' * 60}")
|
||||
print(f"宏平均 recall@{k}={macro['recall@k']:.3f} "
|
||||
f"precision@{k}={macro['precision@k']:.3f} "
|
||||
f"MRR={macro['mrr']:.3f} 漏召回率(1-recall@{k})={macro['miss_rate@k']:.3f}")
|
||||
return {"k": k, "per_query": {q: {kk: vv for kk, vv in m.items() if kk != "retrieved"}
|
||||
for q, m in per_query.items()},
|
||||
"macro": macro}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ANN 后端对比（实验 3-4 的重点）：复用服务端 indexing.py 里的 ANNOY / HNSW 实现，
|
||||
# 在一批合成单位向量上对比二者相对"精确暴力检索"的召回率、建索引耗时与查询延迟。
|
||||
# 用合成向量而非真实文本嵌入，是为了 (a) 完全离线、无需下载模型；(b) 语料足够大时
|
||||
# ANN 的"近似"才会显现出与精确检索的差距，从而看清两类算法的取舍。
|
||||
# ---------------------------------------------------------------------------
|
||||
def _exact_topk(queries: np.ndarray, base: np.ndarray, k: int) -> List[Set[int]]:
|
||||
"""精确暴力最近邻（余弦），作为 ANN 召回率的 ground truth。"""
|
||||
sims = queries @ base.T
|
||||
idx = np.argsort(-sims, axis=1)[:, :k]
|
||||
return [set(row.tolist()) for row in idx]
|
||||
|
||||
|
||||
def _sanity_ok(index, base: np.ndarray) -> bool:
|
||||
"""自检：用库中已存在的向量查询，应能召回它自己。用于识别环境中损坏的索引后端。"""
|
||||
probe = min(5, len(base))
|
||||
for i in range(probe):
|
||||
ids, _ = index.search(base[i], min(10, len(base)))
|
||||
if f"v{i}" not in set(ids):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def compare_ann(base: np.ndarray, queries: np.ndarray, top_k: int, backends: List[str],
|
||||
annoy_n_trees: int, hnsw_M: int, hnsw_ef_search: int,
|
||||
hnsw_ef_construction: int) -> Dict:
|
||||
dim = base.shape[1]
|
||||
n = len(base)
|
||||
exact_sets = _exact_topk(queries, base, top_k)
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"ANN 后端对比：{n} 条 {dim} 维向量，{len(queries)} 条查询，top-{top_k}")
|
||||
print(f"指标：recall@{top_k} 相对精确暴力检索 / 建索引耗时 / 平均查询延迟")
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
report: Dict[str, Dict] = {}
|
||||
for backend in backends:
|
||||
if backend == "annoy":
|
||||
index = AnnoyIndex(dimension=dim, n_trees=annoy_n_trees,
|
||||
metric="angular", logger=None)
|
||||
else:
|
||||
index = HNSWIndex(dimension=dim, max_elements=n + 16,
|
||||
ef_construction=hnsw_ef_construction, M=hnsw_M,
|
||||
ef_search=hnsw_ef_search, space="cosine", logger=None)
|
||||
|
||||
t0 = time.time()
|
||||
for i, v in enumerate(base):
|
||||
index.add_item(f"v{i}", v)
|
||||
if backend == "annoy":
|
||||
index.rebuild_index()
|
||||
build_time = time.time() - t0
|
||||
|
||||
healthy = _sanity_ok(index, base)
|
||||
|
||||
recalls: List[float] = []
|
||||
qtimes: List[float] = []
|
||||
for qi, q in enumerate(queries):
|
||||
ts = time.time()
|
||||
ids, _ = index.search(q, top_k)
|
||||
qtimes.append(time.time() - ts)
|
||||
got = {int(d[1:]) for d in ids}
|
||||
recalls.append(len(got & exact_sets[qi]) / top_k)
|
||||
|
||||
mean_recall = float(np.mean(recalls))
|
||||
mean_qms = float(np.mean(qtimes) * 1000)
|
||||
params = (f"n_trees={annoy_n_trees}" if backend == "annoy"
|
||||
else f"M={hnsw_M}, ef_search={hnsw_ef_search}, ef_construction={hnsw_ef_construction}")
|
||||
report[backend] = {
|
||||
"recall@k": mean_recall,
|
||||
"build_time_s": build_time,
|
||||
"mean_query_ms": mean_qms,
|
||||
"params": params,
|
||||
"healthy": healthy,
|
||||
}
|
||||
warn = "" if healthy else " [警告] 该后端连自身向量都召回不到，疑似当前环境下损坏，下列数字不可信"
|
||||
print(f"\n[{backend.upper()}] {params}{warn}")
|
||||
print(f" recall@{top_k} = {mean_recall:.3f}")
|
||||
print(f" 建索引耗时 = {build_time * 1000:.1f} ms")
|
||||
print(f" 平均查询延迟 = {mean_qms:.3f} ms")
|
||||
|
||||
if "annoy" in report and "hnsw" in report:
|
||||
print(f"\n{'-' * 60}")
|
||||
print("小结：HNSW 图结构通常召回率更高、支持增量插入，代价是更高内存与建索引开销；")
|
||||
print(" ANNOY 树结构建索引快、内存省，但删除需重建，召回随 n_trees 调节。")
|
||||
return report
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="cli.py",
|
||||
description="稠密检索命令行工具（实验 3-4）：在小型语料上运行稠密嵌入检索并评测检索质量，"
|
||||
"并对比 ANNOY / HNSW 两种 ANN 索引后端。",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""示例：
|
||||
python cli.py # 默认演示（查询 "a cat playing"，需嵌入模型）
|
||||
python cli.py -q "model distillation" -k 3 # 单条稠密查询
|
||||
python cli.py --eval # 在标注集上算 recall/precision/MRR
|
||||
python cli.py --embedding-model sentence-transformers/all-MiniLM-L6-v2 --eval # 离线小模型
|
||||
python cli.py --compare-ann # ANNOY vs HNSW 召回率对比（合成向量，无需模型）
|
||||
python cli.py --compare-ann --ann-base 5000 --annoy-n-trees 5 -k 10 -o ann.json
|
||||
""",
|
||||
)
|
||||
parser.add_argument("-q", "--query", default=DEFAULT_QUERY,
|
||||
help=f"查询字符串（默认: '{DEFAULT_QUERY}'）")
|
||||
parser.add_argument("-c", "--corpus", default=None,
|
||||
help="语料文件路径（.json 文档数组 或 .jsonl 每行一篇）；缺省用内置示例语料")
|
||||
parser.add_argument("-k", "--top-k", type=int, default=5,
|
||||
help="返回前 k 条结果（默认: 5）")
|
||||
parser.add_argument("-o", "--output", default=None,
|
||||
help="把结果/评测指标以 JSON 写入该文件")
|
||||
parser.add_argument("--embedding-model", default=DEFAULT_MODEL,
|
||||
help=f"稠密嵌入模型名（默认: {DEFAULT_MODEL}）；"
|
||||
f"离线可用已缓存的 {OFFLINE_HINT_MODEL}")
|
||||
parser.add_argument("--pooling", choices=["auto", "mean", "cls"], default="auto",
|
||||
help="句向量池化方式：auto(bge*用cls，其余用mean) / mean / cls")
|
||||
parser.add_argument("--device", default="cpu",
|
||||
help="推理设备（cpu / cuda / mps，默认: cpu）")
|
||||
parser.add_argument("--eval", action="store_true",
|
||||
help="在标注集上评测 recall@k / precision@k / MRR，而非只跑单条查询")
|
||||
parser.add_argument("--labels", default=None,
|
||||
help="评测标注文件 {query: [相关doc_id,...]}；缺省用内置标注")
|
||||
|
||||
ann = parser.add_argument_group("ANN 后端对比（--compare-ann）")
|
||||
ann.add_argument("--compare-ann", action="store_true",
|
||||
help="对比 ANNOY 与 HNSW 的召回率/耗时（复用 indexing.py，用合成向量，无需模型）")
|
||||
ann.add_argument("--backend", choices=["annoy", "hnsw", "both"], default="both",
|
||||
help="参与对比的 ANN 后端（默认: both）")
|
||||
ann.add_argument("--ann-base", type=int, default=3000,
|
||||
help="合成底库向量数量（默认: 3000，越大 ANN 近似误差越明显）")
|
||||
ann.add_argument("--ann-queries", type=int, default=100,
|
||||
help="合成查询向量数量（默认: 100）")
|
||||
ann.add_argument("--ann-dim", type=int, default=128,
|
||||
help="合成向量维度（默认: 128）")
|
||||
ann.add_argument("--annoy-n-trees", type=int, default=10,
|
||||
help="ANNOY 树数量（默认: 10；越多越准越慢）")
|
||||
ann.add_argument("--hnsw-M", type=int, default=16,
|
||||
help="HNSW 每节点连接数 M（默认: 16；越大召回越高越占内存）")
|
||||
ann.add_argument("--hnsw-ef-search", type=int, default=20,
|
||||
help="HNSW 查询期动态候选表大小 ef_search（默认: 20）")
|
||||
ann.add_argument("--hnsw-ef-construction", type=int, default=100,
|
||||
help="HNSW 建索引期动态候选表大小 ef_construction（默认: 100）")
|
||||
ann.add_argument("--seed", type=int, default=42,
|
||||
help="合成向量随机种子（默认: 42）")
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Optional[List[str]] = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
payload: Dict = {"top_k": args.top_k}
|
||||
|
||||
# --- ANN 后端对比：合成向量，无需嵌入模型，完全离线 ---
|
||||
if args.compare_ann:
|
||||
rng = np.random.default_rng(args.seed)
|
||||
base = rng.standard_normal((args.ann_base, args.ann_dim)).astype("float32")
|
||||
base /= np.linalg.norm(base, axis=1, keepdims=True)
|
||||
queries = rng.standard_normal((args.ann_queries, args.ann_dim)).astype("float32")
|
||||
queries /= np.linalg.norm(queries, axis=1, keepdims=True)
|
||||
backends = ["annoy", "hnsw"] if args.backend == "both" else [args.backend]
|
||||
payload["compare_ann"] = compare_ann(
|
||||
base, queries, args.top_k, backends,
|
||||
annoy_n_trees=args.annoy_n_trees, hnsw_M=args.hnsw_M,
|
||||
hnsw_ef_search=args.hnsw_ef_search, hnsw_ef_construction=args.hnsw_ef_construction)
|
||||
if args.output:
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
print(f"\n已写入结果：{args.output}")
|
||||
return 0
|
||||
|
||||
# --- 稠密检索 / 评测：需要嵌入模型 ---
|
||||
corpus = load_corpus(args.corpus)
|
||||
print(f"已加载语料：{len(corpus)} 篇文档"
|
||||
+ ("（内置示例）" if not args.corpus else f"（来自 {args.corpus}）"))
|
||||
|
||||
encoder = load_encoder(args.embedding_model, args.pooling, args.device)
|
||||
if encoder is None:
|
||||
return 0 # 已给出模型缺失提示，视为正常退出（参数解析已验证）
|
||||
|
||||
doc_matrix = encoder.encode([d["text"] for d in corpus])
|
||||
payload["embedding_model"] = args.embedding_model
|
||||
payload["query"] = args.query
|
||||
|
||||
if args.eval:
|
||||
labels = load_labels(args.labels)
|
||||
payload["eval"] = run_eval(encoder, corpus, doc_matrix, labels, args.top_k)
|
||||
else:
|
||||
payload["results"] = run_search(encoder, corpus, doc_matrix, args.query, args.top_k)
|
||||
|
||||
if args.output:
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
print(f"\n已写入结果：{args.output}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Configuration for the dense embedding service."""
|
||||
|
||||
import os
|
||||
from enum import Enum
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
class IndexType(Enum):
|
||||
"""Supported index types."""
|
||||
ANNOY = "annoy"
|
||||
HNSW = "hnsw"
|
||||
|
||||
@dataclass
|
||||
class ServiceConfig:
|
||||
"""Service configuration."""
|
||||
# Server settings
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 4240 # Default port for dense embedding service
|
||||
|
||||
# Model settings
|
||||
model_name: str = "BAAI/bge-m3"
|
||||
use_fp16: bool = True
|
||||
max_seq_length: int = 8192 # Increased to match HARD_LIMIT in chunking
|
||||
|
||||
# Index settings
|
||||
index_type: IndexType = IndexType.HNSW
|
||||
max_documents: int = 100000
|
||||
|
||||
# HNSW specific settings
|
||||
hnsw_ef_construction: int = 200
|
||||
hnsw_M: int = 16
|
||||
hnsw_ef_search: int = 50
|
||||
hnsw_space: str = "cosine"
|
||||
|
||||
# Annoy specific settings
|
||||
annoy_n_trees: int = 50
|
||||
annoy_metric: str = "angular"
|
||||
|
||||
# Logging settings
|
||||
log_level: str = "INFO"
|
||||
debug: bool = False
|
||||
show_embeddings: bool = False
|
||||
|
||||
@classmethod
|
||||
def from_env(cls):
|
||||
"""Create config from environment variables."""
|
||||
config = cls()
|
||||
|
||||
# Override with environment variables if present
|
||||
if os.getenv("DENSE_PORT"):
|
||||
config.port = int(os.getenv("DENSE_PORT"))
|
||||
if os.getenv("DENSE_HOST"):
|
||||
config.host = os.getenv("DENSE_HOST")
|
||||
if os.getenv("DENSE_MODEL"):
|
||||
config.model_name = os.getenv("DENSE_MODEL")
|
||||
if os.getenv("DEBUG"):
|
||||
config.debug = os.getenv("DEBUG").lower() == "true"
|
||||
|
||||
return config
|
||||
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Linux-isolated ANNOY measurement used when the host ARM wheel is broken."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
from annoy import AnnoyIndex
|
||||
|
||||
|
||||
def latency_stats(values):
|
||||
return {
|
||||
"mean": statistics.mean(values),
|
||||
"p50": float(np.percentile(values, 50)),
|
||||
"p95": float(np.percentile(values, 95)),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
input_path, output_path = sys.argv[1:3]
|
||||
data = np.load(input_path, allow_pickle=False)
|
||||
ids = [str(x) for x in data["ids"]]
|
||||
vectors = data["vectors"].astype("float32")
|
||||
queries = data["queries"].astype("float32")
|
||||
initial_truth = data["initial_truth"]
|
||||
full_truth = data["full_truth"]
|
||||
initial_n, k, repeats = (int(x) for x in data["parameters"])
|
||||
dimension = vectors.shape[1]
|
||||
|
||||
index = AnnoyIndex(dimension, "angular")
|
||||
started = time.perf_counter()
|
||||
for i, vector in enumerate(vectors[:initial_n]):
|
||||
index.add_item(i, vector.tolist())
|
||||
index.build(50)
|
||||
build_ms = (time.perf_counter() - started) * 1000
|
||||
|
||||
recalls, latencies, rankings = [], [], []
|
||||
for q_idx, query in enumerate(queries):
|
||||
first = None
|
||||
for _ in range(repeats):
|
||||
started = time.perf_counter()
|
||||
found = index.get_nns_by_vector(query.tolist(), k, -1, False)
|
||||
latencies.append((time.perf_counter() - started) * 1000)
|
||||
if first is None:
|
||||
first = found
|
||||
recalls.append(len(set(first) & set(initial_truth[q_idx].tolist())) / k)
|
||||
rankings.append({"query_index": q_idx, "doc_ids": [ids[i] for i in first]})
|
||||
|
||||
with tempfile.NamedTemporaryFile() as handle:
|
||||
index.save(handle.name)
|
||||
serialized_bytes = os.path.getsize(handle.name)
|
||||
|
||||
# ANNOY cannot mutate a built index: full update means rebuilding a fresh tree.
|
||||
started = time.perf_counter()
|
||||
updated = AnnoyIndex(dimension, "angular")
|
||||
for i, vector in enumerate(vectors):
|
||||
updated.add_item(i, vector.tolist())
|
||||
updated.build(50)
|
||||
update_ms = (time.perf_counter() - started) * 1000
|
||||
update_recalls = []
|
||||
for q_idx, query in enumerate(queries):
|
||||
found = updated.get_nns_by_vector(query.tolist(), k, -1, False)
|
||||
update_recalls.append(len(set(found) & set(full_truth[q_idx].tolist())) / k)
|
||||
|
||||
payload = {
|
||||
"build_ms": round(build_ms, 3),
|
||||
"recall_at_k": statistics.mean(recalls),
|
||||
"query_latency_ms": latency_stats(latencies),
|
||||
"serialized_bytes": serialized_bytes,
|
||||
"rankings": rankings,
|
||||
"incremental_update": {
|
||||
"items_added": len(ids) - initial_n,
|
||||
"latency_ms": round(update_ms, 3),
|
||||
"requires_full_rebuild": True,
|
||||
"recall_at_k_after_update": statistics.mean(update_recalls),
|
||||
},
|
||||
}
|
||||
with open(output_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, indent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,193 @@
|
||||
"""In-memory document store for managing documents."""
|
||||
|
||||
from typing import Dict, Optional, List
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
import uuid
|
||||
from logger import VectorSearchLogger
|
||||
|
||||
|
||||
@dataclass
|
||||
class Document:
|
||||
"""Document data class."""
|
||||
id: str
|
||||
text: str
|
||||
metadata: Dict = field(default_factory=dict)
|
||||
created_at: datetime = field(default_factory=datetime.now)
|
||||
embedding: Optional[List[float]] = None
|
||||
|
||||
|
||||
class DocumentStore:
|
||||
"""In-memory document storage."""
|
||||
|
||||
def __init__(self, logger: Optional[VectorSearchLogger] = None):
|
||||
"""
|
||||
Initialize the document store.
|
||||
|
||||
Args:
|
||||
logger: Logger instance for educational output
|
||||
"""
|
||||
self.documents: Dict[str, Document] = {}
|
||||
self.logger = logger
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info("📦 Initialized in-memory document store")
|
||||
|
||||
def add_document(self, text: str, doc_id: Optional[str] = None,
|
||||
metadata: Optional[Dict] = None) -> str:
|
||||
"""
|
||||
Add a document to the store.
|
||||
|
||||
Args:
|
||||
text: Document text
|
||||
doc_id: Optional document ID (will be generated if not provided)
|
||||
metadata: Optional metadata dictionary
|
||||
|
||||
Returns:
|
||||
Document ID
|
||||
"""
|
||||
# Generate ID if not provided
|
||||
if doc_id is None:
|
||||
doc_id = str(uuid.uuid4())
|
||||
|
||||
# Check if document already exists
|
||||
if doc_id in self.documents:
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Document {doc_id} already exists, updating...")
|
||||
|
||||
# Create document
|
||||
doc = Document(
|
||||
id=doc_id,
|
||||
text=text,
|
||||
metadata=metadata or {}
|
||||
)
|
||||
|
||||
# Store document
|
||||
self.documents[doc_id] = doc
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"📄 Stored document")
|
||||
self.logger.logger.debug(f" - ID: {doc_id}")
|
||||
self.logger.logger.debug(f" - Text length: {len(text)} chars")
|
||||
self.logger.logger.debug(f" - Metadata keys: {list(metadata.keys()) if metadata else []}")
|
||||
self.logger.logger.debug(f" - Total documents: {len(self.documents)}")
|
||||
|
||||
return doc_id
|
||||
|
||||
def get_document(self, doc_id: str) -> Optional[Document]:
|
||||
"""
|
||||
Retrieve a document by ID.
|
||||
|
||||
Args:
|
||||
doc_id: Document ID
|
||||
|
||||
Returns:
|
||||
Document or None if not found
|
||||
"""
|
||||
doc = self.documents.get(doc_id)
|
||||
|
||||
if self.logger:
|
||||
if doc:
|
||||
self.logger.logger.debug(f"✅ Retrieved document {doc_id}")
|
||||
else:
|
||||
self.logger.logger.warning(f"❌ Document {doc_id} not found")
|
||||
|
||||
return doc
|
||||
|
||||
def delete_document(self, doc_id: str) -> bool:
|
||||
"""
|
||||
Delete a document from the store.
|
||||
|
||||
Args:
|
||||
doc_id: Document ID
|
||||
|
||||
Returns:
|
||||
True if deleted, False if not found
|
||||
"""
|
||||
if doc_id in self.documents:
|
||||
del self.documents[doc_id]
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"🗑️ Deleted document {doc_id}")
|
||||
self.logger.logger.debug(f" Remaining documents: {len(self.documents)}")
|
||||
|
||||
return True
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Document {doc_id} not found for deletion")
|
||||
|
||||
return False
|
||||
|
||||
def list_documents(self, limit: Optional[int] = None) -> List[Document]:
|
||||
"""
|
||||
List all documents in the store.
|
||||
|
||||
Args:
|
||||
limit: Maximum number of documents to return
|
||||
|
||||
Returns:
|
||||
List of documents
|
||||
"""
|
||||
docs = list(self.documents.values())
|
||||
|
||||
if limit:
|
||||
docs = docs[:limit]
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"📋 Listing {len(docs)} documents")
|
||||
|
||||
return docs
|
||||
|
||||
def get_size(self) -> int:
|
||||
"""Get the number of documents in the store."""
|
||||
return len(self.documents)
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Clear all documents from the store."""
|
||||
count = len(self.documents)
|
||||
self.documents.clear()
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"🧹 Cleared {count} documents from store")
|
||||
|
||||
def get_documents_by_ids(self, doc_ids: List[str]) -> List[Document]:
|
||||
"""
|
||||
Retrieve multiple documents by their IDs.
|
||||
|
||||
Args:
|
||||
doc_ids: List of document IDs
|
||||
|
||||
Returns:
|
||||
List of documents (only those found)
|
||||
"""
|
||||
docs = []
|
||||
for doc_id in doc_ids:
|
||||
doc = self.documents.get(doc_id)
|
||||
if doc:
|
||||
docs.append(doc)
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"Retrieved {len(docs)}/{len(doc_ids)} documents")
|
||||
|
||||
return docs
|
||||
|
||||
def update_document_embedding(self, doc_id: str, embedding: List[float]) -> bool:
|
||||
"""
|
||||
Update the embedding for a document.
|
||||
|
||||
Args:
|
||||
doc_id: Document ID
|
||||
embedding: Embedding vector
|
||||
|
||||
Returns:
|
||||
True if updated, False if document not found
|
||||
"""
|
||||
if doc_id in self.documents:
|
||||
self.documents[doc_id].embedding = embedding
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"Updated embedding for document {doc_id}")
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Embedding service using BGE-M3 model."""
|
||||
|
||||
import time
|
||||
import numpy as np
|
||||
from typing import List, Dict, Optional
|
||||
from FlagEmbedding import BGEM3FlagModel
|
||||
from logger import VectorSearchLogger, log_execution_time
|
||||
import logging
|
||||
|
||||
|
||||
class EmbeddingService:
|
||||
"""Service for generating embeddings using BGE-M3 model."""
|
||||
|
||||
def __init__(self, model_name: str = "BAAI/bge-m3", use_fp16: bool = True,
|
||||
max_seq_length: int = 512, logger: Optional[VectorSearchLogger] = None):
|
||||
"""
|
||||
Initialize the embedding service with BGE-M3 model.
|
||||
|
||||
Args:
|
||||
model_name: Name of the BGE-M3 model
|
||||
use_fp16: Whether to use FP16 for inference
|
||||
max_seq_length: Maximum sequence length
|
||||
logger: Logger instance for educational output
|
||||
"""
|
||||
self.model_name = model_name
|
||||
self.use_fp16 = use_fp16
|
||||
self.max_seq_length = max_seq_length
|
||||
self.logger = logger
|
||||
self.std_logger = logging.getLogger("vector_search")
|
||||
|
||||
# Initialize the model
|
||||
self._initialize_model()
|
||||
|
||||
def _initialize_model(self):
|
||||
"""Initialize the BGE-M3 model."""
|
||||
start_time = time.time()
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"🚀 Initializing BGE-M3 model: {self.model_name}")
|
||||
self.logger.logger.debug(f" - Using FP16: {self.use_fp16}")
|
||||
self.logger.logger.debug(f" - Max sequence length: {self.max_seq_length}")
|
||||
|
||||
try:
|
||||
self.model = BGEM3FlagModel(
|
||||
self.model_name,
|
||||
use_fp16=self.use_fp16
|
||||
)
|
||||
|
||||
# Get embedding dimension by encoding a test sentence
|
||||
test_embedding = self.model.encode(["test"])
|
||||
if isinstance(test_embedding, dict):
|
||||
self.embedding_dim = test_embedding['dense_vecs'].shape[1]
|
||||
else:
|
||||
self.embedding_dim = test_embedding.shape[1]
|
||||
|
||||
load_time = time.time() - start_time
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"✅ Model loaded successfully in {load_time:.2f} seconds")
|
||||
self.logger.logger.debug(f" - Embedding dimension: {self.embedding_dim}")
|
||||
self.logger.logger.debug(f" - Model supports: dense, sparse, and multi-vector retrieval")
|
||||
|
||||
except Exception as e:
|
||||
if self.logger:
|
||||
self.logger.logger.error(f"Failed to load model: {e}")
|
||||
raise
|
||||
|
||||
@log_execution_time()
|
||||
def encode_text(self, text: str, return_sparse: bool = False,
|
||||
return_colbert: bool = False) -> Dict[str, np.ndarray]:
|
||||
"""
|
||||
Encode a single text into embeddings.
|
||||
|
||||
Args:
|
||||
text: Input text to encode
|
||||
return_sparse: Whether to return sparse embeddings
|
||||
return_colbert: Whether to return ColBERT embeddings
|
||||
|
||||
Returns:
|
||||
Dictionary containing different types of embeddings
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"📝 Encoding text (length: {len(text)} chars)")
|
||||
self.logger.logger.debug(f" Text preview: {text[:100]}..." if len(text) > 100 else f" Text: {text}")
|
||||
|
||||
# Encode the text
|
||||
embeddings = self.model.encode(
|
||||
[text],
|
||||
return_dense=True,
|
||||
return_sparse=return_sparse,
|
||||
return_colbert_vecs=return_colbert
|
||||
)
|
||||
|
||||
# Extract dense embeddings
|
||||
dense_vec = embeddings['dense_vecs'][0]
|
||||
|
||||
result = {
|
||||
'dense': dense_vec,
|
||||
'dimension': len(dense_vec)
|
||||
}
|
||||
|
||||
# Add sparse embeddings if requested
|
||||
if return_sparse and 'lexical_weights' in embeddings:
|
||||
result['sparse'] = embeddings['lexical_weights'][0]
|
||||
if self.logger:
|
||||
num_tokens = len(result['sparse'])
|
||||
self.logger.logger.debug(f" Sparse embedding: {num_tokens} non-zero tokens")
|
||||
|
||||
# Add ColBERT embeddings if requested
|
||||
if return_colbert and 'colbert_vecs' in embeddings:
|
||||
result['colbert'] = embeddings['colbert_vecs'][0]
|
||||
if self.logger:
|
||||
colbert_shape = result['colbert'].shape
|
||||
self.logger.logger.debug(f" ColBERT embedding shape: {colbert_shape}")
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"✅ Encoding completed in {encoding_time:.4f} seconds")
|
||||
self.logger.log_embedding_vector(dense_vec, sample_size=10)
|
||||
|
||||
return result
|
||||
|
||||
@log_execution_time()
|
||||
def encode_batch(self, texts: List[str], return_sparse: bool = False,
|
||||
return_colbert: bool = False) -> Dict[str, np.ndarray]:
|
||||
"""
|
||||
Encode multiple texts into embeddings.
|
||||
|
||||
Args:
|
||||
texts: List of input texts to encode
|
||||
return_sparse: Whether to return sparse embeddings
|
||||
return_colbert: Whether to return ColBERT embeddings
|
||||
|
||||
Returns:
|
||||
Dictionary containing different types of embeddings for all texts
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"📚 Batch encoding {len(texts)} texts")
|
||||
total_chars = sum(len(t) for t in texts)
|
||||
self.logger.logger.debug(f" Total characters: {total_chars}")
|
||||
avg_len = total_chars / len(texts) if texts else 0.0
|
||||
self.logger.logger.debug(f" Average text length: {avg_len:.1f} chars")
|
||||
|
||||
# Encode all texts
|
||||
embeddings = self.model.encode(
|
||||
texts,
|
||||
return_dense=True,
|
||||
return_sparse=return_sparse,
|
||||
return_colbert_vecs=return_colbert
|
||||
)
|
||||
|
||||
result = {
|
||||
'dense': embeddings['dense_vecs'],
|
||||
'dimension': embeddings['dense_vecs'].shape[1],
|
||||
'num_texts': len(texts)
|
||||
}
|
||||
|
||||
# Add sparse embeddings if requested
|
||||
if return_sparse and 'lexical_weights' in embeddings:
|
||||
result['sparse'] = embeddings['lexical_weights']
|
||||
|
||||
# Add ColBERT embeddings if requested
|
||||
if return_colbert and 'colbert_vecs' in embeddings:
|
||||
result['colbert'] = embeddings['colbert_vecs']
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"✅ Batch encoding completed in {encoding_time:.4f} seconds")
|
||||
avg_time = encoding_time / len(texts) if texts else 0.0
|
||||
self.logger.logger.debug(f" Average time per text: {avg_time:.4f} seconds")
|
||||
|
||||
return result
|
||||
|
||||
def get_embedding_dimension(self) -> int:
|
||||
"""Get the dimension of the embeddings."""
|
||||
return self.embedding_dim
|
||||
|
||||
def compute_similarity(self, vec1: np.ndarray, vec2: np.ndarray,
|
||||
metric: str = "cosine") -> float:
|
||||
"""
|
||||
Compute similarity between two vectors.
|
||||
|
||||
Args:
|
||||
vec1: First vector
|
||||
vec2: Second vector
|
||||
metric: Similarity metric ('cosine', 'euclidean', 'dot')
|
||||
|
||||
Returns:
|
||||
Similarity score
|
||||
"""
|
||||
if metric == "cosine":
|
||||
# Cosine similarity
|
||||
dot_product = np.dot(vec1, vec2)
|
||||
norm1 = np.linalg.norm(vec1)
|
||||
norm2 = np.linalg.norm(vec2)
|
||||
similarity = dot_product / (norm1 * norm2)
|
||||
|
||||
elif metric == "euclidean":
|
||||
# Euclidean distance (negative for similarity)
|
||||
similarity = -np.linalg.norm(vec1 - vec2)
|
||||
|
||||
elif metric == "dot":
|
||||
# Dot product
|
||||
similarity = np.dot(vec1, vec2)
|
||||
|
||||
else:
|
||||
raise ValueError(f"Unknown metric: {metric}")
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f" Similarity ({metric}): {similarity:.6f}")
|
||||
|
||||
return float(similarity)
|
||||
@@ -0,0 +1,390 @@
|
||||
"""Vector index implementations using ANNOY and HNSW."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import List, Tuple, Dict, Optional
|
||||
import numpy as np
|
||||
import annoy
|
||||
import hnswlib
|
||||
import time
|
||||
from logger import VectorSearchLogger
|
||||
|
||||
|
||||
class VectorIndex(ABC):
|
||||
"""Abstract base class for vector indexes."""
|
||||
|
||||
@abstractmethod
|
||||
def add_item(self, doc_id: str, vector: np.ndarray) -> None:
|
||||
"""Add an item to the index."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete_item(self, doc_id: str) -> bool:
|
||||
"""Delete an item from the index."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def search(self, query_vector: np.ndarray, top_k: int) -> Tuple[List[str], List[float]]:
|
||||
"""Search for top-k similar items."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_size(self) -> int:
|
||||
"""Get the current number of items in the index."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def rebuild_index(self) -> None:
|
||||
"""Rebuild the index if necessary."""
|
||||
pass
|
||||
|
||||
|
||||
class AnnoyIndex(VectorIndex):
|
||||
"""ANNOY-based vector index implementation."""
|
||||
|
||||
def __init__(self, dimension: int, n_trees: int = 50, metric: str = "angular",
|
||||
logger: Optional[VectorSearchLogger] = None):
|
||||
"""
|
||||
Initialize ANNOY index.
|
||||
|
||||
Args:
|
||||
dimension: Dimension of vectors
|
||||
n_trees: Number of trees for ANNOY (affects precision/speed tradeoff)
|
||||
metric: Distance metric ('angular', 'euclidean', 'manhattan', 'hamming', 'dot')
|
||||
logger: Logger instance for educational output
|
||||
"""
|
||||
self.dimension = dimension
|
||||
self.n_trees = n_trees
|
||||
self.metric = metric
|
||||
self.logger = logger
|
||||
|
||||
# Create index
|
||||
self.index = annoy.AnnoyIndex(dimension, metric)
|
||||
|
||||
# Mapping between internal indices and document IDs
|
||||
self.id_to_index: Dict[str, int] = {}
|
||||
self.index_to_id: Dict[int, str] = {}
|
||||
self.vectors_cache: Dict[int, np.ndarray] = {}
|
||||
self.next_index = 0
|
||||
self.is_built = False
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"📚 Initialized ANNOY index")
|
||||
self.logger.logger.debug(f" - Dimension: {dimension}")
|
||||
self.logger.logger.debug(f" - Number of trees: {n_trees}")
|
||||
self.logger.logger.debug(f" - Metric: {metric}")
|
||||
|
||||
def add_item(self, doc_id: str, vector: np.ndarray) -> None:
|
||||
"""Add an item to the ANNOY index."""
|
||||
start_time = time.time()
|
||||
|
||||
if doc_id in self.id_to_index:
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Document {doc_id} already exists in index, updating...")
|
||||
# Remove old entry
|
||||
old_index = self.id_to_index[doc_id]
|
||||
del self.index_to_id[old_index]
|
||||
del self.vectors_cache[old_index]
|
||||
|
||||
# Cache the vector only. ANNOY refuses new items once build() has run,
|
||||
# so the underlying index is (re)created from vectors_cache in
|
||||
# rebuild_index() instead of being mutated in place here.
|
||||
current_index = self.next_index
|
||||
|
||||
# Update mappings
|
||||
self.id_to_index[doc_id] = current_index
|
||||
self.index_to_id[current_index] = doc_id
|
||||
self.vectors_cache[current_index] = vector.copy()
|
||||
self.next_index += 1
|
||||
|
||||
# Mark index as needing rebuild
|
||||
self.is_built = False
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f"✅ Added document to ANNOY index in {time_taken:.4f}s")
|
||||
self.logger.logger.debug(f" - Document ID: {doc_id}")
|
||||
self.logger.logger.debug(f" - Internal index: {current_index}")
|
||||
self.logger.logger.debug(f" - Index needs rebuild: True")
|
||||
|
||||
def delete_item(self, doc_id: str) -> bool:
|
||||
"""
|
||||
Delete an item from the index.
|
||||
Note: ANNOY doesn't support deletion, so we need to rebuild without the item.
|
||||
"""
|
||||
if doc_id not in self.id_to_index:
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Document {doc_id} not found in index")
|
||||
return False
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"🗑️ Deleting from ANNOY index (requires rebuild)")
|
||||
|
||||
# Remove from mappings
|
||||
old_index = self.id_to_index[doc_id]
|
||||
del self.id_to_index[doc_id]
|
||||
del self.index_to_id[old_index]
|
||||
del self.vectors_cache[old_index]
|
||||
|
||||
# Rebuild index without the deleted item
|
||||
self._rebuild_without_deleted()
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.debug(f"✅ Document {doc_id} deleted and index rebuilt")
|
||||
|
||||
return True
|
||||
|
||||
def _rebuild_without_deleted(self):
|
||||
"""Rebuild the index without deleted items."""
|
||||
start_time = time.time()
|
||||
|
||||
# Create new index
|
||||
new_index = annoy.AnnoyIndex(self.dimension, self.metric)
|
||||
|
||||
# Create new mappings
|
||||
new_id_to_index = {}
|
||||
new_index_to_id = {}
|
||||
new_vectors_cache = {}
|
||||
|
||||
# Add all remaining items to new index
|
||||
new_idx = 0
|
||||
for old_idx, doc_id in self.index_to_id.items():
|
||||
if old_idx in self.vectors_cache:
|
||||
vector = self.vectors_cache[old_idx]
|
||||
new_index.add_item(new_idx, vector.tolist())
|
||||
new_id_to_index[doc_id] = new_idx
|
||||
new_index_to_id[new_idx] = doc_id
|
||||
new_vectors_cache[new_idx] = vector
|
||||
new_idx += 1
|
||||
|
||||
# Build the new index
|
||||
new_index.build(self.n_trees)
|
||||
|
||||
# Replace old index with new one
|
||||
self.index = new_index
|
||||
self.id_to_index = new_id_to_index
|
||||
self.index_to_id = new_index_to_id
|
||||
self.vectors_cache = new_vectors_cache
|
||||
self.next_index = new_idx
|
||||
self.is_built = True
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f" Rebuild completed in {time_taken:.4f}s")
|
||||
self.logger.logger.debug(f" New index size: {len(self.id_to_index)} documents")
|
||||
|
||||
def search(self, query_vector: np.ndarray, top_k: int) -> Tuple[List[str], List[float]]:
|
||||
"""Search for top-k similar items in the ANNOY index."""
|
||||
# Build index if needed
|
||||
if not self.is_built:
|
||||
self.rebuild_index()
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Ensure we don't request more items than we have
|
||||
actual_k = min(top_k, len(self.index_to_id))
|
||||
|
||||
if actual_k == 0:
|
||||
if self.logger:
|
||||
self.logger.logger.warning("Index is empty, returning no results")
|
||||
return [], []
|
||||
|
||||
# Search in index
|
||||
indices, distances = self.index.get_nns_by_vector(
|
||||
query_vector.tolist(), actual_k, include_distances=True
|
||||
)
|
||||
|
||||
# Convert indices to document IDs
|
||||
doc_ids = [self.index_to_id[idx] for idx in indices if idx in self.index_to_id]
|
||||
valid_distances = distances[:len(doc_ids)]
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f"⚡ ANNOY search completed in {time_taken:.4f}s")
|
||||
self.logger.logger.debug(f" Retrieved {len(doc_ids)} results")
|
||||
|
||||
return doc_ids, valid_distances
|
||||
|
||||
def get_size(self) -> int:
|
||||
"""Get the current number of items in the index."""
|
||||
return len(self.id_to_index)
|
||||
|
||||
def rebuild_index(self) -> None:
|
||||
"""Build/rebuild the ANNOY index."""
|
||||
if self.is_built:
|
||||
if self.logger:
|
||||
self.logger.logger.debug("Index already built, skipping rebuild")
|
||||
return
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"🏗️ Building ANNOY index with {self.n_trees} trees")
|
||||
|
||||
# ANNOY can neither accept items after a build nor be built twice, so
|
||||
# always construct a fresh index from the cached vectors.
|
||||
new_index = annoy.AnnoyIndex(self.dimension, self.metric)
|
||||
for internal_index, vector in self.vectors_cache.items():
|
||||
new_index.add_item(internal_index, vector.tolist())
|
||||
new_index.build(self.n_trees)
|
||||
self.index = new_index
|
||||
self.is_built = True
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f"✅ Index built in {time_taken:.4f}s")
|
||||
|
||||
|
||||
class HNSWIndex(VectorIndex):
|
||||
"""HNSW-based vector index implementation."""
|
||||
|
||||
def __init__(self, dimension: int, max_elements: int = 100000,
|
||||
ef_construction: int = 200, M: int = 32, ef_search: int = 100,
|
||||
space: str = "cosine", logger: Optional[VectorSearchLogger] = None):
|
||||
"""
|
||||
Initialize HNSW index.
|
||||
|
||||
Args:
|
||||
dimension: Dimension of vectors
|
||||
max_elements: Maximum number of elements
|
||||
ef_construction: Size of the dynamic list (affects build time/accuracy)
|
||||
M: Number of bi-directional links (affects memory/accuracy)
|
||||
ef_search: Size of the dynamic list for search (affects search time/accuracy)
|
||||
space: Distance metric ('l2', 'ip', 'cosine')
|
||||
logger: Logger instance for educational output
|
||||
"""
|
||||
self.dimension = dimension
|
||||
self.max_elements = max_elements
|
||||
self.ef_construction = ef_construction
|
||||
self.M = M
|
||||
self.ef_search = ef_search
|
||||
self.space = space
|
||||
self.logger = logger
|
||||
|
||||
# Create index
|
||||
self.index = hnswlib.Index(space=space, dim=dimension)
|
||||
self.index.init_index(max_elements=max_elements, ef_construction=ef_construction, M=M)
|
||||
self.index.set_ef(ef_search)
|
||||
|
||||
# Mapping between document IDs and internal labels
|
||||
self.id_to_label: Dict[str, int] = {}
|
||||
self.label_to_id: Dict[int, str] = {}
|
||||
self.available_labels: List[int] = []
|
||||
self.next_label = 0
|
||||
|
||||
if self.logger:
|
||||
self.logger.logger.info(f"📚 Initialized HNSW index")
|
||||
self.logger.logger.debug(f" - Dimension: {dimension}")
|
||||
self.logger.logger.debug(f" - Max elements: {max_elements}")
|
||||
self.logger.logger.debug(f" - ef_construction: {ef_construction}")
|
||||
self.logger.logger.debug(f" - M: {M}")
|
||||
self.logger.logger.debug(f" - ef_search: {ef_search}")
|
||||
self.logger.logger.debug(f" - Space: {space}")
|
||||
|
||||
def add_item(self, doc_id: str, vector: np.ndarray) -> None:
|
||||
"""Add an item to the HNSW index."""
|
||||
start_time = time.time()
|
||||
|
||||
# Check if document already exists
|
||||
if doc_id in self.id_to_label:
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Document {doc_id} already exists, updating...")
|
||||
# Remove old entry first
|
||||
self.delete_item(doc_id)
|
||||
|
||||
# Get a label for this document
|
||||
if self.available_labels:
|
||||
label = self.available_labels.pop()
|
||||
else:
|
||||
label = self.next_label
|
||||
self.next_label += 1
|
||||
|
||||
# Add to index
|
||||
self.index.add_items(vector.reshape(1, -1), np.array([label]))
|
||||
|
||||
# Update mappings
|
||||
self.id_to_label[doc_id] = label
|
||||
self.label_to_id[label] = doc_id
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f"✅ Added document to HNSW index in {time_taken:.4f}s")
|
||||
self.logger.logger.debug(f" - Document ID: {doc_id}")
|
||||
self.logger.logger.debug(f" - Internal label: {label}")
|
||||
self.logger.logger.debug(f" - Current index size: {self.index.get_current_count()}")
|
||||
|
||||
def delete_item(self, doc_id: str) -> bool:
|
||||
"""Delete an item from the HNSW index."""
|
||||
if doc_id not in self.id_to_label:
|
||||
if self.logger:
|
||||
self.logger.logger.warning(f"Document {doc_id} not found in index")
|
||||
return False
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Get label and mark for deletion
|
||||
label = self.id_to_label[doc_id]
|
||||
|
||||
try:
|
||||
# Mark as deleted in HNSW (soft delete)
|
||||
self.index.mark_deleted(label)
|
||||
|
||||
# Update mappings
|
||||
del self.id_to_label[doc_id]
|
||||
del self.label_to_id[label]
|
||||
|
||||
# Add label back to available labels for reuse
|
||||
self.available_labels.append(label)
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f"✅ Deleted document from HNSW index in {time_taken:.4f}s")
|
||||
self.logger.logger.debug(f" - Document ID: {doc_id}")
|
||||
self.logger.logger.debug(f" - Internal label: {label} (marked for reuse)")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
if self.logger:
|
||||
self.logger.logger.error(f"Error deleting document: {e}")
|
||||
return False
|
||||
|
||||
def search(self, query_vector: np.ndarray, top_k: int) -> Tuple[List[str], List[float]]:
|
||||
"""Search for top-k similar items in the HNSW index."""
|
||||
start_time = time.time()
|
||||
|
||||
# Ensure we don't request more items than we have
|
||||
actual_k = min(top_k, len(self.label_to_id))
|
||||
|
||||
if actual_k == 0:
|
||||
if self.logger:
|
||||
self.logger.logger.warning("Index is empty, returning no results")
|
||||
return [], []
|
||||
|
||||
# Search in index
|
||||
labels, distances = self.index.knn_query(query_vector.reshape(1, -1), k=actual_k)
|
||||
|
||||
# Convert labels to document IDs
|
||||
doc_ids = []
|
||||
valid_distances = []
|
||||
for label, distance in zip(labels[0], distances[0]):
|
||||
if label in self.label_to_id:
|
||||
doc_ids.append(self.label_to_id[label])
|
||||
valid_distances.append(float(distance))
|
||||
|
||||
if self.logger:
|
||||
time_taken = time.time() - start_time
|
||||
self.logger.logger.debug(f"⚡ HNSW search completed in {time_taken:.4f}s")
|
||||
self.logger.logger.debug(f" Retrieved {len(doc_ids)} results")
|
||||
self.logger.logger.debug(f" Search ef parameter: {self.ef_search}")
|
||||
|
||||
return doc_ids, valid_distances
|
||||
|
||||
def get_size(self) -> int:
|
||||
"""Get the current number of items in the index."""
|
||||
return len(self.id_to_label)
|
||||
|
||||
def rebuild_index(self) -> None:
|
||||
"""HNSW doesn't require explicit rebuild."""
|
||||
if self.logger:
|
||||
self.logger.logger.debug("HNSW index doesn't require explicit rebuild")
|
||||
pass
|
||||
@@ -0,0 +1,157 @@
|
||||
"""Educational logging configuration with extensive debug information."""
|
||||
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from typing import Optional
|
||||
import colorlog
|
||||
from functools import wraps
|
||||
|
||||
|
||||
def setup_logger(name: str = "vector_search", level: str = "DEBUG") -> logging.Logger:
|
||||
"""
|
||||
Set up a colorful and informative logger for educational purposes.
|
||||
|
||||
Args:
|
||||
name: Logger name
|
||||
level: Logging level (DEBUG, INFO, WARNING, ERROR)
|
||||
|
||||
Returns:
|
||||
Configured logger instance
|
||||
"""
|
||||
# Create logger
|
||||
logger = logging.getLogger(name)
|
||||
logger.setLevel(getattr(logging, level))
|
||||
|
||||
# Clear existing handlers
|
||||
logger.handlers = []
|
||||
|
||||
# Create console handler with colors
|
||||
console_handler = colorlog.StreamHandler(sys.stdout)
|
||||
console_handler.setLevel(getattr(logging, level))
|
||||
|
||||
# Create detailed formatter for educational purposes
|
||||
log_format = (
|
||||
"%(log_color)s%(asctime)s - %(name)s - [%(levelname)s] - "
|
||||
"%(filename)s:%(lineno)d - %(funcName)s() - %(message)s%(reset)s"
|
||||
)
|
||||
|
||||
formatter = colorlog.ColoredFormatter(
|
||||
log_format,
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
reset=True,
|
||||
log_colors={
|
||||
'DEBUG': 'cyan',
|
||||
'INFO': 'green',
|
||||
'WARNING': 'yellow',
|
||||
'ERROR': 'red',
|
||||
'CRITICAL': 'red,bg_white',
|
||||
}
|
||||
)
|
||||
|
||||
console_handler.setFormatter(formatter)
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
return logger
|
||||
|
||||
|
||||
def log_execution_time(logger: Optional[logging.Logger] = None):
|
||||
"""
|
||||
Decorator to log function execution time for educational purposes.
|
||||
|
||||
Args:
|
||||
logger: Logger instance to use
|
||||
"""
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
nonlocal logger
|
||||
if logger is None:
|
||||
logger = logging.getLogger("vector_search")
|
||||
|
||||
logger.debug(f"Starting execution of {func.__name__}")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
result = func(*args, **kwargs)
|
||||
execution_time = time.time() - start_time
|
||||
logger.info(
|
||||
f"✅ {func.__name__} completed successfully in {execution_time:.4f} seconds"
|
||||
)
|
||||
return result
|
||||
except Exception as e:
|
||||
execution_time = time.time() - start_time
|
||||
logger.error(
|
||||
f"❌ {func.__name__} failed after {execution_time:.4f} seconds: {str(e)}"
|
||||
)
|
||||
raise
|
||||
|
||||
return wrapper
|
||||
return decorator
|
||||
|
||||
|
||||
class VectorSearchLogger:
|
||||
"""Educational logger for vector search operations with detailed debugging."""
|
||||
|
||||
def __init__(self, logger: logging.Logger, show_embeddings: bool = False):
|
||||
self.logger = logger
|
||||
self.show_embeddings = show_embeddings
|
||||
|
||||
def log_indexing_start(self, doc_id: str, text: str):
|
||||
"""Log the start of document indexing."""
|
||||
self.logger.debug("=" * 80)
|
||||
self.logger.info(f"📝 Starting INDEXING operation")
|
||||
self.logger.debug(f"Document ID: {doc_id}")
|
||||
self.logger.debug(f"Text length: {len(text)} characters")
|
||||
self.logger.debug(f"Text preview: {text[:100]}..." if len(text) > 100 else f"Text: {text}")
|
||||
|
||||
def log_embedding_generation(self, text: str, embedding_shape: tuple, time_taken: float):
|
||||
"""Log embedding generation details."""
|
||||
self.logger.debug(f"🧮 Generating embeddings using BGE-M3 model")
|
||||
self.logger.debug(f"Input text length: {len(text)} characters")
|
||||
self.logger.debug(f"Embedding shape: {embedding_shape}")
|
||||
self.logger.debug(f"Embedding generation time: {time_taken:.4f} seconds")
|
||||
|
||||
def log_embedding_vector(self, embedding, sample_size: int = 10):
|
||||
"""Log embedding vector details for educational purposes."""
|
||||
if self.show_embeddings:
|
||||
self.logger.debug(f"Embedding vector (first {sample_size} dimensions): {embedding[:sample_size]}")
|
||||
self.logger.debug(f"Embedding statistics - Min: {embedding.min():.6f}, Max: {embedding.max():.6f}, Mean: {embedding.mean():.6f}")
|
||||
|
||||
def log_index_update(self, index_type: str, doc_id: str, current_size: int):
|
||||
"""Log index update operations."""
|
||||
self.logger.info(f"📊 Updating {index_type.upper()} index")
|
||||
self.logger.debug(f"Adding document {doc_id} to index")
|
||||
self.logger.debug(f"Current index size: {current_size} documents")
|
||||
|
||||
def log_search_start(self, query: str, top_k: int):
|
||||
"""Log the start of search operation."""
|
||||
self.logger.debug("=" * 80)
|
||||
self.logger.info(f"🔍 Starting SEARCH operation")
|
||||
self.logger.debug(f"Query: {query}")
|
||||
self.logger.debug(f"Retrieving top {top_k} results")
|
||||
|
||||
def log_search_results(self, results: list, distances: list, time_taken: float):
|
||||
"""Log search results with detailed information."""
|
||||
self.logger.info(f"✨ Search completed in {time_taken:.4f} seconds")
|
||||
self.logger.debug(f"Found {len(results)} matching documents")
|
||||
|
||||
for i, (doc_id, distance) in enumerate(zip(results, distances), 1):
|
||||
self.logger.debug(f" Rank {i}: Document {doc_id} (distance: {distance:.6f})")
|
||||
|
||||
def log_deletion(self, doc_id: str):
|
||||
"""Log document deletion."""
|
||||
self.logger.debug("=" * 80)
|
||||
self.logger.info(f"🗑️ Starting DELETE operation")
|
||||
self.logger.debug(f"Deleting document: {doc_id}")
|
||||
|
||||
def log_error(self, operation: str, error: Exception):
|
||||
"""Log errors with context."""
|
||||
self.logger.error(f"❌ Error during {operation}: {type(error).__name__}: {str(error)}")
|
||||
self.logger.debug(f"Full error details:", exc_info=True)
|
||||
|
||||
def log_index_build(self, index_type: str, num_documents: int, parameters: dict):
|
||||
"""Log index building process."""
|
||||
self.logger.info(f"🏗️ Building {index_type.upper()} index")
|
||||
self.logger.debug(f"Number of documents: {num_documents}")
|
||||
self.logger.debug(f"Index parameters: {parameters}")
|
||||
@@ -0,0 +1,436 @@
|
||||
"""Main FastAPI application for vector similarity search service."""
|
||||
|
||||
import time
|
||||
import argparse
|
||||
from typing import List, Optional, Dict, Any
|
||||
from contextlib import asynccontextmanager
|
||||
import uvicorn
|
||||
from fastapi import FastAPI, HTTPException, Query
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel, Field
|
||||
import numpy as np
|
||||
|
||||
from config import ServiceConfig, IndexType
|
||||
from logger import setup_logger, VectorSearchLogger
|
||||
from embedding_service import EmbeddingService
|
||||
from indexing import AnnoyIndex, HNSWIndex, VectorIndex
|
||||
from document_store import DocumentStore
|
||||
|
||||
|
||||
# Request/Response models
|
||||
class IndexRequest(BaseModel):
|
||||
"""Request model for indexing documents."""
|
||||
text: str = Field(..., description="Text content to index")
|
||||
doc_id: Optional[str] = Field(None, description="Optional document ID")
|
||||
metadata: Optional[Dict[str, Any]] = Field(default_factory=dict, description="Optional metadata")
|
||||
|
||||
|
||||
class SearchRequest(BaseModel):
|
||||
"""Request model for searching documents."""
|
||||
query: str = Field(..., description="Search query text")
|
||||
top_k: int = Field(default=10, ge=1, le=100, description="Number of results to return")
|
||||
return_documents: bool = Field(default=True, description="Whether to return full documents")
|
||||
|
||||
|
||||
class DeleteRequest(BaseModel):
|
||||
"""Request model for deleting documents."""
|
||||
doc_id: str = Field(..., description="Document ID to delete")
|
||||
|
||||
|
||||
class SearchResult(BaseModel):
|
||||
"""Search result model."""
|
||||
doc_id: str
|
||||
score: float
|
||||
text: Optional[str] = None
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
rank: int
|
||||
|
||||
|
||||
class IndexResponse(BaseModel):
|
||||
"""Response model for indexing operations."""
|
||||
success: bool
|
||||
doc_id: str
|
||||
message: str
|
||||
index_size: int
|
||||
|
||||
|
||||
class DeleteResponse(BaseModel):
|
||||
"""Response model for deletion operations."""
|
||||
success: bool
|
||||
message: str
|
||||
index_size: int
|
||||
|
||||
|
||||
class SearchResponse(BaseModel):
|
||||
"""Response model for search operations."""
|
||||
success: bool
|
||||
query: str
|
||||
results: List[SearchResult]
|
||||
total_results: int
|
||||
search_time_ms: float
|
||||
|
||||
|
||||
class StatsResponse(BaseModel):
|
||||
"""Response model for service statistics."""
|
||||
index_type: str
|
||||
index_size: int
|
||||
document_count: int
|
||||
embedding_dimension: int
|
||||
model_name: str
|
||||
|
||||
|
||||
# Global instances
|
||||
config: ServiceConfig = None
|
||||
logger = None
|
||||
vec_logger: VectorSearchLogger = None
|
||||
embedding_service: EmbeddingService = None
|
||||
vector_index: VectorIndex = None
|
||||
document_store: DocumentStore = None
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Manage application lifecycle."""
|
||||
# Startup
|
||||
global config, logger, vec_logger, embedding_service, vector_index, document_store
|
||||
|
||||
logger.info("=" * 80)
|
||||
logger.info("🚀 Starting Vector Similarity Search Service")
|
||||
logger.info("=" * 80)
|
||||
|
||||
# Initialize embedding service
|
||||
logger.info("Initializing BGE-M3 embedding service...")
|
||||
embedding_service = EmbeddingService(
|
||||
model_name=config.model_name,
|
||||
use_fp16=config.use_fp16,
|
||||
max_seq_length=config.max_seq_length,
|
||||
logger=vec_logger
|
||||
)
|
||||
|
||||
# Initialize vector index based on configuration
|
||||
embedding_dim = embedding_service.get_embedding_dimension()
|
||||
logger.info(f"Initializing {config.index_type.value.upper()} vector index...")
|
||||
|
||||
if config.index_type == IndexType.ANNOY:
|
||||
vector_index = AnnoyIndex(
|
||||
dimension=embedding_dim,
|
||||
n_trees=config.annoy_n_trees,
|
||||
metric=config.annoy_metric,
|
||||
logger=vec_logger
|
||||
)
|
||||
else: # HNSW
|
||||
vector_index = HNSWIndex(
|
||||
dimension=embedding_dim,
|
||||
max_elements=config.max_documents,
|
||||
ef_construction=config.hnsw_ef_construction,
|
||||
M=config.hnsw_M,
|
||||
ef_search=config.hnsw_ef_search,
|
||||
space=config.hnsw_space,
|
||||
logger=vec_logger
|
||||
)
|
||||
|
||||
# Initialize document store
|
||||
logger.info("Initializing document store...")
|
||||
document_store = DocumentStore(logger=vec_logger)
|
||||
|
||||
logger.info("=" * 80)
|
||||
logger.info("✅ Service initialized successfully!")
|
||||
logger.info(f"📍 API available at http://{config.host}:{config.port}")
|
||||
logger.info(f"📚 Docs available at http://{config.host}:{config.port}/docs")
|
||||
logger.info("=" * 80)
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
logger.info("Shutting down service...")
|
||||
|
||||
|
||||
# Create FastAPI app
|
||||
app = FastAPI(
|
||||
title="Vector Similarity Search Service",
|
||||
description="Educational service for vector similarity search using BGE-M3 embeddings with ANNOY/HNSW indexing",
|
||||
version="1.0.0",
|
||||
lifespan=lifespan
|
||||
)
|
||||
|
||||
# Add CORS middleware
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
|
||||
@app.get("/", response_model=Dict[str, str])
|
||||
async def root():
|
||||
"""Root endpoint."""
|
||||
return {
|
||||
"service": "Vector Similarity Search",
|
||||
"status": "running",
|
||||
"index_type": config.index_type.value,
|
||||
"model": config.model_name
|
||||
}
|
||||
|
||||
|
||||
@app.post("/index", response_model=IndexResponse)
|
||||
async def index_document(request: IndexRequest):
|
||||
"""
|
||||
Index a new document.
|
||||
|
||||
This endpoint:
|
||||
1. Generates embeddings using BGE-M3
|
||||
2. Adds the document to the document store
|
||||
3. Adds the embedding to the vector index
|
||||
"""
|
||||
try:
|
||||
vec_logger.log_indexing_start(request.doc_id or "auto-generated", request.text)
|
||||
|
||||
# Generate embedding
|
||||
start_time = time.time()
|
||||
embedding_result = embedding_service.encode_text(request.text)
|
||||
embedding = embedding_result['dense']
|
||||
embedding_time = time.time() - start_time
|
||||
|
||||
vec_logger.log_embedding_generation(
|
||||
request.text,
|
||||
embedding.shape,
|
||||
embedding_time
|
||||
)
|
||||
|
||||
# Store document
|
||||
doc_id = document_store.add_document(
|
||||
text=request.text,
|
||||
doc_id=request.doc_id,
|
||||
metadata=request.metadata
|
||||
)
|
||||
|
||||
# Update document with embedding
|
||||
document_store.update_document_embedding(doc_id, embedding.tolist())
|
||||
|
||||
# Add to vector index
|
||||
vector_index.add_item(doc_id, embedding)
|
||||
vec_logger.log_index_update(
|
||||
config.index_type.value,
|
||||
doc_id,
|
||||
vector_index.get_size()
|
||||
)
|
||||
|
||||
# Rebuild index if necessary (for ANNOY)
|
||||
if config.index_type == IndexType.ANNOY:
|
||||
vector_index.rebuild_index()
|
||||
|
||||
return IndexResponse(
|
||||
success=True,
|
||||
doc_id=doc_id,
|
||||
message=f"Document indexed successfully using {config.index_type.value.upper()}",
|
||||
index_size=vector_index.get_size()
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
vec_logger.log_error("indexing", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.post("/search", response_model=SearchResponse)
|
||||
async def search_documents(request: SearchRequest):
|
||||
"""
|
||||
Search for similar documents.
|
||||
|
||||
This endpoint:
|
||||
1. Generates query embedding using BGE-M3
|
||||
2. Searches the vector index for similar documents
|
||||
3. Returns ranked results with scores
|
||||
"""
|
||||
try:
|
||||
vec_logger.log_search_start(request.query, request.top_k)
|
||||
|
||||
# Generate query embedding
|
||||
start_time = time.time()
|
||||
embedding_result = embedding_service.encode_text(request.query)
|
||||
query_embedding = embedding_result['dense']
|
||||
embedding_time = time.time() - start_time
|
||||
|
||||
logger.debug(f"Query embedding generated in {embedding_time:.4f}s")
|
||||
vec_logger.log_embedding_vector(query_embedding, sample_size=10)
|
||||
|
||||
# Search in index
|
||||
search_start = time.time()
|
||||
doc_ids, distances = vector_index.search(query_embedding, request.top_k)
|
||||
search_time = time.time() - search_start
|
||||
|
||||
vec_logger.log_search_results(doc_ids, distances, search_time)
|
||||
|
||||
# Prepare results
|
||||
results = []
|
||||
if request.return_documents:
|
||||
documents = document_store.get_documents_by_ids(doc_ids)
|
||||
doc_map = {doc.id: doc for doc in documents}
|
||||
|
||||
for rank, (doc_id, distance) in enumerate(zip(doc_ids, distances), 1):
|
||||
doc = doc_map.get(doc_id)
|
||||
if doc:
|
||||
results.append(SearchResult(
|
||||
doc_id=doc_id,
|
||||
score=float(1.0 / (1.0 + distance)), # Convert distance to similarity score
|
||||
text=doc.text,
|
||||
metadata=doc.metadata,
|
||||
rank=rank
|
||||
))
|
||||
else:
|
||||
for rank, (doc_id, distance) in enumerate(zip(doc_ids, distances), 1):
|
||||
results.append(SearchResult(
|
||||
doc_id=doc_id,
|
||||
score=float(1.0 / (1.0 + distance)),
|
||||
rank=rank
|
||||
))
|
||||
|
||||
total_time_ms = (time.time() - start_time) * 1000
|
||||
|
||||
return SearchResponse(
|
||||
success=True,
|
||||
query=request.query,
|
||||
results=results,
|
||||
total_results=len(results),
|
||||
search_time_ms=total_time_ms
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
vec_logger.log_error("search", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.delete("/index", response_model=DeleteResponse)
|
||||
async def delete_document(request: DeleteRequest):
|
||||
"""
|
||||
Delete a document from the index.
|
||||
|
||||
This endpoint:
|
||||
1. Removes the document from the document store
|
||||
2. Removes the embedding from the vector index
|
||||
"""
|
||||
try:
|
||||
vec_logger.log_deletion(request.doc_id)
|
||||
|
||||
# Delete from document store
|
||||
doc_deleted = document_store.delete_document(request.doc_id)
|
||||
|
||||
if not doc_deleted:
|
||||
return DeleteResponse(
|
||||
success=False,
|
||||
message=f"Document {request.doc_id} not found",
|
||||
index_size=vector_index.get_size()
|
||||
)
|
||||
|
||||
# Delete from vector index
|
||||
index_deleted = vector_index.delete_item(request.doc_id)
|
||||
|
||||
if index_deleted:
|
||||
return DeleteResponse(
|
||||
success=True,
|
||||
message=f"Document {request.doc_id} deleted successfully",
|
||||
index_size=vector_index.get_size()
|
||||
)
|
||||
else:
|
||||
return DeleteResponse(
|
||||
success=False,
|
||||
message=f"Document {request.doc_id} deleted from store but not from index",
|
||||
index_size=vector_index.get_size()
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
vec_logger.log_error("deletion", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.get("/stats", response_model=StatsResponse)
|
||||
async def get_stats():
|
||||
"""Get service statistics."""
|
||||
return StatsResponse(
|
||||
index_type=config.index_type.value,
|
||||
index_size=vector_index.get_size(),
|
||||
document_count=document_store.get_size(),
|
||||
embedding_dimension=embedding_service.get_embedding_dimension(),
|
||||
model_name=config.model_name
|
||||
)
|
||||
|
||||
|
||||
@app.get("/documents", response_model=List[Dict[str, Any]])
|
||||
async def list_documents(limit: int = Query(default=10, ge=1, le=100)):
|
||||
"""List documents in the store."""
|
||||
docs = document_store.list_documents(limit=limit)
|
||||
return [
|
||||
{
|
||||
"id": doc.id,
|
||||
"text": doc.text[:200] + "..." if len(doc.text) > 200 else doc.text,
|
||||
"metadata": doc.metadata,
|
||||
"created_at": doc.created_at.isoformat()
|
||||
}
|
||||
for doc in docs
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
global config, logger, vec_logger
|
||||
|
||||
# Parse command line arguments
|
||||
parser = argparse.ArgumentParser(description="Vector Similarity Search Service")
|
||||
parser.add_argument(
|
||||
"--index-type",
|
||||
type=str,
|
||||
choices=["annoy", "hnsw"],
|
||||
default="hnsw",
|
||||
help="Type of index to use (default: hnsw)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--host",
|
||||
type=str,
|
||||
default="0.0.0.0",
|
||||
help="Host to bind to (default: 0.0.0.0)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--port",
|
||||
type=int,
|
||||
default=4240,
|
||||
help="Port to bind to (default: 4240)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--debug",
|
||||
action="store_true",
|
||||
help="Enable debug mode"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--show-embeddings",
|
||||
action="store_true",
|
||||
help="Show embedding vectors in logs"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Create configuration
|
||||
config = ServiceConfig(
|
||||
index_type=IndexType(args.index_type),
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
debug=args.debug,
|
||||
show_embeddings=args.show_embeddings
|
||||
)
|
||||
|
||||
# Setup logging
|
||||
logger = setup_logger("vector_search", config.log_level)
|
||||
vec_logger = VectorSearchLogger(logger, config.show_embeddings)
|
||||
|
||||
# Run the service
|
||||
uvicorn.run(
|
||||
app,
|
||||
host=config.host,
|
||||
port=config.port,
|
||||
log_level=config.log_level.lower(),
|
||||
reload=False
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Quick demo script to showcase the vector similarity search service."""
|
||||
|
||||
import time
|
||||
import sys
|
||||
|
||||
|
||||
def print_section(title):
|
||||
"""Print a formatted section header."""
|
||||
print("\n" + "=" * 60)
|
||||
print(f" {title}")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def main():
|
||||
"""Run a quick demo of the service."""
|
||||
|
||||
print_section("Vector Similarity Search - Quick Demo")
|
||||
|
||||
print("""
|
||||
This educational service demonstrates vector similarity search
|
||||
using BGE-M3 embeddings with ANNOY/HNSW indexing.
|
||||
|
||||
EDUCATIONAL CONCEPTS DEMONSTRATED:
|
||||
1. Text → Vector embedding generation
|
||||
2. Approximate nearest neighbor search
|
||||
3. Cosine similarity for semantic matching
|
||||
4. Trade-offs between index types (ANNOY vs HNSW)
|
||||
""")
|
||||
|
||||
print("\n📚 STEP 1: Start the service")
|
||||
print("-" * 40)
|
||||
print("\nOption A - Using HNSW (high precision):")
|
||||
print(" python main.py --index-type hnsw --debug")
|
||||
|
||||
print("\nOption B - Using ANNOY (fast, memory-efficient):")
|
||||
print(" python main.py --index-type annoy --debug")
|
||||
|
||||
print("\nOption C - Using the startup script:")
|
||||
print(" ./start_service.sh hnsw 8000 true")
|
||||
|
||||
print("\n📝 STEP 2: Index some documents")
|
||||
print("-" * 40)
|
||||
print("""
|
||||
Example using curl:
|
||||
|
||||
curl -X POST http://localhost:8000/index \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{
|
||||
"text": "Machine learning is a subset of AI that enables systems to learn from data.",
|
||||
"metadata": {"category": "AI", "level": "beginner"}
|
||||
}'
|
||||
""")
|
||||
|
||||
print("\n🔍 STEP 3: Search for similar documents")
|
||||
print("-" * 40)
|
||||
print("""
|
||||
Example search:
|
||||
|
||||
curl -X POST http://localhost:8000/search \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{
|
||||
"query": "What is deep learning?",
|
||||
"top_k": 5
|
||||
}'
|
||||
""")
|
||||
|
||||
print("\n🎯 STEP 4: Run the test client")
|
||||
print("-" * 40)
|
||||
print("""
|
||||
The test client will:
|
||||
- Index 10 sample documents about AI, programming, and DevOps
|
||||
- Perform 5 different similarity searches
|
||||
- Demonstrate document deletion
|
||||
- Show performance metrics
|
||||
|
||||
Run it with:
|
||||
python test_client.py
|
||||
|
||||
For performance testing (100 documents):
|
||||
python test_client.py --performance
|
||||
""")
|
||||
|
||||
print("\n📊 KEY LEARNING POINTS")
|
||||
print("-" * 40)
|
||||
print("""
|
||||
1. EMBEDDINGS: BGE-M3 converts text → 1024-dimensional vectors
|
||||
- Semantic meaning is captured in vector space
|
||||
- Similar texts have similar vectors
|
||||
|
||||
2. INDEXING: Two algorithms for efficient similarity search
|
||||
- ANNOY: Tree-based, fast but approximate
|
||||
- HNSW: Graph-based, slower but more accurate
|
||||
|
||||
3. SIMILARITY: Cosine distance measures semantic similarity
|
||||
- Score close to 1.0 = very similar
|
||||
- Score close to 0.0 = not similar
|
||||
|
||||
4. TRADE-OFFS:
|
||||
- Speed vs Accuracy (ANNOY vs HNSW)
|
||||
- Memory vs Performance (index parameters)
|
||||
- Build time vs Search time
|
||||
""")
|
||||
|
||||
print("\n🔗 USEFUL ENDPOINTS")
|
||||
print("-" * 40)
|
||||
print("""
|
||||
- API Documentation: http://localhost:8000/docs
|
||||
- Service Status: http://localhost:8000/
|
||||
- Statistics: http://localhost:8000/stats
|
||||
- List Documents: http://localhost:8000/documents
|
||||
""")
|
||||
|
||||
print("\n💡 EXPERIMENT IDEAS")
|
||||
print("-" * 40)
|
||||
print("""
|
||||
1. Compare ANNOY vs HNSW accuracy on same queries
|
||||
2. Measure indexing time for different document sizes
|
||||
3. Test multilingual search (BGE-M3 supports 100+ languages)
|
||||
4. Analyze how different parameters affect performance
|
||||
5. Try searching with synonyms and paraphrases
|
||||
""")
|
||||
|
||||
print_section("Ready to Start!")
|
||||
print("\nNext steps:")
|
||||
print("1. Start the service: python main.py --debug")
|
||||
print("2. Run the demo: python test_client.py")
|
||||
print("3. Explore the API: http://localhost:8000/docs")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,20 @@
|
||||
# Core dependencies for vector similarity search service
|
||||
fastapi==0.110.0
|
||||
uvicorn==0.27.0
|
||||
pydantic==2.6.0
|
||||
python-multipart==0.0.9
|
||||
|
||||
# BGE-M3 model and embeddings
|
||||
FlagEmbedding==1.2.11
|
||||
torch>=2.0.0
|
||||
transformers>=4.36.0
|
||||
sentencepiece>=0.1.99
|
||||
|
||||
# Vector indexing libraries
|
||||
annoy==1.17.3
|
||||
hnswlib==0.8.0
|
||||
numpy>=1.24.0
|
||||
|
||||
# Utilities
|
||||
python-dotenv==1.0.0
|
||||
colorlog==6.8.0
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Vector Similarity Search Service Startup Script
|
||||
|
||||
echo "========================================"
|
||||
echo "Vector Similarity Search Service"
|
||||
echo "========================================"
|
||||
|
||||
# Default values
|
||||
INDEX_TYPE=${1:-hnsw}
|
||||
PORT=${2:-8000}
|
||||
DEBUG=${3:-true}
|
||||
|
||||
echo ""
|
||||
echo "Configuration:"
|
||||
echo " Index Type: $INDEX_TYPE"
|
||||
echo " Port: $PORT"
|
||||
echo " Debug: $DEBUG"
|
||||
echo ""
|
||||
|
||||
# Check if virtual environment exists
|
||||
if [ -d "venv" ]; then
|
||||
echo "Activating virtual environment..."
|
||||
source venv/bin/activate
|
||||
fi
|
||||
|
||||
# Install dependencies if needed
|
||||
echo "Checking dependencies..."
|
||||
pip list | grep -q "FlagEmbedding" || {
|
||||
echo "Installing dependencies..."
|
||||
pip install -r requirements.txt
|
||||
}
|
||||
|
||||
# Start the service
|
||||
echo ""
|
||||
echo "Starting service..."
|
||||
echo "========================================"
|
||||
|
||||
if [ "$DEBUG" = "true" ]; then
|
||||
python main.py --index-type $INDEX_TYPE --port $PORT --debug
|
||||
else
|
||||
python main.py --index-type $INDEX_TYPE --port $PORT
|
||||
fi
|
||||
@@ -0,0 +1,252 @@
|
||||
"""Test client for the vector similarity search service."""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
from typing import List, Dict, Any
|
||||
|
||||
|
||||
class VectorSearchClient:
|
||||
"""Client for testing the vector search service."""
|
||||
|
||||
def __init__(self, base_url: str = "http://localhost:8000"):
|
||||
"""Initialize the client."""
|
||||
self.base_url = base_url
|
||||
|
||||
def index_document(self, text: str, doc_id: str = None, metadata: Dict = None) -> Dict:
|
||||
"""Index a document."""
|
||||
response = requests.post(
|
||||
f"{self.base_url}/index",
|
||||
json={
|
||||
"text": text,
|
||||
"doc_id": doc_id,
|
||||
"metadata": metadata or {}
|
||||
}, timeout=30
|
||||
)
|
||||
return response.json()
|
||||
|
||||
def search(self, query: str, top_k: int = 5, return_documents: bool = True) -> Dict:
|
||||
"""Search for similar documents."""
|
||||
response = requests.post(
|
||||
f"{self.base_url}/search",
|
||||
json={
|
||||
"query": query,
|
||||
"top_k": top_k,
|
||||
"return_documents": return_documents
|
||||
}, timeout=30
|
||||
)
|
||||
return response.json()
|
||||
|
||||
def delete_document(self, doc_id: str) -> Dict:
|
||||
"""Delete a document."""
|
||||
response = requests.delete(
|
||||
f"{self.base_url}/index",
|
||||
json={"doc_id": doc_id}, timeout=30
|
||||
)
|
||||
return response.json()
|
||||
|
||||
def get_stats(self) -> Dict:
|
||||
"""Get service statistics."""
|
||||
response = requests.get(f"{self.base_url}/stats", timeout=30)
|
||||
return response.json()
|
||||
|
||||
def list_documents(self, limit: int = 10) -> List[Dict]:
|
||||
"""List documents in the store."""
|
||||
response = requests.get(f"{self.base_url}/documents", params={"limit": limit}, timeout=30)
|
||||
return response.json()
|
||||
|
||||
|
||||
def run_demo():
|
||||
"""Run a comprehensive demo of the vector search service."""
|
||||
print("=" * 80)
|
||||
print("🚀 Vector Similarity Search Service - Demo Client")
|
||||
print("=" * 80)
|
||||
|
||||
# Initialize client
|
||||
client = VectorSearchClient()
|
||||
|
||||
# Check service status
|
||||
print("\n📊 Checking service status...")
|
||||
try:
|
||||
response = requests.get("http://localhost:8000/", timeout=30)
|
||||
status = response.json()
|
||||
print(f"✅ Service is running")
|
||||
print(f" - Index type: {status['index_type']}")
|
||||
print(f" - Model: {status['model']}")
|
||||
except Exception as e:
|
||||
print(f"❌ Service is not running: {e}")
|
||||
print("Please start the service first with: python main.py")
|
||||
return
|
||||
|
||||
# Sample documents
|
||||
documents = [
|
||||
{
|
||||
"text": "Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed.",
|
||||
"metadata": {"category": "AI", "topic": "machine_learning"}
|
||||
},
|
||||
{
|
||||
"text": "Deep learning is a type of machine learning based on artificial neural networks with multiple layers that progressively extract higher-level features from raw input.",
|
||||
"metadata": {"category": "AI", "topic": "deep_learning"}
|
||||
},
|
||||
{
|
||||
"text": "Natural language processing (NLP) is a branch of AI that helps computers understand, interpret and manipulate human language.",
|
||||
"metadata": {"category": "AI", "topic": "nlp"}
|
||||
},
|
||||
{
|
||||
"text": "Computer vision enables machines to interpret and understand visual information from the world, similar to how humans use their eyes and brains.",
|
||||
"metadata": {"category": "AI", "topic": "computer_vision"}
|
||||
},
|
||||
{
|
||||
"text": "Reinforcement learning is an area of machine learning where an agent learns to make decisions by taking actions in an environment to maximize cumulative reward.",
|
||||
"metadata": {"category": "AI", "topic": "reinforcement_learning"}
|
||||
},
|
||||
{
|
||||
"text": "Python is a high-level programming language known for its simplicity and readability, widely used in data science and web development.",
|
||||
"metadata": {"category": "Programming", "topic": "python"}
|
||||
},
|
||||
{
|
||||
"text": "JavaScript is a versatile programming language primarily used for creating interactive web applications and running code in browsers.",
|
||||
"metadata": {"category": "Programming", "topic": "javascript"}
|
||||
},
|
||||
{
|
||||
"text": "Docker is a platform that uses containerization to package applications with their dependencies, ensuring consistency across different environments.",
|
||||
"metadata": {"category": "DevOps", "topic": "containerization"}
|
||||
},
|
||||
{
|
||||
"text": "Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications.",
|
||||
"metadata": {"category": "DevOps", "topic": "orchestration"}
|
||||
},
|
||||
{
|
||||
"text": "The transformer architecture revolutionized NLP by introducing self-attention mechanisms that process sequences in parallel rather than sequentially.",
|
||||
"metadata": {"category": "AI", "topic": "transformers"}
|
||||
}
|
||||
]
|
||||
|
||||
# Index documents
|
||||
print("\n📝 Indexing documents...")
|
||||
doc_ids = []
|
||||
for i, doc in enumerate(documents, 1):
|
||||
print(f" [{i}/{len(documents)}] Indexing: {doc['text'][:50]}...")
|
||||
result = client.index_document(
|
||||
text=doc["text"],
|
||||
metadata=doc["metadata"]
|
||||
)
|
||||
doc_ids.append(result["doc_id"])
|
||||
time.sleep(0.1) # Small delay for demonstration
|
||||
|
||||
print(f"\n✅ Indexed {len(documents)} documents")
|
||||
|
||||
# Show statistics
|
||||
print("\n📊 Current statistics:")
|
||||
stats = client.get_stats()
|
||||
for key, value in stats.items():
|
||||
print(f" - {key}: {value}")
|
||||
|
||||
# Perform searches
|
||||
queries = [
|
||||
"What is deep learning and neural networks?",
|
||||
"How to deploy applications with containers?",
|
||||
"Programming languages for web development",
|
||||
"Learning from environment and rewards",
|
||||
"Understanding human language with computers"
|
||||
]
|
||||
|
||||
print("\n🔍 Performing searches...")
|
||||
for query in queries:
|
||||
print(f"\n Query: '{query}'")
|
||||
print(" " + "-" * 60)
|
||||
|
||||
result = client.search(query, top_k=3)
|
||||
|
||||
if result["success"]:
|
||||
print(f" Found {result['total_results']} results in {result['search_time_ms']:.2f}ms:")
|
||||
for res in result["results"]:
|
||||
print(f"\n Rank {res['rank']}: (Score: {res['score']:.4f})")
|
||||
print(f" Text: {res['text'][:100]}...")
|
||||
if res.get("metadata"):
|
||||
print(f" Metadata: {res['metadata']}")
|
||||
else:
|
||||
print(f" ❌ Search failed")
|
||||
|
||||
time.sleep(0.5) # Delay for readability
|
||||
|
||||
# Test deletion
|
||||
print("\n🗑️ Testing document deletion...")
|
||||
if doc_ids:
|
||||
doc_to_delete = doc_ids[0]
|
||||
print(f" Deleting document: {doc_to_delete}")
|
||||
delete_result = client.delete_document(doc_to_delete)
|
||||
print(f" Result: {delete_result['message']}")
|
||||
print(f" New index size: {delete_result['index_size']}")
|
||||
|
||||
# List remaining documents
|
||||
print("\n📋 Listing documents (first 5)...")
|
||||
docs = client.list_documents(limit=5)
|
||||
for i, doc in enumerate(docs, 1):
|
||||
print(f" {i}. ID: {doc['id'][:8]}... | Text: {doc['text'][:50]}...")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("✅ Demo completed successfully!")
|
||||
print("=" * 80)
|
||||
|
||||
|
||||
def run_performance_test():
|
||||
"""Run a performance test."""
|
||||
print("\n" + "=" * 80)
|
||||
print("⚡ Performance Test")
|
||||
print("=" * 80)
|
||||
|
||||
client = VectorSearchClient()
|
||||
|
||||
# Generate test documents
|
||||
num_docs = 100
|
||||
print(f"\n📝 Indexing {num_docs} documents for performance testing...")
|
||||
|
||||
start_time = time.time()
|
||||
for i in range(num_docs):
|
||||
text = f"This is test document number {i}. It contains various information about topic {i % 10}. " \
|
||||
f"The content is randomly generated for testing purposes. Keywords: test, document, {i}, performance."
|
||||
client.index_document(text)
|
||||
|
||||
if (i + 1) % 20 == 0:
|
||||
print(f" Indexed {i + 1}/{num_docs} documents...")
|
||||
|
||||
index_time = time.time() - start_time
|
||||
print(f"\n✅ Indexing completed in {index_time:.2f} seconds")
|
||||
print(f" Average: {index_time/num_docs*1000:.2f}ms per document")
|
||||
|
||||
# Test search performance
|
||||
print(f"\n🔍 Testing search performance with 20 queries...")
|
||||
search_times = []
|
||||
|
||||
for i in range(20):
|
||||
query = f"Find information about topic {i % 10} and document testing"
|
||||
start_time = time.time()
|
||||
result = client.search(query, top_k=10, return_documents=False)
|
||||
search_time = time.time() - start_time
|
||||
search_times.append(search_time)
|
||||
|
||||
avg_search_time = sum(search_times) / len(search_times)
|
||||
min_search_time = min(search_times)
|
||||
max_search_time = max(search_times)
|
||||
|
||||
print(f"\n📊 Search Performance Results:")
|
||||
print(f" - Average search time: {avg_search_time*1000:.2f}ms")
|
||||
print(f" - Min search time: {min_search_time*1000:.2f}ms")
|
||||
print(f" - Max search time: {max_search_time*1000:.2f}ms")
|
||||
|
||||
# Final stats
|
||||
stats = client.get_stats()
|
||||
print(f"\n📊 Final Statistics:")
|
||||
print(f" - Total documents: {stats['document_count']}")
|
||||
print(f" - Index size: {stats['index_size']}")
|
||||
print(f" - Index type: {stats['index_type']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "--performance":
|
||||
run_performance_test()
|
||||
else:
|
||||
run_demo()
|
||||
@@ -0,0 +1,34 @@
|
||||
"""
|
||||
Test suite locking out ZeroDivisionError in EmbeddingService.encode_batch
|
||||
when an empty texts list is provided.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Mock third-party dependencies before importing embedding_service
|
||||
sys.modules['FlagEmbedding'] = MagicMock()
|
||||
sys.modules['colorlog'] = MagicMock()
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
|
||||
|
||||
from embedding_service import EmbeddingService
|
||||
|
||||
|
||||
def test_encode_batch_empty_texts_logger_zero_division():
|
||||
"""
|
||||
Ensure encode_batch with an empty list of texts does not raise ZeroDivisionError
|
||||
during logging.
|
||||
"""
|
||||
service = EmbeddingService.__new__(EmbeddingService)
|
||||
mock_logger = MagicMock()
|
||||
service.logger = mock_logger
|
||||
service.model = MagicMock()
|
||||
service.model.encode.return_value = {
|
||||
'dense_vecs': MagicMock(shape=(0, 768))
|
||||
}
|
||||
|
||||
result = service.encode_batch([])
|
||||
assert result['num_texts'] == 0
|
||||
assert result['dimension'] == 768
|
||||
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"schema_version": "chapter3-evidence-v1",
|
||||
"experiment": "3-4",
|
||||
"run_id": "20260729T182946Z-3_4-2555ea60",
|
||||
"created_at": "2026-07-29T18:29:46.110155+00:00",
|
||||
"status": "passed",
|
||||
"run_dir": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/validation/runs/20260729T182946Z-3_4-2555ea60",
|
||||
"artifacts": {
|
||||
"evidence.json": "0670be3c6fe6e203dbacc926ac0a4cecc6b4902d49db2bc4205bc0d14d3b171f",
|
||||
"receipts.json": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570",
|
||||
"manifest.json": "b992300835bd2cfcd03e820164a8c0972bc2b85e06f587f0602f48cd03d3c06c"
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/indexing.py",
|
||||
"sha256": "80ca520a78f044b367edfe71ff4e4bb22fa80f00d4803831d17b2b8c14ba1f57",
|
||||
"bytes": 14977
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/benchmark.py",
|
||||
"sha256": "d5d72267ca683ec513acd422287ff6f7cd65c54c6eaad5d636b67b983dda7904",
|
||||
"bytes": 13883
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/docker_annoy_runner.py",
|
||||
"sha256": "f75ee2f5f81365143a6f8ad68c6706933f8012ca4792a643d8ae79bfc9621c70",
|
||||
"bytes": 2980
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"embedding_latency_ms": 55429.937,
|
||||
"annoy": {
|
||||
"build_ms": 13.203,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.2336004702374339,
|
||||
"p50": 0.22620847448706627,
|
||||
"p95": 0.2634833101183176
|
||||
},
|
||||
"serialized_bytes": 1396720,
|
||||
"incremental_update": {
|
||||
"items_added": 60,
|
||||
"latency_ms": 8.274,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 77.682,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.25732293259352446,
|
||||
"p50": 0.2595204859972,
|
||||
"p95": 0.2800690243020654
|
||||
},
|
||||
"serialized_bytes": 1018716,
|
||||
"incremental_update": {
|
||||
"items_added": 60,
|
||||
"latency_ms": 33.833,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": true,
|
||||
"same_vectors_and_queries": true,
|
||||
"exact_search_ground_truth": true,
|
||||
"recall_latency_build_size_measured": true,
|
||||
"incremental_behavior_measured": true,
|
||||
"both_backends_recall_at_least_0_8": true,
|
||||
"passed": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,499 @@
|
||||
{
|
||||
"schema_version": "chapter3-evidence-v1",
|
||||
"experiment": "3-4",
|
||||
"run_id": "20260729T182108Z-3_4-7353a50f",
|
||||
"provenance": {
|
||||
"captured_at": "2026-07-29T18:21:08.357431+00:00",
|
||||
"git_revision": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
|
||||
"python": "3.11.4 (main, Jul 5 2023, 08:40:20) [Clang 14.0.6 ]",
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"credential_presence": {
|
||||
"ARK_API_KEY": true,
|
||||
"MOONSHOT_API_KEY": true,
|
||||
"OPENAI_API_KEY": true,
|
||||
"GEMINI_API_KEY": true,
|
||||
"SILICONFLOW_API_KEY": true
|
||||
}
|
||||
},
|
||||
"status": "partial",
|
||||
"configuration": {
|
||||
"embedding_model": "Qwen/Qwen3-Embedding-0.6B",
|
||||
"model_revision": "97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3",
|
||||
"device": "cpu",
|
||||
"seed": 37,
|
||||
"dimension": 1024,
|
||||
"documents": 40,
|
||||
"queries": 20,
|
||||
"top_k": 3
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": true,
|
||||
"same_vectors_and_queries": true,
|
||||
"exact_search_ground_truth": true,
|
||||
"recall_latency_build_size_measured": true,
|
||||
"incremental_behavior_measured": true,
|
||||
"both_backends_recall_at_least_0_8": false,
|
||||
"passed": false
|
||||
},
|
||||
"summary": {
|
||||
"embedding_latency_ms": 9437.365,
|
||||
"annoy": {
|
||||
"build_ms": 1.214,
|
||||
"recall_at_k": 0.16666666666666666,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.027637556195259094,
|
||||
"p50": 0.026833033189177513,
|
||||
"p95": 0.030818302184343352
|
||||
},
|
||||
"serialized_bytes": 542256,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 1.247,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 0.13333333333333333
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 1.398,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.04115619231015444,
|
||||
"p50": 0.03870879299938679,
|
||||
"p95": 0.05208926741033793
|
||||
},
|
||||
"serialized_bytes": 135912,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 0.74,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"corpus": {
|
||||
"doc_ids": [
|
||||
"doc_0000",
|
||||
"doc_0001",
|
||||
"doc_0002",
|
||||
"doc_0003",
|
||||
"doc_0004",
|
||||
"doc_0005",
|
||||
"doc_0006",
|
||||
"doc_0007",
|
||||
"doc_0008",
|
||||
"doc_0009",
|
||||
"doc_0010",
|
||||
"doc_0011",
|
||||
"doc_0012",
|
||||
"doc_0013",
|
||||
"doc_0014",
|
||||
"doc_0015",
|
||||
"doc_0016",
|
||||
"doc_0017",
|
||||
"doc_0018",
|
||||
"doc_0019",
|
||||
"doc_0020",
|
||||
"doc_0021",
|
||||
"doc_0022",
|
||||
"doc_0023",
|
||||
"doc_0024",
|
||||
"doc_0025",
|
||||
"doc_0026",
|
||||
"doc_0027",
|
||||
"doc_0028",
|
||||
"doc_0029",
|
||||
"doc_0030",
|
||||
"doc_0031",
|
||||
"doc_0032",
|
||||
"doc_0033",
|
||||
"doc_0034",
|
||||
"doc_0035",
|
||||
"doc_0036",
|
||||
"doc_0037",
|
||||
"doc_0038",
|
||||
"doc_0039"
|
||||
],
|
||||
"texts": [
|
||||
"Topic: vector search. Approximate nearest-neighbor indexes accelerate semantic vector retrieval. A concise technical overview. Document revision 0000.",
|
||||
"Topic: database transactions. Database transactions use atomicity, consistency, isolation and durability. A concise technical overview. Document revision 0001.",
|
||||
"Topic: photosynthesis. Green plants turn sunlight and carbon dioxide into chemical energy. A concise technical overview. Document revision 0002.",
|
||||
"Topic: quantum entanglement. Entangled particles exhibit correlated quantum measurement outcomes. A concise technical overview. Document revision 0003.",
|
||||
"Topic: contract law. A valid contract generally requires offer acceptance and consideration. A concise technical overview. Document revision 0004.",
|
||||
"Topic: neural networks. Deep neural networks learn layered nonlinear representations from data. A concise technical overview. Document revision 0005.",
|
||||
"Topic: cybersecurity. Zero trust security continuously verifies identity and device posture. A concise technical overview. Document revision 0006.",
|
||||
"Topic: volcanoes. Volcanoes form when magma rises through fractures in the planetary crust. A concise technical overview. Document revision 0007.",
|
||||
"Topic: water cycle. Evaporation condensation precipitation and runoff form the water cycle. A concise technical overview. Document revision 0008.",
|
||||
"Topic: operating systems. An operating system schedules processes and manages memory and devices. A concise technical overview. Document revision 0009.",
|
||||
"Topic: HTTP errors. HTTP status 403 means a server understood but refused a request. A concise technical overview. Document revision 0010.",
|
||||
"Topic: machine translation. Multilingual models translate meaning between natural languages. A concise technical overview. Document revision 0011.",
|
||||
"Topic: financial risk. Portfolio diversification reduces exposure to idiosyncratic financial risk. A concise technical overview. Document revision 0012.",
|
||||
"Topic: medical imaging. Radiology systems analyze X-rays CT scans and magnetic resonance images. A concise technical overview. Document revision 0013.",
|
||||
"Topic: supply chains. Supply chain planning coordinates inventory logistics demand and suppliers. A concise technical overview. Document revision 0014.",
|
||||
"Topic: climate science. Climate models simulate long-term interactions among atmosphere ocean and land. A concise technical overview. Document revision 0015.",
|
||||
"Topic: CPU instructions. SIMD instructions apply one operation to several packed numeric values. A concise technical overview. Document revision 0016.",
|
||||
"Topic: compiler design. A compiler parses source code optimizes intermediate form and emits machine code. A concise technical overview. Document revision 0017.",
|
||||
"Topic: graph theory. Graph algorithms traverse vertices and edges to discover paths and communities. A concise technical overview. Document revision 0018.",
|
||||
"Topic: astronomy. Astronomers infer stellar properties from spectra luminosity and orbital motion. A concise technical overview. Document revision 0019.",
|
||||
"Topic: vector search. Approximate nearest-neighbor indexes accelerate semantic vector retrieval. This passage explains the central mechanism and its practical use. Document revision 0020.",
|
||||
"Topic: database transactions. Database transactions use atomicity, consistency, isolation and durability. This passage explains the central mechanism and its practical use. Document revision 0021.",
|
||||
"Topic: photosynthesis. Green plants turn sunlight and carbon dioxide into chemical energy. This passage explains the central mechanism and its practical use. Document revision 0022.",
|
||||
"Topic: quantum entanglement. Entangled particles exhibit correlated quantum measurement outcomes. This passage explains the central mechanism and its practical use. Document revision 0023.",
|
||||
"Topic: contract law. A valid contract generally requires offer acceptance and consideration. This passage explains the central mechanism and its practical use. Document revision 0024.",
|
||||
"Topic: neural networks. Deep neural networks learn layered nonlinear representations from data. This passage explains the central mechanism and its practical use. Document revision 0025.",
|
||||
"Topic: cybersecurity. Zero trust security continuously verifies identity and device posture. This passage explains the central mechanism and its practical use. Document revision 0026.",
|
||||
"Topic: volcanoes. Volcanoes form when magma rises through fractures in the planetary crust. This passage explains the central mechanism and its practical use. Document revision 0027.",
|
||||
"Topic: water cycle. Evaporation condensation precipitation and runoff form the water cycle. This passage explains the central mechanism and its practical use. Document revision 0028.",
|
||||
"Topic: operating systems. An operating system schedules processes and manages memory and devices. This passage explains the central mechanism and its practical use. Document revision 0029.",
|
||||
"Topic: HTTP errors. HTTP status 403 means a server understood but refused a request. This passage explains the central mechanism and its practical use. Document revision 0030.",
|
||||
"Topic: machine translation. Multilingual models translate meaning between natural languages. This passage explains the central mechanism and its practical use. Document revision 0031.",
|
||||
"Topic: financial risk. Portfolio diversification reduces exposure to idiosyncratic financial risk. This passage explains the central mechanism and its practical use. Document revision 0032.",
|
||||
"Topic: medical imaging. Radiology systems analyze X-rays CT scans and magnetic resonance images. This passage explains the central mechanism and its practical use. Document revision 0033.",
|
||||
"Topic: supply chains. Supply chain planning coordinates inventory logistics demand and suppliers. This passage explains the central mechanism and its practical use. Document revision 0034.",
|
||||
"Topic: climate science. Climate models simulate long-term interactions among atmosphere ocean and land. This passage explains the central mechanism and its practical use. Document revision 0035.",
|
||||
"Topic: CPU instructions. SIMD instructions apply one operation to several packed numeric values. This passage explains the central mechanism and its practical use. Document revision 0036.",
|
||||
"Topic: compiler design. A compiler parses source code optimizes intermediate form and emits machine code. This passage explains the central mechanism and its practical use. Document revision 0037.",
|
||||
"Topic: graph theory. Graph algorithms traverse vertices and edges to discover paths and communities. This passage explains the central mechanism and its practical use. Document revision 0038.",
|
||||
"Topic: astronomy. Astronomers infer stellar properties from spectra luminosity and orbital motion. This passage explains the central mechanism and its practical use. Document revision 0039."
|
||||
],
|
||||
"queries": [
|
||||
"Find technical information about vector search.",
|
||||
"Find technical information about database transactions.",
|
||||
"Find technical information about photosynthesis.",
|
||||
"Find technical information about quantum entanglement.",
|
||||
"Find technical information about contract law.",
|
||||
"Find technical information about neural networks.",
|
||||
"Find technical information about cybersecurity.",
|
||||
"Find technical information about volcanoes.",
|
||||
"Find technical information about water cycle.",
|
||||
"Find technical information about operating systems.",
|
||||
"Find technical information about HTTP errors.",
|
||||
"Find technical information about machine translation.",
|
||||
"Find technical information about financial risk.",
|
||||
"Find technical information about medical imaging.",
|
||||
"Find technical information about supply chains.",
|
||||
"Find technical information about climate science.",
|
||||
"Find technical information about CPU instructions.",
|
||||
"Find technical information about compiler design.",
|
||||
"Find technical information about graph theory.",
|
||||
"Find technical information about astronomy."
|
||||
]
|
||||
},
|
||||
"results": {
|
||||
"annoy": {
|
||||
"build_ms": 1.214,
|
||||
"recall_at_k": 0.16666666666666666,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.027637556195259094,
|
||||
"p50": 0.026833033189177513,
|
||||
"p95": 0.030818302184343352
|
||||
},
|
||||
"serialized_bytes": 542256,
|
||||
"rankings": [
|
||||
{
|
||||
"query_index": 0,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 1,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 2,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 3,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 4,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 5,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 6,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 7,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 8,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 9,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 10,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 11,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 12,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 13,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 14,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 15,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 16,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 17,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 18,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 19,
|
||||
"doc_ids": [
|
||||
"doc_0000"
|
||||
]
|
||||
}
|
||||
],
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 1.247,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 0.13333333333333333
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 1.398,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.04115619231015444,
|
||||
"p50": 0.03870879299938679,
|
||||
"p95": 0.05208926741033793
|
||||
},
|
||||
"serialized_bytes": 135912,
|
||||
"rankings": [
|
||||
{
|
||||
"query_index": 0,
|
||||
"doc_ids": [
|
||||
"doc_0000",
|
||||
"doc_0020",
|
||||
"doc_0018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 1,
|
||||
"doc_ids": [
|
||||
"doc_0001",
|
||||
"doc_0021",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 2,
|
||||
"doc_ids": [
|
||||
"doc_0002",
|
||||
"doc_0022",
|
||||
"doc_0008"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 3,
|
||||
"doc_ids": [
|
||||
"doc_0003",
|
||||
"doc_0023",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 4,
|
||||
"doc_ids": [
|
||||
"doc_0024",
|
||||
"doc_0004",
|
||||
"doc_0003"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 5,
|
||||
"doc_ids": [
|
||||
"doc_0005",
|
||||
"doc_0025",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 6,
|
||||
"doc_ids": [
|
||||
"doc_0006",
|
||||
"doc_0026",
|
||||
"doc_0018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 7,
|
||||
"doc_ids": [
|
||||
"doc_0007",
|
||||
"doc_0027",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 8,
|
||||
"doc_ids": [
|
||||
"doc_0008",
|
||||
"doc_0028",
|
||||
"doc_0015"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 9,
|
||||
"doc_ids": [
|
||||
"doc_0009",
|
||||
"doc_0029",
|
||||
"doc_0016"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 10,
|
||||
"doc_ids": [
|
||||
"doc_0010",
|
||||
"doc_0030",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 11,
|
||||
"doc_ids": [
|
||||
"doc_0011",
|
||||
"doc_0031",
|
||||
"doc_0025"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 12,
|
||||
"doc_ids": [
|
||||
"doc_0012",
|
||||
"doc_0018",
|
||||
"doc_0005"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 13,
|
||||
"doc_ids": [
|
||||
"doc_0013",
|
||||
"doc_0000",
|
||||
"doc_0005"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 14,
|
||||
"doc_ids": [
|
||||
"doc_0014",
|
||||
"doc_0008",
|
||||
"doc_0028"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 15,
|
||||
"doc_ids": [
|
||||
"doc_0015",
|
||||
"doc_0008",
|
||||
"doc_0028"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 16,
|
||||
"doc_ids": [
|
||||
"doc_0016",
|
||||
"doc_0029",
|
||||
"doc_0017"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 17,
|
||||
"doc_ids": [
|
||||
"doc_0017",
|
||||
"doc_0016",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 18,
|
||||
"doc_ids": [
|
||||
"doc_0018",
|
||||
"doc_0005",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 19,
|
||||
"doc_ids": [
|
||||
"doc_0019",
|
||||
"doc_0000",
|
||||
"doc_0013"
|
||||
]
|
||||
}
|
||||
],
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 0.74,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"schema_version": "chapter3-evidence-v1",
|
||||
"experiment": "3-4",
|
||||
"run_id": "20260729T182108Z-3_4-7353a50f",
|
||||
"created_at": "2026-07-29T18:21:08.387916+00:00",
|
||||
"status": "partial",
|
||||
"run_dir": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/validation/runs/20260729T182108Z-3_4-7353a50f",
|
||||
"artifacts": {
|
||||
"evidence.json": "598decd9d5a757f70c27db255e8117e2621aa6799e4bb8a871083c97503d03c8",
|
||||
"receipts.json": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570"
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/indexing.py",
|
||||
"sha256": "80ca520a78f044b367edfe71ff4e4bb22fa80f00d4803831d17b2b8c14ba1f57",
|
||||
"bytes": 14977
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/benchmark.py",
|
||||
"sha256": "9780724566260f034a5bba3143a303a7e934d4b8ee0f8055adacc5a51bdbf8a8",
|
||||
"bytes": 10855
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"embedding_latency_ms": 9437.365,
|
||||
"annoy": {
|
||||
"build_ms": 1.214,
|
||||
"recall_at_k": 0.16666666666666666,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.027637556195259094,
|
||||
"p50": 0.026833033189177513,
|
||||
"p95": 0.030818302184343352
|
||||
},
|
||||
"serialized_bytes": 542256,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 1.247,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 0.13333333333333333
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 1.398,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.04115619231015444,
|
||||
"p50": 0.03870879299938679,
|
||||
"p95": 0.05208926741033793
|
||||
},
|
||||
"serialized_bytes": 135912,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 0.74,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": true,
|
||||
"same_vectors_and_queries": true,
|
||||
"exact_search_ground_truth": true,
|
||||
"recall_latency_build_size_measured": true,
|
||||
"incremental_behavior_measured": true,
|
||||
"both_backends_recall_at_least_0_8": false,
|
||||
"passed": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
[]
|
||||
@@ -0,0 +1,548 @@
|
||||
{
|
||||
"schema_version": "chapter3-evidence-v1",
|
||||
"experiment": "3-4",
|
||||
"run_id": "20260729T182754Z-3_4-d21199a0",
|
||||
"provenance": {
|
||||
"captured_at": "2026-07-29T18:27:54.233593+00:00",
|
||||
"git_revision": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
|
||||
"python": "3.11.4 (main, Jul 5 2023, 08:40:20) [Clang 14.0.6 ]",
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"credential_presence": {
|
||||
"ARK_API_KEY": true,
|
||||
"MOONSHOT_API_KEY": true,
|
||||
"OPENAI_API_KEY": true,
|
||||
"GEMINI_API_KEY": true,
|
||||
"SILICONFLOW_API_KEY": true
|
||||
}
|
||||
},
|
||||
"status": "partial",
|
||||
"configuration": {
|
||||
"embedding_model": "Qwen/Qwen3-Embedding-0.6B",
|
||||
"model_revision": "97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3",
|
||||
"device": "cpu",
|
||||
"seed": 37,
|
||||
"dimension": 1024,
|
||||
"documents": 40,
|
||||
"queries": 20,
|
||||
"top_k": 3,
|
||||
"annoy_runtime": {
|
||||
"kind": "docker-linux-aarch64",
|
||||
"reason": "host macOS ARM ANNOY extension failed health check (returned fewer than k items)",
|
||||
"base_image": "python:3.11-slim",
|
||||
"base_image_repo_digests": [
|
||||
"python@sha256:db3ff2e1800a8581e2c48a27c3995339d47bdf046da21c7627accd3d51053a93"
|
||||
],
|
||||
"container_setup_and_run_wall_ms": 38303.133
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": true,
|
||||
"same_vectors_and_queries": true,
|
||||
"exact_search_ground_truth": true,
|
||||
"recall_latency_build_size_measured": true,
|
||||
"incremental_behavior_measured": true,
|
||||
"both_backends_recall_at_least_0_8": true,
|
||||
"passed": false
|
||||
},
|
||||
"summary": {
|
||||
"embedding_latency_ms": 8671.983,
|
||||
"annoy": {
|
||||
"build_ms": 5.385,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.10229384060949087,
|
||||
"p50": 0.09818747639656067,
|
||||
"p95": 0.12266857083886863
|
||||
},
|
||||
"serialized_bytes": 542256,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 2.763,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 1.375,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.03809796180576086,
|
||||
"p50": 0.03702123649418354,
|
||||
"p95": 0.042385491542518146
|
||||
},
|
||||
"serialized_bytes": 135912,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 0.919,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"corpus": {
|
||||
"doc_ids": [
|
||||
"doc_0000",
|
||||
"doc_0001",
|
||||
"doc_0002",
|
||||
"doc_0003",
|
||||
"doc_0004",
|
||||
"doc_0005",
|
||||
"doc_0006",
|
||||
"doc_0007",
|
||||
"doc_0008",
|
||||
"doc_0009",
|
||||
"doc_0010",
|
||||
"doc_0011",
|
||||
"doc_0012",
|
||||
"doc_0013",
|
||||
"doc_0014",
|
||||
"doc_0015",
|
||||
"doc_0016",
|
||||
"doc_0017",
|
||||
"doc_0018",
|
||||
"doc_0019",
|
||||
"doc_0020",
|
||||
"doc_0021",
|
||||
"doc_0022",
|
||||
"doc_0023",
|
||||
"doc_0024",
|
||||
"doc_0025",
|
||||
"doc_0026",
|
||||
"doc_0027",
|
||||
"doc_0028",
|
||||
"doc_0029",
|
||||
"doc_0030",
|
||||
"doc_0031",
|
||||
"doc_0032",
|
||||
"doc_0033",
|
||||
"doc_0034",
|
||||
"doc_0035",
|
||||
"doc_0036",
|
||||
"doc_0037",
|
||||
"doc_0038",
|
||||
"doc_0039"
|
||||
],
|
||||
"texts": [
|
||||
"Topic: vector search. Approximate nearest-neighbor indexes accelerate semantic vector retrieval. A concise technical overview. Document revision 0000.",
|
||||
"Topic: database transactions. Database transactions use atomicity, consistency, isolation and durability. A concise technical overview. Document revision 0001.",
|
||||
"Topic: photosynthesis. Green plants turn sunlight and carbon dioxide into chemical energy. A concise technical overview. Document revision 0002.",
|
||||
"Topic: quantum entanglement. Entangled particles exhibit correlated quantum measurement outcomes. A concise technical overview. Document revision 0003.",
|
||||
"Topic: contract law. A valid contract generally requires offer acceptance and consideration. A concise technical overview. Document revision 0004.",
|
||||
"Topic: neural networks. Deep neural networks learn layered nonlinear representations from data. A concise technical overview. Document revision 0005.",
|
||||
"Topic: cybersecurity. Zero trust security continuously verifies identity and device posture. A concise technical overview. Document revision 0006.",
|
||||
"Topic: volcanoes. Volcanoes form when magma rises through fractures in the planetary crust. A concise technical overview. Document revision 0007.",
|
||||
"Topic: water cycle. Evaporation condensation precipitation and runoff form the water cycle. A concise technical overview. Document revision 0008.",
|
||||
"Topic: operating systems. An operating system schedules processes and manages memory and devices. A concise technical overview. Document revision 0009.",
|
||||
"Topic: HTTP errors. HTTP status 403 means a server understood but refused a request. A concise technical overview. Document revision 0010.",
|
||||
"Topic: machine translation. Multilingual models translate meaning between natural languages. A concise technical overview. Document revision 0011.",
|
||||
"Topic: financial risk. Portfolio diversification reduces exposure to idiosyncratic financial risk. A concise technical overview. Document revision 0012.",
|
||||
"Topic: medical imaging. Radiology systems analyze X-rays CT scans and magnetic resonance images. A concise technical overview. Document revision 0013.",
|
||||
"Topic: supply chains. Supply chain planning coordinates inventory logistics demand and suppliers. A concise technical overview. Document revision 0014.",
|
||||
"Topic: climate science. Climate models simulate long-term interactions among atmosphere ocean and land. A concise technical overview. Document revision 0015.",
|
||||
"Topic: CPU instructions. SIMD instructions apply one operation to several packed numeric values. A concise technical overview. Document revision 0016.",
|
||||
"Topic: compiler design. A compiler parses source code optimizes intermediate form and emits machine code. A concise technical overview. Document revision 0017.",
|
||||
"Topic: graph theory. Graph algorithms traverse vertices and edges to discover paths and communities. A concise technical overview. Document revision 0018.",
|
||||
"Topic: astronomy. Astronomers infer stellar properties from spectra luminosity and orbital motion. A concise technical overview. Document revision 0019.",
|
||||
"Topic: vector search. Approximate nearest-neighbor indexes accelerate semantic vector retrieval. This passage explains the central mechanism and its practical use. Document revision 0020.",
|
||||
"Topic: database transactions. Database transactions use atomicity, consistency, isolation and durability. This passage explains the central mechanism and its practical use. Document revision 0021.",
|
||||
"Topic: photosynthesis. Green plants turn sunlight and carbon dioxide into chemical energy. This passage explains the central mechanism and its practical use. Document revision 0022.",
|
||||
"Topic: quantum entanglement. Entangled particles exhibit correlated quantum measurement outcomes. This passage explains the central mechanism and its practical use. Document revision 0023.",
|
||||
"Topic: contract law. A valid contract generally requires offer acceptance and consideration. This passage explains the central mechanism and its practical use. Document revision 0024.",
|
||||
"Topic: neural networks. Deep neural networks learn layered nonlinear representations from data. This passage explains the central mechanism and its practical use. Document revision 0025.",
|
||||
"Topic: cybersecurity. Zero trust security continuously verifies identity and device posture. This passage explains the central mechanism and its practical use. Document revision 0026.",
|
||||
"Topic: volcanoes. Volcanoes form when magma rises through fractures in the planetary crust. This passage explains the central mechanism and its practical use. Document revision 0027.",
|
||||
"Topic: water cycle. Evaporation condensation precipitation and runoff form the water cycle. This passage explains the central mechanism and its practical use. Document revision 0028.",
|
||||
"Topic: operating systems. An operating system schedules processes and manages memory and devices. This passage explains the central mechanism and its practical use. Document revision 0029.",
|
||||
"Topic: HTTP errors. HTTP status 403 means a server understood but refused a request. This passage explains the central mechanism and its practical use. Document revision 0030.",
|
||||
"Topic: machine translation. Multilingual models translate meaning between natural languages. This passage explains the central mechanism and its practical use. Document revision 0031.",
|
||||
"Topic: financial risk. Portfolio diversification reduces exposure to idiosyncratic financial risk. This passage explains the central mechanism and its practical use. Document revision 0032.",
|
||||
"Topic: medical imaging. Radiology systems analyze X-rays CT scans and magnetic resonance images. This passage explains the central mechanism and its practical use. Document revision 0033.",
|
||||
"Topic: supply chains. Supply chain planning coordinates inventory logistics demand and suppliers. This passage explains the central mechanism and its practical use. Document revision 0034.",
|
||||
"Topic: climate science. Climate models simulate long-term interactions among atmosphere ocean and land. This passage explains the central mechanism and its practical use. Document revision 0035.",
|
||||
"Topic: CPU instructions. SIMD instructions apply one operation to several packed numeric values. This passage explains the central mechanism and its practical use. Document revision 0036.",
|
||||
"Topic: compiler design. A compiler parses source code optimizes intermediate form and emits machine code. This passage explains the central mechanism and its practical use. Document revision 0037.",
|
||||
"Topic: graph theory. Graph algorithms traverse vertices and edges to discover paths and communities. This passage explains the central mechanism and its practical use. Document revision 0038.",
|
||||
"Topic: astronomy. Astronomers infer stellar properties from spectra luminosity and orbital motion. This passage explains the central mechanism and its practical use. Document revision 0039."
|
||||
],
|
||||
"queries": [
|
||||
"Find technical information about vector search.",
|
||||
"Find technical information about database transactions.",
|
||||
"Find technical information about photosynthesis.",
|
||||
"Find technical information about quantum entanglement.",
|
||||
"Find technical information about contract law.",
|
||||
"Find technical information about neural networks.",
|
||||
"Find technical information about cybersecurity.",
|
||||
"Find technical information about volcanoes.",
|
||||
"Find technical information about water cycle.",
|
||||
"Find technical information about operating systems.",
|
||||
"Find technical information about HTTP errors.",
|
||||
"Find technical information about machine translation.",
|
||||
"Find technical information about financial risk.",
|
||||
"Find technical information about medical imaging.",
|
||||
"Find technical information about supply chains.",
|
||||
"Find technical information about climate science.",
|
||||
"Find technical information about CPU instructions.",
|
||||
"Find technical information about compiler design.",
|
||||
"Find technical information about graph theory.",
|
||||
"Find technical information about astronomy."
|
||||
]
|
||||
},
|
||||
"results": {
|
||||
"annoy": {
|
||||
"build_ms": 5.385,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.10229384060949087,
|
||||
"p50": 0.09818747639656067,
|
||||
"p95": 0.12266857083886863
|
||||
},
|
||||
"serialized_bytes": 542256,
|
||||
"rankings": [
|
||||
{
|
||||
"query_index": 0,
|
||||
"doc_ids": [
|
||||
"doc_0000",
|
||||
"doc_0020",
|
||||
"doc_0018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 1,
|
||||
"doc_ids": [
|
||||
"doc_0001",
|
||||
"doc_0021",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 2,
|
||||
"doc_ids": [
|
||||
"doc_0002",
|
||||
"doc_0022",
|
||||
"doc_0008"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 3,
|
||||
"doc_ids": [
|
||||
"doc_0003",
|
||||
"doc_0023",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 4,
|
||||
"doc_ids": [
|
||||
"doc_0024",
|
||||
"doc_0004",
|
||||
"doc_0003"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 5,
|
||||
"doc_ids": [
|
||||
"doc_0005",
|
||||
"doc_0025",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 6,
|
||||
"doc_ids": [
|
||||
"doc_0006",
|
||||
"doc_0026",
|
||||
"doc_0018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 7,
|
||||
"doc_ids": [
|
||||
"doc_0007",
|
||||
"doc_0027",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 8,
|
||||
"doc_ids": [
|
||||
"doc_0008",
|
||||
"doc_0028",
|
||||
"doc_0015"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 9,
|
||||
"doc_ids": [
|
||||
"doc_0009",
|
||||
"doc_0029",
|
||||
"doc_0016"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 10,
|
||||
"doc_ids": [
|
||||
"doc_0010",
|
||||
"doc_0030",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 11,
|
||||
"doc_ids": [
|
||||
"doc_0011",
|
||||
"doc_0031",
|
||||
"doc_0025"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 12,
|
||||
"doc_ids": [
|
||||
"doc_0012",
|
||||
"doc_0018",
|
||||
"doc_0005"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 13,
|
||||
"doc_ids": [
|
||||
"doc_0013",
|
||||
"doc_0000",
|
||||
"doc_0005"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 14,
|
||||
"doc_ids": [
|
||||
"doc_0014",
|
||||
"doc_0008",
|
||||
"doc_0028"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 15,
|
||||
"doc_ids": [
|
||||
"doc_0015",
|
||||
"doc_0008",
|
||||
"doc_0028"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 16,
|
||||
"doc_ids": [
|
||||
"doc_0016",
|
||||
"doc_0029",
|
||||
"doc_0017"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 17,
|
||||
"doc_ids": [
|
||||
"doc_0017",
|
||||
"doc_0016",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 18,
|
||||
"doc_ids": [
|
||||
"doc_0018",
|
||||
"doc_0005",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 19,
|
||||
"doc_ids": [
|
||||
"doc_0019",
|
||||
"doc_0000",
|
||||
"doc_0013"
|
||||
]
|
||||
}
|
||||
],
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 2.763,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 1.375,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.03809796180576086,
|
||||
"p50": 0.03702123649418354,
|
||||
"p95": 0.042385491542518146
|
||||
},
|
||||
"serialized_bytes": 135912,
|
||||
"rankings": [
|
||||
{
|
||||
"query_index": 0,
|
||||
"doc_ids": [
|
||||
"doc_0000",
|
||||
"doc_0020",
|
||||
"doc_0018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 1,
|
||||
"doc_ids": [
|
||||
"doc_0001",
|
||||
"doc_0021",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 2,
|
||||
"doc_ids": [
|
||||
"doc_0002",
|
||||
"doc_0022",
|
||||
"doc_0008"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 3,
|
||||
"doc_ids": [
|
||||
"doc_0003",
|
||||
"doc_0023",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 4,
|
||||
"doc_ids": [
|
||||
"doc_0024",
|
||||
"doc_0004",
|
||||
"doc_0003"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 5,
|
||||
"doc_ids": [
|
||||
"doc_0005",
|
||||
"doc_0025",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 6,
|
||||
"doc_ids": [
|
||||
"doc_0006",
|
||||
"doc_0026",
|
||||
"doc_0018"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 7,
|
||||
"doc_ids": [
|
||||
"doc_0007",
|
||||
"doc_0027",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 8,
|
||||
"doc_ids": [
|
||||
"doc_0008",
|
||||
"doc_0028",
|
||||
"doc_0015"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 9,
|
||||
"doc_ids": [
|
||||
"doc_0009",
|
||||
"doc_0029",
|
||||
"doc_0016"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 10,
|
||||
"doc_ids": [
|
||||
"doc_0010",
|
||||
"doc_0030",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 11,
|
||||
"doc_ids": [
|
||||
"doc_0011",
|
||||
"doc_0031",
|
||||
"doc_0025"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 12,
|
||||
"doc_ids": [
|
||||
"doc_0012",
|
||||
"doc_0018",
|
||||
"doc_0005"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 13,
|
||||
"doc_ids": [
|
||||
"doc_0013",
|
||||
"doc_0000",
|
||||
"doc_0005"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 14,
|
||||
"doc_ids": [
|
||||
"doc_0014",
|
||||
"doc_0008",
|
||||
"doc_0028"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 15,
|
||||
"doc_ids": [
|
||||
"doc_0015",
|
||||
"doc_0008",
|
||||
"doc_0028"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 16,
|
||||
"doc_ids": [
|
||||
"doc_0016",
|
||||
"doc_0029",
|
||||
"doc_0017"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 17,
|
||||
"doc_ids": [
|
||||
"doc_0017",
|
||||
"doc_0016",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 18,
|
||||
"doc_ids": [
|
||||
"doc_0018",
|
||||
"doc_0005",
|
||||
"doc_0000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"query_index": 19,
|
||||
"doc_ids": [
|
||||
"doc_0019",
|
||||
"doc_0000",
|
||||
"doc_0013"
|
||||
]
|
||||
}
|
||||
],
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 0.919,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"schema_version": "chapter3-evidence-v1",
|
||||
"experiment": "3-4",
|
||||
"run_id": "20260729T182754Z-3_4-d21199a0",
|
||||
"created_at": "2026-07-29T18:27:54.269138+00:00",
|
||||
"status": "partial",
|
||||
"run_dir": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/validation/runs/20260729T182754Z-3_4-d21199a0",
|
||||
"artifacts": {
|
||||
"evidence.json": "d04ecc4f2612d3559d402ff7c6d67e2e13cda27eb4f8e63f610e7a10b9052a7e",
|
||||
"receipts.json": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570"
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/indexing.py",
|
||||
"sha256": "80ca520a78f044b367edfe71ff4e4bb22fa80f00d4803831d17b2b8c14ba1f57",
|
||||
"bytes": 14977
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/benchmark.py",
|
||||
"sha256": "d5d72267ca683ec513acd422287ff6f7cd65c54c6eaad5d636b67b983dda7904",
|
||||
"bytes": 13883
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/docker_annoy_runner.py",
|
||||
"sha256": "f75ee2f5f81365143a6f8ad68c6706933f8012ca4792a643d8ae79bfc9621c70",
|
||||
"bytes": 2980
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"embedding_latency_ms": 8671.983,
|
||||
"annoy": {
|
||||
"build_ms": 5.385,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.10229384060949087,
|
||||
"p50": 0.09818747639656067,
|
||||
"p95": 0.12266857083886863
|
||||
},
|
||||
"serialized_bytes": 542256,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 2.763,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 1.375,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.03809796180576086,
|
||||
"p50": 0.03702123649418354,
|
||||
"p95": 0.042385491542518146
|
||||
},
|
||||
"serialized_bytes": 135912,
|
||||
"incremental_update": {
|
||||
"items_added": 8,
|
||||
"latency_ms": 0.919,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": true,
|
||||
"same_vectors_and_queries": true,
|
||||
"exact_search_ground_truth": true,
|
||||
"recall_latency_build_size_measured": true,
|
||||
"incremental_behavior_measured": true,
|
||||
"both_backends_recall_at_least_0_8": true,
|
||||
"passed": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
[]
|
||||
+1348
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"schema_version": "chapter3-evidence-v1",
|
||||
"experiment": "3-4",
|
||||
"run_id": "20260729T182946Z-3_4-2555ea60",
|
||||
"created_at": "2026-07-29T18:29:46.110155+00:00",
|
||||
"status": "passed",
|
||||
"run_dir": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/validation/runs/20260729T182946Z-3_4-2555ea60",
|
||||
"artifacts": {
|
||||
"evidence.json": "0670be3c6fe6e203dbacc926ac0a4cecc6b4902d49db2bc4205bc0d14d3b171f",
|
||||
"receipts.json": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570"
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/indexing.py",
|
||||
"sha256": "80ca520a78f044b367edfe71ff4e4bb22fa80f00d4803831d17b2b8c14ba1f57",
|
||||
"bytes": 14977
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/benchmark.py",
|
||||
"sha256": "d5d72267ca683ec513acd422287ff6f7cd65c54c6eaad5d636b67b983dda7904",
|
||||
"bytes": 13883
|
||||
},
|
||||
{
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter3/dense-embedding/docker_annoy_runner.py",
|
||||
"sha256": "f75ee2f5f81365143a6f8ad68c6706933f8012ca4792a643d8ae79bfc9621c70",
|
||||
"bytes": 2980
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"embedding_latency_ms": 55429.937,
|
||||
"annoy": {
|
||||
"build_ms": 13.203,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.2336004702374339,
|
||||
"p50": 0.22620847448706627,
|
||||
"p95": 0.2634833101183176
|
||||
},
|
||||
"serialized_bytes": 1396720,
|
||||
"incremental_update": {
|
||||
"items_added": 60,
|
||||
"latency_ms": 8.274,
|
||||
"requires_full_rebuild": true,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
},
|
||||
"hnsw": {
|
||||
"build_ms": 77.682,
|
||||
"recall_at_k": 1.0,
|
||||
"query_latency_ms": {
|
||||
"mean": 0.25732293259352446,
|
||||
"p50": 0.2595204859972,
|
||||
"p95": 0.2800690243020654
|
||||
},
|
||||
"serialized_bytes": 1018716,
|
||||
"incremental_update": {
|
||||
"items_added": 60,
|
||||
"latency_ms": 33.833,
|
||||
"requires_full_rebuild": false,
|
||||
"recall_at_k_after_update": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"real_embedding_model": true,
|
||||
"same_vectors_and_queries": true,
|
||||
"exact_search_ground_truth": true,
|
||||
"recall_latency_build_size_measured": true,
|
||||
"incremental_behavior_measured": true,
|
||||
"both_backends_recall_at_least_0_8": true,
|
||||
"passed": true
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user