ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# Data files
|
||||
*.json
|
||||
*.jsonl
|
||||
*.csv
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Generated visualizations
|
||||
*.png
|
||||
*.html
|
||||
!index.html
|
||||
|
||||
# Canonical Experiment 7-7 evidence is intentionally versioned. The public
|
||||
# Arena input remains ignored because it is ~2 GB; manifests bind it by URL,
|
||||
# byte size, record count, and SHA-256 instead.
|
||||
!validation/
|
||||
!validation/**/*.json
|
||||
!validation/**/*.png
|
||||
!validation/**/*.html
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
*.ipynb
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
@@ -0,0 +1,786 @@
|
||||
# Elo Rating Leaderboard from Pairwise Comparisons
|
||||
|
||||
## English
|
||||
|
||||
**Experiment 7-7**: Building Model Leaderboard from Pairwise Comparison Data
|
||||
|
||||
This project implements an Elo rating system from scratch to analyze model performance using Chatbot Arena's public voting data. The implementation demonstrates how the Bradley-Terry model extracts relative model capabilities from millions of pairwise comparison votes.
|
||||
|
||||
## Overview
|
||||
|
||||
The Elo rating system is a method for calculating the relative skill levels of players (or in this case, AI models) in zero-sum games. Originally developed for chess, it has been adapted to rank AI language models based on head-to-head comparisons from user votes.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **High-performance implementation**: NumPy + Numba JIT + parallel processing for optimal speed
|
||||
- **Real voting data analysis**: Uses actual Chatbot Arena voting data with millions of pairwise comparisons
|
||||
- **Win rate prediction**: Calculates expected win probabilities between any two models
|
||||
- **Historical tracking**: Builds time-series snapshots showing ranking evolution
|
||||
- **Interactive visualizations**: Multiple visualization types including animated bar chart races
|
||||
- **Scalable**: Efficiently handles 2GB datasets with hundreds of thousands of matches
|
||||
|
||||
## Mathematical Foundation
|
||||
|
||||
The Elo system is based on the Bradley-Terry model, which models the probability that model A beats model B as:
|
||||
|
||||
```
|
||||
P(A beats B) = 1 / (1 + 10^((R_B - R_A) / 400))
|
||||
```
|
||||
|
||||
After each match, ratings are updated using:
|
||||
|
||||
```
|
||||
R_A_new = R_A + K * (S_A - E_A)
|
||||
```
|
||||
|
||||
Where:
|
||||
- `R_A` is the current rating of model A
|
||||
- `K` is the learning rate (K-factor)
|
||||
- `S_A` is the actual score (1 for win, 0 for loss, 0.5 for tie)
|
||||
- `E_A` is the expected score (predicted win probability)
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Disk Space**: At least 3GB free (2GB for data file, 1GB for processing)
|
||||
- **RAM**: 4GB+ recommended for full dataset analysis
|
||||
- **Internet**: Stable connection for ~2GB download
|
||||
- **Python**: 3.12 for the root `ch6` install
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 6 environment
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch6]"
|
||||
|
||||
cd chapter7/elo-leaderboard
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# From the repository root, include the shared test tooling:
|
||||
uv sync --locked --python 3.12 --extra ch6 --extra dev
|
||||
|
||||
# Activate it before changing directories:
|
||||
source .venv/bin/activate
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch6,dev]"
|
||||
|
||||
cd chapter7/elo-leaderboard
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
## Canonical full-data validation
|
||||
|
||||
The accepted Experiment 7-7 run uses the complete public 2024-08-14 Arena
|
||||
snapshot rather than the synthetic quickstart or a small sample. The 2.0 GB
|
||||
input is not committed to git; the manifest records its official URL, byte
|
||||
size, 1,799,991-row count, and SHA-256.
|
||||
|
||||
```bash
|
||||
curl -L \
|
||||
https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json \
|
||||
-o /path/to/arena_data.json
|
||||
|
||||
python validation/run_experiment.py \
|
||||
--input /path/to/arena_data.json \
|
||||
--output-dir validation/runs/exp7-7-arena-20260731-v1 \
|
||||
--bootstrap-rounds 20
|
||||
|
||||
python validation/validate_evidence.py \
|
||||
validation/runs/exp7-7-arena-20260731-v1 \
|
||||
--input /path/to/arena_data.json
|
||||
```
|
||||
|
||||
The retained canonical run accepted 1,670,250 anonymous, deduplicated votes
|
||||
over 129 models. Chronological online Elo (initial 1000, K=4) and the
|
||||
Bradley-Terry reconstruction reached Spearman 0.787, Kendall 0.606, and 12/20
|
||||
top-model overlap. This is the expected broad agreement, not score identity:
|
||||
online Elo is order-dependent while Bradley-Terry fits all comparisons at
|
||||
once. Seventeen cumulative monthly snapshots drive the retained D3 animation.
|
||||
|
||||
Canonical evidence: [`validation/latest.json`](validation/latest.json).
|
||||
|
||||
## 命令行工具 / Command-Line Interface (`cli.py`)
|
||||
|
||||
`cli.py` 是本实验统一的 argparse 命令行入口（中文 `--help`），把整条流水线拆成子命令：
|
||||
**对战 (battle) -> 计算评分 (elo) -> 展示排行榜 (leaderboard)**，并提供 `pipeline` 一步到位。
|
||||
|
||||
```bash
|
||||
python cli.py --help # 查看全部子命令
|
||||
python cli.py battle --help # 查看某个子命令的参数
|
||||
|
||||
# 默认离线端到端演示：模拟对战 -> 在线 Elo -> 最终排行榜表格（无需任何数据/API）
|
||||
python cli.py # 等价于 python cli.py pipeline
|
||||
```
|
||||
|
||||
### 子命令
|
||||
|
||||
| 子命令 | 作用 | 关键参数 |
|
||||
|--------|------|----------|
|
||||
| `battle` | 生成两两对战结果 | `--source {simulate,arena,llm}`、`--num-battles`、`--tie-prob`、`--seed`、`--sample`、`--output` |
|
||||
| `elo` | 从对战结果计算评分 | `--method {online-elo,bradley-terry}`、`--k`、`--bootstrap`、`--input`、`--output` |
|
||||
| `leaderboard` | 渲染最终排行榜表格 | `--input`（对战或评分文件）、`--method`、`--bootstrap`、`--top-n` |
|
||||
| `pipeline` | 一步跑完 对战 -> Elo -> 排行榜 | 上述参数的并集 |
|
||||
|
||||
### 三种对战来源（`--source`）
|
||||
|
||||
- **`simulate`（默认，纯离线）**：从已知的潜在实力分模拟对战。因为真值已知，可用来**校验**恢复出的排行榜排序是否正确；`--tie-prob` 控制平局比例，用于演练平局处理。
|
||||
- **`arena`（离线）**：加载真实 Chatbot Arena 投票数据（默认 `arena_data.json`，约 2GB），可用 `--sample N` 抽样。
|
||||
- **`llm`（需 API）**：用 LLM 做配对评判，并内置**位置偏差消除**——每对交换顺序各评一次，两次判决一致才计胜负、否则记为平局（对应书中 6.4 位置偏差讨论）。仅此来源需要 LLM API Key。
|
||||
|
||||
**两种评判后端（`--judge-backend {anthropic,openrouter,auto}`，默认 `auto`）**：
|
||||
- `anthropic`：官方 `anthropic` SDK，用 `ANTHROPIC_API_KEY`。
|
||||
- `openrouter`：OpenAI 兼容 SDK 指向 `https://openrouter.ai/api/v1`，用 `OPENROUTER_API_KEY`。内部 Claude 名字会自动映射为 OpenRouter id（`claude-opus-4-8` → `anthropic/claude-opus-4.8`，`claude-haiku-4-5` → `anthropic/claude-haiku-4.5`）；已含 `/` 的 id（如 `openai/gpt-5.6-luna`）原样透传。当直连 Anthropic key 缺失或失效时用它兜底。
|
||||
- `auto`（默认）：有 `ANTHROPIC_API_KEY` 走 anthropic，否则回退 openrouter。注意 `auto` 只看 key 是否存在、不校验有效性；若 `ANTHROPIC_API_KEY` 存在但已失效，请显式 `--judge-backend openrouter`。
|
||||
|
||||
位置偏差消除与 A/B/tie 解析逻辑与后端无关，两条路径完全一致。
|
||||
|
||||
### 分步示例
|
||||
|
||||
```bash
|
||||
# 1) 模拟 5000 场对战（含 10% 平局）
|
||||
python cli.py battle --source simulate --num-battles 5000 --output battles.json
|
||||
|
||||
# 2) 用官方 Bradley-Terry MLE + 100 轮 bootstrap 置信区间计算评分
|
||||
python cli.py elo --input battles.json --method bradley-terry --bootstrap 100
|
||||
|
||||
# 3) 展示前 20 名排行榜（也可直接读评分文件）
|
||||
python cli.py leaderboard --input battles.json --top-n 20
|
||||
|
||||
# 用真实 Arena 数据抽样跑（离线）
|
||||
python cli.py pipeline --source arena --arena-file arena_data.json --sample 50000 --method bradley-terry --bootstrap 100
|
||||
|
||||
# LLM 评判对战（需要 API Key）——官方 Anthropic
|
||||
export ANTHROPIC_API_KEY=your-anthropic-api-key
|
||||
python cli.py battle --source llm --candidate-models claude-opus-4-8 claude-haiku-4-5
|
||||
|
||||
# LLM 评判对战——通过 OpenRouter 兜底（直连 Anthropic key 缺失/失效时）
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python cli.py battle --source llm --judge-backend openrouter \
|
||||
--judge-model claude-opus-4-8 \
|
||||
--candidate-models anthropic/claude-haiku-4.5 openai/gpt-5.6-luna
|
||||
```
|
||||
|
||||
模拟来源会同时打印真值潜在实力，方便和恢复出的排行榜对照；在线 Elo 与 Bradley-Terry 两种方法都应恢复出与真值一致的排名（分值不必精确对齐，见下文说明）。
|
||||
|
||||
## Quick Start
|
||||
|
||||
The project implements **two ranking methods** following official Chatbot Arena:
|
||||
|
||||
### 1. Bradley-Terry Model (Default - Recommended)
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
# or explicitly:
|
||||
python main.py bradley-terry
|
||||
```
|
||||
|
||||
**Use this for**: Official leaderboard, stable rankings, production use
|
||||
|
||||
**Key features**:
|
||||
- ✅ Official Chatbot Arena method
|
||||
- ✅ Uses sklearn LogisticRegression for Maximum Likelihood Estimation
|
||||
- ✅ Order-independent (processes all matches simultaneously)
|
||||
- ✅ Includes 95% confidence intervals via bootstrap (100 samples)
|
||||
- ✅ More stable and reliable rankings
|
||||
|
||||
**Processing time**: ~2-3 minutes (including bootstrap)
|
||||
|
||||
### 2. Online Elo (K=4)
|
||||
|
||||
```bash
|
||||
python main.py online-elo
|
||||
```
|
||||
|
||||
**Use this for**: Understanding Elo mechanics, educational purposes, faster computation
|
||||
|
||||
**Key features**:
|
||||
- ✅ K-factor = 4 (official value used by Chatbot Arena)
|
||||
- ✅ Simple sequential rating updates
|
||||
- ✅ Order-dependent (processes matches chronologically)
|
||||
- ✅ Faster computation (~30 seconds)
|
||||
- ⚠️ Less stable, can vary based on match order
|
||||
|
||||
### Method Comparison
|
||||
|
||||
| Feature | Bradley-Terry | Online Elo |
|
||||
|---------|--------------|------------|
|
||||
| **Stability** | High (MLE fit) | Medium (sequential) |
|
||||
| **Order dependence** | None | High |
|
||||
| **Confidence intervals** | Yes (bootstrap) | No |
|
||||
| **Speed** | Slower (~3 min) | Faster (~30 sec) |
|
||||
| **Official method** | ✅ Yes | For comparison only |
|
||||
| **Recommended** | ✅ Production | Educational |
|
||||
|
||||
### What Both Methods Do
|
||||
|
||||
1. Download Chatbot Arena voting data (~2GB, 5-15 minutes depending on connection)
|
||||
2. Apply official filters:
|
||||
- Anonymous votes only (blind evaluation)
|
||||
- Deduplication (removes top 0.1% redundant prompts)
|
||||
3. Compute model ratings using selected method
|
||||
4. Calculate predicted win rates between all model pairs
|
||||
5. Generate visualizations:
|
||||
- `leaderboard.png` - Top 20 models ranked by rating
|
||||
- `rating_distribution.png` - Rating histogram and statistics
|
||||
- `win_rate_matrix.png` - Predicted win rates (top 30 models)
|
||||
|
||||
**Note**: The initial data download is ~2GB and may take several minutes. A progress bar shows download status.
|
||||
|
||||
### Quick Demo (Synthetic Data)
|
||||
|
||||
To quickly understand Elo mechanics without downloading 2GB:
|
||||
|
||||
```bash
|
||||
python quickstart.py
|
||||
```
|
||||
|
||||
This runs a small demo with synthetic matchups between GPT-4, Claude, Llama, and Gemini.
|
||||
|
||||
### Benchmark
|
||||
|
||||
To compare both methods:
|
||||
|
||||
```bash
|
||||
python benchmark.py
|
||||
```
|
||||
|
||||
This shows performance and accuracy differences between online Elo and Bradley-Terry approaches.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
elo-leaderboard/
|
||||
├── cli.py # Unified argparse CLI (battle / elo / leaderboard / pipeline)
|
||||
├── battle_simulator.py # Offline synthetic pairwise-battle generator
|
||||
├── llm_judge.py # LLM-as-judge battles with position-bias mitigation (needs API)
|
||||
├── main.py # Main analysis script
|
||||
├── optimized_elo.py # NumPy + Numba Elo rating system
|
||||
├── parallel_processing.py # Multi-core parallel processing utilities
|
||||
├── data_loader.py # Data download and preprocessing
|
||||
├── leaderboard.py # Leaderboard calculation and analysis
|
||||
├── visualization.py # Static and interactive visualizations
|
||||
├── animation.py # Animated bar chart race generator
|
||||
├── benchmark.py # Performance benchmark tool
|
||||
├── quickstart.py # Quick demo with synthetic data
|
||||
├── elo_rating.py # Reference implementation (for comparison)
|
||||
├── tests/ # Unit and regression tests
|
||||
├── requirements.txt # Python dependencies
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Building Elo Leaderboard
|
||||
|
||||
```python
|
||||
from optimized_elo import build_leaderboard_optimized
|
||||
|
||||
# Build Elo leaderboard from DataFrame
|
||||
elo = build_leaderboard_optimized(
|
||||
df, # DataFrame with columns: model_a, model_b, winner
|
||||
initial_rating=1000.0,
|
||||
k_factor=32.0,
|
||||
show_progress=True
|
||||
)
|
||||
|
||||
# Get leaderboard
|
||||
leaderboard = elo.get_leaderboard()
|
||||
for rank, (model, rating, matches, wins) in enumerate(leaderboard[:10], 1):
|
||||
win_rate = wins / matches * 100 if matches > 0 else 0
|
||||
print(f"{rank}. {model}: {rating:.1f} ({matches} matches, {win_rate:.1f}% win rate)")
|
||||
```
|
||||
|
||||
### Loading and Filtering Data
|
||||
|
||||
```python
|
||||
from data_loader import load_arena_data, filter_data
|
||||
|
||||
# Load data
|
||||
df = load_arena_data("arena_data.json")
|
||||
|
||||
# Filter for blind votes only (reduces bias)
|
||||
df_filtered = filter_data(
|
||||
df,
|
||||
anony_only=True, # Only anonymous votes
|
||||
language="English", # Specific language
|
||||
min_turn=1 # Minimum conversation turn
|
||||
)
|
||||
```
|
||||
|
||||
### Building Historical Leaderboards
|
||||
|
||||
```python
|
||||
from data_loader import get_time_slices
|
||||
from leaderboard import build_historical_leaderboards, get_rating_history
|
||||
|
||||
# Create weekly time slices
|
||||
time_slices = get_time_slices(df, interval='W')
|
||||
|
||||
# Build leaderboard for each time point
|
||||
historical_leaderboards = build_historical_leaderboards(
|
||||
df, time_slices, initial_rating=1000.0, k_factor=32.0
|
||||
)
|
||||
|
||||
# Get rating history DataFrame
|
||||
history_df = get_rating_history(historical_leaderboards)
|
||||
```
|
||||
|
||||
### Creating Visualizations
|
||||
|
||||
```python
|
||||
from visualization import (
|
||||
plot_leaderboard,
|
||||
plot_win_rate_matrix,
|
||||
plot_rating_history,
|
||||
create_interactive_leaderboard
|
||||
)
|
||||
|
||||
# Static leaderboard chart
|
||||
plot_leaderboard(leaderboard, top_n=20, save_path="leaderboard.png")
|
||||
|
||||
# Win rate heatmap
|
||||
win_rate_df = calculate_win_rate_matrix_from_data(df)
|
||||
plot_win_rate_matrix(win_rate_df, top_n=15, save_path="matrix.png")
|
||||
|
||||
# Rating evolution
|
||||
plot_rating_history(history_df, models=["gpt-4", "claude-v1"],
|
||||
save_path="history.png")
|
||||
|
||||
# Interactive chart
|
||||
fig = create_interactive_leaderboard(history_df, top_n=15)
|
||||
fig.write_html("interactive.html")
|
||||
```
|
||||
|
||||
### Creating Animated Bar Chart Race
|
||||
|
||||
```python
|
||||
from animation import create_simple_animation
|
||||
|
||||
# Generate animated HTML
|
||||
animation_file = create_simple_animation(
|
||||
history_df,
|
||||
output_path="animation.html",
|
||||
top_n=15
|
||||
)
|
||||
|
||||
# Open animation.html in browser to view
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
After running `main.py`, the following files are generated:
|
||||
|
||||
### Static Images (PNG)
|
||||
- `leaderboard.png` - Current top 20 models ranked by Elo rating
|
||||
- `rating_distribution.png` - Histogram and box plot of rating distribution
|
||||
- `win_rate_matrix.png` - Heatmap showing pairwise win rates
|
||||
- `rating_history.png` - Line chart showing rating evolution over time
|
||||
|
||||
### Interactive Visualizations (HTML)
|
||||
- `interactive_rating_evolution.html` - Interactive chart with zoom/pan
|
||||
- `interactive_rank_evolution.html` - Interactive rank tracking
|
||||
- `leaderboard_animation.html` - **Animated bar chart race** showing ranking evolution
|
||||
|
||||
## Key Parameters
|
||||
|
||||
### Elo System Parameters (Online Elo Method)
|
||||
|
||||
- **initial_rating** (default: 1000.0): Starting rating for all models
|
||||
- **k_factor** (default: 4.0): Learning rate controlling update magnitude
|
||||
- Official Chatbot Arena uses K=4 for stability
|
||||
- Higher K-factor (e.g., 32): More volatile, faster adaptation to new data
|
||||
- Lower K-factor (e.g., 4): More stable, less influenced by recent matches
|
||||
|
||||
### Bradley-Terry Parameters
|
||||
|
||||
- **SCALE** (400): Elo scale parameter - determines rating point interpretation
|
||||
- **BASE** (10): Base for logistic function - standard for Elo calculations
|
||||
- **INIT_RATING** (1000): Initial rating for all models
|
||||
- **bootstrap_rounds** (100): Number of bootstrap samples for confidence intervals
|
||||
|
||||
### Time Slice Intervals
|
||||
|
||||
For historical analysis, you can adjust the time granularity:
|
||||
- `'D'` - Daily snapshots
|
||||
- `'W'` - Weekly snapshots (recommended)
|
||||
- `'M'` - Monthly snapshots
|
||||
|
||||
### Visualization Parameters
|
||||
|
||||
- **top_n**: Number of top models to display (10-20 recommended)
|
||||
- **Animation speed**: Adjustable in the HTML interface (1x to 10x)
|
||||
|
||||
## Data Format
|
||||
|
||||
The Chatbot Arena data includes the following fields:
|
||||
|
||||
- `model_a`: Identifier for first model
|
||||
- `model_b`: Identifier for second model
|
||||
- `winner`: Match outcome ('model_a', 'model_b', or 'tie')
|
||||
- `tstamp`: Unix timestamp of the vote
|
||||
- `judge`: User who made the vote
|
||||
- `turn`: Conversation turn number
|
||||
- `anony`: Whether vote was anonymous/blind
|
||||
- `language`: Language of the conversation
|
||||
|
||||
## Validation
|
||||
|
||||
The implementation validates the Elo predictions against empirical win rates:
|
||||
|
||||
```python
|
||||
from leaderboard import compare_win_rates
|
||||
|
||||
# Compare predicted vs actual win rates
|
||||
comparison = compare_win_rates(elo_system, empirical_win_rates)
|
||||
mean_error = comparison['error'].mean()
|
||||
print(f"Mean Absolute Error: {mean_error:.4f}")
|
||||
```
|
||||
|
||||
A low MAE (< 0.05) indicates the Elo model fits the data well.
|
||||
|
||||
## Analysis Insights
|
||||
|
||||
The project helps identify:
|
||||
|
||||
1. **Current Rankings**: Which models are currently strongest
|
||||
2. **Rating Trends**: How model performance evolves over time
|
||||
3. **Breakthrough Moments**: When new models enter or shake up rankings
|
||||
4. **Competitive Dynamics**: Which models are closely matched
|
||||
5. **Long-term Trajectories**: Models in ascent vs. decline
|
||||
6. **Rating Stability**: Volatility in model performance
|
||||
|
||||
## Performance Architecture
|
||||
|
||||
The implementation is designed for high performance on large datasets (2GB+).
|
||||
|
||||
### Core Optimizations
|
||||
|
||||
#### 1. **NumPy + Numba JIT Compilation**
|
||||
|
||||
Uses NumPy arrays and Numba's just-in-time compilation:
|
||||
- **NumPy arrays** for O(1) integer indexing (vs O(n) dictionary lookups)
|
||||
- **Numba JIT** compiles hot loops to machine code (50-100x speedup)
|
||||
- **Pre-allocated arrays** eliminate dynamic memory allocation overhead
|
||||
- **Integer indices** instead of string model names for cache-friendly access
|
||||
|
||||
#### 2. **Multi-Core Parallel Processing**
|
||||
|
||||
Parallelizes independent operations across all CPU cores:
|
||||
- **Historical analysis**: Each time slice processed independently
|
||||
- **Win rate matrices**: Model pairs computed in parallel chunks
|
||||
- **Data filtering**: DataFrame operations distributed across cores
|
||||
|
||||
```python
|
||||
from parallel_processing import build_historical_leaderboards_parallel
|
||||
|
||||
# Automatically uses all available CPU cores
|
||||
historical_lb = build_historical_leaderboards_parallel(
|
||||
df, time_slices, n_jobs=-1
|
||||
)
|
||||
```
|
||||
|
||||
#### 3. **Memory Optimization**
|
||||
|
||||
Reduces memory footprint through intelligent data types:
|
||||
- Downcasts numeric types (int64 → int32, float64 → float32)
|
||||
- Converts repetitive strings to categorical types
|
||||
- Achieves 30-50% memory reduction
|
||||
|
||||
```python
|
||||
from parallel_processing import optimize_dataframe
|
||||
|
||||
df = optimize_dataframe(df) # Automatic memory optimization
|
||||
```
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
On typical hardware (4-8 core CPU) with the full 2GB dataset:
|
||||
|
||||
| Component | Technique | Impact |
|
||||
|---|---|---|
|
||||
| Elo Computation | NumPy + Numba JIT | 50-100x faster |
|
||||
| Historical Analysis | Multi-core parallel | 4-8x faster |
|
||||
| Win Rate Matrix | Parallel processing | 4-8x faster |
|
||||
| Memory Usage | Type optimization | 30-50% reduction |
|
||||
| **Overall** | **Combined** | **~10-15x speedup** |
|
||||
|
||||
**Processing time**: 1-2 minutes for full dataset (hundreds of thousands of matches)
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Custom Analysis
|
||||
|
||||
The modular design allows for flexible customization:
|
||||
|
||||
### Focused Analysis
|
||||
|
||||
```python
|
||||
from optimized_elo import build_leaderboard_optimized
|
||||
from data_loader import load_arena_data, filter_data
|
||||
|
||||
df = load_arena_data("arena_data.json")
|
||||
|
||||
# Analyze only recent data
|
||||
df_recent = filter_data(df, min_date="2024-01-01")
|
||||
elo_recent = build_leaderboard_optimized(df_recent)
|
||||
|
||||
# Analyze specific model family
|
||||
gpt_models = [m for m in df['model_a'].unique() if 'gpt' in m.lower()]
|
||||
df_gpt = df[df['model_a'].isin(gpt_models) & df['model_b'].isin(gpt_models)]
|
||||
elo_gpt = build_leaderboard_optimized(df_gpt)
|
||||
```
|
||||
|
||||
### Export Results
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
# Export leaderboard to CSV
|
||||
lb_df = pd.DataFrame(leaderboard, columns=['model', 'rating', 'matches', 'wins'])
|
||||
lb_df.to_csv('leaderboard.csv', index=False)
|
||||
|
||||
# Export rating history
|
||||
history_df.to_csv('rating_history.csv', index=False)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Data Download Issues
|
||||
|
||||
If automatic download fails:
|
||||
1. Manually download from: https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json
|
||||
2. Save as `arena_data.json` in the project directory
|
||||
3. Run `python main.py` again
|
||||
|
||||
### Memory Issues
|
||||
|
||||
The dataset is large (~2GB, hundreds of thousands of battles). If you encounter memory issues on systems with limited RAM:
|
||||
|
||||
```python
|
||||
from data_loader import load_arena_data, filter_data, get_time_slices
|
||||
from optimized_elo import build_leaderboard_optimized
|
||||
|
||||
# Load and immediately filter to reduce memory usage
|
||||
df = load_arena_data("arena_data.json")
|
||||
|
||||
# Filter to recent data only
|
||||
df_filtered = filter_data(df, min_date="2024-01-01", anony_only=True)
|
||||
|
||||
# Use monthly instead of weekly intervals for historical analysis
|
||||
time_slices = get_time_slices(df_filtered, interval='M') # vs 'W' for weekly
|
||||
|
||||
# Analyze with smaller top_n for visualizations
|
||||
elo = build_leaderboard_optimized(df_filtered)
|
||||
```
|
||||
|
||||
The built-in memory optimization reduces footprint by 30-50%, but very large analyses may still require 4GB+ RAM.
|
||||
|
||||
### Visualization Issues
|
||||
|
||||
- Ensure matplotlib, seaborn, and plotly are installed via the root `ch6` extra or the compatibility `requirements.txt` path.
|
||||
- For HTML animations, use a modern web browser (Chrome, Firefox, Safari, Edge)
|
||||
- If plots don't display in Jupyter, use `%matplotlib inline` or save to file
|
||||
|
||||
## References
|
||||
|
||||
- **Chatbot Arena**: https://chat.lmsys.org/
|
||||
- **Elo Rating System**: https://en.wikipedia.org/wiki/Elo_rating_system
|
||||
- **Bradley-Terry Model**: https://en.wikipedia.org/wiki/Bradley–Terry_model
|
||||
- **LMSYS Paper**: "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"
|
||||
|
||||
## Learning Objectives
|
||||
|
||||
This experiment demonstrates:
|
||||
|
||||
1. **Statistical Modeling**: How pairwise comparisons reveal relative abilities
|
||||
2. **Online Learning**: Incremental rating updates as new data arrives
|
||||
3. **Probabilistic Prediction**: Converting rating differences to win probabilities
|
||||
4. **Data Visualization**: Effective techniques for showing temporal dynamics
|
||||
5. **Model Evaluation**: Alternative to traditional benchmark approaches
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Why Elo Computation is Hard to Parallelize
|
||||
|
||||
Elo rating computation is **inherently sequential** because each match's rating update depends on the current ratings, which were modified by all previous matches. This is why we can't simply split matches into chunks and process them independently.
|
||||
|
||||
However, we can still achieve significant speedups through:
|
||||
|
||||
1. **Algorithmic optimization**: NumPy arrays + Numba JIT
|
||||
2. **Parallelizing independent operations**: Historical analysis, win rate matrices
|
||||
3. **Memory efficiency**: Better cache utilization
|
||||
4. **Data structure optimization**: Integer indexing, pre-allocation
|
||||
|
||||
### Numba JIT Compilation
|
||||
|
||||
The core Elo update loop is compiled to machine code using Numba:
|
||||
|
||||
```python
|
||||
@jit(nopython=True)
|
||||
def process_elo_updates_vectorized(ratings, model_a_indices, model_b_indices,
|
||||
outcomes, k_factor, match_counts, win_counts):
|
||||
for i in range(len(model_a_indices)):
|
||||
# This loop runs at C speed, not Python speed
|
||||
# Typical speedup: 50-100x over pure Python
|
||||
...
|
||||
```
|
||||
|
||||
### Memory Layout
|
||||
|
||||
Using NumPy arrays with proper data types:
|
||||
- `ratings`: float64 array (8 bytes per model)
|
||||
- `match_counts`: int32 array (4 bytes per model)
|
||||
- `model_indices`: int32 array (4 bytes per match)
|
||||
|
||||
For 500 models and 500K matches: ~10 MB vs ~500 MB for dictionaries.
|
||||
|
||||
## Extensions
|
||||
|
||||
Potential enhancements:
|
||||
|
||||
- Implement Glicko or Glicko-2 rating systems (account for rating uncertainty)
|
||||
- Add confidence intervals for rating estimates
|
||||
- Analyze rating by language or task type
|
||||
- Compare with other ranking methods (e.g., TrueSkill, PageRank)
|
||||
- Implement time-decay for older matches
|
||||
- Add statistical significance testing
|
||||
- Build prediction model for future rankings
|
||||
- GPU acceleration using CuPy for even larger datasets
|
||||
- Distributed processing using Dask for multi-machine scaling
|
||||
|
||||
## License
|
||||
|
||||
This project is part of the AI Agent practical training course materials.
|
||||
|
||||
## Contact
|
||||
|
||||
For questions or issues, please refer to the course materials or discussion forums.
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
该项目围绕**配对比较（pairwise）数据**构建 Elo/Bradley-Terry 排名流程，目标是用公开的模型对战投票数据（重点是 Chatbot Arena）形成可复现的模型排行榜与可视化分析。
|
||||
|
||||
### 实验导向背景
|
||||
|
||||
Elo 本质上用于“成对对局中的胜率”学习相对能力，最初用于棋类，现被广泛用于语言模型两两对比的排序。
|
||||
|
||||
### 关键特性
|
||||
|
||||
- 高性能实现：NumPy + Numba JIT + 并行处理。
|
||||
- 真实数据分析：接入大规模公开投票数据。
|
||||
- 胜率推断：可预测任意两个模型的胜率。
|
||||
- 历史追踪：可输出时间序列排行快照。
|
||||
- 交互可视化：支持静态图与动态动画。
|
||||
- 可扩展：可承接较大规模比赛集合。
|
||||
|
||||
### 数学原理
|
||||
|
||||
与 AndroidWorld 风格一致，评分来自 Bradley-Terry：
|
||||
|
||||
```
|
||||
P(A 胜过 B) = 1 / (1 + 10^((R_B - R_A) / 400))
|
||||
```
|
||||
|
||||
单步更新：
|
||||
|
||||
```
|
||||
R_A_new = R_A + K * (S_A - E_A)
|
||||
```
|
||||
|
||||
### 安装与运行
|
||||
|
||||
```bash
|
||||
# 在仓库根目录使用统一的第 6 章环境
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch6]"
|
||||
|
||||
cd chapter7/elo-leaderboard
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 测试
|
||||
|
||||
```bash
|
||||
# 在仓库根目录安装测试工具：
|
||||
uv sync --locked --python 3.12 --extra ch6 --extra dev
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
source .venv/bin/activate
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch6,dev]"
|
||||
|
||||
cd chapter7/elo-leaderboard
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
### 命令行（`cli.py`）
|
||||
|
||||
`cli.py` 是统一入口：
|
||||
- `battle`：生成/采集两两对战
|
||||
- `elo`：计算评级
|
||||
- `leaderboard`：出榜
|
||||
- `pipeline`：端到端一条龙
|
||||
|
||||
```bash
|
||||
python cli.py --help
|
||||
python cli.py battle --help
|
||||
python cli.py # 等价于 python cli.py pipeline
|
||||
```
|
||||
|
||||
### 三类对战源
|
||||
|
||||
- `simulate`：合成对战（有真值），用于验证是否恢复出正确排序。
|
||||
- `arena`：离线加载 `arena_data.json`（约 2GB）；可用 `--sample` 抽样。
|
||||
- `llm`：调用 LLM 判断对战，带位置偏差消除；支持 `anthropic`、`openrouter` 和 `auto`。
|
||||
|
||||
`auto` 会优先使用 Anthropic key，失败时回退 OpenRouter；位置消偏策略与 A/B/tie 判定和后端无关。
|
||||
|
||||
### 两种核心评分方法
|
||||
|
||||
- Bradley-Terry（推荐）：更稳定，适合正式排行。
|
||||
- Online Elo：更贴近课程里的机制讲解，速度快但对顺序敏感。
|
||||
|
||||
### 项目结构
|
||||
|
||||
同上方英文学段落中的文件列表。
|
||||
|
||||
### 使用示例
|
||||
|
||||
核心示例同上英文学：
|
||||
- `python cli.py battle ...`
|
||||
- `python cli.py elo ...`
|
||||
- `python cli.py leaderboard ...`
|
||||
- `python demo.py` / `python benchmark.py`
|
||||
|
||||
### 注意
|
||||
|
||||
- `--sample`、`--pipeline`、`--top-n` 等参数见命令行帮助。
|
||||
- 建议先看 CLI 输出再对照 `leaderboard` 与可视化文件确认理解。
|
||||
@@ -0,0 +1,436 @@
|
||||
"""
|
||||
Create animated bar chart race showing leaderboard evolution over time
|
||||
"""
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from typing import List, Tuple
|
||||
import json
|
||||
import os
|
||||
|
||||
|
||||
def prepare_animation_data(history_df: pd.DataFrame, top_n: int = 15) -> dict:
|
||||
"""
|
||||
Prepare data for D3.js bar chart race animation.
|
||||
|
||||
Args:
|
||||
history_df: DataFrame with columns: date, model, rating, rank
|
||||
top_n: Number of top models to show at each time point
|
||||
|
||||
Returns:
|
||||
Dictionary with animation data
|
||||
"""
|
||||
# Convert date column to Timestamp to support ISO date strings and date objects
|
||||
if history_df is not None and len(history_df) > 0 and 'date' in history_df.columns:
|
||||
history_df = history_df.copy()
|
||||
history_df['date'] = pd.to_datetime(history_df['date'])
|
||||
# Get all unique dates
|
||||
dates = sorted(history_df['date'].unique())
|
||||
if len(dates) == 0:
|
||||
return {
|
||||
'frames': [],
|
||||
'total_frames': 0,
|
||||
'top_n': top_n,
|
||||
'start_date': None,
|
||||
'end_date': None,
|
||||
}
|
||||
|
||||
# For each date, get top N models
|
||||
frames = []
|
||||
for date in dates:
|
||||
date_data = history_df[history_df['date'] == date].nlargest(top_n, 'rating')
|
||||
|
||||
frame = {
|
||||
'date': date.strftime('%Y-%m-%d'),
|
||||
'timestamp': int(date.timestamp()),
|
||||
'models': []
|
||||
}
|
||||
|
||||
for rank, row in enumerate(date_data.itertuples(), 1):
|
||||
frame['models'].append({
|
||||
'rank': rank,
|
||||
'name': row.model,
|
||||
'rating': float(row.rating),
|
||||
'matches': int(row.matches),
|
||||
'wins': float(row.wins)
|
||||
})
|
||||
|
||||
frames.append(frame)
|
||||
|
||||
animation_data = {
|
||||
'frames': frames,
|
||||
'total_frames': len(frames),
|
||||
'top_n': top_n,
|
||||
'start_date': dates[0].strftime('%Y-%m-%d'),
|
||||
'end_date': dates[-1].strftime('%Y-%m-%d')
|
||||
}
|
||||
|
||||
return animation_data
|
||||
|
||||
|
||||
def generate_html_animation(animation_data: dict, output_path: str = "leaderboard_animation.html"):
|
||||
"""
|
||||
Generate standalone HTML file with D3.js bar chart race animation.
|
||||
|
||||
Args:
|
||||
animation_data: Dictionary from prepare_animation_data
|
||||
output_path: Path to save HTML file
|
||||
"""
|
||||
html_template = """<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Model Leaderboard Evolution</title>
|
||||
<script src="https://d3js.org/d3.v7.min.js"></script>
|
||||
<style>
|
||||
body {
|
||||
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background: #f5f5f5;
|
||||
}
|
||||
|
||||
#container {
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
padding: 30px;
|
||||
border-radius: 10px;
|
||||
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
|
||||
}
|
||||
|
||||
h1 {
|
||||
text-align: center;
|
||||
color: #333;
|
||||
margin-bottom: 10px;
|
||||
}
|
||||
|
||||
#date-display {
|
||||
text-align: center;
|
||||
font-size: 24px;
|
||||
font-weight: bold;
|
||||
color: #666;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
#chart {
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.bar {
|
||||
fill: steelblue;
|
||||
cursor: pointer;
|
||||
transition: fill 0.3s;
|
||||
}
|
||||
|
||||
.bar:hover {
|
||||
fill: #4682b4;
|
||||
}
|
||||
|
||||
.bar-label {
|
||||
font-size: 14px;
|
||||
fill: white;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.bar-value {
|
||||
font-size: 12px;
|
||||
fill: #333;
|
||||
}
|
||||
|
||||
.rank-label {
|
||||
font-size: 18px;
|
||||
fill: #666;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
#controls {
|
||||
text-align: center;
|
||||
margin-top: 30px;
|
||||
}
|
||||
|
||||
button {
|
||||
padding: 10px 20px;
|
||||
margin: 0 5px;
|
||||
font-size: 16px;
|
||||
cursor: pointer;
|
||||
border: none;
|
||||
border-radius: 5px;
|
||||
background: #4CAF50;
|
||||
color: white;
|
||||
transition: background 0.3s;
|
||||
}
|
||||
|
||||
button:hover {
|
||||
background: #45a049;
|
||||
}
|
||||
|
||||
button:disabled {
|
||||
background: #ccc;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
#progress-bar {
|
||||
width: 100%;
|
||||
height: 5px;
|
||||
background: #e0e0e0;
|
||||
margin-top: 20px;
|
||||
border-radius: 3px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#progress {
|
||||
height: 100%;
|
||||
background: #4CAF50;
|
||||
width: 0%;
|
||||
transition: width 0.5s;
|
||||
}
|
||||
|
||||
#speed-control {
|
||||
margin-top: 20px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
#speed-slider {
|
||||
width: 300px;
|
||||
margin: 0 10px;
|
||||
}
|
||||
|
||||
.info-box {
|
||||
background: #f9f9f9;
|
||||
padding: 15px;
|
||||
border-radius: 5px;
|
||||
margin-top: 20px;
|
||||
font-size: 14px;
|
||||
color: #666;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="container">
|
||||
<h1>🏆 Model Leaderboard Evolution</h1>
|
||||
<div id="date-display">Loading...</div>
|
||||
<div id="chart"></div>
|
||||
<div id="controls">
|
||||
<button id="play-btn">▶ Play</button>
|
||||
<button id="pause-btn" disabled>⏸ Pause</button>
|
||||
<button id="reset-btn">↺ Reset</button>
|
||||
</div>
|
||||
<div id="progress-bar">
|
||||
<div id="progress"></div>
|
||||
</div>
|
||||
<div id="speed-control">
|
||||
<label>Speed: </label>
|
||||
<input type="range" id="speed-slider" min="1" max="10" value="5">
|
||||
<span id="speed-value">5x</span>
|
||||
</div>
|
||||
<div class="info-box">
|
||||
<strong>About:</strong> This animation shows the evolution of model rankings based on Elo ratings
|
||||
calculated from Chatbot Arena voting data. Each frame represents a snapshot in time,
|
||||
with models ranked by their current Elo rating. Bars show the rating value,
|
||||
and the animation reveals how models compete and evolve over time.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const data = """ + json.dumps(animation_data, indent=2) + """;
|
||||
|
||||
// Configuration
|
||||
const margin = {top: 20, right: 100, bottom: 40, left: 50};
|
||||
const width = 1100 - margin.left - margin.right;
|
||||
const height = 600 - margin.top - margin.bottom;
|
||||
const barHeight = height / data.top_n - 5;
|
||||
|
||||
// Create SVG
|
||||
const svg = d3.select("#chart")
|
||||
.append("svg")
|
||||
.attr("width", width + margin.left + margin.right)
|
||||
.attr("height", height + margin.top + margin.bottom)
|
||||
.append("g")
|
||||
.attr("transform", `translate(${margin.left},${margin.top})`);
|
||||
|
||||
// Scales
|
||||
const xScale = d3.scaleLinear()
|
||||
.domain([0, d3.max(data.frames.flatMap(f => f.models.map(m => m.rating)))])
|
||||
.range([0, width - 200]);
|
||||
|
||||
// Color scale
|
||||
const colorScale = d3.scaleOrdinal(d3.schemeCategory10);
|
||||
|
||||
// Animation state
|
||||
let currentFrame = 0;
|
||||
let isPlaying = false;
|
||||
let animationInterval = null;
|
||||
let animationSpeed = 500; // milliseconds per frame
|
||||
|
||||
// Update speed based on slider
|
||||
d3.select("#speed-slider").on("input", function() {
|
||||
const speed = +this.value;
|
||||
animationSpeed = 1000 / speed;
|
||||
d3.select("#speed-value").text(`${speed}x`);
|
||||
if (isPlaying) {
|
||||
stopAnimation();
|
||||
startAnimation();
|
||||
}
|
||||
});
|
||||
|
||||
function updateChart(frameIndex) {
|
||||
const frame = data.frames[frameIndex];
|
||||
|
||||
// Update date display
|
||||
d3.select("#date-display").text(frame.date);
|
||||
|
||||
// Update progress bar
|
||||
const progress = ((frameIndex + 1) / data.total_frames) * 100;
|
||||
d3.select("#progress").style("width", `${progress}%`);
|
||||
|
||||
// Update max value for scale
|
||||
const maxRating = d3.max(frame.models, d => d.rating);
|
||||
xScale.domain([0, maxRating * 1.1]);
|
||||
|
||||
// Bind data
|
||||
const bars = svg.selectAll(".bar-group")
|
||||
.data(frame.models, d => d.name);
|
||||
|
||||
// Remove old bars
|
||||
bars.exit()
|
||||
.transition()
|
||||
.duration(animationSpeed * 0.8)
|
||||
.style("opacity", 0)
|
||||
.remove();
|
||||
|
||||
// Add new bars
|
||||
const enter = bars.enter()
|
||||
.append("g")
|
||||
.attr("class", "bar-group")
|
||||
.style("opacity", 0);
|
||||
|
||||
enter.append("rect")
|
||||
.attr("class", "bar")
|
||||
.attr("height", barHeight);
|
||||
|
||||
enter.append("text")
|
||||
.attr("class", "bar-label")
|
||||
.attr("x", 10)
|
||||
.attr("y", barHeight / 2)
|
||||
.attr("dy", "0.35em");
|
||||
|
||||
enter.append("text")
|
||||
.attr("class", "bar-value")
|
||||
.attr("y", barHeight / 2)
|
||||
.attr("dy", "0.35em");
|
||||
|
||||
enter.append("text")
|
||||
.attr("class", "rank-label")
|
||||
.attr("x", -40)
|
||||
.attr("y", barHeight / 2)
|
||||
.attr("dy", "0.35em")
|
||||
.attr("text-anchor", "middle");
|
||||
|
||||
// Update all bars
|
||||
const merged = enter.merge(bars);
|
||||
|
||||
merged.transition()
|
||||
.duration(animationSpeed * 0.8)
|
||||
.style("opacity", 1)
|
||||
.attr("transform", (d, i) => `translate(0,${i * (barHeight + 5)})`);
|
||||
|
||||
merged.select(".bar")
|
||||
.transition()
|
||||
.duration(animationSpeed * 0.8)
|
||||
.attr("width", d => xScale(d.rating))
|
||||
.attr("fill", d => colorScale(d.name));
|
||||
|
||||
merged.select(".bar-label")
|
||||
.text(d => d.name);
|
||||
|
||||
merged.select(".bar-value")
|
||||
.transition()
|
||||
.duration(animationSpeed * 0.8)
|
||||
.attr("x", d => xScale(d.rating) + 10)
|
||||
.text(d => `${Math.round(d.rating)} (${d.matches} matches)`);
|
||||
|
||||
merged.select(".rank-label")
|
||||
.text(d => `#${d.rank}`);
|
||||
}
|
||||
|
||||
function startAnimation() {
|
||||
if (currentFrame >= data.total_frames - 1) {
|
||||
currentFrame = 0;
|
||||
}
|
||||
|
||||
isPlaying = true;
|
||||
d3.select("#play-btn").property("disabled", true);
|
||||
d3.select("#pause-btn").property("disabled", false);
|
||||
|
||||
animationInterval = setInterval(() => {
|
||||
updateChart(currentFrame);
|
||||
currentFrame++;
|
||||
|
||||
if (currentFrame >= data.total_frames) {
|
||||
stopAnimation();
|
||||
currentFrame = data.total_frames - 1;
|
||||
}
|
||||
}, animationSpeed);
|
||||
}
|
||||
|
||||
function stopAnimation() {
|
||||
isPlaying = false;
|
||||
d3.select("#play-btn").property("disabled", false);
|
||||
d3.select("#pause-btn").property("disabled", true);
|
||||
|
||||
if (animationInterval) {
|
||||
clearInterval(animationInterval);
|
||||
animationInterval = null;
|
||||
}
|
||||
}
|
||||
|
||||
function resetAnimation() {
|
||||
stopAnimation();
|
||||
currentFrame = 0;
|
||||
updateChart(currentFrame);
|
||||
d3.select("#progress").style("width", "0%");
|
||||
}
|
||||
|
||||
// Button handlers
|
||||
d3.select("#play-btn").on("click", startAnimation);
|
||||
d3.select("#pause-btn").on("click", stopAnimation);
|
||||
d3.select("#reset-btn").on("click", resetAnimation);
|
||||
|
||||
// Initialize with first frame
|
||||
updateChart(0);
|
||||
|
||||
// Auto-play on load
|
||||
setTimeout(startAnimation, 1000);
|
||||
</script>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
# Keep generated evidence friendly to `git diff --check` and deterministic
|
||||
# across editors that otherwise strip indentation-only lines.
|
||||
html_template = "\n".join(line.rstrip() for line in html_template.splitlines()) + "\n"
|
||||
|
||||
# Write to file
|
||||
with open(output_path, 'w', encoding='utf-8') as f:
|
||||
f.write(html_template)
|
||||
|
||||
print(f"Generated animation HTML at: {output_path}")
|
||||
print(f"Open the file in a web browser to view the animation.")
|
||||
|
||||
|
||||
def create_simple_animation(history_df: pd.DataFrame, output_path: str = "leaderboard_animation.html", top_n: int = 15):
|
||||
"""
|
||||
Convenience function to create animation in one step.
|
||||
|
||||
Args:
|
||||
history_df: DataFrame with rating history
|
||||
output_path: Path to save HTML file
|
||||
top_n: Number of top models to show
|
||||
"""
|
||||
print("Preparing animation data...")
|
||||
animation_data = prepare_animation_data(history_df, top_n)
|
||||
|
||||
print(f"Generating HTML animation with {animation_data['total_frames']} frames...")
|
||||
generate_html_animation(animation_data, output_path)
|
||||
|
||||
return output_path
|
||||
@@ -0,0 +1,77 @@
|
||||
"""
|
||||
Synthetic pairwise battle generator (offline).
|
||||
|
||||
Generates head-to-head "battle" outcomes from a set of known latent skill
|
||||
scores, so the whole battles -> Elo -> leaderboard pipeline can be demonstrated
|
||||
end-to-end without downloading the 2GB Chatbot Arena dataset or calling any API.
|
||||
|
||||
Because the ground-truth skills are known, the recovered Elo leaderboard can be
|
||||
checked against them: the ranking should match, which validates the
|
||||
implementation. Ties are produced with a configurable probability to exercise
|
||||
the tie-handling paths in both the online Elo and Bradley-Terry code.
|
||||
"""
|
||||
import random
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
|
||||
# Default roster with plausible latent skills (in Elo points). The exact numbers
|
||||
# are only used to *generate* battles; the experiment then tries to recover them.
|
||||
DEFAULT_TRUE_SKILLS: Dict[str, float] = {
|
||||
"gpt-4": 1250.0,
|
||||
"claude-3-opus": 1225.0,
|
||||
"gemini-1.5-pro": 1180.0,
|
||||
"llama-3-70b": 1120.0,
|
||||
"mixtral-8x7b": 1075.0,
|
||||
"gpt-3.5-turbo": 1035.0,
|
||||
"llama-2-13b": 980.0,
|
||||
"vicuna-13b": 935.0,
|
||||
}
|
||||
|
||||
|
||||
def expected_score(rating_a: float, rating_b: float,
|
||||
base: float = 10.0, scale: float = 400.0) -> float:
|
||||
"""Bradley-Terry / Elo win probability of A against B."""
|
||||
return 1.0 / (1.0 + base ** ((rating_b - rating_a) / scale))
|
||||
|
||||
|
||||
def simulate_battles(true_skills: Dict[str, float],
|
||||
num_battles: int,
|
||||
tie_prob: float = 0.1,
|
||||
seed: Optional[int] = None) -> List[dict]:
|
||||
"""
|
||||
Simulate `num_battles` random pairwise battles.
|
||||
|
||||
For each battle two distinct models are drawn uniformly at random. With
|
||||
probability `tie_prob` the outcome is a tie; otherwise the winner is sampled
|
||||
according to the Bradley-Terry win probability implied by the latent skills
|
||||
(so upsets happen, but stronger models win more often).
|
||||
|
||||
Args:
|
||||
true_skills: Mapping of model name -> latent skill (Elo points).
|
||||
num_battles: Number of battles to generate.
|
||||
tie_prob: Probability that a battle ends in a tie.
|
||||
seed: Optional RNG seed for reproducibility.
|
||||
|
||||
Returns:
|
||||
List of dicts with keys 'model_a', 'model_b', 'winner'
|
||||
(winner in {'model_a', 'model_b', 'tie'}), matching the Chatbot Arena
|
||||
schema consumed by the Elo / Bradley-Terry code.
|
||||
"""
|
||||
if len(true_skills) < 2:
|
||||
raise ValueError("Need at least 2 models to simulate battles")
|
||||
|
||||
rng = random.Random(seed)
|
||||
models = list(true_skills.keys())
|
||||
battles: List[dict] = []
|
||||
|
||||
for _ in range(num_battles):
|
||||
model_a, model_b = rng.sample(models, 2)
|
||||
if rng.random() < tie_prob:
|
||||
winner = "tie"
|
||||
elif rng.random() < expected_score(true_skills[model_a], true_skills[model_b]):
|
||||
winner = "model_a"
|
||||
else:
|
||||
winner = "model_b"
|
||||
battles.append({"model_a": model_a, "model_b": model_b, "winner": winner})
|
||||
|
||||
return battles
|
||||
@@ -0,0 +1,144 @@
|
||||
"""
|
||||
Benchmark script to compare performance of different Elo implementations
|
||||
"""
|
||||
import time
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from elo_rating import EloRatingSystem
|
||||
from optimized_elo import build_leaderboard_optimized
|
||||
from data_loader import load_arena_data, filter_data
|
||||
|
||||
|
||||
def benchmark_basic_elo(df: pd.DataFrame) -> float:
|
||||
"""Benchmark the basic Elo implementation."""
|
||||
print("\n" + "="*80)
|
||||
print("Benchmarking Basic Elo Implementation (Python dict)")
|
||||
print("="*80)
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
for _, row in df.iterrows():
|
||||
elo.update_ratings(row['model_a'], row['model_b'], row['winner'])
|
||||
|
||||
end_time = time.time()
|
||||
elapsed = end_time - start_time
|
||||
|
||||
leaderboard = elo.get_leaderboard()
|
||||
|
||||
print(f"✓ Processed {len(df)} matches in {elapsed:.2f} seconds")
|
||||
print(f" Speed: {len(df)/elapsed:.0f} matches/second")
|
||||
print(f" Top 3 models: {[m[0] for m in leaderboard[:3]]}")
|
||||
|
||||
return elapsed
|
||||
|
||||
|
||||
def benchmark_optimized_elo(df: pd.DataFrame) -> float:
|
||||
"""Benchmark the NumPy + Numba optimized implementation."""
|
||||
print("\n" + "="*80)
|
||||
print("Benchmarking Optimized Elo Implementation (NumPy + Numba JIT)")
|
||||
print("="*80)
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
elo = build_leaderboard_optimized(
|
||||
df,
|
||||
initial_rating=1000.0,
|
||||
k_factor=32.0,
|
||||
show_progress=False
|
||||
)
|
||||
|
||||
end_time = time.time()
|
||||
elapsed = end_time - start_time
|
||||
|
||||
leaderboard = elo.get_leaderboard()
|
||||
|
||||
print(f"✓ Processed {len(df)} matches in {elapsed:.2f} seconds")
|
||||
print(f" Speed: {len(df)/elapsed:.0f} matches/second")
|
||||
print(f" Top 3 models: {[m[0] for m in leaderboard[:3]]}")
|
||||
|
||||
return elapsed
|
||||
|
||||
|
||||
def main():
|
||||
"""Run benchmark comparison."""
|
||||
print("="*80)
|
||||
print("ELO RATING COMPUTATION BENCHMARK")
|
||||
print("="*80)
|
||||
print("\nThis benchmark compares the performance of different Elo implementations")
|
||||
print("on Chatbot Arena voting data.\n")
|
||||
|
||||
# Load data
|
||||
print("Loading data...")
|
||||
try:
|
||||
df = load_arena_data("arena_data.json")
|
||||
except FileNotFoundError:
|
||||
print("Error: arena_data.json not found. Please run main.py first to download the data.")
|
||||
return
|
||||
|
||||
# Filter for blind votes
|
||||
print("Filtering data...")
|
||||
df_filtered = filter_data(df, anony_only=True, min_turn=1)
|
||||
|
||||
# Use a subset for quick benchmarking (can change to full dataset)
|
||||
sample_size = 50000
|
||||
if len(df_filtered) > sample_size:
|
||||
print(f"\nUsing a sample of {sample_size} matches for benchmarking")
|
||||
print("(To benchmark on full dataset, set sample_size = len(df_filtered))")
|
||||
df_sample = df_filtered.head(sample_size).copy()
|
||||
else:
|
||||
df_sample = df_filtered.copy()
|
||||
|
||||
print(f"\nBenchmark dataset: {len(df_sample)} matches")
|
||||
print(f"Unique models: {len(set(df_sample['model_a'].unique()) | set(df_sample['model_b'].unique()))}")
|
||||
|
||||
# Warm up Numba JIT (first run compiles the functions)
|
||||
print("\n" + "-"*80)
|
||||
print("Warming up Numba JIT compiler (first run)...")
|
||||
print("-"*80)
|
||||
df_tiny = df_sample.head(1000)
|
||||
build_leaderboard_optimized(df_tiny, show_progress=False)
|
||||
print("✓ JIT compilation complete")
|
||||
|
||||
# Run benchmarks
|
||||
time_basic = benchmark_basic_elo(df_sample)
|
||||
time_optimized = benchmark_optimized_elo(df_sample)
|
||||
|
||||
# Results summary
|
||||
print("\n" + "="*80)
|
||||
print("BENCHMARK RESULTS")
|
||||
print("="*80)
|
||||
|
||||
speedup = time_basic / time_optimized if time_optimized > 0 else 0
|
||||
|
||||
print(f"\nBasic Implementation: {time_basic:8.2f} seconds")
|
||||
print(f"Optimized Implementation: {time_optimized:8.2f} seconds")
|
||||
print(f"\nSpeedup: {speedup:.1f}x faster")
|
||||
pct_reduction = (1 - time_optimized / time_basic) * 100 if time_basic > 0 else 0.0
|
||||
print(f"Time saved: {time_basic - time_optimized:.2f} seconds ({pct_reduction:.1f}% reduction)")
|
||||
|
||||
# Extrapolate to full dataset
|
||||
if len(df_sample) > 0 and len(df_sample) < len(df_filtered):
|
||||
full_time_basic = time_basic * (len(df_filtered) / len(df_sample))
|
||||
full_time_optimized = time_optimized * (len(df_filtered) / len(df_sample))
|
||||
|
||||
print(f"\nExtrapolated times for full dataset ({len(df_filtered)} matches):")
|
||||
print(f" Basic: ~{full_time_basic/60:.1f} minutes")
|
||||
print(f" Optimized: ~{full_time_optimized/60:.1f} minutes")
|
||||
print(f" Time saved: ~{(full_time_basic - full_time_optimized)/60:.1f} minutes")
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("\nOptimization Techniques Applied:")
|
||||
print(" • NumPy arrays instead of Python dicts (O(1) integer indexing)")
|
||||
print(" • Numba JIT compilation (compiles hot loops to machine code)")
|
||||
print(" • Pre-allocated arrays (no dynamic memory allocation)")
|
||||
print(" • Integer model indices (no string lookups)")
|
||||
print(" • Vectorized operations where possible")
|
||||
print("\nFor the full optimized pipeline with parallel processing, run main_optimized.py")
|
||||
print("="*80 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
Bradley-Terry Model Implementation
|
||||
Official Chatbot Arena leaderboard calculation method
|
||||
"""
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
|
||||
|
||||
def compute_mle_elo(df: pd.DataFrame,
|
||||
SCALE: int = 400,
|
||||
BASE: int = 10,
|
||||
INIT_RATING: int = 1000,
|
||||
calibration_model: str | None = None,
|
||||
calibration_rating: int | None = None) -> pd.Series:
|
||||
"""
|
||||
Compute Elo ratings using Bradley-Terry model with Maximum Likelihood Estimation.
|
||||
|
||||
This is the official method used by Chatbot Arena for their leaderboard.
|
||||
It uses sklearn's LogisticRegression to fit a Bradley-Terry model.
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
SCALE: Elo scale parameter (default 400)
|
||||
BASE: Base for logistic function (default 10)
|
||||
INIT_RATING: Initial rating (default 1000)
|
||||
calibration_model: Model name to calibrate ratings to
|
||||
calibration_rating: Target rating for calibration model
|
||||
|
||||
Returns:
|
||||
Series of Elo ratings indexed by model name
|
||||
"""
|
||||
# Empty battle frame (e.g. --num-battles 0 or fully filtered input) is valid.
|
||||
if df is None or len(df) == 0:
|
||||
return pd.Series(dtype=float)
|
||||
|
||||
models = sorted({m for m in (set(df["model_a"]) | set(df["model_b"])) if pd.notna(m)})
|
||||
if len(models) <= 1:
|
||||
res_dict = {}
|
||||
for m in models:
|
||||
if calibration_model == m and calibration_rating is not None:
|
||||
res_dict[m] = float(calibration_rating)
|
||||
else:
|
||||
res_dict[m] = float(INIT_RATING)
|
||||
return pd.Series(res_dict, index=pd.Index(models), dtype=float)
|
||||
# Create pivot tables for wins
|
||||
ptbl_a_win = pd.pivot_table(
|
||||
df[df["winner"] == "model_a"],
|
||||
index="model_a",
|
||||
columns="model_b",
|
||||
aggfunc="size",
|
||||
fill_value=0,
|
||||
observed=False
|
||||
)
|
||||
|
||||
# Handle ties (including "tie (bothbad)"). Symmetrize only after aligning to
|
||||
# the full model square; (pivot + pivot.T) on a one-sided A×B pivot zeroes
|
||||
# every cell via pandas index/column alignment.
|
||||
if sum(df["winner"].isin(["tie", "tie (bothbad)"])) == 0:
|
||||
ptbl_tie = pd.DataFrame(0, index=ptbl_a_win.index, columns=ptbl_a_win.columns)
|
||||
else:
|
||||
ptbl_tie = pd.pivot_table(
|
||||
df[df["winner"].isin(["tie", "tie (bothbad)"])],
|
||||
index="model_a",
|
||||
columns="model_b",
|
||||
aggfunc="size",
|
||||
fill_value=0,
|
||||
observed=False
|
||||
)
|
||||
|
||||
ptbl_b_win = pd.pivot_table(
|
||||
df[df["winner"] == "model_b"],
|
||||
index="model_a",
|
||||
columns="model_b",
|
||||
aggfunc="size",
|
||||
fill_value=0,
|
||||
observed=False
|
||||
)
|
||||
|
||||
# Align pivots on the full model universe (small samples otherwise leave NaNs).
|
||||
models = sorted({m for m in (set(df["model_a"]) | set(df["model_b"])) if pd.notna(m)})
|
||||
ptbl_a_win = ptbl_a_win.reindex(index=models, columns=models, fill_value=0)
|
||||
ptbl_b_win = ptbl_b_win.reindex(index=models, columns=models, fill_value=0)
|
||||
ptbl_tie = ptbl_tie.reindex(index=models, columns=models, fill_value=0)
|
||||
ptbl_tie = ptbl_tie + ptbl_tie.T
|
||||
|
||||
# Compute win matrix (A wins * 2 + B wins * 2 + ties)
|
||||
ptbl_win = (ptbl_a_win * 2 + ptbl_b_win.T * 2 + ptbl_tie).fillna(0)
|
||||
|
||||
# Map models to indices
|
||||
models = pd.Series(np.arange(len(ptbl_win.index)), index=ptbl_win.index)
|
||||
|
||||
p = len(models)
|
||||
X = np.zeros([p * (p - 1) * 2, p])
|
||||
Y = np.zeros(p * (p - 1) * 2)
|
||||
|
||||
cur_row = 0
|
||||
sample_weights = []
|
||||
|
||||
for m_a in ptbl_win.index:
|
||||
for m_b in ptbl_win.columns:
|
||||
if m_a == m_b:
|
||||
continue
|
||||
# Skip if nan
|
||||
if math.isnan(ptbl_win.loc[m_a, m_b]) or math.isnan(ptbl_win.loc[m_b, m_a]):
|
||||
continue
|
||||
|
||||
X[cur_row, models[m_a]] = +math.log(BASE)
|
||||
X[cur_row, models[m_b]] = -math.log(BASE)
|
||||
Y[cur_row] = 1.0
|
||||
sample_weights.append(ptbl_win.loc[m_a, m_b])
|
||||
|
||||
X[cur_row + 1, models[m_a]] = math.log(BASE)
|
||||
X[cur_row + 1, models[m_b]] = -math.log(BASE)
|
||||
Y[cur_row + 1] = 0.0
|
||||
sample_weights.append(ptbl_win.loc[m_b, m_a])
|
||||
cur_row += 2
|
||||
|
||||
X = X[:cur_row]
|
||||
Y = Y[:cur_row]
|
||||
|
||||
# Fit logistic regression
|
||||
lr = LogisticRegression(fit_intercept=False, penalty=None, tol=1e-6)
|
||||
lr.fit(X, Y, sample_weight=sample_weights)
|
||||
|
||||
# Convert to Elo scores
|
||||
elo_scores = SCALE * lr.coef_[0] + INIT_RATING
|
||||
|
||||
# Calibrate to reference model if provided
|
||||
if calibration_model and calibration_model in models.index:
|
||||
target_rating = INIT_RATING if calibration_rating is None else calibration_rating
|
||||
elo_scores += target_rating - elo_scores[models[calibration_model]]
|
||||
|
||||
return pd.Series(elo_scores, index=models.index).sort_values(ascending=False)
|
||||
|
||||
|
||||
def predict_win_rate(elo_ratings: dict[str, float],
|
||||
SCALE: int = 400,
|
||||
BASE: int = 10) -> pd.DataFrame:
|
||||
"""
|
||||
Predict win rates between all model pairs using Elo ratings.
|
||||
|
||||
Args:
|
||||
elo_ratings: Dictionary of model names to Elo ratings
|
||||
SCALE: Elo scale parameter
|
||||
BASE: Base for logistic function
|
||||
|
||||
Returns:
|
||||
DataFrame with predicted win rates (row vs column)
|
||||
"""
|
||||
from collections import defaultdict
|
||||
|
||||
names = sorted(elo_ratings)
|
||||
wins = defaultdict(lambda: defaultdict(lambda: 0))
|
||||
|
||||
for a in names:
|
||||
for b in names:
|
||||
ea = 1 / (1 + BASE ** ((elo_ratings[b] - elo_ratings[a]) / SCALE))
|
||||
wins[a][b] = ea
|
||||
wins[b][a] = 1 - ea
|
||||
|
||||
data = {
|
||||
# np.nan, not np.NAN: the upper-case aliases were removed in NumPy 2.0
|
||||
# and requirements.txt allows numpy>=1.24 (i.e. 2.x).
|
||||
a: [wins[a][b] if a != b else np.nan for b in names]
|
||||
for a in names
|
||||
}
|
||||
|
||||
df = pd.DataFrame(data, index=names)
|
||||
df.index.name = "model_a"
|
||||
df.columns.name = "model_b"
|
||||
return df.T
|
||||
|
||||
|
||||
def get_bootstrap_result(battles: pd.DataFrame,
|
||||
func_compute_elo,
|
||||
num_round: int = 100,
|
||||
random_seed: int = 0) -> pd.DataFrame:
|
||||
"""
|
||||
Compute bootstrap confidence intervals for Elo ratings.
|
||||
|
||||
Args:
|
||||
battles: DataFrame with battle data
|
||||
func_compute_elo: Function to compute Elo ratings
|
||||
num_round: Number of bootstrap rounds
|
||||
|
||||
Returns:
|
||||
DataFrame with ratings from each bootstrap round
|
||||
"""
|
||||
from tqdm import tqdm
|
||||
|
||||
rows = []
|
||||
for i in tqdm(range(num_round), desc="Bootstrap sampling"):
|
||||
rows.append(
|
||||
func_compute_elo(
|
||||
battles.sample(frac=1.0, replace=True, random_state=random_seed + i)
|
||||
)
|
||||
)
|
||||
df = pd.DataFrame(rows)
|
||||
return df[df.median().sort_values(ascending=False).index]
|
||||
|
||||
|
||||
def compute_bradley_terry_leaderboard(df: pd.DataFrame,
|
||||
bootstrap_rounds: int = 0) -> pd.DataFrame:
|
||||
"""
|
||||
Compute leaderboard using Bradley-Terry model (official Chatbot Arena method).
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
bootstrap_rounds: Number of bootstrap rounds for confidence intervals (0 = no bootstrap)
|
||||
|
||||
Returns:
|
||||
DataFrame with model ratings (and confidence intervals if bootstrap > 0)
|
||||
"""
|
||||
print("Computing Bradley-Terry model ratings...")
|
||||
|
||||
# Compute MLE Elo ratings
|
||||
elo_ratings = compute_mle_elo(df)
|
||||
|
||||
if bootstrap_rounds > 0:
|
||||
print(f"Computing {bootstrap_rounds} bootstrap samples for confidence intervals...")
|
||||
bootstrap_df = get_bootstrap_result(df, compute_mle_elo, bootstrap_rounds)
|
||||
|
||||
# Compute confidence intervals
|
||||
result = pd.DataFrame({
|
||||
'rating': bootstrap_df.quantile(0.5),
|
||||
'lower_ci': bootstrap_df.quantile(0.025),
|
||||
'upper_ci': bootstrap_df.quantile(0.975)
|
||||
}).sort_values('rating', ascending=False)
|
||||
else:
|
||||
result = pd.DataFrame({
|
||||
'rating': elo_ratings
|
||||
}).sort_values('rating', ascending=False)
|
||||
|
||||
result.index.name = 'model'
|
||||
return result.reset_index()
|
||||
@@ -0,0 +1,343 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
实验 7-7：从配对比较数据构建模型排行榜 —— 命令行入口
|
||||
|
||||
统一的 argparse 命令行工具，把整个流程拆成三个子命令：
|
||||
|
||||
battle 运行两两对战，生成对战结果（模拟 / Chatbot Arena 真实数据 / LLM 评判）
|
||||
elo 从对战结果计算 Elo 或 Bradley-Terry 评分
|
||||
leaderboard 把对战结果或评分渲染成最终排行榜表格
|
||||
pipeline 一步跑完 对战 -> Elo -> 排行榜（默认离线可复现）
|
||||
|
||||
其中 battle 的 simulate/arena 来源与 elo、leaderboard、pipeline 均为纯离线计算，
|
||||
无需任何 API；只有 --source llm（LLM 评判对战）需要 LLM API Key：优先用官方
|
||||
Anthropic（ANTHROPIC_API_KEY），若无则自动回退到 OpenRouter（OPENROUTER_API_KEY），
|
||||
也可用 --judge-backend openrouter 强制走 OpenRouter（direct key 失效时）。
|
||||
|
||||
示例：
|
||||
# 离线一条龙：模拟对战 -> Elo -> 排行榜
|
||||
python cli.py pipeline
|
||||
|
||||
# 分步运行
|
||||
python cli.py battle --source simulate --num-battles 5000 --output battles.json
|
||||
python cli.py elo --input battles.json --method bradley-terry --bootstrap 100
|
||||
python cli.py leaderboard --input battles.json --top-n 20
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import warnings
|
||||
from typing import List, Optional
|
||||
|
||||
import pandas as pd
|
||||
|
||||
# Bradley-Terry 的 LogisticRegression 在新版 sklearn 会对 penalty=None 抛
|
||||
# FutureWarning；bootstrap 会重复上百次，这里静音以保持排行榜输出整洁。
|
||||
warnings.filterwarnings("ignore", category=FutureWarning, module="sklearn")
|
||||
|
||||
from battle_simulator import DEFAULT_TRUE_SKILLS, simulate_battles
|
||||
from elo_rating import EloRatingSystem
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 通用辅助函数
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _load_battles(path: str) -> pd.DataFrame:
|
||||
"""从 JSON 文件加载对战结果，返回带 model_a/model_b/winner 列的 DataFrame。"""
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
df = pd.DataFrame(data)
|
||||
# `[]` from --num-battles 0 has no columns; treat as empty battle frame.
|
||||
if len(df) == 0:
|
||||
return pd.DataFrame(columns=["model_a", "model_b", "winner"])
|
||||
required = {"model_a", "model_b", "winner"}
|
||||
if not required.issubset(df.columns):
|
||||
raise ValueError(
|
||||
f"对战文件 {path} 缺少必要字段 {required}，实际字段：{list(df.columns)}"
|
||||
)
|
||||
return df
|
||||
|
||||
|
||||
def _save_json(obj, path: str) -> None:
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(obj, f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def _battle_stats(df: pd.DataFrame) -> dict:
|
||||
"""统计每个模型的对战场数与胜场（平局按 0.5 计）。"""
|
||||
matches: dict = {}
|
||||
wins: dict = {}
|
||||
for model_a, model_b, winner in zip(df["model_a"], df["model_b"], df["winner"]):
|
||||
matches[model_a] = matches.get(model_a, 0) + 1
|
||||
matches[model_b] = matches.get(model_b, 0) + 1
|
||||
if winner == "model_a":
|
||||
wins[model_a] = wins.get(model_a, 0) + 1.0
|
||||
elif winner == "model_b":
|
||||
wins[model_b] = wins.get(model_b, 0) + 1.0
|
||||
else: # tie / tie (bothbad)
|
||||
wins[model_a] = wins.get(model_a, 0) + 0.5
|
||||
wins[model_b] = wins.get(model_b, 0) + 0.5
|
||||
return {"matches": matches, "wins": wins}
|
||||
|
||||
|
||||
def _compute_online_elo(df: pd.DataFrame, k: float, init_rating: float) -> pd.DataFrame:
|
||||
"""在线增量 Elo（按记录顺序处理），返回带 model/rating 列的 DataFrame。"""
|
||||
elo = EloRatingSystem(initial_rating=init_rating, k_factor=k)
|
||||
for model_a, model_b, winner in zip(df["model_a"], df["model_b"], df["winner"]):
|
||||
elo.update_ratings(model_a, model_b, winner)
|
||||
rows = [(m, r) for m, r, *_ in elo.get_leaderboard()]
|
||||
return pd.DataFrame(rows, columns=["model", "rating"])
|
||||
|
||||
|
||||
def _compute_bradley_terry(df: pd.DataFrame, bootstrap: int) -> pd.DataFrame:
|
||||
"""Bradley-Terry MLE 评分（可选 bootstrap 置信区间）。"""
|
||||
# 延迟导入：Bradley-Terry 依赖 scikit-learn，仅在需要时加载。
|
||||
from bradley_terry import compute_bradley_terry_leaderboard
|
||||
return compute_bradley_terry_leaderboard(df, bootstrap_rounds=bootstrap)
|
||||
|
||||
|
||||
def _compute_ratings(df: pd.DataFrame, method: str, k: float,
|
||||
init_rating: float, bootstrap: int) -> pd.DataFrame:
|
||||
if method == "bradley-terry":
|
||||
return _compute_bradley_terry(df, bootstrap)
|
||||
return _compute_online_elo(df, k, init_rating)
|
||||
|
||||
|
||||
def _print_leaderboard(ratings: pd.DataFrame, df: Optional[pd.DataFrame],
|
||||
top_n: int, title: str) -> None:
|
||||
"""打印最终排行榜表格。若评分含置信区间则展示 95% CI 列。"""
|
||||
has_ci = {"lower_ci", "upper_ci"}.issubset(ratings.columns)
|
||||
stats = _battle_stats(df) if df is not None else {"matches": {}, "wins": {}}
|
||||
|
||||
ratings = ratings.sort_values("rating", ascending=False).reset_index(drop=True)
|
||||
|
||||
print("=" * 78)
|
||||
print(title)
|
||||
print("=" * 78)
|
||||
if has_ci:
|
||||
header = f"{'排名':<6}{'模型':<24}{'Elo':>8} {'95% 置信区间':<20}{'场数':>7}{'胜率':>9}"
|
||||
else:
|
||||
header = f"{'排名':<6}{'模型':<24}{'Elo':>8} {'场数':>7}{'胜率':>9}"
|
||||
print(header)
|
||||
print("-" * 78)
|
||||
|
||||
for idx, row in ratings.head(top_n).iterrows():
|
||||
model = str(row["model"])
|
||||
n = stats["matches"].get(model, 0)
|
||||
w = stats["wins"].get(model, 0.0)
|
||||
win_rate = (w / n * 100.0) if n else 0.0
|
||||
if has_ci:
|
||||
ci = f"[{row['lower_ci']:.0f}, {row['upper_ci']:.0f}]"
|
||||
print(f"{idx + 1:<6}{model:<24}{row['rating']:>8.1f} "
|
||||
f"{ci:<20}{n:>7}{win_rate:>8.1f}%")
|
||||
else:
|
||||
print(f"{idx + 1:<6}{model:<24}{row['rating']:>8.1f} "
|
||||
f"{n:>7}{win_rate:>8.1f}%")
|
||||
print("-" * 78)
|
||||
print(f"共 {len(ratings)} 个模型，"
|
||||
f"评分范围 {ratings['rating'].min():.1f} ~ {ratings['rating'].max():.1f}")
|
||||
if has_ci:
|
||||
avg_ci = (ratings["upper_ci"] - ratings["lower_ci"]).mean()
|
||||
print(f"平均 95% 置信区间宽度：{avg_ci:.1f} 分")
|
||||
print()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 子命令实现
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _make_battles(args) -> List[dict]:
|
||||
if args.source == "simulate":
|
||||
skills = DEFAULT_TRUE_SKILLS
|
||||
if args.models:
|
||||
# 用户指定模型名时，围绕 1000 分等距分配潜在实力。
|
||||
n = len(args.models)
|
||||
skills = {m: 1000.0 + (n - 1 - 2 * i) * 40.0 for i, m in enumerate(args.models)}
|
||||
print(f"模拟 {args.num_battles} 场对战（{len(skills)} 个模型，"
|
||||
f"平局概率 {args.tie_prob}，随机种子 {args.seed}）...")
|
||||
battles = simulate_battles(skills, args.num_battles,
|
||||
tie_prob=args.tie_prob, seed=args.seed)
|
||||
print("真实潜在实力（用于事后对照）：")
|
||||
for m, s in sorted(skills.items(), key=lambda kv: -kv[1]):
|
||||
print(f" {m:<24}{s:>8.1f}")
|
||||
return battles
|
||||
|
||||
if args.source == "arena":
|
||||
from data_loader import load_arena_data, filter_data
|
||||
from parallel_processing import optimize_dataframe
|
||||
if not os.path.exists(args.arena_file):
|
||||
print(f"错误：找不到 Chatbot Arena 数据文件 {args.arena_file}。", file=sys.stderr)
|
||||
print("可从以下地址下载并保存为该文件名：", file=sys.stderr)
|
||||
print("https://storage.googleapis.com/arena_external_data/public/"
|
||||
"clean_battle_20240814_public.json", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
df = load_arena_data(args.arena_file)
|
||||
df = optimize_dataframe(df)
|
||||
df = filter_data(df, anony_only=True, use_dedup=True, min_turn=1)
|
||||
if args.sample and args.sample < len(df):
|
||||
df = df.sample(n=args.sample, random_state=args.seed).reset_index(drop=True)
|
||||
print(f"采样 {args.sample} 场对战。")
|
||||
return df[["model_a", "model_b", "winner"]].to_dict("records")
|
||||
|
||||
# source == "llm"
|
||||
from llm_judge import run_llm_battles
|
||||
print("运行 LLM 评判对战（顺序交换以消除位置偏差）...")
|
||||
return run_llm_battles(
|
||||
candidate_models=args.candidate_models,
|
||||
judge_model=args.judge_model,
|
||||
backend=args.judge_backend,
|
||||
)
|
||||
|
||||
|
||||
def cmd_battle(args) -> None:
|
||||
battles = _make_battles(args)
|
||||
_save_json(battles, args.output)
|
||||
print(f"\n已生成 {len(battles)} 场对战，写入 {args.output}")
|
||||
|
||||
|
||||
def cmd_elo(args) -> None:
|
||||
df = _load_battles(args.input)
|
||||
print(f"从 {args.input} 加载 {len(df)} 场对战，方法：{args.method}")
|
||||
ratings = _compute_ratings(df, args.method, args.k, args.init_rating, args.bootstrap)
|
||||
_print_leaderboard(ratings, df, top_n=args.top_n,
|
||||
title=f"Elo 评分（{args.method}）")
|
||||
if args.output:
|
||||
_save_json(ratings.to_dict("records"), args.output)
|
||||
print(f"评分已写入 {args.output}")
|
||||
|
||||
|
||||
def cmd_leaderboard(args) -> None:
|
||||
with open(args.input, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
sample = data[0] if isinstance(data, list) and data else {}
|
||||
if "rating" in sample: # 输入已是评分文件，直接展示。
|
||||
ratings = pd.DataFrame(data)
|
||||
_print_leaderboard(ratings, None, top_n=args.top_n, title="模型排行榜")
|
||||
return
|
||||
# 否则视为对战文件：先计算评分再展示。
|
||||
df = _load_battles(args.input)
|
||||
print(f"从 {args.input} 加载 {len(df)} 场对战，方法：{args.method}")
|
||||
ratings = _compute_ratings(df, args.method, args.k, args.init_rating, args.bootstrap)
|
||||
_print_leaderboard(ratings, df, top_n=args.top_n, title="模型排行榜")
|
||||
|
||||
|
||||
def cmd_pipeline(args) -> None:
|
||||
print("=" * 78)
|
||||
print("实验 7-7：对战 -> Elo -> 排行榜（端到端）")
|
||||
print("=" * 78)
|
||||
battles = _make_battles(args)
|
||||
if args.output:
|
||||
_save_json(battles, args.output)
|
||||
print(f"对战结果写入 {args.output}")
|
||||
df = pd.DataFrame(battles)
|
||||
print(f"\n用 {args.method} 方法从 {len(df)} 场对战计算评分...")
|
||||
ratings = _compute_ratings(df, args.method, args.k, args.init_rating, args.bootstrap)
|
||||
_print_leaderboard(ratings, df, top_n=args.top_n, title="最终排行榜")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 参数解析
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _add_source_args(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--source", choices=["simulate", "arena", "llm"],
|
||||
default="simulate",
|
||||
help="对战来源：simulate=离线模拟(默认)，arena=真实 Chatbot Arena 数据，"
|
||||
"llm=LLM 评判(需 API)")
|
||||
parser.add_argument("--models", nargs="+", default=None,
|
||||
help="simulate：自定义模型名列表（默认使用内置 8 个模型）")
|
||||
parser.add_argument("--num-battles", type=int, default=3000,
|
||||
help="simulate：模拟对战场数（默认 3000）")
|
||||
parser.add_argument("--tie-prob", type=float, default=0.1,
|
||||
help="simulate：平局概率（默认 0.1）")
|
||||
parser.add_argument("--seed", type=int, default=42,
|
||||
help="随机种子（默认 42）")
|
||||
parser.add_argument("--arena-file", default="arena_data.json",
|
||||
help="arena：Chatbot Arena 数据文件路径（默认 arena_data.json）")
|
||||
parser.add_argument("--sample", type=int, default=0,
|
||||
help="arena：随机采样 N 场对战，0 表示全部（默认 0）")
|
||||
parser.add_argument("--candidate-models", nargs="+", default=None,
|
||||
help="llm：参与对战的候选模型（默认 Claude 系列）")
|
||||
parser.add_argument("--judge-model", default="claude-opus-4-8",
|
||||
help="llm：评判模型（默认 claude-opus-4-8）")
|
||||
parser.add_argument("--judge-backend", choices=["anthropic", "openrouter", "auto"],
|
||||
default="auto",
|
||||
help="llm：评判后端。auto=有 ANTHROPIC_API_KEY 用官方 Anthropic，"
|
||||
"否则回退到 OpenRouter（OPENROUTER_API_KEY）；"
|
||||
"openrouter=强制走 OpenRouter（direct key 失效时用）")
|
||||
|
||||
|
||||
def _add_rating_args(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--method", choices=["online-elo", "bradley-terry"],
|
||||
default="online-elo",
|
||||
help="评分方法：online-elo=在线增量 Elo(默认)，"
|
||||
"bradley-terry=官方 MLE 拟合")
|
||||
parser.add_argument("--k", type=float, default=4.0,
|
||||
help="online-elo：K 因子/学习率（默认 4.0，官方取值）")
|
||||
parser.add_argument("--init-rating", type=float, default=1000.0,
|
||||
help="初始评分（默认 1000）")
|
||||
parser.add_argument("--bootstrap", type=int, default=0,
|
||||
help="bradley-terry：bootstrap 轮数以估计 95%% 置信区间（默认 0=不估计）")
|
||||
parser.add_argument("--top-n", type=int, default=20,
|
||||
help="排行榜展示的模型数量（默认 20）")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="cli.py",
|
||||
description="实验 7-7：从配对比较数据构建模型排行榜（对战 -> Elo -> 排行榜）",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=__doc__,
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", metavar="{battle,elo,leaderboard,pipeline}")
|
||||
|
||||
# battle
|
||||
p_battle = sub.add_parser("battle", help="运行两两对战，生成对战结果")
|
||||
_add_source_args(p_battle)
|
||||
p_battle.add_argument("--output", default="battles.json",
|
||||
help="对战结果输出文件（默认 battles.json）")
|
||||
p_battle.set_defaults(func=cmd_battle)
|
||||
|
||||
# elo
|
||||
p_elo = sub.add_parser("elo", help="从对战结果计算 Elo / Bradley-Terry 评分")
|
||||
p_elo.add_argument("--input", default="battles.json",
|
||||
help="对战结果输入文件（默认 battles.json）")
|
||||
_add_rating_args(p_elo)
|
||||
p_elo.add_argument("--output", default=None,
|
||||
help="把评分写入 JSON 文件（可选）")
|
||||
p_elo.set_defaults(func=cmd_elo)
|
||||
|
||||
# leaderboard
|
||||
p_lb = sub.add_parser("leaderboard", help="显示最终排行榜表格")
|
||||
p_lb.add_argument("--input", default="battles.json",
|
||||
help="对战结果或评分输入文件（默认 battles.json）")
|
||||
_add_rating_args(p_lb)
|
||||
p_lb.set_defaults(func=cmd_leaderboard)
|
||||
|
||||
# pipeline
|
||||
p_pipe = sub.add_parser("pipeline", help="一步跑完 对战 -> Elo -> 排行榜（默认离线）")
|
||||
_add_source_args(p_pipe)
|
||||
_add_rating_args(p_pipe)
|
||||
p_pipe.add_argument("--output", default=None,
|
||||
help="把对战结果写入 JSON 文件（可选）")
|
||||
p_pipe.set_defaults(func=cmd_pipeline)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Optional[List[str]] = None) -> None:
|
||||
parser = build_parser()
|
||||
# 无子命令时默认运行离线端到端演示，保留开箱即用体验。
|
||||
args = parser.parse_args(argv if argv is not None else (sys.argv[1:] or ["pipeline"]))
|
||||
try:
|
||||
args.func(args)
|
||||
except (RuntimeError, FileNotFoundError, ValueError) as exc:
|
||||
print(f"错误：{exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except Exception as exc: # 例如无效 ANTHROPIC_API_KEY 触发的 anthropic.AuthenticationError
|
||||
print(f"错误：{type(exc).__name__}: {exc}", file=sys.stderr)
|
||||
print("（若为 LLM 评审路径，请检查对应 provider 的 API key 是否有效）", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,223 @@
|
||||
"""
|
||||
Data loading and preprocessing for Chatbot Arena voting data
|
||||
"""
|
||||
import pandas as pd
|
||||
import requests
|
||||
import os
|
||||
from typing import Optional
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
def download_arena_data(output_path: str = "arena_data.json", force_download: bool = False) -> str:
|
||||
"""
|
||||
Download Chatbot Arena voting data via HTTPS.
|
||||
|
||||
Args:
|
||||
output_path: Path to save downloaded file
|
||||
force_download: If True, re-download even if file exists
|
||||
|
||||
Returns:
|
||||
Path to downloaded file
|
||||
"""
|
||||
if os.path.exists(output_path) and not force_download:
|
||||
print(f"Data file already exists at {output_path}")
|
||||
file_size = os.path.getsize(output_path) / (1024 * 1024)
|
||||
print(f"File size: {file_size:.2f} MB")
|
||||
return output_path
|
||||
|
||||
print("Downloading Chatbot Arena voting data...")
|
||||
url = "https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json"
|
||||
|
||||
try:
|
||||
# Stream download with progress bar
|
||||
response = requests.get(url, stream=True)
|
||||
response.raise_for_status()
|
||||
|
||||
# Get total file size
|
||||
total_size = int(response.headers.get('content-length', 0))
|
||||
|
||||
# Download with progress bar
|
||||
with open(output_path, 'wb') as f, tqdm(
|
||||
desc=output_path,
|
||||
total=total_size,
|
||||
unit='B',
|
||||
unit_scale=True,
|
||||
unit_divisor=1024,
|
||||
) as pbar:
|
||||
for chunk in response.iter_content(chunk_size=8192):
|
||||
if chunk:
|
||||
f.write(chunk)
|
||||
pbar.update(len(chunk))
|
||||
|
||||
file_size = os.path.getsize(output_path) / (1024 * 1024)
|
||||
print(f"\nDownloaded data to {output_path} ({file_size:.2f} MB)")
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error downloading data: {e}")
|
||||
print("Please ensure you have internet connection and the URL is accessible.")
|
||||
raise
|
||||
|
||||
|
||||
def load_arena_data(filepath: str) -> pd.DataFrame:
|
||||
"""
|
||||
Load and preprocess Chatbot Arena voting data.
|
||||
|
||||
Expected columns:
|
||||
- model_a: Identifier for first model
|
||||
- model_b: Identifier for second model
|
||||
- winner: Which model won ('model_a', 'model_b', or 'tie')
|
||||
- tstamp: Unix timestamp of the vote
|
||||
- judge: User who made the vote
|
||||
- turn: Conversation turn
|
||||
- anony: Whether vote was anonymous (blind)
|
||||
- language: Language of the conversation
|
||||
|
||||
Args:
|
||||
filepath: Path to data file
|
||||
|
||||
Returns:
|
||||
Preprocessed DataFrame sorted by timestamp
|
||||
"""
|
||||
print(f"Loading data from {filepath}...")
|
||||
print("Note: This is a large file (~2GB), loading may take 1-2 minutes...")
|
||||
|
||||
# Try different file formats
|
||||
if filepath.endswith('.json'):
|
||||
df = pd.read_json(filepath)
|
||||
elif filepath.endswith('.jsonl'):
|
||||
df = pd.read_json(filepath, lines=True)
|
||||
elif filepath.endswith('.csv'):
|
||||
df = pd.read_csv(filepath)
|
||||
else:
|
||||
# Try JSON by default
|
||||
try:
|
||||
df = pd.read_json(filepath)
|
||||
except (ValueError, KeyError):
|
||||
df = pd.read_json(filepath, lines=True)
|
||||
|
||||
print(f"Loaded {len(df)} records")
|
||||
print(f"Columns: {df.columns.tolist()}")
|
||||
|
||||
# Sort by timestamp
|
||||
if 'tstamp' in df.columns:
|
||||
df = df.sort_values('tstamp', ascending=True).reset_index(drop=True)
|
||||
print(f"Data spans from {pd.to_datetime(df['tstamp'].min(), unit='s')} to {pd.to_datetime(df['tstamp'].max(), unit='s')}")
|
||||
|
||||
# Basic statistics
|
||||
if 'winner' in df.columns:
|
||||
print(f"\nOutcome distribution:")
|
||||
print(df['winner'].value_counts())
|
||||
|
||||
if 'model_a' in df.columns and 'model_b' in df.columns:
|
||||
all_models = set(df['model_a'].unique()) | set(df['model_b'].unique())
|
||||
print(f"\nTotal unique models: {len(all_models)}")
|
||||
print(f"Top 10 models by appearance:")
|
||||
model_counts = pd.concat([df['model_a'], df['model_b']]).value_counts().head(10)
|
||||
print(model_counts)
|
||||
|
||||
return df
|
||||
|
||||
|
||||
def filter_data(df: pd.DataFrame,
|
||||
min_date: Optional[str] = None,
|
||||
max_date: Optional[str] = None,
|
||||
anony_only: bool = True,
|
||||
language: Optional[str] = None,
|
||||
min_turn: int = 1,
|
||||
use_dedup: bool = True) -> pd.DataFrame:
|
||||
"""
|
||||
Filter voting data based on various criteria (following official Chatbot Arena method).
|
||||
|
||||
Args:
|
||||
df: Input DataFrame
|
||||
min_date: Minimum date (YYYY-MM-DD format)
|
||||
max_date: Maximum date (YYYY-MM-DD format)
|
||||
anony_only: If True, only include anonymous (blind) votes
|
||||
language: If specified, filter by language
|
||||
min_turn: Minimum conversation turn
|
||||
use_dedup: If True, apply deduplication filter (official Arena method)
|
||||
|
||||
Returns:
|
||||
Filtered DataFrame
|
||||
"""
|
||||
filtered = df.copy()
|
||||
|
||||
print(f"Before filtering: {len(filtered)} records")
|
||||
|
||||
# Filter by anonymous votes only (official method)
|
||||
if anony_only and 'anony' in filtered.columns:
|
||||
filtered = filtered[filtered['anony'] == True]
|
||||
print(f" After anony filter: {len(filtered)} records")
|
||||
|
||||
# Apply deduplication (official method removes top 0.1% redundant prompts)
|
||||
if use_dedup and 'dedup_tag' in filtered.columns:
|
||||
try:
|
||||
filtered = filtered[filtered["dedup_tag"].apply(lambda x: x.get("sampled", False) if isinstance(x, dict) else False)]
|
||||
print(f" After dedup filter: {len(filtered)} records")
|
||||
except Exception as e:
|
||||
print(f" Warning: Could not apply dedup filter: {e}")
|
||||
|
||||
# Filter by date
|
||||
if 'tstamp' in filtered.columns:
|
||||
if min_date:
|
||||
min_timestamp = pd.to_datetime(min_date).timestamp()
|
||||
filtered = filtered[filtered['tstamp'] >= min_timestamp]
|
||||
if max_date:
|
||||
max_timestamp = pd.to_datetime(max_date).timestamp()
|
||||
filtered = filtered[filtered['tstamp'] <= max_timestamp]
|
||||
|
||||
# Filter by language
|
||||
if language and 'language' in filtered.columns:
|
||||
filtered = filtered[filtered['language'] == language]
|
||||
|
||||
# Filter by turn
|
||||
if 'turn' in filtered.columns:
|
||||
filtered = filtered[filtered['turn'] >= min_turn]
|
||||
|
||||
pct = (len(filtered) / len(df) * 100) if len(df) else 0.0
|
||||
print(f"After filtering: {len(filtered)} records ({pct:.1f}% of original)")
|
||||
return filtered.reset_index(drop=True)
|
||||
|
||||
|
||||
def get_time_slices(df: pd.DataFrame, interval: str = 'W') -> list:
|
||||
"""
|
||||
Split data into time slices for historical analysis.
|
||||
|
||||
Args:
|
||||
df: Input DataFrame with 'tstamp' column
|
||||
interval: Pandas frequency string ('D' for daily, 'W' for weekly, 'M' for monthly)
|
||||
|
||||
Returns:
|
||||
List of (end_date, dataframe_slice) tuples
|
||||
"""
|
||||
if 'tstamp' not in df.columns:
|
||||
raise ValueError("DataFrame must have 'tstamp' column")
|
||||
|
||||
if len(df) == 0:
|
||||
print(f"Created 0 time slices with interval '{interval}'")
|
||||
return []
|
||||
|
||||
df['datetime'] = pd.to_datetime(df['tstamp'], unit='s')
|
||||
min_date = df['datetime'].min()
|
||||
max_date = df['datetime'].max()
|
||||
|
||||
# Pandas 2.2+ removed 'M' (month-end); keep the documented monthly alias.
|
||||
freq = "ME" if interval == "M" else interval
|
||||
|
||||
# Generate date ranges
|
||||
date_ranges = pd.date_range(start=min_date, end=max_date, freq=freq)
|
||||
|
||||
slices = []
|
||||
for end_date in date_ranges:
|
||||
slice_df = df[df['datetime'] <= end_date].copy()
|
||||
if len(slice_df) > 0:
|
||||
slices.append((end_date, slice_df))
|
||||
|
||||
# Empty date_ranges when span < interval; also cover trailing gap to max_date.
|
||||
if len(date_ranges) == 0 or date_ranges[-1] < max_date:
|
||||
slices.append((max_date, df.copy()))
|
||||
|
||||
print(f"Created {len(slices)} time slices with interval '{interval}'")
|
||||
return slices
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
"""
|
||||
Elo Rating System Implementation
|
||||
Based on Bradley-Terry model for pairwise comparison
|
||||
"""
|
||||
import numpy as np
|
||||
from typing import Dict, Tuple, Optional
|
||||
|
||||
|
||||
class EloRatingSystem:
|
||||
"""
|
||||
Implementation of Elo rating system for model comparison.
|
||||
|
||||
The Elo system updates ratings based on pairwise comparison outcomes,
|
||||
where the rating difference between two models determines expected win probability.
|
||||
"""
|
||||
|
||||
def __init__(self, initial_rating: float = 1000.0, k_factor: float = 4.0):
|
||||
"""
|
||||
Initialize Elo rating system.
|
||||
|
||||
Args:
|
||||
initial_rating: Starting rating for all models
|
||||
k_factor: Learning rate controlling magnitude of rating updates
|
||||
"""
|
||||
self.initial_rating = initial_rating
|
||||
self.k_factor = k_factor
|
||||
self.ratings: Dict[str, float] = {}
|
||||
self.match_counts: Dict[str, int] = {}
|
||||
self.win_counts: Dict[str, float] = {} # ties add 0.5, so this is float
|
||||
|
||||
def get_rating(self, model: str) -> float:
|
||||
"""Get current rating for a model, initializing if necessary."""
|
||||
if model not in self.ratings:
|
||||
self.ratings[model] = self.initial_rating
|
||||
self.match_counts[model] = 0
|
||||
self.win_counts[model] = 0
|
||||
return self.ratings[model]
|
||||
|
||||
def expected_score(self, rating_a: float, rating_b: float) -> float:
|
||||
"""
|
||||
Calculate expected win probability for model A against model B.
|
||||
|
||||
Uses logistic function: P(A wins) = 1 / (1 + 10^((R_B - R_A)/400))
|
||||
|
||||
Args:
|
||||
rating_a: Rating of model A
|
||||
rating_b: Rating of model B
|
||||
|
||||
Returns:
|
||||
Expected probability that A wins (between 0 and 1)
|
||||
"""
|
||||
return 1.0 / (1.0 + 10.0 ** ((rating_b - rating_a) / 400.0))
|
||||
|
||||
def update_ratings(self, model_a: str, model_b: str, outcome: str) -> Tuple[float, float]:
|
||||
"""
|
||||
Update ratings after a match between two models.
|
||||
|
||||
Args:
|
||||
model_a: Identifier for first model
|
||||
model_b: Identifier for second model
|
||||
outcome: Match result ('model_a', 'model_b', or 'tie')
|
||||
|
||||
Returns:
|
||||
Tuple of (new_rating_a, new_rating_b)
|
||||
"""
|
||||
# Get current ratings
|
||||
rating_a = self.get_rating(model_a)
|
||||
rating_b = self.get_rating(model_b)
|
||||
|
||||
# Calculate expected scores
|
||||
expected_a = self.expected_score(rating_a, rating_b)
|
||||
expected_b = 1.0 - expected_a
|
||||
|
||||
# Determine actual scores
|
||||
if outcome == 'model_a':
|
||||
score_a, score_b = 1.0, 0.0
|
||||
self.win_counts[model_a] = self.win_counts.get(model_a, 0) + 1
|
||||
elif outcome == 'model_b':
|
||||
score_a, score_b = 0.0, 1.0
|
||||
self.win_counts[model_b] = self.win_counts.get(model_b, 0) + 1
|
||||
else: # tie
|
||||
score_a, score_b = 0.5, 0.5
|
||||
# A tie counts as half a win for each side, keeping win_counts (and
|
||||
# the win-rate derived from it) consistent with the 0.5-per-tie
|
||||
# convention used elsewhere (leaderboard win-rate matrix, CLI stats).
|
||||
self.win_counts[model_a] = self.win_counts.get(model_a, 0) + 0.5
|
||||
self.win_counts[model_b] = self.win_counts.get(model_b, 0) + 0.5
|
||||
|
||||
# Update ratings using Elo formula
|
||||
new_rating_a = rating_a + self.k_factor * (score_a - expected_a)
|
||||
new_rating_b = rating_b + self.k_factor * (score_b - expected_b)
|
||||
|
||||
# Store updated ratings
|
||||
self.ratings[model_a] = new_rating_a
|
||||
self.ratings[model_b] = new_rating_b
|
||||
|
||||
# Update match counts
|
||||
self.match_counts[model_a] = self.match_counts.get(model_a, 0) + 1
|
||||
self.match_counts[model_b] = self.match_counts.get(model_b, 0) + 1
|
||||
|
||||
return new_rating_a, new_rating_b
|
||||
|
||||
def get_leaderboard(self) -> list:
|
||||
"""
|
||||
Get current leaderboard sorted by rating.
|
||||
|
||||
Returns:
|
||||
List of tuples (model, rating, matches, wins) sorted by rating descending
|
||||
"""
|
||||
leaderboard = []
|
||||
for model in self.ratings:
|
||||
leaderboard.append((
|
||||
model,
|
||||
self.ratings[model],
|
||||
self.match_counts.get(model, 0),
|
||||
self.win_counts.get(model, 0)
|
||||
))
|
||||
|
||||
# Sort by rating descending
|
||||
leaderboard.sort(key=lambda x: x[1], reverse=True)
|
||||
return leaderboard
|
||||
|
||||
def calculate_win_probability(self, model_a: str, model_b: str) -> float:
|
||||
"""
|
||||
Calculate win probability of model_a against model_b based on current ratings.
|
||||
|
||||
Args:
|
||||
model_a: First model identifier
|
||||
model_b: Second model identifier
|
||||
|
||||
Returns:
|
||||
Probability that model_a wins (between 0 and 1)
|
||||
"""
|
||||
rating_a = self.get_rating(model_a)
|
||||
rating_b = self.get_rating(model_b)
|
||||
return self.expected_score(rating_a, rating_b)
|
||||
|
||||
def get_win_rate_matrix(self) -> Dict[Tuple[str, str], float]:
|
||||
"""
|
||||
Calculate pairwise win probability matrix for all models.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping (model_a, model_b) to win probability of model_a
|
||||
"""
|
||||
models = sorted(self.ratings.keys())
|
||||
matrix = {}
|
||||
|
||||
for model_a in models:
|
||||
for model_b in models:
|
||||
if model_a != model_b:
|
||||
prob = self.calculate_win_probability(model_a, model_b)
|
||||
matrix[(model_a, model_b)] = prob
|
||||
|
||||
return matrix
|
||||
|
||||
def reset(self):
|
||||
"""Reset all ratings to initial values."""
|
||||
self.ratings.clear()
|
||||
self.match_counts.clear()
|
||||
self.win_counts.clear()
|
||||
|
||||
def copy(self) -> 'EloRatingSystem':
|
||||
"""Create a deep copy of the current rating system."""
|
||||
new_system = EloRatingSystem(self.initial_rating, self.k_factor)
|
||||
new_system.ratings = self.ratings.copy()
|
||||
new_system.match_counts = self.match_counts.copy()
|
||||
new_system.win_counts = self.win_counts.copy()
|
||||
return new_system
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# 实验 7-7 环境变量示例
|
||||
#
|
||||
# 只有 `--source llm`（LLM 评判对战）需要 API Key；
|
||||
# simulate / arena / elo / leaderboard / pipeline 均为纯离线，无需任何 Key。
|
||||
#
|
||||
# 用法：把本文件复制为 .env 并填入真实 Key，或直接 `export` 到 shell。
|
||||
|
||||
# --- 评判后端 1：官方 Anthropic（默认优先）---
|
||||
# 有此 Key 时 --judge-backend auto 会走官方 Anthropic SDK。
|
||||
ANTHROPIC_API_KEY=your-anthropic-api-key
|
||||
|
||||
# --- 评判后端 2：OpenRouter 兜底 ---
|
||||
# 当 ANTHROPIC_API_KEY 缺失或失效时使用；用 OpenAI 兼容 SDK 指向 OpenRouter。
|
||||
# 内部 Claude 名字会自动映射为 OpenRouter id：
|
||||
# claude-opus-4-8 -> anthropic/claude-opus-4.8
|
||||
# claude-haiku-4-5 -> anthropic/claude-haiku-4.5
|
||||
# claude-sonnet-4-6 -> anthropic/claude-sonnet-4.6
|
||||
# 已含 '/' 的 id（如 openai/gpt-5.6-luna）原样透传。
|
||||
#
|
||||
# 强制走 OpenRouter：
|
||||
# python cli.py battle --source llm --judge-backend openrouter \
|
||||
# --judge-model claude-opus-4-8 \
|
||||
# --candidate-models anthropic/claude-haiku-4.5 openai/gpt-5.6-luna
|
||||
OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
@@ -0,0 +1,243 @@
|
||||
"""
|
||||
Leaderboard calculation and analysis
|
||||
"""
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from typing import Dict, List, Tuple
|
||||
from tqdm import tqdm
|
||||
from elo_rating import EloRatingSystem
|
||||
|
||||
|
||||
def build_leaderboard(df: pd.DataFrame,
|
||||
initial_rating: float = 1000.0,
|
||||
k_factor: float = 32.0,
|
||||
show_progress: bool = True) -> EloRatingSystem:
|
||||
"""
|
||||
Build Elo leaderboard from voting data.
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
initial_rating: Starting rating for all models
|
||||
k_factor: Elo learning rate
|
||||
show_progress: Whether to show progress bar
|
||||
|
||||
Returns:
|
||||
EloRatingSystem with final ratings
|
||||
"""
|
||||
elo = EloRatingSystem(initial_rating=initial_rating, k_factor=k_factor)
|
||||
|
||||
iterator = tqdm(df.iterrows(), total=len(df), desc="Processing matches") if show_progress else df.iterrows()
|
||||
|
||||
for idx, row in iterator:
|
||||
model_a = row['model_a']
|
||||
model_b = row['model_b']
|
||||
winner = row['winner']
|
||||
|
||||
elo.update_ratings(model_a, model_b, winner)
|
||||
|
||||
return elo
|
||||
|
||||
|
||||
def calculate_win_rate_matrix_from_data(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Calculate empirical win rate matrix directly from vote data.
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
|
||||
Returns:
|
||||
DataFrame with win rates (rows beat columns)
|
||||
"""
|
||||
# Get all unique models
|
||||
all_models = sorted(set(df['model_a'].unique()) | set(df['model_b'].unique()))
|
||||
|
||||
# Initialize counts
|
||||
wins = {model: {opponent: 0 for opponent in all_models} for model in all_models}
|
||||
total = {model: {opponent: 0 for opponent in all_models} for model in all_models}
|
||||
|
||||
# Count wins and totals
|
||||
for _, row in df.iterrows():
|
||||
model_a = row['model_a']
|
||||
model_b = row['model_b']
|
||||
winner = row['winner']
|
||||
|
||||
total[model_a][model_b] += 1
|
||||
total[model_b][model_a] += 1
|
||||
|
||||
if winner == 'model_a':
|
||||
wins[model_a][model_b] += 1
|
||||
elif winner == 'model_b':
|
||||
wins[model_b][model_a] += 1
|
||||
else: # tie
|
||||
wins[model_a][model_b] += 0.5
|
||||
wins[model_b][model_a] += 0.5
|
||||
|
||||
# Calculate win rates
|
||||
win_rates = {}
|
||||
for model in all_models:
|
||||
win_rates[model] = {}
|
||||
for opponent in all_models:
|
||||
if model == opponent:
|
||||
win_rates[model][opponent] = 0.5
|
||||
elif total[model][opponent] > 0:
|
||||
win_rates[model][opponent] = wins[model][opponent] / total[model][opponent]
|
||||
else:
|
||||
win_rates[model][opponent] = np.nan
|
||||
|
||||
# Convert to DataFrame
|
||||
win_rate_df = pd.DataFrame(win_rates).T
|
||||
win_rate_df = win_rate_df[all_models] # Ensure consistent ordering
|
||||
|
||||
return win_rate_df
|
||||
|
||||
|
||||
def compare_win_rates(elo_system: EloRatingSystem, empirical_df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Compare predicted win rates from Elo with empirical win rates.
|
||||
|
||||
Args:
|
||||
elo_system: Trained Elo rating system
|
||||
empirical_df: DataFrame with empirical win rates
|
||||
|
||||
Returns:
|
||||
DataFrame with comparison statistics
|
||||
"""
|
||||
models = empirical_df.index.tolist()
|
||||
|
||||
comparisons = []
|
||||
for model_a in models:
|
||||
for model_b in models:
|
||||
if model_a != model_b:
|
||||
empirical = empirical_df.loc[model_a, model_b]
|
||||
if not np.isnan(empirical):
|
||||
predicted = elo_system.calculate_win_probability(model_a, model_b)
|
||||
error = abs(predicted - empirical)
|
||||
comparisons.append({
|
||||
'model_a': model_a,
|
||||
'model_b': model_b,
|
||||
'empirical': empirical,
|
||||
'predicted': predicted,
|
||||
'error': error
|
||||
})
|
||||
|
||||
cols = ['model_a', 'model_b', 'empirical', 'predicted', 'error']
|
||||
if not comparisons:
|
||||
return pd.DataFrame(columns=cols)
|
||||
comparison_df = pd.DataFrame(comparisons, columns=cols)
|
||||
return comparison_df
|
||||
|
||||
|
||||
def build_historical_leaderboards(df: pd.DataFrame,
|
||||
time_slices: List[Tuple],
|
||||
initial_rating: float = 1000.0,
|
||||
k_factor: float = 32.0) -> List[Tuple]:
|
||||
"""
|
||||
Build leaderboard snapshots at different time points.
|
||||
|
||||
Args:
|
||||
df: Full voting DataFrame
|
||||
time_slices: List of (end_date, slice_df) tuples from get_time_slices
|
||||
initial_rating: Starting rating
|
||||
k_factor: Elo learning rate
|
||||
|
||||
Returns:
|
||||
List of (date, leaderboard_data) tuples
|
||||
"""
|
||||
historical_leaderboards = []
|
||||
|
||||
for end_date, slice_df in tqdm(time_slices, desc="Building historical leaderboards"):
|
||||
elo = build_leaderboard(slice_df, initial_rating, k_factor, show_progress=False)
|
||||
leaderboard = elo.get_leaderboard()
|
||||
|
||||
# Convert to DataFrame for easier handling
|
||||
lb_df = pd.DataFrame(leaderboard, columns=['model', 'rating', 'matches', 'wins'])
|
||||
lb_df['date'] = end_date
|
||||
lb_df['rank'] = range(1, len(lb_df) + 1)
|
||||
|
||||
historical_leaderboards.append((end_date, lb_df))
|
||||
|
||||
return historical_leaderboards
|
||||
|
||||
|
||||
def get_rating_history(historical_leaderboards: List[Tuple]) -> pd.DataFrame:
|
||||
"""
|
||||
Extract rating history for all models over time.
|
||||
|
||||
Args:
|
||||
historical_leaderboards: List of (date, leaderboard_df) tuples
|
||||
|
||||
Returns:
|
||||
DataFrame with columns: date, model, rating, rank
|
||||
"""
|
||||
all_data = []
|
||||
|
||||
for date, lb_df in historical_leaderboards:
|
||||
for _, row in lb_df.iterrows():
|
||||
all_data.append({
|
||||
'date': date,
|
||||
'model': row['model'],
|
||||
'rating': row['rating'],
|
||||
'rank': row['rank'],
|
||||
'matches': row['matches'],
|
||||
'wins': row['wins']
|
||||
})
|
||||
|
||||
cols = ['date', 'model', 'rating', 'rank', 'matches', 'wins']
|
||||
if not all_data:
|
||||
return pd.DataFrame(columns=cols)
|
||||
history_df = pd.DataFrame(all_data)
|
||||
return history_df
|
||||
|
||||
|
||||
def analyze_rating_changes(history_df: pd.DataFrame, top_n: int = 20) -> pd.DataFrame:
|
||||
"""
|
||||
Analyze rating changes over time for top models.
|
||||
|
||||
Args:
|
||||
history_df: DataFrame from get_rating_history
|
||||
top_n: Number of top models to analyze
|
||||
|
||||
Returns:
|
||||
DataFrame with change statistics
|
||||
"""
|
||||
stats_cols = [
|
||||
'model', 'final_rating', 'initial_rating', 'rating_change',
|
||||
'max_rating', 'min_rating', 'volatility', 'total_matches',
|
||||
]
|
||||
if history_df is None or len(history_df) == 0:
|
||||
return pd.DataFrame(columns=stats_cols)
|
||||
|
||||
# Get final ratings
|
||||
final_date = history_df['date'].max()
|
||||
final_ratings = history_df[history_df['date'] == final_date].nlargest(top_n, 'rating')
|
||||
top_models = final_ratings['model'].tolist()
|
||||
|
||||
# Calculate statistics for each model
|
||||
stats = []
|
||||
for model in top_models:
|
||||
model_data = history_df[history_df['model'] == model].sort_values('date')
|
||||
|
||||
if len(model_data) > 0:
|
||||
initial_rating = model_data.iloc[0]['rating']
|
||||
final_rating = model_data.iloc[-1]['rating']
|
||||
max_rating = model_data['rating'].max()
|
||||
min_rating = model_data['rating'].min()
|
||||
rating_change = final_rating - initial_rating
|
||||
volatility = model_data['rating'].std()
|
||||
|
||||
stats.append({
|
||||
'model': model,
|
||||
'final_rating': final_rating,
|
||||
'initial_rating': initial_rating,
|
||||
'rating_change': rating_change,
|
||||
'max_rating': max_rating,
|
||||
'min_rating': min_rating,
|
||||
'volatility': volatility,
|
||||
'total_matches': model_data.iloc[-1]['matches']
|
||||
})
|
||||
|
||||
if not stats:
|
||||
return pd.DataFrame(columns=stats_cols)
|
||||
stats_df = pd.DataFrame(stats).sort_values('final_rating', ascending=False)
|
||||
return stats_df
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
"""
|
||||
LLM-as-judge pairwise battles with position-bias mitigation.
|
||||
|
||||
This is the only battle source that needs network access; `simulate` and
|
||||
`arena` run fully offline.
|
||||
|
||||
Two backends are supported, selected automatically or via ``backend=``:
|
||||
|
||||
* ``anthropic`` – the official ``anthropic`` SDK, using ``ANTHROPIC_API_KEY``
|
||||
(the default when that key is present).
|
||||
* ``openrouter`` – the OpenAI-compatible ``openai`` SDK pointed at
|
||||
``https://openrouter.ai/api/v1`` with ``OPENROUTER_API_KEY``. Internal
|
||||
Claude ids (e.g. ``claude-opus-4-8``) are mapped to their OpenRouter ids
|
||||
(``anthropic/claude-opus-4.8``); ids that already contain a ``/`` such as
|
||||
``openai/gpt-5.6-luna`` are passed through untouched. This lets the judge run
|
||||
when a direct Anthropic key is missing or invalid.
|
||||
|
||||
The two backends are interchangeable: the position-bias swap-and-agree logic and
|
||||
the A/B/tie response parsing are identical regardless of which one is used.
|
||||
|
||||
The book (实验 7-7, 位置偏差 discussion) notes that an LLM judge systematically
|
||||
favours whichever answer appears in a fixed slot (usually the first). The
|
||||
standard mitigation, implemented here, is to judge each pair twice with the
|
||||
answers swapped and only record a winner when both judgements agree; a
|
||||
disagreement is counted as a tie. This cancels the position bias instead of
|
||||
letting it leak into the ratings.
|
||||
|
||||
The resulting battle list uses the same {'model_a', 'model_b', 'winner'} schema
|
||||
as the simulated and Chatbot Arena data, so it feeds straight into the Elo /
|
||||
Bradley-Terry pipeline.
|
||||
"""
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv()
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Default candidate roster and judge (Claude models). Kept small because every
|
||||
# battle costs several API calls (two responses + two swapped judgements).
|
||||
DEFAULT_CANDIDATE_MODELS = ["claude-opus-4-8", "claude-haiku-4-5"]
|
||||
DEFAULT_JUDGE_MODEL = "claude-opus-4-8"
|
||||
|
||||
DEFAULT_PROMPTS = [
|
||||
"用一句话解释什么是 Transformer 的自注意力机制。",
|
||||
"Write a haiku about distributed systems.",
|
||||
"给出快速排序的时间复杂度，并简要说明最坏情况。",
|
||||
]
|
||||
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
|
||||
# Map internal Claude ids -> OpenRouter model ids. Any id already containing a
|
||||
# '/' (e.g. 'openai/gpt-5.6-luna') is treated as a native OpenRouter id and used
|
||||
# verbatim; unknown ids are also passed through unchanged.
|
||||
_OPENROUTER_MODEL_MAP = {
|
||||
"claude-opus-4-8": "anthropic/claude-opus-4.8",
|
||||
"claude-opus-4-1": "anthropic/claude-opus-4.1",
|
||||
"claude-sonnet-4-6": "anthropic/claude-sonnet-4.6",
|
||||
"claude-sonnet-4-5": "anthropic/claude-sonnet-4.5",
|
||||
"claude-haiku-4-5": "anthropic/claude-haiku-4.5",
|
||||
}
|
||||
|
||||
_JUDGE_SYSTEM = (
|
||||
"你是一个严格的评委。用户会给你一个问题和两个候选回答（回答 A 和回答 B）。"
|
||||
"请只根据回答质量判断哪个更好，忽略它们出现的顺序。"
|
||||
"只输出一个词：A、B 或 tie。"
|
||||
)
|
||||
|
||||
|
||||
def _to_openrouter_model(model: str) -> str:
|
||||
"""Translate an internal model id into an OpenRouter model id."""
|
||||
if "/" in model: # already a native OpenRouter id
|
||||
return model
|
||||
return _OPENROUTER_MODEL_MAP.get(model, model)
|
||||
|
||||
|
||||
class JudgeClient:
|
||||
"""
|
||||
Thin adapter over either the Anthropic SDK or the OpenAI-compatible
|
||||
OpenRouter endpoint, exposing a single ``chat()`` method so the rest of the
|
||||
module is backend-agnostic.
|
||||
"""
|
||||
|
||||
def __init__(self, backend: str, impl):
|
||||
self.backend = backend
|
||||
self.impl = impl
|
||||
|
||||
def chat(self, model: str, user: str, max_tokens: int,
|
||||
system: Optional[str] = None) -> str:
|
||||
"""Send a single-turn chat and return the assistant's text reply."""
|
||||
if self.backend == "anthropic":
|
||||
kwargs = {
|
||||
"model": model,
|
||||
"max_tokens": max_tokens,
|
||||
"messages": [{"role": "user", "content": user}],
|
||||
}
|
||||
if system is not None:
|
||||
kwargs["system"] = system
|
||||
response = self.impl.messages.create(**kwargs)
|
||||
return "".join(
|
||||
block.text for block in response.content if block.type == "text"
|
||||
).strip()
|
||||
|
||||
# openrouter (OpenAI-compatible chat.completions)
|
||||
messages = []
|
||||
if system is not None:
|
||||
messages.append({"role": "system", "content": system})
|
||||
messages.append({"role": "user", "content": user})
|
||||
response = self.impl.chat.completions.create(
|
||||
model=_to_openrouter_model(model),
|
||||
max_tokens=max_tokens,
|
||||
messages=messages,
|
||||
)
|
||||
return (response.choices[0].message.content or "").strip()
|
||||
|
||||
|
||||
def _resolve_backend(backend: str = "auto") -> str:
|
||||
"""
|
||||
Resolve the effective backend.
|
||||
|
||||
``auto`` -> ``anthropic`` if ANTHROPIC_API_KEY is set, else ``openrouter``
|
||||
if OPENROUTER_API_KEY is set. Raises if neither key is available.
|
||||
"""
|
||||
if backend not in ("anthropic", "openrouter", "auto"):
|
||||
raise ValueError(
|
||||
f"Unknown judge backend {backend!r}; expected 'anthropic', "
|
||||
"'openrouter' or 'auto'."
|
||||
)
|
||||
if backend != "auto":
|
||||
return backend
|
||||
if os.environ.get("ANTHROPIC_API_KEY"):
|
||||
return "anthropic"
|
||||
if os.environ.get("OPENROUTER_API_KEY"):
|
||||
return "openrouter"
|
||||
raise RuntimeError(
|
||||
"No LLM-judge credentials found. Set ANTHROPIC_API_KEY (direct Anthropic) "
|
||||
"or OPENROUTER_API_KEY (OpenRouter fallback); or use --source simulate / "
|
||||
"--source arena to run the experiment fully offline."
|
||||
)
|
||||
|
||||
|
||||
def _get_client(backend: str = "auto") -> JudgeClient:
|
||||
"""Create a JudgeClient for the resolved backend, with clear errors."""
|
||||
backend = _resolve_backend(backend)
|
||||
|
||||
if backend == "anthropic":
|
||||
try:
|
||||
import anthropic
|
||||
except ImportError as exc: # pragma: no cover - depends on environment
|
||||
raise RuntimeError(
|
||||
"The 'anthropic' package is required for the anthropic judge "
|
||||
"backend. Install it with: pip install anthropic"
|
||||
) from exc
|
||||
if not os.environ.get("ANTHROPIC_API_KEY"):
|
||||
raise RuntimeError(
|
||||
"ANTHROPIC_API_KEY is not set. Set it, or use "
|
||||
"--judge-backend openrouter with OPENROUTER_API_KEY, or run "
|
||||
"--source simulate / --source arena fully offline."
|
||||
)
|
||||
return JudgeClient("anthropic", anthropic.Anthropic())
|
||||
|
||||
# backend == "openrouter"
|
||||
try:
|
||||
import openai
|
||||
except ImportError as exc: # pragma: no cover - depends on environment
|
||||
raise RuntimeError(
|
||||
"The 'openai' package is required for the openrouter judge backend. "
|
||||
"Install it with: pip install openai"
|
||||
) from exc
|
||||
if not os.environ.get("OPENROUTER_API_KEY"):
|
||||
raise RuntimeError(
|
||||
"OPENROUTER_API_KEY is not set. Set it, or use --judge-backend "
|
||||
"anthropic with ANTHROPIC_API_KEY, or run --source simulate / "
|
||||
"--source arena fully offline."
|
||||
)
|
||||
return JudgeClient(
|
||||
"openrouter",
|
||||
openai.OpenAI(
|
||||
base_url=OPENROUTER_BASE_URL,
|
||||
api_key=os.environ["OPENROUTER_API_KEY"],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def generate_response(client: JudgeClient, model: str, prompt: str,
|
||||
max_tokens: int = 1024) -> str:
|
||||
"""Generate a single model answer for a prompt."""
|
||||
return client.chat(model, prompt, max_tokens=max_tokens)
|
||||
|
||||
|
||||
def _judge_once(client: JudgeClient, judge_model: str, prompt: str,
|
||||
answer_first: str, answer_second: str) -> str:
|
||||
"""Ask the judge which slot is better; returns 'first', 'second' or 'tie'."""
|
||||
user = (
|
||||
f"问题：\n{prompt}\n\n"
|
||||
f"回答 A：\n{answer_first}\n\n"
|
||||
f"回答 B：\n{answer_second}\n\n"
|
||||
"哪个回答更好？只输出 A、B 或 tie。"
|
||||
)
|
||||
verdict = client.chat(judge_model, user, max_tokens=8, system=_JUDGE_SYSTEM).lower()
|
||||
if verdict.startswith("a"):
|
||||
return "first"
|
||||
if verdict.startswith("b"):
|
||||
return "second"
|
||||
return "tie"
|
||||
|
||||
|
||||
def judge_pair(client: JudgeClient, judge_model: str, prompt: str,
|
||||
answer_a: str, answer_b: str) -> str:
|
||||
"""
|
||||
Judge a pair with position-bias mitigation (swap order, tie on disagreement).
|
||||
|
||||
Returns 'model_a', 'model_b', or 'tie'.
|
||||
"""
|
||||
# First pass: A in slot 1, B in slot 2.
|
||||
first_pass = _judge_once(client, judge_model, prompt, answer_a, answer_b)
|
||||
# Second pass: swap the slots so B is now in slot 1.
|
||||
second_pass = _judge_once(client, judge_model, prompt, answer_b, answer_a)
|
||||
|
||||
# Translate both judgements into "which real model won", then require
|
||||
# agreement. Slot 1 in the first pass is A; slot 1 in the second pass is B.
|
||||
winner_first = {"first": "model_a", "second": "model_b", "tie": "tie"}[first_pass]
|
||||
winner_second = {"first": "model_b", "second": "model_a", "tie": "tie"}[second_pass]
|
||||
|
||||
if winner_first == winner_second:
|
||||
return winner_first
|
||||
return "tie" # inconsistent under swap -> position bias, count as tie
|
||||
|
||||
|
||||
def run_llm_battles(candidate_models: Optional[List[str]] = None,
|
||||
prompts: Optional[List[str]] = None,
|
||||
judge_model: str = DEFAULT_JUDGE_MODEL,
|
||||
backend: str = "auto") -> List[dict]:
|
||||
"""
|
||||
Run LLM-judged battles between every model pair over every prompt.
|
||||
|
||||
Args:
|
||||
candidate_models: Models to compare (default: DEFAULT_CANDIDATE_MODELS).
|
||||
prompts: Prompts to battle on (default: DEFAULT_PROMPTS).
|
||||
judge_model: Model used as the judge.
|
||||
backend: 'anthropic', 'openrouter', or 'auto' (anthropic if
|
||||
ANTHROPIC_API_KEY else openrouter).
|
||||
|
||||
Returns:
|
||||
List of battle dicts ({'model_a', 'model_b', 'winner'}).
|
||||
"""
|
||||
candidate_models = candidate_models or DEFAULT_CANDIDATE_MODELS
|
||||
prompts = prompts or DEFAULT_PROMPTS
|
||||
if len(candidate_models) < 2:
|
||||
raise ValueError("Need at least 2 candidate models for LLM-judge battles")
|
||||
|
||||
client = _get_client(backend)
|
||||
battles: List[dict] = []
|
||||
|
||||
for prompt in prompts:
|
||||
# Cache each model's answer per prompt so it is generated only once.
|
||||
answers: Dict[str, str] = {
|
||||
model: generate_response(client, model, prompt) for model in candidate_models
|
||||
}
|
||||
for i, model_a in enumerate(candidate_models):
|
||||
for model_b in candidate_models[i + 1:]:
|
||||
winner = judge_pair(
|
||||
client, judge_model, prompt, answers[model_a], answers[model_b]
|
||||
)
|
||||
battles.append(
|
||||
{"model_a": model_a, "model_b": model_b, "winner": winner}
|
||||
)
|
||||
|
||||
return battles
|
||||
@@ -0,0 +1,242 @@
|
||||
"""
|
||||
Main script for Model Leaderboard Calculation
|
||||
Experiment 7-7: Building Model Leaderboard from Pairwise Comparison Data
|
||||
|
||||
Supports two methods (following official Chatbot Arena):
|
||||
1. Online Elo (K=4) - Simple but order-dependent
|
||||
2. Bradley-Terry MLE - Official leaderboard method (more stable)
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pandas as pd
|
||||
from bradley_terry import (
|
||||
compute_bradley_terry_leaderboard,
|
||||
predict_win_rate,
|
||||
)
|
||||
from data_loader import download_arena_data, filter_data, load_arena_data
|
||||
from elo_rating import EloRatingSystem
|
||||
from parallel_processing import optimize_dataframe
|
||||
from visualization import plot_leaderboard, plot_rating_distribution, plot_win_rate_matrix
|
||||
|
||||
|
||||
def compute_online_elo_leaderboard(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Compute leaderboard using online Elo updates (K=4, official value).
|
||||
|
||||
This method updates ratings sequentially as matches are processed.
|
||||
It's simpler but can be unstable and order-dependent.
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
|
||||
Returns:
|
||||
DataFrame with model ratings
|
||||
"""
|
||||
from tqdm import tqdm
|
||||
|
||||
print("Computing online Elo ratings (K=4)...")
|
||||
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=4.0)
|
||||
|
||||
# Process matches sequentially
|
||||
for _, row in tqdm(df.iterrows(), total=len(df), desc="Processing matches"):
|
||||
elo.update_ratings(row['model_a'], row['model_b'], row['winner'])
|
||||
|
||||
# Get leaderboard
|
||||
leaderboard = elo.get_leaderboard()
|
||||
|
||||
# Convert to DataFrame
|
||||
result = pd.DataFrame(leaderboard, columns=['model', 'rating', 'matches', 'wins'])
|
||||
return result
|
||||
|
||||
|
||||
def main(method: str = 'bradley-terry'):
|
||||
"""
|
||||
Run model leaderboard calculation.
|
||||
|
||||
Args:
|
||||
method: 'bradley-terry' (default, official) or 'online-elo' (simple)
|
||||
"""
|
||||
|
||||
print("="*80)
|
||||
print("Experiment 7-7: Building Model Leaderboard from Pairwise Comparisons")
|
||||
if method == 'bradley-terry':
|
||||
print("Method: Bradley-Terry Model with MLE (Official Chatbot Arena)")
|
||||
else:
|
||||
print("Method: Online Elo Updates (K=4)")
|
||||
print("="*80)
|
||||
print()
|
||||
|
||||
# Step 1: Download and load data
|
||||
print("Step 1: Loading Chatbot Arena voting data...")
|
||||
print("-" * 80)
|
||||
|
||||
data_file = "arena_data.json"
|
||||
|
||||
try:
|
||||
# Download data if not exists
|
||||
if not os.path.exists(data_file):
|
||||
data_file = download_arena_data(data_file)
|
||||
|
||||
# Load data
|
||||
df = load_arena_data(data_file)
|
||||
|
||||
# Optimize memory usage
|
||||
df = optimize_dataframe(df)
|
||||
|
||||
except Exception as e: # noqa: BLE001 - surface loader/provider diagnostics to CLI users
|
||||
print(f"Error loading data: {e}")
|
||||
print("\nNote: If the data download fails, you can manually download the file from:")
|
||||
print("https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json")
|
||||
print("and save it as 'arena_data.json' in the current directory.")
|
||||
return
|
||||
|
||||
print()
|
||||
|
||||
# Step 2: Filter data (official Chatbot Arena method)
|
||||
print("Step 2: Filtering data (following official Arena method)...")
|
||||
print("-" * 80)
|
||||
|
||||
df_filtered = filter_data(
|
||||
df,
|
||||
anony_only=True, # Only anonymous/blind votes
|
||||
use_dedup=True, # Apply deduplication (removes top 0.1% redundant prompts)
|
||||
min_turn=1
|
||||
)
|
||||
|
||||
print()
|
||||
|
||||
# Step 3: Compute ratings using selected method
|
||||
print(f"Step 3: Computing ratings using {method} method...")
|
||||
print("-" * 80)
|
||||
|
||||
if method == 'bradley-terry':
|
||||
print("Note: Bradley-Terry model uses sklearn LogisticRegression for MLE.")
|
||||
print("This is the official Chatbot Arena method - more stable than online Elo.")
|
||||
print()
|
||||
|
||||
# Compute ratings with bootstrap for confidence intervals
|
||||
leaderboard_df = compute_bradley_terry_leaderboard(df_filtered, bootstrap_rounds=100)
|
||||
|
||||
print("\nTop 20 models by Bradley-Terry rating:")
|
||||
print("-" * 80)
|
||||
print(f"{'Rank':<6}{'Model':<35}{'Rating':<10}{'95% CI':<20}")
|
||||
print("-" * 80)
|
||||
for idx, row in leaderboard_df.head(20).iterrows():
|
||||
if 'lower_ci' in row and 'upper_ci' in row:
|
||||
ci_str = f"[{row['lower_ci']:.1f}, {row['upper_ci']:.1f}]"
|
||||
else:
|
||||
ci_str = "N/A"
|
||||
print(f"{idx+1:<6}{row['model']:<35}{row['rating']:7.1f} {ci_str:<20}")
|
||||
|
||||
else: # online-elo
|
||||
print("Note: Online Elo uses K=4 (official value) for stable ratings.")
|
||||
print("Processes matches sequentially - simpler but can be order-dependent.")
|
||||
print()
|
||||
|
||||
# Compute online Elo ratings
|
||||
leaderboard_df = compute_online_elo_leaderboard(df_filtered)
|
||||
|
||||
print("\nTop 20 models by Online Elo rating:")
|
||||
print("-" * 80)
|
||||
print(f"{'Rank':<6}{'Model':<35}{'Rating':<10}{'Matches':<10}{'Win Rate':<10}")
|
||||
print("-" * 80)
|
||||
for idx, row in leaderboard_df.head(20).iterrows():
|
||||
win_rate = row['wins'] / row['matches'] * 100 if row['matches'] > 0 else 0
|
||||
print(f"{idx+1:<6}{row['model']:<35}{row['rating']:7.1f} {row['matches']:<10}{win_rate:6.1f}%")
|
||||
|
||||
print()
|
||||
|
||||
# Step 4: Predict win rates using Bradley-Terry model
|
||||
print("Step 4: Calculating predicted win rates...")
|
||||
print("-" * 80)
|
||||
|
||||
# Get ratings as dictionary
|
||||
ratings_dict = dict(zip(leaderboard_df['model'], leaderboard_df['rating']))
|
||||
|
||||
# Predict win rates
|
||||
predicted_win_rates = predict_win_rate(ratings_dict)
|
||||
|
||||
print(f"Calculated predicted win rates for {len(ratings_dict)} models")
|
||||
print()
|
||||
|
||||
# Step 5: Create visualizations
|
||||
print("Step 5: Creating visualizations...")
|
||||
print("-" * 80)
|
||||
|
||||
# Convert leaderboard_df to format expected by visualization functions
|
||||
leaderboard_tuples = [(row['model'], row['rating'], 0, 0) for _, row in leaderboard_df.iterrows()]
|
||||
|
||||
plot_leaderboard(leaderboard_tuples, top_n=20, save_path="leaderboard.png")
|
||||
plot_rating_distribution(leaderboard_tuples, save_path="rating_distribution.png")
|
||||
|
||||
# Plot win rate matrix
|
||||
top_30_models = leaderboard_df.head(30)['model'].tolist()
|
||||
plot_win_rate_matrix(predicted_win_rates.loc[top_30_models, top_30_models],
|
||||
top_n=30, save_path="win_rate_matrix.png")
|
||||
|
||||
print()
|
||||
|
||||
# Summary
|
||||
print("="*80)
|
||||
print("Analysis complete!")
|
||||
print("="*80)
|
||||
print("\nGenerated files:")
|
||||
print(" - leaderboard.png : Top 20 models by Bradley-Terry rating")
|
||||
print(" - rating_distribution.png : Distribution of ratings")
|
||||
print(" - win_rate_matrix.png : Predicted win rate matrix (top 30 models)")
|
||||
print()
|
||||
|
||||
# Method summary
|
||||
print("Method:")
|
||||
print("-" * 80)
|
||||
if method == 'bradley-terry':
|
||||
print(" ✓ Bradley-Terry model with Maximum Likelihood Estimation")
|
||||
print(" ✓ sklearn LogisticRegression for stable rating computation")
|
||||
print(" ✓ Bootstrap confidence intervals (100 samples)")
|
||||
else:
|
||||
print(" ✓ Online Elo with K=4 (official value)")
|
||||
print(" ✓ Sequential match processing")
|
||||
print(" ✓ Simple but order-dependent")
|
||||
print(" ✓ Deduplication filter (removes top 0.1% redundant prompts)")
|
||||
print(" ✓ Anonymous votes only (blind evaluation)")
|
||||
print()
|
||||
|
||||
# Key insights
|
||||
print("Key Insights:")
|
||||
print("-" * 80)
|
||||
print(f" • Total models analyzed: {len(leaderboard_df)}")
|
||||
print(f" • Total battles: {len(df_filtered):,}")
|
||||
print(f" • Rating range: {leaderboard_df['rating'].min():.1f} - {leaderboard_df['rating'].max():.1f}")
|
||||
print(f" • Top model: {leaderboard_df.iloc[0]['model']} ({leaderboard_df.iloc[0]['rating']:.1f})")
|
||||
|
||||
if 'lower_ci' in leaderboard_df.columns:
|
||||
avg_ci_width = (leaderboard_df['upper_ci'] - leaderboard_df['lower_ci']).mean()
|
||||
print(f" • Average confidence interval width: {avg_ci_width:.1f} rating points")
|
||||
|
||||
print()
|
||||
print("This implementation matches the official Chatbot Arena leaderboard calculation!")
|
||||
print("Source: https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Allow method selection via command line argument
|
||||
import sys
|
||||
|
||||
method = 'bradley-terry' # Default to official method
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
if sys.argv[1] in ['bradley-terry', 'bt', 'mle']:
|
||||
method = 'bradley-terry'
|
||||
elif sys.argv[1] in ['online-elo', 'elo', 'online']:
|
||||
method = 'online-elo'
|
||||
else:
|
||||
print(f"Unknown method: {sys.argv[1]}")
|
||||
print("Usage: python main.py [bradley-terry|online-elo]")
|
||||
print(" bradley-terry (default): Official Arena method, more stable")
|
||||
print(" online-elo: Simple Elo updates with K=4")
|
||||
sys.exit(1)
|
||||
|
||||
main(method)
|
||||
@@ -0,0 +1,303 @@
|
||||
"""
|
||||
Optimized Elo rating system using NumPy vectorization and Numba JIT
|
||||
"""
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from typing import Dict, Tuple, List
|
||||
from numba import jit
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
@jit(nopython=True)
|
||||
def expected_score_fast(rating_a: float, rating_b: float) -> float:
|
||||
"""
|
||||
Fast expected score calculation using Numba JIT.
|
||||
|
||||
Args:
|
||||
rating_a: Rating of model A
|
||||
rating_b: Rating of model B
|
||||
|
||||
Returns:
|
||||
Expected probability that A wins
|
||||
"""
|
||||
return 1.0 / (1.0 + 10.0 ** ((rating_b - rating_a) / 400.0))
|
||||
|
||||
|
||||
@jit(nopython=True)
|
||||
def process_elo_updates_vectorized(ratings: np.ndarray,
|
||||
model_a_indices: np.ndarray,
|
||||
model_b_indices: np.ndarray,
|
||||
outcomes: np.ndarray,
|
||||
k_factor: float,
|
||||
match_counts: np.ndarray,
|
||||
win_counts: np.ndarray) -> np.ndarray:
|
||||
"""
|
||||
Process Elo updates using vectorized NumPy operations with Numba JIT.
|
||||
|
||||
This is the core hot loop optimized with Numba for maximum performance.
|
||||
|
||||
Args:
|
||||
ratings: Array of current ratings for all models
|
||||
model_a_indices: Indices of model A for each match
|
||||
model_b_indices: Indices of model B for each match
|
||||
outcomes: Match outcomes (1.0 = A wins, 0.0 = B wins, 0.5 = tie)
|
||||
k_factor: Elo K-factor
|
||||
match_counts: Array to track match counts per model
|
||||
win_counts: Array to track win counts per model
|
||||
|
||||
Returns:
|
||||
Updated ratings array
|
||||
"""
|
||||
n_matches = len(model_a_indices)
|
||||
|
||||
for i in range(n_matches):
|
||||
idx_a = model_a_indices[i]
|
||||
idx_b = model_b_indices[i]
|
||||
outcome = outcomes[i]
|
||||
|
||||
# Get current ratings
|
||||
rating_a = ratings[idx_a]
|
||||
rating_b = ratings[idx_b]
|
||||
|
||||
# Calculate expected scores
|
||||
expected_a = 1.0 / (1.0 + 10.0 ** ((rating_b - rating_a) / 400.0))
|
||||
expected_b = 1.0 - expected_a
|
||||
|
||||
# Update ratings
|
||||
ratings[idx_a] += k_factor * (outcome - expected_a)
|
||||
ratings[idx_b] += k_factor * ((1.0 - outcome) - expected_b)
|
||||
|
||||
# Update counts
|
||||
match_counts[idx_a] += 1
|
||||
match_counts[idx_b] += 1
|
||||
win_counts[idx_a] += outcome
|
||||
win_counts[idx_b] += (1.0 - outcome)
|
||||
|
||||
return ratings
|
||||
|
||||
|
||||
@jit(nopython=True)
|
||||
def calculate_expected_scores_vectorized(ratings_a: np.ndarray,
|
||||
ratings_b: np.ndarray) -> np.ndarray:
|
||||
"""
|
||||
Vectorized calculation of expected scores for multiple matches.
|
||||
|
||||
Args:
|
||||
ratings_a: Array of ratings for model A
|
||||
ratings_b: Array of ratings for model B
|
||||
|
||||
Returns:
|
||||
Array of expected scores for model A
|
||||
"""
|
||||
return 1.0 / (1.0 + np.power(10.0, (ratings_b - ratings_a) / 400.0))
|
||||
|
||||
|
||||
class NumpyEloRatingSystem:
|
||||
"""
|
||||
Highly optimized Elo rating system using NumPy arrays and Numba JIT.
|
||||
|
||||
Optimizations:
|
||||
- NumPy arrays for O(1) indexing instead of dictionary lookups
|
||||
- Numba JIT compilation of hot loops
|
||||
- Pre-allocated arrays to avoid memory reallocation
|
||||
- Integer indexing for models instead of string lookups
|
||||
"""
|
||||
|
||||
def __init__(self, initial_rating: float = 1000.0, k_factor: float = 4.0):
|
||||
"""Initialize NumPy-based Elo system."""
|
||||
self.initial_rating = initial_rating
|
||||
self.k_factor = k_factor
|
||||
|
||||
# Model name to index mapping
|
||||
self.model_to_idx: Dict[str, int] = {}
|
||||
self.idx_to_model: Dict[int, str] = {}
|
||||
|
||||
# NumPy arrays for fast access
|
||||
self.ratings: np.ndarray = None
|
||||
self.match_counts: np.ndarray = None
|
||||
self.win_counts: np.ndarray = None
|
||||
|
||||
self.n_models = 0
|
||||
|
||||
def _prepare_data(self, df: pd.DataFrame):
|
||||
"""
|
||||
Prepare NumPy arrays from DataFrame for fast processing.
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
"""
|
||||
print("Preparing data structures...")
|
||||
|
||||
# Get all unique models
|
||||
all_models = sorted(set(df['model_a'].unique()) | set(df['model_b'].unique()))
|
||||
self.n_models = len(all_models)
|
||||
|
||||
print(f"Found {self.n_models} unique models")
|
||||
|
||||
# Create model mappings
|
||||
for idx, model in enumerate(all_models):
|
||||
self.model_to_idx[model] = idx
|
||||
self.idx_to_model[idx] = model
|
||||
|
||||
# Initialize arrays
|
||||
self.ratings = np.full(self.n_models, self.initial_rating, dtype=np.float64)
|
||||
self.match_counts = np.zeros(self.n_models, dtype=np.int32)
|
||||
self.win_counts = np.zeros(self.n_models, dtype=np.float64)
|
||||
|
||||
# Convert DataFrame columns to NumPy arrays with integer indices
|
||||
print("Converting model names to indices...")
|
||||
model_a_indices = df['model_a'].map(self.model_to_idx).values.astype(np.int32)
|
||||
model_b_indices = df['model_b'].map(self.model_to_idx).values.astype(np.int32)
|
||||
|
||||
# Convert outcomes to numeric (1.0 for A wins, 0.0 for B wins, 0.5 for tie)
|
||||
print("Converting outcomes to numeric...")
|
||||
# 'tie (bothbad)' is a real Arena outcome; unmapped values become NaN and
|
||||
# would silently poison every rating they touch, so fall back to a tie
|
||||
# (same as EloRatingSystem.update_ratings).
|
||||
outcome_map = {'model_a': 1.0, 'model_b': 0.0,
|
||||
'tie': 0.5, 'tie (bothbad)': 0.5}
|
||||
outcomes = df['winner'].map(outcome_map).fillna(0.5).values.astype(np.float64)
|
||||
|
||||
return model_a_indices, model_b_indices, outcomes
|
||||
|
||||
def process_matches_vectorized(self, df: pd.DataFrame, show_progress: bool = True):
|
||||
"""
|
||||
Process all matches using vectorized NumPy operations and Numba JIT.
|
||||
|
||||
This is the fastest way to compute Elo ratings for large datasets.
|
||||
|
||||
Args:
|
||||
df: DataFrame with columns 'model_a', 'model_b', 'winner'
|
||||
show_progress: Whether to show progress bar
|
||||
"""
|
||||
# Prepare data
|
||||
model_a_indices, model_b_indices, outcomes = self._prepare_data(df)
|
||||
|
||||
print(f"\nProcessing {len(df)} matches with NumPy + Numba JIT...")
|
||||
|
||||
# Process all matches using JIT-compiled function
|
||||
# This is where the magic happens - Numba compiles this to machine code
|
||||
if show_progress:
|
||||
# Process in chunks to show progress
|
||||
chunk_size = 50000
|
||||
n_chunks = (len(model_a_indices) + chunk_size - 1) // chunk_size
|
||||
|
||||
for i in tqdm(range(n_chunks), desc="Processing matches"):
|
||||
start_idx = i * chunk_size
|
||||
end_idx = min((i + 1) * chunk_size, len(model_a_indices))
|
||||
|
||||
self.ratings = process_elo_updates_vectorized(
|
||||
self.ratings,
|
||||
model_a_indices[start_idx:end_idx],
|
||||
model_b_indices[start_idx:end_idx],
|
||||
outcomes[start_idx:end_idx],
|
||||
self.k_factor,
|
||||
self.match_counts,
|
||||
self.win_counts
|
||||
)
|
||||
else:
|
||||
self.ratings = process_elo_updates_vectorized(
|
||||
self.ratings,
|
||||
model_a_indices,
|
||||
model_b_indices,
|
||||
outcomes,
|
||||
self.k_factor,
|
||||
self.match_counts,
|
||||
self.win_counts
|
||||
)
|
||||
|
||||
print("✓ Processing complete!")
|
||||
|
||||
def get_leaderboard(self) -> List[Tuple]:
|
||||
"""
|
||||
Get sorted leaderboard using NumPy's fast sorting.
|
||||
|
||||
Returns:
|
||||
List of tuples (model, rating, matches, wins)
|
||||
"""
|
||||
# Use NumPy's argsort for fast sorting
|
||||
sorted_indices = np.argsort(-self.ratings) # Negative for descending order
|
||||
|
||||
leaderboard = []
|
||||
for idx in sorted_indices:
|
||||
model = self.idx_to_model[idx]
|
||||
rating = float(self.ratings[idx])
|
||||
matches = int(self.match_counts[idx])
|
||||
wins = float(self.win_counts[idx])
|
||||
leaderboard.append((model, rating, matches, wins))
|
||||
|
||||
return leaderboard
|
||||
|
||||
def calculate_win_probability(self, model_a: str, model_b: str) -> float:
|
||||
"""
|
||||
Calculate win probability using fast NumPy operations.
|
||||
|
||||
Args:
|
||||
model_a: First model identifier
|
||||
model_b: Second model identifier
|
||||
|
||||
Returns:
|
||||
Probability that model_a wins
|
||||
"""
|
||||
if model_a not in self.model_to_idx or model_b not in self.model_to_idx:
|
||||
return 0.5
|
||||
|
||||
idx_a = self.model_to_idx[model_a]
|
||||
idx_b = self.model_to_idx[model_b]
|
||||
|
||||
rating_a = self.ratings[idx_a]
|
||||
rating_b = self.ratings[idx_b]
|
||||
|
||||
return expected_score_fast(rating_a, rating_b)
|
||||
|
||||
def get_win_rate_matrix(self) -> Dict[Tuple[str, str], float]:
|
||||
"""
|
||||
Calculate pairwise win probability matrix using vectorized operations.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping (model_a, model_b) to win probability
|
||||
"""
|
||||
matrix = {}
|
||||
|
||||
# Vectorized calculation for all pairs
|
||||
for i in range(self.n_models):
|
||||
model_a = self.idx_to_model[i]
|
||||
|
||||
# Calculate win probabilities against all other models at once
|
||||
ratings_a = np.full(self.n_models, self.ratings[i])
|
||||
win_probs = calculate_expected_scores_vectorized(ratings_a, self.ratings)
|
||||
|
||||
for j in range(self.n_models):
|
||||
if i != j:
|
||||
model_b = self.idx_to_model[j]
|
||||
matrix[(model_a, model_b)] = float(win_probs[j])
|
||||
|
||||
return matrix
|
||||
|
||||
|
||||
def build_leaderboard_optimized(df: pd.DataFrame,
|
||||
initial_rating: float = 1000.0,
|
||||
k_factor: float = 4.0,
|
||||
show_progress: bool = True) -> NumpyEloRatingSystem:
|
||||
"""
|
||||
Build Elo leaderboard using highly optimized NumPy + Numba algorithm.
|
||||
|
||||
This implementation is significantly faster than the basic version:
|
||||
- Uses NumPy arrays for O(1) indexing
|
||||
- Numba JIT compilation for hot loops
|
||||
- Pre-allocated arrays to avoid memory overhead
|
||||
- Integer-based model indexing instead of string lookups
|
||||
|
||||
Args:
|
||||
df: DataFrame with match data (columns: model_a, model_b, winner)
|
||||
initial_rating: Starting rating for all models
|
||||
k_factor: Elo learning rate (K-factor)
|
||||
show_progress: Whether to display progress bar
|
||||
|
||||
Returns:
|
||||
NumpyEloRatingSystem with final ratings
|
||||
"""
|
||||
elo = NumpyEloRatingSystem(initial_rating=initial_rating, k_factor=k_factor)
|
||||
elo.process_matches_vectorized(df, show_progress=show_progress)
|
||||
return elo
|
||||
|
||||
@@ -0,0 +1,322 @@
|
||||
"""
|
||||
Parallel processing utilities for Elo rating computation
|
||||
"""
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from multiprocessing import Pool, cpu_count
|
||||
from functools import partial
|
||||
from typing import List, Tuple
|
||||
from tqdm import tqdm
|
||||
from elo_rating import EloRatingSystem
|
||||
|
||||
|
||||
def process_time_slice(args: Tuple) -> Tuple:
|
||||
"""
|
||||
Process a single time slice to build leaderboard.
|
||||
|
||||
Args:
|
||||
args: Tuple of (end_date, slice_df, initial_rating, k_factor)
|
||||
|
||||
Returns:
|
||||
Tuple of (end_date, leaderboard_data)
|
||||
"""
|
||||
end_date, slice_df, initial_rating, k_factor = args
|
||||
|
||||
# Build Elo system for this time slice
|
||||
elo = EloRatingSystem(initial_rating=initial_rating, k_factor=k_factor)
|
||||
|
||||
# Process all matches in this slice
|
||||
for _, row in slice_df.iterrows():
|
||||
elo.update_ratings(row['model_a'], row['model_b'], row['winner'])
|
||||
|
||||
# Get leaderboard
|
||||
leaderboard = elo.get_leaderboard()
|
||||
|
||||
# Convert to list of dicts for easier handling
|
||||
lb_data = []
|
||||
for rank, (model, rating, matches, wins) in enumerate(leaderboard, 1):
|
||||
lb_data.append({
|
||||
'model': model,
|
||||
'rating': rating,
|
||||
'matches': matches,
|
||||
'wins': wins,
|
||||
'rank': rank,
|
||||
'date': end_date
|
||||
})
|
||||
|
||||
return (end_date, lb_data)
|
||||
|
||||
|
||||
def build_historical_leaderboards_parallel(df: pd.DataFrame,
|
||||
time_slices: List[Tuple],
|
||||
initial_rating: float = 1000.0,
|
||||
k_factor: float = 32.0,
|
||||
n_jobs: int = -1) -> List[Tuple]:
|
||||
"""
|
||||
Build historical leaderboards using parallel processing.
|
||||
|
||||
Args:
|
||||
df: Full voting DataFrame
|
||||
time_slices: List of (end_date, slice_df) tuples
|
||||
initial_rating: Starting rating
|
||||
k_factor: Elo learning rate
|
||||
n_jobs: Number of parallel jobs (-1 for all cores)
|
||||
|
||||
Returns:
|
||||
List of (date, leaderboard_data) tuples
|
||||
"""
|
||||
if n_jobs == -1:
|
||||
n_jobs = cpu_count()
|
||||
|
||||
print(f"Building historical leaderboards using {n_jobs} cores...")
|
||||
|
||||
# Prepare arguments for parallel processing
|
||||
args_list = [
|
||||
(end_date, slice_df, initial_rating, k_factor)
|
||||
for end_date, slice_df in time_slices
|
||||
]
|
||||
|
||||
# Process in parallel
|
||||
with Pool(processes=n_jobs) as pool:
|
||||
results = list(tqdm(
|
||||
pool.imap(process_time_slice, args_list),
|
||||
total=len(args_list),
|
||||
desc="Processing time slices"
|
||||
))
|
||||
|
||||
# Convert results to expected format
|
||||
historical_leaderboards = []
|
||||
for end_date, lb_data in results:
|
||||
lb_df = pd.DataFrame(lb_data)
|
||||
historical_leaderboards.append((end_date, lb_df))
|
||||
|
||||
# Sort by date
|
||||
historical_leaderboards.sort(key=lambda x: x[0])
|
||||
|
||||
return historical_leaderboards
|
||||
|
||||
|
||||
def calculate_pairwise_win_rates_chunk(args: Tuple) -> List[dict]:
|
||||
"""
|
||||
Calculate win rates for a chunk of model pairs.
|
||||
|
||||
Args:
|
||||
args: Tuple of (model_pairs, df)
|
||||
|
||||
Returns:
|
||||
List of win rate dictionaries
|
||||
"""
|
||||
model_pairs, df = args
|
||||
|
||||
results = []
|
||||
for model_a, model_b in model_pairs:
|
||||
# Filter matches between these two models
|
||||
matches = df[
|
||||
((df['model_a'] == model_a) & (df['model_b'] == model_b)) |
|
||||
((df['model_a'] == model_b) & (df['model_b'] == model_a))
|
||||
]
|
||||
|
||||
if len(matches) == 0:
|
||||
continue
|
||||
|
||||
wins_a = 0
|
||||
total = len(matches)
|
||||
|
||||
for _, row in matches.iterrows():
|
||||
# Arena data has four winner values; any non-win outcome
|
||||
# ('tie' and 'tie (bothbad)') is worth 0.5, matching the
|
||||
# serial calculate_win_rate_matrix_from_data.
|
||||
if row['model_a'] == model_a:
|
||||
if row['winner'] == 'model_a':
|
||||
wins_a += 1
|
||||
elif row['winner'] != 'model_b':
|
||||
wins_a += 0.5
|
||||
else: # model_a is model_b in the row
|
||||
if row['winner'] == 'model_b':
|
||||
wins_a += 1
|
||||
elif row['winner'] != 'model_a':
|
||||
wins_a += 0.5
|
||||
|
||||
win_rate = wins_a / total if total > 0 else 0.5
|
||||
|
||||
results.append({
|
||||
'model_a': model_a,
|
||||
'model_b': model_b,
|
||||
'win_rate': win_rate,
|
||||
'total_matches': total
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def calculate_win_rate_matrix_parallel(df: pd.DataFrame,
|
||||
models: List[str] = None,
|
||||
n_jobs: int = -1) -> pd.DataFrame:
|
||||
"""
|
||||
Calculate win rate matrix using parallel processing.
|
||||
|
||||
Args:
|
||||
df: DataFrame with match data
|
||||
models: List of models to include (if None, use all)
|
||||
n_jobs: Number of parallel jobs
|
||||
|
||||
Returns:
|
||||
DataFrame with win rates
|
||||
"""
|
||||
if n_jobs == -1:
|
||||
n_jobs = cpu_count()
|
||||
|
||||
if models is None:
|
||||
models = sorted(set(df['model_a'].unique()) | set(df['model_b'].unique()))
|
||||
|
||||
print(f"Calculating win rate matrix for {len(models)} models using {n_jobs} cores...")
|
||||
|
||||
# Generate all model pairs
|
||||
model_pairs = [(m1, m2) for i, m1 in enumerate(models) for m2 in models[i+1:]]
|
||||
|
||||
# Split pairs into chunks for parallel processing
|
||||
chunk_size = max(1, len(model_pairs) // (n_jobs * 4))
|
||||
chunks = [model_pairs[i:i+chunk_size] for i in range(0, len(model_pairs), chunk_size)]
|
||||
|
||||
# Prepare arguments
|
||||
args_list = [(chunk, df) for chunk in chunks]
|
||||
|
||||
# Process in parallel
|
||||
with Pool(processes=n_jobs) as pool:
|
||||
results_chunks = list(tqdm(
|
||||
pool.imap(calculate_pairwise_win_rates_chunk, args_list),
|
||||
total=len(args_list),
|
||||
desc="Calculating win rates"
|
||||
))
|
||||
|
||||
# Flatten results
|
||||
all_results = [item for chunk in results_chunks for item in chunk]
|
||||
|
||||
# Build matrix. Pairs with no data stay NaN (the serial version's
|
||||
# convention) — 0.5 would misreport "no data" as an even record;
|
||||
# the diagonal is 0.5 by definition.
|
||||
win_rates = {model: {opponent: (0.5 if opponent == model else np.nan)
|
||||
for opponent in models} for model in models}
|
||||
|
||||
for result in all_results:
|
||||
model_a = result['model_a']
|
||||
model_b = result['model_b']
|
||||
win_rate = result['win_rate']
|
||||
|
||||
win_rates[model_a][model_b] = win_rate
|
||||
win_rates[model_b][model_a] = 1.0 - win_rate
|
||||
|
||||
# Convert to DataFrame
|
||||
win_rate_df = pd.DataFrame(win_rates).T
|
||||
win_rate_df = win_rate_df[models]
|
||||
|
||||
return win_rate_df
|
||||
|
||||
|
||||
def filter_data_parallel(df: pd.DataFrame,
|
||||
filters: dict,
|
||||
n_jobs: int = -1) -> pd.DataFrame:
|
||||
"""
|
||||
Filter large DataFrame using parallel processing.
|
||||
|
||||
Args:
|
||||
df: Input DataFrame
|
||||
filters: Dictionary of filter conditions
|
||||
n_jobs: Number of parallel jobs
|
||||
|
||||
Returns:
|
||||
Filtered DataFrame
|
||||
"""
|
||||
if n_jobs == -1:
|
||||
n_jobs = min(cpu_count(), 4) # Cap at 4 for filtering
|
||||
|
||||
if len(df) == 0:
|
||||
return df.copy()
|
||||
|
||||
n_jobs = max(1, min(n_jobs, len(df)))
|
||||
|
||||
# Split DataFrame into chunks
|
||||
chunk_size = max(1, len(df) // n_jobs)
|
||||
chunks = [df.iloc[i:i+chunk_size] for i in range(0, len(df), chunk_size)]
|
||||
|
||||
def apply_filters(chunk):
|
||||
filtered = chunk.copy()
|
||||
|
||||
# Apply each filter
|
||||
if 'anony_only' in filters and filters['anony_only'] and 'anony' in filtered.columns:
|
||||
filtered = filtered[filtered['anony'] == True]
|
||||
|
||||
if 'language' in filters and filters['language'] and 'language' in filtered.columns:
|
||||
filtered = filtered[filtered['language'] == filters['language']]
|
||||
|
||||
if 'min_turn' in filters and 'turn' in filtered.columns:
|
||||
filtered = filtered[filtered['turn'] >= filters['min_turn']]
|
||||
|
||||
if 'min_date' in filters and 'tstamp' in filtered.columns:
|
||||
min_timestamp = pd.to_datetime(filters['min_date']).timestamp()
|
||||
filtered = filtered[filtered['tstamp'] >= min_timestamp]
|
||||
|
||||
if 'max_date' in filters and 'tstamp' in filtered.columns:
|
||||
max_timestamp = pd.to_datetime(filters['max_date']).timestamp()
|
||||
filtered = filtered[filtered['tstamp'] <= max_timestamp]
|
||||
|
||||
return filtered
|
||||
|
||||
# Process chunks in parallel
|
||||
with Pool(processes=n_jobs) as pool:
|
||||
filtered_chunks = pool.map(apply_filters, chunks)
|
||||
|
||||
# Combine results
|
||||
result = pd.concat(filtered_chunks, ignore_index=True)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def optimize_dataframe(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Optimize DataFrame memory usage by downcasting numeric types.
|
||||
|
||||
Args:
|
||||
df: Input DataFrame
|
||||
|
||||
Returns:
|
||||
Optimized DataFrame
|
||||
"""
|
||||
print("Optimizing DataFrame memory usage...")
|
||||
|
||||
initial_memory = df.memory_usage(deep=True).sum() / 1024**2
|
||||
|
||||
# Optimize numeric columns
|
||||
for col in df.columns:
|
||||
col_type = df[col].dtype
|
||||
|
||||
if col_type == 'int64':
|
||||
df[col] = pd.to_numeric(df[col], downcast='integer')
|
||||
elif col_type == 'float64':
|
||||
df[col] = pd.to_numeric(df[col], downcast='float')
|
||||
|
||||
# Convert string columns to category if they have few unique values
|
||||
for col in df.select_dtypes(include=['object']).columns:
|
||||
try:
|
||||
# Check if column contains hashable types (not dict, list, etc.)
|
||||
# Try to get unique values - will fail if unhashable
|
||||
num_unique = df[col].nunique()
|
||||
num_total = len(df[col])
|
||||
if num_total == 0:
|
||||
continue
|
||||
|
||||
# If less than 50% unique values, convert to category
|
||||
if num_unique / num_total < 0.5:
|
||||
df[col] = df[col].astype('category')
|
||||
except (TypeError, AttributeError):
|
||||
# Column contains unhashable types (dicts, lists), skip optimization
|
||||
print(f" Skipping column '{col}' (contains complex data types)")
|
||||
continue
|
||||
|
||||
final_memory = df.memory_usage(deep=True).sum() / 1024**2
|
||||
reduction = 0.0 if initial_memory == 0 else (1 - final_memory / initial_memory) * 100
|
||||
|
||||
print(f"Memory usage reduced from {initial_memory:.2f} MB to {final_memory:.2f} MB ({reduction:.1f}% reduction)")
|
||||
|
||||
return df
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
"""
|
||||
Quick start demo - minimal example to get started quickly
|
||||
"""
|
||||
from elo_rating import EloRatingSystem
|
||||
|
||||
|
||||
def demo_basic_elo():
|
||||
"""Demonstrate basic Elo rating calculation with synthetic data."""
|
||||
|
||||
print("="*60)
|
||||
print("Quick Start: Elo Rating System Demo")
|
||||
print("="*60)
|
||||
print()
|
||||
|
||||
# Initialize Elo system
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
# Simulate some matches
|
||||
matches = [
|
||||
("GPT-4", "Claude-v1", "GPT-4"),
|
||||
("GPT-4", "Llama-2", "GPT-4"),
|
||||
("Claude-v1", "Llama-2", "Claude-v1"),
|
||||
("GPT-4", "Claude-v1", "tie"),
|
||||
("Llama-2", "Gemini", "Gemini"),
|
||||
("GPT-4", "Gemini", "GPT-4"),
|
||||
("Claude-v1", "Gemini", "Claude-v1"),
|
||||
("GPT-4", "Llama-2", "GPT-4"),
|
||||
("Claude-v1", "Llama-2", "Claude-v1"),
|
||||
("Gemini", "Llama-2", "Gemini"),
|
||||
]
|
||||
|
||||
print("Processing matches:")
|
||||
print("-" * 60)
|
||||
for i, (model_a, model_b, winner) in enumerate(matches, 1):
|
||||
old_rating_a = elo.get_rating(model_a)
|
||||
old_rating_b = elo.get_rating(model_b)
|
||||
|
||||
# update_ratings expects 'model_a' / 'model_b' / 'tie', not the
|
||||
# winning model's name (anything unrecognized is scored as a tie).
|
||||
outcome = ("model_a" if winner == model_a
|
||||
else "model_b" if winner == model_b else "tie")
|
||||
new_rating_a, new_rating_b = elo.update_ratings(model_a, model_b, outcome)
|
||||
|
||||
print(f"Match {i}: {model_a} vs {model_b} -> {winner} wins")
|
||||
print(f" {model_a}: {old_rating_a:.1f} → {new_rating_a:.1f} ({new_rating_a-old_rating_a:+.1f})")
|
||||
print(f" {model_b}: {old_rating_b:.1f} → {new_rating_b:.1f} ({new_rating_b-old_rating_b:+.1f})")
|
||||
print()
|
||||
|
||||
# Show final leaderboard
|
||||
print("=" * 60)
|
||||
print("Final Leaderboard:")
|
||||
print("=" * 60)
|
||||
leaderboard = elo.get_leaderboard()
|
||||
for rank, (model, rating, matches, wins) in enumerate(leaderboard, 1):
|
||||
win_rate = (wins / matches * 100) if matches > 0 else 0
|
||||
print(f"{rank}. {model:15s} - Rating: {rating:7.1f} | "
|
||||
f"Matches: {matches:2d} | Wins: {wins:4.1f} | Win Rate: {win_rate:5.1f}%")
|
||||
|
||||
print()
|
||||
|
||||
# Show win probability predictions
|
||||
print("=" * 60)
|
||||
print("Win Probability Predictions:")
|
||||
print("=" * 60)
|
||||
|
||||
models = [m[0] for m in leaderboard]
|
||||
for i, model_a in enumerate(models):
|
||||
for model_b in models[i+1:]:
|
||||
prob = elo.calculate_win_probability(model_a, model_b)
|
||||
print(f"{model_a} vs {model_b}: {prob*100:.1f}% - {(1-prob)*100:.1f}%")
|
||||
|
||||
print()
|
||||
print("=" * 60)
|
||||
print("Demo complete! Check main.py for full analysis with real data.")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
demo_basic_elo()
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
pandas>=2.0.0
|
||||
numpy>=1.24.0
|
||||
matplotlib>=3.7.0
|
||||
seaborn>=0.12.0
|
||||
requests>=2.31.0
|
||||
plotly>=5.14.0
|
||||
tqdm>=4.65.0
|
||||
scikit-learn>=1.3.0
|
||||
python-dotenv>=1.0.0
|
||||
@@ -0,0 +1,10 @@
|
||||
"""Helpers for direct execution of tests moved under tests/."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
def bootstrap_experiment_root() -> None:
|
||||
experiment_root = Path(__file__).resolve().parents[1]
|
||||
if str(experiment_root) not in sys.path:
|
||||
sys.path.insert(0, str(experiment_root))
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Test import bootstrap for the elo-leaderboard experiment."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
EXPERIMENT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(EXPERIMENT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(EXPERIMENT_ROOT))
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Empty rating history must not crash analyze_rating_changes / get_rating_history."""
|
||||
import pandas as pd
|
||||
|
||||
from animation import prepare_animation_data
|
||||
from leaderboard import (
|
||||
analyze_rating_changes,
|
||||
build_historical_leaderboards,
|
||||
get_rating_history,
|
||||
)
|
||||
|
||||
|
||||
def test_get_rating_history_empty_keeps_columns():
|
||||
hist = build_historical_leaderboards(
|
||||
pd.DataFrame(columns=["model_a", "model_b", "winner"]),
|
||||
[(pd.Timestamp("2020-01-01"), pd.DataFrame(columns=["model_a", "model_b", "winner"]))],
|
||||
)
|
||||
rh = get_rating_history(hist)
|
||||
assert list(rh.columns) == ["date", "model", "rating", "rank", "matches", "wins"]
|
||||
assert len(rh) == 0
|
||||
|
||||
|
||||
def test_analyze_empty_history_returns_empty_frame():
|
||||
empty = pd.DataFrame(columns=["date", "model", "rating", "rank", "matches", "wins"])
|
||||
stats = analyze_rating_changes(empty)
|
||||
assert len(stats) == 0
|
||||
assert "model" in stats.columns
|
||||
|
||||
|
||||
def test_analyze_after_empty_historical_leaderboards():
|
||||
hist = build_historical_leaderboards(
|
||||
pd.DataFrame(columns=["model_a", "model_b", "winner"]),
|
||||
[(pd.Timestamp("2020-01-01"), pd.DataFrame(columns=["model_a", "model_b", "winner"]))],
|
||||
)
|
||||
rh = get_rating_history(hist)
|
||||
stats = analyze_rating_changes(rh)
|
||||
assert len(stats) == 0
|
||||
anim = prepare_animation_data(rh)
|
||||
assert anim["frames"] == []
|
||||
assert anim["total_frames"] == 0
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Regression: prepare_animation_data must tolerate empty history."""
|
||||
import pandas as pd
|
||||
from animation import prepare_animation_data
|
||||
|
||||
|
||||
def test_empty_history_returns_empty_frames():
|
||||
df = pd.DataFrame(columns=["date", "model", "rating", "rank", "matches", "wins"])
|
||||
data = prepare_animation_data(df)
|
||||
assert data["frames"] == []
|
||||
assert data["total_frames"] == 0
|
||||
assert data["start_date"] is None
|
||||
@@ -0,0 +1,35 @@
|
||||
"""prepare_animation_data must keep fractional wins from Elo ties."""
|
||||
import pandas as pd
|
||||
from animation import prepare_animation_data
|
||||
|
||||
|
||||
def test_tie_half_wins_are_not_truncated():
|
||||
history = pd.DataFrame(
|
||||
{
|
||||
"date": pd.to_datetime(["2024-01-07", "2024-01-07"]),
|
||||
"model": ["A", "B"],
|
||||
"rating": [1000.0, 1000.0],
|
||||
"rank": [1, 2],
|
||||
"matches": [1, 1],
|
||||
"wins": [0.5, 0.5],
|
||||
}
|
||||
)
|
||||
data = prepare_animation_data(history, top_n=2)
|
||||
wins = {m["name"]: m["wins"] for m in data["frames"][0]["models"]}
|
||||
assert wins["A"] == 0.5
|
||||
assert wins["B"] == 0.5
|
||||
|
||||
|
||||
def test_whole_wins_still_serialize():
|
||||
history = pd.DataFrame(
|
||||
{
|
||||
"date": pd.to_datetime(["2024-01-07"]),
|
||||
"model": ["A"],
|
||||
"rating": [1010.0],
|
||||
"rank": [1],
|
||||
"matches": [2],
|
||||
"wins": [2.0],
|
||||
}
|
||||
)
|
||||
data = prepare_animation_data(history, top_n=1)
|
||||
assert data["frames"][0]["models"][0]["wins"] == 2.0
|
||||
@@ -0,0 +1,23 @@
|
||||
"""
|
||||
Test suite locking out ZeroDivisionError in benchmark summary print logic
|
||||
when time_basic is 0.0 or df_sample is empty.
|
||||
"""
|
||||
|
||||
def test_benchmark_pct_reduction_zero_division():
|
||||
"""
|
||||
Ensure zero time_basic does not raise ZeroDivisionError during benchmark calculation.
|
||||
"""
|
||||
time_basic = 0.0
|
||||
time_optimized = 0.0
|
||||
pct_reduction = (1 - time_optimized / time_basic) * 100 if time_basic > 0 else 0.0
|
||||
assert pct_reduction == 0.0
|
||||
|
||||
|
||||
def test_benchmark_extrapolation_zero_sample():
|
||||
"""
|
||||
Ensure empty df_sample does not raise ZeroDivisionError during extrapolation check.
|
||||
"""
|
||||
df_sample = []
|
||||
df_filtered = [1, 2, 3]
|
||||
should_extrapolate = len(df_sample) > 0 and len(df_sample) < len(df_filtered)
|
||||
assert not should_extrapolate
|
||||
@@ -0,0 +1,16 @@
|
||||
import pandas as pd
|
||||
from bradley_terry import compute_mle_elo, get_bootstrap_result
|
||||
|
||||
|
||||
def test_bootstrap_is_reproducible():
|
||||
battles = pd.DataFrame(
|
||||
[
|
||||
{"model_a": "a", "model_b": "b", "winner": "model_a"},
|
||||
{"model_a": "a", "model_b": "b", "winner": "model_b"},
|
||||
{"model_a": "a", "model_b": "b", "winner": "tie"},
|
||||
{"model_a": "b", "model_b": "a", "winner": "model_a"},
|
||||
]
|
||||
)
|
||||
first = get_bootstrap_result(battles, compute_mle_elo, num_round=3)
|
||||
second = get_bootstrap_result(battles, compute_mle_elo, num_round=3)
|
||||
pd.testing.assert_frame_equal(first, second)
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Regression: compute_mle_elo must work on small Arena-shaped battle sets."""
|
||||
import pandas as pd
|
||||
from battle_simulator import simulate_battles
|
||||
from bradley_terry import compute_mle_elo
|
||||
|
||||
|
||||
def test_small_two_model_sample():
|
||||
df = pd.DataFrame(simulate_battles({"gpt-4": 1200.0, "llama-3": 1000.0}, 10, seed=1))
|
||||
ratings = compute_mle_elo(df)
|
||||
assert len(ratings) == 2
|
||||
assert set(ratings.index) == {"gpt-4", "llama-3"}
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Ties must contribute to Bradley-Terry weights (not be zeroed by pivot+T)."""
|
||||
import pandas as pd
|
||||
|
||||
from bradley_terry import compute_mle_elo
|
||||
|
||||
|
||||
def test_all_ties_rates_models_instead_of_sample_weight_error():
|
||||
df = pd.DataFrame(
|
||||
[
|
||||
{"model_a": "A", "model_b": "B", "winner": "tie"},
|
||||
{"model_a": "A", "model_b": "C", "winner": "tie (bothbad)"},
|
||||
{"model_a": "B", "model_b": "C", "winner": "tie"},
|
||||
]
|
||||
)
|
||||
ratings = compute_mle_elo(df)
|
||||
assert set(ratings.index) == {"A", "B", "C"}
|
||||
# Pure ties -> equal latent skills under BT.
|
||||
assert abs(float(ratings["A"]) - float(ratings["B"])) < 1e-6
|
||||
assert abs(float(ratings["A"]) - float(ratings["C"])) < 1e-6
|
||||
|
||||
|
||||
def test_ties_change_ratings_versus_wins_only():
|
||||
wins_only = pd.DataFrame(
|
||||
[
|
||||
{"model_a": "A", "model_b": "B", "winner": "model_a"},
|
||||
{"model_a": "B", "model_b": "C", "winner": "model_a"},
|
||||
]
|
||||
)
|
||||
with_ties = pd.concat(
|
||||
[
|
||||
wins_only,
|
||||
pd.DataFrame(
|
||||
[{"model_a": "A", "model_b": "C", "winner": "tie"}] * 8
|
||||
),
|
||||
],
|
||||
ignore_index=True,
|
||||
)
|
||||
r1 = compute_mle_elo(wins_only)
|
||||
r2 = compute_mle_elo(with_ties)
|
||||
# Extra A–C ties pull A and C together relative to the wins-only fit.
|
||||
assert abs(float(r2["A"]) - float(r2["C"])) < abs(float(r1["A"]) - float(r1["C"]))
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Regression test for prepare_animation_data with string or date objects in history_df."""
|
||||
import pandas as pd
|
||||
from animation import prepare_animation_data
|
||||
|
||||
|
||||
def test_prepare_animation_data_string_date():
|
||||
"""prepare_animation_data must handle string dates without raising AttributeError."""
|
||||
history = pd.DataFrame([
|
||||
{
|
||||
"date": "2024-08-01",
|
||||
"model": "model_a",
|
||||
"rating": 1050.0,
|
||||
"rank": 1,
|
||||
"matches": 10,
|
||||
"wins": 7.0,
|
||||
},
|
||||
{
|
||||
"date": "2024-08-01",
|
||||
"model": "model_b",
|
||||
"rating": 950.0,
|
||||
"rank": 2,
|
||||
"matches": 10,
|
||||
"wins": 3.0,
|
||||
},
|
||||
])
|
||||
data = prepare_animation_data(history, top_n=2)
|
||||
assert data["total_frames"] == 1
|
||||
assert data["start_date"] == "2024-08-01"
|
||||
assert data["end_date"] == "2024-08-01"
|
||||
assert len(data["frames"]) == 1
|
||||
assert data["frames"][0]["date"] == "2024-08-01"
|
||||
assert data["frames"][0]["timestamp"] == 1722470400
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Regression test for compare_win_rates when comparisons list is empty."""
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from elo_rating import EloRatingSystem
|
||||
from leaderboard import compare_win_rates
|
||||
|
||||
|
||||
def test_compare_win_rates_empty_has_required_columns():
|
||||
"""compare_win_rates must return a DataFrame with required columns when no valid comparisons exist."""
|
||||
elo = EloRatingSystem()
|
||||
empirical_df = pd.DataFrame(np.nan, index=["model_a", "model_b"], columns=["model_a", "model_b"])
|
||||
df_comp = compare_win_rates(elo, empirical_df)
|
||||
assert list(df_comp.columns) == ["model_a", "model_b", "empirical", "predicted", "error"]
|
||||
assert len(df_comp) == 0
|
||||
# Accessing columns on empty result must not raise KeyError
|
||||
assert "error" in df_comp
|
||||
assert df_comp["error"].empty
|
||||
@@ -0,0 +1,170 @@
|
||||
"""
|
||||
Unit tests for Elo rating system
|
||||
"""
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
from _bootstrap import bootstrap_experiment_root
|
||||
|
||||
bootstrap_experiment_root()
|
||||
|
||||
from elo_rating import EloRatingSystem
|
||||
|
||||
|
||||
def test_initial_rating():
|
||||
"""Test that models start with initial rating."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0)
|
||||
assert elo.get_rating("model_a") == 1000.0
|
||||
assert elo.get_rating("model_b") == 1000.0
|
||||
|
||||
|
||||
def test_expected_score():
|
||||
"""Test expected score calculation."""
|
||||
elo = EloRatingSystem()
|
||||
|
||||
# Equal ratings should give 50% probability
|
||||
assert elo.expected_score(1000, 1000) == 0.5
|
||||
|
||||
# Higher rated player should have > 50% probability
|
||||
assert elo.expected_score(1200, 1000) > 0.5
|
||||
assert elo.expected_score(1000, 1200) < 0.5
|
||||
|
||||
# 400 point difference should give ~91% probability
|
||||
prob = elo.expected_score(1400, 1000)
|
||||
assert 0.90 < prob < 0.92
|
||||
|
||||
|
||||
def test_rating_update_win():
|
||||
"""Test rating update when model_a wins."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
new_a, new_b = elo.update_ratings("model_a", "model_b", "model_a")
|
||||
|
||||
# Winner should gain rating, loser should lose rating
|
||||
assert new_a > 1000.0
|
||||
assert new_b < 1000.0
|
||||
|
||||
# Total rating should be conserved (zero-sum)
|
||||
assert abs((new_a + new_b) - 2000.0) < 0.01
|
||||
|
||||
|
||||
def test_rating_update_tie():
|
||||
"""Test rating update for a tie."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
new_a, new_b = elo.update_ratings("model_a", "model_b", "tie")
|
||||
|
||||
# With equal ratings, tie should not change ratings much
|
||||
assert abs(new_a - 1000.0) < 0.01
|
||||
assert abs(new_b - 1000.0) < 0.01
|
||||
|
||||
|
||||
def test_upset_gives_larger_change():
|
||||
"""Test that unexpected results cause larger rating changes."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
# Give model_a higher rating
|
||||
elo.ratings["model_a"] = 1200.0
|
||||
elo.ratings["model_b"] = 1000.0
|
||||
|
||||
# If weaker model wins (upset), changes should be larger
|
||||
new_a_upset, new_b_upset = elo.update_ratings("model_a", "model_b", "model_b")
|
||||
|
||||
# Reset
|
||||
elo.ratings["model_a"] = 1200.0
|
||||
elo.ratings["model_b"] = 1000.0
|
||||
|
||||
# If stronger model wins (expected), changes should be smaller
|
||||
new_a_expected, new_b_expected = elo.update_ratings("model_a", "model_b", "model_a")
|
||||
|
||||
# Upset should cause larger change
|
||||
change_upset = abs(new_a_upset - 1200.0)
|
||||
change_expected = abs(new_a_expected - 1200.0)
|
||||
|
||||
assert change_upset > change_expected
|
||||
|
||||
|
||||
def test_leaderboard_sorting():
|
||||
"""Test that leaderboard is sorted by rating."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
# Create some matches to differentiate ratings
|
||||
elo.update_ratings("model_a", "model_b", "model_a")
|
||||
elo.update_ratings("model_a", "model_c", "model_a")
|
||||
elo.update_ratings("model_b", "model_c", "model_b")
|
||||
|
||||
leaderboard = elo.get_leaderboard()
|
||||
|
||||
# Check descending order
|
||||
for i in range(len(leaderboard) - 1):
|
||||
assert leaderboard[i][1] >= leaderboard[i+1][1]
|
||||
|
||||
# model_a should be first (won all matches)
|
||||
assert leaderboard[0][0] == "model_a"
|
||||
|
||||
|
||||
def test_win_probability_symmetry():
|
||||
"""Test that win probabilities sum to 1."""
|
||||
elo = EloRatingSystem()
|
||||
elo.ratings["model_a"] = 1200.0
|
||||
elo.ratings["model_b"] = 1000.0
|
||||
|
||||
prob_a = elo.calculate_win_probability("model_a", "model_b")
|
||||
prob_b = elo.calculate_win_probability("model_b", "model_a")
|
||||
|
||||
# Should sum to 1
|
||||
assert abs(prob_a + prob_b - 1.0) < 0.001
|
||||
|
||||
|
||||
def test_match_counting():
|
||||
"""Test that match and win counts are tracked correctly."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
elo.update_ratings("model_a", "model_b", "model_a") # model_a wins
|
||||
elo.update_ratings("model_a", "model_c", "model_b") # model_a loses (2nd slot wins)
|
||||
elo.update_ratings("model_a", "model_b", "tie") # tie -> 0.5 each
|
||||
|
||||
# model_a played 3 matches
|
||||
assert elo.match_counts["model_a"] == 3
|
||||
|
||||
# model_a won 1 match and tied 1 (1.5 total)
|
||||
assert elo.win_counts["model_a"] == 1.5
|
||||
|
||||
# model_b played 2 matches
|
||||
assert elo.match_counts["model_b"] == 2
|
||||
|
||||
|
||||
def test_copy():
|
||||
"""Test that copy creates independent instance."""
|
||||
elo1 = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
elo1.update_ratings("model_a", "model_b", "model_a")
|
||||
|
||||
elo2 = elo1.copy()
|
||||
|
||||
# Modify elo2
|
||||
elo2.update_ratings("model_a", "model_b", "model_b")
|
||||
|
||||
# elo1 should be unchanged
|
||||
assert elo1.ratings["model_a"] != elo2.ratings["model_a"]
|
||||
|
||||
|
||||
def test_reset():
|
||||
"""Test that reset clears all data."""
|
||||
elo = EloRatingSystem(initial_rating=1000.0, k_factor=32.0)
|
||||
|
||||
elo.update_ratings("model_a", "model_b", "model_a")
|
||||
elo.update_ratings("model_a", "model_c", "model_a")
|
||||
|
||||
assert len(elo.ratings) > 0
|
||||
|
||||
elo.reset()
|
||||
|
||||
assert len(elo.ratings) == 0
|
||||
assert len(elo.match_counts) == 0
|
||||
assert len(elo.win_counts) == 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Run tests
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,21 @@
|
||||
"""Regression: filter_data_parallel must tolerate n_jobs > len(df)."""
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from parallel_processing import filter_data_parallel
|
||||
|
||||
|
||||
def test_n_jobs_larger_than_rows():
|
||||
df = pd.DataFrame({"anony": [True, False, True], "turn": [1, 2, 1]})
|
||||
|
||||
def map_inline(fn, chunks):
|
||||
return [fn(c) for c in chunks]
|
||||
|
||||
pool = MagicMock()
|
||||
pool.__enter__.return_value.map.side_effect = map_inline
|
||||
pool.__exit__.return_value = False
|
||||
|
||||
with patch("parallel_processing.Pool", return_value=pool):
|
||||
out = filter_data_parallel(df, {"anony_only": True}, n_jobs=8)
|
||||
assert len(out) == 2
|
||||
@@ -0,0 +1,37 @@
|
||||
"""
|
||||
Regression test for filter_data on empty input (实验 7-7 排行榜).
|
||||
|
||||
An empty arena data file (e.g. a failed/truncated download saved as `[]`) used to
|
||||
crash with ZeroDivisionError at the "After filtering" percentage print.
|
||||
"""
|
||||
import pandas as pd
|
||||
import pytest
|
||||
|
||||
from _bootstrap import bootstrap_experiment_root
|
||||
|
||||
bootstrap_experiment_root()
|
||||
|
||||
from data_loader import filter_data
|
||||
|
||||
|
||||
def test_filter_data_tolerates_empty_dataframe():
|
||||
"""Empty input no longer raises ZeroDivisionError; returns an empty DataFrame."""
|
||||
empty = pd.DataFrame({"model_a": [], "model_b": [], "winner": []})
|
||||
result = filter_data(empty)
|
||||
assert len(result) == 0
|
||||
|
||||
|
||||
def test_filter_data_normal_case_unchanged():
|
||||
"""Non-empty input still filters and reports normally."""
|
||||
df = pd.DataFrame({
|
||||
"model_a": ["a", "b", "a"],
|
||||
"model_b": ["b", "a", "c"],
|
||||
"winner": ["model_a", "model_b", "tie"],
|
||||
"anony": [True, True, False],
|
||||
})
|
||||
result = filter_data(df, anony_only=True, use_dedup=False)
|
||||
assert len(result) == 2 # 非匿名的一条被过滤
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Empty battles JSON array [] must load as an empty battle frame."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import cli
|
||||
|
||||
|
||||
def test_load_battles_empty_json_array(tmp_path):
|
||||
path = tmp_path / "battles.json"
|
||||
path.write_text("[]", encoding="utf-8")
|
||||
df = cli._load_battles(str(path))
|
||||
assert list(df.columns) == ["model_a", "model_b", "winner"]
|
||||
assert len(df) == 0
|
||||
|
||||
|
||||
def test_load_battles_nonempty_still_requires_columns(tmp_path):
|
||||
path = tmp_path / "bad.json"
|
||||
path.write_text(json.dumps([{"x": 1}]), encoding="utf-8")
|
||||
try:
|
||||
cli._load_battles(str(path))
|
||||
assert False, "expected ValueError"
|
||||
except ValueError as e:
|
||||
assert "model_a" in str(e)
|
||||
|
||||
|
||||
def test_load_battles_normal(tmp_path):
|
||||
path = tmp_path / "ok.json"
|
||||
path.write_text(
|
||||
json.dumps([{"model_a": "A", "model_b": "B", "winner": "model_a"}]),
|
||||
encoding="utf-8",
|
||||
)
|
||||
df = cli._load_battles(str(path))
|
||||
assert len(df) == 1
|
||||
assert df.iloc[0]["winner"] == "model_a"
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Regression: optimize_dataframe must tolerate empty object columns."""
|
||||
import pandas as pd
|
||||
from parallel_processing import optimize_dataframe
|
||||
|
||||
|
||||
def test_optimize_empty_object_columns():
|
||||
df = pd.DataFrame({
|
||||
"model_a": pd.Series([], dtype=object),
|
||||
"model_b": pd.Series([], dtype=object),
|
||||
"winner": pd.Series([], dtype=object),
|
||||
})
|
||||
out = optimize_dataframe(df)
|
||||
assert len(out) == 0
|
||||
@@ -0,0 +1,49 @@
|
||||
"""
|
||||
Regression test for 'tie (bothbad)' handling in optimized_elo (实验 7-7 排行榜).
|
||||
|
||||
Chatbot Arena battle data has four outcomes; 'tie (bothbad)' was missing from
|
||||
the outcome map, so Series.map produced NaN. NaN then propagated through the
|
||||
rating updates and spread to every model that later faced an affected one,
|
||||
leaving the whole leaderboard NaN.
|
||||
"""
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from optimized_elo import NumpyEloRatingSystem
|
||||
|
||||
|
||||
def test_tie_bothbad_does_not_produce_nan_outcomes():
|
||||
"""'tie (bothbad)' maps to a tie instead of NaN."""
|
||||
df = pd.DataFrame({
|
||||
"model_a": ["a", "a"],
|
||||
"model_b": ["b", "b"],
|
||||
"winner": ["model_a", "tie (bothbad)"],
|
||||
})
|
||||
_, _, outcomes = NumpyEloRatingSystem()._prepare_data(df)
|
||||
assert not np.isnan(outcomes).any()
|
||||
assert outcomes[1] == 0.5
|
||||
|
||||
|
||||
def test_tie_bothbad_does_not_poison_the_leaderboard():
|
||||
"""One 'tie (bothbad)' battle used to NaN every rating, including model c."""
|
||||
df = pd.DataFrame({
|
||||
"model_a": ["a", "a", "b"],
|
||||
"model_b": ["b", "b", "c"],
|
||||
"winner": ["model_a", "tie (bothbad)", "model_a"],
|
||||
})
|
||||
system = NumpyEloRatingSystem()
|
||||
system.process_matches_vectorized(df, show_progress=False)
|
||||
ratings = [rating for _, rating, _, _ in system.get_leaderboard()]
|
||||
assert len(ratings) == 3
|
||||
assert not any(np.isnan(r) for r in ratings)
|
||||
|
||||
|
||||
def test_unknown_outcome_falls_back_to_tie():
|
||||
"""An unrecognized label degrades to a tie rather than NaN."""
|
||||
df = pd.DataFrame({
|
||||
"model_a": ["a"],
|
||||
"model_b": ["b"],
|
||||
"winner": ["something_new"],
|
||||
})
|
||||
_, _, outcomes = NumpyEloRatingSystem()._prepare_data(df)
|
||||
assert outcomes[0] == 0.5
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Regression: documented interval='M' must work on modern pandas."""
|
||||
import pandas as pd
|
||||
from data_loader import get_time_slices
|
||||
|
||||
|
||||
def test_monthly_interval_alias():
|
||||
df = pd.DataFrame({"tstamp": [1_700_000_000, 1_710_000_000]})
|
||||
slices = get_time_slices(df, interval="M")
|
||||
assert len(slices) >= 1
|
||||
@@ -0,0 +1,55 @@
|
||||
"""
|
||||
Regression: get_time_slices must not IndexError when the tstamp span is
|
||||
shorter than the requested interval (default weekly).
|
||||
|
||||
Chatbot Arena samples, same-second dumps, and single-row demos all produce an
|
||||
empty pd.date_range for freq='W'; the old code then crashed on date_ranges[-1].
|
||||
"""
|
||||
import pandas as pd
|
||||
|
||||
from _bootstrap import bootstrap_experiment_root
|
||||
|
||||
bootstrap_experiment_root()
|
||||
|
||||
from data_loader import get_time_slices
|
||||
|
||||
|
||||
def test_identical_timestamps_return_one_slice():
|
||||
"""Two battles at the same unix second (weekly interval) -> one slice."""
|
||||
ts = 1_700_000_000
|
||||
df = pd.DataFrame({
|
||||
"tstamp": [ts, ts],
|
||||
"model_a": ["a", "c"],
|
||||
"model_b": ["b", "d"],
|
||||
"winner": ["model_a", "model_b"],
|
||||
})
|
||||
slices = get_time_slices(df, interval="W")
|
||||
assert len(slices) == 1
|
||||
end_date, slice_df = slices[0]
|
||||
assert len(slice_df) == 2
|
||||
assert end_date == pd.to_datetime(ts, unit="s")
|
||||
|
||||
|
||||
def test_empty_dataframe_returns_empty_list():
|
||||
"""Empty input returns [] instead of NaT ValueError."""
|
||||
df = pd.DataFrame({"tstamp": pd.Series(dtype="float64")})
|
||||
assert get_time_slices(df, interval="W") == []
|
||||
|
||||
|
||||
def test_multi_week_span_still_produces_buckets():
|
||||
"""A span covering multiple weeks still yields intermediate buckets."""
|
||||
# ~3 weeks apart
|
||||
df = pd.DataFrame({
|
||||
"tstamp": [1_700_000_000, 1_700_000_000 + 21 * 86400],
|
||||
"model_a": ["a", "c"],
|
||||
"model_b": ["b", "d"],
|
||||
"winner": ["model_a", "model_b"],
|
||||
})
|
||||
slices = get_time_slices(df, interval="W")
|
||||
assert len(slices) >= 2
|
||||
assert all(len(s[1]) > 0 for s in slices)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import pytest
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,20 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def test_canonical_manifest_is_hash_complete():
|
||||
run_dir = Path(__file__).resolve().parents[1] / "validation" / "runs" / "exp7-7-arena-20260731-v1"
|
||||
manifest_path = run_dir / "manifest.json"
|
||||
assert manifest_path.exists()
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
assert manifest["experiment"] == "7-7"
|
||||
assert manifest["official_complete"] is True
|
||||
assert all(manifest["gates"].values())
|
||||
assert set(manifest["artifacts"]) >= {
|
||||
"summary.json",
|
||||
"online_elo.json",
|
||||
"bradley_terry.json",
|
||||
"win_rate_matrix.json",
|
||||
"rating_history.json",
|
||||
"leaderboard_animation.html",
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Empty battle DataFrame must not crash Bradley-Terry LogisticRegression."""
|
||||
import pandas as pd
|
||||
|
||||
from bradley_terry import compute_bradley_terry_leaderboard, compute_mle_elo
|
||||
|
||||
|
||||
def test_compute_mle_elo_empty_battles():
|
||||
df = pd.DataFrame(columns=["model_a", "model_b", "winner"])
|
||||
ratings = compute_mle_elo(df)
|
||||
assert isinstance(ratings, pd.Series)
|
||||
assert len(ratings) == 0
|
||||
|
||||
|
||||
def test_compute_bradley_terry_leaderboard_empty():
|
||||
df = pd.DataFrame(columns=["model_a", "model_b", "winner"])
|
||||
board = compute_bradley_terry_leaderboard(df)
|
||||
assert isinstance(board, pd.DataFrame)
|
||||
assert len(board) == 0
|
||||
|
||||
|
||||
def test_nonempty_still_rates():
|
||||
df = pd.DataFrame(
|
||||
[
|
||||
{"model_a": "A", "model_b": "B", "winner": "model_a"},
|
||||
{"model_a": "A", "model_b": "B", "winner": "model_a"},
|
||||
{"model_a": "B", "model_b": "C", "winner": "model_b"},
|
||||
{"model_a": "A", "model_b": "C", "winner": "model_a"},
|
||||
]
|
||||
)
|
||||
ratings = compute_mle_elo(df)
|
||||
assert set(ratings.index) >= {"A", "B", "C"}
|
||||
assert ratings["A"] > ratings["C"]
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"experiment": "7-7",
|
||||
"manifest_sha256": "b67f8b15a088e694ae356322e0ccd676b54ed79a0dc1c787ca1960ef670ba2e8",
|
||||
"official_complete": true,
|
||||
"run": "runs/exp7-7-arena-20260731-v1",
|
||||
"status": "passed"
|
||||
}
|
||||
@@ -0,0 +1,389 @@
|
||||
"""Run the complete, evidence-producing Experiment 7-7 campaign.
|
||||
|
||||
The public Arena file is deliberately not copied into git. A canonical run
|
||||
binds the exact input by URL, size, record count, and SHA-256, then retains all
|
||||
derived tables, visualizations, the D3 history animation, and a manifest that
|
||||
hashes every output and the source used to create it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
os.environ.setdefault("MPLBACKEND", "Agg")
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import seaborn as sns
|
||||
from scipy.stats import kendalltau, spearmanr
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
PROJECT = HERE.parent
|
||||
sys.path.insert(0, str(PROJECT))
|
||||
|
||||
from animation import create_simple_animation
|
||||
from bradley_terry import compute_bradley_terry_leaderboard
|
||||
from optimized_elo import (
|
||||
NumpyEloRatingSystem,
|
||||
process_elo_updates_vectorized,
|
||||
)
|
||||
|
||||
DATASET_URL = (
|
||||
"https://storage.googleapis.com/arena_external_data/public/"
|
||||
"clean_battle_20240814_public.json"
|
||||
)
|
||||
REQUIRED_COLUMNS = ["model_a", "model_b", "winner", "tstamp", "anony", "turn"]
|
||||
ALLOWED_OUTCOMES = {"model_a", "model_b", "tie", "tie (bothbad)"}
|
||||
|
||||
|
||||
def sha256_file(path: Path, chunk_size: int = 8 * 1024 * 1024) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(chunk_size):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, value: Any) -> None:
|
||||
path.write_text(
|
||||
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def json_records(frame: pd.DataFrame) -> list[dict[str, Any]]:
|
||||
return json.loads(frame.to_json(orient="records", date_format="iso"))
|
||||
|
||||
|
||||
def load_and_filter(path: Path, max_records: int) -> tuple[pd.DataFrame, dict[str, Any]]:
|
||||
started = time.perf_counter()
|
||||
raw = pd.read_json(path)
|
||||
missing = sorted(set(REQUIRED_COLUMNS) - set(raw.columns))
|
||||
if missing:
|
||||
raise ValueError(f"Arena input is missing columns: {missing}")
|
||||
|
||||
source_records = len(raw)
|
||||
frame = raw[REQUIRED_COLUMNS + (["dedup_tag"] if "dedup_tag" in raw else [])].copy()
|
||||
del raw
|
||||
frame = frame[frame["anony"].eq(True) & frame["turn"].ge(1)]
|
||||
if "dedup_tag" in frame:
|
||||
sampled = frame["dedup_tag"].map(
|
||||
lambda value: bool(value.get("sampled", False)) if isinstance(value, dict) else False
|
||||
)
|
||||
frame = frame[sampled]
|
||||
frame = frame[frame["winner"].isin(ALLOWED_OUTCOMES)]
|
||||
frame = frame.sort_values("tstamp", kind="stable").reset_index(drop=True)
|
||||
if max_records:
|
||||
frame = frame.head(max_records).copy()
|
||||
if frame.empty:
|
||||
raise ValueError("Arena filtering produced no accepted blind votes")
|
||||
|
||||
metadata = {
|
||||
"source_records": source_records,
|
||||
"accepted_records": len(frame),
|
||||
"model_count": len(set(frame["model_a"]) | set(frame["model_b"])),
|
||||
"start_utc": datetime.fromtimestamp(float(frame["tstamp"].min()), timezone.utc).isoformat(),
|
||||
"end_utc": datetime.fromtimestamp(float(frame["tstamp"].max()), timezone.utc).isoformat(),
|
||||
"outcomes": {str(k): int(v) for k, v in frame["winner"].value_counts().items()},
|
||||
"load_filter_seconds": round(time.perf_counter() - started, 3),
|
||||
"bounded_test_run": bool(max_records),
|
||||
}
|
||||
return frame, metadata
|
||||
|
||||
|
||||
def online_elo_and_history(
|
||||
frame: pd.DataFrame,
|
||||
) -> tuple[pd.DataFrame, pd.DataFrame, NumpyEloRatingSystem, float]:
|
||||
started = time.perf_counter()
|
||||
system = NumpyEloRatingSystem(initial_rating=1000.0, k_factor=4.0)
|
||||
model_a, model_b, outcomes = system._prepare_data(frame)
|
||||
|
||||
months = (
|
||||
pd.to_datetime(frame["tstamp"], unit="s", utc=True)
|
||||
.dt.tz_localize(None)
|
||||
.dt.to_period("M")
|
||||
)
|
||||
boundaries = np.flatnonzero(months.to_numpy()[1:] != months.to_numpy()[:-1]) + 1
|
||||
boundaries = np.append(boundaries, len(frame))
|
||||
start = 0
|
||||
history_rows: list[dict[str, Any]] = []
|
||||
for stop in boundaries:
|
||||
process_elo_updates_vectorized(
|
||||
system.ratings,
|
||||
model_a[start:stop],
|
||||
model_b[start:stop],
|
||||
outcomes[start:stop],
|
||||
system.k_factor,
|
||||
system.match_counts,
|
||||
system.win_counts,
|
||||
)
|
||||
snapshot = system.get_leaderboard()
|
||||
date = pd.to_datetime(float(frame.iloc[stop - 1]["tstamp"]), unit="s", utc=True)
|
||||
for rank, (model, rating, matches, wins) in enumerate(snapshot, 1):
|
||||
history_rows.append(
|
||||
{
|
||||
"date": date.tz_localize(None),
|
||||
"model": model,
|
||||
"rating": rating,
|
||||
"rank": rank,
|
||||
"matches": matches,
|
||||
"wins": wins,
|
||||
}
|
||||
)
|
||||
start = int(stop)
|
||||
|
||||
leaderboard = pd.DataFrame(
|
||||
system.get_leaderboard(), columns=["model", "rating", "matches", "wins"]
|
||||
)
|
||||
leaderboard.insert(0, "rank", range(1, len(leaderboard) + 1))
|
||||
history = pd.DataFrame(history_rows)
|
||||
return leaderboard, history, system, round(time.perf_counter() - started, 3)
|
||||
|
||||
|
||||
def rank_comparison(online: pd.DataFrame, official_method: pd.DataFrame) -> dict[str, Any]:
|
||||
online_rank = online.set_index("model")["rank"]
|
||||
official = official_method.sort_values("rating", ascending=False).reset_index(drop=True)
|
||||
official["rank"] = np.arange(1, len(official) + 1)
|
||||
official_rank = official.set_index("model")["rank"]
|
||||
common = sorted(set(online_rank.index) & set(official_rank.index))
|
||||
rho = spearmanr(online_rank.loc[common], official_rank.loc[common]).statistic
|
||||
tau = kendalltau(online_rank.loc[common], official_rank.loc[common]).statistic
|
||||
online_top = online.nsmallest(20, "rank")["model"].tolist()
|
||||
official_top = official.nsmallest(20, "rank")["model"].tolist()
|
||||
return {
|
||||
"comparison_target": "Bradley-Terry MLE reconstruction used by Chatbot Arena",
|
||||
"claim_boundary": (
|
||||
"This is a same-snapshot reconstruction of the official method, not a scrape of "
|
||||
"the mutable live leaderboard. Scores need not match the live service."
|
||||
),
|
||||
"common_models": len(common),
|
||||
"spearman_rank_correlation": round(float(rho), 6),
|
||||
"kendall_rank_correlation": round(float(tau), 6),
|
||||
"top_20_overlap": len(set(online_top) & set(official_top)),
|
||||
"online_top_20": online_top,
|
||||
"official_method_top_20": official_top,
|
||||
}
|
||||
|
||||
|
||||
def empirical_matrix(frame: pd.DataFrame, models: list[str]) -> pd.DataFrame:
|
||||
subset = frame[frame["model_a"].isin(models) & frame["model_b"].isin(models)].copy()
|
||||
rows: list[tuple[str, str, float]] = []
|
||||
for a, b, winner in subset[["model_a", "model_b", "winner"]].itertuples(index=False):
|
||||
score = 1.0 if winner == "model_a" else 0.0 if winner == "model_b" else 0.5
|
||||
rows.append((a, b, score))
|
||||
rows.append((b, a, 1.0 - score))
|
||||
scored = pd.DataFrame(rows, columns=["model", "opponent", "score"])
|
||||
matrix = scored.pivot_table(index="model", columns="opponent", values="score", aggfunc="mean")
|
||||
matrix = matrix.reindex(index=models, columns=models)
|
||||
np.fill_diagonal(matrix.values, 0.5)
|
||||
return matrix
|
||||
|
||||
|
||||
def plot_artifacts(
|
||||
out: Path,
|
||||
online: pd.DataFrame,
|
||||
history: pd.DataFrame,
|
||||
empirical: pd.DataFrame,
|
||||
) -> None:
|
||||
top = online.head(20).sort_values("rating")
|
||||
fig, ax = plt.subplots(figsize=(11, 8))
|
||||
ax.barh(top["model"], top["rating"], color="#3b82f6")
|
||||
ax.set_title("Experiment 7-7: Online Elo leaderboard")
|
||||
ax.set_xlabel("Elo rating (K=4, chronological)")
|
||||
fig.tight_layout()
|
||||
fig.savefig(out / "leaderboard.png", dpi=180)
|
||||
plt.close(fig)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(13, 11))
|
||||
sns.heatmap(empirical, cmap="RdYlGn", center=0.5, vmin=0, vmax=1, ax=ax)
|
||||
ax.set_title("Empirical pairwise win rate — final online-Elo top 20")
|
||||
fig.tight_layout()
|
||||
fig.savefig(out / "win_rate_matrix.png", dpi=180)
|
||||
plt.close(fig)
|
||||
|
||||
top_models = online.head(10)["model"].tolist()
|
||||
fig, ax = plt.subplots(figsize=(13, 7))
|
||||
for model in top_models:
|
||||
values = history[history["model"].eq(model)].sort_values("date")
|
||||
ax.plot(values["date"], values["rating"], label=model, linewidth=1.8)
|
||||
ax.set_title("Monthly online-Elo evolution — final top 10")
|
||||
ax.set_ylabel("Elo rating")
|
||||
ax.legend(fontsize=7, ncol=2)
|
||||
fig.autofmt_xdate()
|
||||
fig.tight_layout()
|
||||
fig.savefig(out / "rating_history.png", dpi=180)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--input", type=Path, required=True, help="Downloaded public Arena JSON")
|
||||
parser.add_argument("--output-dir", type=Path, required=True)
|
||||
parser.add_argument("--bootstrap-rounds", type=int, default=20)
|
||||
parser.add_argument("--max-records", type=int, default=0, help="Noncanonical bounded test only")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
input_path = args.input.resolve()
|
||||
if not input_path.is_file():
|
||||
raise SystemExit(f"Arena input not found: {input_path}")
|
||||
|
||||
run_started = time.perf_counter()
|
||||
input_hash = sha256_file(input_path)
|
||||
frame, dataset = load_and_filter(input_path, args.max_records)
|
||||
online, history, online_system, online_seconds = online_elo_and_history(frame)
|
||||
|
||||
bt_started = time.perf_counter()
|
||||
official_method = compute_bradley_terry_leaderboard(
|
||||
frame[["model_a", "model_b", "winner"]],
|
||||
bootstrap_rounds=args.bootstrap_rounds,
|
||||
)
|
||||
bt_seconds = round(time.perf_counter() - bt_started, 3)
|
||||
official_method = official_method.sort_values("rating", ascending=False).reset_index(drop=True)
|
||||
official_method.insert(0, "rank", range(1, len(official_method) + 1))
|
||||
|
||||
comparison = rank_comparison(online, official_method)
|
||||
top_models = online.head(20)["model"].tolist()
|
||||
empirical = empirical_matrix(frame, top_models)
|
||||
predicted = pd.DataFrame(
|
||||
{
|
||||
opponent: {
|
||||
model: online_system.calculate_win_probability(model, opponent)
|
||||
for model in top_models
|
||||
}
|
||||
for opponent in top_models
|
||||
}
|
||||
).reindex(index=top_models, columns=top_models)
|
||||
|
||||
write_json(args.output_dir / "online_elo.json", json_records(online))
|
||||
write_json(args.output_dir / "bradley_terry.json", json_records(official_method))
|
||||
write_json(
|
||||
args.output_dir / "win_rate_matrix.json",
|
||||
{
|
||||
"models": top_models,
|
||||
"empirical": empirical.where(pd.notna(empirical), None).to_dict(orient="index"),
|
||||
"online_elo_predicted": predicted.to_dict(orient="index"),
|
||||
},
|
||||
)
|
||||
write_json(args.output_dir / "rating_history.json", json_records(history))
|
||||
plot_artifacts(args.output_dir, online, history, empirical)
|
||||
create_simple_animation(history, str(args.output_dir / "leaderboard_animation.html"), top_n=15)
|
||||
|
||||
gates = {
|
||||
"official_public_arena_snapshot_hashed": not args.max_records,
|
||||
"millions_of_blind_votes_loaded": dataset["source_records"] >= 1_000_000,
|
||||
"chronological_online_elo_k4_completed": len(online) == dataset["model_count"],
|
||||
"bradley_terry_official_method_completed": len(official_method) == dataset["model_count"],
|
||||
"online_vs_official_method_rank_agreement_observed": (
|
||||
comparison["spearman_rank_correlation"] >= 0.70
|
||||
and comparison["top_20_overlap"] >= 10
|
||||
),
|
||||
"pairwise_empirical_and_predicted_matrix_saved": len(empirical) == 20,
|
||||
"monthly_history_saved": history["date"].nunique() >= 2,
|
||||
"d3_animation_saved": (args.output_dir / "leaderboard_animation.html").is_file(),
|
||||
"static_visualizations_saved": all(
|
||||
(args.output_dir / name).is_file()
|
||||
for name in ["leaderboard.png", "win_rate_matrix.png", "rating_history.png"]
|
||||
),
|
||||
}
|
||||
accepted = all(gates.values())
|
||||
summary = {
|
||||
"schema_version": 1,
|
||||
"experiment": "7-7",
|
||||
"status": "passed" if accepted else "noncanonical_test",
|
||||
"official_complete": accepted,
|
||||
"generated_at_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"dataset": {
|
||||
"url": DATASET_URL,
|
||||
"path_recorded_as": input_path.name,
|
||||
"bytes": input_path.stat().st_size,
|
||||
"sha256": input_hash,
|
||||
**dataset,
|
||||
},
|
||||
"protocol": {
|
||||
"online_elo": "initial=1000, K=4, stable chronological order",
|
||||
"official_method": "Bradley-Terry maximum-likelihood reconstruction",
|
||||
"history_interval": "monthly cumulative snapshots",
|
||||
"bootstrap_rounds": args.bootstrap_rounds,
|
||||
"bootstrap_random_seed": 0,
|
||||
},
|
||||
"results": {
|
||||
"online_top_20": json_records(online.head(20)),
|
||||
"official_method_top_20": json_records(official_method.head(20)),
|
||||
"rank_comparison": comparison,
|
||||
},
|
||||
"timing_seconds": {
|
||||
"online_and_history": online_seconds,
|
||||
"bradley_terry": bt_seconds,
|
||||
"total": round(time.perf_counter() - run_started, 3),
|
||||
},
|
||||
"gates": gates,
|
||||
}
|
||||
write_json(args.output_dir / "summary.json", summary)
|
||||
|
||||
artifact_names = [
|
||||
"online_elo.json",
|
||||
"bradley_terry.json",
|
||||
"win_rate_matrix.json",
|
||||
"rating_history.json",
|
||||
"leaderboard.png",
|
||||
"win_rate_matrix.png",
|
||||
"rating_history.png",
|
||||
"leaderboard_animation.html",
|
||||
"summary.json",
|
||||
]
|
||||
source_names = [
|
||||
"animation.py",
|
||||
"bradley_terry.py",
|
||||
"optimized_elo.py",
|
||||
"validation/run_experiment.py",
|
||||
"validation/validate_evidence.py",
|
||||
]
|
||||
manifest = {
|
||||
"schema_version": 1,
|
||||
"experiment": "7-7",
|
||||
"status": summary["status"],
|
||||
"official_complete": accepted,
|
||||
"input": {
|
||||
"url": DATASET_URL,
|
||||
"filename": input_path.name,
|
||||
"bytes": input_path.stat().st_size,
|
||||
"sha256": input_hash,
|
||||
},
|
||||
"artifacts": {
|
||||
name: {"bytes": (args.output_dir / name).stat().st_size, "sha256": sha256_file(args.output_dir / name)}
|
||||
for name in artifact_names
|
||||
},
|
||||
"sources": {
|
||||
name: sha256_file(PROJECT / name)
|
||||
for name in source_names
|
||||
},
|
||||
"runtime": {
|
||||
"python": sys.version.split()[0],
|
||||
"platform": platform.platform(),
|
||||
"pandas": pd.__version__,
|
||||
"numpy": np.__version__,
|
||||
},
|
||||
"gates": gates,
|
||||
}
|
||||
write_json(args.output_dir / "manifest.json", manifest)
|
||||
print(json.dumps({"status": manifest["status"], "output": str(args.output_dir)}, indent=2))
|
||||
if not accepted:
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,905 @@
|
||||
[
|
||||
{
|
||||
"lower_ci": 1199.4356201692,
|
||||
"model": "chatgpt-4o-latest",
|
||||
"rank": 1,
|
||||
"rating": 1202.8722324768,
|
||||
"upper_ci": 1207.3805726125
|
||||
},
|
||||
{
|
||||
"lower_ci": 1182.767114327,
|
||||
"model": "gemini-1.5-pro-exp-0801",
|
||||
"rank": 2,
|
||||
"rating": 1187.3727762743,
|
||||
"upper_ci": 1191.7564879767
|
||||
},
|
||||
{
|
||||
"lower_ci": 1173.1270556478,
|
||||
"model": "gpt-4o-2024-05-13",
|
||||
"rank": 3,
|
||||
"rating": 1174.5475812185,
|
||||
"upper_ci": 1177.0403252386
|
||||
},
|
||||
{
|
||||
"lower_ci": 1161.1978031325,
|
||||
"model": "gpt-4o-mini-2024-07-18",
|
||||
"rank": 4,
|
||||
"rating": 1162.9712796813,
|
||||
"upper_ci": 1166.568334134
|
||||
},
|
||||
{
|
||||
"lower_ci": 1156.0224735749,
|
||||
"model": "claude-3-5-sonnet-20240620",
|
||||
"rank": 5,
|
||||
"rating": 1159.6692177384,
|
||||
"upper_ci": 1161.8458744744
|
||||
},
|
||||
{
|
||||
"lower_ci": 1153.6447817793,
|
||||
"model": "gemini-advanced-0514",
|
||||
"rank": 6,
|
||||
"rating": 1155.6305094344,
|
||||
"upper_ci": 1157.1030226933
|
||||
},
|
||||
{
|
||||
"lower_ci": 1150.4614200682,
|
||||
"model": "llama-3.1-405b-instruct",
|
||||
"rank": 7,
|
||||
"rating": 1153.3950109278,
|
||||
"upper_ci": 1157.0848375454
|
||||
},
|
||||
{
|
||||
"lower_ci": 1146.348878889,
|
||||
"model": "gpt-4o-2024-08-06",
|
||||
"rank": 8,
|
||||
"rating": 1150.7428051987,
|
||||
"upper_ci": 1156.3245456986
|
||||
},
|
||||
{
|
||||
"lower_ci": 1146.7772301944,
|
||||
"model": "gemini-1.5-pro-api-0514",
|
||||
"rank": 9,
|
||||
"rating": 1149.1265569601,
|
||||
"upper_ci": 1151.5010539806
|
||||
},
|
||||
{
|
||||
"lower_ci": 1143.6682527667,
|
||||
"model": "gemini-1.5-pro-api-0409-preview",
|
||||
"rank": 10,
|
||||
"rating": 1146.1756037638,
|
||||
"upper_ci": 1147.8895040945
|
||||
},
|
||||
{
|
||||
"lower_ci": 1142.3990049162,
|
||||
"model": "gpt-4-turbo-2024-04-09",
|
||||
"rank": 11,
|
||||
"rating": 1144.9187340632,
|
||||
"upper_ci": 1146.8362809167
|
||||
},
|
||||
{
|
||||
"lower_ci": 1138.1303324947,
|
||||
"model": "gpt-4-1106-preview",
|
||||
"rank": 12,
|
||||
"rating": 1139.5004505155,
|
||||
"upper_ci": 1140.97644635
|
||||
},
|
||||
{
|
||||
"lower_ci": 1136.3852091186,
|
||||
"model": "mistral-large-2407",
|
||||
"rank": 13,
|
||||
"rating": 1139.3566231011,
|
||||
"upper_ci": 1143.8960727747
|
||||
},
|
||||
{
|
||||
"lower_ci": 1135.0624903381,
|
||||
"model": "athene-70b-0725",
|
||||
"rank": 14,
|
||||
"rating": 1138.2016223593,
|
||||
"upper_ci": 1141.397916447
|
||||
},
|
||||
{
|
||||
"lower_ci": 1135.0278139027,
|
||||
"model": "claude-3-opus-20240229",
|
||||
"rank": 15,
|
||||
"rating": 1136.5739464324,
|
||||
"upper_ci": 1137.5663039742
|
||||
},
|
||||
{
|
||||
"lower_ci": 1131.7219814044,
|
||||
"model": "llama-3.1-70b-instruct",
|
||||
"rank": 16,
|
||||
"rating": 1134.8942150397,
|
||||
"upper_ci": 1139.3028524197
|
||||
},
|
||||
{
|
||||
"lower_ci": 1132.1800230236,
|
||||
"model": "gpt-4-0125-preview",
|
||||
"rank": 17,
|
||||
"rating": 1133.974390087,
|
||||
"upper_ci": 1135.7552442263
|
||||
},
|
||||
{
|
||||
"lower_ci": 1127.3300289872,
|
||||
"model": "yi-large-preview",
|
||||
"rank": 18,
|
||||
"rating": 1128.1692229165,
|
||||
"upper_ci": 1131.371743138
|
||||
},
|
||||
{
|
||||
"lower_ci": 1111.850158929,
|
||||
"model": "reka-core-20240722",
|
||||
"rank": 19,
|
||||
"rating": 1117.1464671073,
|
||||
"upper_ci": 1121.9537941313
|
||||
},
|
||||
{
|
||||
"lower_ci": 1114.0244965884,
|
||||
"model": "gemini-1.5-flash-api-0514",
|
||||
"rank": 20,
|
||||
"rating": 1116.6161359294,
|
||||
"upper_ci": 1118.5223962538
|
||||
},
|
||||
{
|
||||
"lower_ci": 1103.6704090106,
|
||||
"model": "deepseek-v2-api-0628",
|
||||
"rank": 21,
|
||||
"rating": 1107.2808888587,
|
||||
"upper_ci": 1112.0417779712
|
||||
},
|
||||
{
|
||||
"lower_ci": 1103.9432951297,
|
||||
"model": "gemma-2-27b-it",
|
||||
"rank": 22,
|
||||
"rating": 1105.9132519337,
|
||||
"upper_ci": 1108.0293102426
|
||||
},
|
||||
{
|
||||
"lower_ci": 1099.3459906048,
|
||||
"model": "deepseek-coder-v2-0724",
|
||||
"rank": 23,
|
||||
"rating": 1105.3954927337,
|
||||
"upper_ci": 1113.0353361228
|
||||
},
|
||||
{
|
||||
"lower_ci": 1099.9821845459,
|
||||
"model": "yi-large",
|
||||
"rank": 24,
|
||||
"rating": 1102.4018397329,
|
||||
"upper_ci": 1104.8525831946
|
||||
},
|
||||
{
|
||||
"lower_ci": 1094.5198632757,
|
||||
"model": "nemotron-4-340b-instruct",
|
||||
"rank": 25,
|
||||
"rating": 1098.0765768899,
|
||||
"upper_ci": 1100.3082488796
|
||||
},
|
||||
{
|
||||
"lower_ci": 1092.3225621341,
|
||||
"model": "bard-jan-24-gemini-pro",
|
||||
"rank": 26,
|
||||
"rating": 1096.589845604,
|
||||
"upper_ci": 1104.5333605578
|
||||
},
|
||||
{
|
||||
"lower_ci": 1089.9500461084,
|
||||
"model": "glm-4-0520",
|
||||
"rank": 27,
|
||||
"rating": 1095.5341569364,
|
||||
"upper_ci": 1100.1005750187
|
||||
},
|
||||
{
|
||||
"lower_ci": 1093.6731280724,
|
||||
"model": "llama-3-70b-instruct",
|
||||
"rank": 28,
|
||||
"rating": 1094.9281802329,
|
||||
"upper_ci": 1095.7283536945
|
||||
},
|
||||
{
|
||||
"lower_ci": 1088.0897564085,
|
||||
"model": "claude-3-sonnet-20240229",
|
||||
"rank": 29,
|
||||
"rating": 1090.1434308802,
|
||||
"upper_ci": 1091.5131310821
|
||||
},
|
||||
{
|
||||
"lower_ci": 1086.224736458,
|
||||
"model": "reka-core-20240501",
|
||||
"rank": 30,
|
||||
"rating": 1088.4321613308,
|
||||
"upper_ci": 1090.272779249
|
||||
},
|
||||
{
|
||||
"lower_ci": 1082.4844014079,
|
||||
"model": "reka-flash-20240722",
|
||||
"rank": 31,
|
||||
"rating": 1088.2114217704,
|
||||
"upper_ci": 1094.1409244278
|
||||
},
|
||||
{
|
||||
"lower_ci": 1076.2505646464,
|
||||
"model": "command-r-plus",
|
||||
"rank": 32,
|
||||
"rating": 1078.6392855764,
|
||||
"upper_ci": 1080.5241685262
|
||||
},
|
||||
{
|
||||
"lower_ci": 1072.978332193,
|
||||
"model": "gemma-2-9b-it",
|
||||
"rank": 33,
|
||||
"rating": 1076.1200490762,
|
||||
"upper_ci": 1078.284506667
|
||||
},
|
||||
{
|
||||
"lower_ci": 1073.5327980749,
|
||||
"model": "qwen2-72b-instruct",
|
||||
"rank": 34,
|
||||
"rating": 1075.9397511888,
|
||||
"upper_ci": 1077.9102798056
|
||||
},
|
||||
{
|
||||
"lower_ci": 1073.0022184434,
|
||||
"model": "gpt-4-0314",
|
||||
"rank": 35,
|
||||
"rating": 1074.4688060969,
|
||||
"upper_ci": 1077.3841437021
|
||||
},
|
||||
{
|
||||
"lower_ci": 1069.6849684012,
|
||||
"model": "qwen-max-0428",
|
||||
"rank": 36,
|
||||
"rating": 1072.288830872,
|
||||
"upper_ci": 1075.2788486953
|
||||
},
|
||||
{
|
||||
"lower_ci": 1067.4064473159,
|
||||
"model": "glm-4-0116",
|
||||
"rank": 37,
|
||||
"rating": 1072.1854778301,
|
||||
"upper_ci": 1078.0278828583
|
||||
},
|
||||
{
|
||||
"lower_ci": 1065.2785446605,
|
||||
"model": "claude-3-haiku-20240307",
|
||||
"rank": 38,
|
||||
"rating": 1067.2876085746,
|
||||
"upper_ci": 1068.5180263057
|
||||
},
|
||||
{
|
||||
"lower_ci": 1063.4499078695,
|
||||
"model": "deepseek-coder-v2",
|
||||
"rank": 39,
|
||||
"rating": 1066.5634197273,
|
||||
"upper_ci": 1071.0372811761
|
||||
},
|
||||
{
|
||||
"lower_ci": 1052.4060997176,
|
||||
"model": "llama-3.1-8b-instruct",
|
||||
"rank": 40,
|
||||
"rating": 1057.2617029177,
|
||||
"upper_ci": 1062.4851344029
|
||||
},
|
||||
{
|
||||
"lower_ci": 1050.6516231153,
|
||||
"model": "reka-flash-preview-20240611",
|
||||
"rank": 41,
|
||||
"rating": 1053.4286082128,
|
||||
"upper_ci": 1056.9047978837
|
||||
},
|
||||
{
|
||||
"lower_ci": 1049.878634752,
|
||||
"model": "gpt-4-0613",
|
||||
"rank": 42,
|
||||
"rating": 1051.2288727429,
|
||||
"upper_ci": 1053.3859970522
|
||||
},
|
||||
{
|
||||
"lower_ci": 1047.658821981,
|
||||
"model": "qwen1.5-110b-chat",
|
||||
"rank": 43,
|
||||
"rating": 1050.1472801797,
|
||||
"upper_ci": 1054.1870464263
|
||||
},
|
||||
{
|
||||
"lower_ci": 1043.4416581432,
|
||||
"model": "yi-1.5-34b-chat",
|
||||
"rank": 44,
|
||||
"rating": 1046.5798307425,
|
||||
"upper_ci": 1048.4870877901
|
||||
},
|
||||
{
|
||||
"lower_ci": 1043.9717227179,
|
||||
"model": "mistral-large-2402",
|
||||
"rank": 45,
|
||||
"rating": 1045.8549746499,
|
||||
"upper_ci": 1048.6558688014
|
||||
},
|
||||
{
|
||||
"lower_ci": 1039.8435107486,
|
||||
"model": "reka-flash-21b-20240226-online",
|
||||
"rank": 46,
|
||||
"rating": 1044.4541273285,
|
||||
"upper_ci": 1047.5722059799
|
||||
},
|
||||
{
|
||||
"lower_ci": 1038.3457593513,
|
||||
"model": "llama-3-8b-instruct",
|
||||
"rank": 47,
|
||||
"rating": 1040.8943517232,
|
||||
"upper_ci": 1042.1727089946
|
||||
},
|
||||
{
|
||||
"lower_ci": 1034.3532315209,
|
||||
"model": "command-r",
|
||||
"rank": 48,
|
||||
"rating": 1037.5954245649,
|
||||
"upper_ci": 1039.801976874
|
||||
},
|
||||
{
|
||||
"lower_ci": 1033.8210405932,
|
||||
"model": "claude-1",
|
||||
"rank": 49,
|
||||
"rating": 1037.2264989242,
|
||||
"upper_ci": 1039.8445566211
|
||||
},
|
||||
{
|
||||
"lower_ci": 1032.4181357042,
|
||||
"model": "reka-flash-21b-20240226",
|
||||
"rank": 50,
|
||||
"rating": 1036.2083492306,
|
||||
"upper_ci": 1038.9143450059
|
||||
},
|
||||
{
|
||||
"lower_ci": 1033.5592570692,
|
||||
"model": "mistral-medium",
|
||||
"rank": 51,
|
||||
"rating": 1035.9830773208,
|
||||
"upper_ci": 1038.0506828269
|
||||
},
|
||||
{
|
||||
"lower_ci": 1033.4716682328,
|
||||
"model": "mixtral-8x22b-instruct-v0.1",
|
||||
"rank": 52,
|
||||
"rating": 1035.6187089038,
|
||||
"upper_ci": 1037.3385699205
|
||||
},
|
||||
{
|
||||
"lower_ci": 1032.3748800585,
|
||||
"model": "qwen1.5-72b-chat",
|
||||
"rank": 53,
|
||||
"rating": 1035.5734020095,
|
||||
"upper_ci": 1037.8584457516
|
||||
},
|
||||
{
|
||||
"lower_ci": 1016.6859168483,
|
||||
"model": "claude-2.0",
|
||||
"rank": 54,
|
||||
"rating": 1020.4032512576,
|
||||
"upper_ci": 1023.0900087939
|
||||
},
|
||||
{
|
||||
"lower_ci": 1016.4818650296,
|
||||
"model": "gemini-pro-dev-api",
|
||||
"rank": 55,
|
||||
"rating": 1018.8216738996,
|
||||
"upper_ci": 1022.0575603351
|
||||
},
|
||||
{
|
||||
"lower_ci": 1013.9644553596,
|
||||
"model": "gemma-2-2b-it",
|
||||
"rank": 56,
|
||||
"rating": 1018.7496539694,
|
||||
"upper_ci": 1022.6257555787
|
||||
},
|
||||
{
|
||||
"lower_ci": 1009.8189712109,
|
||||
"model": "zephyr-orpo-141b-A35b-v0.1",
|
||||
"rank": 57,
|
||||
"rating": 1017.4768858149,
|
||||
"upper_ci": 1023.4231756618
|
||||
},
|
||||
{
|
||||
"lower_ci": 1011.7444717478,
|
||||
"model": "qwen1.5-32b-chat",
|
||||
"rank": 58,
|
||||
"rating": 1014.2192047782,
|
||||
"upper_ci": 1017.4848181067
|
||||
},
|
||||
{
|
||||
"lower_ci": 1009.527805507,
|
||||
"model": "mistral-next",
|
||||
"rank": 59,
|
||||
"rating": 1013.8190347615,
|
||||
"upper_ci": 1016.756252395
|
||||
},
|
||||
{
|
||||
"lower_ci": 1008.0043265533,
|
||||
"model": "phi-3-medium-4k-instruct",
|
||||
"rank": 60,
|
||||
"rating": 1011.6064259691,
|
||||
"upper_ci": 1014.8977711664
|
||||
},
|
||||
{
|
||||
"lower_ci": 1003.3664596337,
|
||||
"model": "claude-2.1",
|
||||
"rank": 61,
|
||||
"rating": 1006.9894951694,
|
||||
"upper_ci": 1008.7578243389
|
||||
},
|
||||
{
|
||||
"lower_ci": 1003.5743630599,
|
||||
"model": "starling-lm-7b-beta",
|
||||
"rank": 62,
|
||||
"rating": 1006.9783218473,
|
||||
"upper_ci": 1009.6647795891
|
||||
},
|
||||
{
|
||||
"lower_ci": 1003.3560004715,
|
||||
"model": "gpt-3.5-turbo-0613",
|
||||
"rank": 63,
|
||||
"rating": 1004.9730864576,
|
||||
"upper_ci": 1008.7646121872
|
||||
},
|
||||
{
|
||||
"lower_ci": 1000.3011942689,
|
||||
"model": "mixtral-8x7b-instruct-v0.1",
|
||||
"rank": 64,
|
||||
"rating": 1002.12545233,
|
||||
"upper_ci": 1004.4555563028
|
||||
},
|
||||
{
|
||||
"lower_ci": 996.8355186436,
|
||||
"model": "yi-34b-chat",
|
||||
"rank": 65,
|
||||
"rating": 1000.1263930743,
|
||||
"upper_ci": 1002.9767969377
|
||||
},
|
||||
{
|
||||
"lower_ci": 996.5513388826,
|
||||
"model": "claude-instant-1",
|
||||
"rank": 66,
|
||||
"rating": 999.4536204497,
|
||||
"upper_ci": 1003.0420312018
|
||||
},
|
||||
{
|
||||
"lower_ci": 991.5815494729,
|
||||
"model": "gemini-pro",
|
||||
"rank": 67,
|
||||
"rating": 998.3909699055,
|
||||
"upper_ci": 1005.6665144072
|
||||
},
|
||||
{
|
||||
"lower_ci": 994.5272838271,
|
||||
"model": "qwen1.5-14b-chat",
|
||||
"rank": 68,
|
||||
"rating": 997.6714948516,
|
||||
"upper_ci": 1000.3276398407
|
||||
},
|
||||
{
|
||||
"lower_ci": 988.7882364736,
|
||||
"model": "gpt-3.5-turbo-0314",
|
||||
"rank": 69,
|
||||
"rating": 997.4790766208,
|
||||
"upper_ci": 1005.2920277339
|
||||
},
|
||||
{
|
||||
"lower_ci": 989.2933547328,
|
||||
"model": "wizardlm-70b",
|
||||
"rank": 70,
|
||||
"rating": 995.5069328559,
|
||||
"upper_ci": 1000.0203225061
|
||||
},
|
||||
{
|
||||
"lower_ci": 992.3677866278,
|
||||
"model": "gpt-3.5-turbo-0125",
|
||||
"rank": 71,
|
||||
"rating": 994.221194786,
|
||||
"upper_ci": 996.0922832004
|
||||
},
|
||||
{
|
||||
"lower_ci": 989.517752069,
|
||||
"model": "dbrx-instruct-preview",
|
||||
"rank": 72,
|
||||
"rating": 991.6558051925,
|
||||
"upper_ci": 994.3414735308
|
||||
},
|
||||
{
|
||||
"lower_ci": 988.3935167102,
|
||||
"model": "phi-3-small-8k-instruct",
|
||||
"rank": 73,
|
||||
"rating": 990.3926872181,
|
||||
"upper_ci": 993.4532484928
|
||||
},
|
||||
{
|
||||
"lower_ci": 980.2003929739,
|
||||
"model": "tulu-2-dpo-70b",
|
||||
"rank": 74,
|
||||
"rating": 986.8619167507,
|
||||
"upper_ci": 995.6082960114
|
||||
},
|
||||
{
|
||||
"lower_ci": 979.2393563337,
|
||||
"model": "llama-2-70b-chat",
|
||||
"rank": 75,
|
||||
"rating": 982.0382072414,
|
||||
"upper_ci": 983.1882888775
|
||||
},
|
||||
{
|
||||
"lower_ci": 976.2082709482,
|
||||
"model": "openchat-3.5-0106",
|
||||
"rank": 76,
|
||||
"rating": 980.6486425506,
|
||||
"upper_ci": 984.7448454329
|
||||
},
|
||||
{
|
||||
"lower_ci": 976.21123294,
|
||||
"model": "vicuna-33b",
|
||||
"rank": 77,
|
||||
"rating": 978.8419077515,
|
||||
"upper_ci": 983.0972163635
|
||||
},
|
||||
{
|
||||
"lower_ci": 976.387496134,
|
||||
"model": "snowflake-arctic-instruct",
|
||||
"rank": 78,
|
||||
"rating": 978.4458039182,
|
||||
"upper_ci": 981.5060581663
|
||||
},
|
||||
{
|
||||
"lower_ci": 969.5806601345,
|
||||
"model": "starling-lm-7b-alpha",
|
||||
"rank": 79,
|
||||
"rating": 976.025808172,
|
||||
"upper_ci": 980.086762811
|
||||
},
|
||||
{
|
||||
"lower_ci": 965.8992419629,
|
||||
"model": "nous-hermes-2-mixtral-8x7b-dpo",
|
||||
"rank": 80,
|
||||
"rating": 975.4699172429,
|
||||
"upper_ci": 981.307231701
|
||||
},
|
||||
{
|
||||
"lower_ci": 969.0894947313,
|
||||
"model": "gemma-1.1-7b-it",
|
||||
"rank": 81,
|
||||
"rating": 971.847848073,
|
||||
"upper_ci": 975.4703266199
|
||||
},
|
||||
{
|
||||
"lower_ci": 966.6172052307,
|
||||
"model": "llama2-70b-steerlm-chat",
|
||||
"rank": 82,
|
||||
"rating": 971.6537111318,
|
||||
"upper_ci": 979.8268878882
|
||||
},
|
||||
{
|
||||
"lower_ci": 961.617441776,
|
||||
"model": "pplx-70b-online",
|
||||
"rank": 83,
|
||||
"rating": 967.4367908935,
|
||||
"upper_ci": 972.8126406202
|
||||
},
|
||||
{
|
||||
"lower_ci": 959.5692952959,
|
||||
"model": "openchat-3.5",
|
||||
"rank": 84,
|
||||
"rating": 966.0510063396,
|
||||
"upper_ci": 971.8466170437
|
||||
},
|
||||
{
|
||||
"lower_ci": 956.7287406383,
|
||||
"model": "deepseek-llm-67b-chat",
|
||||
"rank": 85,
|
||||
"rating": 965.1147523375,
|
||||
"upper_ci": 971.4330625559
|
||||
},
|
||||
{
|
||||
"lower_ci": 956.3934469962,
|
||||
"model": "openhermes-2.5-mistral-7b",
|
||||
"rank": 86,
|
||||
"rating": 963.454411119,
|
||||
"upper_ci": 968.2145476373
|
||||
},
|
||||
{
|
||||
"lower_ci": 957.6795493747,
|
||||
"model": "mistral-7b-instruct-v0.2",
|
||||
"rank": 87,
|
||||
"rating": 961.195768793,
|
||||
"upper_ci": 964.3273838331
|
||||
},
|
||||
{
|
||||
"lower_ci": 955.9914563912,
|
||||
"model": "qwen1.5-7b-chat",
|
||||
"rank": 88,
|
||||
"rating": 959.7566395303,
|
||||
"upper_ci": 963.8434392791
|
||||
},
|
||||
{
|
||||
"lower_ci": 953.2396686903,
|
||||
"model": "phi-3-mini-4k-instruct-june-2024",
|
||||
"rank": 89,
|
||||
"rating": 958.8907631692,
|
||||
"upper_ci": 964.0199135738
|
||||
},
|
||||
{
|
||||
"lower_ci": 953.2564762147,
|
||||
"model": "gpt-3.5-turbo-1106",
|
||||
"rank": 90,
|
||||
"rating": 955.8557887224,
|
||||
"upper_ci": 959.2854635955
|
||||
},
|
||||
{
|
||||
"lower_ci": 951.2318268117,
|
||||
"model": "phi-3-mini-4k-instruct",
|
||||
"rank": 91,
|
||||
"rating": 955.5964299136,
|
||||
"upper_ci": 958.4767635868
|
||||
},
|
||||
{
|
||||
"lower_ci": 939.9598426545,
|
||||
"model": "dolphin-2.2.1-mistral-7b",
|
||||
"rank": 92,
|
||||
"rating": 952.2051167375,
|
||||
"upper_ci": 962.2676381777
|
||||
},
|
||||
{
|
||||
"lower_ci": 945.2431328588,
|
||||
"model": "solar-10.7b-instruct-v1.0",
|
||||
"rank": 93,
|
||||
"rating": 951.1224233106,
|
||||
"upper_ci": 958.6524028214
|
||||
},
|
||||
{
|
||||
"lower_ci": 948.6812080723,
|
||||
"model": "llama-2-13b-chat",
|
||||
"rank": 94,
|
||||
"rating": 951.0465511917,
|
||||
"upper_ci": 953.7276982632
|
||||
},
|
||||
{
|
||||
"lower_ci": 940.2487365654,
|
||||
"model": "wizardlm-13b",
|
||||
"rank": 95,
|
||||
"rating": 945.6170139554,
|
||||
"upper_ci": 952.3153173674
|
||||
},
|
||||
{
|
||||
"lower_ci": 937.3497020159,
|
||||
"model": "zephyr-7b-beta",
|
||||
"rank": 96,
|
||||
"rating": 941.7856240916,
|
||||
"upper_ci": 945.7193614882
|
||||
},
|
||||
{
|
||||
"lower_ci": 925.3514960917,
|
||||
"model": "mpt-30b-chat",
|
||||
"rank": 97,
|
||||
"rating": 933.4637447927,
|
||||
"upper_ci": 942.1936901157
|
||||
},
|
||||
{
|
||||
"lower_ci": 928.7552755121,
|
||||
"model": "pplx-7b-online",
|
||||
"rank": 98,
|
||||
"rating": 932.9740226673,
|
||||
"upper_ci": 940.2890736975
|
||||
},
|
||||
{
|
||||
"lower_ci": 924.8240945839,
|
||||
"model": "codellama-34b-instruct",
|
||||
"rank": 99,
|
||||
"rating": 931.25075507,
|
||||
"upper_ci": 936.6721012017
|
||||
},
|
||||
{
|
||||
"lower_ci": 919.9794338309,
|
||||
"model": "zephyr-7b-alpha",
|
||||
"rank": 100,
|
||||
"rating": 930.8024303718,
|
||||
"upper_ci": 940.3904190626
|
||||
},
|
||||
{
|
||||
"lower_ci": 927.6228696967,
|
||||
"model": "vicuna-13b",
|
||||
"rank": 101,
|
||||
"rating": 930.129806663,
|
||||
"upper_ci": 933.8873904378
|
||||
},
|
||||
{
|
||||
"lower_ci": 916.3079081804,
|
||||
"model": "codellama-70b-instruct",
|
||||
"rank": 102,
|
||||
"rating": 929.7441814853,
|
||||
"upper_ci": 939.4638589466
|
||||
},
|
||||
{
|
||||
"lower_ci": 919.0275736817,
|
||||
"model": "gemma-7b-it",
|
||||
"rank": 103,
|
||||
"rating": 925.6063141152,
|
||||
"upper_ci": 931.2321796245
|
||||
},
|
||||
{
|
||||
"lower_ci": 921.4335696267,
|
||||
"model": "llama-2-7b-chat",
|
||||
"rank": 104,
|
||||
"rating": 925.1226871516,
|
||||
"upper_ci": 927.4211774328
|
||||
},
|
||||
{
|
||||
"lower_ci": 919.914948524,
|
||||
"model": "phi-3-mini-128k-instruct",
|
||||
"rank": 105,
|
||||
"rating": 925.0941128748,
|
||||
"upper_ci": 928.953409783
|
||||
},
|
||||
{
|
||||
"lower_ci": 917.0649637172,
|
||||
"model": "qwen-14b-chat",
|
||||
"rank": 106,
|
||||
"rating": 923.0565707958,
|
||||
"upper_ci": 927.5147584562
|
||||
},
|
||||
{
|
||||
"lower_ci": 910.6316722798,
|
||||
"model": "falcon-180b-chat",
|
||||
"rank": 107,
|
||||
"rating": 922.8642075646,
|
||||
"upper_ci": 938.170426621
|
||||
},
|
||||
{
|
||||
"lower_ci": 912.8204058608,
|
||||
"model": "guanaco-33b",
|
||||
"rank": 108,
|
||||
"rating": 921.0201740806,
|
||||
"upper_ci": 929.2566479601
|
||||
},
|
||||
{
|
||||
"lower_ci": 903.2450471973,
|
||||
"model": "gemma-1.1-2b-it",
|
||||
"rank": 109,
|
||||
"rating": 908.7494526802,
|
||||
"upper_ci": 913.4741035764
|
||||
},
|
||||
{
|
||||
"lower_ci": 899.9846491667,
|
||||
"model": "stripedhyena-nous-7b",
|
||||
"rank": 110,
|
||||
"rating": 905.1534113535,
|
||||
"upper_ci": 912.9382997405
|
||||
},
|
||||
{
|
||||
"lower_ci": 899.9839053334,
|
||||
"model": "olmo-7b-instruct",
|
||||
"rank": 111,
|
||||
"rating": 903.629872537,
|
||||
"upper_ci": 908.2570306538
|
||||
},
|
||||
{
|
||||
"lower_ci": 890.8398870858,
|
||||
"model": "mistral-7b-instruct",
|
||||
"rank": 112,
|
||||
"rating": 897.2720855909,
|
||||
"upper_ci": 902.9292885859
|
||||
},
|
||||
{
|
||||
"lower_ci": 887.0047037055,
|
||||
"model": "vicuna-7b",
|
||||
"rank": 113,
|
||||
"rating": 893.825818387,
|
||||
"upper_ci": 899.7593423485
|
||||
},
|
||||
{
|
||||
"lower_ci": 887.0036385995,
|
||||
"model": "palm-2",
|
||||
"rank": 114,
|
||||
"rating": 891.3072880052,
|
||||
"upper_ci": 897.1963496269
|
||||
},
|
||||
{
|
||||
"lower_ci": 872.1578125039,
|
||||
"model": "gemma-2b-it",
|
||||
"rank": 115,
|
||||
"rating": 880.4830688196,
|
||||
"upper_ci": 884.0952002573
|
||||
},
|
||||
{
|
||||
"lower_ci": 868.4676861275,
|
||||
"model": "qwen1.5-4b-chat",
|
||||
"rank": 116,
|
||||
"rating": 877.0664166393,
|
||||
"upper_ci": 883.3511645246
|
||||
},
|
||||
{
|
||||
"lower_ci": 846.7609834499,
|
||||
"model": "koala-13b",
|
||||
"rank": 117,
|
||||
"rating": 852.5741703315,
|
||||
"upper_ci": 859.5370102373
|
||||
},
|
||||
{
|
||||
"lower_ci": 833.5366986653,
|
||||
"model": "chatglm3-6b",
|
||||
"rank": 118,
|
||||
"rating": 843.5467457427,
|
||||
"upper_ci": 853.6979083485
|
||||
},
|
||||
{
|
||||
"lower_ci": 808.672593832,
|
||||
"model": "gpt4all-13b-snoozy",
|
||||
"rank": 119,
|
||||
"rating": 821.0385947815,
|
||||
"upper_ci": 835.6716204673
|
||||
},
|
||||
{
|
||||
"lower_ci": 804.1197078456,
|
||||
"model": "chatglm2-6b",
|
||||
"rank": 120,
|
||||
"rating": 815.3737962537,
|
||||
"upper_ci": 821.0889058237
|
||||
},
|
||||
{
|
||||
"lower_ci": 805.0790145038,
|
||||
"model": "mpt-7b-chat",
|
||||
"rank": 121,
|
||||
"rating": 813.8798502443,
|
||||
"upper_ci": 823.374846381
|
||||
},
|
||||
{
|
||||
"lower_ci": 803.3055604626,
|
||||
"model": "RWKV-4-Raven-14B",
|
||||
"rank": 122,
|
||||
"rating": 810.0683748343,
|
||||
"upper_ci": 819.7203248755
|
||||
},
|
||||
{
|
||||
"lower_ci": 781.8369679949,
|
||||
"model": "alpaca-13b",
|
||||
"rank": 123,
|
||||
"rating": 789.0910768147,
|
||||
"upper_ci": 797.3012759671
|
||||
},
|
||||
{
|
||||
"lower_ci": 778.7113024193,
|
||||
"model": "oasst-pythia-12b",
|
||||
"rank": 124,
|
||||
"rating": 782.6327874047,
|
||||
"upper_ci": 789.0013974908
|
||||
},
|
||||
{
|
||||
"lower_ci": 760.5962982346,
|
||||
"model": "chatglm-6b",
|
||||
"rank": 125,
|
||||
"rating": 767.7699631784,
|
||||
"upper_ci": 776.2247501319
|
||||
},
|
||||
{
|
||||
"lower_ci": 749.711207841,
|
||||
"model": "fastchat-t5-3b",
|
||||
"rank": 126,
|
||||
"rating": 756.8214872887,
|
||||
"upper_ci": 762.6617679978
|
||||
},
|
||||
{
|
||||
"lower_ci": 723.1589751426,
|
||||
"model": "stablelm-tuned-alpha-7b",
|
||||
"rank": 127,
|
||||
"rating": 727.6986691281,
|
||||
"upper_ci": 737.7996673488
|
||||
},
|
||||
{
|
||||
"lower_ci": 701.8523006848,
|
||||
"model": "dolly-v2-12b",
|
||||
"rank": 128,
|
||||
"rating": 710.1023148692,
|
||||
"upper_ci": 717.2721178625
|
||||
},
|
||||
{
|
||||
"lower_ci": 679.2645194103,
|
||||
"model": "llama-13b",
|
||||
"rank": 129,
|
||||
"rating": 689.1631346499,
|
||||
"upper_ci": 694.5590720343
|
||||
}
|
||||
]
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
+2224
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"artifacts": {
|
||||
"bradley_terry.json": {
|
||||
"bytes": 19827,
|
||||
"sha256": "4a861b80c69132fdb3e3d1cfdc9d962444d664623b956595e67798d141ee1c6b"
|
||||
},
|
||||
"leaderboard.png": {
|
||||
"bytes": 118255,
|
||||
"sha256": "386889f573198987400fef7c7803e0779b95e2fa35cb274d04071fed842c6499"
|
||||
},
|
||||
"leaderboard_animation.html": {
|
||||
"bytes": 54322,
|
||||
"sha256": "afb9b0426605dcc5accf9e1ea5067dd885ad9a9226884cc341b3430404ca433a"
|
||||
},
|
||||
"online_elo.json": {
|
||||
"bytes": 16885,
|
||||
"sha256": "41ec9a46bb37457a385d0c4b4036ff8ae3d1affcbae8b9b00689fef494f9f9a6"
|
||||
},
|
||||
"rating_history.json": {
|
||||
"bytes": 354362,
|
||||
"sha256": "3a5e3193cbbe50bf1de3a9e5d9702f2b594f449300d195674ed0da2a76f72574"
|
||||
},
|
||||
"rating_history.png": {
|
||||
"bytes": 193995,
|
||||
"sha256": "cb676a82b38a8876bc1346e58f72b6a9d72c8fa5e405a9d22610f6948382fba4"
|
||||
},
|
||||
"summary.json": {
|
||||
"bytes": 10441,
|
||||
"sha256": "6258914ebdb6be7686985e637903c929ed14535d8572a4b9d42b1256d5cdf261"
|
||||
},
|
||||
"win_rate_matrix.json": {
|
||||
"bytes": 37040,
|
||||
"sha256": "a6ac9580a8d0290d53adf16458ac777d72e214d0771cd0e72d1b2d8a4d86abd2"
|
||||
},
|
||||
"win_rate_matrix.png": {
|
||||
"bytes": 230883,
|
||||
"sha256": "db375436646096bfafdad458b53aba54ac156432cf9e8bb7458a2f733c9ae690"
|
||||
}
|
||||
},
|
||||
"experiment": "7-7",
|
||||
"gates": {
|
||||
"bradley_terry_official_method_completed": true,
|
||||
"chronological_online_elo_k4_completed": true,
|
||||
"d3_animation_saved": true,
|
||||
"millions_of_blind_votes_loaded": true,
|
||||
"monthly_history_saved": true,
|
||||
"official_public_arena_snapshot_hashed": true,
|
||||
"online_vs_official_method_rank_agreement_observed": true,
|
||||
"pairwise_empirical_and_predicted_matrix_saved": true,
|
||||
"static_visualizations_saved": true
|
||||
},
|
||||
"input": {
|
||||
"bytes": 2131976365,
|
||||
"filename": "arena_data.json",
|
||||
"sha256": "747c1c937dfa941d5a455a1fd70e2879d29642058da39b97996b977233b9bd1b",
|
||||
"url": "https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json"
|
||||
},
|
||||
"official_complete": true,
|
||||
"runtime": {
|
||||
"numpy": "1.26.4",
|
||||
"pandas": "2.3.3",
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"python": "3.12.11"
|
||||
},
|
||||
"schema_version": 1,
|
||||
"sources": {
|
||||
"animation.py": "6839a3173a6a9ee8855901ca3c781fd91de13219b52ce6b285778ab89d926376",
|
||||
"bradley_terry.py": "030c716284017b14cff204ecd7036020c05554ae0eacf71af5da5c0963848b54",
|
||||
"optimized_elo.py": "f56b5b4aaba9cda11006bf01742be99fc07ee44ceeed5bd2dee5322a8ce08a57",
|
||||
"validation/run_experiment.py": "6c668081c4fbf04f9dbc84411a2a40391d7b00c76a90336542cc6a88623e991c",
|
||||
"validation/validate_evidence.py": "36fe448c94608cfc6a299f21224870357d109d999ba3391ed7d7c71fc1817e62"
|
||||
},
|
||||
"status": "passed"
|
||||
}
|
||||
@@ -0,0 +1,905 @@
|
||||
[
|
||||
{
|
||||
"matches": 55654,
|
||||
"model": "gemini-1.5-pro-api-0409-preview",
|
||||
"rank": 1,
|
||||
"rating": 1159.8203895968,
|
||||
"wins": 33738.5
|
||||
},
|
||||
{
|
||||
"matches": 20071,
|
||||
"model": "gemini-1.5-pro-exp-0801",
|
||||
"rank": 2,
|
||||
"rating": 1127.8877886511,
|
||||
"wins": 12139.5
|
||||
},
|
||||
{
|
||||
"matches": 14514,
|
||||
"model": "chatgpt-4o-latest",
|
||||
"rank": 3,
|
||||
"rating": 1126.6935165964,
|
||||
"wins": 8999.0
|
||||
},
|
||||
{
|
||||
"matches": 5655,
|
||||
"model": "gpt-3.5-turbo-0314",
|
||||
"rank": 4,
|
||||
"rating": 1103.9751202512,
|
||||
"wins": 3789.0
|
||||
},
|
||||
{
|
||||
"matches": 11827,
|
||||
"model": "bard-jan-24-gemini-pro",
|
||||
"rank": 5,
|
||||
"rating": 1094.2118678517,
|
||||
"wins": 7075.5
|
||||
},
|
||||
{
|
||||
"matches": 21172,
|
||||
"model": "claude-1",
|
||||
"rank": 6,
|
||||
"rating": 1092.7309868534,
|
||||
"wins": 12081.5
|
||||
},
|
||||
{
|
||||
"matches": 52155,
|
||||
"model": "gemini-advanced-0514",
|
||||
"rank": 7,
|
||||
"rating": 1087.5408038371,
|
||||
"wins": 30523.0
|
||||
},
|
||||
{
|
||||
"matches": 13604,
|
||||
"model": "llama-3.1-70b-instruct",
|
||||
"rank": 8,
|
||||
"rating": 1087.3529845908,
|
||||
"wins": 7058.5
|
||||
},
|
||||
{
|
||||
"matches": 77509,
|
||||
"model": "gpt-4o-2024-05-13",
|
||||
"rank": 9,
|
||||
"rating": 1086.7242770205,
|
||||
"wins": 46951.0
|
||||
},
|
||||
{
|
||||
"matches": 9761,
|
||||
"model": "gpt-4o-2024-08-06",
|
||||
"rank": 10,
|
||||
"rating": 1085.601246973,
|
||||
"wins": 5323.0
|
||||
},
|
||||
{
|
||||
"matches": 19370,
|
||||
"model": "gpt-4o-mini-2024-07-18",
|
||||
"rank": 11,
|
||||
"rating": 1079.187026245,
|
||||
"wins": 11082.0
|
||||
},
|
||||
{
|
||||
"matches": 47703,
|
||||
"model": "claude-3-5-sonnet-20240620",
|
||||
"rank": 12,
|
||||
"rating": 1079.0096726066,
|
||||
"wins": 27107.5
|
||||
},
|
||||
{
|
||||
"matches": 155944,
|
||||
"model": "claude-3-opus-20240229",
|
||||
"rank": 13,
|
||||
"rating": 1074.9078256001,
|
||||
"wins": 91432.5
|
||||
},
|
||||
{
|
||||
"matches": 11476,
|
||||
"model": "athene-70b-0725",
|
||||
"rank": 14,
|
||||
"rating": 1073.7607622933,
|
||||
"wins": 6098.0
|
||||
},
|
||||
{
|
||||
"matches": 18790,
|
||||
"model": "gemini-pro-dev-api",
|
||||
"rank": 15,
|
||||
"rating": 1072.3072263884,
|
||||
"wins": 8728.5
|
||||
},
|
||||
{
|
||||
"matches": 12774,
|
||||
"model": "claude-2.0",
|
||||
"rank": 16,
|
||||
"rating": 1069.530762176,
|
||||
"wins": 6814.5
|
||||
},
|
||||
{
|
||||
"matches": 10239,
|
||||
"model": "glm-4-0520",
|
||||
"rank": 17,
|
||||
"rating": 1065.2967580553,
|
||||
"wins": 5154.5
|
||||
},
|
||||
{
|
||||
"matches": 20669,
|
||||
"model": "nemotron-4-340b-instruct",
|
||||
"rank": 18,
|
||||
"rating": 1063.9797181947,
|
||||
"wins": 10567.5
|
||||
},
|
||||
{
|
||||
"matches": 51739,
|
||||
"model": "yi-large-preview",
|
||||
"rank": 19,
|
||||
"rating": 1060.8561460851,
|
||||
"wins": 28793.0
|
||||
},
|
||||
{
|
||||
"matches": 39631,
|
||||
"model": "llama-2-70b-chat",
|
||||
"rank": 20,
|
||||
"rating": 1060.6350445555,
|
||||
"wins": 18272.0
|
||||
},
|
||||
{
|
||||
"matches": 6974,
|
||||
"model": "reka-core-20240722",
|
||||
"rank": 21,
|
||||
"rating": 1059.9327626435,
|
||||
"wins": 3490.0
|
||||
},
|
||||
{
|
||||
"matches": 69418,
|
||||
"model": "gemini-1.5-pro-api-0514",
|
||||
"rank": 22,
|
||||
"rating": 1059.8892020659,
|
||||
"wins": 39302.5
|
||||
},
|
||||
{
|
||||
"matches": 6565,
|
||||
"model": "gemini-pro",
|
||||
"rank": 23,
|
||||
"rating": 1059.2020149016,
|
||||
"wins": 3028.0
|
||||
},
|
||||
{
|
||||
"matches": 18897,
|
||||
"model": "llama-3.1-405b-instruct",
|
||||
"rank": 24,
|
||||
"rating": 1058.4003562569,
|
||||
"wins": 10361.5
|
||||
},
|
||||
{
|
||||
"matches": 11517,
|
||||
"model": "mistral-large-2407",
|
||||
"rank": 25,
|
||||
"rating": 1058.1277726399,
|
||||
"wins": 6038.5
|
||||
},
|
||||
{
|
||||
"matches": 84505,
|
||||
"model": "gpt-4-turbo-2024-04-09",
|
||||
"rank": 26,
|
||||
"rating": 1054.9014485987,
|
||||
"wins": 48844.5
|
||||
},
|
||||
{
|
||||
"matches": 92488,
|
||||
"model": "gpt-4-1106-preview",
|
||||
"rank": 27,
|
||||
"rating": 1054.2069148591,
|
||||
"wins": 57496.5
|
||||
},
|
||||
{
|
||||
"matches": 38959,
|
||||
"model": "gpt-3.5-turbo-0613",
|
||||
"rank": 28,
|
||||
"rating": 1052.6707187439,
|
||||
"wins": 19308.0
|
||||
},
|
||||
{
|
||||
"matches": 85867,
|
||||
"model": "gpt-4-0125-preview",
|
||||
"rank": 29,
|
||||
"rating": 1050.1218586343,
|
||||
"wins": 50802.5
|
||||
},
|
||||
{
|
||||
"matches": 7579,
|
||||
"model": "glm-4-0116",
|
||||
"rank": 30,
|
||||
"rating": 1049.1894798578,
|
||||
"wins": 3566.0
|
||||
},
|
||||
{
|
||||
"matches": 4858,
|
||||
"model": "zephyr-orpo-141b-A35b-v0.1",
|
||||
"rank": 31,
|
||||
"rating": 1045.2216087838,
|
||||
"wins": 2116.5
|
||||
},
|
||||
{
|
||||
"matches": 25748,
|
||||
"model": "qwen-max-0428",
|
||||
"rank": 32,
|
||||
"rating": 1044.8335252342,
|
||||
"wins": 12269.5
|
||||
},
|
||||
{
|
||||
"matches": 20633,
|
||||
"model": "claude-instant-1",
|
||||
"rank": 33,
|
||||
"rating": 1044.7562986355,
|
||||
"wins": 11178.0
|
||||
},
|
||||
{
|
||||
"matches": 16672,
|
||||
"model": "yi-large",
|
||||
"rank": 34,
|
||||
"rating": 1044.7374546104,
|
||||
"wins": 8533.5
|
||||
},
|
||||
{
|
||||
"matches": 3997,
|
||||
"model": "deepseek-coder-v2-0724",
|
||||
"rank": 35,
|
||||
"rating": 1043.8935002553,
|
||||
"wins": 1883.5
|
||||
},
|
||||
{
|
||||
"matches": 16165,
|
||||
"model": "deepseek-v2-api-0628",
|
||||
"rank": 36,
|
||||
"rating": 1042.4852039574,
|
||||
"wins": 7994.5
|
||||
},
|
||||
{
|
||||
"matches": 56129,
|
||||
"model": "gemini-1.5-flash-api-0514",
|
||||
"rank": 37,
|
||||
"rating": 1041.3766839839,
|
||||
"wins": 29778.0
|
||||
},
|
||||
{
|
||||
"matches": 161827,
|
||||
"model": "llama-3-70b-instruct",
|
||||
"rank": 38,
|
||||
"rating": 1040.8719399645,
|
||||
"wins": 84977.0
|
||||
},
|
||||
{
|
||||
"matches": 80911,
|
||||
"model": "command-r-plus",
|
||||
"rank": 39,
|
||||
"rating": 1034.8822379837,
|
||||
"wins": 40967.0
|
||||
},
|
||||
{
|
||||
"matches": 55979,
|
||||
"model": "gpt-4-0314",
|
||||
"rank": 40,
|
||||
"rating": 1033.6412751608,
|
||||
"wins": 31098.0
|
||||
},
|
||||
{
|
||||
"matches": 37702,
|
||||
"model": "claude-2.1",
|
||||
"rank": 41,
|
||||
"rating": 1032.8143823524,
|
||||
"wins": 16848.0
|
||||
},
|
||||
{
|
||||
"matches": 8391,
|
||||
"model": "wizardlm-70b",
|
||||
"rank": 42,
|
||||
"rating": 1032.5111769139,
|
||||
"wins": 4325.0
|
||||
},
|
||||
{
|
||||
"matches": 27838,
|
||||
"model": "gemma-2-27b-it",
|
||||
"rank": 43,
|
||||
"rating": 1030.7976774989,
|
||||
"wins": 13981.5
|
||||
},
|
||||
{
|
||||
"matches": 1714,
|
||||
"model": "dolphin-2.2.1-mistral-7b",
|
||||
"rank": 44,
|
||||
"rating": 1029.9575607586,
|
||||
"wins": 738.5
|
||||
},
|
||||
{
|
||||
"matches": 3003,
|
||||
"model": "guanaco-33b",
|
||||
"rank": 45,
|
||||
"rating": 1027.2582838587,
|
||||
"wins": 1552.0
|
||||
},
|
||||
{
|
||||
"matches": 3843,
|
||||
"model": "nous-hermes-2-mixtral-8x7b-dpo",
|
||||
"rank": 46,
|
||||
"rating": 1025.2145335894,
|
||||
"wins": 1697.5
|
||||
},
|
||||
{
|
||||
"matches": 7192,
|
||||
"model": "wizardlm-13b",
|
||||
"rank": 47,
|
||||
"rating": 1024.793903105,
|
||||
"wins": 3690.0
|
||||
},
|
||||
{
|
||||
"matches": 2650,
|
||||
"model": "mpt-30b-chat",
|
||||
"rank": 48,
|
||||
"rating": 1024.0563241095,
|
||||
"wins": 1351.5
|
||||
},
|
||||
{
|
||||
"matches": 27508,
|
||||
"model": "qwen1.5-110b-chat",
|
||||
"rank": 49,
|
||||
"rating": 1023.7761579902,
|
||||
"wins": 12612.0
|
||||
},
|
||||
{
|
||||
"matches": 113106,
|
||||
"model": "claude-3-sonnet-20240229",
|
||||
"rank": 50,
|
||||
"rating": 1023.6338810846,
|
||||
"wins": 60284.5
|
||||
},
|
||||
{
|
||||
"matches": 12377,
|
||||
"model": "mistral-next",
|
||||
"rank": 51,
|
||||
"rating": 1021.9755255724,
|
||||
"wins": 5952.5
|
||||
},
|
||||
{
|
||||
"matches": 15801,
|
||||
"model": "deepseek-coder-v2",
|
||||
"rank": 52,
|
||||
"rating": 1021.6778455715,
|
||||
"wins": 7243.0
|
||||
},
|
||||
{
|
||||
"matches": 16056,
|
||||
"model": "reka-flash-21b-20240226-online",
|
||||
"rank": 53,
|
||||
"rating": 1020.8238893282,
|
||||
"wins": 7554.5
|
||||
},
|
||||
{
|
||||
"matches": 16667,
|
||||
"model": "starling-lm-7b-beta",
|
||||
"rank": 54,
|
||||
"rating": 1020.1246920766,
|
||||
"wins": 7626.0
|
||||
},
|
||||
{
|
||||
"matches": 3636,
|
||||
"model": "llama2-70b-steerlm-chat",
|
||||
"rank": 55,
|
||||
"rating": 1018.5882277478,
|
||||
"wins": 1587.5
|
||||
},
|
||||
{
|
||||
"matches": 35555,
|
||||
"model": "mistral-medium",
|
||||
"rank": 56,
|
||||
"rating": 1018.0043978294,
|
||||
"wins": 17731.5
|
||||
},
|
||||
{
|
||||
"matches": 19740,
|
||||
"model": "llama-2-13b-chat",
|
||||
"rank": 57,
|
||||
"rating": 1016.6652779795,
|
||||
"wins": 8935.5
|
||||
},
|
||||
{
|
||||
"matches": 6660,
|
||||
"model": "tulu-2-dpo-70b",
|
||||
"rank": 58,
|
||||
"rating": 1014.6804751782,
|
||||
"wins": 3231.0
|
||||
},
|
||||
{
|
||||
"matches": 62688,
|
||||
"model": "reka-core-20240501",
|
||||
"rank": 59,
|
||||
"rating": 1013.8222528866,
|
||||
"wins": 31198.0
|
||||
},
|
||||
{
|
||||
"matches": 89617,
|
||||
"model": "gpt-4-0613",
|
||||
"rank": 60,
|
||||
"rating": 1013.3276942754,
|
||||
"wins": 43878.0
|
||||
},
|
||||
{
|
||||
"matches": 4981,
|
||||
"model": "deepseek-llm-67b-chat",
|
||||
"rank": 61,
|
||||
"rating": 1011.6783077357,
|
||||
"wins": 2017.0
|
||||
},
|
||||
{
|
||||
"matches": 4291,
|
||||
"model": "solar-10.7b-instruct-v1.0",
|
||||
"rank": 62,
|
||||
"rating": 1011.6055213958,
|
||||
"wins": 1814.0
|
||||
},
|
||||
{
|
||||
"matches": 12987,
|
||||
"model": "openchat-3.5-0106",
|
||||
"rank": 63,
|
||||
"rating": 1009.5145969972,
|
||||
"wins": 5974.0
|
||||
},
|
||||
{
|
||||
"matches": 7156,
|
||||
"model": "reka-flash-20240722",
|
||||
"rank": 64,
|
||||
"rating": 1003.5048766039,
|
||||
"wins": 3290.0
|
||||
},
|
||||
{
|
||||
"matches": 25248,
|
||||
"model": "gemma-2-9b-it",
|
||||
"rank": 65,
|
||||
"rating": 1003.026164166,
|
||||
"wins": 11661.5
|
||||
},
|
||||
{
|
||||
"matches": 12671,
|
||||
"model": "llama-3.1-8b-instruct",
|
||||
"rank": 66,
|
||||
"rating": 1002.2631876345,
|
||||
"wins": 5121.0
|
||||
},
|
||||
{
|
||||
"matches": 8117,
|
||||
"model": "openchat-3.5",
|
||||
"rank": 67,
|
||||
"rating": 1001.939993772,
|
||||
"wins": 3783.0
|
||||
},
|
||||
{
|
||||
"matches": 6338,
|
||||
"model": "pplx-7b-online",
|
||||
"rank": 68,
|
||||
"rating": 1001.6927968751,
|
||||
"wins": 2572.0
|
||||
},
|
||||
{
|
||||
"matches": 40643,
|
||||
"model": "qwen1.5-72b-chat",
|
||||
"rank": 69,
|
||||
"rating": 1000.9425509065,
|
||||
"wins": 19941.0
|
||||
},
|
||||
{
|
||||
"matches": 1815,
|
||||
"model": "zephyr-7b-alpha",
|
||||
"rank": 70,
|
||||
"rating": 999.8845726251,
|
||||
"wins": 855.0
|
||||
},
|
||||
{
|
||||
"matches": 109888,
|
||||
"model": "claude-3-haiku-20240307",
|
||||
"rank": 71,
|
||||
"rating": 999.7072031054,
|
||||
"wins": 55074.0
|
||||
},
|
||||
{
|
||||
"matches": 10422,
|
||||
"model": "starling-lm-7b-alpha",
|
||||
"rank": 72,
|
||||
"rating": 998.0936577805,
|
||||
"wins": 4817.0
|
||||
},
|
||||
{
|
||||
"matches": 25792,
|
||||
"model": "reka-flash-21b-20240226",
|
||||
"rank": 73,
|
||||
"rating": 992.8107482264,
|
||||
"wins": 11891.0
|
||||
},
|
||||
{
|
||||
"matches": 64938,
|
||||
"model": "mistral-large-2402",
|
||||
"rank": 74,
|
||||
"rating": 992.5035561283,
|
||||
"wins": 30754.5
|
||||
},
|
||||
{
|
||||
"matches": 17026,
|
||||
"model": "gpt-3.5-turbo-1106",
|
||||
"rank": 75,
|
||||
"rating": 990.5828502496,
|
||||
"wins": 7026.0
|
||||
},
|
||||
{
|
||||
"matches": 4864,
|
||||
"model": "qwen1.5-7b-chat",
|
||||
"rank": 76,
|
||||
"rating": 990.5290268684,
|
||||
"wins": 1903.5
|
||||
},
|
||||
{
|
||||
"matches": 20477,
|
||||
"model": "reka-flash-preview-20240611",
|
||||
"rank": 77,
|
||||
"rating": 990.4289024537,
|
||||
"wins": 9003.5
|
||||
},
|
||||
{
|
||||
"matches": 25156,
|
||||
"model": "yi-1.5-34b-chat",
|
||||
"rank": 78,
|
||||
"rating": 988.8079557694,
|
||||
"wins": 11066.0
|
||||
},
|
||||
{
|
||||
"matches": 5091,
|
||||
"model": "openhermes-2.5-mistral-7b",
|
||||
"rank": 79,
|
||||
"rating": 988.507956639,
|
||||
"wins": 2379.0
|
||||
},
|
||||
{
|
||||
"matches": 7508,
|
||||
"model": "codellama-34b-instruct",
|
||||
"rank": 80,
|
||||
"rating": 986.9963372601,
|
||||
"wins": 3192.5
|
||||
},
|
||||
{
|
||||
"matches": 18666,
|
||||
"model": "qwen1.5-14b-chat",
|
||||
"rank": 81,
|
||||
"rating": 985.79676379,
|
||||
"wins": 8265.0
|
||||
},
|
||||
{
|
||||
"matches": 15946,
|
||||
"model": "yi-34b-chat",
|
||||
"rank": 82,
|
||||
"rating": 984.9789753527,
|
||||
"wins": 7423.5
|
||||
},
|
||||
{
|
||||
"matches": 6890,
|
||||
"model": "pplx-70b-online",
|
||||
"rank": 83,
|
||||
"rating": 984.0082457269,
|
||||
"wins": 3118.0
|
||||
},
|
||||
{
|
||||
"matches": 34410,
|
||||
"model": "qwen2-72b-instruct",
|
||||
"rank": 84,
|
||||
"rating": 983.7228839439,
|
||||
"wins": 16578.5
|
||||
},
|
||||
{
|
||||
"matches": 33736,
|
||||
"model": "dbrx-instruct-preview",
|
||||
"rank": 85,
|
||||
"rating": 983.0655232047,
|
||||
"wins": 14350.5
|
||||
},
|
||||
{
|
||||
"matches": 107100,
|
||||
"model": "llama-3-8b-instruct",
|
||||
"rank": 86,
|
||||
"rating": 978.2506897907,
|
||||
"wins": 49020.5
|
||||
},
|
||||
{
|
||||
"matches": 1325,
|
||||
"model": "falcon-180b-chat",
|
||||
"rank": 87,
|
||||
"rating": 976.0053410687,
|
||||
"wins": 580.0
|
||||
},
|
||||
{
|
||||
"matches": 8745,
|
||||
"model": "palm-2",
|
||||
"rank": 88,
|
||||
"rating": 975.4724123669,
|
||||
"wins": 3851.5
|
||||
},
|
||||
{
|
||||
"matches": 5070,
|
||||
"model": "qwen-14b-chat",
|
||||
"rank": 89,
|
||||
"rating": 974.4393986925,
|
||||
"wins": 2262.5
|
||||
},
|
||||
{
|
||||
"matches": 5270,
|
||||
"model": "stripedhyena-nous-7b",
|
||||
"rank": 90,
|
||||
"rating": 972.5214538963,
|
||||
"wins": 2001.0
|
||||
},
|
||||
{
|
||||
"matches": 22783,
|
||||
"model": "qwen1.5-32b-chat",
|
||||
"rank": 91,
|
||||
"rating": 970.5573357551,
|
||||
"wins": 10038.5
|
||||
},
|
||||
{
|
||||
"matches": 56373,
|
||||
"model": "command-r",
|
||||
"rank": 92,
|
||||
"rating": 968.3429828775,
|
||||
"wins": 25830.5
|
||||
},
|
||||
{
|
||||
"matches": 9178,
|
||||
"model": "gemma-7b-it",
|
||||
"rank": 93,
|
||||
"rating": 966.2526965022,
|
||||
"wins": 3447.5
|
||||
},
|
||||
{
|
||||
"matches": 11330,
|
||||
"model": "zephyr-7b-beta",
|
||||
"rank": 94,
|
||||
"rating": 966.1969243012,
|
||||
"wins": 5268.0
|
||||
},
|
||||
{
|
||||
"matches": 50256,
|
||||
"model": "mixtral-8x22b-instruct-v0.1",
|
||||
"rank": 95,
|
||||
"rating": 964.4141739849,
|
||||
"wins": 22236.0
|
||||
},
|
||||
{
|
||||
"matches": 7022,
|
||||
"model": "vicuna-7b",
|
||||
"rank": 96,
|
||||
"rating": 963.6362226151,
|
||||
"wins": 3187.5
|
||||
},
|
||||
{
|
||||
"matches": 34197,
|
||||
"model": "snowflake-arctic-instruct",
|
||||
"rank": 97,
|
||||
"rating": 961.9084215198,
|
||||
"wins": 13326.5
|
||||
},
|
||||
{
|
||||
"matches": 22938,
|
||||
"model": "vicuna-33b",
|
||||
"rank": 98,
|
||||
"rating": 959.3823284501,
|
||||
"wins": 10924.5
|
||||
},
|
||||
{
|
||||
"matches": 10509,
|
||||
"model": "gemma-2-2b-it",
|
||||
"rank": 99,
|
||||
"rating": 959.2713799268,
|
||||
"wins": 3731.5
|
||||
},
|
||||
{
|
||||
"matches": 7038,
|
||||
"model": "koala-13b",
|
||||
"rank": 100,
|
||||
"rating": 954.395027749,
|
||||
"wins": 3380.0
|
||||
},
|
||||
{
|
||||
"matches": 68914,
|
||||
"model": "gpt-3.5-turbo-0125",
|
||||
"rank": 101,
|
||||
"rating": 951.9181205239,
|
||||
"wins": 29165.5
|
||||
},
|
||||
{
|
||||
"matches": 9145,
|
||||
"model": "mistral-7b-instruct",
|
||||
"rank": 102,
|
||||
"rating": 948.891187234,
|
||||
"wins": 3685.5
|
||||
},
|
||||
{
|
||||
"matches": 14559,
|
||||
"model": "llama-2-7b-chat",
|
||||
"rank": 103,
|
||||
"rating": 948.4212635676,
|
||||
"wins": 6212.0
|
||||
},
|
||||
{
|
||||
"matches": 20074,
|
||||
"model": "mistral-7b-instruct-v0.2",
|
||||
"rank": 104,
|
||||
"rating": 945.8420359878,
|
||||
"wins": 8282.5
|
||||
},
|
||||
{
|
||||
"matches": 25095,
|
||||
"model": "gemma-1.1-7b-it",
|
||||
"rank": 105,
|
||||
"rating": 941.8141470663,
|
||||
"wins": 9687.0
|
||||
},
|
||||
{
|
||||
"matches": 1787,
|
||||
"model": "gpt4all-13b-snoozy",
|
||||
"rank": 106,
|
||||
"rating": 938.6958755482,
|
||||
"wins": 680.0
|
||||
},
|
||||
{
|
||||
"matches": 18505,
|
||||
"model": "phi-3-small-8k-instruct",
|
||||
"rank": 107,
|
||||
"rating": 930.6944270029,
|
||||
"wins": 7100.0
|
||||
},
|
||||
{
|
||||
"matches": 6496,
|
||||
"model": "olmo-7b-instruct",
|
||||
"rank": 108,
|
||||
"rating": 928.8306135323,
|
||||
"wins": 2157.0
|
||||
},
|
||||
{
|
||||
"matches": 21129,
|
||||
"model": "phi-3-mini-4k-instruct",
|
||||
"rank": 109,
|
||||
"rating": 928.4371869598,
|
||||
"wins": 7094.0
|
||||
},
|
||||
{
|
||||
"matches": 21617,
|
||||
"model": "phi-3-mini-128k-instruct",
|
||||
"rank": 110,
|
||||
"rating": 928.3336295806,
|
||||
"wins": 7094.5
|
||||
},
|
||||
{
|
||||
"matches": 4938,
|
||||
"model": "RWKV-4-Raven-14B",
|
||||
"rank": 111,
|
||||
"rating": 927.486785137,
|
||||
"wins": 1965.5
|
||||
},
|
||||
{
|
||||
"matches": 19794,
|
||||
"model": "vicuna-13b",
|
||||
"rank": 112,
|
||||
"rating": 927.1632498042,
|
||||
"wins": 9553.5
|
||||
},
|
||||
{
|
||||
"matches": 1193,
|
||||
"model": "codellama-70b-instruct",
|
||||
"rank": 113,
|
||||
"rating": 926.7141337337,
|
||||
"wins": 411.5
|
||||
},
|
||||
{
|
||||
"matches": 4018,
|
||||
"model": "mpt-7b-chat",
|
||||
"rank": 114,
|
||||
"rating": 922.0268289601,
|
||||
"wins": 1573.5
|
||||
},
|
||||
{
|
||||
"matches": 76064,
|
||||
"model": "mixtral-8x7b-instruct-v0.1",
|
||||
"rank": 115,
|
||||
"rating": 920.8542542685,
|
||||
"wins": 33437.0
|
||||
},
|
||||
{
|
||||
"matches": 22427,
|
||||
"model": "phi-3-medium-4k-instruct",
|
||||
"rank": 116,
|
||||
"rating": 919.7045738753,
|
||||
"wins": 8854.0
|
||||
},
|
||||
{
|
||||
"matches": 4921,
|
||||
"model": "gemma-2b-it",
|
||||
"rank": 117,
|
||||
"rating": 918.581510353,
|
||||
"wins": 1513.0
|
||||
},
|
||||
{
|
||||
"matches": 10484,
|
||||
"model": "phi-3-mini-4k-instruct-june-2024",
|
||||
"rank": 118,
|
||||
"rating": 913.3885931532,
|
||||
"wins": 3097.5
|
||||
},
|
||||
{
|
||||
"matches": 7811,
|
||||
"model": "qwen1.5-4b-chat",
|
||||
"rank": 119,
|
||||
"rating": 910.8121423587,
|
||||
"wins": 2364.5
|
||||
},
|
||||
{
|
||||
"matches": 4995,
|
||||
"model": "chatglm-6b",
|
||||
"rank": 120,
|
||||
"rating": 908.8611775605,
|
||||
"wins": 1834.5
|
||||
},
|
||||
{
|
||||
"matches": 5874,
|
||||
"model": "alpaca-13b",
|
||||
"rank": 121,
|
||||
"rating": 904.2619553966,
|
||||
"wins": 2330.0
|
||||
},
|
||||
{
|
||||
"matches": 11363,
|
||||
"model": "gemma-1.1-2b-it",
|
||||
"rank": 122,
|
||||
"rating": 892.2955737397,
|
||||
"wins": 3282.0
|
||||
},
|
||||
{
|
||||
"matches": 2707,
|
||||
"model": "chatglm2-6b",
|
||||
"rank": 123,
|
||||
"rating": 888.2580854124,
|
||||
"wins": 808.0
|
||||
},
|
||||
{
|
||||
"matches": 3334,
|
||||
"model": "stablelm-tuned-alpha-7b",
|
||||
"rank": 124,
|
||||
"rating": 882.7115402984,
|
||||
"wins": 1141.5
|
||||
},
|
||||
{
|
||||
"matches": 4763,
|
||||
"model": "chatglm3-6b",
|
||||
"rank": 125,
|
||||
"rating": 867.498613261,
|
||||
"wins": 1454.0
|
||||
},
|
||||
{
|
||||
"matches": 6382,
|
||||
"model": "oasst-pythia-12b",
|
||||
"rank": 126,
|
||||
"rating": 865.3424632797,
|
||||
"wins": 2425.0
|
||||
},
|
||||
{
|
||||
"matches": 2443,
|
||||
"model": "llama-13b",
|
||||
"rank": 127,
|
||||
"rating": 852.9506118392,
|
||||
"wins": 745.5
|
||||
},
|
||||
{
|
||||
"matches": 4304,
|
||||
"model": "fastchat-t5-3b",
|
||||
"rank": 128,
|
||||
"rating": 847.3254654092,
|
||||
"wins": 1465.0
|
||||
},
|
||||
{
|
||||
"matches": 3484,
|
||||
"model": "dolly-v2-12b",
|
||||
"rank": 129,
|
||||
"rating": 834.3876685486,
|
||||
"wins": 1099.5
|
||||
}
|
||||
]
|
||||
+17546
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 189 KiB |
@@ -0,0 +1,387 @@
|
||||
{
|
||||
"dataset": {
|
||||
"accepted_records": 1670250,
|
||||
"bounded_test_run": false,
|
||||
"bytes": 2131976365,
|
||||
"end_utc": "2024-08-14T19:52:53.409200+00:00",
|
||||
"load_filter_seconds": 15.133,
|
||||
"model_count": 129,
|
||||
"outcomes": {
|
||||
"model_a": 543065,
|
||||
"model_b": 550810,
|
||||
"tie": 281121,
|
||||
"tie (bothbad)": 295254
|
||||
},
|
||||
"path_recorded_as": "arena_data.json",
|
||||
"sha256": "747c1c937dfa941d5a455a1fd70e2879d29642058da39b97996b977233b9bd1b",
|
||||
"source_records": 1799991,
|
||||
"start_utc": "2023-04-24T15:53:11.132200+00:00",
|
||||
"url": "https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json"
|
||||
},
|
||||
"experiment": "7-7",
|
||||
"gates": {
|
||||
"bradley_terry_official_method_completed": true,
|
||||
"chronological_online_elo_k4_completed": true,
|
||||
"d3_animation_saved": true,
|
||||
"millions_of_blind_votes_loaded": true,
|
||||
"monthly_history_saved": true,
|
||||
"official_public_arena_snapshot_hashed": true,
|
||||
"online_vs_official_method_rank_agreement_observed": true,
|
||||
"pairwise_empirical_and_predicted_matrix_saved": true,
|
||||
"static_visualizations_saved": true
|
||||
},
|
||||
"generated_at_utc": "2026-07-31T09:59:17.815099+00:00",
|
||||
"official_complete": true,
|
||||
"protocol": {
|
||||
"bootstrap_random_seed": 0,
|
||||
"bootstrap_rounds": 20,
|
||||
"history_interval": "monthly cumulative snapshots",
|
||||
"official_method": "Bradley-Terry maximum-likelihood reconstruction",
|
||||
"online_elo": "initial=1000, K=4, stable chronological order"
|
||||
},
|
||||
"results": {
|
||||
"official_method_top_20": [
|
||||
{
|
||||
"lower_ci": 1199.4356201692,
|
||||
"model": "chatgpt-4o-latest",
|
||||
"rank": 1,
|
||||
"rating": 1202.8722324768,
|
||||
"upper_ci": 1207.3805726125
|
||||
},
|
||||
{
|
||||
"lower_ci": 1182.767114327,
|
||||
"model": "gemini-1.5-pro-exp-0801",
|
||||
"rank": 2,
|
||||
"rating": 1187.3727762743,
|
||||
"upper_ci": 1191.7564879767
|
||||
},
|
||||
{
|
||||
"lower_ci": 1173.1270556478,
|
||||
"model": "gpt-4o-2024-05-13",
|
||||
"rank": 3,
|
||||
"rating": 1174.5475812185,
|
||||
"upper_ci": 1177.0403252386
|
||||
},
|
||||
{
|
||||
"lower_ci": 1161.1978031325,
|
||||
"model": "gpt-4o-mini-2024-07-18",
|
||||
"rank": 4,
|
||||
"rating": 1162.9712796813,
|
||||
"upper_ci": 1166.568334134
|
||||
},
|
||||
{
|
||||
"lower_ci": 1156.0224735749,
|
||||
"model": "claude-3-5-sonnet-20240620",
|
||||
"rank": 5,
|
||||
"rating": 1159.6692177384,
|
||||
"upper_ci": 1161.8458744744
|
||||
},
|
||||
{
|
||||
"lower_ci": 1153.6447817793,
|
||||
"model": "gemini-advanced-0514",
|
||||
"rank": 6,
|
||||
"rating": 1155.6305094344,
|
||||
"upper_ci": 1157.1030226933
|
||||
},
|
||||
{
|
||||
"lower_ci": 1150.4614200682,
|
||||
"model": "llama-3.1-405b-instruct",
|
||||
"rank": 7,
|
||||
"rating": 1153.3950109278,
|
||||
"upper_ci": 1157.0848375454
|
||||
},
|
||||
{
|
||||
"lower_ci": 1146.348878889,
|
||||
"model": "gpt-4o-2024-08-06",
|
||||
"rank": 8,
|
||||
"rating": 1150.7428051987,
|
||||
"upper_ci": 1156.3245456986
|
||||
},
|
||||
{
|
||||
"lower_ci": 1146.7772301944,
|
||||
"model": "gemini-1.5-pro-api-0514",
|
||||
"rank": 9,
|
||||
"rating": 1149.1265569601,
|
||||
"upper_ci": 1151.5010539806
|
||||
},
|
||||
{
|
||||
"lower_ci": 1143.6682527667,
|
||||
"model": "gemini-1.5-pro-api-0409-preview",
|
||||
"rank": 10,
|
||||
"rating": 1146.1756037638,
|
||||
"upper_ci": 1147.8895040945
|
||||
},
|
||||
{
|
||||
"lower_ci": 1142.3990049162,
|
||||
"model": "gpt-4-turbo-2024-04-09",
|
||||
"rank": 11,
|
||||
"rating": 1144.9187340632,
|
||||
"upper_ci": 1146.8362809167
|
||||
},
|
||||
{
|
||||
"lower_ci": 1138.1303324947,
|
||||
"model": "gpt-4-1106-preview",
|
||||
"rank": 12,
|
||||
"rating": 1139.5004505155,
|
||||
"upper_ci": 1140.97644635
|
||||
},
|
||||
{
|
||||
"lower_ci": 1136.3852091186,
|
||||
"model": "mistral-large-2407",
|
||||
"rank": 13,
|
||||
"rating": 1139.3566231011,
|
||||
"upper_ci": 1143.8960727747
|
||||
},
|
||||
{
|
||||
"lower_ci": 1135.0624903381,
|
||||
"model": "athene-70b-0725",
|
||||
"rank": 14,
|
||||
"rating": 1138.2016223593,
|
||||
"upper_ci": 1141.397916447
|
||||
},
|
||||
{
|
||||
"lower_ci": 1135.0278139027,
|
||||
"model": "claude-3-opus-20240229",
|
||||
"rank": 15,
|
||||
"rating": 1136.5739464324,
|
||||
"upper_ci": 1137.5663039742
|
||||
},
|
||||
{
|
||||
"lower_ci": 1131.7219814044,
|
||||
"model": "llama-3.1-70b-instruct",
|
||||
"rank": 16,
|
||||
"rating": 1134.8942150397,
|
||||
"upper_ci": 1139.3028524197
|
||||
},
|
||||
{
|
||||
"lower_ci": 1132.1800230236,
|
||||
"model": "gpt-4-0125-preview",
|
||||
"rank": 17,
|
||||
"rating": 1133.974390087,
|
||||
"upper_ci": 1135.7552442263
|
||||
},
|
||||
{
|
||||
"lower_ci": 1127.3300289872,
|
||||
"model": "yi-large-preview",
|
||||
"rank": 18,
|
||||
"rating": 1128.1692229165,
|
||||
"upper_ci": 1131.371743138
|
||||
},
|
||||
{
|
||||
"lower_ci": 1111.850158929,
|
||||
"model": "reka-core-20240722",
|
||||
"rank": 19,
|
||||
"rating": 1117.1464671073,
|
||||
"upper_ci": 1121.9537941313
|
||||
},
|
||||
{
|
||||
"lower_ci": 1114.0244965884,
|
||||
"model": "gemini-1.5-flash-api-0514",
|
||||
"rank": 20,
|
||||
"rating": 1116.6161359294,
|
||||
"upper_ci": 1118.5223962538
|
||||
}
|
||||
],
|
||||
"online_top_20": [
|
||||
{
|
||||
"matches": 55654,
|
||||
"model": "gemini-1.5-pro-api-0409-preview",
|
||||
"rank": 1,
|
||||
"rating": 1159.8203895968,
|
||||
"wins": 33738.5
|
||||
},
|
||||
{
|
||||
"matches": 20071,
|
||||
"model": "gemini-1.5-pro-exp-0801",
|
||||
"rank": 2,
|
||||
"rating": 1127.8877886511,
|
||||
"wins": 12139.5
|
||||
},
|
||||
{
|
||||
"matches": 14514,
|
||||
"model": "chatgpt-4o-latest",
|
||||
"rank": 3,
|
||||
"rating": 1126.6935165964,
|
||||
"wins": 8999.0
|
||||
},
|
||||
{
|
||||
"matches": 5655,
|
||||
"model": "gpt-3.5-turbo-0314",
|
||||
"rank": 4,
|
||||
"rating": 1103.9751202512,
|
||||
"wins": 3789.0
|
||||
},
|
||||
{
|
||||
"matches": 11827,
|
||||
"model": "bard-jan-24-gemini-pro",
|
||||
"rank": 5,
|
||||
"rating": 1094.2118678517,
|
||||
"wins": 7075.5
|
||||
},
|
||||
{
|
||||
"matches": 21172,
|
||||
"model": "claude-1",
|
||||
"rank": 6,
|
||||
"rating": 1092.7309868534,
|
||||
"wins": 12081.5
|
||||
},
|
||||
{
|
||||
"matches": 52155,
|
||||
"model": "gemini-advanced-0514",
|
||||
"rank": 7,
|
||||
"rating": 1087.5408038371,
|
||||
"wins": 30523.0
|
||||
},
|
||||
{
|
||||
"matches": 13604,
|
||||
"model": "llama-3.1-70b-instruct",
|
||||
"rank": 8,
|
||||
"rating": 1087.3529845908,
|
||||
"wins": 7058.5
|
||||
},
|
||||
{
|
||||
"matches": 77509,
|
||||
"model": "gpt-4o-2024-05-13",
|
||||
"rank": 9,
|
||||
"rating": 1086.7242770205,
|
||||
"wins": 46951.0
|
||||
},
|
||||
{
|
||||
"matches": 9761,
|
||||
"model": "gpt-4o-2024-08-06",
|
||||
"rank": 10,
|
||||
"rating": 1085.601246973,
|
||||
"wins": 5323.0
|
||||
},
|
||||
{
|
||||
"matches": 19370,
|
||||
"model": "gpt-4o-mini-2024-07-18",
|
||||
"rank": 11,
|
||||
"rating": 1079.187026245,
|
||||
"wins": 11082.0
|
||||
},
|
||||
{
|
||||
"matches": 47703,
|
||||
"model": "claude-3-5-sonnet-20240620",
|
||||
"rank": 12,
|
||||
"rating": 1079.0096726066,
|
||||
"wins": 27107.5
|
||||
},
|
||||
{
|
||||
"matches": 155944,
|
||||
"model": "claude-3-opus-20240229",
|
||||
"rank": 13,
|
||||
"rating": 1074.9078256001,
|
||||
"wins": 91432.5
|
||||
},
|
||||
{
|
||||
"matches": 11476,
|
||||
"model": "athene-70b-0725",
|
||||
"rank": 14,
|
||||
"rating": 1073.7607622933,
|
||||
"wins": 6098.0
|
||||
},
|
||||
{
|
||||
"matches": 18790,
|
||||
"model": "gemini-pro-dev-api",
|
||||
"rank": 15,
|
||||
"rating": 1072.3072263884,
|
||||
"wins": 8728.5
|
||||
},
|
||||
{
|
||||
"matches": 12774,
|
||||
"model": "claude-2.0",
|
||||
"rank": 16,
|
||||
"rating": 1069.530762176,
|
||||
"wins": 6814.5
|
||||
},
|
||||
{
|
||||
"matches": 10239,
|
||||
"model": "glm-4-0520",
|
||||
"rank": 17,
|
||||
"rating": 1065.2967580553,
|
||||
"wins": 5154.5
|
||||
},
|
||||
{
|
||||
"matches": 20669,
|
||||
"model": "nemotron-4-340b-instruct",
|
||||
"rank": 18,
|
||||
"rating": 1063.9797181947,
|
||||
"wins": 10567.5
|
||||
},
|
||||
{
|
||||
"matches": 51739,
|
||||
"model": "yi-large-preview",
|
||||
"rank": 19,
|
||||
"rating": 1060.8561460851,
|
||||
"wins": 28793.0
|
||||
},
|
||||
{
|
||||
"matches": 39631,
|
||||
"model": "llama-2-70b-chat",
|
||||
"rank": 20,
|
||||
"rating": 1060.6350445555,
|
||||
"wins": 18272.0
|
||||
}
|
||||
],
|
||||
"rank_comparison": {
|
||||
"claim_boundary": "This is a same-snapshot reconstruction of the official method, not a scrape of the mutable live leaderboard. Scores need not match the live service.",
|
||||
"common_models": 129,
|
||||
"comparison_target": "Bradley-Terry MLE reconstruction used by Chatbot Arena",
|
||||
"kendall_rank_correlation": 0.606347,
|
||||
"official_method_top_20": [
|
||||
"chatgpt-4o-latest",
|
||||
"gemini-1.5-pro-exp-0801",
|
||||
"gpt-4o-2024-05-13",
|
||||
"gpt-4o-mini-2024-07-18",
|
||||
"claude-3-5-sonnet-20240620",
|
||||
"gemini-advanced-0514",
|
||||
"llama-3.1-405b-instruct",
|
||||
"gpt-4o-2024-08-06",
|
||||
"gemini-1.5-pro-api-0514",
|
||||
"gemini-1.5-pro-api-0409-preview",
|
||||
"gpt-4-turbo-2024-04-09",
|
||||
"gpt-4-1106-preview",
|
||||
"mistral-large-2407",
|
||||
"athene-70b-0725",
|
||||
"claude-3-opus-20240229",
|
||||
"llama-3.1-70b-instruct",
|
||||
"gpt-4-0125-preview",
|
||||
"yi-large-preview",
|
||||
"reka-core-20240722",
|
||||
"gemini-1.5-flash-api-0514"
|
||||
],
|
||||
"online_top_20": [
|
||||
"gemini-1.5-pro-api-0409-preview",
|
||||
"gemini-1.5-pro-exp-0801",
|
||||
"chatgpt-4o-latest",
|
||||
"gpt-3.5-turbo-0314",
|
||||
"bard-jan-24-gemini-pro",
|
||||
"claude-1",
|
||||
"gemini-advanced-0514",
|
||||
"llama-3.1-70b-instruct",
|
||||
"gpt-4o-2024-05-13",
|
||||
"gpt-4o-2024-08-06",
|
||||
"gpt-4o-mini-2024-07-18",
|
||||
"claude-3-5-sonnet-20240620",
|
||||
"claude-3-opus-20240229",
|
||||
"athene-70b-0725",
|
||||
"gemini-pro-dev-api",
|
||||
"claude-2.0",
|
||||
"glm-4-0520",
|
||||
"nemotron-4-340b-instruct",
|
||||
"yi-large-preview",
|
||||
"llama-2-70b-chat"
|
||||
],
|
||||
"spearman_rank_correlation": 0.786717,
|
||||
"top_20_overlap": 12
|
||||
}
|
||||
},
|
||||
"schema_version": 1,
|
||||
"status": "passed",
|
||||
"timing_seconds": {
|
||||
"bradley_terry": 71.472,
|
||||
"online_and_history": 8.301,
|
||||
"total": 97.248
|
||||
}
|
||||
}
|
||||
+908
@@ -0,0 +1,908 @@
|
||||
{
|
||||
"empirical": {
|
||||
"athene-70b-0725": {
|
||||
"athene-70b-0725": 0.5,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.41589648798521256,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.49077181208053694,
|
||||
"claude-3-opus-20240229": 0.5027472527472527,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.43097014925373134,
|
||||
"gemini-advanced-0514": 0.48249027237354086,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4431239388794567,
|
||||
"gpt-4o-2024-08-06": 0.505249343832021,
|
||||
"gpt-4o-mini-2024-07-18": 0.4753787878787879,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.5317460317460317,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": 0.5294117647058824
|
||||
},
|
||||
"bard-jan-24-gemini-pro": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": 0.5,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": 0.572289156626506,
|
||||
"claude-2.0": 0.6696428571428571,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": 0.3,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": 0.6332794830371568,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": NaN,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": 0.6492537313432836,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"chatgpt-4o-latest": {
|
||||
"athene-70b-0725": 0.5841035120147874,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.5,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.5534351145038168,
|
||||
"claude-3-opus-20240229": 0.5915750915750916,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.5329736211031175,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.5442092154420921,
|
||||
"gpt-4o-2024-08-06": 0.5164835164835165,
|
||||
"gpt-4o-mini-2024-07-18": 0.6010733452593918,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.5793838862559242,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"claude-1": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": 0.42771084337349397,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": 0.5,
|
||||
"claude-2.0": 0.5460750853242321,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": 0.34452296819787986,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": 0.5342465753424658,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": 0.5548780487804879,
|
||||
"gpt-4o-2024-05-13": NaN,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": 0.604,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"claude-2.0": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": 0.33035714285714285,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": 0.4539249146757679,
|
||||
"claude-2.0": 0.5,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": NaN,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": 0.5569620253164557,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": NaN,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": 0.5866834170854272,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"claude-3-5-sonnet-20240620": {
|
||||
"athene-70b-0725": 0.5092281879194631,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.44656488549618323,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.5,
|
||||
"claude-3-opus-20240229": 0.5447761194029851,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.4537037037037037,
|
||||
"gemini-advanced-0514": 0.49312714776632305,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": 0.5800756620428752,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.47875730217737655,
|
||||
"gpt-4o-2024-08-06": 0.5189873417721519,
|
||||
"gpt-4o-mini-2024-07-18": 0.47484909456740443,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.538961038961039,
|
||||
"nemotron-4-340b-instruct": 0.589835728952772,
|
||||
"yi-large-preview": 0.5205761316872428
|
||||
},
|
||||
"claude-3-opus-20240229": {
|
||||
"athene-70b-0725": 0.49725274725274726,
|
||||
"bard-jan-24-gemini-pro": 0.7,
|
||||
"chatgpt-4o-latest": 0.4084249084249084,
|
||||
"claude-1": 0.6554770318021201,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.4552238805970149,
|
||||
"claude-3-opus-20240229": 0.5,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.4815616343490305,
|
||||
"gemini-1.5-pro-exp-0801": 0.4305157593123209,
|
||||
"gemini-advanced-0514": 0.46121231155778897,
|
||||
"gemini-pro-dev-api": 0.6489533011272142,
|
||||
"glm-4-0520": 0.5424311926605505,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4481694338016177,
|
||||
"gpt-4o-2024-08-06": 0.4369627507163324,
|
||||
"gpt-4o-mini-2024-07-18": 0.4437386569872958,
|
||||
"llama-2-70b-chat": 0.6890951276102089,
|
||||
"llama-3.1-70b-instruct": 0.4813519813519814,
|
||||
"nemotron-4-340b-instruct": 0.5431309904153354,
|
||||
"yi-large-preview": 0.5238190552441954
|
||||
},
|
||||
"gemini-1.5-pro-api-0409-preview": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": 0.5184383656509696,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.5,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": NaN,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": 0.6875,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"gemini-1.5-pro-exp-0801": {
|
||||
"athene-70b-0725": 0.5690298507462687,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.4670263788968825,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.5462962962962963,
|
||||
"claude-3-opus-20240229": 0.5694842406876791,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.5,
|
||||
"gemini-advanced-0514": 0.5263157894736842,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.5229445506692161,
|
||||
"gpt-4o-2024-08-06": 0.5241071428571429,
|
||||
"gpt-4o-mini-2024-07-18": 0.529559748427673,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.5773092369477911,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": 0.5862068965517241
|
||||
},
|
||||
"gemini-advanced-0514": {
|
||||
"athene-70b-0725": 0.5175097276264592,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.506872852233677,
|
||||
"claude-3-opus-20240229": 0.538787688442211,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.47368421052631576,
|
||||
"gemini-advanced-0514": 0.5,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": 0.5773381294964028,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4901780233271946,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": 0.4604072398190045,
|
||||
"llama-2-70b-chat": 0.6145833333333334,
|
||||
"llama-3.1-70b-instruct": 0.5423076923076923,
|
||||
"nemotron-4-340b-instruct": 0.5807407407407408,
|
||||
"yi-large-preview": 0.5602636534839924
|
||||
},
|
||||
"gemini-pro-dev-api": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": 0.3667205169628433,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": 0.4657534246575342,
|
||||
"claude-2.0": 0.4430379746835443,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": 0.35104669887278583,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": 0.5,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": NaN,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": 0.5535714285714286,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"glm-4-0520": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.41992433795712486,
|
||||
"claude-3-opus-20240229": 0.45756880733944955,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": 0.4226618705035971,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": 0.5,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.3613861386138614,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": 0.5087025316455697,
|
||||
"yi-large-preview": 0.448943661971831
|
||||
},
|
||||
"gpt-3.5-turbo-0314": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": 0.4451219512195122,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": NaN,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": 0.5,
|
||||
"gpt-4o-2024-05-13": NaN,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"gpt-4o-2024-05-13": {
|
||||
"athene-70b-0725": 0.5568760611205433,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.45579078455790784,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.5212426978226234,
|
||||
"claude-3-opus-20240229": 0.5518305661983823,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.4770554493307839,
|
||||
"gemini-advanced-0514": 0.5098219766728054,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": 0.6386138613861386,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.5,
|
||||
"gpt-4o-2024-08-06": 0.544973544973545,
|
||||
"gpt-4o-mini-2024-07-18": 0.5163472378804961,
|
||||
"llama-2-70b-chat": 0.5607476635514018,
|
||||
"llama-3.1-70b-instruct": 0.5650224215246636,
|
||||
"nemotron-4-340b-instruct": 0.5968559837728195,
|
||||
"yi-large-preview": 0.5756445047489823
|
||||
},
|
||||
"gpt-4o-2024-08-06": {
|
||||
"athene-70b-0725": 0.494750656167979,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.4835164835164835,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.4810126582278481,
|
||||
"claude-3-opus-20240229": 0.5630372492836676,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.4758928571428571,
|
||||
"gemini-advanced-0514": NaN,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.455026455026455,
|
||||
"gpt-4o-2024-08-06": 0.5,
|
||||
"gpt-4o-mini-2024-07-18": 0.5052910052910053,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.5273037542662116,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": NaN
|
||||
},
|
||||
"gpt-4o-mini-2024-07-18": {
|
||||
"athene-70b-0725": 0.5246212121212122,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.39892665474060823,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.5251509054325956,
|
||||
"claude-3-opus-20240229": 0.5562613430127041,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.47044025157232705,
|
||||
"gemini-advanced-0514": 0.5395927601809954,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4836527621195039,
|
||||
"gpt-4o-2024-08-06": 0.4947089947089947,
|
||||
"gpt-4o-mini-2024-07-18": 0.5,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.5584756898817346,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": 0.553125
|
||||
},
|
||||
"llama-2-70b-chat": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": 0.35074626865671643,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": 0.396,
|
||||
"claude-2.0": 0.41331658291457285,
|
||||
"claude-3-5-sonnet-20240620": NaN,
|
||||
"claude-3-opus-20240229": 0.3109048723897912,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3125,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": 0.3854166666666667,
|
||||
"gemini-pro-dev-api": 0.44642857142857145,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4392523364485981,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": 0.5,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": 0.42142857142857143
|
||||
},
|
||||
"llama-3.1-70b-instruct": {
|
||||
"athene-70b-0725": 0.46825396825396826,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": 0.4206161137440758,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.461038961038961,
|
||||
"claude-3-opus-20240229": 0.5186480186480187,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.4226907630522088,
|
||||
"gemini-advanced-0514": 0.4576923076923077,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": NaN,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4349775784753363,
|
||||
"gpt-4o-2024-08-06": 0.4726962457337884,
|
||||
"gpt-4o-mini-2024-07-18": 0.44152431011826543,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": 0.5,
|
||||
"nemotron-4-340b-instruct": NaN,
|
||||
"yi-large-preview": 0.58
|
||||
},
|
||||
"nemotron-4-340b-instruct": {
|
||||
"athene-70b-0725": NaN,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.4101642710472279,
|
||||
"claude-3-opus-20240229": 0.45686900958466453,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": NaN,
|
||||
"gemini-advanced-0514": 0.4192592592592593,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": 0.4912974683544304,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.40314401622718055,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": NaN,
|
||||
"llama-2-70b-chat": NaN,
|
||||
"llama-3.1-70b-instruct": NaN,
|
||||
"nemotron-4-340b-instruct": 0.5,
|
||||
"yi-large-preview": 0.44568690095846647
|
||||
},
|
||||
"yi-large-preview": {
|
||||
"athene-70b-0725": 0.47058823529411764,
|
||||
"bard-jan-24-gemini-pro": NaN,
|
||||
"chatgpt-4o-latest": NaN,
|
||||
"claude-1": NaN,
|
||||
"claude-2.0": NaN,
|
||||
"claude-3-5-sonnet-20240620": 0.4794238683127572,
|
||||
"claude-3-opus-20240229": 0.47618094475580464,
|
||||
"gemini-1.5-pro-api-0409-preview": NaN,
|
||||
"gemini-1.5-pro-exp-0801": 0.41379310344827586,
|
||||
"gemini-advanced-0514": 0.4397363465160075,
|
||||
"gemini-pro-dev-api": NaN,
|
||||
"glm-4-0520": 0.551056338028169,
|
||||
"gpt-3.5-turbo-0314": NaN,
|
||||
"gpt-4o-2024-05-13": 0.4243554952510176,
|
||||
"gpt-4o-2024-08-06": NaN,
|
||||
"gpt-4o-mini-2024-07-18": 0.446875,
|
||||
"llama-2-70b-chat": 0.5785714285714286,
|
||||
"llama-3.1-70b-instruct": 0.42,
|
||||
"nemotron-4-340b-instruct": 0.5543130990415336,
|
||||
"yi-large-preview": 0.5
|
||||
}
|
||||
},
|
||||
"models": [
|
||||
"gemini-1.5-pro-api-0409-preview",
|
||||
"gemini-1.5-pro-exp-0801",
|
||||
"chatgpt-4o-latest",
|
||||
"gpt-3.5-turbo-0314",
|
||||
"bard-jan-24-gemini-pro",
|
||||
"claude-1",
|
||||
"gemini-advanced-0514",
|
||||
"llama-3.1-70b-instruct",
|
||||
"gpt-4o-2024-05-13",
|
||||
"gpt-4o-2024-08-06",
|
||||
"gpt-4o-mini-2024-07-18",
|
||||
"claude-3-5-sonnet-20240620",
|
||||
"claude-3-opus-20240229",
|
||||
"athene-70b-0725",
|
||||
"gemini-pro-dev-api",
|
||||
"claude-2.0",
|
||||
"glm-4-0520",
|
||||
"nemotron-4-340b-instruct",
|
||||
"yi-large-preview",
|
||||
"llama-2-70b-chat"
|
||||
],
|
||||
"online_elo_predicted": {
|
||||
"athene-70b-0725": {
|
||||
"athene-70b-0725": 0.5,
|
||||
"bard-jan-24-gemini-pro": 0.4706024381703526,
|
||||
"chatgpt-4o-latest": 0.4244076072370654,
|
||||
"claude-1": 0.4727267496028405,
|
||||
"claude-2.0": 0.5060871587475949,
|
||||
"claude-3-5-sonnet-20240620": 0.4924467854859755,
|
||||
"claude-3-opus-20240229": 0.4983492492034077,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3786225197416955,
|
||||
"gemini-1.5-pro-exp-0801": 0.4227290744565508,
|
||||
"gemini-advanced-0514": 0.48017931822260296,
|
||||
"gemini-pro-dev-api": 0.5020917941129024,
|
||||
"glm-4-0520": 0.5121782721655728,
|
||||
"gpt-3.5-turbo-0314": 0.45662732701186726,
|
||||
"gpt-4o-2024-05-13": 0.4813526553622729,
|
||||
"gpt-4o-2024-08-06": 0.482966766553394,
|
||||
"gpt-4o-mini-2024-07-18": 0.492191613318456,
|
||||
"llama-2-70b-chat": 0.5188804447839834,
|
||||
"llama-3.1-70b-instruct": 0.4804491928529648,
|
||||
"nemotron-4-340b-instruct": 0.5140723365206418,
|
||||
"yi-large-preview": 0.5185627002158419
|
||||
},
|
||||
"bard-jan-24-gemini-pro": {
|
||||
"athene-70b-0725": 0.5293975618296474,
|
||||
"bard-jan-24-gemini-pro": 0.5,
|
||||
"chatgpt-4o-latest": 0.45339086380344346,
|
||||
"claude-1": 0.5021311461637057,
|
||||
"claude-2.0": 0.5354593391190725,
|
||||
"claude-3-5-sonnet-20240620": 0.5218637663710245,
|
||||
"claude-3-opus-20240229": 0.5277521980731233,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.4066882610443156,
|
||||
"gemini-1.5-pro-exp-0801": 0.45168765548015544,
|
||||
"gemini-advanced-0514": 0.5095992532128728,
|
||||
"gemini-pro-dev-api": 0.5314816122659721,
|
||||
"glm-4-0520": 0.54151638062692,
|
||||
"gpt-3.5-turbo-0314": 0.48595324754769137,
|
||||
"gpt-4o-2024-05-13": 0.5107738414883239,
|
||||
"gpt-4o-2024-08-06": 0.5123891431197954,
|
||||
"gpt-4o-mini-2024-07-18": 0.5216090163334262,
|
||||
"llama-2-70b-chat": 0.5481710593389454,
|
||||
"llama-3.1-70b-instruct": 0.5098694443797749,
|
||||
"nemotron-4-340b-instruct": 0.5433980845222826,
|
||||
"yi-large-preview": 0.5478558027577041
|
||||
},
|
||||
"chatgpt-4o-latest": {
|
||||
"athene-70b-0725": 0.5755923927629346,
|
||||
"bard-jan-24-gemini-pro": 0.5466091361965566,
|
||||
"chatgpt-4o-latest": 0.5,
|
||||
"claude-1": 0.5487209243907352,
|
||||
"claude-2.0": 0.5815294906471586,
|
||||
"claude-3-5-sonnet-20240620": 0.5681949260650095,
|
||||
"claude-3-opus-20240229": 0.5739785673924995,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.45247054063006786,
|
||||
"gemini-1.5-pro-exp-0801": 0.49828131112534013,
|
||||
"gemini-advanced-0514": 0.5561079756808931,
|
||||
"gemini-pro-dev-api": 0.5776350830826076,
|
||||
"glm-4-0520": 0.587448649362509,
|
||||
"gpt-3.5-turbo-0314": 0.5326478829339771,
|
||||
"gpt-4o-2024-05-13": 0.5572679471226406,
|
||||
"gpt-4o-2024-08-06": 0.5588623197754942,
|
||||
"gpt-4o-mini-2024-07-18": 0.5679444241258642,
|
||||
"llama-2-70b-chat": 0.5939365657179976,
|
||||
"llama-3.1-70b-instruct": 0.5563748494378427,
|
||||
"nemotron-4-340b-instruct": 0.5892848182429368,
|
||||
"yi-large-preview": 0.5936295693344523
|
||||
},
|
||||
"claude-1": {
|
||||
"athene-70b-0725": 0.5272732503971594,
|
||||
"bard-jan-24-gemini-pro": 0.4978688538362943,
|
||||
"chatgpt-4o-latest": 0.45127907560926495,
|
||||
"claude-1": 0.5,
|
||||
"claude-2.0": 0.5333382703189705,
|
||||
"claude-3-5-sonnet-20240620": 0.5197362986493318,
|
||||
"claude-3-opus-20240229": 0.5256271146687194,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.4046329739918908,
|
||||
"gemini-1.5-pro-exp-0801": 0.44957727554595445,
|
||||
"gemini-advanced-0514": 0.5074687182117393,
|
||||
"gemini-pro-dev-api": 0.5293583449328568,
|
||||
"glm-4-0520": 0.539399178222487,
|
||||
"gpt-3.5-turbo-0314": 0.483824038337802,
|
||||
"gpt-4o-2024-05-13": 0.5086434891644779,
|
||||
"gpt-4o-2024-08-06": 0.5102590804411641,
|
||||
"gpt-4o-mini-2024-07-18": 0.519481458808123,
|
||||
"llama-2-70b-chat": 0.5460588266923004,
|
||||
"llama-3.1-70b-instruct": 0.5077389493164274,
|
||||
"nemotron-4-340b-instruct": 0.541282210739082,
|
||||
"yi-large-preview": 0.5457433176189294
|
||||
},
|
||||
"claude-2.0": {
|
||||
"athene-70b-0725": 0.4939128412524052,
|
||||
"bard-jan-24-gemini-pro": 0.46454066088092755,
|
||||
"chatgpt-4o-latest": 0.41847050935284147,
|
||||
"claude-1": 0.46666172968102954,
|
||||
"claude-2.0": 0.5,
|
||||
"claude-3-5-sonnet-20240620": 0.4863621348844711,
|
||||
"claude-3-opus-20240229": 0.4922624014572012,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3729109568972296,
|
||||
"gemini-1.5-pro-exp-0801": 0.41679845455453196,
|
||||
"gemini-advanced-0514": 0.474104656737174,
|
||||
"gemini-pro-dev-api": 0.4960044318617072,
|
||||
"glm-4-0520": 0.5060929201168735,
|
||||
"gpt-3.5-turbo-0314": 0.4505923459773481,
|
||||
"gpt-4o-2024-05-13": 0.4752767219072891,
|
||||
"gpt-4o-2024-08-06": 0.4768891926892078,
|
||||
"gpt-4o-mini-2024-07-18": 0.4861070959392006,
|
||||
"llama-2-70b-chat": 0.5127991699819673,
|
||||
"llama-3.1-70b-instruct": 0.47437423287938824,
|
||||
"nemotron-4-340b-instruct": 0.5079879147696204,
|
||||
"yi-large-preview": 0.512481182668407
|
||||
},
|
||||
"claude-3-5-sonnet-20240620": {
|
||||
"athene-70b-0725": 0.5075532145140246,
|
||||
"bard-jan-24-gemini-pro": 0.4781362336289755,
|
||||
"chatgpt-4o-latest": 0.4318050739349906,
|
||||
"claude-1": 0.4802637013506682,
|
||||
"claude-2.0": 0.5136378651155289,
|
||||
"claude-3-5-sonnet-20240620": 0.5,
|
||||
"claude-3-opus-20240229": 0.505902758110997,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.38575678604471353,
|
||||
"gemini-1.5-pro-exp-0801": 0.4301191468425039,
|
||||
"gemini-advanced-0514": 0.4877251820914822,
|
||||
"gemini-pro-dev-api": 0.5096443991097894,
|
||||
"glm-4-0520": 0.519724229338601,
|
||||
"gpt-3.5-turbo-0314": 0.46413354167370846,
|
||||
"gpt-4o-2024-05-13": 0.48889961603567494,
|
||||
"gpt-4o-2024-08-06": 0.4905150999228697,
|
||||
"gpt-4o-mini-2024-07-18": 0.49974476761967085,
|
||||
"llama-2-70b-chat": 0.526418589284047,
|
||||
"llama-3.1-70b-instruct": 0.487995316371302,
|
||||
"nemotron-4-340b-instruct": 0.5216163605038188,
|
||||
"yi-large-preview": 0.5261012762929619
|
||||
},
|
||||
"claude-3-opus-20240229": {
|
||||
"athene-70b-0725": 0.5016507507965924,
|
||||
"bard-jan-24-gemini-pro": 0.47224780192687665,
|
||||
"chatgpt-4o-latest": 0.4260214326075004,
|
||||
"claude-1": 0.47437288533128064,
|
||||
"claude-2.0": 0.5077375985427988,
|
||||
"claude-3-5-sonnet-20240620": 0.49409724188900295,
|
||||
"claude-3-opus-20240229": 0.5,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3801772378799626,
|
||||
"gemini-1.5-pro-exp-0801": 0.4243412226190692,
|
||||
"gemini-advanced-0514": 0.4818276906995902,
|
||||
"gemini-pro-dev-api": 0.5037424932177174,
|
||||
"glm-4-0520": 0.513827911016007,
|
||||
"gpt-3.5-turbo-0314": 0.4582661256484433,
|
||||
"gpt-4o-2024-05-13": 0.4830013131364567,
|
||||
"gpt-4o-2024-08-06": 0.48461578707994823,
|
||||
"gpt-4o-mini-2024-07-18": 0.4938420466183484,
|
||||
"llama-2-70b-chat": 0.520528636323993,
|
||||
"llama-3.1-70b-instruct": 0.48209763256058513,
|
||||
"nemotron-4-340b-instruct": 0.5157216264686881,
|
||||
"yi-large-preview": 0.5202109737620978
|
||||
},
|
||||
"gemini-1.5-pro-api-0409-preview": {
|
||||
"athene-70b-0725": 0.6213774802583045,
|
||||
"bard-jan-24-gemini-pro": 0.5933117389556843,
|
||||
"chatgpt-4o-latest": 0.5475294593699321,
|
||||
"claude-1": 0.5953670260081093,
|
||||
"claude-2.0": 0.6270890431027705,
|
||||
"claude-3-5-sonnet-20240620": 0.6142432139552865,
|
||||
"claude-3-opus-20240229": 0.6198227621200374,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.5,
|
||||
"gemini-1.5-pro-exp-0801": 0.5458257442135583,
|
||||
"gemini-advanced-0514": 0.6025435895577902,
|
||||
"gemini-pro-dev-api": 0.6233440078256695,
|
||||
"glm-4-0520": 0.6327707226950309,
|
||||
"gpt-3.5-turbo-0314": 0.579682755828864,
|
||||
"gpt-4o-2024-05-13": 0.6036686971453942,
|
||||
"gpt-4o-2024-08-06": 0.6052143488362124,
|
||||
"gpt-4o-mini-2024-07-18": 0.6140012780307695,
|
||||
"llama-2-70b-chat": 0.6389839091289585,
|
||||
"llama-3.1-70b-instruct": 0.6028024856995498,
|
||||
"nemotron-4-340b-instruct": 0.6345306684098833,
|
||||
"yi-large-preview": 0.6386902518991052
|
||||
},
|
||||
"gemini-1.5-pro-exp-0801": {
|
||||
"athene-70b-0725": 0.5772709255434492,
|
||||
"bard-jan-24-gemini-pro": 0.5483123445198447,
|
||||
"chatgpt-4o-latest": 0.5017186888746599,
|
||||
"claude-1": 0.5504227244540455,
|
||||
"claude-2.0": 0.5832015454454681,
|
||||
"claude-3-5-sonnet-20240620": 0.569880853157496,
|
||||
"claude-3-opus-20240229": 0.5756587773809309,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.4541742557864416,
|
||||
"gemini-1.5-pro-exp-0801": 0.5,
|
||||
"gemini-advanced-0514": 0.557804367756876,
|
||||
"gemini-pro-dev-api": 0.5793114416789945,
|
||||
"glm-4-0520": 0.5891137641041534,
|
||||
"gpt-3.5-turbo-0314": 0.5343588600926158,
|
||||
"gpt-4o-2024-05-13": 0.5589634219132797,
|
||||
"gpt-4o-2024-08-06": 0.5605565036097535,
|
||||
"gpt-4o-mini-2024-07-18": 0.5696305884618997,
|
||||
"llama-2-70b-chat": 0.5955935211645167,
|
||||
"llama-3.1-70b-instruct": 0.5580710321113055,
|
||||
"nemotron-4-340b-instruct": 0.5909476824028691,
|
||||
"yi-large-preview": 0.5952869238841539
|
||||
},
|
||||
"gemini-advanced-0514": {
|
||||
"athene-70b-0725": 0.519820681777397,
|
||||
"bard-jan-24-gemini-pro": 0.49040074678712725,
|
||||
"chatgpt-4o-latest": 0.4438920243191068,
|
||||
"claude-1": 0.4925312817882607,
|
||||
"claude-2.0": 0.525895343262826,
|
||||
"claude-3-5-sonnet-20240620": 0.5122748179085177,
|
||||
"claude-3-opus-20240229": 0.5181723093004097,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3974564104422098,
|
||||
"gemini-1.5-pro-exp-0801": 0.44219563224312397,
|
||||
"gemini-advanced-0514": 0.5,
|
||||
"gemini-pro-dev-api": 0.5219088424590324,
|
||||
"glm-4-0520": 0.5319680879028288,
|
||||
"gpt-3.5-turbo-0314": 0.4763667410118573,
|
||||
"gpt-4o-2024-05-13": 0.5011750743841354,
|
||||
"gpt-4o-2024-08-06": 0.5027912177061772,
|
||||
"gpt-4o-mini-2024-07-18": 0.5120197361562043,
|
||||
"llama-2-70b-chat": 0.5386432816976552,
|
||||
"llama-3.1-70b-instruct": 0.5002702935966019,
|
||||
"nemotron-4-340b-instruct": 0.5338552462294891,
|
||||
"yi-large-preview": 0.5383269761251527
|
||||
},
|
||||
"gemini-pro-dev-api": {
|
||||
"athene-70b-0725": 0.4979082058870976,
|
||||
"bard-jan-24-gemini-pro": 0.4685183877340279,
|
||||
"chatgpt-4o-latest": 0.4223649169173924,
|
||||
"claude-1": 0.47064165506714317,
|
||||
"claude-2.0": 0.5039955681382927,
|
||||
"claude-3-5-sonnet-20240620": 0.4903556008902106,
|
||||
"claude-3-opus-20240229": 0.4962575067822826,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.37665599217433055,
|
||||
"gemini-1.5-pro-exp-0801": 0.4206885583210056,
|
||||
"gemini-advanced-0514": 0.4780911575409677,
|
||||
"gemini-pro-dev-api": 0.5,
|
||||
"glm-4-0520": 0.5100875059468505,
|
||||
"gpt-3.5-turbo-0314": 0.45455202627804153,
|
||||
"gpt-4o-2024-05-13": 0.47926409658161856,
|
||||
"gpt-4o-2024-08-06": 0.4808776977523424,
|
||||
"gpt-4o-mini-2024-07-18": 0.49010046598318574,
|
||||
"llama-2-70b-chat": 0.5167913032942285,
|
||||
"llama-3.1-70b-instruct": 0.4783609385670775,
|
||||
"nemotron-4-340b-instruct": 0.5119819532314825,
|
||||
"yi-large-preview": 0.516473464718453
|
||||
},
|
||||
"glm-4-0520": {
|
||||
"athene-70b-0725": 0.4878217278344273,
|
||||
"bard-jan-24-gemini-pro": 0.4584836193730801,
|
||||
"chatgpt-4o-latest": 0.41255135063749104,
|
||||
"claude-1": 0.4606008217775131,
|
||||
"claude-2.0": 0.49390707988312654,
|
||||
"claude-3-5-sonnet-20240620": 0.48027577066139915,
|
||||
"claude-3-opus-20240229": 0.48617208898399294,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3672292773049691,
|
||||
"gemini-1.5-pro-exp-0801": 0.4108862358958465,
|
||||
"gemini-advanced-0514": 0.4680319120971712,
|
||||
"gemini-pro-dev-api": 0.4899124940531495,
|
||||
"glm-4-0520": 0.5,
|
||||
"gpt-3.5-turbo-0314": 0.444566176363753,
|
||||
"gpt-4o-2024-05-13": 0.4692023588423809,
|
||||
"gpt-4o-2024-08-06": 0.47081271228911237,
|
||||
"gpt-4o-mini-2024-07-18": 0.4800209406003484,
|
||||
"llama-2-70b-chat": 0.5067083424471972,
|
||||
"llama-3.1-70b-instruct": 0.4683011100825741,
|
||||
"nemotron-4-340b-instruct": 0.5018953636400675,
|
||||
"yi-large-preview": 0.5063902063717687
|
||||
},
|
||||
"gpt-3.5-turbo-0314": {
|
||||
"athene-70b-0725": 0.5433726729881329,
|
||||
"bard-jan-24-gemini-pro": 0.5140467524523086,
|
||||
"chatgpt-4o-latest": 0.4673521170660229,
|
||||
"claude-1": 0.5161759616621979,
|
||||
"claude-2.0": 0.5494076540226519,
|
||||
"claude-3-5-sonnet-20240620": 0.5358664583262915,
|
||||
"claude-3-opus-20240229": 0.5417338743515567,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.4203172441711361,
|
||||
"gemini-1.5-pro-exp-0801": 0.4656411399073841,
|
||||
"gemini-advanced-0514": 0.5236332589881427,
|
||||
"gemini-pro-dev-api": 0.5454479737219585,
|
||||
"glm-4-0520": 0.5554338236362469,
|
||||
"gpt-3.5-turbo-0314": 0.5,
|
||||
"gpt-4o-2024-05-13": 0.5248055778856127,
|
||||
"gpt-4o-2024-08-06": 0.5264175061108259,
|
||||
"gpt-4o-mini-2024-07-18": 0.5356125299744643,
|
||||
"llama-2-70b-chat": 0.5620498683755348,
|
||||
"llama-3.1-70b-instruct": 0.5239029418245605,
|
||||
"nemotron-4-340b-instruct": 0.5573051037168208,
|
||||
"yi-large-preview": 0.5617365533684751
|
||||
},
|
||||
"gpt-4o-2024-05-13": {
|
||||
"athene-70b-0725": 0.5186473446377271,
|
||||
"bard-jan-24-gemini-pro": 0.48922615851167606,
|
||||
"chatgpt-4o-latest": 0.4427320528773593,
|
||||
"claude-1": 0.49135651083552223,
|
||||
"claude-2.0": 0.5247232780927109,
|
||||
"claude-3-5-sonnet-20240620": 0.511100383964325,
|
||||
"claude-3-opus-20240229": 0.5169986868635432,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3963313028546058,
|
||||
"gemini-1.5-pro-exp-0801": 0.44103657808672025,
|
||||
"gemini-advanced-0514": 0.4988249256158645,
|
||||
"gemini-pro-dev-api": 0.5207359034183815,
|
||||
"glm-4-0520": 0.5307976411576192,
|
||||
"gpt-3.5-turbo-0314": 0.4751944221143874,
|
||||
"gpt-4o-2024-05-13": 0.5,
|
||||
"gpt-4o-2024-08-06": 0.501616164525399,
|
||||
"gpt-4o-mini-2024-07-18": 0.5108452744903431,
|
||||
"llama-2-70b-chat": 0.5374750140847713,
|
||||
"llama-3.1-70b-instruct": 0.4990952180629769,
|
||||
"nemotron-4-340b-instruct": 0.5326853730599583,
|
||||
"yi-large-preview": 0.537158595794847
|
||||
},
|
||||
"gpt-4o-2024-08-06": {
|
||||
"athene-70b-0725": 0.517033233446606,
|
||||
"bard-jan-24-gemini-pro": 0.48761085688020456,
|
||||
"chatgpt-4o-latest": 0.44113768022450583,
|
||||
"claude-1": 0.4897409195588359,
|
||||
"claude-2.0": 0.5231108073107922,
|
||||
"claude-3-5-sonnet-20240620": 0.5094849000771304,
|
||||
"claude-3-opus-20240229": 0.5153842129200518,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.39478565116378755,
|
||||
"gemini-1.5-pro-exp-0801": 0.4394434963902465,
|
||||
"gemini-advanced-0514": 0.4972087822938228,
|
||||
"gemini-pro-dev-api": 0.5191223022476575,
|
||||
"glm-4-0520": 0.5291872877108876,
|
||||
"gpt-3.5-turbo-0314": 0.473582493889174,
|
||||
"gpt-4o-2024-05-13": 0.4983838354746009,
|
||||
"gpt-4o-2024-08-06": 0.5,
|
||||
"gpt-4o-mini-2024-07-18": 0.5092297570723646,
|
||||
"llama-2-70b-chat": 0.5358675389424615,
|
||||
"llama-3.1-70b-instruct": 0.49747906828277433,
|
||||
"nemotron-4-340b-instruct": 0.531075774838379,
|
||||
"yi-large-preview": 0.5355509712390535
|
||||
},
|
||||
"gpt-4o-mini-2024-07-18": {
|
||||
"athene-70b-0725": 0.507808386681544,
|
||||
"bard-jan-24-gemini-pro": 0.47839098366657384,
|
||||
"chatgpt-4o-latest": 0.43205557587413584,
|
||||
"claude-1": 0.48051854119187704,
|
||||
"claude-2.0": 0.5138929040607993,
|
||||
"claude-3-5-sonnet-20240620": 0.5002552323803292,
|
||||
"claude-3-opus-20240229": 0.5061579533816516,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.38599872196923046,
|
||||
"gemini-1.5-pro-exp-0801": 0.4303694115381003,
|
||||
"gemini-advanced-0514": 0.4879802638437957,
|
||||
"gemini-pro-dev-api": 0.5098995340168142,
|
||||
"glm-4-0520": 0.5199790593996516,
|
||||
"gpt-3.5-turbo-0314": 0.4643874700255357,
|
||||
"gpt-4o-2024-05-13": 0.4891547255096569,
|
||||
"gpt-4o-2024-08-06": 0.4907702429276354,
|
||||
"gpt-4o-mini-2024-07-18": 0.5,
|
||||
"llama-2-70b-chat": 0.5266731022503263,
|
||||
"llama-3.1-70b-instruct": 0.4882504047493449,
|
||||
"nemotron-4-340b-instruct": 0.5218711102154154,
|
||||
"yi-large-preview": 0.5263558063552674
|
||||
},
|
||||
"llama-2-70b-chat": {
|
||||
"athene-70b-0725": 0.4811195552160165,
|
||||
"bard-jan-24-gemini-pro": 0.4518289406610546,
|
||||
"chatgpt-4o-latest": 0.40606343428200253,
|
||||
"claude-1": 0.4539411733076995,
|
||||
"claude-2.0": 0.4872008300180327,
|
||||
"claude-3-5-sonnet-20240620": 0.4735814107159531,
|
||||
"claude-3-opus-20240229": 0.4794713636760069,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3610160908710415,
|
||||
"gemini-1.5-pro-exp-0801": 0.40440647883548325,
|
||||
"gemini-advanced-0514": 0.4613567183023448,
|
||||
"gemini-pro-dev-api": 0.4832086967057715,
|
||||
"glm-4-0520": 0.4932916575528028,
|
||||
"gpt-3.5-turbo-0314": 0.43795013162446517,
|
||||
"gpt-4o-2024-05-13": 0.4625249859152287,
|
||||
"gpt-4o-2024-08-06": 0.46413246105753847,
|
||||
"gpt-4o-mini-2024-07-18": 0.47332689774967374,
|
||||
"llama-2-70b-chat": 0.5,
|
||||
"llama-3.1-70b-instruct": 0.46162540860366846,
|
||||
"nemotron-4-340b-instruct": 0.49518677639716263,
|
||||
"yi-large-preview": 0.49968180936417794
|
||||
},
|
||||
"llama-3.1-70b-instruct": {
|
||||
"athene-70b-0725": 0.5195508071470352,
|
||||
"bard-jan-24-gemini-pro": 0.4901305556202252,
|
||||
"chatgpt-4o-latest": 0.4436251505621573,
|
||||
"claude-1": 0.4922610506835726,
|
||||
"claude-2.0": 0.5256257671206118,
|
||||
"claude-3-5-sonnet-20240620": 0.512004683628698,
|
||||
"claude-3-opus-20240229": 0.5179023674394149,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3971975143004503,
|
||||
"gemini-1.5-pro-exp-0801": 0.4419289678886945,
|
||||
"gemini-advanced-0514": 0.49972970640339814,
|
||||
"gemini-pro-dev-api": 0.5216390614329225,
|
||||
"glm-4-0520": 0.531698889917426,
|
||||
"gpt-3.5-turbo-0314": 0.47609705817543935,
|
||||
"gpt-4o-2024-05-13": 0.5009047819370231,
|
||||
"gpt-4o-2024-08-06": 0.5025209317172257,
|
||||
"gpt-4o-mini-2024-07-18": 0.5117495952506551,
|
||||
"llama-2-70b-chat": 0.5383745913963316,
|
||||
"llama-3.1-70b-instruct": 0.5,
|
||||
"nemotron-4-340b-instruct": 0.5335861820021834,
|
||||
"yi-large-preview": 0.5380582595922262
|
||||
},
|
||||
"nemotron-4-340b-instruct": {
|
||||
"athene-70b-0725": 0.4859276634793583,
|
||||
"bard-jan-24-gemini-pro": 0.45660191547771745,
|
||||
"chatgpt-4o-latest": 0.4107151817570633,
|
||||
"claude-1": 0.458717789260918,
|
||||
"claude-2.0": 0.4920120852303797,
|
||||
"claude-3-5-sonnet-20240620": 0.47838363949618123,
|
||||
"claude-3-opus-20240229": 0.4842783735313118,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3654693315901167,
|
||||
"gemini-1.5-pro-exp-0801": 0.4090523175971309,
|
||||
"gemini-advanced-0514": 0.4661447537705109,
|
||||
"gemini-pro-dev-api": 0.4880180467685175,
|
||||
"glm-4-0520": 0.49810463635993246,
|
||||
"gpt-3.5-turbo-0314": 0.4426948962831791,
|
||||
"gpt-4o-2024-05-13": 0.4673146269400416,
|
||||
"gpt-4o-2024-08-06": 0.46892422516162097,
|
||||
"gpt-4o-mini-2024-07-18": 0.47812888978458457,
|
||||
"llama-2-70b-chat": 0.5048132236028374,
|
||||
"llama-3.1-70b-instruct": 0.4664138179978165,
|
||||
"nemotron-4-340b-instruct": 0.5,
|
||||
"yi-large-preview": 0.5044950605041638
|
||||
},
|
||||
"yi-large-preview": {
|
||||
"athene-70b-0725": 0.48143729978415806,
|
||||
"bard-jan-24-gemini-pro": 0.4521441972422958,
|
||||
"chatgpt-4o-latest": 0.4063704306655477,
|
||||
"claude-1": 0.45425668238107064,
|
||||
"claude-2.0": 0.48751881733159297,
|
||||
"claude-3-5-sonnet-20240620": 0.4738987237070381,
|
||||
"claude-3-opus-20240229": 0.47978902623790215,
|
||||
"gemini-1.5-pro-api-0409-preview": 0.3613097481008948,
|
||||
"gemini-1.5-pro-exp-0801": 0.40471307611584606,
|
||||
"gemini-advanced-0514": 0.4616730238748474,
|
||||
"gemini-pro-dev-api": 0.48352653528154693,
|
||||
"glm-4-0520": 0.4936097936282313,
|
||||
"gpt-3.5-turbo-0314": 0.4382634466315249,
|
||||
"gpt-4o-2024-05-13": 0.462841404205153,
|
||||
"gpt-4o-2024-08-06": 0.46444902876094657,
|
||||
"gpt-4o-mini-2024-07-18": 0.4736441936447327,
|
||||
"llama-2-70b-chat": 0.5003181906358221,
|
||||
"llama-3.1-70b-instruct": 0.4619417404077737,
|
||||
"nemotron-4-340b-instruct": 0.4955049394958363,
|
||||
"yi-large-preview": 0.5
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 226 KiB |
@@ -0,0 +1,66 @@
|
||||
"""Fail-closed verifier for a saved Experiment 7-7 run."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(8 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("run_dir", type=Path)
|
||||
parser.add_argument("--input", type=Path, help="Optionally re-hash the 2 GB Arena input")
|
||||
args = parser.parse_args()
|
||||
|
||||
manifest_path = args.run_dir / "manifest.json"
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
failures: list[str] = []
|
||||
if manifest.get("experiment") != "7-7":
|
||||
failures.append("wrong experiment id")
|
||||
if manifest.get("status") != "passed" or manifest.get("official_complete") is not True:
|
||||
failures.append("run is not officially complete")
|
||||
false_gates = sorted(name for name, passed in manifest.get("gates", {}).items() if passed is not True)
|
||||
if false_gates:
|
||||
failures.append(f"false gates: {false_gates}")
|
||||
|
||||
for name, expected in manifest.get("artifacts", {}).items():
|
||||
path = args.run_dir / name
|
||||
if not path.is_file():
|
||||
failures.append(f"missing artifact: {name}")
|
||||
continue
|
||||
if path.stat().st_size != expected.get("bytes"):
|
||||
failures.append(f"size mismatch: {name}")
|
||||
if sha256_file(path) != expected.get("sha256"):
|
||||
failures.append(f"sha256 mismatch: {name}")
|
||||
|
||||
project = Path(__file__).resolve().parents[1]
|
||||
for name, expected_hash in manifest.get("sources", {}).items():
|
||||
path = project / name
|
||||
if not path.is_file() or sha256_file(path) != expected_hash:
|
||||
failures.append(f"source mismatch: {name}")
|
||||
|
||||
if args.input:
|
||||
expected = manifest["input"]
|
||||
if args.input.stat().st_size != expected["bytes"]:
|
||||
failures.append("input size mismatch")
|
||||
if sha256_file(args.input) != expected["sha256"]:
|
||||
failures.append("input sha256 mismatch")
|
||||
|
||||
result = {"valid": not failures, "failures": failures}
|
||||
print(json.dumps(result, indent=2))
|
||||
if failures:
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,283 @@
|
||||
"""
|
||||
Visualization tools for Elo leaderboard analysis
|
||||
"""
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
import seaborn as sns
|
||||
from typing import List, Tuple, Optional
|
||||
import plotly.graph_objects as go
|
||||
import plotly.express as px
|
||||
|
||||
|
||||
def plot_leaderboard(leaderboard_data: list, top_n: int = 20, save_path: Optional[str] = None):
|
||||
"""
|
||||
Plot static leaderboard bar chart.
|
||||
|
||||
Args:
|
||||
leaderboard_data: List of (model, rating, matches, wins) tuples
|
||||
top_n: Number of top models to display
|
||||
save_path: If provided, save figure to this path
|
||||
"""
|
||||
# Convert to DataFrame and get top N
|
||||
df = pd.DataFrame(leaderboard_data, columns=['model', 'rating', 'matches', 'wins'])
|
||||
df = df.head(top_n)
|
||||
|
||||
# Create figure
|
||||
fig, ax = plt.subplots(figsize=(12, 8))
|
||||
|
||||
# Create horizontal bar chart
|
||||
bars = ax.barh(range(len(df)), df['rating'], color=plt.cm.viridis(np.linspace(0, 1, len(df))))
|
||||
|
||||
# Customize
|
||||
ax.set_yticks(range(len(df)))
|
||||
ax.set_yticklabels(df['model'])
|
||||
ax.set_xlabel('Elo Rating', fontsize=12)
|
||||
ax.set_title(f'Model Leaderboard - Top {top_n} Models', fontsize=14, fontweight='bold')
|
||||
ax.invert_yaxis() # Highest rating at top
|
||||
|
||||
# Add value labels on bars
|
||||
for i, (rating, matches) in enumerate(zip(df['rating'], df['matches'])):
|
||||
ax.text(rating + 10, i, f'{rating:.0f} ({matches} matches)',
|
||||
va='center', fontsize=9)
|
||||
|
||||
ax.grid(axis='x', alpha=0.3)
|
||||
plt.tight_layout()
|
||||
|
||||
if save_path:
|
||||
plt.savefig(save_path, dpi=300, bbox_inches='tight')
|
||||
print(f"Saved leaderboard to {save_path}")
|
||||
|
||||
plt.show()
|
||||
|
||||
|
||||
def plot_win_rate_matrix(win_rate_df: pd.DataFrame,
|
||||
top_n: int = 15,
|
||||
save_path: Optional[str] = None):
|
||||
"""
|
||||
Plot heatmap of win rate matrix.
|
||||
|
||||
Args:
|
||||
win_rate_df: DataFrame with win rates (rows beat columns)
|
||||
top_n: Number of models to include
|
||||
save_path: If provided, save figure to this path
|
||||
"""
|
||||
# Get top N models by average win rate
|
||||
avg_win_rates = win_rate_df.mean(axis=1).sort_values(ascending=False)
|
||||
top_models = avg_win_rates.head(top_n).index.tolist()
|
||||
|
||||
# Subset matrix
|
||||
subset = win_rate_df.loc[top_models, top_models]
|
||||
|
||||
# Create figure
|
||||
fig, ax = plt.subplots(figsize=(14, 12))
|
||||
|
||||
# Plot heatmap
|
||||
sns.heatmap(subset, annot=True, fmt='.2f', cmap='RdYlGn', center=0.5,
|
||||
vmin=0, vmax=1, square=True, linewidths=0.5,
|
||||
cbar_kws={'label': 'Win Rate'}, ax=ax)
|
||||
|
||||
ax.set_title(f'Win Rate Matrix - Top {top_n} Models\n(Row vs Column)',
|
||||
fontsize=14, fontweight='bold')
|
||||
ax.set_xlabel('Opponent (Column)', fontsize=12)
|
||||
ax.set_ylabel('Model (Row)', fontsize=12)
|
||||
|
||||
plt.tight_layout()
|
||||
|
||||
if save_path:
|
||||
plt.savefig(save_path, dpi=300, bbox_inches='tight')
|
||||
print(f"Saved win rate matrix to {save_path}")
|
||||
|
||||
plt.show()
|
||||
|
||||
|
||||
def plot_rating_history(history_df: pd.DataFrame,
|
||||
models: Optional[List[str]] = None,
|
||||
top_n: int = 10,
|
||||
save_path: Optional[str] = None):
|
||||
"""
|
||||
Plot rating evolution over time for selected models.
|
||||
|
||||
Args:
|
||||
history_df: DataFrame with columns: date, model, rating
|
||||
models: List of specific models to plot (if None, plot top N)
|
||||
top_n: If models not specified, plot top N models by final rating
|
||||
save_path: If provided, save figure to this path
|
||||
"""
|
||||
if models is None:
|
||||
# Get top N models by final rating
|
||||
final_date = history_df['date'].max()
|
||||
final_ratings = history_df[history_df['date'] == final_date].nlargest(top_n, 'rating')
|
||||
models = final_ratings['model'].tolist()
|
||||
|
||||
# Filter data
|
||||
plot_data = history_df[history_df['model'].isin(models)].copy()
|
||||
|
||||
# Create figure
|
||||
fig, ax = plt.subplots(figsize=(14, 8))
|
||||
|
||||
# Plot each model
|
||||
for model in models:
|
||||
model_data = plot_data[plot_data['model'] == model].sort_values('date')
|
||||
ax.plot(model_data['date'], model_data['rating'], marker='o',
|
||||
label=model, linewidth=2, markersize=4)
|
||||
|
||||
ax.set_xlabel('Date', fontsize=12)
|
||||
ax.set_ylabel('Elo Rating', fontsize=12)
|
||||
ax.set_title('Model Rating Evolution Over Time', fontsize=14, fontweight='bold')
|
||||
ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left', fontsize=9)
|
||||
ax.grid(alpha=0.3)
|
||||
|
||||
plt.xticks(rotation=45)
|
||||
plt.tight_layout()
|
||||
|
||||
if save_path:
|
||||
plt.savefig(save_path, dpi=300, bbox_inches='tight')
|
||||
print(f"Saved rating history to {save_path}")
|
||||
|
||||
plt.show()
|
||||
|
||||
|
||||
def plot_rating_distribution(leaderboard_data: list, save_path: Optional[str] = None):
|
||||
"""
|
||||
Plot distribution of ratings across all models.
|
||||
|
||||
Args:
|
||||
leaderboard_data: List of (model, rating, matches, wins) tuples
|
||||
save_path: If provided, save figure to this path
|
||||
"""
|
||||
df = pd.DataFrame(leaderboard_data, columns=['model', 'rating', 'matches', 'wins'])
|
||||
|
||||
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
|
||||
|
||||
# Histogram
|
||||
ax1.hist(df['rating'], bins=30, color='steelblue', edgecolor='black', alpha=0.7)
|
||||
ax1.axvline(df['rating'].mean(), color='red', linestyle='--',
|
||||
linewidth=2, label=f'Mean: {df["rating"].mean():.1f}')
|
||||
ax1.axvline(df['rating'].median(), color='green', linestyle='--',
|
||||
linewidth=2, label=f'Median: {df["rating"].median():.1f}')
|
||||
ax1.set_xlabel('Elo Rating', fontsize=12)
|
||||
ax1.set_ylabel('Count', fontsize=12)
|
||||
ax1.set_title('Rating Distribution', fontsize=13, fontweight='bold')
|
||||
ax1.legend()
|
||||
ax1.grid(alpha=0.3)
|
||||
|
||||
# Box plot
|
||||
ax2.boxplot(df['rating'], vert=True)
|
||||
ax2.set_ylabel('Elo Rating', fontsize=12)
|
||||
ax2.set_title('Rating Box Plot', fontsize=13, fontweight='bold')
|
||||
ax2.grid(alpha=0.3)
|
||||
|
||||
plt.tight_layout()
|
||||
|
||||
if save_path:
|
||||
plt.savefig(save_path, dpi=300, bbox_inches='tight')
|
||||
print(f"Saved rating distribution to {save_path}")
|
||||
|
||||
plt.show()
|
||||
|
||||
|
||||
def create_interactive_leaderboard(history_df: pd.DataFrame, top_n: int = 15) -> go.Figure:
|
||||
"""
|
||||
Create interactive Plotly visualization of ranking evolution.
|
||||
|
||||
Args:
|
||||
history_df: DataFrame with columns: date, model, rating, rank
|
||||
top_n: Number of top models to include
|
||||
|
||||
Returns:
|
||||
Plotly Figure object
|
||||
"""
|
||||
# Get top N models by final rating
|
||||
final_date = history_df['date'].max()
|
||||
final_ratings = history_df[history_df['date'] == final_date].nlargest(top_n, 'rating')
|
||||
top_models = final_ratings['model'].tolist()
|
||||
|
||||
# Filter data
|
||||
plot_data = history_df[history_df['model'].isin(top_models)].copy()
|
||||
|
||||
# Create figure
|
||||
fig = go.Figure()
|
||||
|
||||
for model in top_models:
|
||||
model_data = plot_data[plot_data['model'] == model].sort_values('date')
|
||||
|
||||
fig.add_trace(go.Scatter(
|
||||
x=model_data['date'],
|
||||
y=model_data['rating'],
|
||||
mode='lines+markers',
|
||||
name=model,
|
||||
hovertemplate='<b>%{fullData.name}</b><br>' +
|
||||
'Date: %{x}<br>' +
|
||||
'Rating: %{y:.0f}<br>' +
|
||||
'<extra></extra>'
|
||||
))
|
||||
|
||||
fig.update_layout(
|
||||
title='Interactive Model Rating Evolution',
|
||||
xaxis_title='Date',
|
||||
yaxis_title='Elo Rating',
|
||||
hovermode='closest',
|
||||
height=600,
|
||||
legend=dict(
|
||||
yanchor="top",
|
||||
y=0.99,
|
||||
xanchor="left",
|
||||
x=0.01
|
||||
)
|
||||
)
|
||||
|
||||
return fig
|
||||
|
||||
|
||||
def create_rank_evolution_chart(history_df: pd.DataFrame, top_n: int = 15) -> go.Figure:
|
||||
"""
|
||||
Create interactive rank evolution chart (lower rank number is better).
|
||||
|
||||
Args:
|
||||
history_df: DataFrame with columns: date, model, rating, rank
|
||||
top_n: Number of models to track
|
||||
|
||||
Returns:
|
||||
Plotly Figure object
|
||||
"""
|
||||
# Get models that were ever in top N
|
||||
models_in_top = history_df[history_df['rank'] <= top_n]['model'].unique()
|
||||
|
||||
# Filter data
|
||||
plot_data = history_df[history_df['model'].isin(models_in_top)].copy()
|
||||
|
||||
# Create figure
|
||||
fig = go.Figure()
|
||||
|
||||
for model in models_in_top:
|
||||
model_data = plot_data[plot_data['model'] == model].sort_values('date')
|
||||
|
||||
fig.add_trace(go.Scatter(
|
||||
x=model_data['date'],
|
||||
y=model_data['rank'],
|
||||
mode='lines+markers',
|
||||
name=model,
|
||||
hovertemplate='<b>%{fullData.name}</b><br>' +
|
||||
'Date: %{x}<br>' +
|
||||
'Rank: #%{y}<br>' +
|
||||
'<extra></extra>'
|
||||
))
|
||||
|
||||
fig.update_layout(
|
||||
title=f'Model Rank Evolution (Top {top_n})',
|
||||
xaxis_title='Date',
|
||||
yaxis_title='Rank',
|
||||
yaxis=dict(autorange='reversed'), # Lower rank at top
|
||||
hovermode='closest',
|
||||
height=600,
|
||||
legend=dict(
|
||||
yanchor="top",
|
||||
y=0.99,
|
||||
xanchor="right",
|
||||
x=0.99
|
||||
)
|
||||
)
|
||||
|
||||
return fig
|
||||
|
||||
Reference in New Issue
Block a user