Training Setup

Data & Batching

• WMT 2014 English–German: 4.5M sentence pairs; English–French: 36M sentences.
• Byte-pair encoding with a shared source–target vocabulary of ~37K tokens.
• Batches of ~25,000 source and ~25,000 target tokens, grouped by approximate sequence length.

Optimization & Regularization

• Adam (β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹) with warmup: 4,000 linearly increasing steps, then inverse-square-root decay.
• Residual dropout 0.1 (base) / 0.3 (big); attention dropout on each sub-layer.
• Label smoothing ε_ls = 0.1 — hurts perplexity but improves accuracy and BLEU.