ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter10/book-translation/sample_book/chapter3.md

@@ -0,0 +1,25 @@
+# Chapter 3: Optimizing Inference Latency
+
+Once a model works, the next battle is speed. The goal is to lower **latency**
+while raising **throughput**, the number of requests the system finishes per
+second. These two often trade off against each other.
+
+The most important trick is the **KV cache**. During inference the model would
+otherwise recompute attention over every previous token at each step. By caching
+the key and value vectors of past tokens, the model only processes the newest
+token, which cuts latency dramatically for long prompts.
+
+```python
+def decode_step(new_token, kv_cache):
+    q, k, v = project(new_token)         # only the new token
+    kv_cache.append(k, v)                # reuse past keys and values
+    return attention(q, kv_cache.keys, kv_cache.values)
+```
+
+A second trick is **batching**: grouping several prompts together so the
+hardware stays busy. Larger batches raise throughput but can hurt the latency of
+any single request, so serving systems tune the batch size carefully.
+
+The lesson is that inference performance is a balance. Every token we avoid
+recomputing, and every prompt we batch well, moves the system toward lower
+latency and higher throughput at the same time.