ai-agent-book 精选快照(<2MB 代码与文档,来自 github.com/bojieli/ai-agent-book)
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
# Live diagnosis experiment PRD
|
||||
|
||||
- R1 (P0): Every refund must call `verify_refund_eligibility` before
|
||||
`process_refund`; a refund without the check is a policy violation.
|
||||
- R2 (P1): `check_stock` must complete within 250 ms. On origin timeout the
|
||||
orchestrator must use the degraded cache route; it must not simply fail.
|
||||
- R3 (P1): Regression cases must cite the source trajectory ID and the exact
|
||||
observed turn where the violation is visible.
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
# Live diagnosis experiment architecture
|
||||
|
||||
The orchestrator calls a local HTTP order service. Refund flows MUST call
|
||||
`verify_refund_eligibility` before `process_refund`. Inventory origin calls
|
||||
have a 250 ms client deadline; on timeout the orchestrator MUST call the same
|
||||
`check_stock` operation through the degraded cache route and finish normally.
|
||||
Every trajectory turn records its measured HTTP latency and raw response.
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"problems": [
|
||||
{
|
||||
"title": "Refund flow missing verify_refund_eligibility check",
|
||||
"priority": "P0",
|
||||
"module": "http_order_system",
|
||||
"description": "Refund flow did not call `verify_refund_eligibility` before `process_refund`, violating the policy requirement that every refund must include this check.",
|
||||
"suggestion": "Add a call to the `verify_refund_eligibility` tool prior to the `process_refund` step in the refund workflow.",
|
||||
"prd_ref": "R1",
|
||||
"trajectory_ids": [
|
||||
"HTTP-RF-001"
|
||||
],
|
||||
"focus_turns": [
|
||||
2
|
||||
],
|
||||
"suggested_assignee": "order-service-team"
|
||||
},
|
||||
{
|
||||
"title": "check_stock origin timeout not handled via degraded cache route",
|
||||
"priority": "P1",
|
||||
"module": "http_order_system",
|
||||
"description": "check_stock call timed out (latency 350.414 ms exceeds 250 ms client deadline) but orchestrator failed to use the degraded cache route, resulting in flow failure.",
|
||||
"suggestion": "Implement logic to invoke `check_stock` through the degraded cache route when the origin call times out, ensuring the flow completes normally.",
|
||||
"prd_ref": "R2",
|
||||
"trajectory_ids": [
|
||||
"HTTP-INV-001"
|
||||
],
|
||||
"focus_turns": [
|
||||
2
|
||||
],
|
||||
"suggested_assignee": "inventory-service-team"
|
||||
}
|
||||
]
|
||||
}
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"server": "official github/github-mcp-server",
|
||||
"transport": "stdio",
|
||||
"tool": "issue_write(method=create)",
|
||||
"request": {
|
||||
"method": "create",
|
||||
"owner": "bojieli",
|
||||
"repo": "ai-agent-book",
|
||||
"title": "[Experiment 5-8][auto-diagnosis] Live HTTP regression findings (exp5-8-live-http-mcp-20260730-053403)",
|
||||
"body": "This issue was created automatically by the Chapter 5 Experiment 5-8 acceptance campaign.\n\n## Evidence-backed diagnosis\n\n- **P0 Refund flow missing verify_refund_eligibility check** (R1): Refund flow did not call `verify_refund_eligibility` before `process_refund`, violating the policy requirement that every refund must include this check. Trajectories: HTTP-RF-001; turns: [2].\n- **P1 check_stock origin timeout not handled via degraded cache route** (R2): check_stock call timed out (latency 350.414 ms exceeds 250 ms client deadline) but orchestrator failed to use the degraded cache route, resulting in flow failure. Trajectories: HTTP-INV-001; turns: [2].\n\n## Generated regression tests\n\n- `TC-R1-HTTP-RF-001-T2` cites `HTTP-RF-001` turn 2: `{\"type\": \"step_present\", \"params\": {\"tool\": \"verify_refund_eligibility\"}}`\n- `TC-R2-LATENCY-HTTP-INV-001-T2` cites `HTTP-INV-001` turn 2: `{\"type\": \"latency_under\", \"params\": {\"tool\": \"check_stock\", \"threshold_ms\": 250}}`\n- `TC-R2-STATUS-HTTP-INV-001-T2` cites `HTTP-INV-001` turn 2: `{\"type\": \"final_status_is\", \"params\": {\"value\": \"success\"}}`\n\nThe campaign replayed each test against the live buggy and fixed HTTP orchestrators: all tests failed on buggy behavior and passed on fixed behavior."
|
||||
},
|
||||
"response": {
|
||||
"is_error": false,
|
||||
"content": "{\"id\":\"5013792084\",\"url\":\"https://github.com/bojieli/ai-agent-book/issues/502\"}"
|
||||
},
|
||||
"issue_url": "https://github.com/bojieli/ai-agent-book/issues/502",
|
||||
"credential_free": true
|
||||
}
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
"""Small real HTTP system used by the Experiment 5-8 campaign.
|
||||
|
||||
The service intentionally exposes ordinary order, refund, and inventory
|
||||
endpoints. The buggy/fixed distinction lives in the orchestrator under test:
|
||||
the buggy orchestrator skips a required endpoint and mishandles an inventory
|
||||
timeout, while the fixed orchestrator makes the required calls and degrades.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
server_version = "Experiment58HTTP/1.0"
|
||||
|
||||
def log_message(self, fmt: str, *args: object) -> None:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"client": self.client_address[0],
|
||||
"message": fmt % args,
|
||||
"time": time.time(),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
def _json(self, status: int, value: object) -> None:
|
||||
raw = json.dumps(value, ensure_ascii=False).encode("utf-8")
|
||||
self.send_response(status)
|
||||
self.send_header("content-type", "application/json; charset=utf-8")
|
||||
self.send_header("content-length", str(len(raw)))
|
||||
self.end_headers()
|
||||
try:
|
||||
self.wfile.write(raw)
|
||||
except BrokenPipeError:
|
||||
# A timed-out client is an expected part of the observed buggy run.
|
||||
pass
|
||||
|
||||
def _body(self) -> dict[str, object]:
|
||||
length = int(self.headers.get("content-length") or 0)
|
||||
if not length:
|
||||
return {}
|
||||
value = json.loads(self.rfile.read(length))
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
|
||||
parsed = urlparse(self.path)
|
||||
if parsed.path == "/health":
|
||||
self._json(200, {"ok": True})
|
||||
return
|
||||
if parsed.path.startswith("/orders/"):
|
||||
order_id = parsed.path.rsplit("/", 1)[-1]
|
||||
self._json(200, {"order_id": order_id, "status": "paid", "sku": "SKU-42"})
|
||||
return
|
||||
if parsed.path.startswith("/inventory/"):
|
||||
sku = parsed.path.rsplit("/", 1)[-1]
|
||||
degraded = parse_qs(parsed.query).get("degraded", ["0"])[0] == "1"
|
||||
if degraded:
|
||||
self._json(200, {"sku": sku, "stock": 12, "source": "cache", "degraded": True})
|
||||
else:
|
||||
# Longer than the campaign's real client deadline.
|
||||
time.sleep(0.8)
|
||||
self._json(200, {"sku": sku, "stock": 12, "source": "origin", "degraded": False})
|
||||
return
|
||||
self._json(404, {"error": "not_found"})
|
||||
|
||||
def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
|
||||
body = self._body()
|
||||
if self.path == "/refund/eligibility":
|
||||
self._json(200, {"order_id": body.get("order_id"), "eligible": True})
|
||||
return
|
||||
if self.path == "/refund/process":
|
||||
self._json(200, {"order_id": body.get("order_id"), "refund_id": "RF-LIVE-1"})
|
||||
return
|
||||
if self.path == "/notifications":
|
||||
self._json(200, {"sent": True, "status": body.get("status")})
|
||||
return
|
||||
self._json(404, {"error": "not_found"})
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--port", type=int, required=True)
|
||||
args = parser.parse_args()
|
||||
server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
|
||||
print(json.dumps({"listening": args.port}), flush=True)
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+529
@@ -0,0 +1,529 @@
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"test": {
|
||||
"test_id": "TC-R1-HTTP-RF-001-T2",
|
||||
"trajectory_id": "HTTP-RF-001",
|
||||
"focus_turn": 2,
|
||||
"description": "Refund flow must include verify_refund_eligibility before process_refund (R1)",
|
||||
"assertion": {
|
||||
"type": "step_present",
|
||||
"params": {
|
||||
"tool": "verify_refund_eligibility"
|
||||
}
|
||||
}
|
||||
},
|
||||
"buggy": {
|
||||
"passed": false,
|
||||
"detail": "verify_refund_eligibility calls=0",
|
||||
"trajectory": {
|
||||
"trajectory_id": "HTTP-RF-001::buggy",
|
||||
"source_trajectory_id": "HTTP-RF-001",
|
||||
"implementation": "buggy",
|
||||
"task_input": {
|
||||
"intent": "refund",
|
||||
"order_id": "ORD-58-A"
|
||||
},
|
||||
"final_status": "success",
|
||||
"turns": [
|
||||
{
|
||||
"index": 0,
|
||||
"role": "user",
|
||||
"content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "query_order",
|
||||
"method": "GET",
|
||||
"path": "/orders/ORD-58-A",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 1.833,
|
||||
"response": {
|
||||
"order_id": "ORD-58-A",
|
||||
"status": "paid",
|
||||
"sku": "SKU-42"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "process_refund",
|
||||
"method": "POST",
|
||||
"path": "/refund/process",
|
||||
"request": {
|
||||
"order_id": "ORD-58-A"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.833,
|
||||
"response": {
|
||||
"order_id": "ORD-58-A",
|
||||
"refund_id": "RF-LIVE-1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "notify_user",
|
||||
"method": "POST",
|
||||
"path": "/notifications",
|
||||
"request": {
|
||||
"status": "success"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.678,
|
||||
"response": {
|
||||
"sent": true,
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"fixed": {
|
||||
"passed": true,
|
||||
"detail": "verify_refund_eligibility calls=1",
|
||||
"trajectory": {
|
||||
"trajectory_id": "HTTP-RF-001::fixed",
|
||||
"source_trajectory_id": "HTTP-RF-001",
|
||||
"implementation": "fixed",
|
||||
"task_input": {
|
||||
"intent": "refund",
|
||||
"order_id": "ORD-58-A"
|
||||
},
|
||||
"final_status": "success",
|
||||
"turns": [
|
||||
{
|
||||
"index": 0,
|
||||
"role": "user",
|
||||
"content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "query_order",
|
||||
"method": "GET",
|
||||
"path": "/orders/ORD-58-A",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.498,
|
||||
"response": {
|
||||
"order_id": "ORD-58-A",
|
||||
"status": "paid",
|
||||
"sku": "SKU-42"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "verify_refund_eligibility",
|
||||
"method": "POST",
|
||||
"path": "/refund/eligibility",
|
||||
"request": {
|
||||
"order_id": "ORD-58-A"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.379,
|
||||
"response": {
|
||||
"order_id": "ORD-58-A",
|
||||
"eligible": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "process_refund",
|
||||
"method": "POST",
|
||||
"path": "/refund/process",
|
||||
"request": {
|
||||
"order_id": "ORD-58-A"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.37,
|
||||
"response": {
|
||||
"order_id": "ORD-58-A",
|
||||
"refund_id": "RF-LIVE-1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "notify_user",
|
||||
"method": "POST",
|
||||
"path": "/notifications",
|
||||
"request": {
|
||||
"status": "success"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.315,
|
||||
"response": {
|
||||
"sent": true,
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"test": {
|
||||
"test_id": "TC-R2-LATENCY-HTTP-INV-001-T2",
|
||||
"trajectory_id": "HTTP-INV-001",
|
||||
"focus_turn": 2,
|
||||
"description": "check_stock must complete within 250 ms (R2)",
|
||||
"assertion": {
|
||||
"type": "latency_under",
|
||||
"params": {
|
||||
"tool": "check_stock",
|
||||
"threshold_ms": 250
|
||||
}
|
||||
}
|
||||
},
|
||||
"buggy": {
|
||||
"passed": false,
|
||||
"detail": "check_stock max_latency_ms=350.383, threshold_ms=250.000",
|
||||
"trajectory": {
|
||||
"trajectory_id": "HTTP-INV-001::buggy",
|
||||
"source_trajectory_id": "HTTP-INV-001",
|
||||
"implementation": "buggy",
|
||||
"task_input": {
|
||||
"intent": "order_status",
|
||||
"order_id": "ORD-58-B",
|
||||
"sku": "SKU-42"
|
||||
},
|
||||
"final_status": "failed",
|
||||
"turns": [
|
||||
{
|
||||
"index": 0,
|
||||
"role": "user",
|
||||
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "query_order",
|
||||
"method": "GET",
|
||||
"path": "/orders/ORD-58-B",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.308,
|
||||
"response": {
|
||||
"order_id": "ORD-58-B",
|
||||
"status": "paid",
|
||||
"sku": "SKU-42"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "check_stock",
|
||||
"method": "GET",
|
||||
"path": "/inventory/SKU-42",
|
||||
"request": null,
|
||||
"http_status": null,
|
||||
"status": "error",
|
||||
"latency_ms": 350.383,
|
||||
"error": "TimeoutError: timed out"
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "notify_user",
|
||||
"method": "POST",
|
||||
"path": "/notifications",
|
||||
"request": {
|
||||
"status": "failed"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 1.481,
|
||||
"response": {
|
||||
"sent": true,
|
||||
"status": "failed"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"fixed": {
|
||||
"passed": true,
|
||||
"detail": "check_stock max_latency_ms=101.234, threshold_ms=250.000",
|
||||
"trajectory": {
|
||||
"trajectory_id": "HTTP-INV-001::fixed",
|
||||
"source_trajectory_id": "HTTP-INV-001",
|
||||
"implementation": "fixed",
|
||||
"task_input": {
|
||||
"intent": "order_status",
|
||||
"order_id": "ORD-58-B",
|
||||
"sku": "SKU-42"
|
||||
},
|
||||
"final_status": "success",
|
||||
"turns": [
|
||||
{
|
||||
"index": 0,
|
||||
"role": "user",
|
||||
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "query_order",
|
||||
"method": "GET",
|
||||
"path": "/orders/ORD-58-B",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.843,
|
||||
"response": {
|
||||
"order_id": "ORD-58-B",
|
||||
"status": "paid",
|
||||
"sku": "SKU-42"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "check_stock",
|
||||
"method": "GET",
|
||||
"path": "/inventory/SKU-42",
|
||||
"request": null,
|
||||
"http_status": null,
|
||||
"status": "error",
|
||||
"latency_ms": 101.234,
|
||||
"error": "TimeoutError: timed out"
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "check_stock",
|
||||
"method": "GET",
|
||||
"path": "/inventory/SKU-42?degraded=1",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 1.289,
|
||||
"response": {
|
||||
"sku": "SKU-42",
|
||||
"stock": 12,
|
||||
"source": "cache",
|
||||
"degraded": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "notify_user",
|
||||
"method": "POST",
|
||||
"path": "/notifications",
|
||||
"request": {
|
||||
"status": "success"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.78,
|
||||
"response": {
|
||||
"sent": true,
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"test": {
|
||||
"test_id": "TC-R2-STATUS-HTTP-INV-001-T2",
|
||||
"trajectory_id": "HTTP-INV-001",
|
||||
"focus_turn": 2,
|
||||
"description": "check_stock origin timeout must use degraded cache route, resulting in successful flow (R2)",
|
||||
"assertion": {
|
||||
"type": "final_status_is",
|
||||
"params": {
|
||||
"value": "success"
|
||||
}
|
||||
}
|
||||
},
|
||||
"buggy": {
|
||||
"passed": false,
|
||||
"detail": "final_status=failed, expected=success",
|
||||
"trajectory": {
|
||||
"trajectory_id": "HTTP-INV-001::buggy",
|
||||
"source_trajectory_id": "HTTP-INV-001",
|
||||
"implementation": "buggy",
|
||||
"task_input": {
|
||||
"intent": "order_status",
|
||||
"order_id": "ORD-58-B",
|
||||
"sku": "SKU-42"
|
||||
},
|
||||
"final_status": "failed",
|
||||
"turns": [
|
||||
{
|
||||
"index": 0,
|
||||
"role": "user",
|
||||
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "query_order",
|
||||
"method": "GET",
|
||||
"path": "/orders/ORD-58-B",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.608,
|
||||
"response": {
|
||||
"order_id": "ORD-58-B",
|
||||
"status": "paid",
|
||||
"sku": "SKU-42"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "check_stock",
|
||||
"method": "GET",
|
||||
"path": "/inventory/SKU-42",
|
||||
"request": null,
|
||||
"http_status": null,
|
||||
"status": "error",
|
||||
"latency_ms": 355.612,
|
||||
"error": "TimeoutError: timed out"
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "notify_user",
|
||||
"method": "POST",
|
||||
"path": "/notifications",
|
||||
"request": {
|
||||
"status": "failed"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 1.19,
|
||||
"response": {
|
||||
"sent": true,
|
||||
"status": "failed"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"fixed": {
|
||||
"passed": true,
|
||||
"detail": "final_status=success, expected=success",
|
||||
"trajectory": {
|
||||
"trajectory_id": "HTTP-INV-001::fixed",
|
||||
"source_trajectory_id": "HTTP-INV-001",
|
||||
"implementation": "fixed",
|
||||
"task_input": {
|
||||
"intent": "order_status",
|
||||
"order_id": "ORD-58-B",
|
||||
"sku": "SKU-42"
|
||||
},
|
||||
"final_status": "success",
|
||||
"turns": [
|
||||
{
|
||||
"index": 0,
|
||||
"role": "user",
|
||||
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
|
||||
},
|
||||
{
|
||||
"index": 1,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "query_order",
|
||||
"method": "GET",
|
||||
"path": "/orders/ORD-58-B",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.895,
|
||||
"response": {
|
||||
"order_id": "ORD-58-B",
|
||||
"status": "paid",
|
||||
"sku": "SKU-42"
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "check_stock",
|
||||
"method": "GET",
|
||||
"path": "/inventory/SKU-42",
|
||||
"request": null,
|
||||
"http_status": null,
|
||||
"status": "error",
|
||||
"latency_ms": 102.492,
|
||||
"error": "TimeoutError: timed out"
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "check_stock",
|
||||
"method": "GET",
|
||||
"path": "/inventory/SKU-42?degraded=1",
|
||||
"request": null,
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 1.493,
|
||||
"response": {
|
||||
"sku": "SKU-42",
|
||||
"stock": 12,
|
||||
"source": "cache",
|
||||
"degraded": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"index": 4,
|
||||
"role": "tool",
|
||||
"module": "http_order_system",
|
||||
"tool": "notify_user",
|
||||
"method": "POST",
|
||||
"path": "/notifications",
|
||||
"request": {
|
||||
"status": "success"
|
||||
},
|
||||
"http_status": 200,
|
||||
"status": "success",
|
||||
"latency_ms": 0.843,
|
||||
"response": {
|
||||
"sent": true,
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-8",
|
||||
"run_id": "exp5-8-live-http-mcp-20260730-053403",
|
||||
"generated_at_utc": "2026-07-29T21:37:08Z",
|
||||
"provider": "ark",
|
||||
"model": "doubao-seed-1-6-250615",
|
||||
"service": {
|
||||
"kind": "real local HTTP subprocess",
|
||||
"base_url": "http://127.0.0.1:56938",
|
||||
"pid": 96265
|
||||
},
|
||||
"source_trajectory_ids": [
|
||||
"HTTP-INV-001",
|
||||
"HTTP-RF-001"
|
||||
],
|
||||
"model_call_count": 2,
|
||||
"github_issue_url": "https://github.com/bojieli/ai-agent-book/issues/502",
|
||||
"gates": {
|
||||
"real_local_http_trajectories": true,
|
||||
"measured_latency_and_raw_http": true,
|
||||
"live_model_diagnosis": true,
|
||||
"diagnosis_references_trajectories_and_turns": true,
|
||||
"live_model_generated_executable_tests": true,
|
||||
"buggy_failures_reproduced": true,
|
||||
"fixed_system_passes": true,
|
||||
"official_github_mcp_issue_created": true,
|
||||
"raw_provider_receipts_complete": true
|
||||
},
|
||||
"artifacts": {
|
||||
"PRD.md": {
|
||||
"sha256": "d21f65b3f5243f79b6ab46525945b8e882a6646d75177062cb4d976192369353",
|
||||
"bytes": 448
|
||||
},
|
||||
"architecture.md": {
|
||||
"sha256": "e76c20c6ac826325db0c7b35204b0ce7235973dc33e3b62220aae607a60f49e4",
|
||||
"bytes": 422
|
||||
},
|
||||
"diagnosis.json": {
|
||||
"sha256": "4232addd060d6ace66814d53625d6b08b8511e77c34253c3d9e1529199d4b570",
|
||||
"bytes": 1334
|
||||
},
|
||||
"github_mcp_receipt.json": {
|
||||
"sha256": "30c42f32006eaf0fda5350747ce2bbf44d61af79a6020a62ab12bfb28836976e",
|
||||
"bytes": 1882
|
||||
},
|
||||
"http_service.py": {
|
||||
"sha256": "bbe0170e342a34fb70dc2477fede68cd393e1fd9a6d98140974a61677413c997",
|
||||
"bytes": 3701
|
||||
},
|
||||
"live_replays.json": {
|
||||
"sha256": "ef8401bed39405e0398d47a2b3553d50c50d1199b359873afa180b60153b29a8",
|
||||
"bytes": 16057
|
||||
},
|
||||
"production_trajectories.jsonl": {
|
||||
"sha256": "9d45f81e64ff8b6ccaa109dbdc838bb016832bcae14633fc86a9a917fccb0e6e",
|
||||
"bytes": 2300
|
||||
},
|
||||
"provider_receipts.checkpoint.json": {
|
||||
"sha256": "20e466f383f807d9ad1d6e0a1a699733f7e7e4da2d459a3add1a976578fb2bd0",
|
||||
"bytes": 12171
|
||||
},
|
||||
"provider_receipts.json": {
|
||||
"sha256": "20e466f383f807d9ad1d6e0a1a699733f7e7e4da2d459a3add1a976578fb2bd0",
|
||||
"bytes": 12171
|
||||
},
|
||||
"regression_tests.json": {
|
||||
"sha256": "642d9037754d16665be684f69eb6decd74e8db25181b214bce806e34ceeb7c3a",
|
||||
"bytes": 1091
|
||||
},
|
||||
"service.stderr.log": {
|
||||
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
"bytes": 0
|
||||
},
|
||||
"service.stdout.jsonl": {
|
||||
"sha256": "776fe22d507f08f0b56e017599777a60c0165a70e57556e34b3540a27b25fe5a",
|
||||
"bytes": 2987
|
||||
}
|
||||
},
|
||||
"official_complete": true
|
||||
}
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
{"trajectory_id": "HTTP-RF-001::buggy", "source_trajectory_id": "HTTP-RF-001", "implementation": "buggy", "task_input": {"intent": "refund", "order_id": "ORD-58-A"}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"}, {"index": 1, "role": "tool", "module": "http_order_system", "tool": "query_order", "method": "GET", "path": "/orders/ORD-58-A", "request": null, "http_status": 200, "status": "success", "latency_ms": 0.463, "response": {"order_id": "ORD-58-A", "status": "paid", "sku": "SKU-42"}}, {"index": 2, "role": "tool", "module": "http_order_system", "tool": "process_refund", "method": "POST", "path": "/refund/process", "request": {"order_id": "ORD-58-A"}, "http_status": 200, "status": "success", "latency_ms": 0.393, "response": {"order_id": "ORD-58-A", "refund_id": "RF-LIVE-1"}}, {"index": 3, "role": "tool", "module": "http_order_system", "tool": "notify_user", "method": "POST", "path": "/notifications", "request": {"status": "success"}, "http_status": 200, "status": "success", "latency_ms": 0.444, "response": {"sent": true, "status": "success"}}]}
|
||||
{"trajectory_id": "HTTP-INV-001::buggy", "source_trajectory_id": "HTTP-INV-001", "implementation": "buggy", "task_input": {"intent": "order_status", "order_id": "ORD-58-B", "sku": "SKU-42"}, "final_status": "failed", "turns": [{"index": 0, "role": "user", "content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"}, {"index": 1, "role": "tool", "module": "http_order_system", "tool": "query_order", "method": "GET", "path": "/orders/ORD-58-B", "request": null, "http_status": 200, "status": "success", "latency_ms": 0.391, "response": {"order_id": "ORD-58-B", "status": "paid", "sku": "SKU-42"}}, {"index": 2, "role": "tool", "module": "http_order_system", "tool": "check_stock", "method": "GET", "path": "/inventory/SKU-42", "request": null, "http_status": null, "status": "error", "latency_ms": 350.414, "error": "TimeoutError: timed out"}, {"index": 3, "role": "tool", "module": "http_order_system", "tool": "notify_user", "method": "POST", "path": "/notifications", "request": {"status": "failed"}, "http_status": 200, "status": "success", "latency_ms": 2.443, "response": {"sent": true, "status": "failed"}}]}
|
||||
+72
File diff suppressed because one or more lines are too long
+72
File diff suppressed because one or more lines are too long
+41
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"test_cases": [
|
||||
{
|
||||
"test_id": "TC-R1-HTTP-RF-001-T2",
|
||||
"trajectory_id": "HTTP-RF-001",
|
||||
"focus_turn": 2,
|
||||
"description": "Refund flow must include verify_refund_eligibility before process_refund (R1)",
|
||||
"assertion": {
|
||||
"type": "step_present",
|
||||
"params": {
|
||||
"tool": "verify_refund_eligibility"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"test_id": "TC-R2-LATENCY-HTTP-INV-001-T2",
|
||||
"trajectory_id": "HTTP-INV-001",
|
||||
"focus_turn": 2,
|
||||
"description": "check_stock must complete within 250 ms (R2)",
|
||||
"assertion": {
|
||||
"type": "latency_under",
|
||||
"params": {
|
||||
"tool": "check_stock",
|
||||
"threshold_ms": 250
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"test_id": "TC-R2-STATUS-HTTP-INV-001-T2",
|
||||
"trajectory_id": "HTTP-INV-001",
|
||||
"focus_turn": 2,
|
||||
"description": "check_stock origin timeout must use degraded cache route, resulting in successful flow (R2)",
|
||||
"assertion": {
|
||||
"type": "final_status_is",
|
||||
"params": {
|
||||
"value": "success"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
{"listening": 56938}
|
||||
{"client": "127.0.0.1", "message": "\"GET /health HTTP/1.1\" 200 -", "time": 1785360843.868107}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785360843.8689358}
|
||||
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785360843.869437}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785360843.8698602}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785360843.8703701}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785360844.223199}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785360844.6740758}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785361025.351637}
|
||||
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785361025.353014}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.3537788}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785361025.3544052}
|
||||
{"client": "127.0.0.1", "message": "\"POST /refund/eligibility HTTP/1.1\" 200 -", "time": 1785361025.354883}
|
||||
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785361025.3552582}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.355627}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361025.355964}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.707724}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361025.708798}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42?degraded=1 HTTP/1.1\" 200 -", "time": 1785361025.811397}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.812402}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361025.813149}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.157606}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361026.169961}
|
||||
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361026.1710122}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42?degraded=1 HTTP/1.1\" 200 -", "time": 1785361026.275081}
|
||||
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361026.276181}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.5155609}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.6163042}
|
||||
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.972992}
|
||||
Reference in New Issue
Block a user