[CLAUDE] Docs: adopt Harness-16 MFE (memory-fidelity-EVAL) — 2-workflow + email AI_INFRA
All checks were successful
Deploy SOLUTION_ERP / build-deploy (push) Successful in 5m2s

Adopt AI_INFRA Harness-16 (3 broadcast 2026-06-29) qua 2-workflow mandate:
WF1 implement wf_4c63e1bd-99e + WF2 review wf_13e3d35a-023 (PASS 0-blocking).

MFE = coverage/retention eval (do nap-vs-nho-vs-dung bang SO DO). DISTINCT
voi H6.7 memoryDelta-routing-fidelity (vocab-fork WF1 reviewer bat duoc).

Built (em-main single-writer, 0 production code):
- scripts/mfe-eval.ps1 deterministic NO-API (ASCII #30, exit 0): LEAD coverage-FIT
  (token-RANGE, cap live-read) + age-band flag-not-cut + Goodhart-anchor strikes/RCA;
  SUB per-role coverage do-that (prose->N/A khong 0%); sub-workflow N/A. Smoke 3-tier PASS.
- memory-budget.json :mfe config + eval/mfe/ (seed sample-questions stable-id + README)
  + engine §H + artifact-row + C3-comment + wire session-start §2.1.6 / session-end §L.b(c)
  opt-in + agents/README S93. Judge layer = SCAFFOLD-only (honest nac).

adap-report harness-16-mfe + harness-15-v3 (covered/folded) + email outbox/ai_infra.
AS-10 dogfood: WF1 residual-write caught+reverted. State GIU NGUYEN (Mig 59 · 88 · 434 · gotcha 76).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
pqhuy1987
2026-06-29 18:35:14 +07:00
parent 8ee8f3e248
commit 958009858e
15 changed files with 602 additions and 1 deletions

46
eval/mfe/README.md Normal file
View File

@ -0,0 +1,46 @@
# MFE — Memory-Fidelity-EVAL (Harness-16)
> **Adopt S93 (2026-06-29)** from AI_INFRA broadcasts `2026-06-29-Governance-harness-16-*` (memory-fidelity-eval + mfe-update + cd-judge-sub-update). Canonical mechanism → [`docs/governance/harness-11-engine.md §H`](../../docs/governance/harness-11-engine.md).
## What MFE is (and is NOT)
MFE measures **provisioned ≠ remembered ≠ applied**. The session `%-print` (Harness-15) tells you *how much you stuffed into* hot-memory (by length). MFE asks the next question: does the agent actually **retain/use** what was stuffed in.
🔴 **DISTINCT from H6.7 "memoryDelta-routing-fidelity"** (the right delta landing in the right `agent-memory/<role>` under single-writer). Same word "fidelity", two senses — always say **memory-fidelity-EVAL / MFE** for this one. The pair is recorded in `governance-detectors.ps1` C3 alias-map so it is not flagged as drift.
## Two layers
| Layer | What | Cost | Status (honest) |
|---|---|---|---|
| **Deterministic ($0)** | `scripts/mfe-eval.ps1` — LEAD coverage-FIT + age-band + Goodhart-anchor; SUB per-role coverage; SUB-workflow N/A | $0, NO-API | ✅ built + wired (opt-in) |
| **Judge (Branch A)** | recall/apply — give the agent the sample-questions, score recall + applied | $0 scaffold (quota only when a real scorer is wired) | 🟡 **SCAFFOLD only** — seed exists, **no scorer wired** |
## How to run
```
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 # all tiers
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Tier lead
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Tier sub
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Judge # show judge scaffold (no scoring)
```
Opt-in at session ends: `/session-start … eval` (baseline) and `/session-end … eval` (retention) — see `session-start.md §2.1.6` / `session-end.md §L.b(c)`. Default (no `eval`) = unchanged behaviour.
## Operational decision (the point)
- **Coverage < 100% or set over-cap** = *lack-of-SPACE* **INCREASE budget** (owner decides the number).
- **Low recall despite the set fitting** = *rot/noise* **REORGANIZE** (value-priority, cut low-value) adding space does NOT fix rot.
## 🔴 Honest caveats (do not hide)
1. **Token sizing is a RANGE, not a number.** `char/4` is not real tokenization; Vietnamese-diacritic hot-memory is ~3.03.5 byte/token, so `byte/4` is an upper bound on headroom. The analyzer reports `[bytes/4 … bytes/3.0]` and uses the worst-case end for the FIT verdict. Real tokenizer count is inside the band.
2. **The judge layer measures nothing yet.** A same-session self-grade is meaningless (the agent just read the answers). Real numbers need (a) the sample-questions to **mature in age** and (b) an **independent cross-session / different-model** judge. Until both, judge output is plumbing-smoke.
3. **Age is a flag, never a cut** (mark `RC-…10-29-11`). An item leaves the must-remember set only on **status-change** (mark Disabled, guard retired, AS-row deleted), never by age.
4. **No self-grading.** Coverage% is anchored to the real recurring-error signal (error-ledger strikes + RCA count). A high score next to rising strikes = the score lies.
## Files
- `scripts/mfe-eval.ps1` deterministic analyzer (NO-API, ASCII-only, exit 0, READ-ONLY on `token_governor`).
- `.claude/agent-memory/memory-budget.json` `mfe` block single-source config (denominator sources, stop-list, toggles, caveats).
- `eval/mfe/sample-questions.json` immutable seed (stable-id anchored), append-only.
- `.claude/agent-memory/.mfe-state.json` last-run strikes (cross-run Goodhart compare); MFE-only, never touches the budget.