[CLAUDE] Docs: adopt Harness-16 MFE (memory-fidelity-EVAL) — 2-workflow + email AI_INFRA
All checks were successful
Deploy SOLUTION_ERP / build-deploy (push) Successful in 5m2s
All checks were successful
Deploy SOLUTION_ERP / build-deploy (push) Successful in 5m2s
Adopt AI_INFRA Harness-16 (3 broadcast 2026-06-29) qua 2-workflow mandate: WF1 implement wf_4c63e1bd-99e + WF2 review wf_13e3d35a-023 (PASS 0-blocking). MFE = coverage/retention eval (do nap-vs-nho-vs-dung bang SO DO). DISTINCT voi H6.7 memoryDelta-routing-fidelity (vocab-fork WF1 reviewer bat duoc). Built (em-main single-writer, 0 production code): - scripts/mfe-eval.ps1 deterministic NO-API (ASCII #30, exit 0): LEAD coverage-FIT (token-RANGE, cap live-read) + age-band flag-not-cut + Goodhart-anchor strikes/RCA; SUB per-role coverage do-that (prose->N/A khong 0%); sub-workflow N/A. Smoke 3-tier PASS. - memory-budget.json :mfe config + eval/mfe/ (seed sample-questions stable-id + README) + engine §H + artifact-row + C3-comment + wire session-start §2.1.6 / session-end §L.b(c) opt-in + agents/README S93. Judge layer = SCAFFOLD-only (honest nac). adap-report harness-16-mfe + harness-15-v3 (covered/folded) + email outbox/ai_infra. AS-10 dogfood: WF1 residual-write caught+reverted. State GIU NGUYEN (Mig 59 · 88 · 434 · gotcha 76). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
46
eval/mfe/README.md
Normal file
46
eval/mfe/README.md
Normal file
@ -0,0 +1,46 @@
|
||||
# MFE — Memory-Fidelity-EVAL (Harness-16)
|
||||
|
||||
> **Adopt S93 (2026-06-29)** from AI_INFRA broadcasts `2026-06-29-Governance-harness-16-*` (memory-fidelity-eval + mfe-update + cd-judge-sub-update). Canonical mechanism → [`docs/governance/harness-11-engine.md §H`](../../docs/governance/harness-11-engine.md).
|
||||
|
||||
## What MFE is (and is NOT)
|
||||
|
||||
MFE measures **provisioned ≠ remembered ≠ applied**. The session `%-print` (Harness-15) tells you *how much you stuffed into* hot-memory (by length). MFE asks the next question: does the agent actually **retain/use** what was stuffed in.
|
||||
|
||||
🔴 **DISTINCT from H6.7 "memoryDelta-routing-fidelity"** (the right delta landing in the right `agent-memory/<role>` under single-writer). Same word "fidelity", two senses — always say **memory-fidelity-EVAL / MFE** for this one. The pair is recorded in `governance-detectors.ps1` C3 alias-map so it is not flagged as drift.
|
||||
|
||||
## Two layers
|
||||
|
||||
| Layer | What | Cost | Status (honest) |
|
||||
|---|---|---|---|
|
||||
| **Deterministic ($0)** | `scripts/mfe-eval.ps1` — LEAD coverage-FIT + age-band + Goodhart-anchor; SUB per-role coverage; SUB-workflow N/A | $0, NO-API | ✅ built + wired (opt-in) |
|
||||
| **Judge (Branch A)** | recall/apply — give the agent the sample-questions, score recall + applied | $0 scaffold (quota only when a real scorer is wired) | 🟡 **SCAFFOLD only** — seed exists, **no scorer wired** |
|
||||
|
||||
## How to run
|
||||
|
||||
```
|
||||
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 # all tiers
|
||||
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Tier lead
|
||||
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Tier sub
|
||||
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Judge # show judge scaffold (no scoring)
|
||||
```
|
||||
|
||||
Opt-in at session ends: `/session-start … eval` (baseline) and `/session-end … eval` (retention) — see `session-start.md §2.1.6` / `session-end.md §L.b(c)`. Default (no `eval`) = unchanged behaviour.
|
||||
|
||||
## Operational decision (the point)
|
||||
|
||||
- **Coverage < 100% or set over-cap** = *lack-of-SPACE* → **INCREASE budget** (owner decides the number).
|
||||
- **Low recall despite the set fitting** = *rot/noise* → **REORGANIZE** (value-priority, cut low-value) — adding space does NOT fix rot.
|
||||
|
||||
## 🔴 Honest caveats (do not hide)
|
||||
|
||||
1. **Token sizing is a RANGE, not a number.** `char/4` is not real tokenization; Vietnamese-diacritic hot-memory is ~3.0–3.5 byte/token, so `byte/4` is an upper bound on headroom. The analyzer reports `[bytes/4 … bytes/3.0]` and uses the worst-case end for the FIT verdict. Real tokenizer count is inside the band.
|
||||
2. **The judge layer measures nothing yet.** A same-session self-grade is meaningless (the agent just read the answers). Real numbers need (a) the sample-questions to **mature in age** and (b) an **independent cross-session / different-model** judge. Until both, judge output is plumbing-smoke.
|
||||
3. **Age is a flag, never a cut** (mark `RC-…10-29-11`). An item leaves the must-remember set only on **status-change** (mark → Disabled, guard → retired, AS-row deleted), never by age.
|
||||
4. **No self-grading.** Coverage% is anchored to the real recurring-error signal (error-ledger strikes + RCA count). A high score next to rising strikes = the score lies.
|
||||
|
||||
## Files
|
||||
|
||||
- `scripts/mfe-eval.ps1` — deterministic analyzer (NO-API, ASCII-only, exit 0, READ-ONLY on `token_governor`).
|
||||
- `.claude/agent-memory/memory-budget.json` → `mfe` block — single-source config (denominator sources, stop-list, toggles, caveats).
|
||||
- `eval/mfe/sample-questions.json` — immutable seed (stable-id anchored), append-only.
|
||||
- `.claude/agent-memory/.mfe-state.json` — last-run strikes (cross-run Goodhart compare); MFE-only, never touches the budget.
|
||||
Reference in New Issue
Block a user