vietreport-admin/solution-erp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
92de58f329167b530f9b1fd842674a2636a3b5ef
solution-erp/eval/mfe
History
pqhuy1987 958009858e
All checks were successful
Deploy SOLUTION_ERP / build-deploy (push) Successful in 5m2s
Details
[CLAUDE] Docs: adopt Harness-16 MFE (memory-fidelity-EVAL) — 2-workflow + email AI_INFRA 
Adopt AI_INFRA Harness-16 (3 broadcast 2026-06-29) qua 2-workflow mandate:
WF1 implement wf_4c63e1bd-99e + WF2 review wf_13e3d35a-023 (PASS 0-blocking).

MFE = coverage/retention eval (do nap-vs-nho-vs-dung bang SO DO). DISTINCT
voi H6.7 memoryDelta-routing-fidelity (vocab-fork WF1 reviewer bat duoc).

Built (em-main single-writer, 0 production code):
- scripts/mfe-eval.ps1 deterministic NO-API (ASCII #30, exit 0): LEAD coverage-FIT
  (token-RANGE, cap live-read) + age-band flag-not-cut + Goodhart-anchor strikes/RCA;
  SUB per-role coverage do-that (prose->N/A khong 0%); sub-workflow N/A. Smoke 3-tier PASS.
- memory-budget.json :mfe config + eval/mfe/ (seed sample-questions stable-id + README)
  + engine §H + artifact-row + C3-comment + wire session-start §2.1.6 / session-end §L.b(c)
  opt-in + agents/README S93. Judge layer = SCAFFOLD-only (honest nac).

adap-report harness-16-mfe + harness-15-v3 (covered/folded) + email outbox/ai_infra.
AS-10 dogfood: WF1 residual-write caught+reverted. State GIU NGUYEN (Mig 59 · 88 · 434 · gotcha 76).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 18:35:14 +07:00
..
README.md
[CLAUDE] Docs: adopt Harness-16 MFE (memory-fidelity-EVAL) — 2-workflow + email AI_INFRA
2026-06-29 18:35:14 +07:00
sample-questions.json
[CLAUDE] Docs: adopt Harness-16 MFE (memory-fidelity-EVAL) — 2-workflow + email AI_INFRA
2026-06-29 18:35:14 +07:00

README.md

MFE — Memory-Fidelity-EVAL (Harness-16)

Adopt S93 (2026-06-29) from AI_INFRA broadcasts 2026-06-29-Governance-harness-16-* (memory-fidelity-eval + mfe-update + cd-judge-sub-update). Canonical mechanism → docs/governance/harness-11-engine.md §H.

What MFE is (and is NOT)

MFE measures provisioned ≠ remembered ≠ applied. The session %-print (Harness-15) tells you how much you stuffed into hot-memory (by length). MFE asks the next question: does the agent actually retain/use what was stuffed in.

🔴 DISTINCT from H6.7 "memoryDelta-routing-fidelity" (the right delta landing in the right agent-memory/<role> under single-writer). Same word "fidelity", two senses — always say memory-fidelity-EVAL / MFE for this one. The pair is recorded in governance-detectors.ps1 C3 alias-map so it is not flagged as drift.

Two layers

Layer What Cost Status (honest)
Deterministic ($0) scripts/mfe-eval.ps1 — LEAD coverage-FIT + age-band + Goodhart-anchor; SUB per-role coverage; SUB-workflow N/A $0, NO-API ✅ built + wired (opt-in)
Judge (Branch A) recall/apply — give the agent the sample-questions, score recall + applied $0 scaffold (quota only when a real scorer is wired) 🟡 SCAFFOLD only — seed exists, no scorer wired

How to run

powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1            # all tiers
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Tier lead
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Tier sub
powershell.exe -ExecutionPolicy Bypass -File scripts\mfe-eval.ps1 -Judge     # show judge scaffold (no scoring)

Opt-in at session ends: /session-start … eval (baseline) and /session-end … eval (retention) — see session-start.md §2.1.6 / session-end.md §L.b(c). Default (no eval) = unchanged behaviour.

Operational decision (the point)

  • Coverage < 100% or set over-cap = lack-of-SPACE → INCREASE budget (owner decides the number).
  • Low recall despite the set fitting = rot/noise → REORGANIZE (value-priority, cut low-value) — adding space does NOT fix rot.

🔴 Honest caveats (do not hide)

  1. Token sizing is a RANGE, not a number. char/4 is not real tokenization; Vietnamese-diacritic hot-memory is ~3.0–3.5 byte/token, so byte/4 is an upper bound on headroom. The analyzer reports [bytes/4 … bytes/3.0] and uses the worst-case end for the FIT verdict. Real tokenizer count is inside the band.
  2. The judge layer measures nothing yet. A same-session self-grade is meaningless (the agent just read the answers). Real numbers need (a) the sample-questions to mature in age and (b) an independent cross-session / different-model judge. Until both, judge output is plumbing-smoke.
  3. Age is a flag, never a cut (mark RC-…10-29-11). An item leaves the must-remember set only on status-change (mark → Disabled, guard → retired, AS-row deleted), never by age.
  4. No self-grading. Coverage% is anchored to the real recurring-error signal (error-ledger strikes + RCA count). A high score next to rising strikes = the score lies.

Files

  • scripts/mfe-eval.ps1 — deterministic analyzer (NO-API, ASCII-only, exit 0, READ-ONLY on token_governor).
  • .claude/agent-memory/memory-budget.json → mfe block — single-source config (denominator sources, stop-list, toggles, caveats).
  • eval/mfe/sample-questions.json — immutable seed (stable-id anchored), append-only.
  • .claude/agent-memory/.mfe-state.json — last-run strikes (cross-run Goodhart compare); MFE-only, never touches the budget.