<# .SYNOPSIS memory-selfimprove-audit.ps1 - Harness-17 PART-C spec-compliance checker (KIEM / Audit arm). .DESCRIPTION NO-API, DETECT-and-FLAG-only spec-compliance net for the Harness-17 memory self-improvement loop (Do -> Tinh-chinh -> Kiem). Same house rules as governance-detectors.ps1: (1) NO-API - only Get-Content / Get-Item / child-invoke. NEVER calls model/API. (2) FLAG-only - prints [SPEC] criteria/gaps, NEVER edits files (single-writer = em-main D9). (3) PowerShell 5.1 compatible. ASCII-only script body (gotcha #30). Target files read -Encoding UTF8. Exit code ALWAYS 0 (advisory net, not a build gate). (4) LIVE every run - emits a CRITERIA-SET + GAP-SET, NOT a frozen pass-tally. The gap-set IS the actionable output (H17 tip: a frozen pass number drifts between sessions because the audit measures a LIVE artifact state). (5) SINGLE-SOURCE live-read - caps come from memory-budget.json at runtime (NEVER hardcode: the S94 owner-override raised lead 220K->380K; a hardcoded copy would silently rot. Also budget.measured is STALE - re-measure on disk). Three C-faces (H17 mandate: write-governance + change-governance + distillation): (a) WRITE-GOVERNANCE - single-writer D9 wired via TOOL-SCOPE + SCHEMA (A1/A2 mechanized) + propose-only declaration (A3 proxy). (b) CHANGE-GOVERNANCE - COMPOSE governance-detectors.ps1 (D8 one-way + C2/B3 derived); do NOT re-implement (that would fork C3). (c) DISTILLATION - every LIVE over-cap agent has raw MEMORY.md -> _INDEX.md -> gist. Plus an eval-arm proxy (HCV = harvest-coverage-value, run-trace completeness) and an always-emit CAVEAT / self-blind-spot disclosure (mandated honest surface). HONEST header: D9 single-writer is enforced by TOOL-SCOPE + SCHEMA (A1/A2) - NOT by an OS settings hook (.claude/settings.json has none; the store_memory allowlist-strip is AS-10/E-006 fails-open). A3 propose-only is a PROXY (phrase-presence != enforcement). This checker proves the guards are WIRED, not that runtime CANNOT violate. .PARAMETER RepoRoot Repo root. Default = resolved 2 levels up from this script (scripts/ -> repo root). .EXAMPLE powershell.exe -ExecutionPolicy Bypass -File scripts/memory-selfimprove-audit.ps1 #> param( [string]$RepoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path ) $ErrorActionPreference = 'Continue' # --------------------------------------------------------------------------- # Emit helpers (mirror governance-detectors.ps1 grammar so output composes) # --------------------------------------------------------------------------- $script:Criteria = New-Object System.Collections.Generic.List[string] $script:Gaps = New-Object System.Collections.Generic.List[string] function Write-Section($title) { Write-Host '' Write-Host ("===== $title =====") -ForegroundColor Cyan } # [SPEC] face | CRITERION | PASS | desc (records criterion, green) function Write-Pass { param([string]$Face, [string]$Criterion, [string]$Desc) Write-Host ("[SPEC] {0,-16} | {1,-14} | PASS | {2}" -f $Face, $Criterion, $Desc) -ForegroundColor Green $script:Criteria.Add($Criterion) | Out-Null } # [SPEC] face | CRITERION | GAP:SEV | desc | resolve: (records criterion + gap) function Write-Gap { param( [string]$Face, [string]$Criterion, [ValidateSet('HIGH', 'MED', 'LOW')] [string]$Severity, [string]$Desc, [string]$Resolve ) $color = switch ($Severity) { 'HIGH' { 'Red' } 'MED' { 'Yellow' } default { 'Gray' } } Write-Host ("[SPEC] {0,-16} | {1,-14} | GAP:{2,-4} | {3} | resolve: {4}" -f $Face, $Criterion, $Severity, $Desc, $Resolve) -ForegroundColor $color $script:Criteria.Add($Criterion) | Out-Null $script:Gaps.Add(("{0}:{1}" -f $Criterion, $Severity)) | Out-Null } # [SPEC] face | CRITERION | N/A | reason (records nothing - honest not-applicable) function Write-Na { param([string]$Face, [string]$Criterion, [string]$Reason) Write-Host ("[SPEC] {0,-16} | {1,-14} | N/A | {2}" -f $Face, $Criterion, $Reason) -ForegroundColor DarkGray } function Rel($full) { $r = $full if ($full.StartsWith($RepoRoot, [StringComparison]::OrdinalIgnoreCase)) { $r = $full.Substring($RepoRoot.Length).TrimStart('\', '/') } return ($r -replace '\\', '/') } function Read-Text($path) { if (-not (Test-Path $path)) { return $null } return (Get-Content -Path $path -Raw -Encoding UTF8) } # --------------------------------------------------------------------------- # SINGLE-SOURCE live-read: caps from memory-budget.json (NEVER hardcode) # --------------------------------------------------------------------------- Write-Section 'H17 spec-audit - single-source config live-read' $budgetPath = Join-Path $RepoRoot '.claude\agent-memory\memory-budget.json' $autoCap = $null $softCap = $null if (Test-Path $budgetPath) { try { $budget = Get-Content -Path $budgetPath -Raw -Encoding UTF8 | ConvertFrom-Json $autoCap = [int]$budget.tiers.l1_hot.autoinject_cap_bytes $softCap = [int]$budget.tiers.l1_hot.soft_cap_bytes Write-Host (" live-read caps: autoinject_cap={0}B soft_cap={1}B (memory-budget.json)" -f $autoCap, $softCap) Write-Host ' [OK] caps read LIVE from single source (NOT hardcoded; budget.measured is stale - ignored)' -ForegroundColor Green } catch { Write-Host (" [!] could not parse memory-budget.json: {0}" -f $_.Exception.Message) -ForegroundColor Red } } else { Write-Host (" [!] memory-budget.json not found at {0}" -f (Rel $budgetPath)) -ForegroundColor Red } if ($null -eq $autoCap) { $autoCap = 25600; Write-Host ' [note] falling back to autoinject_cap=25600 (config unreadable)' -ForegroundColor DarkGray } if ($null -eq $softCap) { $softCap = 30720 } $agentsDir = Join-Path $RepoRoot '.claude\agents' $agentMemDir = Join-Path $RepoRoot '.claude\agent-memory' $hmwPath = Join-Path $RepoRoot '.claude\workflows\hmw.js' # --------------------------------------------------------------------------- # FACE (a) - WRITE-GOVERNANCE / single-writer D9 # --------------------------------------------------------------------------- Write-Section 'FACE (a) WRITE-GOVERNANCE (single-writer D9)' # Collect agent .md (exclude README). Split by tool-capability: read-only vs write. $agentFiles = @() if (Test-Path $agentsDir) { $agentFiles = Get-ChildItem -Path $agentsDir -Filter *.md -File -ErrorAction SilentlyContinue | Where-Object { $_.BaseName -ne 'README' } } # --- A1: sub-agent tool-scope excludes memory-WRITE verbs (parse the `tools:` line ONLY) --- # NOTE: scan the frontmatter `tools:` line, NOT the whole file. Several agent BODIES say # "KHONG store_memory" (that is their A3 propose-only declaration); grepping the body would # false-positive. The D9 tool-scope guard is about what the tools[] array EXPOSES. $writeVerbRx = 'store_memory|delete_memory|update_memory|promote_to_shared' $a1Offenders = @() $readOnlyRoles = @() # roles whose tools[] lack Write AND Edit -> must carry A3 declaration foreach ($af in $agentFiles) { $lines = Get-Content -Path $af.FullName -Encoding UTF8 $toolsLine = ($lines | Where-Object { $_ -match '^tools:' } | Select-Object -First 1) if ($null -eq $toolsLine) { continue } if ($toolsLine -match $writeVerbRx) { $a1Offenders += $af.BaseName } $hasWrite = ($toolsLine -match '\bWrite\b') -or ($toolsLine -match '\bEdit\b') if (-not $hasWrite) { $readOnlyRoles += $af.BaseName } } if ($a1Offenders.Count -eq 0) { Write-Pass 'a:tool-scope' 'A1' ("{0} sub-agent tools[] carry only READ RAG verbs (no store_memory/delete/update)" -f $agentFiles.Count) } else { Write-Gap 'a:tool-scope' 'A1' 'HIGH' ` ("agent(s) expose a memory-WRITE verb: {0}" -f ($a1Offenders -join ', ')) ` 'remove store_memory/delete_memory/update_memory from that agent .md tools[] (single-writer = em-main D9)' } # --- A2: hmw.js return-delta-only SCHEMA intact --- $hmw = Read-Text $hmwPath if ($null -eq $hmw) { Write-Gap 'a:schema' 'A2' 'HIGH' 'hmw.js not found' 'restore .claude/workflows/hmw.js (D9 mechanized return-delta-only channel)' } else { $hasRequired = ($hmw -match "required:\s*\[\s*'findings'\s*,\s*'memoryDelta'\s*\]") $hasNoStore = ($hmw -match 'store_memory/RAG') # writeGuard forbidding line if ($hasRequired -and $hasNoStore) { Write-Pass 'a:schema' 'A2' "hmw.js SCHEMA required=[findings,memoryDelta] + writeGuard forbids store_memory/RAG" } else { $miss = @() if (-not $hasRequired) { $miss += "required=[findings,memoryDelta]" } if (-not $hasNoStore) { $miss += "writeGuard 'KHONG store_memory/RAG'" } Write-Gap 'a:schema' 'A2' 'HIGH' ` ("hmw.js missing return-delta guard literal(s): {0}" -f ($miss -join '; ')) ` 'restore the return-delta-only SCHEMA + writeGuard in hmw.js (D9 mechanized channel)' } } # --- A3: read-only-role agents carry a propose-only / single-writer declaration (PROXY) --- $declRx = 'propose-only|single-writer|read-only|READ only|KHONG store_memory|store_memory' $a3Missing = @() foreach ($role in $readOnlyRoles) { $body = Read-Text (Join-Path $agentsDir ("{0}.md" -f $role)) if ($null -eq $body) { continue } if ($body -notmatch $declRx) { $a3Missing += $role } } if ($readOnlyRoles.Count -eq 0) { Write-Na 'a:propose-only' 'A3' 'no read-only-role agents detected' } elseif ($a3Missing.Count -eq 0) { Write-Pass 'a:propose-only' 'A3' ("{0} read-only-role agents carry a propose-only/single-writer declaration (PROXY: phrase-presence)" -f $readOnlyRoles.Count) } else { Write-Gap 'a:propose-only' 'A3' 'MED' ` ("read-only-role agent(s) missing propose-only declaration: {0}" -f ($a3Missing -join ', ')) ` 'add "propose-only / single-writer" to that agent .md (PROXY guard)' } # --------------------------------------------------------------------------- # FACE (b) - CHANGE-GOVERNANCE / derived-pointer one-way D8 (COMPOSE, do NOT duplicate) # --------------------------------------------------------------------------- Write-Section 'FACE (b) CHANGE-GOVERNANCE (composed - D8 one-way + C2/B3)' # --- B1: reference-and-run governance-detectors.ps1 (delegate, do not re-grep counts) --- $detPath = Join-Path $RepoRoot 'scripts\governance-detectors.ps1' if (-not (Test-Path $detPath)) { Write-Gap 'b:detectors' 'B1' 'HIGH' 'governance-detectors.ps1 missing (owner of D8/C2/B3)' 'restore scripts/governance-detectors.ps1' } else { $detOut = & powershell.exe -NoProfile -ExecutionPolicy Bypass -File $detPath 2>&1 $detExit = $LASTEXITCODE $flagCount = @($detOut | Select-String -Pattern '\[DETECTOR\]').Count if ($detExit -eq 0) { Write-Pass 'b:detectors' 'B1' ("governance-detectors.ps1 present + ran (exit 0; {0} advisory FLAG surfaced - detector owns the verdict)" -f $flagCount) } else { Write-Gap 'b:detectors' 'B1' 'MED' ` ("governance-detectors.ps1 exited {0} (expected 0)" -f $detExit) ` 'investigate detector run failure (it is the D8/C2 owner)' } } # --- B2: change-process anchor exists (rule-change routes through a documented process) --- $rulesText = Read-Text (Join-Path $RepoRoot 'docs\rules.md') $engineText = Read-Text (Join-Path $RepoRoot 'docs\governance\harness-11-engine.md') $anchorHit = $false if ($rulesText -and ($rulesText -match 'report-before-stamp|report-tr\w*-\w*ng-d\w*u|6\.6')) { $anchorHit = $true } if ($engineText -and ($engineText -match 'D8|one-way|OWNER-APPROVE|canonical')) { $anchorHit = $true } if ($anchorHit) { Write-Pass 'b:change-proc' 'B2' 'change-governance route documented (D7 owner-approve / D8 one-way / report-before-stamp)' } else { Write-Gap 'b:change-proc' 'B2' 'LOW' 'no documented rule-change process anchor found' ` 'document D7 owner-approve + D8 one-way in rules.md or harness-11-engine.md' } # --------------------------------------------------------------------------- # FACE (c) - DISTILLATION-SPECTRUM / raw MEMORY.md -> _INDEX.md -> gist per over-cap agent # --------------------------------------------------------------------------- Write-Section 'FACE (c) DISTILLATION-SPECTRUM (live byte-measure per agent)' $overCapAny = $false if (Test-Path $agentMemDir) { $memDirs = Get-ChildItem -Path $agentMemDir -Directory -ErrorAction SilentlyContinue foreach ($md in $memDirs) { $memFile = Join-Path $md.FullName 'MEMORY.md' if (-not (Test-Path $memFile)) { continue } $len = (Get-Item $memFile).Length if ($len -le $autoCap) { continue } # under cap -> no spectrum required (trivial PASS) $overCapAny = $true $hasIndex = Test-Path (Join-Path $md.FullName 'archive\_INDEX.md') $gistCount = @(Get-ChildItem -Path (Join-Path $md.FullName 'archive') -Filter '*.gist.md' -File -ErrorAction SilentlyContinue).Count $crit = ("C-spectrum:{0}" -f $md.Name) if ($hasIndex -and $gistCount -gt 0) { Write-Pass 'c:distill' $crit ("{0} over cap ({1}B>{2}B) but has _INDEX + {3} gist(s)" -f $md.Name, $len, $autoCap, $gistCount) } else { $sev = if ($len -gt $softCap) { 'HIGH' } else { 'MED' } $miss = @() if (-not $hasIndex) { $miss += '_INDEX.md' } if ($gistCount -eq 0) { $miss += 'gist' } Write-Gap 'c:distill' $crit $sev ` ("{0} MEMORY.md={1}B over cap {2}B but missing {3}" -f $md.Name, $len, $autoCap, ($miss -join ' + ')) ` 'build distillation-spectrum (_INDEX + gist) or curate L1->L2 (/sleep-recovery-memory-l2)' } } } if (-not $overCapAny) { Write-Na 'c:distill' 'C-spectrum' 'no over-cap agent this run (all MEMORY.md under autoinject_cap - no spectrum required)' } # --------------------------------------------------------------------------- # EVAL-ARM proxy - HCV (harvest-coverage-value): run-trace harvest completeness # NOTE: named HCV to avoid the "coverage" 3-way vocab collision (H2 harvest-curator # "Coverage" = 0-silent-miss-of-spawns ; H16 MFE SUB "coverage" = role-floor retention ; # H17 A1 HCV = per-run sub-said vs harvested). Deterministic proxy = file-exists over the # most recent run-trace; the said-vs-kept RATIO is calibration-interim (H17 caveat 3). # --------------------------------------------------------------------------- Write-Section 'EVAL-ARM HCV proxy (run-trace harvest completeness)' $runsDir = Join-Path $RepoRoot '.claude\workflows\runs' if (-not (Test-Path $runsDir)) { Write-Na 'eval:hcv' 'HCV' 'no runs/ dir (no hmw run-trace this repo yet) - said-vs-kept proxy N/A' } else { $runDirs = Get-ChildItem -Path $runsDir -Directory -ErrorAction SilentlyContinue | Sort-Object LastWriteTime -Descending if ($runDirs.Count -eq 0) { Write-Na 'eval:hcv' 'HCV' 'runs/ empty - no run-trace to score' } else { $latest = $runDirs[0] # @S180 W1 FIX (naming-standard.md section 4): the filter used to be the literal glob # 'sub-*.md', so every '-return.md' was INVISIBLE here - including the three trio # returns (harness-eval / harness-refine / harness-audit), i.e. the most expensive lanes. # governance-detectors.ps1:1101-1102 already fixed exactly this at S159 and measured it on # the S159 bookend: 9 spawns ran, the glob saw 6. That fix was never mirrored into THIS # file even though both scan the same directory - a diagnosed, named, patched bug still # alive one file over. Regex below is a byte-for-byte mirror of the detectors one; if you # change one, change both (naming-standard.md section 12 lists all 3 sites). $subMd = @(Get-ChildItem -Path $latest.FullName -File -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^(sub-.+|.+-return)\.md$' }) $synth = @(Get-ChildItem -Path $latest.FullName -Filter '*-synthesis.md' -File -ErrorAction SilentlyContinue) if ($subMd.Count -eq 0) { Write-Na 'eval:hcv' 'HCV' ("latest run {0} has no sub-*.md / *-return.md (return-delta-only mode) - proxy N/A" -f $latest.Name) } elseif ($synth.Count -gt 0) { Write-Pass 'eval:hcv' 'HCV' ("latest run {0}: {1} sub-md + {2} synthesis present (harvest-complete proxy; ratio = calibration-interim)" -f $latest.Name, $subMd.Count, $synth.Count) } else { Write-Gap 'eval:hcv' 'HCV' 'MED' ` ("latest run {0}: {1} sub-md but 0 synthesis - possible silent harvest-loss" -f $latest.Name, $subMd.Count) ` 'harvest the run-trace (write *-synthesis.md) before closing (H2 harvest-curator gate)' } } } # --------------------------------------------------------------------------- # CAVEATS / SELF-BLIND-SPOT (always-emit disclosure - NOT a check; H17 mandate) # --------------------------------------------------------------------------- Write-Section 'CAVEATS / self-blind-spot (honest surface - always emitted)' $caveats = @( 'C1. Loop "runs" = MANUAL walkthrough chained by em-main (detect -> classify -> record). This script + governance-detectors run exit-0 SEPARATELY; there is NO single auto-run that closes the whole loop each session.', 'C2. Recall/apply JUDGE layer = STUB (mfe-eval -Judge scaffold). Numbers meaningless until sample-questions mature AND an independent cross-session judge scores. Not measured here.', 'C3. HCV harvest-coverage threshold = calibration-interim (single-peak distribution -> tail-flag, NOT a chosen quality bar).', 'C4. Meta-blind-spot: this checker audits the WORKER artifacts (agents/memory/hmw). It does NOT audit the eval/refine/audit meta-arms themselves (the measurer is not yet self-measured).', 'C5. Distillation = compress-only so far. The verify-BEFORE-keep step (validate content before a distill deletes the raw) is NOT built.' ) foreach ($c in $caveats) { Write-Host (" - {0}" -f $c) -ForegroundColor DarkGray } Write-Host ' - SCOPE: this checker covers C-face (a)(b)(c) DETERMINISTICALLY + HCV proxy. It does NOT verify B1 signal->action MAP application, B2 CG-1 termination decisions, or A2 load-fidelity NUMBERS (those are separate arms: mfe-eval.ps1 + the reinject-ledger + em-main judgement).' -ForegroundColor DarkGray Write-Host ' - BUILD-GAP honest: B1/B2 refine mechanics are convention (em-main hand); the reinject-ledger is a git-tracked append-only record, NOT an auto-writer. Do not read "checker silent" as "fully compliant".' -ForegroundColor DarkGray Write-Host ' - D9 ENFORCEMENT honest: single-writer is wired by TOOL-SCOPE (A1) + SCHEMA (A2) - NOT by an OS hook (.claude/settings.json has none; store_memory allowlist-strip is AS-10/E-006 fails-open). A3 is a PROXY. This proves guards are WIRED, not that runtime CANNOT violate.' -ForegroundColor DarkGray # --------------------------------------------------------------------------- # Summary - CRITERIA-SET + GAP-SET (LIVE, no frozen pass-tally) # --------------------------------------------------------------------------- Write-Section 'Summary (criteria-set + gap-set - re-computed LIVE each run)' Write-Host ("CRITERIA (this run, LIVE): {0}" -f ($script:Criteria -join ' ')) if ($script:Gaps.Count -eq 0) { Write-Host 'GAPS (this run): none' -ForegroundColor Green } else { Write-Host ("GAPS (this run): {0}" -f ($script:Gaps -join ' ')) -ForegroundColor Yellow } Write-Host '' Write-Host 'NOTE: H17 PART-C spec-audit. CRITERIA+GAP set is the output (NOT a frozen score - it drifts as artifacts self-heal). Exit 0 always (advisory). Cite the GAP-set, not a tally.' -ForegroundColor DarkGray exit 0