[CLAUDE] Docs: S103 AI_INFRA 6-broadcast adap — crystallized-backfill (measure-then-fill) + Harness-20 fix-all-structure + memory-hygiene mapping
All checks were successful
Deploy SOLUTION_ERP / build-deploy (push) Successful in 5m26s
All checks were successful
Deploy SOLUTION_ERP / build-deploy (push) Successful in 5m26s
Adap 6 broadcast (hash MATCH cả 6 indep-recompute) qua 2-workflow Harness-9: - Harness-20 fix-all-structure -> engine §L + dogfood model-note 4-tầng-sweep (incl PERSONA) + 220K→380K neo-supersede + 2-đầu-catch (review bắt 2 DẪN-XUẤT stale-trailer sweep-sót → fixed) - F4 crystallized-backfill = NEW scripts/crystallized-backfill.ps1 (measure-then-fill DRY NO-API; default target=0=OFF owner-authority; calibrated-range; precondition; file-floor=UPPER-BOUND honest) + budget.json config + budget-table wire - F1/F3 already-met (0 hardcoded-violation persona-clean; BƯỚC 0.6 bắt S103=Opus real-metadata) - session-cmd budget-table (Up1) + run-loop 2-tầng (Up2 largely-already-met §L.b(c)) - H16 no-key Applied% = NEW scripts/applied-eval-nokey.ps1 (adversarial-gate fail-closed 3-ca + fault-inject FLIP; mechanism-built numbers-DEFERRED) - memory-hygiene A/B already-met (archive-gate DRY-RUN + governance-detectors 4-layer 1:1); C n/a JS-sandbox no-CLI-launcher - consolidation reference-ack (crystallization≠backfill) engine §L+§M · 6 adap-report + 1 adap-request (#53-garble reasoning-lane mitigation) · send-email fc1e22ac selftest-MATCH · check-email 3 pulled+verified+STAGE-2. 2-workflow: WF1 implement wf_55ef6778-a0d (3 lane schema-CLEAN, selftest exit-0 ×2 + fault-inject teeth) + WF2 review wf_3c5c37cb-aa2 fable-clone (Lane-C PASS returned + Lane-B recover-from-memory PASS; 0-BLOCKING 2-CONCERN-addressed). #53 garble ×3 → em-main recover; investigator AS-10 garble-curation REVERTED (git-backstop). Runtime S103 = Opus (Fable outage lại, per-session flip; H5-fallback revert-free). State GIỮ NGUYÊN: Mig 61 · 89 bảng · 440 test · gotcha 79 · menu 54. 0 prod-code. Co-Authored-By: Claude Opus 4.8 (1M) <noreply@anthropic.com>
This commit is contained in:
pqhuy1987
362
scripts/applied-eval-nokey.ps1
Normal file
362
scripts/applied-eval-nokey.ps1
Normal file
@ -0,0 +1,362 @@
|
||||
# applied-eval-nokey.ps1 - Harness-16 UPDATE: Applied% WITHOUT an API key + adversarial gate - S102 (2026-07-06)
|
||||
#
|
||||
# ADAPT of AI_INFRA broadcast H16-update (owner-directive, 2026-07-04/05). SISTER-side KHUNG,
|
||||
# tailored to SOLUTION_ERP (self=se). Function-floor kept; form tailored. NO copy-paste.
|
||||
#
|
||||
# ============================================================================
|
||||
# WHAT THIS IS (and, loudly, what it is NOT)
|
||||
# ============================================================================
|
||||
# Two SEPARATE things live in this file, and they must never be confused:
|
||||
#
|
||||
# [A] SCRIPTABLE / DETERMINISTIC ($0) <-- the REAL teeth of this script, runs here, now.
|
||||
# An ADVERSARIAL GATE + a MERGE + a 3-case SELFTEST. Given a set of judge
|
||||
# verdicts (each {id, score, is_control, control_expected_low}), the gate
|
||||
# decides BLOCK / PASS, and the merge folds surviving real questions into a
|
||||
# single Applied% number. Zero model calls. Pure arithmetic + set logic.
|
||||
#
|
||||
# [B] RUNTIME / 2-AGENT-PASS <-- DOCUMENTED ONLY here; runs later, via a workflow,
|
||||
# when the sample-questions corpus is mature. NOT executed by this script.
|
||||
# This block exists so the mechanism is written down and auditable, not so a
|
||||
# number is produced today.
|
||||
#
|
||||
# The honesty ceiling (printed by -Honest and at the end of -Selftest):
|
||||
# "We have an adversarial GATE (MANUAL: you supply the verdicts + the control
|
||||
# list) + a 3-case selftest. We do NOT claim any live number has passed this
|
||||
# gate. Applied% numbers are DEFERRED until the sample-questions mature AND an
|
||||
# independent, different-session judge scores them. This file is a MECHANISM,
|
||||
# NOT a number."
|
||||
#
|
||||
# ============================================================================
|
||||
# [B] THE NO-KEY APPLIED% FLOW (documented; RUNTIME, not run here)
|
||||
# ============================================================================
|
||||
# The point: measure "does the hot-feed actually get APPLIED" without any paid
|
||||
# API / external key -- using two in-harness sub-agents instead of an oracle key.
|
||||
#
|
||||
# STEP 1 - PROBE with the ANSWER BLANKED (the "candidate" sub-agent)
|
||||
# * Take a sample question that has a known answer-anchor (the gotcha #, the
|
||||
# mark-id, the file:line, the root-cause phrase). BLANK IT OUT: strip the
|
||||
# answer-anchor from the probe so the candidate cannot read the answer back.
|
||||
# * Spawn a sub-agent IN THE RIGHT PERSONA for that question (the role whose
|
||||
# diary is supposed to carry that memory), carrying that role's diary as it
|
||||
# normally would. The candidate is told NOTHING about being graded and sees
|
||||
# NO answer key -- it just does the task and emits its working (a "diary" of
|
||||
# what it recalled / applied for that role).
|
||||
#
|
||||
# STEP 2 - JUDGE in a CLEAN, SEPARATE CONTEXT (the "referee" sub-agent)
|
||||
# * A DIFFERENT sub-agent, in a FRESH context window, DIFFERENT session/turn
|
||||
# from the candidate (no shared scratch, no candidate's chain-of-thought),
|
||||
# is GIVEN the ground-truth answer-anchor and scores the candidate 0..1 on
|
||||
# whether the required memory was actually recalled/applied.
|
||||
# * Control/sentinel probes are mixed in (see below): questions the referee
|
||||
# SHOULD score low (answer deliberately absent from any diary). They are the
|
||||
# adversarial tripwire for a too-lenient judge.
|
||||
#
|
||||
# STEP 3 - MERGE LOCALLY ($0) -> Applied%
|
||||
# * The verdicts come back here. The GATE (block A, deterministic) runs FIRST:
|
||||
# drop the whole batch if any control scored high, or if any control was
|
||||
# silently skipped. Only then MERGE: strip controls, average the real
|
||||
# scores, Applied% = round(mean * 100). Unscored (null) real questions are
|
||||
# NOT invented -- excluded + counted (N excluded).
|
||||
#
|
||||
# ============================================================================
|
||||
# 4 CAVEATS (MANDATORY -- this measurement is weak in exactly these 4 ways)
|
||||
# ============================================================================
|
||||
# (a) POINT-IN-TIME PROXY, not real behavior. A judged score at one instant is a
|
||||
# proxy for "was it applied", NOT observed downstream behavior over a real
|
||||
# task. High Applied% != the memory changed what the system actually did.
|
||||
#
|
||||
# (b) SHARED MODEL-FAMILY NOT NEUTRALIZED. Candidate and referee share the same
|
||||
# base model + the same corpus/RAG. Blanking the ANSWER-ANCHOR does NOT blank
|
||||
# the WHOLE CORPUS -- the referee (and even the candidate) may reconstruct the
|
||||
# answer from common training / shared store rather than from the diary under
|
||||
# test. The ONLY model-independent signal is RECURRENCE-FREQUENCY anchoring
|
||||
# (does the SAME bug/mark stop recurring over sessions) -- that lives in the
|
||||
# OLD measurable-now block (recurrence tokens / AS-class registry), NOT here.
|
||||
# This no-key score is the WEAKER, model-entangled cousin of that.
|
||||
#
|
||||
# (c) BLINDING IS AN INSTRUCTION, NOT AN ENFORCEMENT. "Candidate doesn't see the
|
||||
# answer / doesn't know it's graded" and "referee is a clean separate context"
|
||||
# are DIRECTIVES to the spawn, not a sandbox that makes leakage impossible. A
|
||||
# shared file, a chatty prompt, or a resumed session can defeat it. Treat the
|
||||
# separation as best-effort, and prefer the control tripwire over trusting it.
|
||||
#
|
||||
# (d) NO AGE DIMENSION YET, SMALL SAMPLE. This does not measure retention-over-age
|
||||
# (does an OLD memory still get applied), and the sample-question set is small.
|
||||
# So even a clean pass is a thin, young signal -- a FLAG, never a grade.
|
||||
#
|
||||
# ============================================================================
|
||||
# NON-NEGOTIABLES (Harness-11 / Harness-16 house rules)
|
||||
# ============================================================================
|
||||
# (1) NO-API : deterministic arithmetic + set logic ONLY. NEVER calls a model.
|
||||
# (The 2-agent pass in block [B] is RUNTIME via a workflow, not here.)
|
||||
# (2) READ-ONLY on the budget : this script NEVER reads-to-tune nor writes the
|
||||
# memory-budget.json caps. Budget numbers are the OWNER's authority (anh /
|
||||
# chu-du-an); scripts only ever measure-and-report. This file does not even
|
||||
# need the budget -- it touches NO owner-set number.
|
||||
# (3) PS 5.1, ASCII-only script body (gotcha #30). Any glyph via [char]0xXXXX
|
||||
# code-point; any target file read with -Encoding UTF8 explicitly.
|
||||
# (4) Exit 0 in normal + -Honest runs (measure-and-report, NOT a build gate).
|
||||
# -Selftest is the ONE exception: it exits 0 only if all 3 cases behave as
|
||||
# specified, non-zero otherwise, so CI/a human can trust the gate has teeth.
|
||||
#
|
||||
# Usage:
|
||||
# powershell.exe -ExecutionPolicy Bypass -File scripts\applied-eval-nokey.ps1 -Selftest
|
||||
# powershell.exe -ExecutionPolicy Bypass -File scripts\applied-eval-nokey.ps1 -Honest
|
||||
# powershell.exe -ExecutionPolicy Bypass -File scripts\applied-eval-nokey.ps1 # prints honesty rung + usage
|
||||
|
||||
param(
|
||||
[switch]$Selftest = $false,
|
||||
[switch]$Honest = $false
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$CHECK = [char]0x2705 # green check, kept as code-point (ASCII body, gotcha #30)
|
||||
$CROSS = [char]0x274C # red cross, kept as code-point
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constant: a control (sentinel) probe should score BELOW this to be "correctly
|
||||
# rejected". At-or-above = the referee wrongly credited a control = too lenient.
|
||||
# ---------------------------------------------------------------------------
|
||||
Set-Variable -Name CONTROL_HIGH_THRESHOLD -Value 0.5 -Option Constant
|
||||
|
||||
# ===========================================================================
|
||||
# [A] ADVERSARIAL GATE (deterministic -- the real, tested teeth)
|
||||
# ===========================================================================
|
||||
# INPUT : array of verdict objects (PSCustomObject or hashtable), each with
|
||||
# id : string, probe id
|
||||
# score : double 0..1, OR $null = judge did not score it
|
||||
# is_control : bool, true = sentinel/control probe
|
||||
# control_expected_low : bool, (advisory) a control is expected low
|
||||
# OUTPUT : PSCustomObject { Decision = 'BLOCK'|'PASS'; Reasons = string[];
|
||||
# Controls = int; ControlsHighOrMissing = int }
|
||||
#
|
||||
# LOGIC (fail-closed -- any doubt about the referee's rigor => BLOCK the batch):
|
||||
# * If ANY control has a null/missing score -> BLOCK
|
||||
# (a control that was silently dropped means the judge quietly skipped the
|
||||
# tripwire; we do NOT let a batch through with an un-checked control.)
|
||||
# * If ANY control scored >= CONTROL_HIGH_THRESHOLD -> BLOCK
|
||||
# (the judge credited a probe whose answer is deliberately absent = too
|
||||
# lenient => every real score from this same judge is suspect => drop all.)
|
||||
# * Else (every control present AND every control < threshold) -> PASS.
|
||||
# ---------------------------------------------------------------------------
|
||||
function Get-ScoreOrNull($v) {
|
||||
# Read .score whether $v is a hashtable or a PSCustomObject; return $null if
|
||||
# absent or explicitly null. Do NOT coerce a missing key into 0 (that would
|
||||
# hide a skipped control).
|
||||
if ($null -eq $v) { return $null }
|
||||
if ($v -is [System.Collections.IDictionary]) {
|
||||
if (-not $v.Contains('score')) { return $null }
|
||||
$s = $v['score']
|
||||
} else {
|
||||
$prop = $v.PSObject.Properties['score']
|
||||
if ($null -eq $prop) { return $null }
|
||||
$s = $prop.Value
|
||||
}
|
||||
if ($null -eq $s) { return $null }
|
||||
return [double]$s
|
||||
}
|
||||
|
||||
function Get-BoolField($v, [string]$name) {
|
||||
if ($null -eq $v) { return $false }
|
||||
if ($v -is [System.Collections.IDictionary]) {
|
||||
if (-not $v.Contains($name)) { return $false }
|
||||
return [bool]$v[$name]
|
||||
}
|
||||
$prop = $v.PSObject.Properties[$name]
|
||||
if ($null -eq $prop) { return $false }
|
||||
return [bool]$prop.Value
|
||||
}
|
||||
|
||||
function Invoke-AdversarialGate {
|
||||
param([object[]]$Verdicts)
|
||||
|
||||
$reasons = New-Object System.Collections.Generic.List[string]
|
||||
$controls = 0
|
||||
$controlsHighOrMissing = 0
|
||||
|
||||
foreach ($v in $Verdicts) {
|
||||
$isControl = Get-BoolField $v 'is_control'
|
||||
if (-not $isControl) { continue }
|
||||
$controls++
|
||||
$id = if (Get-BoolField $v 'id') { $v.id } else { $v.id } # id passthrough (may be $null)
|
||||
$score = Get-ScoreOrNull $v
|
||||
|
||||
if ($null -eq $score) {
|
||||
$controlsHighOrMissing++
|
||||
$reasons.Add("control '$($v.id)' has NO score (silently skipped) -> BLOCK")
|
||||
continue
|
||||
}
|
||||
if ($score -ge $CONTROL_HIGH_THRESHOLD) {
|
||||
$controlsHighOrMissing++
|
||||
$reasons.Add("control '$($v.id)' scored $score >= $CONTROL_HIGH_THRESHOLD (judge too lenient) -> BLOCK")
|
||||
}
|
||||
}
|
||||
|
||||
if ($controls -eq 0) {
|
||||
# No tripwire at all: cannot certify the judge's rigor. Fail-closed.
|
||||
$reasons.Add("no control probes present -> cannot certify judge rigor -> BLOCK")
|
||||
return [pscustomobject]@{
|
||||
Decision = 'BLOCK'; Reasons = $reasons.ToArray()
|
||||
Controls = 0; ControlsHighOrMissing = 0
|
||||
}
|
||||
}
|
||||
|
||||
$decision = if ($controlsHighOrMissing -gt 0) { 'BLOCK' } else { 'PASS' }
|
||||
return [pscustomobject]@{
|
||||
Decision = $decision; Reasons = $reasons.ToArray()
|
||||
Controls = $controls; ControlsHighOrMissing = $controlsHighOrMissing
|
||||
}
|
||||
}
|
||||
|
||||
# ===========================================================================
|
||||
# MERGE (deterministic -- only reached AFTER the gate returns PASS)
|
||||
# ===========================================================================
|
||||
# Strip controls, average the REAL scores, Applied% = round(mean * 100).
|
||||
# A real question the judge did not score (null) is NOT invented -> excluded +
|
||||
# counted in ExcludedNull. If, after excluding controls + nulls, no real scored
|
||||
# question remains, AppliedPct = $null (report "no scored real questions"),
|
||||
# NEVER a fabricated 0.
|
||||
# ---------------------------------------------------------------------------
|
||||
function Merge-AppliedScore {
|
||||
param([object[]]$Verdicts)
|
||||
|
||||
$realScores = New-Object System.Collections.Generic.List[double]
|
||||
$excludedNull = 0
|
||||
$controlCount = 0
|
||||
|
||||
foreach ($v in $Verdicts) {
|
||||
if (Get-BoolField $v 'is_control') { $controlCount++; continue }
|
||||
$score = Get-ScoreOrNull $v
|
||||
if ($null -eq $score) { $excludedNull++; continue } # do NOT fabricate a number
|
||||
$realScores.Add($score)
|
||||
}
|
||||
|
||||
if ($realScores.Count -eq 0) {
|
||||
return [pscustomobject]@{
|
||||
AppliedPct = $null; RealN = 0; ExcludedNull = $excludedNull
|
||||
ControlN = $controlCount; Mean = $null
|
||||
}
|
||||
}
|
||||
$mean = ($realScores | Measure-Object -Average).Average
|
||||
return [pscustomobject]@{
|
||||
AppliedPct = [int][Math]::Round($mean * 100)
|
||||
RealN = $realScores.Count
|
||||
ExcludedNull = $excludedNull
|
||||
ControlN = $controlCount
|
||||
Mean = [Math]::Round($mean, 4)
|
||||
}
|
||||
}
|
||||
|
||||
# ===========================================================================
|
||||
# Honesty rung (printed on plain run + at end of -Selftest)
|
||||
# ===========================================================================
|
||||
function Write-HonestyRung {
|
||||
Write-Host ""
|
||||
Write-Host "--- HONESTY RUNG (applied-eval-nokey) ---"
|
||||
Write-Host "We HAVE : an adversarial GATE (MANUAL -- you supply the verdicts + the control list)"
|
||||
Write-Host " + a 3-case selftest proving the gate BLOCKs a lenient/dropped control."
|
||||
Write-Host "We do NOT claim any live number has passed this gate."
|
||||
Write-Host "Applied% numbers are DEFERRED until (a) the sample-questions mature AND"
|
||||
Write-Host " (b) an independent, DIFFERENT-SESSION judge scores them."
|
||||
Write-Host "This file is a MECHANISM, NOT a number."
|
||||
Write-Host ""
|
||||
Write-Host "4 CAVEATS baked in: (a) point-in-time proxy != real behavior; (b) shared model-family"
|
||||
Write-Host " not neutralized (blank-ANSWER != blank-CORPUS; only recurrence-freq anchoring"
|
||||
Write-Host " is model-independent, and that lives in the OLD measurable-now block);"
|
||||
Write-Host " (c) blinding is an instruction, not an enforcement; (d) no age dimension yet,"
|
||||
Write-Host " small sample -- a FLAG, never a grade."
|
||||
}
|
||||
|
||||
# ===========================================================================
|
||||
# SELFTEST (3 cases, internal fake data -- the ONE path that may exit non-zero)
|
||||
# ===========================================================================
|
||||
function Invoke-Selftest {
|
||||
# $script:-scoped so the nested Assert-Case can flip it and the flip survives
|
||||
# back here (a plain local would NOT propagate from the nested function --
|
||||
# caught by fault-injection: a sabotaged gate printed CASE FAIL yet the summary
|
||||
# still said ALL PASS + exit 0 when this was function-local).
|
||||
$script:allPass = $true
|
||||
|
||||
function Assert-Case {
|
||||
param([string]$Name, [bool]$Condition, [string]$Detail)
|
||||
if ($Condition) {
|
||||
Write-Host ("{0} CASE {1}: PASS ({2})" -f $CHECK, $Name, $Detail)
|
||||
} else {
|
||||
Write-Host ("{0} CASE {1}: FAIL ({2})" -f $CROSS, $Name, $Detail)
|
||||
$script:allPass = $false
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "=== applied-eval-nokey SELFTEST (3 cases) ==="
|
||||
|
||||
# -- Case (a): a control scored HIGH (0.9) -> MUST BLOCK -------------------
|
||||
$caseA = @(
|
||||
[pscustomobject]@{ id='q1'; score=0.8; is_control=$false; control_expected_low=$false }
|
||||
[pscustomobject]@{ id='q2'; score=0.7; is_control=$false; control_expected_low=$false }
|
||||
[pscustomobject]@{ id='ctlA'; score=0.9; is_control=$true; control_expected_low=$true }
|
||||
)
|
||||
$rA = Invoke-AdversarialGate -Verdicts $caseA
|
||||
Assert-Case -Name 'a-control-HIGH' -Condition ($rA.Decision -eq 'BLOCK') `
|
||||
-Detail ("control 0.9 -> Decision=$($rA.Decision) (expect BLOCK); reason=" + ($rA.Reasons -join '; '))
|
||||
|
||||
# -- Case (b): all controls LOW (0.1) -> MUST PASS + produce Applied% ------
|
||||
$caseB = @(
|
||||
[pscustomobject]@{ id='q1'; score=0.8; is_control=$false; control_expected_low=$false }
|
||||
[pscustomobject]@{ id='q2'; score=0.6; is_control=$false; control_expected_low=$false }
|
||||
[pscustomobject]@{ id='ctlB'; score=0.1; is_control=$true; control_expected_low=$true }
|
||||
)
|
||||
$rB = Invoke-AdversarialGate -Verdicts $caseB
|
||||
$mB = $null
|
||||
if ($rB.Decision -eq 'PASS') { $mB = Merge-AppliedScore -Verdicts $caseB }
|
||||
# real scores 0.8,0.6 -> mean 0.7 -> Applied% = 70; control excluded.
|
||||
$bOk = ($rB.Decision -eq 'PASS') -and ($null -ne $mB) -and ($mB.AppliedPct -eq 70) -and ($mB.ControlN -eq 1)
|
||||
$bDetail = if ($null -ne $mB) {
|
||||
"Decision=$($rB.Decision); Applied%=$($mB.AppliedPct) (expect 70); RealN=$($mB.RealN); ControlN=$($mB.ControlN)"
|
||||
} else {
|
||||
"Decision=$($rB.Decision) (expect PASS) -- merge not reached"
|
||||
}
|
||||
Assert-Case -Name 'b-control-LOW' -Condition $bOk -Detail $bDetail
|
||||
|
||||
# -- Case (c): a control was FORGOTTEN (null score) -> MUST BLOCK ----------
|
||||
$caseC = @(
|
||||
[pscustomobject]@{ id='q1'; score=0.9; is_control=$false; control_expected_low=$false }
|
||||
[pscustomobject]@{ id='q2'; score=0.8; is_control=$false; control_expected_low=$false }
|
||||
[pscustomobject]@{ id='ctlC'; score=$null; is_control=$true; control_expected_low=$true }
|
||||
)
|
||||
$rC = Invoke-AdversarialGate -Verdicts $caseC
|
||||
Assert-Case -Name 'c-control-FORGOTTEN' -Condition ($rC.Decision -eq 'BLOCK') `
|
||||
-Detail ("control score=null -> Decision=$($rC.Decision) (expect BLOCK); reason=" + ($rC.Reasons -join '; '))
|
||||
|
||||
Write-Host ""
|
||||
if ($script:allPass) {
|
||||
Write-Host ("{0} SELFTEST: ALL 3 CASES PASS" -f $CHECK)
|
||||
} else {
|
||||
Write-Host ("{0} SELFTEST: FAILURE -- at least one case did not behave as specified" -f $CROSS)
|
||||
}
|
||||
|
||||
Write-HonestyRung
|
||||
return $script:allPass
|
||||
}
|
||||
|
||||
# ===========================================================================
|
||||
# Entry point
|
||||
# ===========================================================================
|
||||
if ($Selftest) {
|
||||
$ok = Invoke-Selftest
|
||||
if ($ok) { exit 0 } else { exit 1 }
|
||||
}
|
||||
|
||||
if ($Honest) {
|
||||
Write-HonestyRung
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Plain run: no live data path exists yet (numbers DEFERRED). Print the rung + usage.
|
||||
Write-Host "applied-eval-nokey.ps1 -- MECHANISM for a no-API Applied% measurement (adversarial gate + merge)."
|
||||
Write-Host "No live numbers are produced: the 2-agent probe/judge pass is RUNTIME (via a workflow) and"
|
||||
Write-Host "runs only when the sample-questions corpus matures. This script contributes the DETERMINISTIC"
|
||||
Write-Host "gate + merge + selftest that the runtime verdicts flow through."
|
||||
Write-HonestyRung
|
||||
Write-Host ""
|
||||
Write-Host "Run the selftest: powershell.exe -ExecutionPolicy Bypass -File scripts\applied-eval-nokey.ps1 -Selftest"
|
||||
exit 0
|
||||
254
scripts/crystallized-backfill.ps1
Normal file
254
scripts/crystallized-backfill.ps1
Normal file
@ -0,0 +1,254 @@
|
||||
# crystallized-backfill.ps1 - Harness-... crystallized-backfill = "measure-then-fill"
|
||||
# Adopt: AI_INFRA broadcast F4 + session-cmd-budget-display Upgrade-1 (SE adapt 07-06).
|
||||
#
|
||||
# WHAT THIS IS
|
||||
# A PLANNER (DRY, NO-API) that answers ONE question for the LEAD hot-feed:
|
||||
# "After the always-loaded hot-load sources are in Tier-1, how much REAL-TOKEN
|
||||
# headroom is left under the live cap, and how many tokens of crystallized
|
||||
# (value-gated) backfill may we pull in WITHOUT over-stuffing?"
|
||||
# It MEASURES (byte-count the live hot-load files) then computes an EXPECTED
|
||||
# backfill under a hard precondition. It does NOT pour anything (no memory write,
|
||||
# no context change) - exactly like memory-archive-gate.ps1 it is a DRY planner.
|
||||
#
|
||||
# NON-NEGOTIABLES
|
||||
# (1) OWNER-READ-ONLY : every budget number (lead cap + backfill target) is the
|
||||
# OWNER's (chu-du-an) to set in memory-budget.json. This
|
||||
# script ONLY reads them LIVE. It NEVER computes, writes,
|
||||
# auto-tunes, or persists a budget number. (role_boundary_note)
|
||||
# (2) DEFAULT 0 = OFF : if the optional crystallized_backfill block is ABSENT (or
|
||||
# target<=0) the expected backfill is 0 (TAT / turned-off).
|
||||
# Backfill only "opens" when owner sets target>0 AND measured
|
||||
# headroom>0 (precondition). Standalone default = OFF.
|
||||
# (3) NOT-FOR-SUB : this measures the LEAD hot-feed only. Sub packs are
|
||||
# hand-distilled by em-main per task; blind-pouring backfill
|
||||
# into a sub dilutes relevance. If invoked -Tier sub => WARN.
|
||||
# (4) CALIBRATED RANGE: token headroom is a RANGE [bytes/4 .. bytes/3.0], NOT the
|
||||
# false-precise byte/4. VN-diacritic hot-memory ~3.0-3.5
|
||||
# byte/tok, so bytes/3.0 = worst-case (fewest tokens hidden
|
||||
# behind the bytes). measured_headroom uses the worst case.
|
||||
# (5) NO-API : Select-String + byte/file measure ONLY. NEVER calls a model.
|
||||
# (6) PS 5.1 / ASCII : script BODY is ASCII-only (gotcha #30); any glyph via
|
||||
# [char]0xXXXX code-point; target files read -Encoding UTF8.
|
||||
# (7) exit 0 ALWAYS : measure-and-report, NOT a build-gate.
|
||||
#
|
||||
# Usage:
|
||||
# powershell.exe -ExecutionPolicy Bypass -File scripts\crystallized-backfill.ps1
|
||||
# powershell.exe -ExecutionPolicy Bypass -File scripts\crystallized-backfill.ps1 -Json
|
||||
#
|
||||
# See [TAILOR] block at the bottom for the honest simplifications vs the maximal spec.
|
||||
|
||||
param(
|
||||
[string]$RepoRoot = "$PSScriptRoot\..",
|
||||
[switch]$Json = $false,
|
||||
[string]$Tier = "lead" # NOT-for-sub guard: anything but 'lead' => WARN (still measures lead)
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# ---- glyphs as code-points (ASCII body, gotcha #30) -----------------------
|
||||
$WARN = [char]0x26A0 # warning sign
|
||||
$ARROW = [char]0x2192 # rightwards arrow (source-order display)
|
||||
|
||||
# ---- resolve paths --------------------------------------------------------
|
||||
$RepoRoot = (Resolve-Path $RepoRoot).Path
|
||||
$memRoot = Join-Path $RepoRoot '.claude\agent-memory'
|
||||
$budgetPath = Join-Path $memRoot 'memory-budget.json'
|
||||
|
||||
if (-not (Test-Path $budgetPath)) { Write-Error "memory-budget.json not found: $budgetPath"; exit 1 }
|
||||
|
||||
# ---- load LIVE budget -----------------------------------------------------
|
||||
$budget = Get-Content $budgetPath -Raw | ConvertFrom-Json
|
||||
|
||||
# (a) lead cap : LIVE-read, never hardcode (owner authority).
|
||||
$tg = $budget.token_governor
|
||||
if ($null -eq $tg -or $null -eq $tg.tier1_hotfeed_tokens) {
|
||||
Write-Error "memory-budget.json missing token_governor.tier1_hotfeed_tokens (lead cap source)"; exit 1
|
||||
}
|
||||
$leadCap = [int]$tg.tier1_hotfeed_tokens.lead_tokens # LIVE. owner-set. read-only here.
|
||||
|
||||
# (b) crystallized_backfill : OPTIONAL block. Absent => SAFE DEFAULT (target=0=OFF,
|
||||
# default hotload list). em-main adds the config block later; script runs
|
||||
# standalone with defaults now.
|
||||
$cb = $budget.crystallized_backfill
|
||||
$defaultHotload = @(
|
||||
'docs/STATUS.md',
|
||||
'docs/HANDOFF.md',
|
||||
'.claude/governance/ACTIVE-MARKS.md',
|
||||
'docs/changelog/migration-todos.md'
|
||||
)
|
||||
if ($null -eq $cb) {
|
||||
$target = 0 # DEFAULT OFF (TAT)
|
||||
$hotloadSources = $defaultHotload
|
||||
$cfgState = "ABSENT (safe default: target=0=OFF, default hotload list)"
|
||||
} else {
|
||||
# target : owner number. Read LIVE; default 0 if the field is missing.
|
||||
if ($null -ne $cb.target) { $target = [int]$cb.target } else { $target = 0 }
|
||||
# hotload_sources : owner list; fall back to default if missing/empty.
|
||||
$hotloadSources = @()
|
||||
if ($cb.hotload_sources) { foreach ($h in $cb.hotload_sources) { if ($h) { $hotloadSources += [string]$h } } }
|
||||
if ($hotloadSources.Count -eq 0) { $hotloadSources = $defaultHotload }
|
||||
$cfgState = "present (target + hotload_sources read LIVE)"
|
||||
}
|
||||
|
||||
# ---- NOT-FOR-SUB guard (invariant 3) --------------------------------------
|
||||
$subWarn = $false
|
||||
if ($Tier -ne 'lead') {
|
||||
$subWarn = $true
|
||||
}
|
||||
|
||||
# ==========================================================================
|
||||
# MEASURE : byte-count each LIVE hot-load source (UTF8), sum, derive RANGE.
|
||||
# ==========================================================================
|
||||
$srcRows = @()
|
||||
$totalBytes = 0
|
||||
$utf8 = New-Object System.Text.UTF8Encoding($false)
|
||||
foreach ($rel in $hotloadSources) {
|
||||
$full = Join-Path $RepoRoot ($rel -replace '/', '\')
|
||||
if (Test-Path $full) {
|
||||
# measure via UTF8 decode-length-in-bytes = on-disk byte size (explicit UTF8, gotcha #30)
|
||||
$b = ([System.IO.File]::ReadAllBytes($full)).Length
|
||||
$totalBytes += $b
|
||||
$srcRows += [pscustomobject]@{ src = $rel; bytes = $b; state = 'ok' }
|
||||
} else {
|
||||
$srcRows += [pscustomobject]@{ src = $rel; bytes = 0; state = 'MISSING' }
|
||||
}
|
||||
}
|
||||
|
||||
# calibrated RANGE (invariant 4): tok_low = optimistic (bytes/4), tok_high = worst (bytes/3.0)
|
||||
$tokLow = [int][math]::Round($totalBytes / 4.0)
|
||||
$tokHigh = [int][math]::Round($totalBytes / 3.0)
|
||||
|
||||
# measured_headroom = cap - worst-case-load (can be NEGATIVE = hot-feed near full)
|
||||
$measuredHeadroom = $leadCap - $tokHigh
|
||||
|
||||
# ---- expected_backfill (invariant 2 + precondition) -----------------------
|
||||
# precondition: only OPEN backfill when headroom>0 AND target>0. Else 0.
|
||||
if ($target -gt 0 -and $measuredHeadroom -gt 0) {
|
||||
$expectedBackfill = [math]::Min($target, $measuredHeadroom)
|
||||
$preconditionMet = $true
|
||||
} else {
|
||||
$expectedBackfill = 0
|
||||
$preconditionMet = $false
|
||||
}
|
||||
|
||||
# reason string (why backfill is what it is)
|
||||
if ($target -le 0) {
|
||||
$backReason = "target=0 => OFF (TAT, default until owner sets target>0)"
|
||||
} elseif ($measuredHeadroom -le 0) {
|
||||
$backReason = "measured_headroom<=0 => NO room (hot-feed at/over worst-case cap)"
|
||||
} elseif ($expectedBackfill -eq $target) {
|
||||
$backReason = "min(target,headroom)=target => full target fits under headroom"
|
||||
} else {
|
||||
$backReason = "min(target,headroom)=headroom => headroom binds (target exceeds room)"
|
||||
}
|
||||
|
||||
# ==========================================================================
|
||||
# -Json : one machine-readable line, then exit. (numbers all live-derived)
|
||||
# ==========================================================================
|
||||
if ($Json) {
|
||||
$out = [ordered]@{
|
||||
cap = $leadCap
|
||||
bytes = $totalBytes
|
||||
tok_low = $tokLow
|
||||
tok_high = $tokHigh
|
||||
headroom = $measuredHeadroom
|
||||
target = $target
|
||||
backfill = $expectedBackfill
|
||||
}
|
||||
Write-Output (($out | ConvertTo-Json -Compress))
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ==========================================================================
|
||||
# HUMAN REPORT
|
||||
# ==========================================================================
|
||||
$dash = [string]([char]45) # '-' as a value, never a bare '--' run (PS 5.1 decrement-op trap)
|
||||
|
||||
Write-Output "============================================================"
|
||||
Write-Output " crystallized-backfill.ps1 - measure-then-fill (DRY planner)"
|
||||
Write-Output " tier : $Tier"
|
||||
Write-Output " budget.json : $budgetPath"
|
||||
Write-Output " cb config : $cfgState"
|
||||
Write-Output "============================================================"
|
||||
|
||||
if ($subWarn) {
|
||||
Write-Output ""
|
||||
Write-Output (" {0} NOT-FOR-SUB: -Tier '$Tier' requested. This planner measures the LEAD" -f $WARN)
|
||||
Write-Output " hot-feed ONLY. Sub packs are hand-distilled by em-main per task; blind"
|
||||
Write-Output " backfill into a sub dilutes relevance. Reporting LEAD numbers regardless."
|
||||
}
|
||||
|
||||
# ---- source measure table -------------------------------------------------
|
||||
Write-Output ""
|
||||
Write-Output "### hot-load sources (LIVE byte-measure, UTF8)"
|
||||
Write-Output ""
|
||||
Write-Output ("{0,-44} {1,10} {2,8}" -f 'source', 'bytes', 'state')
|
||||
Write-Output ("{0,-44} {1,10} {2,8}" -f ($dash*44), ($dash*10), ($dash*8))
|
||||
foreach ($r in $srcRows) {
|
||||
Write-Output ("{0,-44} {1,10} {2,8}" -f $r.src, $r.bytes, $r.state)
|
||||
}
|
||||
Write-Output ("{0,-44} {1,10} {2,8}" -f 'TOTAL', $totalBytes, '')
|
||||
|
||||
# ---- the budget number table (Upgrade-1) ----------------------------------
|
||||
# ALL numbers read-live; ZERO hardcoded budget values.
|
||||
Write-Output ""
|
||||
Write-Output "### budget numbers (Upgrade-1) - ALL live-read, 0 hardcoded"
|
||||
Write-Output ""
|
||||
$hr = if ($measuredHeadroom -ge 0) { "$measuredHeadroom" } else { "$measuredHeadroom (NEG=near-full)" }
|
||||
Write-Output ("{0,-22} : {1}" -f 'Tran tong (live cap)', "$leadCap tok")
|
||||
Write-Output ("{0,-22} : {1}" -f 'hotload measured', "$totalBytes bytes")
|
||||
Write-Output ("{0,-22} : {1}" -f 'tok RANGE [low..high]', "[$tokLow .. $tokHigh] tok (bytes/4 .. bytes/3.0)")
|
||||
Write-Output ("{0,-22} : {1}" -f 'measured_headroom', "$hr tok (cap - tok_high, worst-case)")
|
||||
Write-Output ("{0,-22} : {1}" -f 'target (owner)', "$target tok")
|
||||
Write-Output ("{0,-22} : {1}" -f 'expected_backfill', "$expectedBackfill tok ($backReason)")
|
||||
|
||||
# ---- value-gated source-order plan (invariant, inert while target=0) ------
|
||||
# When backfill>0 we would PULL crystallized content in this priority order,
|
||||
# dedup vs the hot-load already counted. We NEVER actually pour it here (DRY,
|
||||
# like archive-gate) - we only PRINT the order the fill WOULD follow.
|
||||
Write-Output ""
|
||||
Write-Output "### value-gated backfill source-order (planner / DRY - no pour)"
|
||||
Write-Output ""
|
||||
if ($preconditionMet) {
|
||||
Write-Output (" precondition MET (target>0 AND headroom>0): would pull up to $expectedBackfill tok in order:")
|
||||
Write-Output (" 1. gist {0} archive/*.gist.md (4-field distilled)" -f $ARROW)
|
||||
Write-Output (" 2. value-marked {0} archive lines carrying gotcha# / anti-pattern / root-cause" -f $ARROW)
|
||||
Write-Output (" 3. curated RAG {0} search_memory high-value hits" -f $ARROW)
|
||||
Write-Output " (each stage DEDUP vs the hot-load sources above; stop at expected_backfill)"
|
||||
} else {
|
||||
Write-Output (" precondition NOT met -> source-order INERT (no pull). Opens only when target>0 AND headroom>0.")
|
||||
Write-Output (" order (for reference): 1.gist {0} 2.value-marked-archive {0} 3.curated-RAG (dedup vs hot-load)" -f $ARROW)
|
||||
}
|
||||
|
||||
# ---- honest caveat (invariant, always printed) ----------------------------
|
||||
Write-Output ""
|
||||
Write-Output "------------------------------------------------------------"
|
||||
Write-Output " HONEST CAVEAT"
|
||||
Write-Output " char/4 is NOT a real token count; VN-diacritic hot-memory ~3.0-3.5 byte/tok."
|
||||
Write-Output " Headroom is a RANGE [bytes/4 .. bytes/3.0]; this planner uses bytes/3.0 (worst)."
|
||||
Write-Output " backfill defaults to 0 = TAT until the OWNER sets target>0 AND measured headroom>0."
|
||||
Write-Output " Every budget number is the OWNER's authority; this script is READ-ONLY (never tunes/writes)."
|
||||
Write-Output " HEADROOM is measured over a STABLE FILE-FLOOR (the configured hotload_sources) = an UPPER BOUND."
|
||||
Write-Output " The PEAK hot-feed also loads session-variable task-context NOT in this list; real headroom is SMALLER."
|
||||
Write-Output "------------------------------------------------------------"
|
||||
|
||||
# measure-and-report; NEVER a build-gate.
|
||||
exit 0
|
||||
|
||||
# ==========================================================================
|
||||
# [TAILOR] honest simplifications vs the maximal spec:
|
||||
# * headroom uses the WORST-CASE end of the RANGE (bytes/3.0 => fewest tokens
|
||||
# hidden behind the bytes => smallest headroom => most conservative "open"
|
||||
# decision). The optimistic bytes/4 is printed for context only.
|
||||
# * byte-measure = [IO.File]::ReadAllBytes(...).Length (true on-disk size);
|
||||
# NOT Get-Content length (which mangles VN/em-dash under PS 5.1 ANSI, gotcha
|
||||
# #30) and NOT a re-encode estimate. Exact bytes, honest tokens-as-range.
|
||||
# * the value-gated source-order is a PRINTED PLAN only (gist -> value-marked
|
||||
# archive -> curated RAG, dedup vs hot-load). It performs NO pull and changes
|
||||
# NO context - identical DRY-planner stance to memory-archive-gate.ps1.
|
||||
# * crystallized_backfill block is OPTIONAL: absent => target=0 (OFF) + the 4
|
||||
# default hotload sources, so the script is fully standalone before em-main
|
||||
# adds the config. Present => target + hotload_sources are read LIVE.
|
||||
# * NOT-FOR-SUB is a WARN (not a hard-exit): -Tier sub still prints LEAD numbers
|
||||
# but flags that sub-packs are hand-distilled, never blind-backfilled.
|
||||
# ==========================================================================
|
||||
Reference in New Issue
Block a user