solution-erp/.claude/agent-memory/cicd-monitor/MEMORY.md

CI/CD Monitor Agent — Persistent Memory

Persistent diary cross-session. Auto-injected first ~200 lines at spawn (L1 HOT). Update BEFORE every stop. Tiered Memory v1: L1 HOT soft-cap ~30KB · L2 archive/ on-demand · L3 RAG search_memory just-in-time. Keep entry ≤ 1.5K chars (gotcha #53). Full verbatim run history pre-S40 → git d2f52ba + archive/2026-05-{runs,q2,q3,q4}.md + archive/2026-06.md. 🗺️ Lookup map (Harness-9 S70): archive/_INDEX.md — 1 dòng/bản-ghi + con-trỏ substring (sha-keyed, Ctrl-F fallback); đọc verbatim + 2026-0{5,6}.gist.md (nén 4-field) theo nhu cầu.

---

🎯 Role baseline

Read-only CI/CD + post-deploy verifier SOLUTION_ERP. Polls Gitea Actions API, verifies test gate + deploy ship + prod health. Tools: Read, Grep, Glob, Bash, WebFetch + 5 RAG MCP. Output: PASS/FAIL + evidence <500 words. Skills: iis-deploy-runbook + dependency-audit-erp + ef-core-migration. Spawn ~150K — trade-off catch fail tự động.

---

🚨 Recurring CI/CD bug patterns (catch priority)

• #39 act_runner github.com TCP timeout — run hang "Set up job" 21s. Log dial tcp github.com:443 i/o timeout. Fix: manual checkout bypass hardcoded .gitea/workflows/deploy.yml (pass #110). KHÔNG revert.
• #40 npm cache tsc not found — build_fe_admin fail post cache: npm. DISABLED rolled back a21790d. KHÔNG re-enable.
• #41 paths-ignore docs-only skip — code commit không trigger CI? Check git diff --name-only HEAD~1 HEAD vs paths-ignore: ['docs/**','**/*.md','.claude/skills/**']. Discovery #3: Gitea evaluates push range commits — nếu ≥1 commit có non-ignored file → toàn range build (BENEFICIAL).
• #25 IIS WebSocket — notification-hub/negotiate 401/404 prod. Fix: WebSocket module enable web.config site api (skill iis-deploy-runbook).
• #48 SQLite tie-break — OrderByDescending(CreatedAt).First() pick wrong khi 2+ .Add() cùng frozen-clock. Fix: discriminator filter .Where(Summary.Contains("Chuyển phase")) BEFORE OrderBy.
• Bundle hash unchanged = ship FAIL — push+action success nhưng prod không đổi. Verify via INDEX.HTML ref (curl -s https://admin.solutions.com.vn/ | grep -oE '/assets/index-[A-Za-z0-9_-]+\.(js|css)'), NOT by GETting a hash-named asset directly. Fix: SSH Restart-WebAppPool. ⚠️ Bundle hash verify MUST sau status=success (Run #242 false-positive lesson: check khi "running" → stale hash).
  • 🔴 #69 (S72 Run #312 OVERTURNS prior invariant) — FE bundle hash is NON-DETERMINISTIC per rebuild. deploy.yml Remove-Item fe-{admin,user}\* -Exclude web.config + Copy-Item dist\* runs UNCONDITIONALLY every run (path-filter gates whole-workflow trigger, NOT per-step). Identical FE source ⟹ DIFFERENT hash each deploy (Vite/rolldown non-reproducible chunk-id). PROVEN: governance-only 18fced6 (0 files in fe-*/src) rotated BOTH bundles admin BgNCjwsG→fc_xkNpJ (+717b) + user CBvh0vtf→DP-tBcg0, index.html Last-Modified matched deploy window. ⟹ "BE-only/governance ⟹ bundle MUST stay frozen" is FALSE; rotation is EXPECTED on every deploy. Bundle-rotate alone is NOT proof of FE content change — to detect real FE change, diff fe-*/src in commit/range, NOT hash delta.
  • ⚠️ SPA-fallback 200 trap (S72): server rewrites /*→/index.html so GET /assets/index-<ANYTHING>.js returns 200 even for non-existent/fake hash (control ZZdoesnotexist0.js→200 confirmed). Old-hash-still-200 is MEANINGLESS as persistence signal. RELIABLE bundle signal = parse index.html <script src>/<link href>, then GET that exact path + check size_download large (not white-screen) + Last-Modified in deploy window.
• Migration drift prod vs repo — compare ls .../Persistence/Migrations/*.cs vs sqlcmd __EFMigrationsHistory. Fix: check Program.cs app.MigrateDatabase() + app pool recycle.

---

📋 5-stage checklist (EVERY run)

• Stage 0 RAG infra: Get-Service Qdrant Running + http://localhost:6333/healthz. Collection proj_solution_erp (prefix proj_* 7 project — Discovery #8).
• Stage 1 Push+filter: git log -1 --format='%H %s' + git log origin/main..HEAD empty + diff vs paths-ignore (docs-only → SKIPPED-DOCS return).
• Stage 2 Gitea poll (max 10 iter × 60s): API .../actions/tasks?limit=5 (NOT /runs 404). Match head_sha. ⚠️ task table updated_at stale ~2min (gotcha #46) → cross-check VPS mtime.
• Stage 3 Test gate: baseline 130 PASS (58 Domain + 72 Infra). Phase 9 UAT exception lower OK (feedback_uat_skip_verify).
• Stage 4 Post-deploy (if SUCCESS): auth login bearer (admin + nv.test gotcha #44; token=accessToken route /api/auth/login) → 3-5 endpoint smoke 2XX (incl new) → FE bundle hash 2 app changed → SignalR negotiate (gotcha #25 if relevant) → EF mig prod==repo.
  • Stage 4.6 (S29 CRITICAL): sqlcmd seed sample verify post-deploy (NOT chỉ schema). sqlcmd -Q "SELECT Code FROM ApprovalWorkflows WHERE Code LIKE 'QT-%-V2-%'" → 0 rows = seed GATE BLOCKED → gotcha #51.
  • Discovery #4: ASP.NET 10 record enum cần numeric input unless JsonStringEnumConverter (SOL has NO converter → FE sends numeric). #5: sqlcmd ssh Windows-auth cần \\\\SQLEXPRESS 4-backslash. #6: INFRASTRUCTURE seed (Roles/Depts/Catalogs/MenuTree/AdminPerms/Templates/SampleWorkflowsV2) MUST run, NOT inside if(!demoSeedDisabled); DEMO seed (DemoUsers/Contracts/PE) OK gated → gotcha #51.
• Stage 5 Report PASS/FAIL + evidence + MEMORY update.

---

⚠️ Anti-patterns (DO NOT)

1. ❌ Push fix code — READ only, escalate em main · 2. ❌ Speculate fail without log · 3. ❌ Skip post-deploy bundle hash (biggest catch) · 4. ❌ Skip MEMORY · 5. ❌ Poll forever (max 10 iter) · 6. ❌ Auto-rollback (escalate + recommend) · 7. ❌ Verify docs-only (SKIPPED-DOCS return ngay)

---

🧠 SOLUTION_ERP CI/CD essentials (S40 verified)

• Gitea: git.baocaogiaoduc.vn/vietreport-admin/solution-erp · workflow .gitea/workflows/deploy.yml · paths-ignore ['docs/**','**/*.md','.claude/skills/**']
• Prod: api/admin/eoffice .solutions.com.vn · SSH ssh vietreport-vps (Administrator, id_ed25519) · IIS site phys paths (S42 verified): API C:\inetpub\solution-erp\api · admin \fe-admin · user \fe-user (3 sites Started). DB .\SQLEXPRESS/SolutionErp/vrapp SQL-auth. Conn string key = ConnectionStrings.Default (NOT DefaultConnection!) — read pw from prod appsettings.Production.json when $env:PROD_DB_PASSWORD empty.
  • SSH→PS quoting (S42 lesson): nested bash→ssh→powershell mangles $var/\". Use iconv UTF-16LE | base64 → powershell -EncodedCommand $B64. Single-quote literal paths.
• Tests baseline: 263 PASS (S62 Run #286 sha 7926c21 spec; 45 Domain + 218 Infra — em-main supplied; supersedes prev 228/240/256). CI gate runs both test projects BEFORE build/deploy → status=success ⟹ test gate passed (tasks endpoint reports terminal as status:success, conclusion field NOT populated). Local grep undercounts (Theory/InlineData) — trust CI conclusion. Phase 9 UAT mode skip per chunk OK.
• Mig latest repo: Mig 53 20260617060207_AddPeUrgentAndCeoApprovalThreshold (S71 Run#308; PE +IsUrgentByPro/+IsUrgentByCcm bit-default-0 + ApprovalWorkflows +CeoApprovalThreshold decimal(18,2)-nullable, 3 AddColumn no new table — VERIFIED APPLIED PROD). Prev Mig 52 AddHoSoLinkToPurchaseEvaluation (S65 PE HoSoLink hyperlink-NAS) + Mig 51 AddDepartmentParentId (S65 org-tree loose-Guid) + Mig 50 ReplaceBudgetModuleWithPeWorkItemBudgets (S61 net-reduce). Path src/Backend/SolutionErp.Infrastructure/Persistence/Migrations/ (53 mig .cs non-designer total). Prod check sqlcmd __EFMigrationsHistory ORDER BY MigrationId DESC TOP 5. ⚠️ Table-count: sys.tables (is_ms_shipped=0, excl mighist) = 88 (S62 Run #286 verified — S61 Budget-replace DROPPED tables 93→88). Narrative-93 is STALE pre-S61 — when commit touches no schema, 88 is correct, don't FAIL on 88↔93. Always cross-ref COMMIT scope vs ambient count.
• Bearer: admin admin@solutions.com.vn/Admin@123456 (full) · UAT nv.test@solutions.com.vn/TestUser@123456 (Drafter CCM, gotcha #44 check)
• Bundle hash live S72: admin fc_xkNpJ + css C2Zvnd2f · user DP-tBcg0 + css 1bS4xeUF (Run #312 sha 18fced6; index.html Last-Modified 2026-06-18T07:07:49Z = deploy window). Prev S71 #308: admin BgNCjwsG/user CBvh0vtf. ⚠️ PER #69 (S72): bundle hash ROTATES on EVERY deploy regardless of FE change (non-deterministic rebuild). So "live hash" value is only a SNAPSHOT for THIS run — do NOT treat it as a frozen-baseline to compare next run against. To detect a real FE ship, diff fe-*/src in commit/range — NOT hash delta. ASYMMETRIC-deploy "anomaly" framing (S66) is RETIRED by #69: both apps rebuilt+rotated every deploy, asymmetry impossible to read from hashes alone. S50 mid-deploy transient lesson still holds: re-confirm hash AFTER status=success ALWAYS (anti-pattern #3).
• DB pw (S42, when $PROD_DB_PASSWORD empty): vrapp/buKL3TGBkD0wDDbYVw65QeX9 read from C:\inetpub\solution-erp\api\appsettings.Production.json→ConnectionStrings.Default. ⚠️ Skill-doc path C:\inetpub\apps\SolutionErp\Api is STALE → real path C:\inetpub\solution-erp\api. sqlcmd over SSH works direct (no UTF-16 encode needed). ⚠️ sys-catalog string-concat queries hit collation conflict (Latin1_General_CI_AS_KS_WS vs SQL_Latin1_General_CP1_CI_AS) → add COLLATE DATABASE_DEFAULT per concatenated column.

🔑 Critical config (flag commit nếu tái xuất)

Node CI 20.x (feedback_node_cicd) · MediatR 12.4.1 (gotcha #1, flag Version="14) · Swashbuckle 6.9.0 (gotcha #2) · act_runner manual checkout (#39) · npm cache DISABLED (#40, flag cache: npm)

---

🎯 Per-NV admin opt-in wire — 10-point checklist (cumulative S22→S23)

Cross-ref feedback_per_nv_permission_scope. Per-NV/per-Level refactor MUST verify: 1 Domain field · 2 EF HasDefaultValue(false) · 3 Mig 3-file · 4 Service read · 5 Domain+App DTO mirror · 6 Designer FE checkbox · 7 AwLevelDto+ToDto · 8 CreateAwLevelInput+Update mutation · 9 Lookup discrimination (FirstOrDefault ADD ApproverUserId==actorId + admin fallback) · 10 Controller body record count == Command record count. Bug latency 2-3 days prod silent khi miss 9-10. Scan grep -n "FirstOrDefault.*Order.*==" *.cs after OR-of-N refactor.

📊 Run stats baseline

BE (test+build) ~90s · FE × 2 ~60s/app · deploy ~30s · total ~3min code / 0s docs-only. >5min → escalate.

---

📅 Recent runs (FIFO — older → archive/git)

• 2026-06-18 S72 Run #312 (run_number 312) sha=18fced6 PASS ~5m16s (GOVERNANCE-ONLY discipline-check: 26 .md + .claude/workflows/hmw.js + .claude/agent-memory/memory-budget.json. Harness-10 flat-refine adoption. ZERO production BE/FE/Mig code. Expected = true no-op deploy. deploy 13/13 session after #297–#308 all PASS): Push origin/main = HEAD 18fced6 (nothing unpushed). git diff --name-only 18fced6~1 18fced6 = 27 files; non-path-ignored = ONLY hmw.js + memory-budget.json (both tooling/governance, NOT prod code) → CI TRIGGERED (correct — .js/.json not in paths-ignore ['docs/**','**/*.md','.claude/skills/**']). Migration files in diff = ZERO. GITEA_TOKEN absent in PS-scope → anon Gitea API (worked, public repo). PROD_DB_PW empty → DB pw from prod appsettings.Production.json→ConnectionStrings.Default (vrapp/buKL3TGBkD0wDDbYVw65QeX9 len24, path C:\inetpub\solution-erp\api). Run IN-PROGRESS at spawn (running 14:04:11→14:06:24) — exact head_sha match RUN#312 (deterministic, not gotcha-#46 stale). Poll-loop iter4 status=success (created 14:04:11 → success 14:09:27 ≈5m16s). CI gate (both proj pre-build ⟹ status=success ⟹ test 306 baseline (45D+261I) passed; conclusion empty — tasks terminal=status:success doesn't populate conclusion, trust success; 306 INFERRED gate-passes-pre-build invariant NOT log-confirmed numeric). Smoke 4×200: api /health/ready+/health/live + admin root + eoffice root (both pre- AND post-success). 🔑★ MIG VERIFIED NO-CHANGE (sqlcmd over SSH ground-truth): prod __EFMigrationsHistory top5 = AddPeUrgentAndCeoApprovalThreshold(Mig53)→AddHoSoLinkToPurchaseEvaluation(52)→AddDepartmentParentId(51)→ReplaceBudgetModuleWithPeWorkItemBudgets(50)→AddWorkItemToPurchaseEvaluation(49) == repo HEAD, NO new mig ✓. sys.tables=88 (excl mighist) UNCHANGED ✓. ⟹ DB side = perfect no-op as expected for governance commit. ⚠️★ BUNDLE SURPRISE → RESOLVED BENIGN (NEW gotcha #69): expected FROZEN admin BgNCjwsG/user CBvh0vtf, but AFTER status=success BOTH ROTATED admin BgNCjwsG→fc_xkNpJ (+717b, 1,593,048b) + user CBvh0vtf→DP-tBcg0 (1,497,701b), stable on 2nd-fetch (not mid-deploy transient). FORENSIC: commits ebd7e1c..18fced6 (9 commits, all [CLAUDE] Docs:/Workflow:) touched 0 files in fe-{admin,user}/src/ + 0 in package.json/lock/vite/tsconfig (verified per-file). ROOT CAUSE: deploy.yml lines 161-167 Remove-Item fe-*\* -Exclude web.config + Copy-Item dist\* runs UNCONDITIONALLY every run (path-filter gates WHOLE-workflow trigger, not per-step) → FE rebuilt every deploy → Vite/rolldown emits NON-DETERMINISTIC hash from identical source. PROOF index.html Last-Modified 07:07:49Z(14:07VN) = inside deploy window → physically rewritten this run. CONTROL TEST: fake /assets/index-ZZdoesnotexist0.js → 200 (SPA /*→index.html fallback) ⟹ "old-hash-still-200" is MEANINGLESS, reliable signal = parse index.html refs (admin fc_xkNpJ.js+C2Zvnd2f.css · user DP-tBcg0.js+1bS4xeUF.css). ⟹ rotation is NOT an accidental FE change — it's the pipeline's normal non-deterministic-rebuild behavior; OVERTURNS prior "BE-only/governance ⟹ bundle frozen" invariant used in every prior verdict (held by coincidence, not mechanism). Assets reachable 200 + large (not white-screen) ⟹ FE healthy. 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only; FE re-copy = expected unconditional deploy side-effect, governance content unchanged). VERDICT PASS (true no-op for the COMMIT's intent): green CI + test 306 + Mig 53 unchanged + tables 88 + health 4×200; bundle rotation flagged-then-cleared as benign pipeline mechanism (#69), NOT accidental ship — confirmed by 0 fe-src diff. LESSON: governance/BE-only verify must NOT FAIL on bundle-rotate; reframe check = (a) Mig+tables unchanged ground-truth, (b) health 200, (c) FE-content-change detection via git diff fe-*/src NOT hash, (d) bundle hash useful only to confirm a DELIBERATE FE deploy DID ship (rotate present), never to assert FE DIDN'T change (rotate is unconditional). TOOLING: bash→$LOCALAPPDATA/Temp/x.ps1 + powershell.exe -File $(cygpath -w ...); Gitea tasks via Invoke-RestMethod with -Body @{limit=10} hashtable (string-interp ?limit=N into encoded path → 'hostname could not be parsed' URI error — use -Body not query-string); poll-loop backgrounded + Monitor grep-until FINAL; SSH→PS base64 -EncodedCommand UTF-16LE for appsettings-read AND sqlcmd; sqlcmd pw via PS here-string $P= + [char]39 for literal object-name quotes; -W -h -1 -s '|'; read DB pw from prod appsettings when $PROD_DB_PASSWORD empty. NEVER fixed code (READ-only). Tag [s72, run312, pass, governance-only-no-prod-code, hmw-js+memory-budget-json-triggers-ci, mig53-unchanged-VERIFIED-prod, tables88-unchanged, health-4x200, test306-inferred, BUNDLE-BOTH-ROTATE-fc_xkNpJ-DP-tBcg0-but-BENIGN, NEW-gotcha-69-fe-hash-nondeterministic-per-rebuild-deploy-unconditional, spa-fallback-200-trap-control-test, frozen-invariant-OVERTURNED, 0-fe-src-diff-confirms-no-accidental-ship, deploy13of13, no-regression]. (id422) sha=ebd7e1c PASS ~4m41s (FULL-STACK: BE Mig 53 + ApproveV2 CCM-finalize-by-threshold + endpoint PUT /urgent + FE 2-app PE cờ-gấp UI + designer ngưỡng. PE "cờ gấp PRO/CCM + CCM duyệt-final theo ngưỡng giá trị" anh Kiệt FDC. 27 files: BE 9 {Controller +SSetUrgent, App PurchaseEvaluationUrgentFeatures.cs NEW, PurchaseEvaluationFeatures, CreateContractFromEvaluation, AwV2AdminFeatures, Dtos, Domain PE+AW, Config} + 3 Mig-file + Service PurchaseEvaluationWorkflowService + FE 6 {PeDetailTabs+ListPage+types ×2-app + ApprovalWorkflowsV2Page admin-only} + 2 test {PeUrgentToggleAuthzTests, PeCcmThresholdFinalizeTests} + 3 agent-memory .md. deploy 12/12 session after #297–#307 all PASS):** Push parent 1f8947e..ebd7e1c (1 commit). git diff --name-only 1f8947e ebd7e1c -- '*Persistence/Migrations*' = 3 files (Mig 53 .cs/.Designer.cs/ModelSnapshot.cs) — REAL EF migration this time (contrast S69b/S70 which had ZERO). .cs/.tsx non-ignored → full pipeline RAN. GITEA_TOKEN present (PS-scope) → authed Gitea API; PROD_DB_PW empty → DB pw từ prod appsettings.Production.json→ConnectionStrings.Default (vrapp/buKL3TGBkD0wDDbYVw65QeX9, path C:\inetpub\solution-erp\api). Run IN-PROGRESS at spawn (running 13:27:58, 1st-seen updated 13:28:54) — pre-deploy NOT snapshotted as baseline (anti#3: prev-live = #306/#307 spec admin Wt54PHYl/user B99fMU6X). Poll-loop iter6 status=success (created 13:27:58 → success-update 13:32:39 ≈4m41s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test 306 baseline (45D+261I; +14 PeUrgentToggleAuthz+PeCcmThresholdFinalize) passed; conclusion empty — tasks terminal=status:success không populate conclusion, trust success; 306 INFERRED gate-passes-pre-build invariant NOT log-confirmed numeric). ★ BUNDLE BOTH ROTATE (FE-both-app PE-UI ⟹ both MUST rotate; verified AFTER status=success + re-confirm STABLE 2nd-fetch identical no-transient — anti#3): admin ROTATE Wt54PHYl→BgNCjwsG ✓ + user ROTATE B99fMU6X→CBvh0vtf ✓. Asset reachable 200: admin js 1,592,331b + user js 1,496,984b (not white). Smoke 4×200 health: api /health/ready+/health/live + admin root + eoffice root. ★ ENDPOINT /urgent PROBE: PUT /api/purchase-evaluations/{guid}/urgent unauth → 401 (NOT 404) ✓ — route EXISTS + class-level [Authorize] any-auth enforced (controller [HttpPut("{id:guid}/urgent")] SetUrgent → SetPurchaseEvaluationUrgentCommand; behavioral PRO/CCM-only authz = unit-test PeUrgentToggleAuthzTests, not probed). PE list unauth → 401 ✓ (reachable, gated). 🔑★ MIG 53 VERIFIED APPLIED PROD (sqlcmd over SSH, ground-truth not inference): prod __EFMigrationsHistory top5 = 20260617060207_AddPeUrgentAndCeoApprovalThreshold(53)→AddHoSoLinkToPurchaseEvaluation(52)→AddDepartmentParentId(51)→ReplaceBudgetModuleWithPeWorkItemBudgets(50)→AddWorkItemToPurchaseEvaluation(49) == repo HEAD ✓ (DbInitializer auto-migrate-on-boot ran; app pool recycled w/ new binary). 3 NEW COLUMNS confirmed via sys.columns: PurchaseEvaluations.IsUrgentByCcm=bit is_nullable=0 ✓ + PurchaseEvaluations.IsUrgentByPro=bit is_nullable=0 ✓ + ApprovalWorkflows.CeoApprovalThreshold=decimal is_nullable=1 precision=18 scale=2 ✓ — ALL match Mig 53 Up() spec exactly. sys.tables=88 (excl mighist) UNCHANGED — 3 AddColumn no new table ✓. 0 regression. NO prod-data mutation (read-only curls + sqlcmd SELECT-only; migration-apply = expected boot-side-effect of deploy). Behavioral CCM-finalize-by-threshold + cờ-gấp UI render = anh Kiệt UAT (em confirm Mig applied + columns + endpoint route + bundles shipped per spec). LESSON: full-stack PE commit w/ REAL Mig = verify ALL 4 axes: (1) BE+FE both-bundle ROTATE (FE 2-app), (2) Mig history-top advance + 3 columns sys.columns ground-truth (NOT just "schema looks right" — query each col type/nullable/precision vs Up() spec), (3) sys.tables stays 88 (AddColumn-only ⟹ no table delta; rotate-to-89 would = unexpected CREATE TABLE leak), (4) new endpoint 401-not-404 unauth probe (route-exists proof; authz-logic = unit-test domain not curl). Mig-applied-on-prod proof = __EFMigrationsHistory top == repo-HEAD (DbInitializer MigrateAsync boot-hook) — if top stuck at Mig 52 ⟹ app pool didn't recycle / migrate-on-boot failed ⟹ FAIL even if status=success+bundle-rotated (deploy shipped binary but DB-side incomplete). Distinguishes this run from S69b/S70 (zero-migration BE/FE — those had NO history-advance EXPECTED). TOOLING: Bash=POSIX → write PS to $LOCALAPPDATA/Temp/x.ps1 + powershell.exe -NoProfile -File "C:\Users\pqhuy\AppData\Local\Temp\x.ps1" (⚠️ bash→Windows path auto-convert mangles /tmp/ → use $LOCALAPPDATA/Temp explicit Windows path); Invoke-RestMethod for Gitea tasks (match head_sha -eq sha, poll-loop in PS not bash — bash jq absent/python3 broken); SSH→PS base64 -EncodedCommand (UTF-16LE Unicode) for BOTH appsettings-read AND sqlcmd; sqlcmd pw-literal via PS here-string $P= var inside encoded payload + query-string built w/ [char]39 for embedded single-quotes (object-name literals); -W -h -1 no-header + -s '|' pipe-delim; read DB pw from prod appsettings when $PROD_DB_PASSWORD empty. NEVER fixed code (READ-only). Tag [s71, run308, pass, full-stack-pe-urgent-ccm-threshold, mig53-AddPeUrgentAndCeoApprovalThreshold-VERIFIED-APPLIED-PROD, 3-cols-sys-columns-confirmed-IsUrgentByPro-IsUrgentByCcm-bit0-CeoApprovalThreshold-decimal18-2-null, history-top-advance-53, tables88-unchanged-addcolumn-only, bundle-BOTH-rotate-BgNCjwsG-CBvh0vtf, asset-200-reachable, endpoint-urgent-401-not-404-route-exists, pe-list-401-gated, health-4x200, test306-inferred, deploy12of12, no-regression, behavioral-ccm-finalize-anh-kiet-uat].

• 2026-06-17 S69b Run #307 (run_number 307, id421) sha=1f8947e PASS ~4m33s (BE-ONLY GOLIVE security hướng-ra-prod "Văn phòng số" public Read+Create mọi role — NEW SeedAllRolesOfficeModulePermissionsAsync DbInitializer.cs chạy lúc API boot SAU RevokeTemporarilyHiddenModulesAsync để THẮNG revoke (mirror S65 HRM/Hồ-sơ-NS pattern), allow-list 16 Office key grant CanRead+CanCreate=true upgrade-only (row tồn tại→nâng, chưa có→tạo R+C=true U/D=false, KHÔNG hạ KHÔNG đụng Update/Delete); +6 test OfficeModulePermissionSeedTests.cs (286→292). KHÔNG FE KHÔNG migration. deploy 11/11 session after #297–#306 all PASS): Push c556f6c..1f8947e (1 commit, 5 files: DbInitializer.cs + 1 test + 3 agent-memory .md cicd/reviewer/test-spec). Diff git diff --name-only c556f6c 1f8947e -- '*Migrations*' '*Persistence/Migrations*' = EMPTY ✓ (DbInitializer.cs ở Persistence/ NOT Persistence/Migrations/ — seed runtime-idempotent NOT EF-migration). .cs non-ignored → full pipeline RAN. GITEA_TOKEN+PROD_DB_PW env empty trong bash-scope (env:GITEA_TOKEN có ở PS-scope; PROD_DB_PW thật rỗng) → anon Gitea API + DB pw từ prod appsettings.Production.json→ConnectionStrings.Default (path C:\inetpub\solution-erp\api, uid vrapp len24). Run IN-PROGRESS at spawn (running 10:33:40→10:38) — pre-deploy bundle baseline captured BEFORE poll-loop (anti#3): admin Wt54PHYl + user B99fMU6X — both == S70 #306 spec baseline (still live, deploy not yet shipped). Polled iter5 status=success (created 10:33:40 → success-update 10:38:13 ≈4m33s). CI gate (both proj pre-deploy ⟹ status=success ⟹ test 292 baseline (45D+247I; +6 OfficeModulePermissionSeedTests) passed; conclusion empty — tasks endpoint terminal=status:success không populate conclusion, trust success; 292 INFERRED gate-passes-pre-build invariant NOT log-confirmed numeric). ★ BUNDLE BOTH FROZEN (BE-only ⟹ both MUST stay = pre-deploy live; verified AFTER status=success): admin Wt54PHYl UNCHANGED ✓ + user B99fMU6X UNCHANGED ✓ (no FE touch → no content-hash rotation; rotate-w/o-FE-change = anomaly, did NOT happen). Smoke 4×200: api /health/ready+/health/live + admin root + eoffice root. NO migration — prod __EFMigrationsHistory top5 = AddHoSoLinkToPurchaseEvaluation(Mig52)→AddDepartmentParentId(51)→ReplaceBudgetModuleWithPeWorkItemBudgets(50)→AddWorkItemToPurchaseEvaluation(49)→AddProjectMasterFields(48) == repo HEAD GIỮ NGUYÊN ✓. sys.tables=88 (sqlcmd COUNT excl mighist — unchanged, seed inserts/updates ROWS not tables). 🔑★ GOLIVE DB VERIFIED THỰC SỰ ÁP PROD (seed chạy lúc boot ⟹ proof = prod Permissions, NOT chỉ binary-shipped): sqlcmd Roles (13 total: Accounting/Admin/AuthorizedSigner/CatalogManager/CostControl/DeptManager/Director/Drafter/Equipment/Finance/HrAdmin/Procurement/ProjectManager). (1) ALLOW-LIST 16/16 R=1∧C=1 cho Drafter (non-admin demo-role) — Off,Off_Dashboard,Off_DanhBa,Off_PhongHop(+View+Book),Off_DeXuat(+List+Create+Inbox),Off_DonTu(+Leave+Ot+Travel),Off_DatXe,Off_ItTicket ✓. (2) CROSS-ROLE: MỌI 13 role count=16 R∧C (golive ÁP TOÀN BỘ không Drafter-fluke) ✓. (3) EXCLUDED-3 (Off_PhongHop_Manage/Off_AttendanceReport/Off_ChamCong) Drafter R=0∧C=0 ✓ + LEAK-check: CHỈ Admin có R=1 trên 3 key này (0 non-admin leak) ✓. (4) HRM/Personal Drafter KHÔNG mở: Hrm_Dashboard=0, all 7 Hrm_Config=0, Personal=0 ✓; Hrm=1 + Hrm_HoSo=1 (S65 public-read GIỮ NGUYÊN không đổi) ✓. (5) Admin KHÔNG hạ: full Office incl 3 excluded all R=1∧C=1 ✓.* ⟹ boot-time seed CHẠY THẬT trên prod (app pool recycled w/ new binary post-deploy; nếu seed CHƯA chạy thì non-admin sẽ vẫn 0 — nhưng 16/16 across 13 roles = đã chạy). Menu-tree non-admin via API NOT tested (DB-query mục 4 đủ proof per spec). 0 regression. NO prod-data mutation ngoài chính golive seed (read-only curls + sqlcmd SELECT-only; seed-write là chủ đích của deploy). Visual menu-render = anh UAT. LESSON: BE-only GOLIVE security-seed verify = both bundles MUST stay FROZEN (no FE → no rotate; rotate=anomaly) + Mig-top + sys.tables MUST stay prev (DbInitializer seed = runtime row insert/update, NOT EF-migration → KHÔNG advance __EFMigrationsHistory NOR sys.tables) + CORE proof = prod Permissions DB query, NOT bundle-frozen alone (frozen chỉ chứng minh FE không đổi, KHÔNG chứng minh API binary mới deploy + seed chạy — phải query Permissions cho non-admin role thấy allow-list grant + cross-role để loại fluke + leak-check excluded chỉ-Admin). upgrade-only seed THẮNG revoke vì chạy SAU trong SeedAsync. golive-state AMBIENT-after-this-deploy (parent commit c556f6c chưa có method này → pre-1f8947e non-admin Office = 0; bằng chứng "đã chạy" = 16/16). TOOLING (re-confirmed S70): Bash=POSIX ($var/$env: mangle through bash→ssh→PS layers) → write PS to /tmp/x.ps1 + powershell.exe -NoProfile -File "C:/.../Temp/x.ps1" (⚠️ heredoc em-dash/non-ASCII corrupts → ASCII-only in PS source); jq ABSENT + system python3 broken → Invoke-RestMethod cho Gitea tasks (match head_sha -eq sha, ?limit=N URI-direct); SSH→PS base64 -EncodedCommand (UTF-16LE [Text.Encoding]::Unicode) cho BOTH appsettings-read AND sqlcmd; sqlcmd pw-literal qua single-quote in B64-payload (no $ survives clean), -U/-P direct + -W -s '|' pipe-delim + -h -1 no-header; read DB pw from prod appsettings when $PROD_DB_PASSWORD empty. NEVER fixed code (READ-only). Tag [s69b, run307, pass, be-only-golive-vanphongso-public-read-create-allowlist16, SeedAllRolesOfficeModulePermissionsAsync-boot-after-revoke, bundle-BOTH-frozen-Wt54PHYl-B99fMU6X, no-mig-top-stays-mig52, tables88, GOLIVE-DB-VERIFIED-16of16-across-13-roles, excluded3-canread0-only-admin-leak0, hrm-personal-still-0, hrm-hoso-public-unchanged, admin-not-downgraded, test292-inferred, deploy11of11, seed-not-migration-no-history-advance, no-regression].

• (S77 #303 sha 6983609, S76 #302, S75/S74 … pre-S78 verbatim → git 764fe70 + archive — FIFO trimmed to keep L1 under soft-cap)

• (S75 Run #301 sha=6df1b2d PASS ~2.5m [FE-both-app PE Link-hồ-sơ auto-detect render, bundle-BOTH-rotate I1fpLeYw/DrQYkzh0, no-mig-mig52, tables88, test286, fastest-streak] → FIFO-trimmed, full verbatim git + archive/2026-06.md)

• (S74 Run #300 sha=91aaf05 PASS [FE-both-app list-redesign flex-row, bundle-BOTH-rotate PxiZQkaw/B36hGoKd, user-unfroze-from-s71-streak, no-mig-mig52, tables88, test286, TOOLING: ps1-file-not-inline-dollar + ssh-encodedcommand-base64] → FIFO-trimmed, full verbatim git + archive/2026-06.md)

• (S67 Run #292 [dept-parentid Mig51-applied-after-#291-fail, retry-chain CS7036-fix, bundle-asymmetric, tree-endpoint-200] + S65 #289 [hrm-hoso public-readonly seed-only, asymmetric read200/write403] → FIFO-trimmed, full verbatim git d2f52ba + archive/2026-06.md)

• Older runs (S66 #290 ← S62 #286 06-13 ← … → S29 #232) full verbatim archived → git d2f52ba + archive/2026-06.md (incl #291 06-16 FAIL forensic [lesson=gotcha #65 + #292-inline] + #383 ex-VITRILAC); pre-S38 → archive/2026-05-{runs,q2,q3,q4}.md.

---

🔄 Curate trigger

• ~30KB → archive recent runs → L2 archive/<period>.md. Dup failure patterns → merge. Stale >3mo → remove.

• Last curate: 2026-06-17 S70 Harness-9 (em-main + Stage-B wf_a58e0d15-beb) (65.2→23.2KB): L2 dark-matter recovery — 10 oldest run-records → archive/2026-06.md (ADDITIVE, original 58378B prefix byte-exact) + archive/_INDEX.md (mục-lục substring sha-keyed, Ctrl-F fallback, no line-hint) + 2026-0{5,6}.gist.md (4-field, distill-gen:1). 0-byte-loss git +13 -0 + sha (Stage C wf_9520d8cd-4fe + em-main self-gate, 2 reviewer no-return). Kept foundation + 2 newest full #308/#307 + stubs.
• Prev curate: 2026-06-16 S66 em main (86.8→29.2KB sed Run #286→#232 incl #291 forensic) · S40 q4.
• Prev curate: 2026-05-29 S40 em main proxy (35.3→~21KB): archived Run #359/#243/#242/#241/#240 + S35/S36 startup → q4 + git d2f52ba; refreshed stale 120→130 test + Mig 34→40. Prev: S34 q3 · S32 q2 · S22 runs.